metadata and JPEG - plans and current status

[Raphaël Quinet]

Since Roman has offered to improve the metadata handling in GIMP, Sven
asked me to document what my plans were for the metadata editor and the
JPEG plug-in.  Well, these plans are best summarized in the mail that I
sent last year:

  http://lists.xcf.berkeley.edu/lists/gimp-developer/2007-November/018992.html
  http://article.gmane.org/gmane.comp.video.gimp.devel/12503/

Basically, every single line of that message is still relevant today.
So I will not quote the whole message again, but I encourage those who
are interested in metadata to read it entirely.

This follow-up message may also be interesting, regarding which Exif
tags should be updated automatically by GIMP and why GIMP should never
ask the user if an image should be rotated automatically or not:

  http://lists.xcf.berkeley.edu/lists/gimp-developer/2007-November/019100.html
  http://article.gmane.org/gmane.comp.video.gimp.devel/12610/

That's about it for the plans.  You should stop reading now if you
are not interested in the details.  :-)


---------- scary details below this line ----------


Maybe a few points from the first message (2.6 roadmap) should be
clarified a bit.  In that message, I wrote:
>  + migrate some parts of the metadata core to a library that can be
>    used by the main app. as well as some plug-ins

Currently, the metadata is stored in an XMP packet inside a GIMP
parasite attached to the image.  While XMP is a good format for
storage in files, it is not ideal for manipulation (updates and
queries).  It would be better it the metadata tree could remain in
memory while the image is open, instead of being converted to/from XMP
all the time.  Also, by having this tree attached to the image and
accessible directly by the core for updates and queries, it would be
possible to have the metadata updated in real-time whenever the image
is modified by some tools or plug-ins.  This would allow some of the
metadata to be displayed in the image info window, or maybe in the
status bar of the image (more and more programs are doing that).

Currently, the metadata tree is only available inside the metadata
editor, which is implemented as a plug-in.  This results in several
round-trips via the PDB whenever something must be updated.  For
example, assuming that Exif would now be handled correctly, the
current implementation would lead to the following scenario when a
JPEG image is saved:

- the JPEG plug-in calls "plug-in-metadata-set" to update some metadata
- this is passed to GIMP
- GIMP starts the metadata editor plug-in
- the metadata editor asks GIMP to get the parasite attached to the image
- the parasite (XMP) is parsed and stored in a tree
- the tree is updated according to what was requested
- the tree is encoded into a parasite (XMP)
- the metadata editor asks GIMP to attach the parasite to the image,
  then it exits
- (if there is some window currently displaying the metadata associated
  with that image, it is *not* updated)
- the JPEG plug-in is informed that the operation was successful
- the JPEG plug-in calls "plug-in-metadata-encode-exif" in order to get
  an Exif block from the updated metadata
- this is passed to GIMP
- GIMP starts the metadata editor plug-in again
- the metadata editor asks GIMP to get the parasite attached to the image
- the parasite (XMP) is parsed and stored in a tree
- the metadata editor updates some metadata that must be updated when an
  image is saved
- the updated tree is encoded into a parasite (XMP)
- the metadata editor asks GIMP to attach the updated parasite
- (if there is some window currently displaying the metadata associated
  with that image, it is *not* updated)
- the contents of the tree are encoded in Exif
- the metadata editor returns an array with the Exif block and exits
- GIMP returns the array to the JPEG plug-in
- the JPEG plug-in can now save the file with the Exif block, then exit
- (if there is some window currently displaying the metadata associated
  with that image, the user must remember to refresh the view)

And here is what would happen if the metadata tree could be managed by
the GIMP core, but we still assume that the conversion to/from Exif is
done by an external plug-in:

- the JPEG plug-in calls "gimp-metadata-set" to update some metadata
- this is passed to GIMP, which updates the metadata attached to the
  image
- (if there is some window currently displaying the metadata associated
  with that image, it *will be* updated)
- the JPEG plug-in is informed that the operation was successful
- the JPEG plug-in calls "gimp-metadata-encode-exif" in order to get an
  Exif block from the updated metadata
- GIMP updates some metadata that must be updated when an image is saved
- (if there is some window currently displaying the metadata associated
  with that image, it *will be* updated)
- GIMP starts the metadata editor plug-in
- the metadata editor makes several calls to "gimp-metadata-get" in order
  to retrieve the parts of the metadata that are relevant for Exif
- GIMP returns the requested data (from its internal tree)
- the metadata editor encodes that information in an Exif block
- the metadata editor returns an array with the Exif block and exits
- GIMP returns the array to the JPEG plug-in
- the JPEG plug-in can now save the file with the Exif block, then exit

Furthermore, here is what would happen if the metadata tree could be
managed by the GIMP core and some common metadata operations (such as
the conversion to/from Exif) would be in a "libgimp-metadata" library
linked directly with the file plug-ins that need it:

- the JPEG plug-in calls "gimp-metadata-set" to update some metadata
- this is passed to GIMP, which updates the metadata attached to the
  image
- (if there is some window currently displaying the metadata associated
  with that image, it *will be* updated)
- the JPEG plug-in is informed that the operation was successful
- the JPEG plug-in calls "gimp-update-metadata-for-save" in order tell
  GIMP that the image is about to be saved
- GIMP updates some metadata that must be updated when an image is saved
- (if there is some window currently displaying the metadata associated
  with that image, it *will be* updated)
- the JPEG plug-in makes several calls to "gimp-metadata-get" in order to
  retrieve the parts of the updated metadata that are relevant for Exif
- the JPEG plug-in calls the library functions for encoding that metadata
  into an Exif block (the previous step could even be performed by the
  library functions so that the JPEG plug-in would not have to care).
- the JPEG plug-in can now save the file with the Exif block, then exit

Sorry for the long details, but using an example like this is probably
the best way to explain why it would make sense to have some parts of
the metadata handling performed by a library, and why it would be
better if the metadata tree could be managed by the GIMP core instead
of being constantly converted to/from XMP and stored in a parasite.


In my message from last year, I also wrote:
>  + convert EXIF to XMP
>  + convert XMP to EXIF

The initial goal of the code that I started writing (and never finished)
for the conversion to/from Exif was to get rid of the dependency on
libexif.  That library had caused several severe stability problems in
the past and most developers (including myself) wanted to replace it by
some code that was more stable.

However, the known bugs in libexif have now been fixed (some updates have
not been released yet, but that's another issue) and after studying how
some people (and the programs they use) are using the Exif data, I am now
convinced that preserving the Exif MakerNotes is very important.  This is
the trickiest part of Exif and this is where most of the time was spent
in libexif or other libraries (exiv2, etc.).  Since parsing and updating
MakerNotes is a difficult task, I think that it is better to leave that
job to libexif instead of re-inventing the wheel.  So contrary to what I
originally planned, I think that the code for converting to/from Exif
should be based on that library.  This will also make our code smaller
and easier to maintain.

-Raphaël
_______________________________________________
Gimp-developer mailing list
Gimp-developer@...
https://lists.XCF.Berkeley.EDU/mailman/listinfo/gimp-developer

[Roman Joost]

Hi Raphaël,

On Tue, Jul 22, 2008 at 04:50:41PM +0200, Raphaël Quinet wrote:

> Since Roman has offered to improve the metadata handling in GIMP, Sven
> asked me to document what my plans were for the metadata editor and the
> JPEG plug-in.  Well, these plans are best summarized in the mail that I
> sent last year:
>
> [...] plans
>
> Maybe a few points from the first message (2.6 roadmap) should be
> clarified a bit.  In that message, I wrote:
> [...]

Thanks for clearing this up a bit. This makes the metadata browser far

more complex than I intentionally thought.

> [...]
>
> Currently, the metadata tree is only available inside the metadata
> editor, which is implemented as a plug-in.  This results in several
> round-trips via the PDB whenever something must be updated.  For
> example, assuming that Exif would now be handled correctly, the
> current implementation would lead to the following scenario when a
> JPEG image is saved:
>
> [...] tremendous amount of calls
>
> And here is what would happen if the metadata tree could be managed by
> the GIMP core, but we still assume that the conversion to/from Exif is
> done by an external plug-in:
>
> [...] less tremendous amount of calls
>
> Furthermore, here is what would happen if the metadata tree could be
> managed by the GIMP core and some common metadata operations (such as
> the conversion to/from Exif) would be in a "libgimp-metadata" library
> linked directly with the file plug-ins that need it:
>
> [...] less, less amount of calls
>
> Sorry for the long details, but using an example like this is probably
> the best way to explain why it would make sense to have some parts of
> the metadata handling performed by a library, and why it would be
> better if the metadata tree could be managed by the GIMP core instead
> of being constantly converted to/from XMP and stored in a parasite.

In my unexperienced point of view, it looks like this should be

clarified first, before I try to improve the current metadata editor.
IMHO your last point seems to be the way to go, all other attempts like
the first one seems to be totally inefficient and error-prune.

> In my message from last year, I also wrote:
> >  + convert EXIF to XMP
> >  + convert XMP to EXIF
>
> The initial goal of the code that I started writing (and never finished)
> for the conversion to/from Exif was to get rid of the dependency on
> libexif.  That library had caused several severe stability problems in
> the past and most developers (including myself) wanted to replace it by
> some code that was more stable.
>
> [...]
> So contrary to what I originally planned, I think that the code for
> converting to/from Exif should be based on that library.  This will
> also make our code smaller and easier to maintain.

My current plan was to provide a read-only view of the currently stored

metadata inkl. a button to update the view. This included the conversion
from Exif to XMP. Currently from my point of view - correct me if I'm
wrong - the conversation will be called from the imagefile plugin (e.g.
jpeg-load), after the Exif data is attached as a parasite. The metadata
exif-decode procedure will convert the Exif data to XMP than, to be able
to display it as XMP data in the metadata browser.

Now after the pros and cons about handling metadata is clarified by your
mail, how different do we set our priorities to move forward for this
editor and the metadata management? I think we should setup a roadmap
first, setup tasks afterwards and start implementing them.

Cheers,
--
Roman Joost
www: http://www.romanofski.de
email: romanofski@...


_______________________________________________
Gimp-developer mailing list
Gimp-developer@...
https://lists.XCF.Berkeley.EDU/mailman/listinfo/gimp-developer

[Raphaël Quinet]

On Wed, 23 Jul 2008 11:20:41 +0200, Roman Joost <romanofski@...> wrote:
> On Tue, Jul 22, 2008 at 04:50:41PM +0200, Raphaël Quinet wrote:
> > [...]
> > Currently, the metadata tree is only available inside the metadata
> > editor, which is implemented as a plug-in.  This results in several
> > round-trips via the PDB whenever something must be updated.
[...]
> In my unexperienced point of view, it looks like this should be
> clarified first, before I try to improve the current metadata editor.
> IMHO your last point seems to be the way to go, all other attempts like
> the first one seems to be totally inefficient and error-prune.

Yes, this is also my point of view.  The best way to go is to have the
metadata tree available in the core with some PDB calls allowing plug-ins
to get/set data, and also have a "libgimp-metadata" library that can be
used by the file plug-ins for performing tasks such as converting between
Exif and XMP.  The editor itself could still be a plug-in, but the
lower-level parts of metadata handling would be included in the library
and another part of it would be in the GIMP core.

To recap, the three scenarios are:
1) metadata editor/viewer + metadata format converter + storage are all
   done by a plug-in (current situation);
2) metadata editor/viewer + metadata format converter done by a plug-in,
   but storage handled in GIMP core;
3) metadata editor/viewer done by a plug-in, metadata format conversions
   in a GIMP metadata library, and storage handled in GIMP core.
In the longer term, we can also think about a scenario 4 in which parts
of the metadata viewer/editor could be handled by the core, but that is
not for the short term anyway.

Although I think that it makes a lot of sense to have the metadata
tree inside the core as in scenario 3, Sven seems to disagree so I
would like to have his opinion before deciding how to proceed.

> > So contrary to what I originally planned, I think that the code for
> > converting to/from Exif should be based on that library.  This will
> > also make our code smaller and easier to maintain.
> My current plan was to provide a read-only view of the currently stored
> metadata inkl. a button to update the view. This included the conversion
> from Exif to XMP. Currently from my point of view - correct me if I'm
> wrong - the conversation will be called from the imagefile plugin (e.g.
> jpeg-load), after the Exif data is attached as a parasite. The metadata
> exif-decode procedure will convert the Exif data to XMP than, to be able
> to display it as XMP data in the metadata browser.

I think that you got it right. :-)  In scenario 1 above, when a file is
loaded the Exif block should be converted to XMP immediately (or more
precisely, it should be converted to a metadata tree, which happens to be
serialized as XMP) and then the metadata parasite is attached to the image.
Later, when the user starts the metadata editor or viewer, the parasite is
retrieved from the image and decoded into a metadata tree that can be
displayed to the user.

Note that in scenarios 2 and 3, it would not be necessary to convert the
metadata tree into XMP and store it in a parasite, because that tree would
be managed by the core.  The metadata viewer would only have to retrieve
the relevant elements for display or editing.  In fact, in scenarios 2 and
3 the metadata viewer/editor would not have to care about the XMP syntax:
it would only need to know the name and data type of the things that it
wants to display.

> Now after the pros and cons about handling metadata is clarified by your
> mail, how different do we set our priorities to move forward for this
> editor and the metadata management? I think we should setup a roadmap
> first, setup tasks afterwards and start implementing them.

Well, the thing that could change the plans significantly is a decision
about whether or not we integrate the metadata tree inside the core.  And
I would like to get Sven's comments about that.

If we start with scenario 2 (for example), then it should not be too hard
to switch to scenario 3 later.  However, if we start with 1, then it is
not easy to migrate to 2 or 3 later.  Most of the new code would have to
be rewritten differently.

-Raphaël
_______________________________________________
Gimp-developer mailing list
Gimp-developer@...
https://lists.XCF.Berkeley.EDU/mailman/listinfo/gimp-developer