You are viewing a plain text version of this content. The canonical link for it is here.
Posted to fop-dev@xmlgraphics.apache.org by Max Berger <ma...@berger.name> on 2006/06/19 16:52:25 UTC
Thoughts about image handling
Dear Fop developers,
after a while and some thinking, here is my concept for extensible
image handlers for fop, or even better for xmlgraphics. If desired, I
can implement this concept for xmlgraphics / fop with support for
imageio, jimi and batik.
Image handling is a three-step process. Step one is detecting the
file format, step two is loading the image, step three is outputting
the image into whatever the output renderer is.
Step 1: Detecting the file format.
In most handlers this currently happens by trying to load the files,
which is very inefficient. Instead, detectors like jmimemagic should
be used. These should return a mime-type or null if the type can not
be detected.
To speed up the process, a mime-type may be guessed from the file
extension, and added as "hint"
Example interface:
public interface MimeTypeDetector {
String detectMimeType(URL file, String probableType) throws
IOException;
}
Step 2: Loading the image.
The image can then be loaded by any image handler that supports this
type.
Example:
public interface ImageHandler {
void setImage(URL file) throws IOException;
// ...
}
Step 3: Outputting the image.
Generally there are three types of images. Vector images (SVG, MML),
bitmap images (GIF, BMP, JPEG), and uninterpreted images (EPS)
- Vector images must supply a paint(Graphics g) function
- Bitmap images must supply a method to get the image contents as a
Raster (similar to java.awt.RenderedImage)
- Bitmap images may supply a method to get the image contents in LZW,
ZLIB, DCT, format, such as in JPEG or TIFF compression (used in PDF)
- Uninterpreted images just provide a method to get the contents in
its original format.
Reflection may be used in the renderer to find out the image type,
rather than checking for the type. Example:
if (image instanceOf VectorImage) {
((VectorImage)image).paint(graphics)
} else if (image instanceOf DCTEncodedImage) {
addResource(image.getDCTData)
// ...
To support extensibility, a registration mechanism is provided. Here
is the basic idea:
Java provides standard mechanisms to find all resources with a given
name in all classpath items. This allows to find all META-INF/
MANIFEST.MF files given in all JAR files in the classpath (1). These
files can be parsed using standard Manifest functionality.
The files contain some attributes that describe classes used. For
image handlers, this could be a classname and the supported image
type. It may contain additional attributes, such as supported
subtypes (e.g. LZW for TIFF). Ideally the exact specification of
these attributes would be coordinated between fop and foray to
support reuse.
This information can be parsed once and stored.
This mechanism requires the user to change only the classpath, and
nothing else.
I have written a short proof-of concept code for the registration,
available at
http://max.berger.name/tmp/extTestMain.jar
and
http://max.berger.name/tmp/extTestProvider.jar
(source is included in the jar files).
To run, try:
java -cp extTestMain.jar name.berger.max.test.ext.Main
or
java -cp extTestMain.jar:extTestProvider.jar
name.berger.max.test.ext.Main
(1) Of course, it doesn't have to be MANIFEST.MF. For resources such
as fonts this may well be META-INF/FontDescriptor.xml or something else.
questions? comments?
Max Berger
e-mail: max@berger.name
--
PGP/GnuPG ID: E81592BC Print: F489F8759D4132923EC4
BC7E072AB73AE81592BC
For information about me or my projects please see http://
max.berger.name
Re: Thoughts about image handling
Posted by Jeremias Maerki <de...@jeremias-maerki.ch>.
On 22.06.2006 17:09:20 Max Berger wrote:
> Jeremias,
>
> > Actually, you left out the pre-loading of the image size. That's
> > important if you want to delay image loading until the rendering stage
> > (or avoid it altogether). Note that not in all cases will it be
> > necessary to load an image. Sometimes only references to the images
> > are
> > put in the output format. Currently, this only applies to RTF but
> > could
> > actually be used in PostScript and maybe AFP output. Special languages
> > such as PPML even go so far an make it their prime purpose not
> > having to
> > handle all the image data but providing them on the targte platform.
>
> Good point. This requires parsing the header without actually parsing
> the whole imagedata. I know that ImageIO provides this capability,
> but I am not sure about JIMI and other solutions.
>
> AAMOF the ImageIO interface provides the exact capabilities required:
> Registering of Image Readers, providing support for setting the image
> data source, then getting all the meta-info without decoding the
> actual image. Unfortunately ImageIO does not seem to support getting
> the "raw" image data, but that functionality is available in fop's
> JpegImage.
Remember that FOP needs to maintain compatibility to JDK 1.3 so entirely
relying on ImageIO is currently not an option. But we already have
classes which handle the pre-loading for the most important image
formats. See org.apache.fop.images.analyser.
> So there are more steps:
> - detect file format (may already read metadata and image if needed)
> - read metadata (may read image if needed)
> - read image data OR get raw image data.
That's actually what we're already doing today. :-)
> > Going towards Raster or RenderedImage for the in-memory representation
> > of the image is certainly a very welcome step.
> >
> > What I'm missing a little is that certain images will be converted
> > before they are processed by the renderer. For example, Barcode4J
> > converts its barcodes to SVG, EPS or Java2D graphics depending on the
> > output format in use. Generally, each renderer will have different
> > preferences how an image will be processed.
>
> That is exactly the problem I see in the current renderers:
>
> To render MathML or plan (in the examples) to pdf they are first
> rendered as SVG and then the SVG is rendered via Java2D into the PDF.
> I don't know how barcode4j does it, but I would assume it is similar.
Actually, Barcode4J is much more advanced. It supports native EPS
construction and painting using Graphics2D. Working today. It's only a
matter of writing an XMLHandler implementation for MathML to do the same.
No witchcraft involved.
>
> However, every one of the renderers (or at least the awt, ps, and pdf
> renderers) support a Java2D compatibility interface, which is
> currently used for SVG images.
...through XMLHandler and AbstractGraphics2DAdapter
> What I propose is:
> - If there is a Java2D interface, offer that directly to all vector
> image providers.
we have that today
> - If there is no Java2D interface (such as in the rtf output) then
> render the vector image into a bitmap image (awt has standard support
> for that, however I do not know if it works headless in 1.3) and use
> that.
See
> > While PDF can embed TIFF
> > CCITT4 files directly, they have to be decoded for PCL. The ideal
> > image
> > subsystem will also cache a preconverted image so the
> > conversion/decoding can be avoided next time the image is used.
>
> Ok. Here's an idea: Use a Map ImageURL-> ImageInfo, where ImageInfo
> contains mime-type, metadata, raw content and decoded content (if
> possible). Every one of them may be null, and will be loaded on
> demand. For the actual imagedata, a SoftReference may be used.
That's not any different from today except that you imply support for
loading raw AND decoded data. I'd actually decouple the actual flavor
(raw or decoded or whatever) from the actual image info. That way the
large data blobs are reclaimable individually.
> > I don't think that'll work considering the above. I rather think the
> > Renderer will have to tell the image subsystem the preferred flavor of
> > the image. It will then receive the image in the right form if that is
> > possible.
>
> How about this (for bitmaps):
>
> An image has a "native" format, which describes the raw stream if
> possible. Typical values are EPS, DCT, RLE, LZW, CCITT, and so on.
> The renderer can then check if it has support for this raw data type.
> If so, it will use it. If not, it will have to use the decoded Raster
> data.
Again, I don't see the difference to today.
> We can provide additional compressors which will take raster data and
> provide raw data in one of the lossless formats (gzip, rle). They can
> be used by the renderer to reduce file size.
Today, this is handled directly by the renderer. Don't see any need to
change it. These compressors are usually simple OutputStream
implementations which are already available.
>
> >
> >>
> >> To support extensibility, a registration mechanism is provided. Here
> >> is the basic idea:
> >>
> >> Java provides standard mechanisms to find all resources with a given
> >> name in all classpath items. This allows to find all META-INF/
> >> MANIFEST.MF files given in all JAR files in the classpath (1). These
> >> files can be parsed using standard Manifest functionality.
> >>
> >> The files contain some attributes that describe classes used. For
> >> image handlers, this could be a classname and the supported image
> >> type. It may contain additional attributes, such as supported
> >> subtypes (e.g. LZW for TIFF). Ideally the exact specification of
> >> these attributes would be coordinated between fop and foray to
> >> support reuse.
> >>
> >> This information can be parsed once and stored.
> >>
> >> This mechanism requires the user to change only the classpath, and
> >> nothing else.
> >
> > Ok, something like that sounds pretty good. Remains to be seen whether
> > the config needs to be in a file or rather in a factory class like
> > we've
> > done it before (example: AbstractRendererMaker). The only thing left
> > might be the question how to handle priorities if two implementations
> > support the same kind of image.
> > What you describe here is already in use in FOP and Batik. We don't
> > use
> > the MANIFEST.MF directly but the class name of the provider
> > class/interface. See [1] and [2]. Let's reuse what we already have.
> > [1] http://svn.apache.org/viewvc/xmlgraphics/fop/trunk/src/java/
> > META-INF/services/
> > [2] http://svn.apache.org/viewvc/xmlgraphics/commons/trunk/src/java/
> > org/apache/xmlgraphics/util/Service.java?view=log
> >
>
> Ok. This solution is much better than the one that I had in mind.
> Especially since it is way more generic and already tested.
>
> Since it does not provide support for additional meta information,
> all of this must be extracted from factory code, which is feasable.
> In the case of ImageIO it is actually better to extract capabilities
> in code: then ImageIO can be asked on the local jdk which image types
> are actually supported.
>
> to check which implementation to prefer, two things must be considered:
>
> - capabilities: ImageIO JPEG can read header data without reading the
> image data, whereas the existing JpegImage provides support for the
> RAW stream, but always reads the whole image. This may be a question
> speed vs. size that can only be answered by a user configuration
> file. Some Image providers may not support all subtypes, such as LZW
> compressed tiff files.
Reading the minimal metadata required to place the image is handled
already. I don't think it's necessary to do anything for this.
> - speed of the implementation: this is a good question, which one is
> faster: ImageIO or JIMI ?
JIMI is Java-only, JAI is Java plus optionally native code which is much
faster. ImageIO can use JAI's codecs. So ImageIO is certainly faster
than JIMI. BTW, JIMI and JAI are no longer interesting if you have
ImageIO available. JIMI and JAI are only interesting for people on JDK
1.3.
> - It may (or may not) make sense to use different implementations in
> different steps of the process or in different renderers. The
> disadvantage is that the URL will have to be re-opened. It may (or
> may not) make sense to use different implementations depending on the
> origin of the image. (Local images are RandomAccessFiles, while
> remote images are usually forward-only files).
The only thing I see is that for PDFRenderer you'd use FOP's native
reader for raw JPEG and for PCLRenderer you'd use some image library
like ImageIO to decode the JPEG image. That's one of the reasons why I'd
split up the basic image info (your term: metadata) and actual flavors
(raw, decoded, DOM...).
>
> >> questions? comments?
> >
> > Have you seen this Wiki page? http://wiki.apache.org/xmlgraphics-
> > fop/ImageSupport
>
> I've just looked at it, addresses pretty much the same issues covered
> in this mail.
>
> >
> > I'm happy to see that you volunteer to work in this area. It's
> > something
> > I wanted to fix for a long time now but it always had a lower
> > priority. I
> > envisioned a slightly different direction as you can guess from my
> > comments but this is still open for discussion.
>
> I'd rather discuss and bounce ideas back and forth first than writing
> tons of code that needs to be revised. This is not my main project
> either, but it is good to have something to divert myself from time
> to time :)
Good. I think you can reduce the whole thing to the following (it's what
I'd do):
- Split the basic image info from the actualy image data and allow for
different flavors of the same image being present (and in the cache).
1:n dependency from image info to image flavor.
- Allow renderers to request a particular flavor of an image.
This is basically it for the moment from me. It's a little difficult to
concentrate in the hackathon room here at ApacheCon with about 40 people
present. I guess I cannot dive into it more deeply before July 12.
Jeremias Maerki
Re: Thoughts about image handling
Posted by Max Berger <ma...@berger.name>.
Jeremias,
> Actually, you left out the pre-loading of the image size. That's
> important if you want to delay image loading until the rendering stage
> (or avoid it altogether). Note that not in all cases will it be
> necessary to load an image. Sometimes only references to the images
> are
> put in the output format. Currently, this only applies to RTF but
> could
> actually be used in PostScript and maybe AFP output. Special languages
> such as PPML even go so far an make it their prime purpose not
> having to
> handle all the image data but providing them on the targte platform.
Good point. This requires parsing the header without actually parsing
the whole imagedata. I know that ImageIO provides this capability,
but I am not sure about JIMI and other solutions.
AAMOF the ImageIO interface provides the exact capabilities required:
Registering of Image Readers, providing support for setting the image
data source, then getting all the meta-info without decoding the
actual image. Unfortunately ImageIO does not seem to support getting
the "raw" image data, but that functionality is available in fop's
JpegImage.
So there are more steps:
- detect file format (may already read metadata and image if needed)
- read metadata (may read image if needed)
- read image data OR get raw image data.
> Going towards Raster or RenderedImage for the in-memory representation
> of the image is certainly a very welcome step.
>
> What I'm missing a little is that certain images will be converted
> before they are processed by the renderer. For example, Barcode4J
> converts its barcodes to SVG, EPS or Java2D graphics depending on the
> output format in use. Generally, each renderer will have different
> preferences how an image will be processed.
That is exactly the problem I see in the current renderers:
To render MathML or plan (in the examples) to pdf they are first
rendered as SVG and then the SVG is rendered via Java2D into the PDF.
I don't know how barcode4j does it, but I would assume it is similar.
However, every one of the renderers (or at least the awt, ps, and pdf
renderers) support a Java2D compatibility interface, which is
currently used for SVG images.
What I propose is:
- If there is a Java2D interface, offer that directly to all vector
image providers.
- If there is no Java2D interface (such as in the rtf output) then
render the vector image into a bitmap image (awt has standard support
for that, however I do not know if it works headless in 1.3) and use
that.
> While PDF can embed TIFF
> CCITT4 files directly, they have to be decoded for PCL. The ideal
> image
> subsystem will also cache a preconverted image so the
> conversion/decoding can be avoided next time the image is used.
Ok. Here's an idea: Use a Map ImageURL-> ImageInfo, where ImageInfo
contains mime-type, metadata, raw content and decoded content (if
possible). Every one of them may be null, and will be loaded on
demand. For the actual imagedata, a SoftReference may be used.
> I don't think that'll work considering the above. I rather think the
> Renderer will have to tell the image subsystem the preferred flavor of
> the image. It will then receive the image in the right form if that is
> possible.
How about this (for bitmaps):
An image has a "native" format, which describes the raw stream if
possible. Typical values are EPS, DCT, RLE, LZW, CCITT, and so on.
The renderer can then check if it has support for this raw data type.
If so, it will use it. If not, it will have to use the decoded Raster
data.
We can provide additional compressors which will take raster data and
provide raw data in one of the lossless formats (gzip, rle). They can
be used by the renderer to reduce file size.
>
>>
>> To support extensibility, a registration mechanism is provided. Here
>> is the basic idea:
>>
>> Java provides standard mechanisms to find all resources with a given
>> name in all classpath items. This allows to find all META-INF/
>> MANIFEST.MF files given in all JAR files in the classpath (1). These
>> files can be parsed using standard Manifest functionality.
>>
>> The files contain some attributes that describe classes used. For
>> image handlers, this could be a classname and the supported image
>> type. It may contain additional attributes, such as supported
>> subtypes (e.g. LZW for TIFF). Ideally the exact specification of
>> these attributes would be coordinated between fop and foray to
>> support reuse.
>>
>> This information can be parsed once and stored.
>>
>> This mechanism requires the user to change only the classpath, and
>> nothing else.
>
> Ok, something like that sounds pretty good. Remains to be seen whether
> the config needs to be in a file or rather in a factory class like
> we've
> done it before (example: AbstractRendererMaker). The only thing left
> might be the question how to handle priorities if two implementations
> support the same kind of image.
> What you describe here is already in use in FOP and Batik. We don't
> use
> the MANIFEST.MF directly but the class name of the provider
> class/interface. See [1] and [2]. Let's reuse what we already have.
> [1] http://svn.apache.org/viewvc/xmlgraphics/fop/trunk/src/java/
> META-INF/services/
> [2] http://svn.apache.org/viewvc/xmlgraphics/commons/trunk/src/java/
> org/apache/xmlgraphics/util/Service.java?view=log
>
Ok. This solution is much better than the one that I had in mind.
Especially since it is way more generic and already tested.
Since it does not provide support for additional meta information,
all of this must be extracted from factory code, which is feasable.
In the case of ImageIO it is actually better to extract capabilities
in code: then ImageIO can be asked on the local jdk which image types
are actually supported.
to check which implementation to prefer, two things must be considered:
- capabilities: ImageIO JPEG can read header data without reading the
image data, whereas the existing JpegImage provides support for the
RAW stream, but always reads the whole image. This may be a question
speed vs. size that can only be answered by a user configuration
file. Some Image providers may not support all subtypes, such as LZW
compressed tiff files.
- speed of the implementation: this is a good question, which one is
faster: ImageIO or JIMI ?
- It may (or may not) make sense to use different implementations in
different steps of the process or in different renderers. The
disadvantage is that the URL will have to be re-opened. It may (or
may not) make sense to use different implementations depending on the
origin of the image. (Local images are RandomAccessFiles, while
remote images are usually forward-only files).
>> questions? comments?
>
> Have you seen this Wiki page? http://wiki.apache.org/xmlgraphics-
> fop/ImageSupport
I've just looked at it, addresses pretty much the same issues covered
in this mail.
>
> I'm happy to see that you volunteer to work in this area. It's
> something
> I wanted to fix for a long time now but it always had a lower
> priority. I
> envisioned a slightly different direction as you can guess from my
> comments but this is still open for discussion.
I'd rather discuss and bounce ideas back and forth first than writing
tons of code that needs to be revised. This is not my main project
either, but it is good to have something to divert myself from time
to time :)
> Jeremias Maerki
>
Max Berger
Re: Thoughts about image handling
Posted by Jeremias Maerki <de...@jeremias-maerki.ch>.
On 19.06.2006 16:52:25 Max Berger wrote:
> Dear Fop developers,
>
> after a while and some thinking, here is my concept for extensible
> image handlers for fop, or even better for xmlgraphics. If desired, I
> can implement this concept for xmlgraphics / fop with support for
> imageio, jimi and batik.
>
> Image handling is a three-step process. Step one is detecting the
> file format, step two is loading the image, step three is outputting
> the image into whatever the output renderer is.
Actually, you left out the pre-loading of the image size. That's
important if you want to delay image loading until the rendering stage
(or avoid it altogether). Note that not in all cases will it be
necessary to load an image. Sometimes only references to the images are
put in the output format. Currently, this only applies to RTF but could
actually be used in PostScript and maybe AFP output. Special languages
such as PPML even go so far an make it their prime purpose not having to
handle all the image data but providing them on the targte platform.
>
> Step 1: Detecting the file format.
>
> In most handlers this currently happens by trying to load the files,
> which is very inefficient. Instead, detectors like jmimemagic should
> be used. These should return a mime-type or null if the type can not
> be detected.
>
> To speed up the process, a mime-type may be guessed from the file
> extension, and added as "hint"
>
> Example interface:
>
> public interface MimeTypeDetector {
> String detectMimeType(URL file, String probableType) throws
> IOException;
> }
Ok, so far I don't see any advantage over the current code.
> Step 2: Loading the image.
>
> The image can then be loaded by any image handler that supports this
> type.
>
> Example:
>
> public interface ImageHandler {
> void setImage(URL file) throws IOException;
> // ...
> }
In your scenario, when would the image actually be loaded?
> Step 3: Outputting the image.
>
> Generally there are three types of images. Vector images (SVG, MML),
> bitmap images (GIF, BMP, JPEG), and uninterpreted images (EPS)
>
> - Vector images must supply a paint(Graphics g) function
> - Bitmap images must supply a method to get the image contents as a
> Raster (similar to java.awt.RenderedImage)
> - Bitmap images may supply a method to get the image contents in LZW,
> ZLIB, DCT, format, such as in JPEG or TIFF compression (used in PDF)
> - Uninterpreted images just provide a method to get the contents in
> its original format.
Going towards Raster or RenderedImage for the in-memory representation
of the image is certainly a very welcome step.
What I'm missing a little is that certain images will be converted
before they are processed by the renderer. For example, Barcode4J
converts its barcodes to SVG, EPS or Java2D graphics depending on the
output format in use. Generally, each renderer will have different
preferences how an image will be processed. While PDF can embed TIFF
CCITT4 files directly, they have to be decoded for PCL. The ideal image
subsystem will also cache a preconverted image so the
conversion/decoding can be avoided next time the image is used.
> Reflection may be used in the renderer to find out the image type,
> rather than checking for the type. Example:
>
> if (image instanceOf VectorImage) {
> ((VectorImage)image).paint(graphics)
> } else if (image instanceOf DCTEncodedImage) {
> addResource(image.getDCTData)
> // ...
I don't think that'll work considering the above. I rather think the
Renderer will have to tell the image subsystem the preferred flavor of
the image. It will then receive the image in the right form if that is
possible.
>
> To support extensibility, a registration mechanism is provided. Here
> is the basic idea:
>
> Java provides standard mechanisms to find all resources with a given
> name in all classpath items. This allows to find all META-INF/
> MANIFEST.MF files given in all JAR files in the classpath (1). These
> files can be parsed using standard Manifest functionality.
>
> The files contain some attributes that describe classes used. For
> image handlers, this could be a classname and the supported image
> type. It may contain additional attributes, such as supported
> subtypes (e.g. LZW for TIFF). Ideally the exact specification of
> these attributes would be coordinated between fop and foray to
> support reuse.
>
> This information can be parsed once and stored.
>
> This mechanism requires the user to change only the classpath, and
> nothing else.
Ok, something like that sounds pretty good. Remains to be seen whether
the config needs to be in a file or rather in a factory class like we've
done it before (example: AbstractRendererMaker). The only thing left
might be the question how to handle priorities if two implementations
support the same kind of image.
> I have written a short proof-of concept code for the registration,
> available at
> http://max.berger.name/tmp/extTestMain.jar
> and
> http://max.berger.name/tmp/extTestProvider.jar
>
> (source is included in the jar files).
>
> To run, try:
> java -cp extTestMain.jar name.berger.max.test.ext.Main
> or
> java -cp extTestMain.jar:extTestProvider.jar
> name.berger.max.test.ext.Main
>
> (1) Of course, it doesn't have to be MANIFEST.MF. For resources such
> as fonts this may well be META-INF/FontDescriptor.xml or something else.
What you describe here is already in use in FOP and Batik. We don't use
the MANIFEST.MF directly but the class name of the provider
class/interface. See [1] and [2]. Let's reuse what we already have.
[1] http://svn.apache.org/viewvc/xmlgraphics/fop/trunk/src/java/META-INF/services/
[2] http://svn.apache.org/viewvc/xmlgraphics/commons/trunk/src/java/org/apache/xmlgraphics/util/Service.java?view=log
> questions? comments?
Have you seen this Wiki page? http://wiki.apache.org/xmlgraphics-fop/ImageSupport
It's there for gathering all the requirement on the image library. I'll
bring it up-to-date in a minute. I saw there are a few things I need to
change.
I'm happy to see that you volunteer to work in this area. It's something
I wanted to fix for a long time now but it always had a lower priority. I
envisioned a slightly different direction as you can guess from my
comments but this is still open for discussion.
Jeremias Maerki