You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@commons.apache.org by Stefan Bodewig <bo...@apache.org> on 2011/09/04 08:04:15 UTC
[compress] Pack200
Hi,
I've just committed Converter*Stream implementations for Pack200[1]
which is a bit unusual in several ways.
First of all it will (by design of the format) only work on compressing
valid jar files. Actually the result isn't likely to be compressed (in
the sense of "smaller than the original") at all but expects another
step of GZip compression in most cases.
The second difference to the other compressors is that the API provided
by the Java classlib doesn't lend itself to streaming at all. There is
a Packer/Unpacker that expects an InputStream and an OutputStream and
converts from one to the other in a single blocking operation (even
closing the input side when done).
I have experimented with Piped*Streams as well as Ant/commons-exec-like
stream pumping in order to provide a streaming experience but always ran
into some edge cases where things broke down. I'll give one example
below.
The current implementation of Pack200CompressorInputStream will
pass the wrapped input and an OutputStream writing to a cache to the
Unpacker synchronously inside the constructor, consuming the input
completely. It will then defer all read-operations to the cache.
Likewise the Pack200CompressorOutputStream will buffer up all write
operations in a cache and once finish() or close() is called the cache
is converted to an InputStream that is then passed together with the
originally wrapped output to the Packer and written synchronously.
Caches can be in-memory (using ByteArray*Stream) or temporary files
controlled by a constructor option with in-memory as the default and
temp-files for cases where the archives are expected to be big.
Because of this design the byte-count methods don't make any sense (do
we count when data is written-to/read-from the cache or while the
(Un)Packer is doing its work?) and haven't been implemented at all.
The class names StreamMode and StreamSwitcher result from my attempts of
using real streams and should be changed unless anybody else comes up
with a working streaming solution.
The biggest hurdle for any streaming solution is that there is always
going to be some sort of intermediate buffer. Something picks up data
written to the output stream and makes it available to the input stream
side. Once the buffer is full, nothing can be written unless anybody
reads input in a timely manner.
In the case of a Pack200CompressorInputStream you don't have any control
over when the user code is going to read the data and whether it is
going to consume all of it at all. For example if the stream is wrapped
in a ZipArchiveInputStream (it represents a JAR, after all), it is never
going to be consumed completely because the archive contains ZIP data at
the end that is ignored by the input stream implementation.
There are more cases where the Pack/Unpack operation would end up
blocked so I decided to only code the more robust indirect solution for
now.
Stefan
[1] http://download.oracle.com/javase/1.5.0/docs/api/java/util/jar/Pack200.html
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
For additional commands, e-mail: dev-help@commons.apache.org
Re: [compress] Pack200
Posted by Stefan Bodewig <bo...@apache.org>.
On 2011-09-15, Emmanuel Bourg wrote:
> Le 15/09/2011 11:46, Stefan Bodewig a écrit :
>> BTW, I've added pack200 support to the Compress Antlib (tasks and a
>> pack200resource) and will push for a release once CC 1.3 is out.
> Pack200 tasks available out of the box with Ant would be really nice,
The Compress Antlib is separate from the Ant distribution.
<http://ant.apache.org/antlibs/compress/index.html> which doesn't say
much. Current trunk has been following Commons Compress closely and
thus already support Unix dump (undump, actually) and Pack200.
Stefan
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
For additional commands, e-mail: dev-help@commons.apache.org
Re: [compress] Pack200
Posted by Emmanuel Bourg <eb...@apache.org>.
Le 15/09/2011 11:46, Stefan Bodewig a écrit :
> BTW, I've added pack200 support to the Compress Antlib (tasks and a
> pack200resource) and will push for a release once CC 1.3 is out.
Pack200 tasks available out of the box with Ant would be really nice,
I'll give it a look.
Emmanuel Bourg
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
For additional commands, e-mail: dev-help@commons.apache.org
Re: [compress] Pack200
Posted by Stefan Bodewig <bo...@apache.org>.
On 2011-09-14, Emmanuel Bourg wrote:
> Le 14/09/2011 17:16, Stefan Bodewig a écrit :
>> Emmanuel, is the Pack200Utils.normalize method I've just committed what
>> you've been looking for?
> Yes that's what I had in mind, thank you.
> You might also want to support repacking a file to itself, that's a
> common use case.
This should be possible by using the same File instance for both
arguments to normalize. Maybe I should add an alternative method
signature with a sngle File argument (and likely methods without a Map
argument as well).
> I implemented this in the pack200 Ant tasks:
> https://svn.java.net/svn/deployment~svn/trunk/ant-tasks/pack200/src/main/java/org/jdesktop/deployment/ant/pack200/PackUtils.java
Yes, looks pretty similar to what I have, I may want to look at my error
handling a little coser, though. 8-)
BTW, I've added pack200 support to the Compress Antlib (tasks and a
pack200resource) and will push for a release once CC 1.3 is out.
Thanks
Stefan
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
For additional commands, e-mail: dev-help@commons.apache.org
Re: [compress] Pack200
Posted by Emmanuel Bourg <eb...@apache.org>.
Le 14/09/2011 17:16, Stefan Bodewig a écrit :
> Emmanuel, is the Pack200Utils.normalize method I've just committed what
> you've been looking for?
Yes that's what I had in mind, thank you.
You might also want to support repacking a file to itself, that's a
common use case. I implemented this in the pack200 Ant tasks:
https://svn.java.net/svn/deployment~svn/trunk/ant-tasks/pack200/src/main/java/org/jdesktop/deployment/ant/pack200/PackUtils.java
Emmanuel Bourg
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
For additional commands, e-mail: dev-help@commons.apache.org
Re: [compress] Pack200
Posted by Stefan Bodewig <bo...@apache.org>.
On 2011-09-05, Stefan Bodewig wrote:
> On 2011-09-05, Emmanuel Bourg wrote:
>> That looks interesting. Does it provide a repack mode suitable for
>> signing compressed jars?
> I assume you mean
> ,----
> | Note that packing and unpacking a JAR will in general alter the bytewise
> | contents of classfiles in the JAR. This means that packing and unpacking
> | will in general invalidate any digital signatures which rely on bytewise
> | images of JAR elements. In order both to sign and to pack a JAR, you
> | must first pack and unpack the JAR to "normalize" it, then compute
> | signatures on the unpacked JAR elements, and finally repack the signed
> | JAR. Both packing steps should use precisely the same options, and the
> | segment limit may also need to be set to "-1", to prevent accidental
> | variation of segment boundaries as class file sizes change slightly.
> `----
> as in
> <http://download.oracle.com/javase/1.5.0/docs/api/java/util/jar/Pack200.Packer.html>
> I think the "normalization" step could be part of a util class inside
> the package. Some utility method that takes a JAR, packs it and unpacks
> it again to a new jar that would then need to get signed. Signing
> itself seems outside of compress' scope to me.
Emmanuel, is the Pack200Utils.normalize method I've just committed what
you've been looking for?
Stefan
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
For additional commands, e-mail: dev-help@commons.apache.org
Re: [compress] Pack200
Posted by Stefan Bodewig <bo...@apache.org>.
On 2011-09-05, Stefan Bodewig wrote:
> On 2011-09-05, Emmanuel Bourg wrote:
>> That looks interesting. Does it provide a repack mode suitable for
>> signing compressed jars?
> I assume you mean
> ,----
> | Note that packing and unpacking a JAR will in general alter the bytewise
> | contents of classfiles in the JAR. This means that packing and unpacking
> | will in general invalidate any digital signatures which rely on bytewise
> | images of JAR elements. In order both to sign and to pack a JAR, you
> | must first pack and unpack the JAR to "normalize" it, then compute
> | signatures on the unpacked JAR elements, and finally repack the signed
> | JAR. Both packing steps should use precisely the same options, and the
> | segment limit may also need to be set to "-1", to prevent accidental
> | variation of segment boundaries as class file sizes change slightly.
> `----
> as in
> <http://download.oracle.com/javase/1.5.0/docs/api/java/util/jar/Pack200.Packer.html>
> I think the "normalization" step could be part of a util class inside
> the package. Some utility method that takes a JAR, packs it and unpacks
> it again to a new jar that would then need to get signed. Signing
> itself seems outside of compress' scope to me.
Emmanuel, is the Pack200Utils.normalize method I've just committed what
you've been looking for?
Stefan
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
For additional commands, e-mail: dev-help@commons.apache.org
Re: [compress] Pack200
Posted by Stefan Bodewig <bo...@apache.org>.
On 2011-09-05, Emmanuel Bourg wrote:
> That looks interesting. Does it provide a repack mode suitable for
> signing compressed jars?
I assume you mean
,----
| Note that packing and unpacking a JAR will in general alter the bytewise
| contents of classfiles in the JAR. This means that packing and unpacking
| will in general invalidate any digital signatures which rely on bytewise
| images of JAR elements. In order both to sign and to pack a JAR, you
| must first pack and unpack the JAR to "normalize" it, then compute
| signatures on the unpacked JAR elements, and finally repack the signed
| JAR. Both packing steps should use precisely the same options, and the
| segment limit may also need to be set to "-1", to prevent accidental
| variation of segment boundaries as class file sizes change slightly.
`----
as in
<http://download.oracle.com/javase/1.5.0/docs/api/java/util/jar/Pack200.Packer.html>
I think the "normalization" step could be part of a util class inside
the package. Some utility method that takes a JAR, packs it and unpacks
it again to a new jar that would then need to get signed. Signing
itself seems outside of compress' scope to me.
Stefan
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
For additional commands, e-mail: dev-help@commons.apache.org
Re: [compress] Pack200
Posted by Emmanuel Bourg <eb...@apache.org>.
That looks interesting. Does it provide a repack mode suitable for
signing compressed jars?
Emmanuel Bourg
Le 04/09/2011 08:04, Stefan Bodewig a écrit :
> Hi,
>
> I've just committed Converter*Stream implementations for Pack200[1]
> which is a bit unusual in several ways.
>
> First of all it will (by design of the format) only work on compressing
> valid jar files. Actually the result isn't likely to be compressed (in
> the sense of "smaller than the original") at all but expects another
> step of GZip compression in most cases.
>
> The second difference to the other compressors is that the API provided
> by the Java classlib doesn't lend itself to streaming at all. There is
> a Packer/Unpacker that expects an InputStream and an OutputStream and
> converts from one to the other in a single blocking operation (even
> closing the input side when done).
>
> I have experimented with Piped*Streams as well as Ant/commons-exec-like
> stream pumping in order to provide a streaming experience but always ran
> into some edge cases where things broke down. I'll give one example
> below.
>
> The current implementation of Pack200CompressorInputStream will
> pass the wrapped input and an OutputStream writing to a cache to the
> Unpacker synchronously inside the constructor, consuming the input
> completely. It will then defer all read-operations to the cache.
>
> Likewise the Pack200CompressorOutputStream will buffer up all write
> operations in a cache and once finish() or close() is called the cache
> is converted to an InputStream that is then passed together with the
> originally wrapped output to the Packer and written synchronously.
>
> Caches can be in-memory (using ByteArray*Stream) or temporary files
> controlled by a constructor option with in-memory as the default and
> temp-files for cases where the archives are expected to be big.
>
> Because of this design the byte-count methods don't make any sense (do
> we count when data is written-to/read-from the cache or while the
> (Un)Packer is doing its work?) and haven't been implemented at all.
>
> The class names StreamMode and StreamSwitcher result from my attempts of
> using real streams and should be changed unless anybody else comes up
> with a working streaming solution.
>
> The biggest hurdle for any streaming solution is that there is always
> going to be some sort of intermediate buffer. Something picks up data
> written to the output stream and makes it available to the input stream
> side. Once the buffer is full, nothing can be written unless anybody
> reads input in a timely manner.
>
> In the case of a Pack200CompressorInputStream you don't have any control
> over when the user code is going to read the data and whether it is
> going to consume all of it at all. For example if the stream is wrapped
> in a ZipArchiveInputStream (it represents a JAR, after all), it is never
> going to be consumed completely because the archive contains ZIP data at
> the end that is ignored by the input stream implementation.
>
> There are more cases where the Pack/Unpack operation would end up
> blocked so I decided to only code the more robust indirect solution for
> now.
>
> Stefan
>
> [1] http://download.oracle.com/javase/1.5.0/docs/api/java/util/jar/Pack200.html
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
> For additional commands, e-mail: dev-help@commons.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
For additional commands, e-mail: dev-help@commons.apache.org
Re: [compress] Pack200
Posted by Stefan Bodewig <bo...@apache.org>.
On 2011-09-04, sebb wrote:
> You must have put a lot of thought into this
Well, I was without network access for almost two weeks 8-)
> it would be useful to record these design decisions and investigations
> in the code somewhere. e.g. as package Javadoc.
The current package.html already contains the "how" but not the "why",
I'll make sure the rest is persisted soemwhere as well.
Stefan
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
For additional commands, e-mail: dev-help@commons.apache.org
Re: [compress] Pack200
Posted by sebb <se...@gmail.com>.
On 4 September 2011 07:04, Stefan Bodewig <bo...@apache.org> wrote:
> Hi,
>
> I've just committed Converter*Stream implementations for Pack200[1]
> which is a bit unusual in several ways.
>
> First of all it will (by design of the format) only work on compressing
> valid jar files. Actually the result isn't likely to be compressed (in
> the sense of "smaller than the original") at all but expects another
> step of GZip compression in most cases.
>
> The second difference to the other compressors is that the API provided
> by the Java classlib doesn't lend itself to streaming at all. There is
> a Packer/Unpacker that expects an InputStream and an OutputStream and
> converts from one to the other in a single blocking operation (even
> closing the input side when done).
>
> I have experimented with Piped*Streams as well as Ant/commons-exec-like
> stream pumping in order to provide a streaming experience but always ran
> into some edge cases where things broke down. I'll give one example
> below.
>
> The current implementation of Pack200CompressorInputStream will
> pass the wrapped input and an OutputStream writing to a cache to the
> Unpacker synchronously inside the constructor, consuming the input
> completely. It will then defer all read-operations to the cache.
>
> Likewise the Pack200CompressorOutputStream will buffer up all write
> operations in a cache and once finish() or close() is called the cache
> is converted to an InputStream that is then passed together with the
> originally wrapped output to the Packer and written synchronously.
>
> Caches can be in-memory (using ByteArray*Stream) or temporary files
> controlled by a constructor option with in-memory as the default and
> temp-files for cases where the archives are expected to be big.
>
> Because of this design the byte-count methods don't make any sense (do
> we count when data is written-to/read-from the cache or while the
> (Un)Packer is doing its work?) and haven't been implemented at all.
>
> The class names StreamMode and StreamSwitcher result from my attempts of
> using real streams and should be changed unless anybody else comes up
> with a working streaming solution.
>
> The biggest hurdle for any streaming solution is that there is always
> going to be some sort of intermediate buffer. Something picks up data
> written to the output stream and makes it available to the input stream
> side. Once the buffer is full, nothing can be written unless anybody
> reads input in a timely manner.
>
> In the case of a Pack200CompressorInputStream you don't have any control
> over when the user code is going to read the data and whether it is
> going to consume all of it at all. For example if the stream is wrapped
> in a ZipArchiveInputStream (it represents a JAR, after all), it is never
> going to be consumed completely because the archive contains ZIP data at
> the end that is ignored by the input stream implementation.
>
> There are more cases where the Pack/Unpack operation would end up
> blocked so I decided to only code the more robust indirect solution for
> now.
Sounds good.
You must have put a lot of thought into this; it would be useful to
record these design decisions and investigations in the code
somewhere. e.g. as package Javadoc.
> Stefan
>
> [1] http://download.oracle.com/javase/1.5.0/docs/api/java/util/jar/Pack200.html
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
> For additional commands, e-mail: dev-help@commons.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
For additional commands, e-mail: dev-help@commons.apache.org