You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@commons.apache.org by Stefan Bodewig <bo...@apache.org> on 2009/02/07 15:10:28 UTC
[compress] Some merging in zip package
Hi,
I've merged ZipEntry into ZipArchiveEntry and ZipOutputStream into
ZipArchiveOutputStream. As a side effect ZipArchiveOutputStream now
supports encoding of file names and the zip file comment (among other
things that it does better than java.util.zip.ZipOutputStream like
providing usable Unix permissions for directory entries).
ZipArchiveOutputStream will now use the platform's native encoding
instead of UTF8 by default now, while JarArchiveOutputStream
explicitly sets the encoding to UTF8. Does anybody consider this a
problem?
Ant never had a ZipInputStream replacement, we only use ZipFile since
the zip format is a lot easier to implement on top of
RandomAccessFile. I suggest we state somewhere in the docs (which
docs?) that you should use ZipFile if you intend to read a zip archive
with non-UTF8 encoding of filenames.
I'll look into merging the Tar streams as well next.
Stefan
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
For additional commands, e-mail: dev-help@commons.apache.org
Re: [compress] Some merging in zip package
Posted by Stefan Bodewig <bo...@apache.org>.
On 2009-02-10, Torsten Curdt <tc...@vafer.org> wrote:
>> Ant never had a ZipInputStream replacement, we only use ZipFile since
>> the zip format is a lot easier to implement on top of
>> RandomAccessFile.
> Hm ... never thought this would be a problem for just reading.
You need to merge data from the local file header with data from the
central directory - and the central directory is at the end of the
archive so you end up jumping between the end of the stream and the
local file header position a lot.
It also means you have to read the stream fully to even read the first
entry.
I think java.util.zip.ZipInputStream simply ignores the central
directory, something I wouldn't like to do myself.
>> that you should use ZipFile if you intend to read a zip archive
>> with non-UTF8 encoding of filenames.
> Why is that?
Our ZipArchiveInputStream uses java.util.zip.ZipInputStream which
doesn't deal with any encoding other than UTF-8.
Stefan
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
For additional commands, e-mail: dev-help@commons.apache.org
Re: [compress] Some merging in zip package
Posted by Stefan Bodewig <bo...@apache.org>.
On 2009-02-11, sebb <se...@gmail.com> wrote:
> That seems likely to produce some confusion - would it not be better
> to default to UTF-8?
Yes, that's been my initial question. Is it OK to have no default at
all or should we keep with UTF-8?
Given that ZipArchiveInputStream doesn't support anything but UTF-8 at
all I'm now leaning towards making UTF-8 the default.
Stefan
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
For additional commands, e-mail: dev-help@commons.apache.org
Re: [compress] Some merging in zip package
Posted by sebb <se...@gmail.com>.
On 11/02/2009, Stefan Bodewig <bo...@apache.org> wrote:
> On 2009-02-11, Torsten Curdt <tc...@apache.org> wrote:
>
> > I am also not so sure this really all that bad. I guess there are 3 scenarios
>
> > 1: the archive standard is known to use a specific encoding
> > 2: the encoding is specified inside the archive (which is similar to 1)
> > 3: we have no clue about the encoding of the strings in the archive
>
> > Unless I am missing something we are fine for 1+2 because as long as
> > we create the strings as we should. It's up to the user of compress to
> > turn this into something he can use on his platform.
>
> > For 3 there is just no point. All we can do is provide a way to get
> > the name and the user needs to figure out the conversion. Nothing we
> > can do about it.
>
> > So I guess we all we need to do is to be sure not to create Strings in
> > the default encoding for 1+2.
>
> > Or what am I missing?
>
>
> Not much, except that java.uti.zip.Zip*putStream and thus the "old"
> ZipArchiveOutputStream always are in your state 1: UTF-8. This also
> means they are unable to create or read anything but UTF-8.
>
> The new ZipArchiveOutputStream uses the platform's default which makes
> your case 3 more likely unless people take care to note the encoding
> when creating the archive.
That seems likely to produce some confusion - would it not be better
to default to UTF-8?
> Note that more modern ZIP implementations provide a way to explicitly
> say "this is UTF-8" inside the archive and SANDBOX-176 contains a
> patch that claims to support that (as does
> <https://issues.apache.org/bugzilla/show_bug.cgi?id=45548>) - I'll be
> looking into it.
>
>
> Stefan
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
> For additional commands, e-mail: dev-help@commons.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
For additional commands, e-mail: dev-help@commons.apache.org
Re: [compress] Some merging in zip package
Posted by Stefan Bodewig <bo...@apache.org>.
On 2009-02-11, Torsten Curdt <tc...@apache.org> wrote:
> I am also not so sure this really all that bad. I guess there are 3 scenarios
> 1: the archive standard is known to use a specific encoding
> 2: the encoding is specified inside the archive (which is similar to 1)
> 3: we have no clue about the encoding of the strings in the archive
> Unless I am missing something we are fine for 1+2 because as long as
> we create the strings as we should. It's up to the user of compress to
> turn this into something he can use on his platform.
> For 3 there is just no point. All we can do is provide a way to get
> the name and the user needs to figure out the conversion. Nothing we
> can do about it.
> So I guess we all we need to do is to be sure not to create Strings in
> the default encoding for 1+2.
> Or what am I missing?
Not much, except that java.uti.zip.Zip*putStream and thus the "old"
ZipArchiveOutputStream always are in your state 1: UTF-8. This also
means they are unable to create or read anything but UTF-8.
The new ZipArchiveOutputStream uses the platform's default which makes
your case 3 more likely unless people take care to note the encoding
when creating the archive.
Note that more modern ZIP implementations provide a way to explicitly
say "this is UTF-8" inside the archive and SANDBOX-176 contains a
patch that claims to support that (as does
<https://issues.apache.org/bugzilla/show_bug.cgi?id=45548>) - I'll be
looking into it.
Stefan
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
For additional commands, e-mail: dev-help@commons.apache.org
Re: [compress] Some merging in zip package
Posted by Torsten Curdt <tc...@apache.org>.
I am also not so sure this really all that bad. I guess there are 3 scenarios
1: the archive standard is known to use a specific encoding
2: the encoding is specified inside the archive (which is similar to 1)
3: we have no clue about the encoding of the strings in the archive
Unless I am missing something we are fine for 1+2 because as long as
we create the strings as we should. It's up to the user of compress to
turn this into something he can use on his platform.
For 3 there is just no point. All we can do is provide a way to get
the name and the user needs to figure out the conversion. Nothing we
can do about it.
So I guess we all we need to do is to be sure not to create Strings in
the default encoding for 1+2.
Or what am I missing?
cheers
--
Torsten
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
For additional commands, e-mail: dev-help@commons.apache.org
Re: [compress] Some merging in zip package
Posted by Stefan Bodewig <bo...@apache.org>.
On 2009-02-10, Christian Grobmeier <gr...@gmail.com> wrote:
> Can this be simulated when creating a file within testcase and then
> changing java.encoding stuff and extracting it again?
Likely. I'm not sure whether String#getBytes consults the system
property for each invocation.
> If so I can write a testcase (i am working on them currently)
Cool.
Stefan
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
For additional commands, e-mail: dev-help@commons.apache.org
Re: [compress] Some merging in zip package
Posted by Christian Grobmeier <gr...@gmail.com>.
>> > ZipArchiveOutputStream will now use the platform's native encoding
>> > instead of UTF8 by default now, while JarArchiveOutputStream
>> > explicitly sets the encoding to UTF8. Does anybody consider this a
>> > problem?
>>
> What happens if you try to take a file that was created on a machine with
> one default encoding, and send it to another machine with a different
> default encoding (i.e. Linux -> Windows)?
Can this be simulated when creating a file within testcase and then
changing java.encoding stuff and extracting it again?
If so I can write a testcase (i am working on them currently)
cheers
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
For additional commands, e-mail: dev-help@commons.apache.org
Re: [compress] Some merging in zip package
Posted by Stefan Bodewig <bo...@apache.org>.
On 2009-02-10, Mark Fortner <ph...@gmail.com> wrote:
> I meant specifically if I was using this library at both ends of the
> equation.
Then if you don't specify an encoding explicitly, you will have a
problem, yes. Of course only, if your file names have non-ASCII
characters in the first place since most native encodings happen to
agree on the ASCII subset.
> I think Linux may default to the UTF-8 encoding, and Windows to the
> ISO-8859-1 encoding.
It's way worse that that 8-)
On Windows, it probably is locale dependent, meaning that you may end
up getting different "i" characters on a Turkish locale, for example.
Stefan
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
For additional commands, e-mail: dev-help@commons.apache.org
Re: [compress] Some merging in zip package
Posted by Mark Fortner <ph...@gmail.com>.
I meant specifically if I was using this library at both ends of the
equation. I assume that if you use different tools at either end, you may
have some challenges. But if I use this library at both ends, and use the
default settings, then I imagine I would have some difficulty on the
decompression side. I think Linux may default to the UTF-8 encoding, and
Windows to the ISO-8859-1 encoding.
Mark
On Tue, Feb 10, 2009 at 8:57 AM, Stefan Bodewig <bo...@apache.org> wrote:
> On 2009-02-10, Mark Fortner <ph...@gmail.com> wrote:
>
>
> >>> ZipArchiveOutputStream will now use the platform's native encoding
> >>> instead of UTF8 by default now, while JarArchiveOutputStream
> >>> explicitly sets the encoding to UTF8. Does anybody consider this a
> >>> problem?
>
> > What happens if you try to take a file that was created on a machine with
> > one default encoding, and send it to another machine with a different
> > default encoding (i.e. Linux -> Windows)?
>
> It depends on what you use to extract the archive, I guess. In a
> situation like this you are much better off specifiying the encoding
> explicitly.
>
> Most tools don't assume UTF-8 (the old default) at all, so you
> wouldn't be able to extract an archive created by the jar command (or
> the prevision version of ZipArchiveOutputStream) in a scenario like
> this either.
>
> Stefan
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
> For additional commands, e-mail: dev-help@commons.apache.org
>
>
--
Mark Fortner
blog: http://feeds.feedburner.com/jroller/ideafactory
Re: [compress] Some merging in zip package
Posted by Stefan Bodewig <bo...@apache.org>.
On 2009-02-10, Mark Fortner <ph...@gmail.com> wrote:
>>> ZipArchiveOutputStream will now use the platform's native encoding
>>> instead of UTF8 by default now, while JarArchiveOutputStream
>>> explicitly sets the encoding to UTF8. Does anybody consider this a
>>> problem?
> What happens if you try to take a file that was created on a machine with
> one default encoding, and send it to another machine with a different
> default encoding (i.e. Linux -> Windows)?
It depends on what you use to extract the archive, I guess. In a
situation like this you are much better off specifiying the encoding
explicitly.
Most tools don't assume UTF-8 (the old default) at all, so you
wouldn't be able to extract an archive created by the jar command (or
the prevision version of ZipArchiveOutputStream) in a scenario like
this either.
Stefan
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
For additional commands, e-mail: dev-help@commons.apache.org
Re: [compress] Some merging in zip package
Posted by Mark Fortner <ph...@gmail.com>.
>
> > ZipArchiveOutputStream will now use the platform's native encoding
> > instead of UTF8 by default now, while JarArchiveOutputStream
> > explicitly sets the encoding to UTF8. Does anybody consider this a
> > problem?
>
>
What happens if you try to take a file that was created on a machine with
one default encoding, and send it to another machine with a different
default encoding (i.e. Linux -> Windows)?
Regards,
Mark Fortner
Re: [compress] Some merging in zip package
Posted by Torsten Curdt <tc...@vafer.org>.
> I've merged ZipEntry into ZipArchiveEntry and ZipOutputStream into
> ZipArchiveOutputStream. As a side effect ZipArchiveOutputStream now
> supports encoding of file names and the zip file comment (among other
> things that it does better than java.util.zip.ZipOutputStream like
> providing usable Unix permissions for directory entries).
Great!
> ZipArchiveOutputStream will now use the platform's native encoding
> instead of UTF8 by default now, while JarArchiveOutputStream
> explicitly sets the encoding to UTF8. Does anybody consider this a
> problem?
I assume that should be OK.
> Ant never had a ZipInputStream replacement, we only use ZipFile since
> the zip format is a lot easier to implement on top of
> RandomAccessFile.
Hm ... never thought this would be a problem for just reading.
> I suggest we state somewhere in the docs (which
> docs?)
:)
> that you should use ZipFile if you intend to read a zip archive
> with non-UTF8 encoding of filenames.
Why is that?
> I'll look into merging the Tar streams as well next.
Cool
cheers
--
Torsten
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@commons.apache.org
For additional commands, e-mail: dev-help@commons.apache.org