You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@camel.apache.org by Gardella juan <ga...@gmail.com> on 2013/07/05 21:50:54 UTC
Re: zip file best practices
Hello Willieem,
It seems there is a problem with the ZipSplitter that you did. I had added a
test "ZipProcessRouteBuilderIT" to reproduce the error at
https://issues.apache.org/jira/browse/CAMEL-6139. If you use my ZipSplitter
the test works.
Thanks,
Juan
--
View this message in context: http://camel.465427.n5.nabble.com/zip-file-best-practices-tp5713437p5735214.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re[2]: zip file best practices
Posted by Serge Shikov <sh...@inbox.ru>.
It will be very good also if all ZipEntry fields (see below) will be added to message as headers.
String name; // entry name
long time = -1; // modification time (in DOS time)
long crc = -1; // crc-32 of entry data
long size = -1; // uncompressed size of entry data
long csize = -1; // compressed size of entry data
int method = -1; // compression method
int flag = 0; // general purpose flag
byte[] extra; // optional extra field data for entry
String comment; // optional comment string for entry
Some of this data could be helpful for processing entries, if available for camel route code. And it should be easy to add - just one line for each field, or even
answer.setHeader("zipEntry", current.clone());
to add entry itself.
Sun, 4 May 2014 02:37:52 -0700 (PDT) от "Claus Ibsen-2 [via Camel]" <ml...@n5.nabble.com>:
>On Sat, May 3, 2014 at 10:08 AM, Serge Shikov < [hidden email] > wrote:
>
>> Here is part of ZipIterator:
>>
>>
>>
>> Note ignored IOException's. I think is should be logged and processed.
>> Probably messages should be generated even if we can't unzip some entries.
>>
>>
>>
>> --
>> View this message in context: http://camel.465427.n5.nabble.com/zip-file-best-practices-tp5713437p5750812.html
>> Sent from the Camel - Users mailing list archive at Nabble.com.
>Hi
>
>Yeah sounds like we should detect those io exceptions and propagate
>them back so Camel can react. Maybe we can have an option on the zip
>iterator to ignore those exceptions so people can partially read
>corrupted jars.
>
>Fell free to log a JIRA ticket
>http://camel.apache.org/support
>
>
>--
>Claus Ibsen
>-----------------
>Red Hat, Inc.
>Email: [hidden email]
>Twitter: davsclaus
>Blog: http://davsclaus.com
>Author of Camel in Action: http://www.manning.com/ibsen
>hawtio: http://hawt.io/
>fabric8: http://fabric8.io/
>
>
>----------------------------------------------------------------------
>If you reply to this email, your message will be added to the discussion below: http://camel.465427.n5.nabble.com/zip-file-best-practices-tp5713437p5750835.html
>To unsubscribe from zip file best practices, click here .
>NAML
--
View this message in context: http://camel.465427.n5.nabble.com/zip-file-best-practices-tp5713437p5750836.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: zip file best practices
Posted by Willem Jiang <wi...@gmail.com>.
I think we should throw the exception out to let the user know there is something wrong with the Zip file. If the ZipIterator eat up the exception, it could be every hard for the user to find out why the camel route doesn’t work as he expects.
I just fill a JIRA[1] to throw the exception out from the ZipIterator.
[1]https://issues.apache.org/jira/browse/CAMEL-7409
--
Willem Jiang
Red Hat, Inc.
Web: http://www.redhat.com
Blog: http://willemjiang.blogspot.com (English)
http://jnn.iteye.com (Chinese)
Twitter: willemjiang
Weibo: 姜宁willem
On May 4, 2014 at 5:37:18 PM, Claus Ibsen (claus.ibsen@gmail.com) wrote:
> On Sat, May 3, 2014 at 10:08 AM, Serge Shikov wrote:
> > Here is part of ZipIterator:
> >
> >
> >
> > Note ignored IOException's. I think is should be logged and processed.
> > Probably messages should be generated even if we can't unzip some entries.
> >
> >
> >
> > --
> > View this message in context: http://camel.465427.n5.nabble.com/zip-file-best-practices-tp5713437p5750812.html
> > Sent from the Camel - Users mailing list archive at Nabble.com.
>
> Hi
>
> Yeah sounds like we should detect those io exceptions and propagate
> them back so Camel can react. Maybe we can have an option on the zip
> iterator to ignore those exceptions so people can partially read
> corrupted jars.
>
> Fell free to log a JIRA ticket
> http://camel.apache.org/support
>
>
> --
> Claus Ibsen
> -----------------
> Red Hat, Inc.
> Email: cibsen@redhat.com
> Twitter: davsclaus
> Blog: http://davsclaus.com
> Author of Camel in Action: http://www.manning.com/ibsen
> hawtio: http://hawt.io/
> fabric8: http://fabric8.io/
>
Re: zip file best practices
Posted by Claus Ibsen <cl...@gmail.com>.
On Sat, May 3, 2014 at 10:08 AM, Serge Shikov <sh...@inbox.ru> wrote:
> Here is part of ZipIterator:
>
>
>
> Note ignored IOException's. I think is should be logged and processed.
> Probably messages should be generated even if we can't unzip some entries.
>
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/zip-file-best-practices-tp5713437p5750812.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
Hi
Yeah sounds like we should detect those io exceptions and propagate
them back so Camel can react. Maybe we can have an option on the zip
iterator to ignore those exceptions so people can partially read
corrupted jars.
Fell free to log a JIRA ticket
http://camel.apache.org/support
--
Claus Ibsen
-----------------
Red Hat, Inc.
Email: cibsen@redhat.com
Twitter: davsclaus
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen
hawtio: http://hawt.io/
fabric8: http://fabric8.io/
Re: zip file best practices
Posted by Serge Shikov <sh...@inbox.ru>.
Here is part of ZipIterator:
Note ignored IOException's. I think is should be logged and processed.
Probably messages should be generated even if we can't unzip some entries.
--
View this message in context: http://camel.465427.n5.nabble.com/zip-file-best-practices-tp5713437p5750812.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: zip file best practices
Posted by Serge Shikov <sh...@inbox.ru>.
Hi, everybody
Could somebody explain, how are errors handled within ZipSplitter and
ZipFileDataFormat?
Just for example, I have a project where I should receive zip files from
external source via http.
What if zip central directory was corrupted during transfer? Does this
component check CRC for zip entries?
Just for example - I have tested some use cases with my camel unzipping
route, and found that if I try to unzip something (not a zip archive at
all), I'll get just one message after splitter, and original file as its
body.
And no exceptions, log messages etc. I don't think this is correct.
Regards
Serge
--
View this message in context: http://camel.465427.n5.nabble.com/zip-file-best-practices-tp5713437p5750811.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: zip file best practices
Posted by Henryk Konsek <he...@gmail.com>.
> Yes, that's good idea. I was thinking about writing another processor to
> do this stuff, but maybe better will be to just improve a bit Camel.
Fell free to create a pull request and ping me when it will be ready.
I'll review it and merge to the master on you behalf.
Cheers.
--
Henryk Konsek
http://henryk-konsek.blogspot.com
Re: zip file best practices
Posted by Jakub Bubin <ja...@datax.pl>.
W dniu 2014-03-03 15:47, hekonsek [via Camel] pisze:
> > I spent a day or two trying the zip component. Finally got it working,
>
> Yeah, Zip File component need some improvements. It lacks some
> functionality and is sometimes a bit unintuitive in a usage.
Good to hear that I was right with my guesses.
>
> > but I
> > think there is still one thing to cover in this component: recursive
> reading
> > entries from zip package.
>
> We love [1] contributors . Our community would highly appreciate
> pull request with such enhancement. I personally think that we don't
> even need 'recursive' option - we can just automatically detected if
> given zip entry is a directory or file.
Yes, that's good idea. I was thinking about writing another processor to
do this stuff, but maybe better will be to just improve a bit Camel.
Cheers
--
DATAX Sp. z o.o., ul. Fabryczna 10, 53-609 Wrocław, Poland, NIP PL 522-01-00-172
Sąd Rejestrowy dla Wrocławia - Fabrycznej, VI Wydz. Gosp. Krajowego Rejestru Sądowego
KRS 0000126740, Kapitał zakładowy: 131.200 PLN (opłacony w całości)
--
View this message in context: http://camel.465427.n5.nabble.com/zip-file-best-practices-tp5713437p5748254.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: zip file best practices
Posted by Henryk Konsek <he...@gmail.com>.
> I spent a day or two trying the zip component. Finally got it working,
Yeah, Zip File component need some improvements. It lacks some
functionality and is sometimes a bit unintuitive in a usage.
> but I
> think there is still one thing to cover in this component: recursive reading
> entries from zip package.
We love [1] contributors :) . Our community would highly appreciate
pull request with such enhancement. I personally think that we don't
even need 'recursive' option - we can just automatically detected if
given zip entry is a directory or file.
Cheers.
[1] https://camel.apache.org/contributing.html
--
Henryk Konsek
http://henryk-konsek.blogspot.com
Re: zip file best practices
Posted by Jakub Bubin <jb...@datax.pl>.
Hi Guys,
I spent a day or two trying the zip component. Finally got it working, but I
think there is still one thing to cover in this component: recursive reading
entries from zip package.
Use case: I want to unzip .zip file which contains many files and process
them separately. But files in package are not packed directly to the zip,
but somewhere in the directory tree (i.e. files.zip -> directory1 ->
directory2 -> file1.txt, file1.txt and so on..).
I think it should be possible with parameter 'recursive' set to 'true'. At
this moment I think it doesn't work this way.
--
View this message in context: http://camel.465427.n5.nabble.com/zip-file-best-practices-tp5713437p5748252.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: zip file best practices
Posted by JonasF <jo...@fugedi.se>.
Ok, thanks I got it working now! The issue was in my missunderstanding that I needed to unmarshall the zip file into a zipFile before calling the split.
On 24 Jan 2014, at 22:00, alexey-s [via Camel] <ml...@n5.nabble.com> wrote:
> I never used to separate files in zip archive.
> I use a component splitter to read nested XML files.
>
> Example use in spring DSL:
> http://stackoverflow.com/questions/20294005/using-zipsplitter-in-apache-camel-spring-dsl
>
> Copy/paste and add comments
>
> <bean id="zipSplitter" class="org.apache.camel.dataformat.zipfile.ZipSplitter" />
> ...
> <split streaming="true" parallelProcessing="true">
> <ref>zipSplitter</ref>
> <!-- Message body is ZipInputStreamWrapper.class extends BufferedInputStream -->
> <!-- Header CamelFileName = include_file_name -->
> <!-- Header zipFileName = include_file_name -->
> <convertBodyTo type="java.lang.String" />
> <to uri="file:foo" />
> </split>
>
>
>
>
> Aleksey
>
> If you reply to this email, your message will be added to the discussion below:
> http://camel.465427.n5.nabble.com/zip-file-best-practices-tp5713437p5746463.html
> To unsubscribe from zip file best practices, click here.
> NAML
SampleZipSplitter.xml (1K) <http://camel.465427.n5.nabble.com/attachment/5746466/0/SampleZipSplitter.xml>
--
View this message in context: http://camel.465427.n5.nabble.com/zip-file-best-practices-tp5713437p5746466.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: zip file best practices
Posted by alexey-s <al...@mail.ru>.
Can be improved.
I thought it would be interesting to the user to remember the name of the
zip file.
Now ZipItaror not give this information.
answer.getHeaders().putAll(inputMessage.getHeaders());
answer.setHeader("zipFileName", current.getName());
answer.setHeader(Exchange.FILE_NAME, current.getName());
It has been
${header.camelFileName} = archive_file.zip
${header.zipFileName} = attached_file_name
It has become
${header.camelFileName} = attached_file_name
${header.zipFileName} = attached_file_name
What for?
Alexey
--
View this message in context: http://camel.465427.n5.nabble.com/zip-file-best-practices-tp5713437p5741856.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: zip file best practices
Posted by alexey-s <al...@mail.ru>.
I never used to separate files in zip archive.
I use a component splitter to read nested XML files.
Example use in spring DSL:
http://stackoverflow.com/questions/20294005/using-zipsplitter-in-apache-camel-spring-dsl
Copy/paste and add comments
<bean id="zipSplitter"
class="org.apache.camel.dataformat.zipfile.ZipSplitter" />
...
<split streaming="true" parallelProcessing="true">
<ref>zipSplitter</ref>
<!-- Message body is ZipInputStreamWrapper.class extends
BufferedInputStream -->
<!-- Header CamelFileName = include_file_name -->
<!-- Header zipFileName = include_file_name -->
<convertBodyTo type="java.lang.String" />
<to uri="file:foo" />
</split>
Aleksey
--
View this message in context: http://camel.465427.n5.nabble.com/zip-file-best-practices-tp5713437p5746463.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: zip file best practices
Posted by JonasF <jo...@fugedi.se>.
Oh, good! Please provide one working spring XML example!
--
View this message in context: http://camel.465427.n5.nabble.com/zip-file-best-practices-tp5713437p5746454.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: zip file best practices
Posted by alexey-s <al...@mail.ru>.
ZipSplitter uses the input stream from the archive file (message body). Sends
output stream for each attached file.
One should not expect from him GenericFile
Nothing complicated.
--
View this message in context: http://camel.465427.n5.nabble.com/zip-file-best-practices-tp5713437p5746434.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: zip file best practices
Posted by JonasF <jo...@fugedi.se>.
I spent a few hours trying to get this to work in Spring DSL as well without
any luck. I tried debugging it and things looked correct but then somehow I
lost the context and the wrong data was coming out (i.e. the ZipSplitter was
called and did its job but the answer got lost somehow). I ran out of
time/patience and implemented a processor which performed the decompression
to a temporary file and passed on each file as a new exchange.
Here is what my xml looked like when it almost worked...
<bean id="zipSplitter"
class="org.apache.camel.dataformat.zipfile.ZipSplitter"/>
...
<unmarshal><zipFile usingIterator="true"/></unmarshal>
<split streaming="true">
<method ref="zipSplitter"/>
<to uri="direct:csv"/>
</split>
--
View this message in context: http://camel.465427.n5.nabble.com/zip-file-best-practices-tp5713437p5746412.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: zip file best practices
Posted by peter <pe...@mtsft.com>.
I believe there was an original request on this post for someone to please
show how to use the ZipSplitter with Spring DSL. But I have not seen this
anywhere and I have not been able to get it to work. Here is a snippet of my
attempt:
<bean id="myZipSplitter"
class="org.apache.camel.dataformat.zipfile.ZipSplitter"/>
<camelContext xmlns="http://camel.apache.org/schema/spring">
<route>
<from uri="file:src/data?noop=true"/>
<choice>
<when>
<simple>${file:name.ext} == 'max'</simple>
<split streaming="true">
<ref>myZipSplitter</ref>
<choice>
<when>
<simple>${header.ZipFileName.ext} ==
'pdz'</simple>
<log loggingLevel="TRACE" message="Processing
PDZ: ${header.ZipFileName.ext}"/>
<to uri="file:target/data/xml"/>
</when>
<otherwise>
<log message="Ignoring found zip entry :
${header.ZipFileName.ext}"/>
</otherwise>
</choice>
</split>
.
.
.
Could someone please post a correct example of usage of the ZipFile
Dataformat (using the ZipSplitter expression) or a statement that it cannot
be used with Spring?
--
View this message in context: http://camel.465427.n5.nabble.com/zip-file-best-practices-tp5713437p5741806.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: zip file best practices
Posted by Willem jiang <wi...@gmail.com>.
Hi Juan,
Thanks for reporting it. I will check out your patch when I get some time next week.
--
Willem Jiang
Red Hat, Inc.
FuseSource is now part of Red Hat
Web: http://www.fusesource.com | http://www.redhat.com
Blog: http://willemjiang.blogspot.com (http://willemjiang.blogspot.com/) (English)
http://jnn.iteye.com (http://jnn.javaeye.com/) (Chinese)
Twitter: willemjiang
Weibo: 姜宁willem
On Saturday, July 6, 2013 at 3:50 AM, Gardella juan wrote:
> Hello Willieem,
>
> It seems there is a problem with the ZipSplitter that you did. I had added a
> test "ZipProcessRouteBuilderIT" to reproduce the error at
> https://issues.apache.org/jira/browse/CAMEL-6139. If you use my ZipSplitter
> the test works.
>
> Thanks,
> Juan
>
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/zip-file-best-practices-tp5713437p5735214.html
> Sent from the Camel - Users mailing list archive at Nabble.com (http://Nabble.com).