You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@camel.apache.org by Tim Dudgeon <td...@gmail.com> on 2014/05/04 17:25:59 UTC

lazyLoad with CSV blows up on last line?

I'm trying to use the CSV data format to handle a large file.
Thinking I should use CsvDataFormat.setLazyLoad(true)
I sort of got it working but it blows up on the last line of the file 
with a java.io.IOException: Stream closed exception.
Is this a bug, or am I dong it wrong? This is with Camel 2.13.0

Here is example of what I'm doing:

CsvDataFormat csv = new CsvDataFormat()
csv.setDelimiter(' ')
csv.setSkipFirstLine(false)
csv.setLazyLoad(true)

CamelContext camelContext = new DefaultCamelContext()
camelContext.addRoutes(new RouteBuilder() {
         def void configure() {
             from('direct:start')
             .unmarshal(csv)
             .split(body()).streaming()
             .log('row: ${body}')
         }
     })
camelContext.start()

ProducerTemplate t = camelContext.createProducerTemplate()
t.sendBody('direct:start', new File('/Users/timbo/data/test.txt'))

camelContext.stop()

And here is exception.

java.lang.IllegalStateException: java.io.IOException: Stream closed
     at 
org.apache.camel.dataformat.csv.CsvIterator.next(CsvIterator.java:61)
     at 
org.apache.camel.processor.Splitter$SplitterIterable$1.next(Splitter.java:170)
     at 
org.apache.camel.processor.Splitter$SplitterIterable$1.next(Splitter.java:146)
     at 
org.apache.camel.processor.MulticastProcessor.doProcessSequential(MulticastProcessor.java:502)
     at 
org.apache.camel.processor.MulticastProcessor.process(MulticastProcessor.java:215)
     at org.apache.camel.processor.Splitter.process(Splitter.java:98)
     at 
org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:72)
     at 
org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:398)
     at 
org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:191)
     at org.apache.camel.processor.Pipeline.process(Pipeline.java:118)
     at org.apache.camel.processor.Pipeline.process(Pipeline.java:80)
     at 
org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:191)
     at 
org.apache.camel.component.direct.DirectProducer.process(DirectProducer.java:51)
     at 
org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:191)
     at 
org.apache.camel.processor.UnitOfWorkProducer.process(UnitOfWorkProducer.java:73)
     at 
org.apache.camel.impl.ProducerCache$2.doInProducer(ProducerCache.java:378)
     at 
org.apache.camel.impl.ProducerCache$2.doInProducer(ProducerCache.java:346)
     at 
org.apache.camel.impl.ProducerCache.doInProducer(ProducerCache.java:242)
     at 
org.apache.camel.impl.ProducerCache.sendExchange(ProducerCache.java:346)
     at org.apache.camel.impl.ProducerCache.send(ProducerCache.java:184)
     at 
org.apache.camel.impl.DefaultProducerTemplate.send(DefaultProducerTemplate.java:124)
     at 
org.apache.camel.impl.DefaultProducerTemplate.sendBody(DefaultProducerTemplate.java:137)
     at 
org.apache.camel.impl.DefaultProducerTemplate.sendBody(DefaultProducerTemplate.java:144)
     at org.apache.camel.ProducerTemplate$sendBody.call(Unknown Source)

Thanks
Tim

Re: lazyLoad with CSV blows up on last line?

Posted by Willem Jiang <wi...@gmail.com>.
I filled JIRA[1] for it and the patch is on the way.

[1]https://issues.apache.org/jira/browse/CAMEL-7415

--  
Willem Jiang

Red Hat, Inc.
Web: http://www.redhat.com
Blog: http://willemjiang.blogspot.com (English)
http://jnn.iteye.com (Chinese)
Twitter: willemjiang  
Weibo: 姜宁willem



On May 6, 2014 at 11:18:53 AM, Willem Jiang (willem.jiang@gmail.com) wrote:
> OK, I managed to reproduce the error.
> I will keep digging the code to see if I can find a solution here.
>  
> --
> Willem Jiang
>  
> Red Hat, Inc.
> Web: http://www.redhat.com
> Blog: http://willemjiang.blogspot.com (English)
> http://jnn.iteye.com (Chinese)
> Twitter: willemjiang
> Weibo: 姜宁willem
>  
>  
>  
> On May 6, 2014 at 1:23:30 AM, Tim Dudgeon (tdudgeon.ml@gmail.com) wrote:
> > I don't think that's the problem. I added long sleep before shutting
> > down the camel context and still get the error (my test file is very
> > small so gets processed almost instantly).
> > So think it must be something else closing the stream prematurely.
> >
> > Tim
> >
> > On 05/05/2014 02:50, Willem Jiang wrote:
> > > The Stream is closed by other thread.
> > > I think you need to let the main thread sleep for few minutes to let the camel route finish  
> > the processing of the message.
> > >
> > > --
> > > Willem Jiang
> > >
> > > Red Hat, Inc.
> > > Web: http://www.redhat.com
> > > Blog: http://willemjiang.blogspot.com (English)
> > > http://jnn.iteye.com (Chinese)
> > > Twitter: willemjiang
> > > Weibo: 姜宁willem
> > >
> > >
> > >
> > > On May 4, 2014 at 11:26:27 PM, Tim Dudgeon (tdudgeon.ml@gmail.com) wrote:
> > >> I'm trying to use the CSV data format to handle a large file.
> > >> Thinking I should use CsvDataFormat.setLazyLoad(true)
> > >> I sort of got it working but it blows up on the last line of the file
> > >> with a java.io.IOException: Stream closed exception.
> > >> Is this a bug, or am I dong it wrong? This is with Camel 2.13.0
> > >>
> > >> Here is example of what I'm doing:
> > >>
> > >> CsvDataFormat csv = new CsvDataFormat()
> > >> csv.setDelimiter(' ')
> > >> csv.setSkipFirstLine(false)
> > >> csv.setLazyLoad(true)
> > >>
> > >> CamelContext camelContext = new DefaultCamelContext()
> > >> camelContext.addRoutes(new RouteBuilder() {
> > >> def void configure() {
> > >> from('direct:start')
> > >> .unmarshal(csv)
> > >> .split(body()).streaming()
> > >> .log('row: ${body}')
> > >> }
> > >> })
> > >> camelContext.start()
> > >>
> > >> ProducerTemplate t = camelContext.createProducerTemplate()
> > >> t.sendBody('direct:start', new File('/Users/timbo/data/test.txt'))
> > >>
> > >> camelContext.stop()
> > >>
> > >> And here is exception.
> > >>
> > >> java.lang.IllegalStateException: java.io.IOException: Stream closed
> > >> at
> > >> org.apache.camel.dataformat.csv.CsvIterator.next(CsvIterator.java:61)  
> > >> at
> > >> org.apache.camel.processor.Splitter$SplitterIterable$1.next(Splitter.java:170)  
> > >> at
> > >> org.apache.camel.processor.Splitter$SplitterIterable$1.next(Splitter.java:146)  
> > >> at
> > >> org.apache.camel.processor.MulticastProcessor.doProcessSequential(MulticastProcessor.java:502)  
> > >> at
> > >> org.apache.camel.processor.MulticastProcessor.process(MulticastProcessor.java:215)  
> > >> at org.apache.camel.processor.Splitter.process(Splitter.java:98)
> > >> at
> > >> org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:72)  
> > >> at
> > >> org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:398)  
> > >> at
> > >> org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:191)  
> > >> at org.apache.camel.processor.Pipeline.process(Pipeline.java:118)
> > >> at org.apache.camel.processor.Pipeline.process(Pipeline.java:80)
> > >> at
> > >> org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:191)  
> > >> at
> > >> org.apache.camel.component.direct.DirectProducer.process(DirectProducer.java:51)  
> > >> at
> > >> org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:191)  
> > >> at
> > >> org.apache.camel.processor.UnitOfWorkProducer.process(UnitOfWorkProducer.java:73)  
> > >> at
> > >> org.apache.camel.impl.ProducerCache$2.doInProducer(ProducerCache.java:378)  
> > >> at
> > >> org.apache.camel.impl.ProducerCache$2.doInProducer(ProducerCache.java:346)  
> > >> at
> > >> org.apache.camel.impl.ProducerCache.doInProducer(ProducerCache.java:242)  
> > >> at
> > >> org.apache.camel.impl.ProducerCache.sendExchange(ProducerCache.java:346)  
> > >> at org.apache.camel.impl.ProducerCache.send(ProducerCache.java:184)
> > >> at
> > >> org.apache.camel.impl.DefaultProducerTemplate.send(DefaultProducerTemplate.java:124)  
> > >> at
> > >> org.apache.camel.impl.DefaultProducerTemplate.sendBody(DefaultProducerTemplate.java:137)  
> > >> at
> > >> org.apache.camel.impl.DefaultProducerTemplate.sendBody(DefaultProducerTemplate.java:144)  
> > >> at org.apache.camel.ProducerTemplate$sendBody.call(Unknown Source)
> > >>
> > >> Thanks
> > >> Tim
> > >>
> > >>
> >
> >
>  
>

Re: lazyLoad with CSV blows up on last line?

Posted by Willem Jiang <wi...@gmail.com>.
OK, I managed to reproduce the error.
I will keep digging the code to see if I can find a solution here.

--  
Willem Jiang

Red Hat, Inc.
Web: http://www.redhat.com
Blog: http://willemjiang.blogspot.com (English)
http://jnn.iteye.com (Chinese)
Twitter: willemjiang  
Weibo: 姜宁willem



On May 6, 2014 at 1:23:30 AM, Tim Dudgeon (tdudgeon.ml@gmail.com) wrote:
> I don't think that's the problem. I added long sleep before shutting
> down the camel context and still get the error (my test file is very
> small so gets processed almost instantly).
> So think it must be something else closing the stream prematurely.
>  
> Tim
>  
> On 05/05/2014 02:50, Willem Jiang wrote:
> > The Stream is closed by other thread.
> > I think you need to let the main thread sleep for few minutes to let the camel route finish  
> the processing of the message.
> >
> > --
> > Willem Jiang
> >
> > Red Hat, Inc.
> > Web: http://www.redhat.com
> > Blog: http://willemjiang.blogspot.com (English)
> > http://jnn.iteye.com (Chinese)
> > Twitter: willemjiang
> > Weibo: 姜宁willem
> >
> >
> >
> > On May 4, 2014 at 11:26:27 PM, Tim Dudgeon (tdudgeon.ml@gmail.com) wrote:
> >> I'm trying to use the CSV data format to handle a large file.
> >> Thinking I should use CsvDataFormat.setLazyLoad(true)
> >> I sort of got it working but it blows up on the last line of the file
> >> with a java.io.IOException: Stream closed exception.
> >> Is this a bug, or am I dong it wrong? This is with Camel 2.13.0
> >>
> >> Here is example of what I'm doing:
> >>
> >> CsvDataFormat csv = new CsvDataFormat()
> >> csv.setDelimiter(' ')
> >> csv.setSkipFirstLine(false)
> >> csv.setLazyLoad(true)
> >>
> >> CamelContext camelContext = new DefaultCamelContext()
> >> camelContext.addRoutes(new RouteBuilder() {
> >> def void configure() {
> >> from('direct:start')
> >> .unmarshal(csv)
> >> .split(body()).streaming()
> >> .log('row: ${body}')
> >> }
> >> })
> >> camelContext.start()
> >>
> >> ProducerTemplate t = camelContext.createProducerTemplate()
> >> t.sendBody('direct:start', new File('/Users/timbo/data/test.txt'))
> >>
> >> camelContext.stop()
> >>
> >> And here is exception.
> >>
> >> java.lang.IllegalStateException: java.io.IOException: Stream closed
> >> at
> >> org.apache.camel.dataformat.csv.CsvIterator.next(CsvIterator.java:61)  
> >> at
> >> org.apache.camel.processor.Splitter$SplitterIterable$1.next(Splitter.java:170)  
> >> at
> >> org.apache.camel.processor.Splitter$SplitterIterable$1.next(Splitter.java:146)  
> >> at
> >> org.apache.camel.processor.MulticastProcessor.doProcessSequential(MulticastProcessor.java:502)  
> >> at
> >> org.apache.camel.processor.MulticastProcessor.process(MulticastProcessor.java:215)  
> >> at org.apache.camel.processor.Splitter.process(Splitter.java:98)
> >> at
> >> org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:72)  
> >> at
> >> org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:398)  
> >> at
> >> org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:191)  
> >> at org.apache.camel.processor.Pipeline.process(Pipeline.java:118)
> >> at org.apache.camel.processor.Pipeline.process(Pipeline.java:80)
> >> at
> >> org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:191)  
> >> at
> >> org.apache.camel.component.direct.DirectProducer.process(DirectProducer.java:51)  
> >> at
> >> org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:191)  
> >> at
> >> org.apache.camel.processor.UnitOfWorkProducer.process(UnitOfWorkProducer.java:73)  
> >> at
> >> org.apache.camel.impl.ProducerCache$2.doInProducer(ProducerCache.java:378)  
> >> at
> >> org.apache.camel.impl.ProducerCache$2.doInProducer(ProducerCache.java:346)  
> >> at
> >> org.apache.camel.impl.ProducerCache.doInProducer(ProducerCache.java:242)  
> >> at
> >> org.apache.camel.impl.ProducerCache.sendExchange(ProducerCache.java:346)  
> >> at org.apache.camel.impl.ProducerCache.send(ProducerCache.java:184)
> >> at
> >> org.apache.camel.impl.DefaultProducerTemplate.send(DefaultProducerTemplate.java:124)  
> >> at
> >> org.apache.camel.impl.DefaultProducerTemplate.sendBody(DefaultProducerTemplate.java:137)  
> >> at
> >> org.apache.camel.impl.DefaultProducerTemplate.sendBody(DefaultProducerTemplate.java:144)  
> >> at org.apache.camel.ProducerTemplate$sendBody.call(Unknown Source)
> >>
> >> Thanks
> >> Tim
> >>
> >>
>  
>

Re: lazyLoad with CSV blows up on last line?

Posted by Tim Dudgeon <td...@gmail.com>.
I don't think that's the problem. I added long sleep before shutting 
down the camel context and still get the error (my test file is very 
small so gets processed almost instantly).
So think it must be something else closing the stream prematurely.

Tim

On 05/05/2014 02:50, Willem Jiang wrote:
> The Stream is closed by other thread.
> I think you need to let the main thread sleep for few minutes to let the camel route finish the processing of the message.
>
> --
> Willem Jiang
>
> Red Hat, Inc.
> Web: http://www.redhat.com
> Blog: http://willemjiang.blogspot.com (English)
> http://jnn.iteye.com (Chinese)
> Twitter: willemjiang
> Weibo: 姜宁willem
>
>
>
> On May 4, 2014 at 11:26:27 PM, Tim Dudgeon (tdudgeon.ml@gmail.com) wrote:
>> I'm trying to use the CSV data format to handle a large file.
>> Thinking I should use CsvDataFormat.setLazyLoad(true)
>> I sort of got it working but it blows up on the last line of the file
>> with a java.io.IOException: Stream closed exception.
>> Is this a bug, or am I dong it wrong? This is with Camel 2.13.0
>>   
>> Here is example of what I'm doing:
>>   
>> CsvDataFormat csv = new CsvDataFormat()
>> csv.setDelimiter(' ')
>> csv.setSkipFirstLine(false)
>> csv.setLazyLoad(true)
>>   
>> CamelContext camelContext = new DefaultCamelContext()
>> camelContext.addRoutes(new RouteBuilder() {
>> def void configure() {
>> from('direct:start')
>> .unmarshal(csv)
>> .split(body()).streaming()
>> .log('row: ${body}')
>> }
>> })
>> camelContext.start()
>>   
>> ProducerTemplate t = camelContext.createProducerTemplate()
>> t.sendBody('direct:start', new File('/Users/timbo/data/test.txt'))
>>   
>> camelContext.stop()
>>   
>> And here is exception.
>>   
>> java.lang.IllegalStateException: java.io.IOException: Stream closed
>> at
>> org.apache.camel.dataformat.csv.CsvIterator.next(CsvIterator.java:61)
>> at
>> org.apache.camel.processor.Splitter$SplitterIterable$1.next(Splitter.java:170)
>> at
>> org.apache.camel.processor.Splitter$SplitterIterable$1.next(Splitter.java:146)
>> at
>> org.apache.camel.processor.MulticastProcessor.doProcessSequential(MulticastProcessor.java:502)
>> at
>> org.apache.camel.processor.MulticastProcessor.process(MulticastProcessor.java:215)
>> at org.apache.camel.processor.Splitter.process(Splitter.java:98)
>> at
>> org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:72)
>> at
>> org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:398)
>> at
>> org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:191)
>> at org.apache.camel.processor.Pipeline.process(Pipeline.java:118)
>> at org.apache.camel.processor.Pipeline.process(Pipeline.java:80)
>> at
>> org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:191)
>> at
>> org.apache.camel.component.direct.DirectProducer.process(DirectProducer.java:51)
>> at
>> org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:191)
>> at
>> org.apache.camel.processor.UnitOfWorkProducer.process(UnitOfWorkProducer.java:73)
>> at
>> org.apache.camel.impl.ProducerCache$2.doInProducer(ProducerCache.java:378)
>> at
>> org.apache.camel.impl.ProducerCache$2.doInProducer(ProducerCache.java:346)
>> at
>> org.apache.camel.impl.ProducerCache.doInProducer(ProducerCache.java:242)
>> at
>> org.apache.camel.impl.ProducerCache.sendExchange(ProducerCache.java:346)
>> at org.apache.camel.impl.ProducerCache.send(ProducerCache.java:184)
>> at
>> org.apache.camel.impl.DefaultProducerTemplate.send(DefaultProducerTemplate.java:124)
>> at
>> org.apache.camel.impl.DefaultProducerTemplate.sendBody(DefaultProducerTemplate.java:137)
>> at
>> org.apache.camel.impl.DefaultProducerTemplate.sendBody(DefaultProducerTemplate.java:144)
>> at org.apache.camel.ProducerTemplate$sendBody.call(Unknown Source)
>>   
>> Thanks
>> Tim
>>   
>>

Re: lazyLoad with CSV blows up on last line?

Posted by Willem Jiang <wi...@gmail.com>.
The Stream is closed by other thread.
I think you need to let the main thread sleep for few minutes to let the camel route finish the processing of the message.

--  
Willem Jiang

Red Hat, Inc.
Web: http://www.redhat.com
Blog: http://willemjiang.blogspot.com (English)
http://jnn.iteye.com (Chinese)
Twitter: willemjiang  
Weibo: 姜宁willem



On May 4, 2014 at 11:26:27 PM, Tim Dudgeon (tdudgeon.ml@gmail.com) wrote:
> I'm trying to use the CSV data format to handle a large file.
> Thinking I should use CsvDataFormat.setLazyLoad(true)
> I sort of got it working but it blows up on the last line of the file
> with a java.io.IOException: Stream closed exception.
> Is this a bug, or am I dong it wrong? This is with Camel 2.13.0
>  
> Here is example of what I'm doing:
>  
> CsvDataFormat csv = new CsvDataFormat()
> csv.setDelimiter(' ')
> csv.setSkipFirstLine(false)
> csv.setLazyLoad(true)
>  
> CamelContext camelContext = new DefaultCamelContext()
> camelContext.addRoutes(new RouteBuilder() {
> def void configure() {
> from('direct:start')
> .unmarshal(csv)
> .split(body()).streaming()
> .log('row: ${body}')
> }
> })
> camelContext.start()
>  
> ProducerTemplate t = camelContext.createProducerTemplate()
> t.sendBody('direct:start', new File('/Users/timbo/data/test.txt'))
>  
> camelContext.stop()
>  
> And here is exception.
>  
> java.lang.IllegalStateException: java.io.IOException: Stream closed
> at
> org.apache.camel.dataformat.csv.CsvIterator.next(CsvIterator.java:61)
> at
> org.apache.camel.processor.Splitter$SplitterIterable$1.next(Splitter.java:170)  
> at
> org.apache.camel.processor.Splitter$SplitterIterable$1.next(Splitter.java:146)  
> at
> org.apache.camel.processor.MulticastProcessor.doProcessSequential(MulticastProcessor.java:502)  
> at
> org.apache.camel.processor.MulticastProcessor.process(MulticastProcessor.java:215)  
> at org.apache.camel.processor.Splitter.process(Splitter.java:98)
> at
> org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:72)  
> at
> org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:398)  
> at
> org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:191)  
> at org.apache.camel.processor.Pipeline.process(Pipeline.java:118)
> at org.apache.camel.processor.Pipeline.process(Pipeline.java:80)
> at
> org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:191)  
> at
> org.apache.camel.component.direct.DirectProducer.process(DirectProducer.java:51)  
> at
> org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:191)  
> at
> org.apache.camel.processor.UnitOfWorkProducer.process(UnitOfWorkProducer.java:73)  
> at
> org.apache.camel.impl.ProducerCache$2.doInProducer(ProducerCache.java:378)  
> at
> org.apache.camel.impl.ProducerCache$2.doInProducer(ProducerCache.java:346)  
> at
> org.apache.camel.impl.ProducerCache.doInProducer(ProducerCache.java:242)  
> at
> org.apache.camel.impl.ProducerCache.sendExchange(ProducerCache.java:346)  
> at org.apache.camel.impl.ProducerCache.send(ProducerCache.java:184)
> at
> org.apache.camel.impl.DefaultProducerTemplate.send(DefaultProducerTemplate.java:124)  
> at
> org.apache.camel.impl.DefaultProducerTemplate.sendBody(DefaultProducerTemplate.java:137)  
> at
> org.apache.camel.impl.DefaultProducerTemplate.sendBody(DefaultProducerTemplate.java:144)  
> at org.apache.camel.ProducerTemplate$sendBody.call(Unknown Source)
>  
> Thanks
> Tim
>  
>