You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@camel.apache.org by shirazi <me...@farhad.eu> on 2009/11/27 12:55:21 UTC
camel casts exception when there is nothing to fetch from the ftp
endpoint
Hi,
Please consider the following route:
from("timer://someTrigger?delay=30000&fixedRate=true&period=300000").
pollEnrich("ftp://cmoreFtp?binary=true&localWorkDirectory=/tmp&idempotent=true&idempotentRepository=#caStreams",
10000).
to("file:///tmp/vw/cmore?tempPrefix=.inprogress");
The pollEnrich cast a NullPointerException in camel-2.0, if there is nothing
to fetch from the ftp. I got rid of the exception by a
"try-catch(NullpointerExcpetion.class).stop()" around the route. Now in
camel-2.1-SNAPSHOT, I get the following exception on the "to" endpoint:
=================
12:37:06,899 INFO [Tracer] >>> -->
file:///tmp/vw/cmore?tempPrefix=.inprogress_, Pattern:InOnly,
Headers:{firedTime=Fri Nov 27 12:36:56 CET 2009},
Exception:org.apache.camel.component.file.GenericFileOperationFailedException:
Cannot store file:
/tmp/vw/cmore/.inprogress_02216f38-8ffc-4aec-b638-27229d857075
12:37:06,899 INFO [Tracer] >>> -->
file:///tmp/vw/cmore?tempPrefix=.inprogress_, Pattern:InOnly,
Headers:{firedTime=Fri Nov 27 12:36:56 CET 2009},
Exception:org.apache.camel.component.file.GenericFileOperationFailedException:
Cannot store file:
/tmp/vw/cmore/.inprogress_02216f38-8ffc-4aec-b638-27229d857075
12:37:06,899 ERROR [TimerConsumer] Cannot store file:
/tmp/vw/cmore/.inprogress_02216f38-8ffc-4aec-b638-27229d857075
org.apache.camel.component.file.GenericFileOperationFailedException: Cannot
store file: /tmp/vw/cmore/.inprogress_02216f38-8ffc-4aec-b638-27229d857075
at
org.apache.camel.component.file.FileOperations.storeFile(FileOperations.java:204)
at
org.apache.camel.component.file.GenericFileProducer.writeFile(GenericFileProducer.java:184)
at
org.apache.camel.component.file.GenericFileProducer.processExchange(GenericFileProducer.java:126)
at
org.apache.camel.component.file.GenericFileProducer.process(GenericFileProducer.java:57)
at
org.apache.camel.processor.SendProcessor$1.doInProducer(SendProcessor.java:97)
at
org.apache.camel.processor.SendProcessor$1.doInProducer(SendProcessor.java:95)
at org.apache.camel.impl.ProducerCache.doInProducer(ProducerCache.java:146)
at
org.apache.camel.processor.SendProcessor.doProcess(SendProcessor.java:94)
at org.apache.camel.processor.SendProcessor.process(SendProcessor.java:82)
at
org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:67)
at
org.apache.camel.processor.DelegateProcessor.processNext(DelegateProcessor.java:53)
at
org.apache.camel.processor.DelegateProcessor.proceed(DelegateProcessor.java:82)
at
org.apache.camel.processor.interceptor.TraceInterceptor.process(TraceInterceptor.java:162)
at
org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:67)
at
org.apache.camel.processor.DefaultChannel.process(DefaultChannel.java:206)
at org.apache.camel.processor.Pipeline.process(Pipeline.java:74)
at org.apache.camel.processor.TryProcessor.process(TryProcessor.java:63)
at
org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:67)
at
org.apache.camel.processor.DelegateProcessor.processNext(DelegateProcessor.java:53)
at
org.apache.camel.processor.DelegateProcessor.proceed(DelegateProcessor.java:82)
at
org.apache.camel.processor.interceptor.TraceInterceptor.process(TraceInterceptor.java:162)
at
org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:67)
at
org.apache.camel.processor.DefaultChannel.process(DefaultChannel.java:206)
at
org.apache.camel.processor.UnitOfWorkProcessor.processNext(UnitOfWorkProcessor.java:54)
at
org.apache.camel.processor.DelegateProcessor.process(DelegateProcessor.java:48)
at
org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:67)
at
org.apache.camel.component.timer.TimerConsumer.sendTimerExchange(TimerConsumer.java:103)
at
org.apache.camel.component.timer.TimerConsumer$1.run(TimerConsumer.java:50)
at java.util.TimerThread.mainLoop(Timer.java:512)
at java.util.TimerThread.run(Timer.java:462)
Caused by: org.apache.camel.InvalidPayloadException: No body available of
type: java.io.InputStream on: Message: [Body is null]. Caused by: No type
converter available to convert from type: null to the required type:
java.io.InputStream with value null on the exchange: Exchange[Message: [Body
is null]]
at
org.apache.camel.impl.MessageSupport.getMandatoryBody(MessageSupport.java:103)
at
org.apache.camel.util.ExchangeHelper.getMandatoryInBody(ExchangeHelper.java:116)
at
org.apache.camel.component.file.FileOperations.storeFile(FileOperations.java:198)
... 29 more
Caused by: org.apache.camel.NoTypeConversionAvailableException: No type
converter available to convert from type: null to the required type:
java.io.InputStream with value null
at
org.apache.camel.impl.converter.DefaultTypeConverter.mandatoryConvertTo(DefaultTypeConverter.java:126)
at
org.apache.camel.impl.MessageSupport.getMandatoryBody(MessageSupport.java:101)
... 31 more
============
Now, I can remedy the situation by a try-catch block or an interceptor, but
in my opinion this should be considered a bug in camel and it should
silently ignore if there was nothing to fetch from the ftp.
Bests,
Farhad Shiraz
--
View this message in context: http://old.nabble.com/camel-casts-exception-when-there-is-nothing-to-fetch-from-the-ftp-endpoint-tp26541447p26541447.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: camel casts exception when there is nothing to fetch from the ftp
endpoint
Posted by Claus Ibsen <cl...@gmail.com>.
Hi
Well its a bug in UseLatestAggregationStrategy which is shipped in Camel.
Its used by default for pollEnrich, enrich if you dont specify a
custom AggregationStrategy.
Normally you would enrich your existing message with what you poll,
and hence why an AggregationStrategy is needed where you can do this
"merge".
The NPE is fixed on trunk so it will be in the upcoming 2.1 release.
Another workaround is to use your own custom AggregationStrategy and
pass it to pollEnrich.
On Fri, Nov 27, 2009 at 12:55 PM, shirazi <me...@farhad.eu> wrote:
>
> Hi,
> Please consider the following route:
>
> from("timer://someTrigger?delay=30000&fixedRate=true&period=300000").
> pollEnrich("ftp://cmoreFtp?binary=true&localWorkDirectory=/tmp&idempotent=true&idempotentRepository=#caStreams",
> 10000).
> to("file:///tmp/vw/cmore?tempPrefix=.inprogress");
>
> The pollEnrich cast a NullPointerException in camel-2.0, if there is nothing
> to fetch from the ftp. I got rid of the exception by a
> "try-catch(NullpointerExcpetion.class).stop()" around the route. Now in
> camel-2.1-SNAPSHOT, I get the following exception on the "to" endpoint:
> =================
> 12:37:06,899 INFO [Tracer] >>> -->
> file:///tmp/vw/cmore?tempPrefix=.inprogress_, Pattern:InOnly,
> Headers:{firedTime=Fri Nov 27 12:36:56 CET 2009},
> Exception:org.apache.camel.component.file.GenericFileOperationFailedException:
> Cannot store file:
> /tmp/vw/cmore/.inprogress_02216f38-8ffc-4aec-b638-27229d857075
> 12:37:06,899 INFO [Tracer] >>> -->
> file:///tmp/vw/cmore?tempPrefix=.inprogress_, Pattern:InOnly,
> Headers:{firedTime=Fri Nov 27 12:36:56 CET 2009},
> Exception:org.apache.camel.component.file.GenericFileOperationFailedException:
> Cannot store file:
> /tmp/vw/cmore/.inprogress_02216f38-8ffc-4aec-b638-27229d857075
> 12:37:06,899 ERROR [TimerConsumer] Cannot store file:
> /tmp/vw/cmore/.inprogress_02216f38-8ffc-4aec-b638-27229d857075
> org.apache.camel.component.file.GenericFileOperationFailedException: Cannot
> store file: /tmp/vw/cmore/.inprogress_02216f38-8ffc-4aec-b638-27229d857075
> at
> org.apache.camel.component.file.FileOperations.storeFile(FileOperations.java:204)
> at
> org.apache.camel.component.file.GenericFileProducer.writeFile(GenericFileProducer.java:184)
> at
> org.apache.camel.component.file.GenericFileProducer.processExchange(GenericFileProducer.java:126)
> at
> org.apache.camel.component.file.GenericFileProducer.process(GenericFileProducer.java:57)
> at
> org.apache.camel.processor.SendProcessor$1.doInProducer(SendProcessor.java:97)
> at
> org.apache.camel.processor.SendProcessor$1.doInProducer(SendProcessor.java:95)
> at org.apache.camel.impl.ProducerCache.doInProducer(ProducerCache.java:146)
> at
> org.apache.camel.processor.SendProcessor.doProcess(SendProcessor.java:94)
> at org.apache.camel.processor.SendProcessor.process(SendProcessor.java:82)
> at
> org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:67)
> at
> org.apache.camel.processor.DelegateProcessor.processNext(DelegateProcessor.java:53)
> at
> org.apache.camel.processor.DelegateProcessor.proceed(DelegateProcessor.java:82)
> at
> org.apache.camel.processor.interceptor.TraceInterceptor.process(TraceInterceptor.java:162)
> at
> org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:67)
> at
> org.apache.camel.processor.DefaultChannel.process(DefaultChannel.java:206)
> at org.apache.camel.processor.Pipeline.process(Pipeline.java:74)
> at org.apache.camel.processor.TryProcessor.process(TryProcessor.java:63)
> at
> org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:67)
> at
> org.apache.camel.processor.DelegateProcessor.processNext(DelegateProcessor.java:53)
> at
> org.apache.camel.processor.DelegateProcessor.proceed(DelegateProcessor.java:82)
> at
> org.apache.camel.processor.interceptor.TraceInterceptor.process(TraceInterceptor.java:162)
> at
> org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:67)
> at
> org.apache.camel.processor.DefaultChannel.process(DefaultChannel.java:206)
> at
> org.apache.camel.processor.UnitOfWorkProcessor.processNext(UnitOfWorkProcessor.java:54)
> at
> org.apache.camel.processor.DelegateProcessor.process(DelegateProcessor.java:48)
> at
> org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:67)
> at
> org.apache.camel.component.timer.TimerConsumer.sendTimerExchange(TimerConsumer.java:103)
> at
> org.apache.camel.component.timer.TimerConsumer$1.run(TimerConsumer.java:50)
> at java.util.TimerThread.mainLoop(Timer.java:512)
> at java.util.TimerThread.run(Timer.java:462)
> Caused by: org.apache.camel.InvalidPayloadException: No body available of
> type: java.io.InputStream on: Message: [Body is null]. Caused by: No type
> converter available to convert from type: null to the required type:
> java.io.InputStream with value null on the exchange: Exchange[Message: [Body
> is null]]
> at
> org.apache.camel.impl.MessageSupport.getMandatoryBody(MessageSupport.java:103)
> at
> org.apache.camel.util.ExchangeHelper.getMandatoryInBody(ExchangeHelper.java:116)
> at
> org.apache.camel.component.file.FileOperations.storeFile(FileOperations.java:198)
> ... 29 more
> Caused by: org.apache.camel.NoTypeConversionAvailableException: No type
> converter available to convert from type: null to the required type:
> java.io.InputStream with value null
> at
> org.apache.camel.impl.converter.DefaultTypeConverter.mandatoryConvertTo(DefaultTypeConverter.java:126)
> at
> org.apache.camel.impl.MessageSupport.getMandatoryBody(MessageSupport.java:101)
> ... 31 more
> ============
> Now, I can remedy the situation by a try-catch block or an interceptor, but
> in my opinion this should be considered a bug in camel and it should
> silently ignore if there was nothing to fetch from the ftp.
>
> Bests,
> Farhad Shiraz
>
> --
> View this message in context: http://old.nabble.com/camel-casts-exception-when-there-is-nothing-to-fetch-from-the-ftp-endpoint-tp26541447p26541447.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>
>
--
Claus Ibsen
Apache Camel Committer
Author of Camel in Action: http://www.manning.com/ibsen/
Open Source Integration: http://fusesource.com
Blog: http://davsclaus.blogspot.com/
Twitter: http://twitter.com/davsclaus
Re: camel casts exception when there is nothing to fetch from the ftp
endpoint
Posted by Claus Ibsen <cl...@gmail.com>.
On Fri, Nov 27, 2009 at 8:17 PM, shirazi <me...@farhad.eu> wrote:
>
> Sorry, my mistake. Using the latest build from trunk, it works just fine. I
> just tried it with a couple of files of 3 and 4 Gigs size.
>
Fantastic. Glad it worked as the code was inspired by your NIO code so
I thought it was odd if it didnt work.
Glad we got that fixed now and thanks for helping out.
> Thanks,
> -Farhad S.
>
>
> Claus Ibsen-2 wrote:
>>
>> On Fri, Nov 27, 2009 at 4:18 PM, shirazi <me...@farhad.eu> wrote:
>>>
>>> Thanks Claus,
>>> I just downloaded the new trunk, and I think there's a new bug in it. I
>>> just
>>> tried the file component with a couple of files around 3-400 MBs and it
>>> works fine. As soon as I try it with a file around 3 Gigs, i get the
>>> following exception (??!!):
>>>
>>
>> That is odd as the stacktrace is off by -1.
>> And you do a full svn up and a mvn clean install -Dtest=false to build
>> all source from scratch?
>>
>>
>> If you enable TRACE logging for: org.apache.camel.component.file
>>
>> You should see this log line when it uses the NIO stuff
>>
>> if (LOG.isTraceEnabled()) {
>> LOG.trace("Using FileChannel to transfer from: " + in
>> + " to: " + out);
>> }
>>
>>
>> I have committed a fix for the NPE in that stacktrace. But the was
>> some other exception thrown that got lost.
>>
>>
>> commited rev: 884894.
>>
>>
>>
>>> 16:13:48,830 ERROR [workflow] null while processing exchange:
>>> Exchange[GenericFileMessage with file: GenericFile[/tmp/vw/cmore/tv5.ts]]
>>> java.lang.NullPointerException
>>> at
>>> org.apache.camel.impl.OnExceptionRouteNode.getLabel(OnExceptionRouteNode.java:41)
>>> at
>>> org.apache.camel.processor.interceptor.DefaultTraceFormatter.getNodeMessage(DefaultTraceFormatter.java:233)
>>> at
>>> org.apache.camel.processor.interceptor.DefaultTraceFormatter.extractBreadCrumb(DefaultTraceFormatter.java:282)
>>> at
>>> org.apache.camel.processor.interceptor.DefaultTraceFormatter.format(DefaultTraceFormatter.java:55)
>>> at
>>> org.apache.camel.processor.interceptor.TraceInterceptor.format(TraceInterceptor.java:232)
>>> at org.apache.camel.processor.Logger.logMessage(Logger.java:283)
>>> at org.apache.camel.processor.Logger.process(Logger.java:166)
>>> at
>>> org.apache.camel.processor.interceptor.TraceInterceptor.logException(TraceInterceptor.java:325)
>>> at
>>> org.apache.camel.processor.interceptor.TraceInterceptor.process(TraceInterceptor.java:171)
>>> at
>>> org.apache.camel.processor.LoggingErrorHandler.process(LoggingErrorHandler.java:58)
>>> at
>>> org.apache.camel.processor.DefaultChannel.process(DefaultChannel.java:206)
>>> at
>>> org.apache.camel.processor.DelegateProcessor.processNext(DelegateProcessor.java:53)
>>> at
>>> org.apache.camel.processor.ChoiceProcessor.process(ChoiceProcessor.java:51)
>>> at
>>> org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:67)
>>> at
>>> org.apache.camel.processor.DelegateProcessor.processNext(DelegateProcessor.java:53)
>>> at
>>> org.apache.camel.processor.DelegateProcessor.proceed(DelegateProcessor.java:82)
>>> at
>>> org.apache.camel.processor.interceptor.TraceInterceptor.process(TraceInterceptor.java:162)
>>> at
>>> org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:67)
>>> at
>>> org.apache.camel.processor.LoggingErrorHandler.process(LoggingErrorHandler.java:58)
>>> at
>>> org.apache.camel.processor.DefaultChannel.process(DefaultChannel.java:206)
>>> at
>>> org.apache.camel.processor.UnitOfWorkProcessor.processNext(UnitOfWorkProcessor.java:70)
>>> at
>>> org.apache.camel.processor.DelegateProcessor.process(DelegateProcessor.java:48)
>>> at
>>> org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:67)
>>> at
>>> org.apache.camel.component.direct.DirectProducer.process(DirectProducer.java:45)
>>> at
>>> org.apache.camel.processor.SendProcessor$1.doInProducer(SendProcessor.java:97)
>>> at
>>> org.apache.camel.processor.SendProcessor$1.doInProducer(SendProcessor.java:95)
>>> at
>>> org.apache.camel.impl.ProducerCache.doInProducer(ProducerCache.java:146)
>>> at
>>> org.apache.camel.processor.SendProcessor.doProcess(SendProcessor.java:94)
>>> at
>>> org.apache.camel.processor.SendProcessor.process(SendProcessor.java:82)
>>> at
>>> org.apache.camel.processor.DelegateProcessor.processNext(DelegateProcessor.java:53)
>>> at
>>> org.apache.camel.processor.DelegateProcessor.proceed(DelegateProcessor.java:82)
>>> at
>>> org.apache.camel.processor.interceptor.TraceInterceptor.process(TraceInterceptor.java:162)
>>> at
>>> org.apache.camel.processor.LoggingErrorHandler.process(LoggingErrorHandler.java:58)
>>> at
>>> org.apache.camel.processor.DefaultChannel.process(DefaultChannel.java:206)
>>> at org.apache.camel.processor.Pipeline.process(Pipeline.java:74)
>>> at
>>> org.apache.camel.processor.ChoiceProcessor.process(ChoiceProcessor.java:56)
>>> at
>>> org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:67)
>>> at
>>> org.apache.camel.processor.DelegateProcessor.processNext(DelegateProcessor.java:53)
>>> at
>>> org.apache.camel.processor.DelegateProcessor.proceed(DelegateProcessor.java:82)
>>> at
>>> org.apache.camel.processor.interceptor.TraceInterceptor.process(TraceInterceptor.java:162)
>>> at
>>> org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:67)
>>> at
>>> org.apache.camel.processor.LoggingErrorHandler.process(LoggingErrorHandler.java:58)
>>> at
>>> org.apache.camel.processor.DefaultChannel.process(DefaultChannel.java:206)
>>> at
>>> org.apache.camel.processor.UnitOfWorkProcessor.processNext(UnitOfWorkProcessor.java:54)
>>> at
>>> org.apache.camel.processor.DelegateProcessor.process(DelegateProcessor.java:48)
>>> at
>>> org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:67)
>>> at
>>> org.apache.camel.component.file.GenericFileConsumer.processExchange(GenericFileConsumer.java:201)
>>> at
>>> org.apache.camel.component.file.GenericFileConsumer.processBatch(GenericFileConsumer.java:122)
>>> at
>>> org.apache.camel.component.file.GenericFileConsumer.poll(GenericFileConsumer.java:95)
>>> at
>>> org.apache.camel.impl.ScheduledPollConsumer.run(ScheduledPollConsumer.java:108)
>>> at
>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:417)
>>> at
>>> java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:280)
>>> at
>>> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:135)
>>> at
>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:65)
>>> at
>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:142)
>>> at
>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:166)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
>>> at java.lang.Thread.run(Thread.java:613)
>>>
>>>
>>>
>>>
>>> Claus Ibsen-2 wrote:
>>>>
>>>> On Fri, Nov 27, 2009 at 2:34 PM, shirazi <me...@farhad.eu> wrote:
>>>>>
>>>>> Oh, sorry. that's right, it is 64 MB. and I tried with 16, 32, 64 and
>>>>> 128
>>>>> MB.
>>>>> I will try the buffer size option of the file-comp. to see if it solves
>>>>> the
>>>>> problem.
>>>>>
>>>>
>>>> I have just committed the fix to camel trunk.
>>>>
>>>> The old code will still have the 2gb limit bug no matter what
>>>> bufferSize you use.
>>>> However you should try the latest code in 2.1 to see if that works for
>>>> you.
>>>> http://camel.apache.org/download.html
>>>>
>>>>
>>>>
>>>>> Thanks,
>>>>> Farhad S.
>>>>>
>>>>>
>>>>> Claus Ibsen-2 wrote:
>>>>>>
>>>>>> On Fri, Nov 27, 2009 at 2:26 PM, shirazi <me...@farhad.eu> wrote:
>>>>>>>
>>>>>>> That's probably the case. I've tried with different buffer sizes
>>>>>>> (though
>>>>>>> not
>>>>>>> 0), from 16 to 128 KB. I got the best performance with buffer of
>>>>>>> 64KB,
>>>>>>> on
>>>>>>> Linux Debian with 4 Gigs RAM, and files larger than 3 Gigs.
>>>>>>> That would be great, if one could specify the buffer-size for the
>>>>>>> "file"
>>>>>>> component.
>>>>>>>
>>>>>>
>>>>>> You buffer size was 64 * 1024 * 1024 which is 64 K * K = 64 MB. Or is
>>>>>> there something that play tricks on me?
>>>>>>
>>>>>> The file component already have a bufferSize option with a default of
>>>>>> 128kb
>>>>>> http://camel.apache.org/file2.html
>>>>>>
>>>>>> So you can use that to set what you want to use.
>>>>>>
>>>>>>
>>>>>>
>>>>>>> -Farhad S.
>>>>>>>
>>>>>>>
>>>>>>> Claus Ibsen-2 wrote:
>>>>>>>>
>>>>>>>> On Fri, Nov 27, 2009 at 2:00 PM, shirazi <me...@farhad.eu> wrote:
>>>>>>>>>
>>>>>>>>> Sure. I have attached the file.
>>>>>>>>> Actually I use from->to components for transfering files and I
>>>>>>>>> guess
>>>>>>>>> it
>>>>>>>>> copies the file, rather than moving.
>>>>>>>>>
>>>>>>>>
>>>>>>>> Ah makes more sense for copying to leverage NIO.
>>>>>>>>
>>>>>>>> Camel does this also but I can see that it may hit that 2gb limit as
>>>>>>>> it doesnt transfer in a loop using a lower buffer.
>>>>>>>>
>>>>>>>> It was doing it in a one liner
>>>>>>>> in.transferTo(0, in.size(), out);
>>>>>>>>
>>>>>>>> I wonder if you really must do that in a loop to avoid any 2gb limit
>>>>>>>> or the likes?
>>>>>>>>
>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Farhad S.
>>>>>>>>> http://old.nabble.com/file/p26542125/Nio.java Nio.java
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Claus Ibsen-2 wrote:
>>>>>>>>>>
>>>>>>>>>> On Fri, Nov 27, 2009 at 1:37 PM, shirazi <me...@farhad.eu> wrote:
>>>>>>>>>>>
>>>>>>>>>>> Wau... thanks for that. I don't understand how I could miss it,
>>>>>>>>>>> cause
>>>>>>>>>>> i
>>>>>>>>>>> searched a lot for such a feature.
>>>>>>>>>>>
>>>>>>>>>>> Another thing that I have experienced in 2.0, is that the "file"
>>>>>>>>>>> component
>>>>>>>>>>> is unable to move/copy files larger than 2Gigs, such large files
>>>>>>>>>>> are
>>>>>>>>>>> cut
>>>>>>>>>>> around 2 Gigs limit. Now, I'm not sure if it has anything to do
>>>>>>>>>>> with
>>>>>>>>>>> the
>>>>>>>>>>> OS
>>>>>>>>>>> or the filesystems or what. I have seen it both in Linux (debian)
>>>>>>>>>>> and
>>>>>>>>>>> Mac
>>>>>>>>>>> OS
>>>>>>>>>>> X. And I don't understand either, why the file component does not
>>>>>>>>>>> use
>>>>>>>>>>> the
>>>>>>>>>>> NIO api or at least we don't have an option to tell it use the
>>>>>>>>>>> NIO
>>>>>>>>>>> api
>>>>>>>>>>> instead.
>>>>>>>>>>> Anyway, for now I've solved the problem by using a smiple
>>>>>>>>>>> component
>>>>>>>>>>> instead
>>>>>>>>>>> of the "file" comp. to move/copy files using the NIO api. It is
>>>>>>>>>>> much
>>>>>>>>>>> faster
>>>>>>>>>>> and it has no problem with very large files.
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> For move it uses java.io.File rename and are you saying its not
>>>>>>>>>> optimized for the underlying OS?
>>>>>>>>>> I would assume that it could move a 2gb file in no time if the
>>>>>>>>>> file
>>>>>>>>>> is
>>>>>>>>>> on the same share.
>>>>>>>>>>
>>>>>>>>>> Can you share your code that you are using?
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> Bests,
>>>>>>>>>>> Farhad S.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Claus Ibsen-2 wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> On Fri, Nov 27, 2009 at 1:12 PM, shirazi <me...@farhad.eu> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>> Ok, I guess I should rather provide an AggregationStrategy.
>>>>>>>>>>>>> The reason why I use a timer is that when I use the ftp
>>>>>>>>>>>>> endpoint,
>>>>>>>>>>>>> it
>>>>>>>>>>>>> starts
>>>>>>>>>>>>> to fetch everything that is on the ftp in the first call and
>>>>>>>>>>>>> that's
>>>>>>>>>>>>> not
>>>>>>>>>>>>> what
>>>>>>>>>>>>> i want, because I'm processing files over 3 Gigs and there are
>>>>>>>>>>>>> many
>>>>>>>>>>>>> of
>>>>>>>>>>>>> them.
>>>>>>>>>>>>> Then I figured that by using a timer, i could have ftp to
>>>>>>>>>>>>> download
>>>>>>>>>>>>> one
>>>>>>>>>>>>> file
>>>>>>>>>>>>> at a time!!
>>>>>>>>>>>>> What I basically need is to somehow tell the ftp, how many
>>>>>>>>>>>>> files
>>>>>>>>>>>>> I
>>>>>>>>>>>>> want
>>>>>>>>>>>>> to
>>>>>>>>>>>>> fetch at a time. This is probably doable now in camel-2.1 by
>>>>>>>>>>>>> using
>>>>>>>>>>>>> the
>>>>>>>>>>>>> "ThrottlingInflightRoutePolicy", which i havn't tried yet.
>>>>>>>>>>>>> Do you have any suggestions?
>>>>>>>>>>>>
>>>>>>>>>>>> Yeah the FTP component extends the File component so you got all
>>>>>>>>>>>> the
>>>>>>>>>>>> options from the file component as well
>>>>>>>>>>>> http://camel.apache.org/file2.html
>>>>>>>>>>>>
>>>>>>>>>>>> And there is a maxMessagesPerPoll you can use to tell how many
>>>>>>>>>>>> files
>>>>>>>>>>>> you want at most to poll. For example = 1 in your use.
>>>>>>>>>>>>
>>>>>>>>>>>> And that option is avail in 2.0 also.
>>>>>>>>>>>>
>>>>>>>>>>>> In fact all these are as they are batch polling consumers
>>>>>>>>>>>> http://camel.apache.org/batch-consumer.html
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Bests,
>>>>>>>>>>>>> Farhad S.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Claus Ibsen-2 wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> BTW why are you using a timer -> ftp when the ftp have a build
>>>>>>>>>>>>>> in
>>>>>>>>>>>>>> timer?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Fri, Nov 27, 2009 at 12:55 PM, shirazi <me...@farhad.eu>
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>> Please consider the following route:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> from("timer://someTrigger?delay=30000&fixedRate=true&period=300000").
>>>>>>>>>>>>>>> pollEnrich("ftp://cmoreFtp?binary=true&localWorkDirectory=/tmp&idempotent=true&idempotentRepository=#caStreams",
>>>>>>>>>>>>>>> 10000).
>>>>>>>>>>>>>>> to("file:///tmp/vw/cmore?tempPrefix=.inprogress");
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> The pollEnrich cast a NullPointerException in camel-2.0, if
>>>>>>>>>>>>>>> there
>>>>>>>>>>>>>>> is
>>>>>>>>>>>>>>> nothing
>>>>>>>>>>>>>>> to fetch from the ftp. I got rid of the exception by a
>>>>>>>>>>>>>>> "try-catch(NullpointerExcpetion.class).stop()" around the
>>>>>>>>>>>>>>> route.
>>>>>>>>>>>>>>> Now
>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>> camel-2.1-SNAPSHOT, I get the following exception on the "to"
>>>>>>>>>>>>>>> endpoint:
>>>>>>>>>>>>>>> =================
>>>>>>>>>>>>>>> 12:37:06,899 INFO [Tracer] >>> -->
>>>>>>>>>>>>>>> file:///tmp/vw/cmore?tempPrefix=.inprogress_, Pattern:InOnly,
>>>>>>>>>>>>>>> Headers:{firedTime=Fri Nov 27 12:36:56 CET 2009},
>>>>>>>>>>>>>>> Exception:org.apache.camel.component.file.GenericFileOperationFailedException:
>>>>>>>>>>>>>>> Cannot store file:
>>>>>>>>>>>>>>> /tmp/vw/cmore/.inprogress_02216f38-8ffc-4aec-b638-27229d857075
>>>>>>>>>>>>>>> 12:37:06,899 INFO [Tracer] >>> -->
>>>>>>>>>>>>>>> file:///tmp/vw/cmore?tempPrefix=.inprogress_, Pattern:InOnly,
>>>>>>>>>>>>>>> Headers:{firedTime=Fri Nov 27 12:36:56 CET 2009},
>>>>>>>>>>>>>>> Exception:org.apache.camel.component.file.GenericFileOperationFailedException:
>>>>>>>>>>>>>>> Cannot store file:
>>>>>>>>>>>>>>> /tmp/vw/cmore/.inprogress_02216f38-8ffc-4aec-b638-27229d857075
>>>>>>>>>>>>>>> 12:37:06,899 ERROR [TimerConsumer] Cannot store file:
>>>>>>>>>>>>>>> /tmp/vw/cmore/.inprogress_02216f38-8ffc-4aec-b638-27229d857075
>>>>>>>>>>>>>>> org.apache.camel.component.file.GenericFileOperationFailedException:
>>>>>>>>>>>>>>> Cannot
>>>>>>>>>>>>>>> store file:
>>>>>>>>>>>>>>> /tmp/vw/cmore/.inprogress_02216f38-8ffc-4aec-b638-27229d857075
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> org.apache.camel.component.file.FileOperations.storeFile(FileOperations.java:204)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> org.apache.camel.component.file.GenericFileProducer.writeFile(GenericFileProducer.java:184)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> org.apache.camel.component.file.GenericFileProducer.processExchange(GenericFileProducer.java:126)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> org.apache.camel.component.file.GenericFileProducer.process(GenericFileProducer.java:57)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> org.apache.camel.processor.SendProcessor$1.doInProducer(SendProcessor.java:97)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> org.apache.camel.processor.SendProcessor$1.doInProducer(SendProcessor.java:95)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> org.apache.camel.impl.ProducerCache.doInProducer(ProducerCache.java:146)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> org.apache.camel.processor.SendProcessor.doProcess(SendProcessor.java:94)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> org.apache.camel.processor.SendProcessor.process(SendProcessor.java:82)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:67)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> org.apache.camel.processor.DelegateProcessor.processNext(DelegateProcessor.java:53)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> org.apache.camel.processor.DelegateProcessor.proceed(DelegateProcessor.java:82)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> org.apache.camel.processor.interceptor.TraceInterceptor.process(TraceInterceptor.java:162)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:67)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> org.apache.camel.processor.DefaultChannel.process(DefaultChannel.java:206)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> org.apache.camel.processor.Pipeline.process(Pipeline.java:74)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> org.apache.camel.processor.TryProcessor.process(TryProcessor.java:63)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:67)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> org.apache.camel.processor.DelegateProcessor.processNext(DelegateProcessor.java:53)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> org.apache.camel.processor.DelegateProcessor.proceed(DelegateProcessor.java:82)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> org.apache.camel.processor.interceptor.TraceInterceptor.process(TraceInterceptor.java:162)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:67)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> org.apache.camel.processor.DefaultChannel.process(DefaultChannel.java:206)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> org.apache.camel.processor.UnitOfWorkProcessor.processNext(UnitOfWorkProcessor.java:54)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> org.apache.camel.processor.DelegateProcessor.process(DelegateProcessor.java:48)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:67)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> org.apache.camel.component.timer.TimerConsumer.sendTimerExchange(TimerConsumer.java:103)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> org.apache.camel.component.timer.TimerConsumer$1.run(TimerConsumer.java:50)
>>>>>>>>>>>>>>> at java.util.TimerThread.mainLoop(Timer.java:512)
>>>>>>>>>>>>>>> at java.util.TimerThread.run(Timer.java:462)
>>>>>>>>>>>>>>> Caused by: org.apache.camel.InvalidPayloadException: No body
>>>>>>>>>>>>>>> available
>>>>>>>>>>>>>>> of
>>>>>>>>>>>>>>> type: java.io.InputStream on: Message: [Body is null]. Caused
>>>>>>>>>>>>>>> by:
>>>>>>>>>>>>>>> No
>>>>>>>>>>>>>>> type
>>>>>>>>>>>>>>> converter available to convert from type: null to the
>>>>>>>>>>>>>>> required
>>>>>>>>>>>>>>> type:
>>>>>>>>>>>>>>> java.io.InputStream with value null on the exchange:
>>>>>>>>>>>>>>> Exchange[Message:
>>>>>>>>>>>>>>> [Body
>>>>>>>>>>>>>>> is null]]
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> org.apache.camel.impl.MessageSupport.getMandatoryBody(MessageSupport.java:103)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> org.apache.camel.util.ExchangeHelper.getMandatoryInBody(ExchangeHelper.java:116)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> org.apache.camel.component.file.FileOperations.storeFile(FileOperations.java:198)
>>>>>>>>>>>>>>> ... 29 more
>>>>>>>>>>>>>>> Caused by:
>>>>>>>>>>>>>>> org.apache.camel.NoTypeConversionAvailableException:
>>>>>>>>>>>>>>> No
>>>>>>>>>>>>>>> type
>>>>>>>>>>>>>>> converter available to convert from type: null to the
>>>>>>>>>>>>>>> required
>>>>>>>>>>>>>>> type:
>>>>>>>>>>>>>>> java.io.InputStream with value null
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> org.apache.camel.impl.converter.DefaultTypeConverter.mandatoryConvertTo(DefaultTypeConverter.java:126)
>>>>>>>>>>>>>>> at
>>>>>>>>>>>>>>> org.apache.camel.impl.MessageSupport.getMandatoryBody(MessageSupport.java:101)
>>>>>>>>>>>>>>> ... 31 more
>>>>>>>>>>>>>>> ============
>>>>>>>>>>>>>>> Now, I can remedy the situation by a try-catch block or an
>>>>>>>>>>>>>>> interceptor,
>>>>>>>>>>>>>>> but
>>>>>>>>>>>>>>> in my opinion this should be considered a bug in camel and it
>>>>>>>>>>>>>>> should
>>>>>>>>>>>>>>> silently ignore if there was nothing to fetch from the ftp.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Bests,
>>>>>>>>>>>>>>> Farhad Shiraz
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>> View this message in context:
>>>>>>>>>>>>>>> http://old.nabble.com/camel-casts-exception-when-there-is-nothing-to-fetch-from-the-ftp-endpoint-tp26541447p26541447.html
>>>>>>>>>>>>>>> Sent from the Camel - Users mailing list archive at
>>>>>>>>>>>>>>> Nabble.com.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> --
>>>>>>>>>>>>>> Claus Ibsen
>>>>>>>>>>>>>> Apache Camel Committer
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Author of Camel in Action: http://www.manning.com/ibsen/
>>>>>>>>>>>>>> Open Source Integration: http://fusesource.com
>>>>>>>>>>>>>> Blog: http://davsclaus.blogspot.com/
>>>>>>>>>>>>>> Twitter: http://twitter.com/davsclaus
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>>> View this message in context:
>>>>>>>>>>>>> http://old.nabble.com/camel-casts-exception-when-there-is-nothing-to-fetch-from-the-ftp-endpoint-tp26541447p26541650.html
>>>>>>>>>>>>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> Claus Ibsen
>>>>>>>>>>>> Apache Camel Committer
>>>>>>>>>>>>
>>>>>>>>>>>> Author of Camel in Action: http://www.manning.com/ibsen/
>>>>>>>>>>>> Open Source Integration: http://fusesource.com
>>>>>>>>>>>> Blog: http://davsclaus.blogspot.com/
>>>>>>>>>>>> Twitter: http://twitter.com/davsclaus
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> View this message in context:
>>>>>>>>>>> http://old.nabble.com/camel-casts-exception-when-there-is-nothing-to-fetch-from-the-ftp-endpoint-tp26541447p26541877.html
>>>>>>>>>>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Claus Ibsen
>>>>>>>>>> Apache Camel Committer
>>>>>>>>>>
>>>>>>>>>> Author of Camel in Action: http://www.manning.com/ibsen/
>>>>>>>>>> Open Source Integration: http://fusesource.com
>>>>>>>>>> Blog: http://davsclaus.blogspot.com/
>>>>>>>>>> Twitter: http://twitter.com/davsclaus
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> View this message in context:
>>>>>>>>> http://old.nabble.com/camel-casts-exception-when-there-is-nothing-to-fetch-from-the-ftp-endpoint-tp26541447p26542125.html
>>>>>>>>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Claus Ibsen
>>>>>>>> Apache Camel Committer
>>>>>>>>
>>>>>>>> Author of Camel in Action: http://www.manning.com/ibsen/
>>>>>>>> Open Source Integration: http://fusesource.com
>>>>>>>> Blog: http://davsclaus.blogspot.com/
>>>>>>>> Twitter: http://twitter.com/davsclaus
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> View this message in context:
>>>>>>> http://old.nabble.com/camel-casts-exception-when-there-is-nothing-to-fetch-from-the-ftp-endpoint-tp26541447p26542396.html
>>>>>>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Claus Ibsen
>>>>>> Apache Camel Committer
>>>>>>
>>>>>> Author of Camel in Action: http://www.manning.com/ibsen/
>>>>>> Open Source Integration: http://fusesource.com
>>>>>> Blog: http://davsclaus.blogspot.com/
>>>>>> Twitter: http://twitter.com/davsclaus
>>>>>>
>>>>>>
>>>>>
>>>>> --
>>>>> View this message in context:
>>>>> http://old.nabble.com/camel-casts-exception-when-there-is-nothing-to-fetch-from-the-ftp-endpoint-tp26541447p26542472.html
>>>>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Claus Ibsen
>>>> Apache Camel Committer
>>>>
>>>> Author of Camel in Action: http://www.manning.com/ibsen/
>>>> Open Source Integration: http://fusesource.com
>>>> Blog: http://davsclaus.blogspot.com/
>>>> Twitter: http://twitter.com/davsclaus
>>>>
>>>>
>>>
>>> --
>>> View this message in context:
>>> http://old.nabble.com/camel-casts-exception-when-there-is-nothing-to-fetch-from-the-ftp-endpoint-tp26541447p26543771.html
>>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>>
>>>
>>
>>
>>
>> --
>> Claus Ibsen
>> Apache Camel Committer
>>
>> Author of Camel in Action: http://www.manning.com/ibsen/
>> Open Source Integration: http://fusesource.com
>> Blog: http://davsclaus.blogspot.com/
>> Twitter: http://twitter.com/davsclaus
>>
>>
>
> --
> View this message in context: http://old.nabble.com/camel-casts-exception-when-there-is-nothing-to-fetch-from-the-ftp-endpoint-tp26541447p26545371.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>
>
--
Claus Ibsen
Apache Camel Committer
Author of Camel in Action: http://www.manning.com/ibsen/
Open Source Integration: http://fusesource.com
Blog: http://davsclaus.blogspot.com/
Twitter: http://twitter.com/davsclaus
Re: camel casts exception when there is nothing to fetch from the
ftp endpoint
Posted by shirazi <me...@farhad.eu>.
Sorry, my mistake. Using the latest build from trunk, it works just fine. I
just tried it with a couple of files of 3 and 4 Gigs size.
Thanks,
-Farhad S.
Claus Ibsen-2 wrote:
>
> On Fri, Nov 27, 2009 at 4:18 PM, shirazi <me...@farhad.eu> wrote:
>>
>> Thanks Claus,
>> I just downloaded the new trunk, and I think there's a new bug in it. I
>> just
>> tried the file component with a couple of files around 3-400 MBs and it
>> works fine. As soon as I try it with a file around 3 Gigs, i get the
>> following exception (??!!):
>>
>
> That is odd as the stacktrace is off by -1.
> And you do a full svn up and a mvn clean install -Dtest=false to build
> all source from scratch?
>
>
> If you enable TRACE logging for: org.apache.camel.component.file
>
> You should see this log line when it uses the NIO stuff
>
> if (LOG.isTraceEnabled()) {
> LOG.trace("Using FileChannel to transfer from: " + in
> + " to: " + out);
> }
>
>
> I have committed a fix for the NPE in that stacktrace. But the was
> some other exception thrown that got lost.
>
>
> commited rev: 884894.
>
>
>
>> 16:13:48,830 ERROR [workflow] null while processing exchange:
>> Exchange[GenericFileMessage with file: GenericFile[/tmp/vw/cmore/tv5.ts]]
>> java.lang.NullPointerException
>> at
>> org.apache.camel.impl.OnExceptionRouteNode.getLabel(OnExceptionRouteNode.java:41)
>> at
>> org.apache.camel.processor.interceptor.DefaultTraceFormatter.getNodeMessage(DefaultTraceFormatter.java:233)
>> at
>> org.apache.camel.processor.interceptor.DefaultTraceFormatter.extractBreadCrumb(DefaultTraceFormatter.java:282)
>> at
>> org.apache.camel.processor.interceptor.DefaultTraceFormatter.format(DefaultTraceFormatter.java:55)
>> at
>> org.apache.camel.processor.interceptor.TraceInterceptor.format(TraceInterceptor.java:232)
>> at org.apache.camel.processor.Logger.logMessage(Logger.java:283)
>> at org.apache.camel.processor.Logger.process(Logger.java:166)
>> at
>> org.apache.camel.processor.interceptor.TraceInterceptor.logException(TraceInterceptor.java:325)
>> at
>> org.apache.camel.processor.interceptor.TraceInterceptor.process(TraceInterceptor.java:171)
>> at
>> org.apache.camel.processor.LoggingErrorHandler.process(LoggingErrorHandler.java:58)
>> at
>> org.apache.camel.processor.DefaultChannel.process(DefaultChannel.java:206)
>> at
>> org.apache.camel.processor.DelegateProcessor.processNext(DelegateProcessor.java:53)
>> at
>> org.apache.camel.processor.ChoiceProcessor.process(ChoiceProcessor.java:51)
>> at
>> org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:67)
>> at
>> org.apache.camel.processor.DelegateProcessor.processNext(DelegateProcessor.java:53)
>> at
>> org.apache.camel.processor.DelegateProcessor.proceed(DelegateProcessor.java:82)
>> at
>> org.apache.camel.processor.interceptor.TraceInterceptor.process(TraceInterceptor.java:162)
>> at
>> org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:67)
>> at
>> org.apache.camel.processor.LoggingErrorHandler.process(LoggingErrorHandler.java:58)
>> at
>> org.apache.camel.processor.DefaultChannel.process(DefaultChannel.java:206)
>> at
>> org.apache.camel.processor.UnitOfWorkProcessor.processNext(UnitOfWorkProcessor.java:70)
>> at
>> org.apache.camel.processor.DelegateProcessor.process(DelegateProcessor.java:48)
>> at
>> org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:67)
>> at
>> org.apache.camel.component.direct.DirectProducer.process(DirectProducer.java:45)
>> at
>> org.apache.camel.processor.SendProcessor$1.doInProducer(SendProcessor.java:97)
>> at
>> org.apache.camel.processor.SendProcessor$1.doInProducer(SendProcessor.java:95)
>> at
>> org.apache.camel.impl.ProducerCache.doInProducer(ProducerCache.java:146)
>> at
>> org.apache.camel.processor.SendProcessor.doProcess(SendProcessor.java:94)
>> at
>> org.apache.camel.processor.SendProcessor.process(SendProcessor.java:82)
>> at
>> org.apache.camel.processor.DelegateProcessor.processNext(DelegateProcessor.java:53)
>> at
>> org.apache.camel.processor.DelegateProcessor.proceed(DelegateProcessor.java:82)
>> at
>> org.apache.camel.processor.interceptor.TraceInterceptor.process(TraceInterceptor.java:162)
>> at
>> org.apache.camel.processor.LoggingErrorHandler.process(LoggingErrorHandler.java:58)
>> at
>> org.apache.camel.processor.DefaultChannel.process(DefaultChannel.java:206)
>> at org.apache.camel.processor.Pipeline.process(Pipeline.java:74)
>> at
>> org.apache.camel.processor.ChoiceProcessor.process(ChoiceProcessor.java:56)
>> at
>> org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:67)
>> at
>> org.apache.camel.processor.DelegateProcessor.processNext(DelegateProcessor.java:53)
>> at
>> org.apache.camel.processor.DelegateProcessor.proceed(DelegateProcessor.java:82)
>> at
>> org.apache.camel.processor.interceptor.TraceInterceptor.process(TraceInterceptor.java:162)
>> at
>> org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:67)
>> at
>> org.apache.camel.processor.LoggingErrorHandler.process(LoggingErrorHandler.java:58)
>> at
>> org.apache.camel.processor.DefaultChannel.process(DefaultChannel.java:206)
>> at
>> org.apache.camel.processor.UnitOfWorkProcessor.processNext(UnitOfWorkProcessor.java:54)
>> at
>> org.apache.camel.processor.DelegateProcessor.process(DelegateProcessor.java:48)
>> at
>> org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:67)
>> at
>> org.apache.camel.component.file.GenericFileConsumer.processExchange(GenericFileConsumer.java:201)
>> at
>> org.apache.camel.component.file.GenericFileConsumer.processBatch(GenericFileConsumer.java:122)
>> at
>> org.apache.camel.component.file.GenericFileConsumer.poll(GenericFileConsumer.java:95)
>> at
>> org.apache.camel.impl.ScheduledPollConsumer.run(ScheduledPollConsumer.java:108)
>> at
>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:417)
>> at
>> java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:280)
>> at
>> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:135)
>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:65)
>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:142)
>> at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:166)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
>> at java.lang.Thread.run(Thread.java:613)
>>
>>
>>
>>
>> Claus Ibsen-2 wrote:
>>>
>>> On Fri, Nov 27, 2009 at 2:34 PM, shirazi <me...@farhad.eu> wrote:
>>>>
>>>> Oh, sorry. that's right, it is 64 MB. and I tried with 16, 32, 64 and
>>>> 128
>>>> MB.
>>>> I will try the buffer size option of the file-comp. to see if it solves
>>>> the
>>>> problem.
>>>>
>>>
>>> I have just committed the fix to camel trunk.
>>>
>>> The old code will still have the 2gb limit bug no matter what
>>> bufferSize you use.
>>> However you should try the latest code in 2.1 to see if that works for
>>> you.
>>> http://camel.apache.org/download.html
>>>
>>>
>>>
>>>> Thanks,
>>>> Farhad S.
>>>>
>>>>
>>>> Claus Ibsen-2 wrote:
>>>>>
>>>>> On Fri, Nov 27, 2009 at 2:26 PM, shirazi <me...@farhad.eu> wrote:
>>>>>>
>>>>>> That's probably the case. I've tried with different buffer sizes
>>>>>> (though
>>>>>> not
>>>>>> 0), from 16 to 128 KB. I got the best performance with buffer of
>>>>>> 64KB,
>>>>>> on
>>>>>> Linux Debian with 4 Gigs RAM, and files larger than 3 Gigs.
>>>>>> That would be great, if one could specify the buffer-size for the
>>>>>> "file"
>>>>>> component.
>>>>>>
>>>>>
>>>>> You buffer size was 64 * 1024 * 1024 which is 64 K * K = 64 MB. Or is
>>>>> there something that play tricks on me?
>>>>>
>>>>> The file component already have a bufferSize option with a default of
>>>>> 128kb
>>>>> http://camel.apache.org/file2.html
>>>>>
>>>>> So you can use that to set what you want to use.
>>>>>
>>>>>
>>>>>
>>>>>> -Farhad S.
>>>>>>
>>>>>>
>>>>>> Claus Ibsen-2 wrote:
>>>>>>>
>>>>>>> On Fri, Nov 27, 2009 at 2:00 PM, shirazi <me...@farhad.eu> wrote:
>>>>>>>>
>>>>>>>> Sure. I have attached the file.
>>>>>>>> Actually I use from->to components for transfering files and I
>>>>>>>> guess
>>>>>>>> it
>>>>>>>> copies the file, rather than moving.
>>>>>>>>
>>>>>>>
>>>>>>> Ah makes more sense for copying to leverage NIO.
>>>>>>>
>>>>>>> Camel does this also but I can see that it may hit that 2gb limit as
>>>>>>> it doesnt transfer in a loop using a lower buffer.
>>>>>>>
>>>>>>> It was doing it in a one liner
>>>>>>> in.transferTo(0, in.size(), out);
>>>>>>>
>>>>>>> I wonder if you really must do that in a loop to avoid any 2gb limit
>>>>>>> or the likes?
>>>>>>>
>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Farhad S.
>>>>>>>> http://old.nabble.com/file/p26542125/Nio.java Nio.java
>>>>>>>>
>>>>>>>>
>>>>>>>> Claus Ibsen-2 wrote:
>>>>>>>>>
>>>>>>>>> On Fri, Nov 27, 2009 at 1:37 PM, shirazi <me...@farhad.eu> wrote:
>>>>>>>>>>
>>>>>>>>>> Wau... thanks for that. I don't understand how I could miss it,
>>>>>>>>>> cause
>>>>>>>>>> i
>>>>>>>>>> searched a lot for such a feature.
>>>>>>>>>>
>>>>>>>>>> Another thing that I have experienced in 2.0, is that the "file"
>>>>>>>>>> component
>>>>>>>>>> is unable to move/copy files larger than 2Gigs, such large files
>>>>>>>>>> are
>>>>>>>>>> cut
>>>>>>>>>> around 2 Gigs limit. Now, I'm not sure if it has anything to do
>>>>>>>>>> with
>>>>>>>>>> the
>>>>>>>>>> OS
>>>>>>>>>> or the filesystems or what. I have seen it both in Linux (debian)
>>>>>>>>>> and
>>>>>>>>>> Mac
>>>>>>>>>> OS
>>>>>>>>>> X. And I don't understand either, why the file component does not
>>>>>>>>>> use
>>>>>>>>>> the
>>>>>>>>>> NIO api or at least we don't have an option to tell it use the
>>>>>>>>>> NIO
>>>>>>>>>> api
>>>>>>>>>> instead.
>>>>>>>>>> Anyway, for now I've solved the problem by using a smiple
>>>>>>>>>> component
>>>>>>>>>> instead
>>>>>>>>>> of the "file" comp. to move/copy files using the NIO api. It is
>>>>>>>>>> much
>>>>>>>>>> faster
>>>>>>>>>> and it has no problem with very large files.
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> For move it uses java.io.File rename and are you saying its not
>>>>>>>>> optimized for the underlying OS?
>>>>>>>>> I would assume that it could move a 2gb file in no time if the
>>>>>>>>> file
>>>>>>>>> is
>>>>>>>>> on the same share.
>>>>>>>>>
>>>>>>>>> Can you share your code that you are using?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> Bests,
>>>>>>>>>> Farhad S.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Claus Ibsen-2 wrote:
>>>>>>>>>>>
>>>>>>>>>>> On Fri, Nov 27, 2009 at 1:12 PM, shirazi <me...@farhad.eu> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> Hi,
>>>>>>>>>>>> Ok, I guess I should rather provide an AggregationStrategy.
>>>>>>>>>>>> The reason why I use a timer is that when I use the ftp
>>>>>>>>>>>> endpoint,
>>>>>>>>>>>> it
>>>>>>>>>>>> starts
>>>>>>>>>>>> to fetch everything that is on the ftp in the first call and
>>>>>>>>>>>> that's
>>>>>>>>>>>> not
>>>>>>>>>>>> what
>>>>>>>>>>>> i want, because I'm processing files over 3 Gigs and there are
>>>>>>>>>>>> many
>>>>>>>>>>>> of
>>>>>>>>>>>> them.
>>>>>>>>>>>> Then I figured that by using a timer, i could have ftp to
>>>>>>>>>>>> download
>>>>>>>>>>>> one
>>>>>>>>>>>> file
>>>>>>>>>>>> at a time!!
>>>>>>>>>>>> What I basically need is to somehow tell the ftp, how many
>>>>>>>>>>>> files
>>>>>>>>>>>> I
>>>>>>>>>>>> want
>>>>>>>>>>>> to
>>>>>>>>>>>> fetch at a time. This is probably doable now in camel-2.1 by
>>>>>>>>>>>> using
>>>>>>>>>>>> the
>>>>>>>>>>>> "ThrottlingInflightRoutePolicy", which i havn't tried yet.
>>>>>>>>>>>> Do you have any suggestions?
>>>>>>>>>>>
>>>>>>>>>>> Yeah the FTP component extends the File component so you got all
>>>>>>>>>>> the
>>>>>>>>>>> options from the file component as well
>>>>>>>>>>> http://camel.apache.org/file2.html
>>>>>>>>>>>
>>>>>>>>>>> And there is a maxMessagesPerPoll you can use to tell how many
>>>>>>>>>>> files
>>>>>>>>>>> you want at most to poll. For example = 1 in your use.
>>>>>>>>>>>
>>>>>>>>>>> And that option is avail in 2.0 also.
>>>>>>>>>>>
>>>>>>>>>>> In fact all these are as they are batch polling consumers
>>>>>>>>>>> http://camel.apache.org/batch-consumer.html
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Bests,
>>>>>>>>>>>> Farhad S.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Claus Ibsen-2 wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> Hi
>>>>>>>>>>>>>
>>>>>>>>>>>>> BTW why are you using a timer -> ftp when the ftp have a build
>>>>>>>>>>>>> in
>>>>>>>>>>>>> timer?
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Fri, Nov 27, 2009 at 12:55 PM, shirazi <me...@farhad.eu>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>> Please consider the following route:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> from("timer://someTrigger?delay=30000&fixedRate=true&period=300000").
>>>>>>>>>>>>>> pollEnrich("ftp://cmoreFtp?binary=true&localWorkDirectory=/tmp&idempotent=true&idempotentRepository=#caStreams",
>>>>>>>>>>>>>> 10000).
>>>>>>>>>>>>>> to("file:///tmp/vw/cmore?tempPrefix=.inprogress");
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> The pollEnrich cast a NullPointerException in camel-2.0, if
>>>>>>>>>>>>>> there
>>>>>>>>>>>>>> is
>>>>>>>>>>>>>> nothing
>>>>>>>>>>>>>> to fetch from the ftp. I got rid of the exception by a
>>>>>>>>>>>>>> "try-catch(NullpointerExcpetion.class).stop()" around the
>>>>>>>>>>>>>> route.
>>>>>>>>>>>>>> Now
>>>>>>>>>>>>>> in
>>>>>>>>>>>>>> camel-2.1-SNAPSHOT, I get the following exception on the "to"
>>>>>>>>>>>>>> endpoint:
>>>>>>>>>>>>>> =================
>>>>>>>>>>>>>> 12:37:06,899 INFO [Tracer] >>> -->
>>>>>>>>>>>>>> file:///tmp/vw/cmore?tempPrefix=.inprogress_, Pattern:InOnly,
>>>>>>>>>>>>>> Headers:{firedTime=Fri Nov 27 12:36:56 CET 2009},
>>>>>>>>>>>>>> Exception:org.apache.camel.component.file.GenericFileOperationFailedException:
>>>>>>>>>>>>>> Cannot store file:
>>>>>>>>>>>>>> /tmp/vw/cmore/.inprogress_02216f38-8ffc-4aec-b638-27229d857075
>>>>>>>>>>>>>> 12:37:06,899 INFO [Tracer] >>> -->
>>>>>>>>>>>>>> file:///tmp/vw/cmore?tempPrefix=.inprogress_, Pattern:InOnly,
>>>>>>>>>>>>>> Headers:{firedTime=Fri Nov 27 12:36:56 CET 2009},
>>>>>>>>>>>>>> Exception:org.apache.camel.component.file.GenericFileOperationFailedException:
>>>>>>>>>>>>>> Cannot store file:
>>>>>>>>>>>>>> /tmp/vw/cmore/.inprogress_02216f38-8ffc-4aec-b638-27229d857075
>>>>>>>>>>>>>> 12:37:06,899 ERROR [TimerConsumer] Cannot store file:
>>>>>>>>>>>>>> /tmp/vw/cmore/.inprogress_02216f38-8ffc-4aec-b638-27229d857075
>>>>>>>>>>>>>> org.apache.camel.component.file.GenericFileOperationFailedException:
>>>>>>>>>>>>>> Cannot
>>>>>>>>>>>>>> store file:
>>>>>>>>>>>>>> /tmp/vw/cmore/.inprogress_02216f38-8ffc-4aec-b638-27229d857075
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> org.apache.camel.component.file.FileOperations.storeFile(FileOperations.java:204)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> org.apache.camel.component.file.GenericFileProducer.writeFile(GenericFileProducer.java:184)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> org.apache.camel.component.file.GenericFileProducer.processExchange(GenericFileProducer.java:126)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> org.apache.camel.component.file.GenericFileProducer.process(GenericFileProducer.java:57)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> org.apache.camel.processor.SendProcessor$1.doInProducer(SendProcessor.java:97)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> org.apache.camel.processor.SendProcessor$1.doInProducer(SendProcessor.java:95)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> org.apache.camel.impl.ProducerCache.doInProducer(ProducerCache.java:146)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> org.apache.camel.processor.SendProcessor.doProcess(SendProcessor.java:94)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> org.apache.camel.processor.SendProcessor.process(SendProcessor.java:82)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:67)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> org.apache.camel.processor.DelegateProcessor.processNext(DelegateProcessor.java:53)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> org.apache.camel.processor.DelegateProcessor.proceed(DelegateProcessor.java:82)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> org.apache.camel.processor.interceptor.TraceInterceptor.process(TraceInterceptor.java:162)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:67)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> org.apache.camel.processor.DefaultChannel.process(DefaultChannel.java:206)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> org.apache.camel.processor.Pipeline.process(Pipeline.java:74)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> org.apache.camel.processor.TryProcessor.process(TryProcessor.java:63)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:67)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> org.apache.camel.processor.DelegateProcessor.processNext(DelegateProcessor.java:53)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> org.apache.camel.processor.DelegateProcessor.proceed(DelegateProcessor.java:82)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> org.apache.camel.processor.interceptor.TraceInterceptor.process(TraceInterceptor.java:162)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:67)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> org.apache.camel.processor.DefaultChannel.process(DefaultChannel.java:206)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> org.apache.camel.processor.UnitOfWorkProcessor.processNext(UnitOfWorkProcessor.java:54)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> org.apache.camel.processor.DelegateProcessor.process(DelegateProcessor.java:48)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:67)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> org.apache.camel.component.timer.TimerConsumer.sendTimerExchange(TimerConsumer.java:103)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> org.apache.camel.component.timer.TimerConsumer$1.run(TimerConsumer.java:50)
>>>>>>>>>>>>>> at java.util.TimerThread.mainLoop(Timer.java:512)
>>>>>>>>>>>>>> at java.util.TimerThread.run(Timer.java:462)
>>>>>>>>>>>>>> Caused by: org.apache.camel.InvalidPayloadException: No body
>>>>>>>>>>>>>> available
>>>>>>>>>>>>>> of
>>>>>>>>>>>>>> type: java.io.InputStream on: Message: [Body is null]. Caused
>>>>>>>>>>>>>> by:
>>>>>>>>>>>>>> No
>>>>>>>>>>>>>> type
>>>>>>>>>>>>>> converter available to convert from type: null to the
>>>>>>>>>>>>>> required
>>>>>>>>>>>>>> type:
>>>>>>>>>>>>>> java.io.InputStream with value null on the exchange:
>>>>>>>>>>>>>> Exchange[Message:
>>>>>>>>>>>>>> [Body
>>>>>>>>>>>>>> is null]]
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> org.apache.camel.impl.MessageSupport.getMandatoryBody(MessageSupport.java:103)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> org.apache.camel.util.ExchangeHelper.getMandatoryInBody(ExchangeHelper.java:116)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> org.apache.camel.component.file.FileOperations.storeFile(FileOperations.java:198)
>>>>>>>>>>>>>> ... 29 more
>>>>>>>>>>>>>> Caused by:
>>>>>>>>>>>>>> org.apache.camel.NoTypeConversionAvailableException:
>>>>>>>>>>>>>> No
>>>>>>>>>>>>>> type
>>>>>>>>>>>>>> converter available to convert from type: null to the
>>>>>>>>>>>>>> required
>>>>>>>>>>>>>> type:
>>>>>>>>>>>>>> java.io.InputStream with value null
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> org.apache.camel.impl.converter.DefaultTypeConverter.mandatoryConvertTo(DefaultTypeConverter.java:126)
>>>>>>>>>>>>>> at
>>>>>>>>>>>>>> org.apache.camel.impl.MessageSupport.getMandatoryBody(MessageSupport.java:101)
>>>>>>>>>>>>>> ... 31 more
>>>>>>>>>>>>>> ============
>>>>>>>>>>>>>> Now, I can remedy the situation by a try-catch block or an
>>>>>>>>>>>>>> interceptor,
>>>>>>>>>>>>>> but
>>>>>>>>>>>>>> in my opinion this should be considered a bug in camel and it
>>>>>>>>>>>>>> should
>>>>>>>>>>>>>> silently ignore if there was nothing to fetch from the ftp.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Bests,
>>>>>>>>>>>>>> Farhad Shiraz
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> --
>>>>>>>>>>>>>> View this message in context:
>>>>>>>>>>>>>> http://old.nabble.com/camel-casts-exception-when-there-is-nothing-to-fetch-from-the-ftp-endpoint-tp26541447p26541447.html
>>>>>>>>>>>>>> Sent from the Camel - Users mailing list archive at
>>>>>>>>>>>>>> Nabble.com.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>>> Claus Ibsen
>>>>>>>>>>>>> Apache Camel Committer
>>>>>>>>>>>>>
>>>>>>>>>>>>> Author of Camel in Action: http://www.manning.com/ibsen/
>>>>>>>>>>>>> Open Source Integration: http://fusesource.com
>>>>>>>>>>>>> Blog: http://davsclaus.blogspot.com/
>>>>>>>>>>>>> Twitter: http://twitter.com/davsclaus
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> View this message in context:
>>>>>>>>>>>> http://old.nabble.com/camel-casts-exception-when-there-is-nothing-to-fetch-from-the-ftp-endpoint-tp26541447p26541650.html
>>>>>>>>>>>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Claus Ibsen
>>>>>>>>>>> Apache Camel Committer
>>>>>>>>>>>
>>>>>>>>>>> Author of Camel in Action: http://www.manning.com/ibsen/
>>>>>>>>>>> Open Source Integration: http://fusesource.com
>>>>>>>>>>> Blog: http://davsclaus.blogspot.com/
>>>>>>>>>>> Twitter: http://twitter.com/davsclaus
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> View this message in context:
>>>>>>>>>> http://old.nabble.com/camel-casts-exception-when-there-is-nothing-to-fetch-from-the-ftp-endpoint-tp26541447p26541877.html
>>>>>>>>>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Claus Ibsen
>>>>>>>>> Apache Camel Committer
>>>>>>>>>
>>>>>>>>> Author of Camel in Action: http://www.manning.com/ibsen/
>>>>>>>>> Open Source Integration: http://fusesource.com
>>>>>>>>> Blog: http://davsclaus.blogspot.com/
>>>>>>>>> Twitter: http://twitter.com/davsclaus
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> View this message in context:
>>>>>>>> http://old.nabble.com/camel-casts-exception-when-there-is-nothing-to-fetch-from-the-ftp-endpoint-tp26541447p26542125.html
>>>>>>>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Claus Ibsen
>>>>>>> Apache Camel Committer
>>>>>>>
>>>>>>> Author of Camel in Action: http://www.manning.com/ibsen/
>>>>>>> Open Source Integration: http://fusesource.com
>>>>>>> Blog: http://davsclaus.blogspot.com/
>>>>>>> Twitter: http://twitter.com/davsclaus
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> --
>>>>>> View this message in context:
>>>>>> http://old.nabble.com/camel-casts-exception-when-there-is-nothing-to-fetch-from-the-ftp-endpoint-tp26541447p26542396.html
>>>>>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Claus Ibsen
>>>>> Apache Camel Committer
>>>>>
>>>>> Author of Camel in Action: http://www.manning.com/ibsen/
>>>>> Open Source Integration: http://fusesource.com
>>>>> Blog: http://davsclaus.blogspot.com/
>>>>> Twitter: http://twitter.com/davsclaus
>>>>>
>>>>>
>>>>
>>>> --
>>>> View this message in context:
>>>> http://old.nabble.com/camel-casts-exception-when-there-is-nothing-to-fetch-from-the-ftp-endpoint-tp26541447p26542472.html
>>>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>>>
>>>>
>>>
>>>
>>>
>>> --
>>> Claus Ibsen
>>> Apache Camel Committer
>>>
>>> Author of Camel in Action: http://www.manning.com/ibsen/
>>> Open Source Integration: http://fusesource.com
>>> Blog: http://davsclaus.blogspot.com/
>>> Twitter: http://twitter.com/davsclaus
>>>
>>>
>>
>> --
>> View this message in context:
>> http://old.nabble.com/camel-casts-exception-when-there-is-nothing-to-fetch-from-the-ftp-endpoint-tp26541447p26543771.html
>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>
>>
>
>
>
> --
> Claus Ibsen
> Apache Camel Committer
>
> Author of Camel in Action: http://www.manning.com/ibsen/
> Open Source Integration: http://fusesource.com
> Blog: http://davsclaus.blogspot.com/
> Twitter: http://twitter.com/davsclaus
>
>
--
View this message in context: http://old.nabble.com/camel-casts-exception-when-there-is-nothing-to-fetch-from-the-ftp-endpoint-tp26541447p26545371.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: camel casts exception when there is nothing to fetch from the ftp
endpoint
Posted by Claus Ibsen <cl...@gmail.com>.
On Fri, Nov 27, 2009 at 4:18 PM, shirazi <me...@farhad.eu> wrote:
>
> Thanks Claus,
> I just downloaded the new trunk, and I think there's a new bug in it. I just
> tried the file component with a couple of files around 3-400 MBs and it
> works fine. As soon as I try it with a file around 3 Gigs, i get the
> following exception (??!!):
>
That is odd as the stacktrace is off by -1.
And you do a full svn up and a mvn clean install -Dtest=false to build
all source from scratch?
If you enable TRACE logging for: org.apache.camel.component.file
You should see this log line when it uses the NIO stuff
if (LOG.isTraceEnabled()) {
LOG.trace("Using FileChannel to transfer from: " + in
+ " to: " + out);
}
I have committed a fix for the NPE in that stacktrace. But the was
some other exception thrown that got lost.
commited rev: 884894.
> 16:13:48,830 ERROR [workflow] null while processing exchange:
> Exchange[GenericFileMessage with file: GenericFile[/tmp/vw/cmore/tv5.ts]]
> java.lang.NullPointerException
> at
> org.apache.camel.impl.OnExceptionRouteNode.getLabel(OnExceptionRouteNode.java:41)
> at
> org.apache.camel.processor.interceptor.DefaultTraceFormatter.getNodeMessage(DefaultTraceFormatter.java:233)
> at
> org.apache.camel.processor.interceptor.DefaultTraceFormatter.extractBreadCrumb(DefaultTraceFormatter.java:282)
> at
> org.apache.camel.processor.interceptor.DefaultTraceFormatter.format(DefaultTraceFormatter.java:55)
> at
> org.apache.camel.processor.interceptor.TraceInterceptor.format(TraceInterceptor.java:232)
> at org.apache.camel.processor.Logger.logMessage(Logger.java:283)
> at org.apache.camel.processor.Logger.process(Logger.java:166)
> at
> org.apache.camel.processor.interceptor.TraceInterceptor.logException(TraceInterceptor.java:325)
> at
> org.apache.camel.processor.interceptor.TraceInterceptor.process(TraceInterceptor.java:171)
> at
> org.apache.camel.processor.LoggingErrorHandler.process(LoggingErrorHandler.java:58)
> at
> org.apache.camel.processor.DefaultChannel.process(DefaultChannel.java:206)
> at
> org.apache.camel.processor.DelegateProcessor.processNext(DelegateProcessor.java:53)
> at
> org.apache.camel.processor.ChoiceProcessor.process(ChoiceProcessor.java:51)
> at
> org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:67)
> at
> org.apache.camel.processor.DelegateProcessor.processNext(DelegateProcessor.java:53)
> at
> org.apache.camel.processor.DelegateProcessor.proceed(DelegateProcessor.java:82)
> at
> org.apache.camel.processor.interceptor.TraceInterceptor.process(TraceInterceptor.java:162)
> at
> org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:67)
> at
> org.apache.camel.processor.LoggingErrorHandler.process(LoggingErrorHandler.java:58)
> at
> org.apache.camel.processor.DefaultChannel.process(DefaultChannel.java:206)
> at
> org.apache.camel.processor.UnitOfWorkProcessor.processNext(UnitOfWorkProcessor.java:70)
> at
> org.apache.camel.processor.DelegateProcessor.process(DelegateProcessor.java:48)
> at
> org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:67)
> at
> org.apache.camel.component.direct.DirectProducer.process(DirectProducer.java:45)
> at
> org.apache.camel.processor.SendProcessor$1.doInProducer(SendProcessor.java:97)
> at
> org.apache.camel.processor.SendProcessor$1.doInProducer(SendProcessor.java:95)
> at org.apache.camel.impl.ProducerCache.doInProducer(ProducerCache.java:146)
> at
> org.apache.camel.processor.SendProcessor.doProcess(SendProcessor.java:94)
> at org.apache.camel.processor.SendProcessor.process(SendProcessor.java:82)
> at
> org.apache.camel.processor.DelegateProcessor.processNext(DelegateProcessor.java:53)
> at
> org.apache.camel.processor.DelegateProcessor.proceed(DelegateProcessor.java:82)
> at
> org.apache.camel.processor.interceptor.TraceInterceptor.process(TraceInterceptor.java:162)
> at
> org.apache.camel.processor.LoggingErrorHandler.process(LoggingErrorHandler.java:58)
> at
> org.apache.camel.processor.DefaultChannel.process(DefaultChannel.java:206)
> at org.apache.camel.processor.Pipeline.process(Pipeline.java:74)
> at
> org.apache.camel.processor.ChoiceProcessor.process(ChoiceProcessor.java:56)
> at
> org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:67)
> at
> org.apache.camel.processor.DelegateProcessor.processNext(DelegateProcessor.java:53)
> at
> org.apache.camel.processor.DelegateProcessor.proceed(DelegateProcessor.java:82)
> at
> org.apache.camel.processor.interceptor.TraceInterceptor.process(TraceInterceptor.java:162)
> at
> org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:67)
> at
> org.apache.camel.processor.LoggingErrorHandler.process(LoggingErrorHandler.java:58)
> at
> org.apache.camel.processor.DefaultChannel.process(DefaultChannel.java:206)
> at
> org.apache.camel.processor.UnitOfWorkProcessor.processNext(UnitOfWorkProcessor.java:54)
> at
> org.apache.camel.processor.DelegateProcessor.process(DelegateProcessor.java:48)
> at
> org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:67)
> at
> org.apache.camel.component.file.GenericFileConsumer.processExchange(GenericFileConsumer.java:201)
> at
> org.apache.camel.component.file.GenericFileConsumer.processBatch(GenericFileConsumer.java:122)
> at
> org.apache.camel.component.file.GenericFileConsumer.poll(GenericFileConsumer.java:95)
> at
> org.apache.camel.impl.ScheduledPollConsumer.run(ScheduledPollConsumer.java:108)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:417)
> at
> java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:280)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:135)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:65)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:142)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:166)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
> at java.lang.Thread.run(Thread.java:613)
>
>
>
>
> Claus Ibsen-2 wrote:
>>
>> On Fri, Nov 27, 2009 at 2:34 PM, shirazi <me...@farhad.eu> wrote:
>>>
>>> Oh, sorry. that's right, it is 64 MB. and I tried with 16, 32, 64 and 128
>>> MB.
>>> I will try the buffer size option of the file-comp. to see if it solves
>>> the
>>> problem.
>>>
>>
>> I have just committed the fix to camel trunk.
>>
>> The old code will still have the 2gb limit bug no matter what
>> bufferSize you use.
>> However you should try the latest code in 2.1 to see if that works for
>> you.
>> http://camel.apache.org/download.html
>>
>>
>>
>>> Thanks,
>>> Farhad S.
>>>
>>>
>>> Claus Ibsen-2 wrote:
>>>>
>>>> On Fri, Nov 27, 2009 at 2:26 PM, shirazi <me...@farhad.eu> wrote:
>>>>>
>>>>> That's probably the case. I've tried with different buffer sizes
>>>>> (though
>>>>> not
>>>>> 0), from 16 to 128 KB. I got the best performance with buffer of 64KB,
>>>>> on
>>>>> Linux Debian with 4 Gigs RAM, and files larger than 3 Gigs.
>>>>> That would be great, if one could specify the buffer-size for the
>>>>> "file"
>>>>> component.
>>>>>
>>>>
>>>> You buffer size was 64 * 1024 * 1024 which is 64 K * K = 64 MB. Or is
>>>> there something that play tricks on me?
>>>>
>>>> The file component already have a bufferSize option with a default of
>>>> 128kb
>>>> http://camel.apache.org/file2.html
>>>>
>>>> So you can use that to set what you want to use.
>>>>
>>>>
>>>>
>>>>> -Farhad S.
>>>>>
>>>>>
>>>>> Claus Ibsen-2 wrote:
>>>>>>
>>>>>> On Fri, Nov 27, 2009 at 2:00 PM, shirazi <me...@farhad.eu> wrote:
>>>>>>>
>>>>>>> Sure. I have attached the file.
>>>>>>> Actually I use from->to components for transfering files and I guess
>>>>>>> it
>>>>>>> copies the file, rather than moving.
>>>>>>>
>>>>>>
>>>>>> Ah makes more sense for copying to leverage NIO.
>>>>>>
>>>>>> Camel does this also but I can see that it may hit that 2gb limit as
>>>>>> it doesnt transfer in a loop using a lower buffer.
>>>>>>
>>>>>> It was doing it in a one liner
>>>>>> in.transferTo(0, in.size(), out);
>>>>>>
>>>>>> I wonder if you really must do that in a loop to avoid any 2gb limit
>>>>>> or the likes?
>>>>>>
>>>>>>
>>>>>>> Thanks,
>>>>>>> Farhad S.
>>>>>>> http://old.nabble.com/file/p26542125/Nio.java Nio.java
>>>>>>>
>>>>>>>
>>>>>>> Claus Ibsen-2 wrote:
>>>>>>>>
>>>>>>>> On Fri, Nov 27, 2009 at 1:37 PM, shirazi <me...@farhad.eu> wrote:
>>>>>>>>>
>>>>>>>>> Wau... thanks for that. I don't understand how I could miss it,
>>>>>>>>> cause
>>>>>>>>> i
>>>>>>>>> searched a lot for such a feature.
>>>>>>>>>
>>>>>>>>> Another thing that I have experienced in 2.0, is that the "file"
>>>>>>>>> component
>>>>>>>>> is unable to move/copy files larger than 2Gigs, such large files
>>>>>>>>> are
>>>>>>>>> cut
>>>>>>>>> around 2 Gigs limit. Now, I'm not sure if it has anything to do
>>>>>>>>> with
>>>>>>>>> the
>>>>>>>>> OS
>>>>>>>>> or the filesystems or what. I have seen it both in Linux (debian)
>>>>>>>>> and
>>>>>>>>> Mac
>>>>>>>>> OS
>>>>>>>>> X. And I don't understand either, why the file component does not
>>>>>>>>> use
>>>>>>>>> the
>>>>>>>>> NIO api or at least we don't have an option to tell it use the NIO
>>>>>>>>> api
>>>>>>>>> instead.
>>>>>>>>> Anyway, for now I've solved the problem by using a smiple component
>>>>>>>>> instead
>>>>>>>>> of the "file" comp. to move/copy files using the NIO api. It is
>>>>>>>>> much
>>>>>>>>> faster
>>>>>>>>> and it has no problem with very large files.
>>>>>>>>>
>>>>>>>>
>>>>>>>> For move it uses java.io.File rename and are you saying its not
>>>>>>>> optimized for the underlying OS?
>>>>>>>> I would assume that it could move a 2gb file in no time if the file
>>>>>>>> is
>>>>>>>> on the same share.
>>>>>>>>
>>>>>>>> Can you share your code that you are using?
>>>>>>>>
>>>>>>>>
>>>>>>>>> Bests,
>>>>>>>>> Farhad S.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Claus Ibsen-2 wrote:
>>>>>>>>>>
>>>>>>>>>> On Fri, Nov 27, 2009 at 1:12 PM, shirazi <me...@farhad.eu> wrote:
>>>>>>>>>>>
>>>>>>>>>>> Hi,
>>>>>>>>>>> Ok, I guess I should rather provide an AggregationStrategy.
>>>>>>>>>>> The reason why I use a timer is that when I use the ftp endpoint,
>>>>>>>>>>> it
>>>>>>>>>>> starts
>>>>>>>>>>> to fetch everything that is on the ftp in the first call and
>>>>>>>>>>> that's
>>>>>>>>>>> not
>>>>>>>>>>> what
>>>>>>>>>>> i want, because I'm processing files over 3 Gigs and there are
>>>>>>>>>>> many
>>>>>>>>>>> of
>>>>>>>>>>> them.
>>>>>>>>>>> Then I figured that by using a timer, i could have ftp to
>>>>>>>>>>> download
>>>>>>>>>>> one
>>>>>>>>>>> file
>>>>>>>>>>> at a time!!
>>>>>>>>>>> What I basically need is to somehow tell the ftp, how many files
>>>>>>>>>>> I
>>>>>>>>>>> want
>>>>>>>>>>> to
>>>>>>>>>>> fetch at a time. This is probably doable now in camel-2.1 by
>>>>>>>>>>> using
>>>>>>>>>>> the
>>>>>>>>>>> "ThrottlingInflightRoutePolicy", which i havn't tried yet.
>>>>>>>>>>> Do you have any suggestions?
>>>>>>>>>>
>>>>>>>>>> Yeah the FTP component extends the File component so you got all
>>>>>>>>>> the
>>>>>>>>>> options from the file component as well
>>>>>>>>>> http://camel.apache.org/file2.html
>>>>>>>>>>
>>>>>>>>>> And there is a maxMessagesPerPoll you can use to tell how many
>>>>>>>>>> files
>>>>>>>>>> you want at most to poll. For example = 1 in your use.
>>>>>>>>>>
>>>>>>>>>> And that option is avail in 2.0 also.
>>>>>>>>>>
>>>>>>>>>> In fact all these are as they are batch polling consumers
>>>>>>>>>> http://camel.apache.org/batch-consumer.html
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Bests,
>>>>>>>>>>> Farhad S.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Claus Ibsen-2 wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> Hi
>>>>>>>>>>>>
>>>>>>>>>>>> BTW why are you using a timer -> ftp when the ftp have a build
>>>>>>>>>>>> in
>>>>>>>>>>>> timer?
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Fri, Nov 27, 2009 at 12:55 PM, shirazi <me...@farhad.eu> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>> Please consider the following route:
>>>>>>>>>>>>>
>>>>>>>>>>>>> from("timer://someTrigger?delay=30000&fixedRate=true&period=300000").
>>>>>>>>>>>>> pollEnrich("ftp://cmoreFtp?binary=true&localWorkDirectory=/tmp&idempotent=true&idempotentRepository=#caStreams",
>>>>>>>>>>>>> 10000).
>>>>>>>>>>>>> to("file:///tmp/vw/cmore?tempPrefix=.inprogress");
>>>>>>>>>>>>>
>>>>>>>>>>>>> The pollEnrich cast a NullPointerException in camel-2.0, if
>>>>>>>>>>>>> there
>>>>>>>>>>>>> is
>>>>>>>>>>>>> nothing
>>>>>>>>>>>>> to fetch from the ftp. I got rid of the exception by a
>>>>>>>>>>>>> "try-catch(NullpointerExcpetion.class).stop()" around the
>>>>>>>>>>>>> route.
>>>>>>>>>>>>> Now
>>>>>>>>>>>>> in
>>>>>>>>>>>>> camel-2.1-SNAPSHOT, I get the following exception on the "to"
>>>>>>>>>>>>> endpoint:
>>>>>>>>>>>>> =================
>>>>>>>>>>>>> 12:37:06,899 INFO [Tracer] >>> -->
>>>>>>>>>>>>> file:///tmp/vw/cmore?tempPrefix=.inprogress_, Pattern:InOnly,
>>>>>>>>>>>>> Headers:{firedTime=Fri Nov 27 12:36:56 CET 2009},
>>>>>>>>>>>>> Exception:org.apache.camel.component.file.GenericFileOperationFailedException:
>>>>>>>>>>>>> Cannot store file:
>>>>>>>>>>>>> /tmp/vw/cmore/.inprogress_02216f38-8ffc-4aec-b638-27229d857075
>>>>>>>>>>>>> 12:37:06,899 INFO [Tracer] >>> -->
>>>>>>>>>>>>> file:///tmp/vw/cmore?tempPrefix=.inprogress_, Pattern:InOnly,
>>>>>>>>>>>>> Headers:{firedTime=Fri Nov 27 12:36:56 CET 2009},
>>>>>>>>>>>>> Exception:org.apache.camel.component.file.GenericFileOperationFailedException:
>>>>>>>>>>>>> Cannot store file:
>>>>>>>>>>>>> /tmp/vw/cmore/.inprogress_02216f38-8ffc-4aec-b638-27229d857075
>>>>>>>>>>>>> 12:37:06,899 ERROR [TimerConsumer] Cannot store file:
>>>>>>>>>>>>> /tmp/vw/cmore/.inprogress_02216f38-8ffc-4aec-b638-27229d857075
>>>>>>>>>>>>> org.apache.camel.component.file.GenericFileOperationFailedException:
>>>>>>>>>>>>> Cannot
>>>>>>>>>>>>> store file:
>>>>>>>>>>>>> /tmp/vw/cmore/.inprogress_02216f38-8ffc-4aec-b638-27229d857075
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.apache.camel.component.file.FileOperations.storeFile(FileOperations.java:204)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.apache.camel.component.file.GenericFileProducer.writeFile(GenericFileProducer.java:184)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.apache.camel.component.file.GenericFileProducer.processExchange(GenericFileProducer.java:126)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.apache.camel.component.file.GenericFileProducer.process(GenericFileProducer.java:57)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.apache.camel.processor.SendProcessor$1.doInProducer(SendProcessor.java:97)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.apache.camel.processor.SendProcessor$1.doInProducer(SendProcessor.java:95)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.apache.camel.impl.ProducerCache.doInProducer(ProducerCache.java:146)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.apache.camel.processor.SendProcessor.doProcess(SendProcessor.java:94)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.apache.camel.processor.SendProcessor.process(SendProcessor.java:82)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:67)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.apache.camel.processor.DelegateProcessor.processNext(DelegateProcessor.java:53)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.apache.camel.processor.DelegateProcessor.proceed(DelegateProcessor.java:82)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.apache.camel.processor.interceptor.TraceInterceptor.process(TraceInterceptor.java:162)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:67)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.apache.camel.processor.DefaultChannel.process(DefaultChannel.java:206)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.apache.camel.processor.Pipeline.process(Pipeline.java:74)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.apache.camel.processor.TryProcessor.process(TryProcessor.java:63)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:67)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.apache.camel.processor.DelegateProcessor.processNext(DelegateProcessor.java:53)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.apache.camel.processor.DelegateProcessor.proceed(DelegateProcessor.java:82)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.apache.camel.processor.interceptor.TraceInterceptor.process(TraceInterceptor.java:162)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:67)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.apache.camel.processor.DefaultChannel.process(DefaultChannel.java:206)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.apache.camel.processor.UnitOfWorkProcessor.processNext(UnitOfWorkProcessor.java:54)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.apache.camel.processor.DelegateProcessor.process(DelegateProcessor.java:48)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:67)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.apache.camel.component.timer.TimerConsumer.sendTimerExchange(TimerConsumer.java:103)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.apache.camel.component.timer.TimerConsumer$1.run(TimerConsumer.java:50)
>>>>>>>>>>>>> at java.util.TimerThread.mainLoop(Timer.java:512)
>>>>>>>>>>>>> at java.util.TimerThread.run(Timer.java:462)
>>>>>>>>>>>>> Caused by: org.apache.camel.InvalidPayloadException: No body
>>>>>>>>>>>>> available
>>>>>>>>>>>>> of
>>>>>>>>>>>>> type: java.io.InputStream on: Message: [Body is null]. Caused
>>>>>>>>>>>>> by:
>>>>>>>>>>>>> No
>>>>>>>>>>>>> type
>>>>>>>>>>>>> converter available to convert from type: null to the required
>>>>>>>>>>>>> type:
>>>>>>>>>>>>> java.io.InputStream with value null on the exchange:
>>>>>>>>>>>>> Exchange[Message:
>>>>>>>>>>>>> [Body
>>>>>>>>>>>>> is null]]
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.apache.camel.impl.MessageSupport.getMandatoryBody(MessageSupport.java:103)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.apache.camel.util.ExchangeHelper.getMandatoryInBody(ExchangeHelper.java:116)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.apache.camel.component.file.FileOperations.storeFile(FileOperations.java:198)
>>>>>>>>>>>>> ... 29 more
>>>>>>>>>>>>> Caused by: org.apache.camel.NoTypeConversionAvailableException:
>>>>>>>>>>>>> No
>>>>>>>>>>>>> type
>>>>>>>>>>>>> converter available to convert from type: null to the required
>>>>>>>>>>>>> type:
>>>>>>>>>>>>> java.io.InputStream with value null
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.apache.camel.impl.converter.DefaultTypeConverter.mandatoryConvertTo(DefaultTypeConverter.java:126)
>>>>>>>>>>>>> at
>>>>>>>>>>>>> org.apache.camel.impl.MessageSupport.getMandatoryBody(MessageSupport.java:101)
>>>>>>>>>>>>> ... 31 more
>>>>>>>>>>>>> ============
>>>>>>>>>>>>> Now, I can remedy the situation by a try-catch block or an
>>>>>>>>>>>>> interceptor,
>>>>>>>>>>>>> but
>>>>>>>>>>>>> in my opinion this should be considered a bug in camel and it
>>>>>>>>>>>>> should
>>>>>>>>>>>>> silently ignore if there was nothing to fetch from the ftp.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Bests,
>>>>>>>>>>>>> Farhad Shiraz
>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>>> View this message in context:
>>>>>>>>>>>>> http://old.nabble.com/camel-casts-exception-when-there-is-nothing-to-fetch-from-the-ftp-endpoint-tp26541447p26541447.html
>>>>>>>>>>>>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> Claus Ibsen
>>>>>>>>>>>> Apache Camel Committer
>>>>>>>>>>>>
>>>>>>>>>>>> Author of Camel in Action: http://www.manning.com/ibsen/
>>>>>>>>>>>> Open Source Integration: http://fusesource.com
>>>>>>>>>>>> Blog: http://davsclaus.blogspot.com/
>>>>>>>>>>>> Twitter: http://twitter.com/davsclaus
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> View this message in context:
>>>>>>>>>>> http://old.nabble.com/camel-casts-exception-when-there-is-nothing-to-fetch-from-the-ftp-endpoint-tp26541447p26541650.html
>>>>>>>>>>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Claus Ibsen
>>>>>>>>>> Apache Camel Committer
>>>>>>>>>>
>>>>>>>>>> Author of Camel in Action: http://www.manning.com/ibsen/
>>>>>>>>>> Open Source Integration: http://fusesource.com
>>>>>>>>>> Blog: http://davsclaus.blogspot.com/
>>>>>>>>>> Twitter: http://twitter.com/davsclaus
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> View this message in context:
>>>>>>>>> http://old.nabble.com/camel-casts-exception-when-there-is-nothing-to-fetch-from-the-ftp-endpoint-tp26541447p26541877.html
>>>>>>>>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Claus Ibsen
>>>>>>>> Apache Camel Committer
>>>>>>>>
>>>>>>>> Author of Camel in Action: http://www.manning.com/ibsen/
>>>>>>>> Open Source Integration: http://fusesource.com
>>>>>>>> Blog: http://davsclaus.blogspot.com/
>>>>>>>> Twitter: http://twitter.com/davsclaus
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> View this message in context:
>>>>>>> http://old.nabble.com/camel-casts-exception-when-there-is-nothing-to-fetch-from-the-ftp-endpoint-tp26541447p26542125.html
>>>>>>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Claus Ibsen
>>>>>> Apache Camel Committer
>>>>>>
>>>>>> Author of Camel in Action: http://www.manning.com/ibsen/
>>>>>> Open Source Integration: http://fusesource.com
>>>>>> Blog: http://davsclaus.blogspot.com/
>>>>>> Twitter: http://twitter.com/davsclaus
>>>>>>
>>>>>>
>>>>>
>>>>> --
>>>>> View this message in context:
>>>>> http://old.nabble.com/camel-casts-exception-when-there-is-nothing-to-fetch-from-the-ftp-endpoint-tp26541447p26542396.html
>>>>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Claus Ibsen
>>>> Apache Camel Committer
>>>>
>>>> Author of Camel in Action: http://www.manning.com/ibsen/
>>>> Open Source Integration: http://fusesource.com
>>>> Blog: http://davsclaus.blogspot.com/
>>>> Twitter: http://twitter.com/davsclaus
>>>>
>>>>
>>>
>>> --
>>> View this message in context:
>>> http://old.nabble.com/camel-casts-exception-when-there-is-nothing-to-fetch-from-the-ftp-endpoint-tp26541447p26542472.html
>>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>>
>>>
>>
>>
>>
>> --
>> Claus Ibsen
>> Apache Camel Committer
>>
>> Author of Camel in Action: http://www.manning.com/ibsen/
>> Open Source Integration: http://fusesource.com
>> Blog: http://davsclaus.blogspot.com/
>> Twitter: http://twitter.com/davsclaus
>>
>>
>
> --
> View this message in context: http://old.nabble.com/camel-casts-exception-when-there-is-nothing-to-fetch-from-the-ftp-endpoint-tp26541447p26543771.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>
>
--
Claus Ibsen
Apache Camel Committer
Author of Camel in Action: http://www.manning.com/ibsen/
Open Source Integration: http://fusesource.com
Blog: http://davsclaus.blogspot.com/
Twitter: http://twitter.com/davsclaus
Re: camel casts exception when there is nothing to fetch from the
ftp endpoint
Posted by shirazi <me...@farhad.eu>.
Thanks Claus,
I just downloaded the new trunk, and I think there's a new bug in it. I just
tried the file component with a couple of files around 3-400 MBs and it
works fine. As soon as I try it with a file around 3 Gigs, i get the
following exception (??!!):
16:13:48,830 ERROR [workflow] null while processing exchange:
Exchange[GenericFileMessage with file: GenericFile[/tmp/vw/cmore/tv5.ts]]
java.lang.NullPointerException
at
org.apache.camel.impl.OnExceptionRouteNode.getLabel(OnExceptionRouteNode.java:41)
at
org.apache.camel.processor.interceptor.DefaultTraceFormatter.getNodeMessage(DefaultTraceFormatter.java:233)
at
org.apache.camel.processor.interceptor.DefaultTraceFormatter.extractBreadCrumb(DefaultTraceFormatter.java:282)
at
org.apache.camel.processor.interceptor.DefaultTraceFormatter.format(DefaultTraceFormatter.java:55)
at
org.apache.camel.processor.interceptor.TraceInterceptor.format(TraceInterceptor.java:232)
at org.apache.camel.processor.Logger.logMessage(Logger.java:283)
at org.apache.camel.processor.Logger.process(Logger.java:166)
at
org.apache.camel.processor.interceptor.TraceInterceptor.logException(TraceInterceptor.java:325)
at
org.apache.camel.processor.interceptor.TraceInterceptor.process(TraceInterceptor.java:171)
at
org.apache.camel.processor.LoggingErrorHandler.process(LoggingErrorHandler.java:58)
at
org.apache.camel.processor.DefaultChannel.process(DefaultChannel.java:206)
at
org.apache.camel.processor.DelegateProcessor.processNext(DelegateProcessor.java:53)
at
org.apache.camel.processor.ChoiceProcessor.process(ChoiceProcessor.java:51)
at
org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:67)
at
org.apache.camel.processor.DelegateProcessor.processNext(DelegateProcessor.java:53)
at
org.apache.camel.processor.DelegateProcessor.proceed(DelegateProcessor.java:82)
at
org.apache.camel.processor.interceptor.TraceInterceptor.process(TraceInterceptor.java:162)
at
org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:67)
at
org.apache.camel.processor.LoggingErrorHandler.process(LoggingErrorHandler.java:58)
at
org.apache.camel.processor.DefaultChannel.process(DefaultChannel.java:206)
at
org.apache.camel.processor.UnitOfWorkProcessor.processNext(UnitOfWorkProcessor.java:70)
at
org.apache.camel.processor.DelegateProcessor.process(DelegateProcessor.java:48)
at
org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:67)
at
org.apache.camel.component.direct.DirectProducer.process(DirectProducer.java:45)
at
org.apache.camel.processor.SendProcessor$1.doInProducer(SendProcessor.java:97)
at
org.apache.camel.processor.SendProcessor$1.doInProducer(SendProcessor.java:95)
at org.apache.camel.impl.ProducerCache.doInProducer(ProducerCache.java:146)
at
org.apache.camel.processor.SendProcessor.doProcess(SendProcessor.java:94)
at org.apache.camel.processor.SendProcessor.process(SendProcessor.java:82)
at
org.apache.camel.processor.DelegateProcessor.processNext(DelegateProcessor.java:53)
at
org.apache.camel.processor.DelegateProcessor.proceed(DelegateProcessor.java:82)
at
org.apache.camel.processor.interceptor.TraceInterceptor.process(TraceInterceptor.java:162)
at
org.apache.camel.processor.LoggingErrorHandler.process(LoggingErrorHandler.java:58)
at
org.apache.camel.processor.DefaultChannel.process(DefaultChannel.java:206)
at org.apache.camel.processor.Pipeline.process(Pipeline.java:74)
at
org.apache.camel.processor.ChoiceProcessor.process(ChoiceProcessor.java:56)
at
org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:67)
at
org.apache.camel.processor.DelegateProcessor.processNext(DelegateProcessor.java:53)
at
org.apache.camel.processor.DelegateProcessor.proceed(DelegateProcessor.java:82)
at
org.apache.camel.processor.interceptor.TraceInterceptor.process(TraceInterceptor.java:162)
at
org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:67)
at
org.apache.camel.processor.LoggingErrorHandler.process(LoggingErrorHandler.java:58)
at
org.apache.camel.processor.DefaultChannel.process(DefaultChannel.java:206)
at
org.apache.camel.processor.UnitOfWorkProcessor.processNext(UnitOfWorkProcessor.java:54)
at
org.apache.camel.processor.DelegateProcessor.process(DelegateProcessor.java:48)
at
org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:67)
at
org.apache.camel.component.file.GenericFileConsumer.processExchange(GenericFileConsumer.java:201)
at
org.apache.camel.component.file.GenericFileConsumer.processBatch(GenericFileConsumer.java:122)
at
org.apache.camel.component.file.GenericFileConsumer.poll(GenericFileConsumer.java:95)
at
org.apache.camel.impl.ScheduledPollConsumer.run(ScheduledPollConsumer.java:108)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:417)
at
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:280)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:135)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:65)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:142)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:166)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
at java.lang.Thread.run(Thread.java:613)
Claus Ibsen-2 wrote:
>
> On Fri, Nov 27, 2009 at 2:34 PM, shirazi <me...@farhad.eu> wrote:
>>
>> Oh, sorry. that's right, it is 64 MB. and I tried with 16, 32, 64 and 128
>> MB.
>> I will try the buffer size option of the file-comp. to see if it solves
>> the
>> problem.
>>
>
> I have just committed the fix to camel trunk.
>
> The old code will still have the 2gb limit bug no matter what
> bufferSize you use.
> However you should try the latest code in 2.1 to see if that works for
> you.
> http://camel.apache.org/download.html
>
>
>
>> Thanks,
>> Farhad S.
>>
>>
>> Claus Ibsen-2 wrote:
>>>
>>> On Fri, Nov 27, 2009 at 2:26 PM, shirazi <me...@farhad.eu> wrote:
>>>>
>>>> That's probably the case. I've tried with different buffer sizes
>>>> (though
>>>> not
>>>> 0), from 16 to 128 KB. I got the best performance with buffer of 64KB,
>>>> on
>>>> Linux Debian with 4 Gigs RAM, and files larger than 3 Gigs.
>>>> That would be great, if one could specify the buffer-size for the
>>>> "file"
>>>> component.
>>>>
>>>
>>> You buffer size was 64 * 1024 * 1024 which is 64 K * K = 64 MB. Or is
>>> there something that play tricks on me?
>>>
>>> The file component already have a bufferSize option with a default of
>>> 128kb
>>> http://camel.apache.org/file2.html
>>>
>>> So you can use that to set what you want to use.
>>>
>>>
>>>
>>>> -Farhad S.
>>>>
>>>>
>>>> Claus Ibsen-2 wrote:
>>>>>
>>>>> On Fri, Nov 27, 2009 at 2:00 PM, shirazi <me...@farhad.eu> wrote:
>>>>>>
>>>>>> Sure. I have attached the file.
>>>>>> Actually I use from->to components for transfering files and I guess
>>>>>> it
>>>>>> copies the file, rather than moving.
>>>>>>
>>>>>
>>>>> Ah makes more sense for copying to leverage NIO.
>>>>>
>>>>> Camel does this also but I can see that it may hit that 2gb limit as
>>>>> it doesnt transfer in a loop using a lower buffer.
>>>>>
>>>>> It was doing it in a one liner
>>>>> in.transferTo(0, in.size(), out);
>>>>>
>>>>> I wonder if you really must do that in a loop to avoid any 2gb limit
>>>>> or the likes?
>>>>>
>>>>>
>>>>>> Thanks,
>>>>>> Farhad S.
>>>>>> http://old.nabble.com/file/p26542125/Nio.java Nio.java
>>>>>>
>>>>>>
>>>>>> Claus Ibsen-2 wrote:
>>>>>>>
>>>>>>> On Fri, Nov 27, 2009 at 1:37 PM, shirazi <me...@farhad.eu> wrote:
>>>>>>>>
>>>>>>>> Wau... thanks for that. I don't understand how I could miss it,
>>>>>>>> cause
>>>>>>>> i
>>>>>>>> searched a lot for such a feature.
>>>>>>>>
>>>>>>>> Another thing that I have experienced in 2.0, is that the "file"
>>>>>>>> component
>>>>>>>> is unable to move/copy files larger than 2Gigs, such large files
>>>>>>>> are
>>>>>>>> cut
>>>>>>>> around 2 Gigs limit. Now, I'm not sure if it has anything to do
>>>>>>>> with
>>>>>>>> the
>>>>>>>> OS
>>>>>>>> or the filesystems or what. I have seen it both in Linux (debian)
>>>>>>>> and
>>>>>>>> Mac
>>>>>>>> OS
>>>>>>>> X. And I don't understand either, why the file component does not
>>>>>>>> use
>>>>>>>> the
>>>>>>>> NIO api or at least we don't have an option to tell it use the NIO
>>>>>>>> api
>>>>>>>> instead.
>>>>>>>> Anyway, for now I've solved the problem by using a smiple component
>>>>>>>> instead
>>>>>>>> of the "file" comp. to move/copy files using the NIO api. It is
>>>>>>>> much
>>>>>>>> faster
>>>>>>>> and it has no problem with very large files.
>>>>>>>>
>>>>>>>
>>>>>>> For move it uses java.io.File rename and are you saying its not
>>>>>>> optimized for the underlying OS?
>>>>>>> I would assume that it could move a 2gb file in no time if the file
>>>>>>> is
>>>>>>> on the same share.
>>>>>>>
>>>>>>> Can you share your code that you are using?
>>>>>>>
>>>>>>>
>>>>>>>> Bests,
>>>>>>>> Farhad S.
>>>>>>>>
>>>>>>>>
>>>>>>>> Claus Ibsen-2 wrote:
>>>>>>>>>
>>>>>>>>> On Fri, Nov 27, 2009 at 1:12 PM, shirazi <me...@farhad.eu> wrote:
>>>>>>>>>>
>>>>>>>>>> Hi,
>>>>>>>>>> Ok, I guess I should rather provide an AggregationStrategy.
>>>>>>>>>> The reason why I use a timer is that when I use the ftp endpoint,
>>>>>>>>>> it
>>>>>>>>>> starts
>>>>>>>>>> to fetch everything that is on the ftp in the first call and
>>>>>>>>>> that's
>>>>>>>>>> not
>>>>>>>>>> what
>>>>>>>>>> i want, because I'm processing files over 3 Gigs and there are
>>>>>>>>>> many
>>>>>>>>>> of
>>>>>>>>>> them.
>>>>>>>>>> Then I figured that by using a timer, i could have ftp to
>>>>>>>>>> download
>>>>>>>>>> one
>>>>>>>>>> file
>>>>>>>>>> at a time!!
>>>>>>>>>> What I basically need is to somehow tell the ftp, how many files
>>>>>>>>>> I
>>>>>>>>>> want
>>>>>>>>>> to
>>>>>>>>>> fetch at a time. This is probably doable now in camel-2.1 by
>>>>>>>>>> using
>>>>>>>>>> the
>>>>>>>>>> "ThrottlingInflightRoutePolicy", which i havn't tried yet.
>>>>>>>>>> Do you have any suggestions?
>>>>>>>>>
>>>>>>>>> Yeah the FTP component extends the File component so you got all
>>>>>>>>> the
>>>>>>>>> options from the file component as well
>>>>>>>>> http://camel.apache.org/file2.html
>>>>>>>>>
>>>>>>>>> And there is a maxMessagesPerPoll you can use to tell how many
>>>>>>>>> files
>>>>>>>>> you want at most to poll. For example = 1 in your use.
>>>>>>>>>
>>>>>>>>> And that option is avail in 2.0 also.
>>>>>>>>>
>>>>>>>>> In fact all these are as they are batch polling consumers
>>>>>>>>> http://camel.apache.org/batch-consumer.html
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Bests,
>>>>>>>>>> Farhad S.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Claus Ibsen-2 wrote:
>>>>>>>>>>>
>>>>>>>>>>> Hi
>>>>>>>>>>>
>>>>>>>>>>> BTW why are you using a timer -> ftp when the ftp have a build
>>>>>>>>>>> in
>>>>>>>>>>> timer?
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Fri, Nov 27, 2009 at 12:55 PM, shirazi <me...@farhad.eu> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> Hi,
>>>>>>>>>>>> Please consider the following route:
>>>>>>>>>>>>
>>>>>>>>>>>> from("timer://someTrigger?delay=30000&fixedRate=true&period=300000").
>>>>>>>>>>>> pollEnrich("ftp://cmoreFtp?binary=true&localWorkDirectory=/tmp&idempotent=true&idempotentRepository=#caStreams",
>>>>>>>>>>>> 10000).
>>>>>>>>>>>> to("file:///tmp/vw/cmore?tempPrefix=.inprogress");
>>>>>>>>>>>>
>>>>>>>>>>>> The pollEnrich cast a NullPointerException in camel-2.0, if
>>>>>>>>>>>> there
>>>>>>>>>>>> is
>>>>>>>>>>>> nothing
>>>>>>>>>>>> to fetch from the ftp. I got rid of the exception by a
>>>>>>>>>>>> "try-catch(NullpointerExcpetion.class).stop()" around the
>>>>>>>>>>>> route.
>>>>>>>>>>>> Now
>>>>>>>>>>>> in
>>>>>>>>>>>> camel-2.1-SNAPSHOT, I get the following exception on the "to"
>>>>>>>>>>>> endpoint:
>>>>>>>>>>>> =================
>>>>>>>>>>>> 12:37:06,899 INFO [Tracer] >>> -->
>>>>>>>>>>>> file:///tmp/vw/cmore?tempPrefix=.inprogress_, Pattern:InOnly,
>>>>>>>>>>>> Headers:{firedTime=Fri Nov 27 12:36:56 CET 2009},
>>>>>>>>>>>> Exception:org.apache.camel.component.file.GenericFileOperationFailedException:
>>>>>>>>>>>> Cannot store file:
>>>>>>>>>>>> /tmp/vw/cmore/.inprogress_02216f38-8ffc-4aec-b638-27229d857075
>>>>>>>>>>>> 12:37:06,899 INFO [Tracer] >>> -->
>>>>>>>>>>>> file:///tmp/vw/cmore?tempPrefix=.inprogress_, Pattern:InOnly,
>>>>>>>>>>>> Headers:{firedTime=Fri Nov 27 12:36:56 CET 2009},
>>>>>>>>>>>> Exception:org.apache.camel.component.file.GenericFileOperationFailedException:
>>>>>>>>>>>> Cannot store file:
>>>>>>>>>>>> /tmp/vw/cmore/.inprogress_02216f38-8ffc-4aec-b638-27229d857075
>>>>>>>>>>>> 12:37:06,899 ERROR [TimerConsumer] Cannot store file:
>>>>>>>>>>>> /tmp/vw/cmore/.inprogress_02216f38-8ffc-4aec-b638-27229d857075
>>>>>>>>>>>> org.apache.camel.component.file.GenericFileOperationFailedException:
>>>>>>>>>>>> Cannot
>>>>>>>>>>>> store file:
>>>>>>>>>>>> /tmp/vw/cmore/.inprogress_02216f38-8ffc-4aec-b638-27229d857075
>>>>>>>>>>>> at
>>>>>>>>>>>> org.apache.camel.component.file.FileOperations.storeFile(FileOperations.java:204)
>>>>>>>>>>>> at
>>>>>>>>>>>> org.apache.camel.component.file.GenericFileProducer.writeFile(GenericFileProducer.java:184)
>>>>>>>>>>>> at
>>>>>>>>>>>> org.apache.camel.component.file.GenericFileProducer.processExchange(GenericFileProducer.java:126)
>>>>>>>>>>>> at
>>>>>>>>>>>> org.apache.camel.component.file.GenericFileProducer.process(GenericFileProducer.java:57)
>>>>>>>>>>>> at
>>>>>>>>>>>> org.apache.camel.processor.SendProcessor$1.doInProducer(SendProcessor.java:97)
>>>>>>>>>>>> at
>>>>>>>>>>>> org.apache.camel.processor.SendProcessor$1.doInProducer(SendProcessor.java:95)
>>>>>>>>>>>> at
>>>>>>>>>>>> org.apache.camel.impl.ProducerCache.doInProducer(ProducerCache.java:146)
>>>>>>>>>>>> at
>>>>>>>>>>>> org.apache.camel.processor.SendProcessor.doProcess(SendProcessor.java:94)
>>>>>>>>>>>> at
>>>>>>>>>>>> org.apache.camel.processor.SendProcessor.process(SendProcessor.java:82)
>>>>>>>>>>>> at
>>>>>>>>>>>> org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:67)
>>>>>>>>>>>> at
>>>>>>>>>>>> org.apache.camel.processor.DelegateProcessor.processNext(DelegateProcessor.java:53)
>>>>>>>>>>>> at
>>>>>>>>>>>> org.apache.camel.processor.DelegateProcessor.proceed(DelegateProcessor.java:82)
>>>>>>>>>>>> at
>>>>>>>>>>>> org.apache.camel.processor.interceptor.TraceInterceptor.process(TraceInterceptor.java:162)
>>>>>>>>>>>> at
>>>>>>>>>>>> org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:67)
>>>>>>>>>>>> at
>>>>>>>>>>>> org.apache.camel.processor.DefaultChannel.process(DefaultChannel.java:206)
>>>>>>>>>>>> at
>>>>>>>>>>>> org.apache.camel.processor.Pipeline.process(Pipeline.java:74)
>>>>>>>>>>>> at
>>>>>>>>>>>> org.apache.camel.processor.TryProcessor.process(TryProcessor.java:63)
>>>>>>>>>>>> at
>>>>>>>>>>>> org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:67)
>>>>>>>>>>>> at
>>>>>>>>>>>> org.apache.camel.processor.DelegateProcessor.processNext(DelegateProcessor.java:53)
>>>>>>>>>>>> at
>>>>>>>>>>>> org.apache.camel.processor.DelegateProcessor.proceed(DelegateProcessor.java:82)
>>>>>>>>>>>> at
>>>>>>>>>>>> org.apache.camel.processor.interceptor.TraceInterceptor.process(TraceInterceptor.java:162)
>>>>>>>>>>>> at
>>>>>>>>>>>> org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:67)
>>>>>>>>>>>> at
>>>>>>>>>>>> org.apache.camel.processor.DefaultChannel.process(DefaultChannel.java:206)
>>>>>>>>>>>> at
>>>>>>>>>>>> org.apache.camel.processor.UnitOfWorkProcessor.processNext(UnitOfWorkProcessor.java:54)
>>>>>>>>>>>> at
>>>>>>>>>>>> org.apache.camel.processor.DelegateProcessor.process(DelegateProcessor.java:48)
>>>>>>>>>>>> at
>>>>>>>>>>>> org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:67)
>>>>>>>>>>>> at
>>>>>>>>>>>> org.apache.camel.component.timer.TimerConsumer.sendTimerExchange(TimerConsumer.java:103)
>>>>>>>>>>>> at
>>>>>>>>>>>> org.apache.camel.component.timer.TimerConsumer$1.run(TimerConsumer.java:50)
>>>>>>>>>>>> at java.util.TimerThread.mainLoop(Timer.java:512)
>>>>>>>>>>>> at java.util.TimerThread.run(Timer.java:462)
>>>>>>>>>>>> Caused by: org.apache.camel.InvalidPayloadException: No body
>>>>>>>>>>>> available
>>>>>>>>>>>> of
>>>>>>>>>>>> type: java.io.InputStream on: Message: [Body is null]. Caused
>>>>>>>>>>>> by:
>>>>>>>>>>>> No
>>>>>>>>>>>> type
>>>>>>>>>>>> converter available to convert from type: null to the required
>>>>>>>>>>>> type:
>>>>>>>>>>>> java.io.InputStream with value null on the exchange:
>>>>>>>>>>>> Exchange[Message:
>>>>>>>>>>>> [Body
>>>>>>>>>>>> is null]]
>>>>>>>>>>>> at
>>>>>>>>>>>> org.apache.camel.impl.MessageSupport.getMandatoryBody(MessageSupport.java:103)
>>>>>>>>>>>> at
>>>>>>>>>>>> org.apache.camel.util.ExchangeHelper.getMandatoryInBody(ExchangeHelper.java:116)
>>>>>>>>>>>> at
>>>>>>>>>>>> org.apache.camel.component.file.FileOperations.storeFile(FileOperations.java:198)
>>>>>>>>>>>> ... 29 more
>>>>>>>>>>>> Caused by: org.apache.camel.NoTypeConversionAvailableException:
>>>>>>>>>>>> No
>>>>>>>>>>>> type
>>>>>>>>>>>> converter available to convert from type: null to the required
>>>>>>>>>>>> type:
>>>>>>>>>>>> java.io.InputStream with value null
>>>>>>>>>>>> at
>>>>>>>>>>>> org.apache.camel.impl.converter.DefaultTypeConverter.mandatoryConvertTo(DefaultTypeConverter.java:126)
>>>>>>>>>>>> at
>>>>>>>>>>>> org.apache.camel.impl.MessageSupport.getMandatoryBody(MessageSupport.java:101)
>>>>>>>>>>>> ... 31 more
>>>>>>>>>>>> ============
>>>>>>>>>>>> Now, I can remedy the situation by a try-catch block or an
>>>>>>>>>>>> interceptor,
>>>>>>>>>>>> but
>>>>>>>>>>>> in my opinion this should be considered a bug in camel and it
>>>>>>>>>>>> should
>>>>>>>>>>>> silently ignore if there was nothing to fetch from the ftp.
>>>>>>>>>>>>
>>>>>>>>>>>> Bests,
>>>>>>>>>>>> Farhad Shiraz
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> View this message in context:
>>>>>>>>>>>> http://old.nabble.com/camel-casts-exception-when-there-is-nothing-to-fetch-from-the-ftp-endpoint-tp26541447p26541447.html
>>>>>>>>>>>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Claus Ibsen
>>>>>>>>>>> Apache Camel Committer
>>>>>>>>>>>
>>>>>>>>>>> Author of Camel in Action: http://www.manning.com/ibsen/
>>>>>>>>>>> Open Source Integration: http://fusesource.com
>>>>>>>>>>> Blog: http://davsclaus.blogspot.com/
>>>>>>>>>>> Twitter: http://twitter.com/davsclaus
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> View this message in context:
>>>>>>>>>> http://old.nabble.com/camel-casts-exception-when-there-is-nothing-to-fetch-from-the-ftp-endpoint-tp26541447p26541650.html
>>>>>>>>>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Claus Ibsen
>>>>>>>>> Apache Camel Committer
>>>>>>>>>
>>>>>>>>> Author of Camel in Action: http://www.manning.com/ibsen/
>>>>>>>>> Open Source Integration: http://fusesource.com
>>>>>>>>> Blog: http://davsclaus.blogspot.com/
>>>>>>>>> Twitter: http://twitter.com/davsclaus
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> View this message in context:
>>>>>>>> http://old.nabble.com/camel-casts-exception-when-there-is-nothing-to-fetch-from-the-ftp-endpoint-tp26541447p26541877.html
>>>>>>>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Claus Ibsen
>>>>>>> Apache Camel Committer
>>>>>>>
>>>>>>> Author of Camel in Action: http://www.manning.com/ibsen/
>>>>>>> Open Source Integration: http://fusesource.com
>>>>>>> Blog: http://davsclaus.blogspot.com/
>>>>>>> Twitter: http://twitter.com/davsclaus
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> --
>>>>>> View this message in context:
>>>>>> http://old.nabble.com/camel-casts-exception-when-there-is-nothing-to-fetch-from-the-ftp-endpoint-tp26541447p26542125.html
>>>>>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Claus Ibsen
>>>>> Apache Camel Committer
>>>>>
>>>>> Author of Camel in Action: http://www.manning.com/ibsen/
>>>>> Open Source Integration: http://fusesource.com
>>>>> Blog: http://davsclaus.blogspot.com/
>>>>> Twitter: http://twitter.com/davsclaus
>>>>>
>>>>>
>>>>
>>>> --
>>>> View this message in context:
>>>> http://old.nabble.com/camel-casts-exception-when-there-is-nothing-to-fetch-from-the-ftp-endpoint-tp26541447p26542396.html
>>>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>>>
>>>>
>>>
>>>
>>>
>>> --
>>> Claus Ibsen
>>> Apache Camel Committer
>>>
>>> Author of Camel in Action: http://www.manning.com/ibsen/
>>> Open Source Integration: http://fusesource.com
>>> Blog: http://davsclaus.blogspot.com/
>>> Twitter: http://twitter.com/davsclaus
>>>
>>>
>>
>> --
>> View this message in context:
>> http://old.nabble.com/camel-casts-exception-when-there-is-nothing-to-fetch-from-the-ftp-endpoint-tp26541447p26542472.html
>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>
>>
>
>
>
> --
> Claus Ibsen
> Apache Camel Committer
>
> Author of Camel in Action: http://www.manning.com/ibsen/
> Open Source Integration: http://fusesource.com
> Blog: http://davsclaus.blogspot.com/
> Twitter: http://twitter.com/davsclaus
>
>
--
View this message in context: http://old.nabble.com/camel-casts-exception-when-there-is-nothing-to-fetch-from-the-ftp-endpoint-tp26541447p26543771.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: camel casts exception when there is nothing to fetch from the ftp
endpoint
Posted by Claus Ibsen <cl...@gmail.com>.
On Fri, Nov 27, 2009 at 2:34 PM, shirazi <me...@farhad.eu> wrote:
>
> Oh, sorry. that's right, it is 64 MB. and I tried with 16, 32, 64 and 128 MB.
> I will try the buffer size option of the file-comp. to see if it solves the
> problem.
>
I have just committed the fix to camel trunk.
The old code will still have the 2gb limit bug no matter what
bufferSize you use.
However you should try the latest code in 2.1 to see if that works for you.
http://camel.apache.org/download.html
> Thanks,
> Farhad S.
>
>
> Claus Ibsen-2 wrote:
>>
>> On Fri, Nov 27, 2009 at 2:26 PM, shirazi <me...@farhad.eu> wrote:
>>>
>>> That's probably the case. I've tried with different buffer sizes (though
>>> not
>>> 0), from 16 to 128 KB. I got the best performance with buffer of 64KB, on
>>> Linux Debian with 4 Gigs RAM, and files larger than 3 Gigs.
>>> That would be great, if one could specify the buffer-size for the "file"
>>> component.
>>>
>>
>> You buffer size was 64 * 1024 * 1024 which is 64 K * K = 64 MB. Or is
>> there something that play tricks on me?
>>
>> The file component already have a bufferSize option with a default of
>> 128kb
>> http://camel.apache.org/file2.html
>>
>> So you can use that to set what you want to use.
>>
>>
>>
>>> -Farhad S.
>>>
>>>
>>> Claus Ibsen-2 wrote:
>>>>
>>>> On Fri, Nov 27, 2009 at 2:00 PM, shirazi <me...@farhad.eu> wrote:
>>>>>
>>>>> Sure. I have attached the file.
>>>>> Actually I use from->to components for transfering files and I guess it
>>>>> copies the file, rather than moving.
>>>>>
>>>>
>>>> Ah makes more sense for copying to leverage NIO.
>>>>
>>>> Camel does this also but I can see that it may hit that 2gb limit as
>>>> it doesnt transfer in a loop using a lower buffer.
>>>>
>>>> It was doing it in a one liner
>>>> in.transferTo(0, in.size(), out);
>>>>
>>>> I wonder if you really must do that in a loop to avoid any 2gb limit
>>>> or the likes?
>>>>
>>>>
>>>>> Thanks,
>>>>> Farhad S.
>>>>> http://old.nabble.com/file/p26542125/Nio.java Nio.java
>>>>>
>>>>>
>>>>> Claus Ibsen-2 wrote:
>>>>>>
>>>>>> On Fri, Nov 27, 2009 at 1:37 PM, shirazi <me...@farhad.eu> wrote:
>>>>>>>
>>>>>>> Wau... thanks for that. I don't understand how I could miss it, cause
>>>>>>> i
>>>>>>> searched a lot for such a feature.
>>>>>>>
>>>>>>> Another thing that I have experienced in 2.0, is that the "file"
>>>>>>> component
>>>>>>> is unable to move/copy files larger than 2Gigs, such large files are
>>>>>>> cut
>>>>>>> around 2 Gigs limit. Now, I'm not sure if it has anything to do with
>>>>>>> the
>>>>>>> OS
>>>>>>> or the filesystems or what. I have seen it both in Linux (debian) and
>>>>>>> Mac
>>>>>>> OS
>>>>>>> X. And I don't understand either, why the file component does not use
>>>>>>> the
>>>>>>> NIO api or at least we don't have an option to tell it use the NIO
>>>>>>> api
>>>>>>> instead.
>>>>>>> Anyway, for now I've solved the problem by using a smiple component
>>>>>>> instead
>>>>>>> of the "file" comp. to move/copy files using the NIO api. It is much
>>>>>>> faster
>>>>>>> and it has no problem with very large files.
>>>>>>>
>>>>>>
>>>>>> For move it uses java.io.File rename and are you saying its not
>>>>>> optimized for the underlying OS?
>>>>>> I would assume that it could move a 2gb file in no time if the file is
>>>>>> on the same share.
>>>>>>
>>>>>> Can you share your code that you are using?
>>>>>>
>>>>>>
>>>>>>> Bests,
>>>>>>> Farhad S.
>>>>>>>
>>>>>>>
>>>>>>> Claus Ibsen-2 wrote:
>>>>>>>>
>>>>>>>> On Fri, Nov 27, 2009 at 1:12 PM, shirazi <me...@farhad.eu> wrote:
>>>>>>>>>
>>>>>>>>> Hi,
>>>>>>>>> Ok, I guess I should rather provide an AggregationStrategy.
>>>>>>>>> The reason why I use a timer is that when I use the ftp endpoint,
>>>>>>>>> it
>>>>>>>>> starts
>>>>>>>>> to fetch everything that is on the ftp in the first call and that's
>>>>>>>>> not
>>>>>>>>> what
>>>>>>>>> i want, because I'm processing files over 3 Gigs and there are many
>>>>>>>>> of
>>>>>>>>> them.
>>>>>>>>> Then I figured that by using a timer, i could have ftp to download
>>>>>>>>> one
>>>>>>>>> file
>>>>>>>>> at a time!!
>>>>>>>>> What I basically need is to somehow tell the ftp, how many files I
>>>>>>>>> want
>>>>>>>>> to
>>>>>>>>> fetch at a time. This is probably doable now in camel-2.1 by using
>>>>>>>>> the
>>>>>>>>> "ThrottlingInflightRoutePolicy", which i havn't tried yet.
>>>>>>>>> Do you have any suggestions?
>>>>>>>>
>>>>>>>> Yeah the FTP component extends the File component so you got all the
>>>>>>>> options from the file component as well
>>>>>>>> http://camel.apache.org/file2.html
>>>>>>>>
>>>>>>>> And there is a maxMessagesPerPoll you can use to tell how many files
>>>>>>>> you want at most to poll. For example = 1 in your use.
>>>>>>>>
>>>>>>>> And that option is avail in 2.0 also.
>>>>>>>>
>>>>>>>> In fact all these are as they are batch polling consumers
>>>>>>>> http://camel.apache.org/batch-consumer.html
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>>
>>>>>>>>> Bests,
>>>>>>>>> Farhad S.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Claus Ibsen-2 wrote:
>>>>>>>>>>
>>>>>>>>>> Hi
>>>>>>>>>>
>>>>>>>>>> BTW why are you using a timer -> ftp when the ftp have a build in
>>>>>>>>>> timer?
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Fri, Nov 27, 2009 at 12:55 PM, shirazi <me...@farhad.eu> wrote:
>>>>>>>>>>>
>>>>>>>>>>> Hi,
>>>>>>>>>>> Please consider the following route:
>>>>>>>>>>>
>>>>>>>>>>> from("timer://someTrigger?delay=30000&fixedRate=true&period=300000").
>>>>>>>>>>> pollEnrich("ftp://cmoreFtp?binary=true&localWorkDirectory=/tmp&idempotent=true&idempotentRepository=#caStreams",
>>>>>>>>>>> 10000).
>>>>>>>>>>> to("file:///tmp/vw/cmore?tempPrefix=.inprogress");
>>>>>>>>>>>
>>>>>>>>>>> The pollEnrich cast a NullPointerException in camel-2.0, if there
>>>>>>>>>>> is
>>>>>>>>>>> nothing
>>>>>>>>>>> to fetch from the ftp. I got rid of the exception by a
>>>>>>>>>>> "try-catch(NullpointerExcpetion.class).stop()" around the route.
>>>>>>>>>>> Now
>>>>>>>>>>> in
>>>>>>>>>>> camel-2.1-SNAPSHOT, I get the following exception on the "to"
>>>>>>>>>>> endpoint:
>>>>>>>>>>> =================
>>>>>>>>>>> 12:37:06,899 INFO [Tracer] >>> -->
>>>>>>>>>>> file:///tmp/vw/cmore?tempPrefix=.inprogress_, Pattern:InOnly,
>>>>>>>>>>> Headers:{firedTime=Fri Nov 27 12:36:56 CET 2009},
>>>>>>>>>>> Exception:org.apache.camel.component.file.GenericFileOperationFailedException:
>>>>>>>>>>> Cannot store file:
>>>>>>>>>>> /tmp/vw/cmore/.inprogress_02216f38-8ffc-4aec-b638-27229d857075
>>>>>>>>>>> 12:37:06,899 INFO [Tracer] >>> -->
>>>>>>>>>>> file:///tmp/vw/cmore?tempPrefix=.inprogress_, Pattern:InOnly,
>>>>>>>>>>> Headers:{firedTime=Fri Nov 27 12:36:56 CET 2009},
>>>>>>>>>>> Exception:org.apache.camel.component.file.GenericFileOperationFailedException:
>>>>>>>>>>> Cannot store file:
>>>>>>>>>>> /tmp/vw/cmore/.inprogress_02216f38-8ffc-4aec-b638-27229d857075
>>>>>>>>>>> 12:37:06,899 ERROR [TimerConsumer] Cannot store file:
>>>>>>>>>>> /tmp/vw/cmore/.inprogress_02216f38-8ffc-4aec-b638-27229d857075
>>>>>>>>>>> org.apache.camel.component.file.GenericFileOperationFailedException:
>>>>>>>>>>> Cannot
>>>>>>>>>>> store file:
>>>>>>>>>>> /tmp/vw/cmore/.inprogress_02216f38-8ffc-4aec-b638-27229d857075
>>>>>>>>>>> at
>>>>>>>>>>> org.apache.camel.component.file.FileOperations.storeFile(FileOperations.java:204)
>>>>>>>>>>> at
>>>>>>>>>>> org.apache.camel.component.file.GenericFileProducer.writeFile(GenericFileProducer.java:184)
>>>>>>>>>>> at
>>>>>>>>>>> org.apache.camel.component.file.GenericFileProducer.processExchange(GenericFileProducer.java:126)
>>>>>>>>>>> at
>>>>>>>>>>> org.apache.camel.component.file.GenericFileProducer.process(GenericFileProducer.java:57)
>>>>>>>>>>> at
>>>>>>>>>>> org.apache.camel.processor.SendProcessor$1.doInProducer(SendProcessor.java:97)
>>>>>>>>>>> at
>>>>>>>>>>> org.apache.camel.processor.SendProcessor$1.doInProducer(SendProcessor.java:95)
>>>>>>>>>>> at
>>>>>>>>>>> org.apache.camel.impl.ProducerCache.doInProducer(ProducerCache.java:146)
>>>>>>>>>>> at
>>>>>>>>>>> org.apache.camel.processor.SendProcessor.doProcess(SendProcessor.java:94)
>>>>>>>>>>> at
>>>>>>>>>>> org.apache.camel.processor.SendProcessor.process(SendProcessor.java:82)
>>>>>>>>>>> at
>>>>>>>>>>> org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:67)
>>>>>>>>>>> at
>>>>>>>>>>> org.apache.camel.processor.DelegateProcessor.processNext(DelegateProcessor.java:53)
>>>>>>>>>>> at
>>>>>>>>>>> org.apache.camel.processor.DelegateProcessor.proceed(DelegateProcessor.java:82)
>>>>>>>>>>> at
>>>>>>>>>>> org.apache.camel.processor.interceptor.TraceInterceptor.process(TraceInterceptor.java:162)
>>>>>>>>>>> at
>>>>>>>>>>> org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:67)
>>>>>>>>>>> at
>>>>>>>>>>> org.apache.camel.processor.DefaultChannel.process(DefaultChannel.java:206)
>>>>>>>>>>> at
>>>>>>>>>>> org.apache.camel.processor.Pipeline.process(Pipeline.java:74)
>>>>>>>>>>> at
>>>>>>>>>>> org.apache.camel.processor.TryProcessor.process(TryProcessor.java:63)
>>>>>>>>>>> at
>>>>>>>>>>> org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:67)
>>>>>>>>>>> at
>>>>>>>>>>> org.apache.camel.processor.DelegateProcessor.processNext(DelegateProcessor.java:53)
>>>>>>>>>>> at
>>>>>>>>>>> org.apache.camel.processor.DelegateProcessor.proceed(DelegateProcessor.java:82)
>>>>>>>>>>> at
>>>>>>>>>>> org.apache.camel.processor.interceptor.TraceInterceptor.process(TraceInterceptor.java:162)
>>>>>>>>>>> at
>>>>>>>>>>> org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:67)
>>>>>>>>>>> at
>>>>>>>>>>> org.apache.camel.processor.DefaultChannel.process(DefaultChannel.java:206)
>>>>>>>>>>> at
>>>>>>>>>>> org.apache.camel.processor.UnitOfWorkProcessor.processNext(UnitOfWorkProcessor.java:54)
>>>>>>>>>>> at
>>>>>>>>>>> org.apache.camel.processor.DelegateProcessor.process(DelegateProcessor.java:48)
>>>>>>>>>>> at
>>>>>>>>>>> org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:67)
>>>>>>>>>>> at
>>>>>>>>>>> org.apache.camel.component.timer.TimerConsumer.sendTimerExchange(TimerConsumer.java:103)
>>>>>>>>>>> at
>>>>>>>>>>> org.apache.camel.component.timer.TimerConsumer$1.run(TimerConsumer.java:50)
>>>>>>>>>>> at java.util.TimerThread.mainLoop(Timer.java:512)
>>>>>>>>>>> at java.util.TimerThread.run(Timer.java:462)
>>>>>>>>>>> Caused by: org.apache.camel.InvalidPayloadException: No body
>>>>>>>>>>> available
>>>>>>>>>>> of
>>>>>>>>>>> type: java.io.InputStream on: Message: [Body is null]. Caused by:
>>>>>>>>>>> No
>>>>>>>>>>> type
>>>>>>>>>>> converter available to convert from type: null to the required
>>>>>>>>>>> type:
>>>>>>>>>>> java.io.InputStream with value null on the exchange:
>>>>>>>>>>> Exchange[Message:
>>>>>>>>>>> [Body
>>>>>>>>>>> is null]]
>>>>>>>>>>> at
>>>>>>>>>>> org.apache.camel.impl.MessageSupport.getMandatoryBody(MessageSupport.java:103)
>>>>>>>>>>> at
>>>>>>>>>>> org.apache.camel.util.ExchangeHelper.getMandatoryInBody(ExchangeHelper.java:116)
>>>>>>>>>>> at
>>>>>>>>>>> org.apache.camel.component.file.FileOperations.storeFile(FileOperations.java:198)
>>>>>>>>>>> ... 29 more
>>>>>>>>>>> Caused by: org.apache.camel.NoTypeConversionAvailableException:
>>>>>>>>>>> No
>>>>>>>>>>> type
>>>>>>>>>>> converter available to convert from type: null to the required
>>>>>>>>>>> type:
>>>>>>>>>>> java.io.InputStream with value null
>>>>>>>>>>> at
>>>>>>>>>>> org.apache.camel.impl.converter.DefaultTypeConverter.mandatoryConvertTo(DefaultTypeConverter.java:126)
>>>>>>>>>>> at
>>>>>>>>>>> org.apache.camel.impl.MessageSupport.getMandatoryBody(MessageSupport.java:101)
>>>>>>>>>>> ... 31 more
>>>>>>>>>>> ============
>>>>>>>>>>> Now, I can remedy the situation by a try-catch block or an
>>>>>>>>>>> interceptor,
>>>>>>>>>>> but
>>>>>>>>>>> in my opinion this should be considered a bug in camel and it
>>>>>>>>>>> should
>>>>>>>>>>> silently ignore if there was nothing to fetch from the ftp.
>>>>>>>>>>>
>>>>>>>>>>> Bests,
>>>>>>>>>>> Farhad Shiraz
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> View this message in context:
>>>>>>>>>>> http://old.nabble.com/camel-casts-exception-when-there-is-nothing-to-fetch-from-the-ftp-endpoint-tp26541447p26541447.html
>>>>>>>>>>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Claus Ibsen
>>>>>>>>>> Apache Camel Committer
>>>>>>>>>>
>>>>>>>>>> Author of Camel in Action: http://www.manning.com/ibsen/
>>>>>>>>>> Open Source Integration: http://fusesource.com
>>>>>>>>>> Blog: http://davsclaus.blogspot.com/
>>>>>>>>>> Twitter: http://twitter.com/davsclaus
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> View this message in context:
>>>>>>>>> http://old.nabble.com/camel-casts-exception-when-there-is-nothing-to-fetch-from-the-ftp-endpoint-tp26541447p26541650.html
>>>>>>>>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Claus Ibsen
>>>>>>>> Apache Camel Committer
>>>>>>>>
>>>>>>>> Author of Camel in Action: http://www.manning.com/ibsen/
>>>>>>>> Open Source Integration: http://fusesource.com
>>>>>>>> Blog: http://davsclaus.blogspot.com/
>>>>>>>> Twitter: http://twitter.com/davsclaus
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> View this message in context:
>>>>>>> http://old.nabble.com/camel-casts-exception-when-there-is-nothing-to-fetch-from-the-ftp-endpoint-tp26541447p26541877.html
>>>>>>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Claus Ibsen
>>>>>> Apache Camel Committer
>>>>>>
>>>>>> Author of Camel in Action: http://www.manning.com/ibsen/
>>>>>> Open Source Integration: http://fusesource.com
>>>>>> Blog: http://davsclaus.blogspot.com/
>>>>>> Twitter: http://twitter.com/davsclaus
>>>>>>
>>>>>>
>>>>>
>>>>> --
>>>>> View this message in context:
>>>>> http://old.nabble.com/camel-casts-exception-when-there-is-nothing-to-fetch-from-the-ftp-endpoint-tp26541447p26542125.html
>>>>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Claus Ibsen
>>>> Apache Camel Committer
>>>>
>>>> Author of Camel in Action: http://www.manning.com/ibsen/
>>>> Open Source Integration: http://fusesource.com
>>>> Blog: http://davsclaus.blogspot.com/
>>>> Twitter: http://twitter.com/davsclaus
>>>>
>>>>
>>>
>>> --
>>> View this message in context:
>>> http://old.nabble.com/camel-casts-exception-when-there-is-nothing-to-fetch-from-the-ftp-endpoint-tp26541447p26542396.html
>>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>>
>>>
>>
>>
>>
>> --
>> Claus Ibsen
>> Apache Camel Committer
>>
>> Author of Camel in Action: http://www.manning.com/ibsen/
>> Open Source Integration: http://fusesource.com
>> Blog: http://davsclaus.blogspot.com/
>> Twitter: http://twitter.com/davsclaus
>>
>>
>
> --
> View this message in context: http://old.nabble.com/camel-casts-exception-when-there-is-nothing-to-fetch-from-the-ftp-endpoint-tp26541447p26542472.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>
>
--
Claus Ibsen
Apache Camel Committer
Author of Camel in Action: http://www.manning.com/ibsen/
Open Source Integration: http://fusesource.com
Blog: http://davsclaus.blogspot.com/
Twitter: http://twitter.com/davsclaus
Re: camel casts exception when there is nothing to fetch from the
ftp endpoint
Posted by shirazi <me...@farhad.eu>.
Oh, sorry. that's right, it is 64 MB. and I tried with 16, 32, 64 and 128 MB.
I will try the buffer size option of the file-comp. to see if it solves the
problem.
Thanks,
Farhad S.
Claus Ibsen-2 wrote:
>
> On Fri, Nov 27, 2009 at 2:26 PM, shirazi <me...@farhad.eu> wrote:
>>
>> That's probably the case. I've tried with different buffer sizes (though
>> not
>> 0), from 16 to 128 KB. I got the best performance with buffer of 64KB, on
>> Linux Debian with 4 Gigs RAM, and files larger than 3 Gigs.
>> That would be great, if one could specify the buffer-size for the "file"
>> component.
>>
>
> You buffer size was 64 * 1024 * 1024 which is 64 K * K = 64 MB. Or is
> there something that play tricks on me?
>
> The file component already have a bufferSize option with a default of
> 128kb
> http://camel.apache.org/file2.html
>
> So you can use that to set what you want to use.
>
>
>
>> -Farhad S.
>>
>>
>> Claus Ibsen-2 wrote:
>>>
>>> On Fri, Nov 27, 2009 at 2:00 PM, shirazi <me...@farhad.eu> wrote:
>>>>
>>>> Sure. I have attached the file.
>>>> Actually I use from->to components for transfering files and I guess it
>>>> copies the file, rather than moving.
>>>>
>>>
>>> Ah makes more sense for copying to leverage NIO.
>>>
>>> Camel does this also but I can see that it may hit that 2gb limit as
>>> it doesnt transfer in a loop using a lower buffer.
>>>
>>> It was doing it in a one liner
>>> in.transferTo(0, in.size(), out);
>>>
>>> I wonder if you really must do that in a loop to avoid any 2gb limit
>>> or the likes?
>>>
>>>
>>>> Thanks,
>>>> Farhad S.
>>>> http://old.nabble.com/file/p26542125/Nio.java Nio.java
>>>>
>>>>
>>>> Claus Ibsen-2 wrote:
>>>>>
>>>>> On Fri, Nov 27, 2009 at 1:37 PM, shirazi <me...@farhad.eu> wrote:
>>>>>>
>>>>>> Wau... thanks for that. I don't understand how I could miss it, cause
>>>>>> i
>>>>>> searched a lot for such a feature.
>>>>>>
>>>>>> Another thing that I have experienced in 2.0, is that the "file"
>>>>>> component
>>>>>> is unable to move/copy files larger than 2Gigs, such large files are
>>>>>> cut
>>>>>> around 2 Gigs limit. Now, I'm not sure if it has anything to do with
>>>>>> the
>>>>>> OS
>>>>>> or the filesystems or what. I have seen it both in Linux (debian) and
>>>>>> Mac
>>>>>> OS
>>>>>> X. And I don't understand either, why the file component does not use
>>>>>> the
>>>>>> NIO api or at least we don't have an option to tell it use the NIO
>>>>>> api
>>>>>> instead.
>>>>>> Anyway, for now I've solved the problem by using a smiple component
>>>>>> instead
>>>>>> of the "file" comp. to move/copy files using the NIO api. It is much
>>>>>> faster
>>>>>> and it has no problem with very large files.
>>>>>>
>>>>>
>>>>> For move it uses java.io.File rename and are you saying its not
>>>>> optimized for the underlying OS?
>>>>> I would assume that it could move a 2gb file in no time if the file is
>>>>> on the same share.
>>>>>
>>>>> Can you share your code that you are using?
>>>>>
>>>>>
>>>>>> Bests,
>>>>>> Farhad S.
>>>>>>
>>>>>>
>>>>>> Claus Ibsen-2 wrote:
>>>>>>>
>>>>>>> On Fri, Nov 27, 2009 at 1:12 PM, shirazi <me...@farhad.eu> wrote:
>>>>>>>>
>>>>>>>> Hi,
>>>>>>>> Ok, I guess I should rather provide an AggregationStrategy.
>>>>>>>> The reason why I use a timer is that when I use the ftp endpoint,
>>>>>>>> it
>>>>>>>> starts
>>>>>>>> to fetch everything that is on the ftp in the first call and that's
>>>>>>>> not
>>>>>>>> what
>>>>>>>> i want, because I'm processing files over 3 Gigs and there are many
>>>>>>>> of
>>>>>>>> them.
>>>>>>>> Then I figured that by using a timer, i could have ftp to download
>>>>>>>> one
>>>>>>>> file
>>>>>>>> at a time!!
>>>>>>>> What I basically need is to somehow tell the ftp, how many files I
>>>>>>>> want
>>>>>>>> to
>>>>>>>> fetch at a time. This is probably doable now in camel-2.1 by using
>>>>>>>> the
>>>>>>>> "ThrottlingInflightRoutePolicy", which i havn't tried yet.
>>>>>>>> Do you have any suggestions?
>>>>>>>
>>>>>>> Yeah the FTP component extends the File component so you got all the
>>>>>>> options from the file component as well
>>>>>>> http://camel.apache.org/file2.html
>>>>>>>
>>>>>>> And there is a maxMessagesPerPoll you can use to tell how many files
>>>>>>> you want at most to poll. For example = 1 in your use.
>>>>>>>
>>>>>>> And that option is avail in 2.0 also.
>>>>>>>
>>>>>>> In fact all these are as they are batch polling consumers
>>>>>>> http://camel.apache.org/batch-consumer.html
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>> Bests,
>>>>>>>> Farhad S.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Claus Ibsen-2 wrote:
>>>>>>>>>
>>>>>>>>> Hi
>>>>>>>>>
>>>>>>>>> BTW why are you using a timer -> ftp when the ftp have a build in
>>>>>>>>> timer?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Fri, Nov 27, 2009 at 12:55 PM, shirazi <me...@farhad.eu> wrote:
>>>>>>>>>>
>>>>>>>>>> Hi,
>>>>>>>>>> Please consider the following route:
>>>>>>>>>>
>>>>>>>>>> from("timer://someTrigger?delay=30000&fixedRate=true&period=300000").
>>>>>>>>>> pollEnrich("ftp://cmoreFtp?binary=true&localWorkDirectory=/tmp&idempotent=true&idempotentRepository=#caStreams",
>>>>>>>>>> 10000).
>>>>>>>>>> to("file:///tmp/vw/cmore?tempPrefix=.inprogress");
>>>>>>>>>>
>>>>>>>>>> The pollEnrich cast a NullPointerException in camel-2.0, if there
>>>>>>>>>> is
>>>>>>>>>> nothing
>>>>>>>>>> to fetch from the ftp. I got rid of the exception by a
>>>>>>>>>> "try-catch(NullpointerExcpetion.class).stop()" around the route.
>>>>>>>>>> Now
>>>>>>>>>> in
>>>>>>>>>> camel-2.1-SNAPSHOT, I get the following exception on the "to"
>>>>>>>>>> endpoint:
>>>>>>>>>> =================
>>>>>>>>>> 12:37:06,899 INFO [Tracer] >>> -->
>>>>>>>>>> file:///tmp/vw/cmore?tempPrefix=.inprogress_, Pattern:InOnly,
>>>>>>>>>> Headers:{firedTime=Fri Nov 27 12:36:56 CET 2009},
>>>>>>>>>> Exception:org.apache.camel.component.file.GenericFileOperationFailedException:
>>>>>>>>>> Cannot store file:
>>>>>>>>>> /tmp/vw/cmore/.inprogress_02216f38-8ffc-4aec-b638-27229d857075
>>>>>>>>>> 12:37:06,899 INFO [Tracer] >>> -->
>>>>>>>>>> file:///tmp/vw/cmore?tempPrefix=.inprogress_, Pattern:InOnly,
>>>>>>>>>> Headers:{firedTime=Fri Nov 27 12:36:56 CET 2009},
>>>>>>>>>> Exception:org.apache.camel.component.file.GenericFileOperationFailedException:
>>>>>>>>>> Cannot store file:
>>>>>>>>>> /tmp/vw/cmore/.inprogress_02216f38-8ffc-4aec-b638-27229d857075
>>>>>>>>>> 12:37:06,899 ERROR [TimerConsumer] Cannot store file:
>>>>>>>>>> /tmp/vw/cmore/.inprogress_02216f38-8ffc-4aec-b638-27229d857075
>>>>>>>>>> org.apache.camel.component.file.GenericFileOperationFailedException:
>>>>>>>>>> Cannot
>>>>>>>>>> store file:
>>>>>>>>>> /tmp/vw/cmore/.inprogress_02216f38-8ffc-4aec-b638-27229d857075
>>>>>>>>>> at
>>>>>>>>>> org.apache.camel.component.file.FileOperations.storeFile(FileOperations.java:204)
>>>>>>>>>> at
>>>>>>>>>> org.apache.camel.component.file.GenericFileProducer.writeFile(GenericFileProducer.java:184)
>>>>>>>>>> at
>>>>>>>>>> org.apache.camel.component.file.GenericFileProducer.processExchange(GenericFileProducer.java:126)
>>>>>>>>>> at
>>>>>>>>>> org.apache.camel.component.file.GenericFileProducer.process(GenericFileProducer.java:57)
>>>>>>>>>> at
>>>>>>>>>> org.apache.camel.processor.SendProcessor$1.doInProducer(SendProcessor.java:97)
>>>>>>>>>> at
>>>>>>>>>> org.apache.camel.processor.SendProcessor$1.doInProducer(SendProcessor.java:95)
>>>>>>>>>> at
>>>>>>>>>> org.apache.camel.impl.ProducerCache.doInProducer(ProducerCache.java:146)
>>>>>>>>>> at
>>>>>>>>>> org.apache.camel.processor.SendProcessor.doProcess(SendProcessor.java:94)
>>>>>>>>>> at
>>>>>>>>>> org.apache.camel.processor.SendProcessor.process(SendProcessor.java:82)
>>>>>>>>>> at
>>>>>>>>>> org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:67)
>>>>>>>>>> at
>>>>>>>>>> org.apache.camel.processor.DelegateProcessor.processNext(DelegateProcessor.java:53)
>>>>>>>>>> at
>>>>>>>>>> org.apache.camel.processor.DelegateProcessor.proceed(DelegateProcessor.java:82)
>>>>>>>>>> at
>>>>>>>>>> org.apache.camel.processor.interceptor.TraceInterceptor.process(TraceInterceptor.java:162)
>>>>>>>>>> at
>>>>>>>>>> org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:67)
>>>>>>>>>> at
>>>>>>>>>> org.apache.camel.processor.DefaultChannel.process(DefaultChannel.java:206)
>>>>>>>>>> at
>>>>>>>>>> org.apache.camel.processor.Pipeline.process(Pipeline.java:74)
>>>>>>>>>> at
>>>>>>>>>> org.apache.camel.processor.TryProcessor.process(TryProcessor.java:63)
>>>>>>>>>> at
>>>>>>>>>> org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:67)
>>>>>>>>>> at
>>>>>>>>>> org.apache.camel.processor.DelegateProcessor.processNext(DelegateProcessor.java:53)
>>>>>>>>>> at
>>>>>>>>>> org.apache.camel.processor.DelegateProcessor.proceed(DelegateProcessor.java:82)
>>>>>>>>>> at
>>>>>>>>>> org.apache.camel.processor.interceptor.TraceInterceptor.process(TraceInterceptor.java:162)
>>>>>>>>>> at
>>>>>>>>>> org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:67)
>>>>>>>>>> at
>>>>>>>>>> org.apache.camel.processor.DefaultChannel.process(DefaultChannel.java:206)
>>>>>>>>>> at
>>>>>>>>>> org.apache.camel.processor.UnitOfWorkProcessor.processNext(UnitOfWorkProcessor.java:54)
>>>>>>>>>> at
>>>>>>>>>> org.apache.camel.processor.DelegateProcessor.process(DelegateProcessor.java:48)
>>>>>>>>>> at
>>>>>>>>>> org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:67)
>>>>>>>>>> at
>>>>>>>>>> org.apache.camel.component.timer.TimerConsumer.sendTimerExchange(TimerConsumer.java:103)
>>>>>>>>>> at
>>>>>>>>>> org.apache.camel.component.timer.TimerConsumer$1.run(TimerConsumer.java:50)
>>>>>>>>>> at java.util.TimerThread.mainLoop(Timer.java:512)
>>>>>>>>>> at java.util.TimerThread.run(Timer.java:462)
>>>>>>>>>> Caused by: org.apache.camel.InvalidPayloadException: No body
>>>>>>>>>> available
>>>>>>>>>> of
>>>>>>>>>> type: java.io.InputStream on: Message: [Body is null]. Caused by:
>>>>>>>>>> No
>>>>>>>>>> type
>>>>>>>>>> converter available to convert from type: null to the required
>>>>>>>>>> type:
>>>>>>>>>> java.io.InputStream with value null on the exchange:
>>>>>>>>>> Exchange[Message:
>>>>>>>>>> [Body
>>>>>>>>>> is null]]
>>>>>>>>>> at
>>>>>>>>>> org.apache.camel.impl.MessageSupport.getMandatoryBody(MessageSupport.java:103)
>>>>>>>>>> at
>>>>>>>>>> org.apache.camel.util.ExchangeHelper.getMandatoryInBody(ExchangeHelper.java:116)
>>>>>>>>>> at
>>>>>>>>>> org.apache.camel.component.file.FileOperations.storeFile(FileOperations.java:198)
>>>>>>>>>> ... 29 more
>>>>>>>>>> Caused by: org.apache.camel.NoTypeConversionAvailableException:
>>>>>>>>>> No
>>>>>>>>>> type
>>>>>>>>>> converter available to convert from type: null to the required
>>>>>>>>>> type:
>>>>>>>>>> java.io.InputStream with value null
>>>>>>>>>> at
>>>>>>>>>> org.apache.camel.impl.converter.DefaultTypeConverter.mandatoryConvertTo(DefaultTypeConverter.java:126)
>>>>>>>>>> at
>>>>>>>>>> org.apache.camel.impl.MessageSupport.getMandatoryBody(MessageSupport.java:101)
>>>>>>>>>> ... 31 more
>>>>>>>>>> ============
>>>>>>>>>> Now, I can remedy the situation by a try-catch block or an
>>>>>>>>>> interceptor,
>>>>>>>>>> but
>>>>>>>>>> in my opinion this should be considered a bug in camel and it
>>>>>>>>>> should
>>>>>>>>>> silently ignore if there was nothing to fetch from the ftp.
>>>>>>>>>>
>>>>>>>>>> Bests,
>>>>>>>>>> Farhad Shiraz
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> View this message in context:
>>>>>>>>>> http://old.nabble.com/camel-casts-exception-when-there-is-nothing-to-fetch-from-the-ftp-endpoint-tp26541447p26541447.html
>>>>>>>>>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Claus Ibsen
>>>>>>>>> Apache Camel Committer
>>>>>>>>>
>>>>>>>>> Author of Camel in Action: http://www.manning.com/ibsen/
>>>>>>>>> Open Source Integration: http://fusesource.com
>>>>>>>>> Blog: http://davsclaus.blogspot.com/
>>>>>>>>> Twitter: http://twitter.com/davsclaus
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> View this message in context:
>>>>>>>> http://old.nabble.com/camel-casts-exception-when-there-is-nothing-to-fetch-from-the-ftp-endpoint-tp26541447p26541650.html
>>>>>>>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Claus Ibsen
>>>>>>> Apache Camel Committer
>>>>>>>
>>>>>>> Author of Camel in Action: http://www.manning.com/ibsen/
>>>>>>> Open Source Integration: http://fusesource.com
>>>>>>> Blog: http://davsclaus.blogspot.com/
>>>>>>> Twitter: http://twitter.com/davsclaus
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> --
>>>>>> View this message in context:
>>>>>> http://old.nabble.com/camel-casts-exception-when-there-is-nothing-to-fetch-from-the-ftp-endpoint-tp26541447p26541877.html
>>>>>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Claus Ibsen
>>>>> Apache Camel Committer
>>>>>
>>>>> Author of Camel in Action: http://www.manning.com/ibsen/
>>>>> Open Source Integration: http://fusesource.com
>>>>> Blog: http://davsclaus.blogspot.com/
>>>>> Twitter: http://twitter.com/davsclaus
>>>>>
>>>>>
>>>>
>>>> --
>>>> View this message in context:
>>>> http://old.nabble.com/camel-casts-exception-when-there-is-nothing-to-fetch-from-the-ftp-endpoint-tp26541447p26542125.html
>>>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>>>
>>>>
>>>
>>>
>>>
>>> --
>>> Claus Ibsen
>>> Apache Camel Committer
>>>
>>> Author of Camel in Action: http://www.manning.com/ibsen/
>>> Open Source Integration: http://fusesource.com
>>> Blog: http://davsclaus.blogspot.com/
>>> Twitter: http://twitter.com/davsclaus
>>>
>>>
>>
>> --
>> View this message in context:
>> http://old.nabble.com/camel-casts-exception-when-there-is-nothing-to-fetch-from-the-ftp-endpoint-tp26541447p26542396.html
>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>
>>
>
>
>
> --
> Claus Ibsen
> Apache Camel Committer
>
> Author of Camel in Action: http://www.manning.com/ibsen/
> Open Source Integration: http://fusesource.com
> Blog: http://davsclaus.blogspot.com/
> Twitter: http://twitter.com/davsclaus
>
>
--
View this message in context: http://old.nabble.com/camel-casts-exception-when-there-is-nothing-to-fetch-from-the-ftp-endpoint-tp26541447p26542472.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: camel casts exception when there is nothing to fetch from the ftp
endpoint
Posted by Claus Ibsen <cl...@gmail.com>.
On Fri, Nov 27, 2009 at 2:26 PM, shirazi <me...@farhad.eu> wrote:
>
> That's probably the case. I've tried with different buffer sizes (though not
> 0), from 16 to 128 KB. I got the best performance with buffer of 64KB, on
> Linux Debian with 4 Gigs RAM, and files larger than 3 Gigs.
> That would be great, if one could specify the buffer-size for the "file"
> component.
>
You buffer size was 64 * 1024 * 1024 which is 64 K * K = 64 MB. Or is
there something that play tricks on me?
The file component already have a bufferSize option with a default of 128kb
http://camel.apache.org/file2.html
So you can use that to set what you want to use.
> -Farhad S.
>
>
> Claus Ibsen-2 wrote:
>>
>> On Fri, Nov 27, 2009 at 2:00 PM, shirazi <me...@farhad.eu> wrote:
>>>
>>> Sure. I have attached the file.
>>> Actually I use from->to components for transfering files and I guess it
>>> copies the file, rather than moving.
>>>
>>
>> Ah makes more sense for copying to leverage NIO.
>>
>> Camel does this also but I can see that it may hit that 2gb limit as
>> it doesnt transfer in a loop using a lower buffer.
>>
>> It was doing it in a one liner
>> in.transferTo(0, in.size(), out);
>>
>> I wonder if you really must do that in a loop to avoid any 2gb limit
>> or the likes?
>>
>>
>>> Thanks,
>>> Farhad S.
>>> http://old.nabble.com/file/p26542125/Nio.java Nio.java
>>>
>>>
>>> Claus Ibsen-2 wrote:
>>>>
>>>> On Fri, Nov 27, 2009 at 1:37 PM, shirazi <me...@farhad.eu> wrote:
>>>>>
>>>>> Wau... thanks for that. I don't understand how I could miss it, cause i
>>>>> searched a lot for such a feature.
>>>>>
>>>>> Another thing that I have experienced in 2.0, is that the "file"
>>>>> component
>>>>> is unable to move/copy files larger than 2Gigs, such large files are
>>>>> cut
>>>>> around 2 Gigs limit. Now, I'm not sure if it has anything to do with
>>>>> the
>>>>> OS
>>>>> or the filesystems or what. I have seen it both in Linux (debian) and
>>>>> Mac
>>>>> OS
>>>>> X. And I don't understand either, why the file component does not use
>>>>> the
>>>>> NIO api or at least we don't have an option to tell it use the NIO api
>>>>> instead.
>>>>> Anyway, for now I've solved the problem by using a smiple component
>>>>> instead
>>>>> of the "file" comp. to move/copy files using the NIO api. It is much
>>>>> faster
>>>>> and it has no problem with very large files.
>>>>>
>>>>
>>>> For move it uses java.io.File rename and are you saying its not
>>>> optimized for the underlying OS?
>>>> I would assume that it could move a 2gb file in no time if the file is
>>>> on the same share.
>>>>
>>>> Can you share your code that you are using?
>>>>
>>>>
>>>>> Bests,
>>>>> Farhad S.
>>>>>
>>>>>
>>>>> Claus Ibsen-2 wrote:
>>>>>>
>>>>>> On Fri, Nov 27, 2009 at 1:12 PM, shirazi <me...@farhad.eu> wrote:
>>>>>>>
>>>>>>> Hi,
>>>>>>> Ok, I guess I should rather provide an AggregationStrategy.
>>>>>>> The reason why I use a timer is that when I use the ftp endpoint, it
>>>>>>> starts
>>>>>>> to fetch everything that is on the ftp in the first call and that's
>>>>>>> not
>>>>>>> what
>>>>>>> i want, because I'm processing files over 3 Gigs and there are many
>>>>>>> of
>>>>>>> them.
>>>>>>> Then I figured that by using a timer, i could have ftp to download
>>>>>>> one
>>>>>>> file
>>>>>>> at a time!!
>>>>>>> What I basically need is to somehow tell the ftp, how many files I
>>>>>>> want
>>>>>>> to
>>>>>>> fetch at a time. This is probably doable now in camel-2.1 by using
>>>>>>> the
>>>>>>> "ThrottlingInflightRoutePolicy", which i havn't tried yet.
>>>>>>> Do you have any suggestions?
>>>>>>
>>>>>> Yeah the FTP component extends the File component so you got all the
>>>>>> options from the file component as well
>>>>>> http://camel.apache.org/file2.html
>>>>>>
>>>>>> And there is a maxMessagesPerPoll you can use to tell how many files
>>>>>> you want at most to poll. For example = 1 in your use.
>>>>>>
>>>>>> And that option is avail in 2.0 also.
>>>>>>
>>>>>> In fact all these are as they are batch polling consumers
>>>>>> http://camel.apache.org/batch-consumer.html
>>>>>>
>>>>>>
>>>>>>
>>>>>>>
>>>>>>> Bests,
>>>>>>> Farhad S.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Claus Ibsen-2 wrote:
>>>>>>>>
>>>>>>>> Hi
>>>>>>>>
>>>>>>>> BTW why are you using a timer -> ftp when the ftp have a build in
>>>>>>>> timer?
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Fri, Nov 27, 2009 at 12:55 PM, shirazi <me...@farhad.eu> wrote:
>>>>>>>>>
>>>>>>>>> Hi,
>>>>>>>>> Please consider the following route:
>>>>>>>>>
>>>>>>>>> from("timer://someTrigger?delay=30000&fixedRate=true&period=300000").
>>>>>>>>> pollEnrich("ftp://cmoreFtp?binary=true&localWorkDirectory=/tmp&idempotent=true&idempotentRepository=#caStreams",
>>>>>>>>> 10000).
>>>>>>>>> to("file:///tmp/vw/cmore?tempPrefix=.inprogress");
>>>>>>>>>
>>>>>>>>> The pollEnrich cast a NullPointerException in camel-2.0, if there
>>>>>>>>> is
>>>>>>>>> nothing
>>>>>>>>> to fetch from the ftp. I got rid of the exception by a
>>>>>>>>> "try-catch(NullpointerExcpetion.class).stop()" around the route.
>>>>>>>>> Now
>>>>>>>>> in
>>>>>>>>> camel-2.1-SNAPSHOT, I get the following exception on the "to"
>>>>>>>>> endpoint:
>>>>>>>>> =================
>>>>>>>>> 12:37:06,899 INFO [Tracer] >>> -->
>>>>>>>>> file:///tmp/vw/cmore?tempPrefix=.inprogress_, Pattern:InOnly,
>>>>>>>>> Headers:{firedTime=Fri Nov 27 12:36:56 CET 2009},
>>>>>>>>> Exception:org.apache.camel.component.file.GenericFileOperationFailedException:
>>>>>>>>> Cannot store file:
>>>>>>>>> /tmp/vw/cmore/.inprogress_02216f38-8ffc-4aec-b638-27229d857075
>>>>>>>>> 12:37:06,899 INFO [Tracer] >>> -->
>>>>>>>>> file:///tmp/vw/cmore?tempPrefix=.inprogress_, Pattern:InOnly,
>>>>>>>>> Headers:{firedTime=Fri Nov 27 12:36:56 CET 2009},
>>>>>>>>> Exception:org.apache.camel.component.file.GenericFileOperationFailedException:
>>>>>>>>> Cannot store file:
>>>>>>>>> /tmp/vw/cmore/.inprogress_02216f38-8ffc-4aec-b638-27229d857075
>>>>>>>>> 12:37:06,899 ERROR [TimerConsumer] Cannot store file:
>>>>>>>>> /tmp/vw/cmore/.inprogress_02216f38-8ffc-4aec-b638-27229d857075
>>>>>>>>> org.apache.camel.component.file.GenericFileOperationFailedException:
>>>>>>>>> Cannot
>>>>>>>>> store file:
>>>>>>>>> /tmp/vw/cmore/.inprogress_02216f38-8ffc-4aec-b638-27229d857075
>>>>>>>>> at
>>>>>>>>> org.apache.camel.component.file.FileOperations.storeFile(FileOperations.java:204)
>>>>>>>>> at
>>>>>>>>> org.apache.camel.component.file.GenericFileProducer.writeFile(GenericFileProducer.java:184)
>>>>>>>>> at
>>>>>>>>> org.apache.camel.component.file.GenericFileProducer.processExchange(GenericFileProducer.java:126)
>>>>>>>>> at
>>>>>>>>> org.apache.camel.component.file.GenericFileProducer.process(GenericFileProducer.java:57)
>>>>>>>>> at
>>>>>>>>> org.apache.camel.processor.SendProcessor$1.doInProducer(SendProcessor.java:97)
>>>>>>>>> at
>>>>>>>>> org.apache.camel.processor.SendProcessor$1.doInProducer(SendProcessor.java:95)
>>>>>>>>> at
>>>>>>>>> org.apache.camel.impl.ProducerCache.doInProducer(ProducerCache.java:146)
>>>>>>>>> at
>>>>>>>>> org.apache.camel.processor.SendProcessor.doProcess(SendProcessor.java:94)
>>>>>>>>> at
>>>>>>>>> org.apache.camel.processor.SendProcessor.process(SendProcessor.java:82)
>>>>>>>>> at
>>>>>>>>> org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:67)
>>>>>>>>> at
>>>>>>>>> org.apache.camel.processor.DelegateProcessor.processNext(DelegateProcessor.java:53)
>>>>>>>>> at
>>>>>>>>> org.apache.camel.processor.DelegateProcessor.proceed(DelegateProcessor.java:82)
>>>>>>>>> at
>>>>>>>>> org.apache.camel.processor.interceptor.TraceInterceptor.process(TraceInterceptor.java:162)
>>>>>>>>> at
>>>>>>>>> org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:67)
>>>>>>>>> at
>>>>>>>>> org.apache.camel.processor.DefaultChannel.process(DefaultChannel.java:206)
>>>>>>>>> at
>>>>>>>>> org.apache.camel.processor.Pipeline.process(Pipeline.java:74)
>>>>>>>>> at
>>>>>>>>> org.apache.camel.processor.TryProcessor.process(TryProcessor.java:63)
>>>>>>>>> at
>>>>>>>>> org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:67)
>>>>>>>>> at
>>>>>>>>> org.apache.camel.processor.DelegateProcessor.processNext(DelegateProcessor.java:53)
>>>>>>>>> at
>>>>>>>>> org.apache.camel.processor.DelegateProcessor.proceed(DelegateProcessor.java:82)
>>>>>>>>> at
>>>>>>>>> org.apache.camel.processor.interceptor.TraceInterceptor.process(TraceInterceptor.java:162)
>>>>>>>>> at
>>>>>>>>> org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:67)
>>>>>>>>> at
>>>>>>>>> org.apache.camel.processor.DefaultChannel.process(DefaultChannel.java:206)
>>>>>>>>> at
>>>>>>>>> org.apache.camel.processor.UnitOfWorkProcessor.processNext(UnitOfWorkProcessor.java:54)
>>>>>>>>> at
>>>>>>>>> org.apache.camel.processor.DelegateProcessor.process(DelegateProcessor.java:48)
>>>>>>>>> at
>>>>>>>>> org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:67)
>>>>>>>>> at
>>>>>>>>> org.apache.camel.component.timer.TimerConsumer.sendTimerExchange(TimerConsumer.java:103)
>>>>>>>>> at
>>>>>>>>> org.apache.camel.component.timer.TimerConsumer$1.run(TimerConsumer.java:50)
>>>>>>>>> at java.util.TimerThread.mainLoop(Timer.java:512)
>>>>>>>>> at java.util.TimerThread.run(Timer.java:462)
>>>>>>>>> Caused by: org.apache.camel.InvalidPayloadException: No body
>>>>>>>>> available
>>>>>>>>> of
>>>>>>>>> type: java.io.InputStream on: Message: [Body is null]. Caused by:
>>>>>>>>> No
>>>>>>>>> type
>>>>>>>>> converter available to convert from type: null to the required
>>>>>>>>> type:
>>>>>>>>> java.io.InputStream with value null on the exchange:
>>>>>>>>> Exchange[Message:
>>>>>>>>> [Body
>>>>>>>>> is null]]
>>>>>>>>> at
>>>>>>>>> org.apache.camel.impl.MessageSupport.getMandatoryBody(MessageSupport.java:103)
>>>>>>>>> at
>>>>>>>>> org.apache.camel.util.ExchangeHelper.getMandatoryInBody(ExchangeHelper.java:116)
>>>>>>>>> at
>>>>>>>>> org.apache.camel.component.file.FileOperations.storeFile(FileOperations.java:198)
>>>>>>>>> ... 29 more
>>>>>>>>> Caused by: org.apache.camel.NoTypeConversionAvailableException: No
>>>>>>>>> type
>>>>>>>>> converter available to convert from type: null to the required
>>>>>>>>> type:
>>>>>>>>> java.io.InputStream with value null
>>>>>>>>> at
>>>>>>>>> org.apache.camel.impl.converter.DefaultTypeConverter.mandatoryConvertTo(DefaultTypeConverter.java:126)
>>>>>>>>> at
>>>>>>>>> org.apache.camel.impl.MessageSupport.getMandatoryBody(MessageSupport.java:101)
>>>>>>>>> ... 31 more
>>>>>>>>> ============
>>>>>>>>> Now, I can remedy the situation by a try-catch block or an
>>>>>>>>> interceptor,
>>>>>>>>> but
>>>>>>>>> in my opinion this should be considered a bug in camel and it
>>>>>>>>> should
>>>>>>>>> silently ignore if there was nothing to fetch from the ftp.
>>>>>>>>>
>>>>>>>>> Bests,
>>>>>>>>> Farhad Shiraz
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> View this message in context:
>>>>>>>>> http://old.nabble.com/camel-casts-exception-when-there-is-nothing-to-fetch-from-the-ftp-endpoint-tp26541447p26541447.html
>>>>>>>>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Claus Ibsen
>>>>>>>> Apache Camel Committer
>>>>>>>>
>>>>>>>> Author of Camel in Action: http://www.manning.com/ibsen/
>>>>>>>> Open Source Integration: http://fusesource.com
>>>>>>>> Blog: http://davsclaus.blogspot.com/
>>>>>>>> Twitter: http://twitter.com/davsclaus
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> View this message in context:
>>>>>>> http://old.nabble.com/camel-casts-exception-when-there-is-nothing-to-fetch-from-the-ftp-endpoint-tp26541447p26541650.html
>>>>>>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Claus Ibsen
>>>>>> Apache Camel Committer
>>>>>>
>>>>>> Author of Camel in Action: http://www.manning.com/ibsen/
>>>>>> Open Source Integration: http://fusesource.com
>>>>>> Blog: http://davsclaus.blogspot.com/
>>>>>> Twitter: http://twitter.com/davsclaus
>>>>>>
>>>>>>
>>>>>
>>>>> --
>>>>> View this message in context:
>>>>> http://old.nabble.com/camel-casts-exception-when-there-is-nothing-to-fetch-from-the-ftp-endpoint-tp26541447p26541877.html
>>>>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Claus Ibsen
>>>> Apache Camel Committer
>>>>
>>>> Author of Camel in Action: http://www.manning.com/ibsen/
>>>> Open Source Integration: http://fusesource.com
>>>> Blog: http://davsclaus.blogspot.com/
>>>> Twitter: http://twitter.com/davsclaus
>>>>
>>>>
>>>
>>> --
>>> View this message in context:
>>> http://old.nabble.com/camel-casts-exception-when-there-is-nothing-to-fetch-from-the-ftp-endpoint-tp26541447p26542125.html
>>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>>
>>>
>>
>>
>>
>> --
>> Claus Ibsen
>> Apache Camel Committer
>>
>> Author of Camel in Action: http://www.manning.com/ibsen/
>> Open Source Integration: http://fusesource.com
>> Blog: http://davsclaus.blogspot.com/
>> Twitter: http://twitter.com/davsclaus
>>
>>
>
> --
> View this message in context: http://old.nabble.com/camel-casts-exception-when-there-is-nothing-to-fetch-from-the-ftp-endpoint-tp26541447p26542396.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>
>
--
Claus Ibsen
Apache Camel Committer
Author of Camel in Action: http://www.manning.com/ibsen/
Open Source Integration: http://fusesource.com
Blog: http://davsclaus.blogspot.com/
Twitter: http://twitter.com/davsclaus
Re: camel casts exception when there is nothing to fetch from the
ftp endpoint
Posted by shirazi <me...@farhad.eu>.
That's probably the case. I've tried with different buffer sizes (though not
0), from 16 to 128 KB. I got the best performance with buffer of 64KB, on
Linux Debian with 4 Gigs RAM, and files larger than 3 Gigs.
That would be great, if one could specify the buffer-size for the "file"
component.
-Farhad S.
Claus Ibsen-2 wrote:
>
> On Fri, Nov 27, 2009 at 2:00 PM, shirazi <me...@farhad.eu> wrote:
>>
>> Sure. I have attached the file.
>> Actually I use from->to components for transfering files and I guess it
>> copies the file, rather than moving.
>>
>
> Ah makes more sense for copying to leverage NIO.
>
> Camel does this also but I can see that it may hit that 2gb limit as
> it doesnt transfer in a loop using a lower buffer.
>
> It was doing it in a one liner
> in.transferTo(0, in.size(), out);
>
> I wonder if you really must do that in a loop to avoid any 2gb limit
> or the likes?
>
>
>> Thanks,
>> Farhad S.
>> http://old.nabble.com/file/p26542125/Nio.java Nio.java
>>
>>
>> Claus Ibsen-2 wrote:
>>>
>>> On Fri, Nov 27, 2009 at 1:37 PM, shirazi <me...@farhad.eu> wrote:
>>>>
>>>> Wau... thanks for that. I don't understand how I could miss it, cause i
>>>> searched a lot for such a feature.
>>>>
>>>> Another thing that I have experienced in 2.0, is that the "file"
>>>> component
>>>> is unable to move/copy files larger than 2Gigs, such large files are
>>>> cut
>>>> around 2 Gigs limit. Now, I'm not sure if it has anything to do with
>>>> the
>>>> OS
>>>> or the filesystems or what. I have seen it both in Linux (debian) and
>>>> Mac
>>>> OS
>>>> X. And I don't understand either, why the file component does not use
>>>> the
>>>> NIO api or at least we don't have an option to tell it use the NIO api
>>>> instead.
>>>> Anyway, for now I've solved the problem by using a smiple component
>>>> instead
>>>> of the "file" comp. to move/copy files using the NIO api. It is much
>>>> faster
>>>> and it has no problem with very large files.
>>>>
>>>
>>> For move it uses java.io.File rename and are you saying its not
>>> optimized for the underlying OS?
>>> I would assume that it could move a 2gb file in no time if the file is
>>> on the same share.
>>>
>>> Can you share your code that you are using?
>>>
>>>
>>>> Bests,
>>>> Farhad S.
>>>>
>>>>
>>>> Claus Ibsen-2 wrote:
>>>>>
>>>>> On Fri, Nov 27, 2009 at 1:12 PM, shirazi <me...@farhad.eu> wrote:
>>>>>>
>>>>>> Hi,
>>>>>> Ok, I guess I should rather provide an AggregationStrategy.
>>>>>> The reason why I use a timer is that when I use the ftp endpoint, it
>>>>>> starts
>>>>>> to fetch everything that is on the ftp in the first call and that's
>>>>>> not
>>>>>> what
>>>>>> i want, because I'm processing files over 3 Gigs and there are many
>>>>>> of
>>>>>> them.
>>>>>> Then I figured that by using a timer, i could have ftp to download
>>>>>> one
>>>>>> file
>>>>>> at a time!!
>>>>>> What I basically need is to somehow tell the ftp, how many files I
>>>>>> want
>>>>>> to
>>>>>> fetch at a time. This is probably doable now in camel-2.1 by using
>>>>>> the
>>>>>> "ThrottlingInflightRoutePolicy", which i havn't tried yet.
>>>>>> Do you have any suggestions?
>>>>>
>>>>> Yeah the FTP component extends the File component so you got all the
>>>>> options from the file component as well
>>>>> http://camel.apache.org/file2.html
>>>>>
>>>>> And there is a maxMessagesPerPoll you can use to tell how many files
>>>>> you want at most to poll. For example = 1 in your use.
>>>>>
>>>>> And that option is avail in 2.0 also.
>>>>>
>>>>> In fact all these are as they are batch polling consumers
>>>>> http://camel.apache.org/batch-consumer.html
>>>>>
>>>>>
>>>>>
>>>>>>
>>>>>> Bests,
>>>>>> Farhad S.
>>>>>>
>>>>>>
>>>>>>
>>>>>> Claus Ibsen-2 wrote:
>>>>>>>
>>>>>>> Hi
>>>>>>>
>>>>>>> BTW why are you using a timer -> ftp when the ftp have a build in
>>>>>>> timer?
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Fri, Nov 27, 2009 at 12:55 PM, shirazi <me...@farhad.eu> wrote:
>>>>>>>>
>>>>>>>> Hi,
>>>>>>>> Please consider the following route:
>>>>>>>>
>>>>>>>> from("timer://someTrigger?delay=30000&fixedRate=true&period=300000").
>>>>>>>> pollEnrich("ftp://cmoreFtp?binary=true&localWorkDirectory=/tmp&idempotent=true&idempotentRepository=#caStreams",
>>>>>>>> 10000).
>>>>>>>> to("file:///tmp/vw/cmore?tempPrefix=.inprogress");
>>>>>>>>
>>>>>>>> The pollEnrich cast a NullPointerException in camel-2.0, if there
>>>>>>>> is
>>>>>>>> nothing
>>>>>>>> to fetch from the ftp. I got rid of the exception by a
>>>>>>>> "try-catch(NullpointerExcpetion.class).stop()" around the route.
>>>>>>>> Now
>>>>>>>> in
>>>>>>>> camel-2.1-SNAPSHOT, I get the following exception on the "to"
>>>>>>>> endpoint:
>>>>>>>> =================
>>>>>>>> 12:37:06,899 INFO [Tracer] >>> -->
>>>>>>>> file:///tmp/vw/cmore?tempPrefix=.inprogress_, Pattern:InOnly,
>>>>>>>> Headers:{firedTime=Fri Nov 27 12:36:56 CET 2009},
>>>>>>>> Exception:org.apache.camel.component.file.GenericFileOperationFailedException:
>>>>>>>> Cannot store file:
>>>>>>>> /tmp/vw/cmore/.inprogress_02216f38-8ffc-4aec-b638-27229d857075
>>>>>>>> 12:37:06,899 INFO [Tracer] >>> -->
>>>>>>>> file:///tmp/vw/cmore?tempPrefix=.inprogress_, Pattern:InOnly,
>>>>>>>> Headers:{firedTime=Fri Nov 27 12:36:56 CET 2009},
>>>>>>>> Exception:org.apache.camel.component.file.GenericFileOperationFailedException:
>>>>>>>> Cannot store file:
>>>>>>>> /tmp/vw/cmore/.inprogress_02216f38-8ffc-4aec-b638-27229d857075
>>>>>>>> 12:37:06,899 ERROR [TimerConsumer] Cannot store file:
>>>>>>>> /tmp/vw/cmore/.inprogress_02216f38-8ffc-4aec-b638-27229d857075
>>>>>>>> org.apache.camel.component.file.GenericFileOperationFailedException:
>>>>>>>> Cannot
>>>>>>>> store file:
>>>>>>>> /tmp/vw/cmore/.inprogress_02216f38-8ffc-4aec-b638-27229d857075
>>>>>>>> at
>>>>>>>> org.apache.camel.component.file.FileOperations.storeFile(FileOperations.java:204)
>>>>>>>> at
>>>>>>>> org.apache.camel.component.file.GenericFileProducer.writeFile(GenericFileProducer.java:184)
>>>>>>>> at
>>>>>>>> org.apache.camel.component.file.GenericFileProducer.processExchange(GenericFileProducer.java:126)
>>>>>>>> at
>>>>>>>> org.apache.camel.component.file.GenericFileProducer.process(GenericFileProducer.java:57)
>>>>>>>> at
>>>>>>>> org.apache.camel.processor.SendProcessor$1.doInProducer(SendProcessor.java:97)
>>>>>>>> at
>>>>>>>> org.apache.camel.processor.SendProcessor$1.doInProducer(SendProcessor.java:95)
>>>>>>>> at
>>>>>>>> org.apache.camel.impl.ProducerCache.doInProducer(ProducerCache.java:146)
>>>>>>>> at
>>>>>>>> org.apache.camel.processor.SendProcessor.doProcess(SendProcessor.java:94)
>>>>>>>> at
>>>>>>>> org.apache.camel.processor.SendProcessor.process(SendProcessor.java:82)
>>>>>>>> at
>>>>>>>> org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:67)
>>>>>>>> at
>>>>>>>> org.apache.camel.processor.DelegateProcessor.processNext(DelegateProcessor.java:53)
>>>>>>>> at
>>>>>>>> org.apache.camel.processor.DelegateProcessor.proceed(DelegateProcessor.java:82)
>>>>>>>> at
>>>>>>>> org.apache.camel.processor.interceptor.TraceInterceptor.process(TraceInterceptor.java:162)
>>>>>>>> at
>>>>>>>> org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:67)
>>>>>>>> at
>>>>>>>> org.apache.camel.processor.DefaultChannel.process(DefaultChannel.java:206)
>>>>>>>> at
>>>>>>>> org.apache.camel.processor.Pipeline.process(Pipeline.java:74)
>>>>>>>> at
>>>>>>>> org.apache.camel.processor.TryProcessor.process(TryProcessor.java:63)
>>>>>>>> at
>>>>>>>> org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:67)
>>>>>>>> at
>>>>>>>> org.apache.camel.processor.DelegateProcessor.processNext(DelegateProcessor.java:53)
>>>>>>>> at
>>>>>>>> org.apache.camel.processor.DelegateProcessor.proceed(DelegateProcessor.java:82)
>>>>>>>> at
>>>>>>>> org.apache.camel.processor.interceptor.TraceInterceptor.process(TraceInterceptor.java:162)
>>>>>>>> at
>>>>>>>> org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:67)
>>>>>>>> at
>>>>>>>> org.apache.camel.processor.DefaultChannel.process(DefaultChannel.java:206)
>>>>>>>> at
>>>>>>>> org.apache.camel.processor.UnitOfWorkProcessor.processNext(UnitOfWorkProcessor.java:54)
>>>>>>>> at
>>>>>>>> org.apache.camel.processor.DelegateProcessor.process(DelegateProcessor.java:48)
>>>>>>>> at
>>>>>>>> org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:67)
>>>>>>>> at
>>>>>>>> org.apache.camel.component.timer.TimerConsumer.sendTimerExchange(TimerConsumer.java:103)
>>>>>>>> at
>>>>>>>> org.apache.camel.component.timer.TimerConsumer$1.run(TimerConsumer.java:50)
>>>>>>>> at java.util.TimerThread.mainLoop(Timer.java:512)
>>>>>>>> at java.util.TimerThread.run(Timer.java:462)
>>>>>>>> Caused by: org.apache.camel.InvalidPayloadException: No body
>>>>>>>> available
>>>>>>>> of
>>>>>>>> type: java.io.InputStream on: Message: [Body is null]. Caused by:
>>>>>>>> No
>>>>>>>> type
>>>>>>>> converter available to convert from type: null to the required
>>>>>>>> type:
>>>>>>>> java.io.InputStream with value null on the exchange:
>>>>>>>> Exchange[Message:
>>>>>>>> [Body
>>>>>>>> is null]]
>>>>>>>> at
>>>>>>>> org.apache.camel.impl.MessageSupport.getMandatoryBody(MessageSupport.java:103)
>>>>>>>> at
>>>>>>>> org.apache.camel.util.ExchangeHelper.getMandatoryInBody(ExchangeHelper.java:116)
>>>>>>>> at
>>>>>>>> org.apache.camel.component.file.FileOperations.storeFile(FileOperations.java:198)
>>>>>>>> ... 29 more
>>>>>>>> Caused by: org.apache.camel.NoTypeConversionAvailableException: No
>>>>>>>> type
>>>>>>>> converter available to convert from type: null to the required
>>>>>>>> type:
>>>>>>>> java.io.InputStream with value null
>>>>>>>> at
>>>>>>>> org.apache.camel.impl.converter.DefaultTypeConverter.mandatoryConvertTo(DefaultTypeConverter.java:126)
>>>>>>>> at
>>>>>>>> org.apache.camel.impl.MessageSupport.getMandatoryBody(MessageSupport.java:101)
>>>>>>>> ... 31 more
>>>>>>>> ============
>>>>>>>> Now, I can remedy the situation by a try-catch block or an
>>>>>>>> interceptor,
>>>>>>>> but
>>>>>>>> in my opinion this should be considered a bug in camel and it
>>>>>>>> should
>>>>>>>> silently ignore if there was nothing to fetch from the ftp.
>>>>>>>>
>>>>>>>> Bests,
>>>>>>>> Farhad Shiraz
>>>>>>>>
>>>>>>>> --
>>>>>>>> View this message in context:
>>>>>>>> http://old.nabble.com/camel-casts-exception-when-there-is-nothing-to-fetch-from-the-ftp-endpoint-tp26541447p26541447.html
>>>>>>>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Claus Ibsen
>>>>>>> Apache Camel Committer
>>>>>>>
>>>>>>> Author of Camel in Action: http://www.manning.com/ibsen/
>>>>>>> Open Source Integration: http://fusesource.com
>>>>>>> Blog: http://davsclaus.blogspot.com/
>>>>>>> Twitter: http://twitter.com/davsclaus
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> --
>>>>>> View this message in context:
>>>>>> http://old.nabble.com/camel-casts-exception-when-there-is-nothing-to-fetch-from-the-ftp-endpoint-tp26541447p26541650.html
>>>>>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Claus Ibsen
>>>>> Apache Camel Committer
>>>>>
>>>>> Author of Camel in Action: http://www.manning.com/ibsen/
>>>>> Open Source Integration: http://fusesource.com
>>>>> Blog: http://davsclaus.blogspot.com/
>>>>> Twitter: http://twitter.com/davsclaus
>>>>>
>>>>>
>>>>
>>>> --
>>>> View this message in context:
>>>> http://old.nabble.com/camel-casts-exception-when-there-is-nothing-to-fetch-from-the-ftp-endpoint-tp26541447p26541877.html
>>>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>>>
>>>>
>>>
>>>
>>>
>>> --
>>> Claus Ibsen
>>> Apache Camel Committer
>>>
>>> Author of Camel in Action: http://www.manning.com/ibsen/
>>> Open Source Integration: http://fusesource.com
>>> Blog: http://davsclaus.blogspot.com/
>>> Twitter: http://twitter.com/davsclaus
>>>
>>>
>>
>> --
>> View this message in context:
>> http://old.nabble.com/camel-casts-exception-when-there-is-nothing-to-fetch-from-the-ftp-endpoint-tp26541447p26542125.html
>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>
>>
>
>
>
> --
> Claus Ibsen
> Apache Camel Committer
>
> Author of Camel in Action: http://www.manning.com/ibsen/
> Open Source Integration: http://fusesource.com
> Blog: http://davsclaus.blogspot.com/
> Twitter: http://twitter.com/davsclaus
>
>
--
View this message in context: http://old.nabble.com/camel-casts-exception-when-there-is-nothing-to-fetch-from-the-ftp-endpoint-tp26541447p26542396.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: camel casts exception when there is nothing to fetch from the ftp
endpoint
Posted by Claus Ibsen <cl...@gmail.com>.
Hi
I have created a ticket about the NIO thingy
https://issues.apache.org/activemq/browse/CAMEL-2234
On Fri, Nov 27, 2009 at 2:14 PM, Claus Ibsen <cl...@gmail.com> wrote:
> On Fri, Nov 27, 2009 at 2:00 PM, shirazi <me...@farhad.eu> wrote:
>>
>> Sure. I have attached the file.
>> Actually I use from->to components for transfering files and I guess it
>> copies the file, rather than moving.
>>
>
> Ah makes more sense for copying to leverage NIO.
>
> Camel does this also but I can see that it may hit that 2gb limit as
> it doesnt transfer in a loop using a lower buffer.
>
> It was doing it in a one liner
> in.transferTo(0, in.size(), out);
>
> I wonder if you really must do that in a loop to avoid any 2gb limit
> or the likes?
>
>
>> Thanks,
>> Farhad S.
>> http://old.nabble.com/file/p26542125/Nio.java Nio.java
>>
>>
>> Claus Ibsen-2 wrote:
>>>
>>> On Fri, Nov 27, 2009 at 1:37 PM, shirazi <me...@farhad.eu> wrote:
>>>>
>>>> Wau... thanks for that. I don't understand how I could miss it, cause i
>>>> searched a lot for such a feature.
>>>>
>>>> Another thing that I have experienced in 2.0, is that the "file"
>>>> component
>>>> is unable to move/copy files larger than 2Gigs, such large files are cut
>>>> around 2 Gigs limit. Now, I'm not sure if it has anything to do with the
>>>> OS
>>>> or the filesystems or what. I have seen it both in Linux (debian) and Mac
>>>> OS
>>>> X. And I don't understand either, why the file component does not use the
>>>> NIO api or at least we don't have an option to tell it use the NIO api
>>>> instead.
>>>> Anyway, for now I've solved the problem by using a smiple component
>>>> instead
>>>> of the "file" comp. to move/copy files using the NIO api. It is much
>>>> faster
>>>> and it has no problem with very large files.
>>>>
>>>
>>> For move it uses java.io.File rename and are you saying its not
>>> optimized for the underlying OS?
>>> I would assume that it could move a 2gb file in no time if the file is
>>> on the same share.
>>>
>>> Can you share your code that you are using?
>>>
>>>
>>>> Bests,
>>>> Farhad S.
>>>>
>>>>
>>>> Claus Ibsen-2 wrote:
>>>>>
>>>>> On Fri, Nov 27, 2009 at 1:12 PM, shirazi <me...@farhad.eu> wrote:
>>>>>>
>>>>>> Hi,
>>>>>> Ok, I guess I should rather provide an AggregationStrategy.
>>>>>> The reason why I use a timer is that when I use the ftp endpoint, it
>>>>>> starts
>>>>>> to fetch everything that is on the ftp in the first call and that's not
>>>>>> what
>>>>>> i want, because I'm processing files over 3 Gigs and there are many of
>>>>>> them.
>>>>>> Then I figured that by using a timer, i could have ftp to download one
>>>>>> file
>>>>>> at a time!!
>>>>>> What I basically need is to somehow tell the ftp, how many files I want
>>>>>> to
>>>>>> fetch at a time. This is probably doable now in camel-2.1 by using the
>>>>>> "ThrottlingInflightRoutePolicy", which i havn't tried yet.
>>>>>> Do you have any suggestions?
>>>>>
>>>>> Yeah the FTP component extends the File component so you got all the
>>>>> options from the file component as well
>>>>> http://camel.apache.org/file2.html
>>>>>
>>>>> And there is a maxMessagesPerPoll you can use to tell how many files
>>>>> you want at most to poll. For example = 1 in your use.
>>>>>
>>>>> And that option is avail in 2.0 also.
>>>>>
>>>>> In fact all these are as they are batch polling consumers
>>>>> http://camel.apache.org/batch-consumer.html
>>>>>
>>>>>
>>>>>
>>>>>>
>>>>>> Bests,
>>>>>> Farhad S.
>>>>>>
>>>>>>
>>>>>>
>>>>>> Claus Ibsen-2 wrote:
>>>>>>>
>>>>>>> Hi
>>>>>>>
>>>>>>> BTW why are you using a timer -> ftp when the ftp have a build in
>>>>>>> timer?
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Fri, Nov 27, 2009 at 12:55 PM, shirazi <me...@farhad.eu> wrote:
>>>>>>>>
>>>>>>>> Hi,
>>>>>>>> Please consider the following route:
>>>>>>>>
>>>>>>>> from("timer://someTrigger?delay=30000&fixedRate=true&period=300000").
>>>>>>>> pollEnrich("ftp://cmoreFtp?binary=true&localWorkDirectory=/tmp&idempotent=true&idempotentRepository=#caStreams",
>>>>>>>> 10000).
>>>>>>>> to("file:///tmp/vw/cmore?tempPrefix=.inprogress");
>>>>>>>>
>>>>>>>> The pollEnrich cast a NullPointerException in camel-2.0, if there is
>>>>>>>> nothing
>>>>>>>> to fetch from the ftp. I got rid of the exception by a
>>>>>>>> "try-catch(NullpointerExcpetion.class).stop()" around the route. Now
>>>>>>>> in
>>>>>>>> camel-2.1-SNAPSHOT, I get the following exception on the "to"
>>>>>>>> endpoint:
>>>>>>>> =================
>>>>>>>> 12:37:06,899 INFO [Tracer] >>> -->
>>>>>>>> file:///tmp/vw/cmore?tempPrefix=.inprogress_, Pattern:InOnly,
>>>>>>>> Headers:{firedTime=Fri Nov 27 12:36:56 CET 2009},
>>>>>>>> Exception:org.apache.camel.component.file.GenericFileOperationFailedException:
>>>>>>>> Cannot store file:
>>>>>>>> /tmp/vw/cmore/.inprogress_02216f38-8ffc-4aec-b638-27229d857075
>>>>>>>> 12:37:06,899 INFO [Tracer] >>> -->
>>>>>>>> file:///tmp/vw/cmore?tempPrefix=.inprogress_, Pattern:InOnly,
>>>>>>>> Headers:{firedTime=Fri Nov 27 12:36:56 CET 2009},
>>>>>>>> Exception:org.apache.camel.component.file.GenericFileOperationFailedException:
>>>>>>>> Cannot store file:
>>>>>>>> /tmp/vw/cmore/.inprogress_02216f38-8ffc-4aec-b638-27229d857075
>>>>>>>> 12:37:06,899 ERROR [TimerConsumer] Cannot store file:
>>>>>>>> /tmp/vw/cmore/.inprogress_02216f38-8ffc-4aec-b638-27229d857075
>>>>>>>> org.apache.camel.component.file.GenericFileOperationFailedException:
>>>>>>>> Cannot
>>>>>>>> store file:
>>>>>>>> /tmp/vw/cmore/.inprogress_02216f38-8ffc-4aec-b638-27229d857075
>>>>>>>> at
>>>>>>>> org.apache.camel.component.file.FileOperations.storeFile(FileOperations.java:204)
>>>>>>>> at
>>>>>>>> org.apache.camel.component.file.GenericFileProducer.writeFile(GenericFileProducer.java:184)
>>>>>>>> at
>>>>>>>> org.apache.camel.component.file.GenericFileProducer.processExchange(GenericFileProducer.java:126)
>>>>>>>> at
>>>>>>>> org.apache.camel.component.file.GenericFileProducer.process(GenericFileProducer.java:57)
>>>>>>>> at
>>>>>>>> org.apache.camel.processor.SendProcessor$1.doInProducer(SendProcessor.java:97)
>>>>>>>> at
>>>>>>>> org.apache.camel.processor.SendProcessor$1.doInProducer(SendProcessor.java:95)
>>>>>>>> at
>>>>>>>> org.apache.camel.impl.ProducerCache.doInProducer(ProducerCache.java:146)
>>>>>>>> at
>>>>>>>> org.apache.camel.processor.SendProcessor.doProcess(SendProcessor.java:94)
>>>>>>>> at
>>>>>>>> org.apache.camel.processor.SendProcessor.process(SendProcessor.java:82)
>>>>>>>> at
>>>>>>>> org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:67)
>>>>>>>> at
>>>>>>>> org.apache.camel.processor.DelegateProcessor.processNext(DelegateProcessor.java:53)
>>>>>>>> at
>>>>>>>> org.apache.camel.processor.DelegateProcessor.proceed(DelegateProcessor.java:82)
>>>>>>>> at
>>>>>>>> org.apache.camel.processor.interceptor.TraceInterceptor.process(TraceInterceptor.java:162)
>>>>>>>> at
>>>>>>>> org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:67)
>>>>>>>> at
>>>>>>>> org.apache.camel.processor.DefaultChannel.process(DefaultChannel.java:206)
>>>>>>>> at
>>>>>>>> org.apache.camel.processor.Pipeline.process(Pipeline.java:74)
>>>>>>>> at
>>>>>>>> org.apache.camel.processor.TryProcessor.process(TryProcessor.java:63)
>>>>>>>> at
>>>>>>>> org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:67)
>>>>>>>> at
>>>>>>>> org.apache.camel.processor.DelegateProcessor.processNext(DelegateProcessor.java:53)
>>>>>>>> at
>>>>>>>> org.apache.camel.processor.DelegateProcessor.proceed(DelegateProcessor.java:82)
>>>>>>>> at
>>>>>>>> org.apache.camel.processor.interceptor.TraceInterceptor.process(TraceInterceptor.java:162)
>>>>>>>> at
>>>>>>>> org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:67)
>>>>>>>> at
>>>>>>>> org.apache.camel.processor.DefaultChannel.process(DefaultChannel.java:206)
>>>>>>>> at
>>>>>>>> org.apache.camel.processor.UnitOfWorkProcessor.processNext(UnitOfWorkProcessor.java:54)
>>>>>>>> at
>>>>>>>> org.apache.camel.processor.DelegateProcessor.process(DelegateProcessor.java:48)
>>>>>>>> at
>>>>>>>> org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:67)
>>>>>>>> at
>>>>>>>> org.apache.camel.component.timer.TimerConsumer.sendTimerExchange(TimerConsumer.java:103)
>>>>>>>> at
>>>>>>>> org.apache.camel.component.timer.TimerConsumer$1.run(TimerConsumer.java:50)
>>>>>>>> at java.util.TimerThread.mainLoop(Timer.java:512)
>>>>>>>> at java.util.TimerThread.run(Timer.java:462)
>>>>>>>> Caused by: org.apache.camel.InvalidPayloadException: No body
>>>>>>>> available
>>>>>>>> of
>>>>>>>> type: java.io.InputStream on: Message: [Body is null]. Caused by: No
>>>>>>>> type
>>>>>>>> converter available to convert from type: null to the required type:
>>>>>>>> java.io.InputStream with value null on the exchange:
>>>>>>>> Exchange[Message:
>>>>>>>> [Body
>>>>>>>> is null]]
>>>>>>>> at
>>>>>>>> org.apache.camel.impl.MessageSupport.getMandatoryBody(MessageSupport.java:103)
>>>>>>>> at
>>>>>>>> org.apache.camel.util.ExchangeHelper.getMandatoryInBody(ExchangeHelper.java:116)
>>>>>>>> at
>>>>>>>> org.apache.camel.component.file.FileOperations.storeFile(FileOperations.java:198)
>>>>>>>> ... 29 more
>>>>>>>> Caused by: org.apache.camel.NoTypeConversionAvailableException: No
>>>>>>>> type
>>>>>>>> converter available to convert from type: null to the required type:
>>>>>>>> java.io.InputStream with value null
>>>>>>>> at
>>>>>>>> org.apache.camel.impl.converter.DefaultTypeConverter.mandatoryConvertTo(DefaultTypeConverter.java:126)
>>>>>>>> at
>>>>>>>> org.apache.camel.impl.MessageSupport.getMandatoryBody(MessageSupport.java:101)
>>>>>>>> ... 31 more
>>>>>>>> ============
>>>>>>>> Now, I can remedy the situation by a try-catch block or an
>>>>>>>> interceptor,
>>>>>>>> but
>>>>>>>> in my opinion this should be considered a bug in camel and it should
>>>>>>>> silently ignore if there was nothing to fetch from the ftp.
>>>>>>>>
>>>>>>>> Bests,
>>>>>>>> Farhad Shiraz
>>>>>>>>
>>>>>>>> --
>>>>>>>> View this message in context:
>>>>>>>> http://old.nabble.com/camel-casts-exception-when-there-is-nothing-to-fetch-from-the-ftp-endpoint-tp26541447p26541447.html
>>>>>>>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Claus Ibsen
>>>>>>> Apache Camel Committer
>>>>>>>
>>>>>>> Author of Camel in Action: http://www.manning.com/ibsen/
>>>>>>> Open Source Integration: http://fusesource.com
>>>>>>> Blog: http://davsclaus.blogspot.com/
>>>>>>> Twitter: http://twitter.com/davsclaus
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> --
>>>>>> View this message in context:
>>>>>> http://old.nabble.com/camel-casts-exception-when-there-is-nothing-to-fetch-from-the-ftp-endpoint-tp26541447p26541650.html
>>>>>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Claus Ibsen
>>>>> Apache Camel Committer
>>>>>
>>>>> Author of Camel in Action: http://www.manning.com/ibsen/
>>>>> Open Source Integration: http://fusesource.com
>>>>> Blog: http://davsclaus.blogspot.com/
>>>>> Twitter: http://twitter.com/davsclaus
>>>>>
>>>>>
>>>>
>>>> --
>>>> View this message in context:
>>>> http://old.nabble.com/camel-casts-exception-when-there-is-nothing-to-fetch-from-the-ftp-endpoint-tp26541447p26541877.html
>>>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>>>
>>>>
>>>
>>>
>>>
>>> --
>>> Claus Ibsen
>>> Apache Camel Committer
>>>
>>> Author of Camel in Action: http://www.manning.com/ibsen/
>>> Open Source Integration: http://fusesource.com
>>> Blog: http://davsclaus.blogspot.com/
>>> Twitter: http://twitter.com/davsclaus
>>>
>>>
>>
>> --
>> View this message in context: http://old.nabble.com/camel-casts-exception-when-there-is-nothing-to-fetch-from-the-ftp-endpoint-tp26541447p26542125.html
>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>
>>
>
>
>
> --
> Claus Ibsen
> Apache Camel Committer
>
> Author of Camel in Action: http://www.manning.com/ibsen/
> Open Source Integration: http://fusesource.com
> Blog: http://davsclaus.blogspot.com/
> Twitter: http://twitter.com/davsclaus
>
--
Claus Ibsen
Apache Camel Committer
Author of Camel in Action: http://www.manning.com/ibsen/
Open Source Integration: http://fusesource.com
Blog: http://davsclaus.blogspot.com/
Twitter: http://twitter.com/davsclaus
Re: camel casts exception when there is nothing to fetch from the ftp
endpoint
Posted by Claus Ibsen <cl...@gmail.com>.
On Fri, Nov 27, 2009 at 2:00 PM, shirazi <me...@farhad.eu> wrote:
>
> Sure. I have attached the file.
> Actually I use from->to components for transfering files and I guess it
> copies the file, rather than moving.
>
Ah makes more sense for copying to leverage NIO.
Camel does this also but I can see that it may hit that 2gb limit as
it doesnt transfer in a loop using a lower buffer.
It was doing it in a one liner
in.transferTo(0, in.size(), out);
I wonder if you really must do that in a loop to avoid any 2gb limit
or the likes?
> Thanks,
> Farhad S.
> http://old.nabble.com/file/p26542125/Nio.java Nio.java
>
>
> Claus Ibsen-2 wrote:
>>
>> On Fri, Nov 27, 2009 at 1:37 PM, shirazi <me...@farhad.eu> wrote:
>>>
>>> Wau... thanks for that. I don't understand how I could miss it, cause i
>>> searched a lot for such a feature.
>>>
>>> Another thing that I have experienced in 2.0, is that the "file"
>>> component
>>> is unable to move/copy files larger than 2Gigs, such large files are cut
>>> around 2 Gigs limit. Now, I'm not sure if it has anything to do with the
>>> OS
>>> or the filesystems or what. I have seen it both in Linux (debian) and Mac
>>> OS
>>> X. And I don't understand either, why the file component does not use the
>>> NIO api or at least we don't have an option to tell it use the NIO api
>>> instead.
>>> Anyway, for now I've solved the problem by using a smiple component
>>> instead
>>> of the "file" comp. to move/copy files using the NIO api. It is much
>>> faster
>>> and it has no problem with very large files.
>>>
>>
>> For move it uses java.io.File rename and are you saying its not
>> optimized for the underlying OS?
>> I would assume that it could move a 2gb file in no time if the file is
>> on the same share.
>>
>> Can you share your code that you are using?
>>
>>
>>> Bests,
>>> Farhad S.
>>>
>>>
>>> Claus Ibsen-2 wrote:
>>>>
>>>> On Fri, Nov 27, 2009 at 1:12 PM, shirazi <me...@farhad.eu> wrote:
>>>>>
>>>>> Hi,
>>>>> Ok, I guess I should rather provide an AggregationStrategy.
>>>>> The reason why I use a timer is that when I use the ftp endpoint, it
>>>>> starts
>>>>> to fetch everything that is on the ftp in the first call and that's not
>>>>> what
>>>>> i want, because I'm processing files over 3 Gigs and there are many of
>>>>> them.
>>>>> Then I figured that by using a timer, i could have ftp to download one
>>>>> file
>>>>> at a time!!
>>>>> What I basically need is to somehow tell the ftp, how many files I want
>>>>> to
>>>>> fetch at a time. This is probably doable now in camel-2.1 by using the
>>>>> "ThrottlingInflightRoutePolicy", which i havn't tried yet.
>>>>> Do you have any suggestions?
>>>>
>>>> Yeah the FTP component extends the File component so you got all the
>>>> options from the file component as well
>>>> http://camel.apache.org/file2.html
>>>>
>>>> And there is a maxMessagesPerPoll you can use to tell how many files
>>>> you want at most to poll. For example = 1 in your use.
>>>>
>>>> And that option is avail in 2.0 also.
>>>>
>>>> In fact all these are as they are batch polling consumers
>>>> http://camel.apache.org/batch-consumer.html
>>>>
>>>>
>>>>
>>>>>
>>>>> Bests,
>>>>> Farhad S.
>>>>>
>>>>>
>>>>>
>>>>> Claus Ibsen-2 wrote:
>>>>>>
>>>>>> Hi
>>>>>>
>>>>>> BTW why are you using a timer -> ftp when the ftp have a build in
>>>>>> timer?
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Fri, Nov 27, 2009 at 12:55 PM, shirazi <me...@farhad.eu> wrote:
>>>>>>>
>>>>>>> Hi,
>>>>>>> Please consider the following route:
>>>>>>>
>>>>>>> from("timer://someTrigger?delay=30000&fixedRate=true&period=300000").
>>>>>>> pollEnrich("ftp://cmoreFtp?binary=true&localWorkDirectory=/tmp&idempotent=true&idempotentRepository=#caStreams",
>>>>>>> 10000).
>>>>>>> to("file:///tmp/vw/cmore?tempPrefix=.inprogress");
>>>>>>>
>>>>>>> The pollEnrich cast a NullPointerException in camel-2.0, if there is
>>>>>>> nothing
>>>>>>> to fetch from the ftp. I got rid of the exception by a
>>>>>>> "try-catch(NullpointerExcpetion.class).stop()" around the route. Now
>>>>>>> in
>>>>>>> camel-2.1-SNAPSHOT, I get the following exception on the "to"
>>>>>>> endpoint:
>>>>>>> =================
>>>>>>> 12:37:06,899 INFO [Tracer] >>> -->
>>>>>>> file:///tmp/vw/cmore?tempPrefix=.inprogress_, Pattern:InOnly,
>>>>>>> Headers:{firedTime=Fri Nov 27 12:36:56 CET 2009},
>>>>>>> Exception:org.apache.camel.component.file.GenericFileOperationFailedException:
>>>>>>> Cannot store file:
>>>>>>> /tmp/vw/cmore/.inprogress_02216f38-8ffc-4aec-b638-27229d857075
>>>>>>> 12:37:06,899 INFO [Tracer] >>> -->
>>>>>>> file:///tmp/vw/cmore?tempPrefix=.inprogress_, Pattern:InOnly,
>>>>>>> Headers:{firedTime=Fri Nov 27 12:36:56 CET 2009},
>>>>>>> Exception:org.apache.camel.component.file.GenericFileOperationFailedException:
>>>>>>> Cannot store file:
>>>>>>> /tmp/vw/cmore/.inprogress_02216f38-8ffc-4aec-b638-27229d857075
>>>>>>> 12:37:06,899 ERROR [TimerConsumer] Cannot store file:
>>>>>>> /tmp/vw/cmore/.inprogress_02216f38-8ffc-4aec-b638-27229d857075
>>>>>>> org.apache.camel.component.file.GenericFileOperationFailedException:
>>>>>>> Cannot
>>>>>>> store file:
>>>>>>> /tmp/vw/cmore/.inprogress_02216f38-8ffc-4aec-b638-27229d857075
>>>>>>> at
>>>>>>> org.apache.camel.component.file.FileOperations.storeFile(FileOperations.java:204)
>>>>>>> at
>>>>>>> org.apache.camel.component.file.GenericFileProducer.writeFile(GenericFileProducer.java:184)
>>>>>>> at
>>>>>>> org.apache.camel.component.file.GenericFileProducer.processExchange(GenericFileProducer.java:126)
>>>>>>> at
>>>>>>> org.apache.camel.component.file.GenericFileProducer.process(GenericFileProducer.java:57)
>>>>>>> at
>>>>>>> org.apache.camel.processor.SendProcessor$1.doInProducer(SendProcessor.java:97)
>>>>>>> at
>>>>>>> org.apache.camel.processor.SendProcessor$1.doInProducer(SendProcessor.java:95)
>>>>>>> at
>>>>>>> org.apache.camel.impl.ProducerCache.doInProducer(ProducerCache.java:146)
>>>>>>> at
>>>>>>> org.apache.camel.processor.SendProcessor.doProcess(SendProcessor.java:94)
>>>>>>> at
>>>>>>> org.apache.camel.processor.SendProcessor.process(SendProcessor.java:82)
>>>>>>> at
>>>>>>> org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:67)
>>>>>>> at
>>>>>>> org.apache.camel.processor.DelegateProcessor.processNext(DelegateProcessor.java:53)
>>>>>>> at
>>>>>>> org.apache.camel.processor.DelegateProcessor.proceed(DelegateProcessor.java:82)
>>>>>>> at
>>>>>>> org.apache.camel.processor.interceptor.TraceInterceptor.process(TraceInterceptor.java:162)
>>>>>>> at
>>>>>>> org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:67)
>>>>>>> at
>>>>>>> org.apache.camel.processor.DefaultChannel.process(DefaultChannel.java:206)
>>>>>>> at
>>>>>>> org.apache.camel.processor.Pipeline.process(Pipeline.java:74)
>>>>>>> at
>>>>>>> org.apache.camel.processor.TryProcessor.process(TryProcessor.java:63)
>>>>>>> at
>>>>>>> org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:67)
>>>>>>> at
>>>>>>> org.apache.camel.processor.DelegateProcessor.processNext(DelegateProcessor.java:53)
>>>>>>> at
>>>>>>> org.apache.camel.processor.DelegateProcessor.proceed(DelegateProcessor.java:82)
>>>>>>> at
>>>>>>> org.apache.camel.processor.interceptor.TraceInterceptor.process(TraceInterceptor.java:162)
>>>>>>> at
>>>>>>> org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:67)
>>>>>>> at
>>>>>>> org.apache.camel.processor.DefaultChannel.process(DefaultChannel.java:206)
>>>>>>> at
>>>>>>> org.apache.camel.processor.UnitOfWorkProcessor.processNext(UnitOfWorkProcessor.java:54)
>>>>>>> at
>>>>>>> org.apache.camel.processor.DelegateProcessor.process(DelegateProcessor.java:48)
>>>>>>> at
>>>>>>> org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:67)
>>>>>>> at
>>>>>>> org.apache.camel.component.timer.TimerConsumer.sendTimerExchange(TimerConsumer.java:103)
>>>>>>> at
>>>>>>> org.apache.camel.component.timer.TimerConsumer$1.run(TimerConsumer.java:50)
>>>>>>> at java.util.TimerThread.mainLoop(Timer.java:512)
>>>>>>> at java.util.TimerThread.run(Timer.java:462)
>>>>>>> Caused by: org.apache.camel.InvalidPayloadException: No body
>>>>>>> available
>>>>>>> of
>>>>>>> type: java.io.InputStream on: Message: [Body is null]. Caused by: No
>>>>>>> type
>>>>>>> converter available to convert from type: null to the required type:
>>>>>>> java.io.InputStream with value null on the exchange:
>>>>>>> Exchange[Message:
>>>>>>> [Body
>>>>>>> is null]]
>>>>>>> at
>>>>>>> org.apache.camel.impl.MessageSupport.getMandatoryBody(MessageSupport.java:103)
>>>>>>> at
>>>>>>> org.apache.camel.util.ExchangeHelper.getMandatoryInBody(ExchangeHelper.java:116)
>>>>>>> at
>>>>>>> org.apache.camel.component.file.FileOperations.storeFile(FileOperations.java:198)
>>>>>>> ... 29 more
>>>>>>> Caused by: org.apache.camel.NoTypeConversionAvailableException: No
>>>>>>> type
>>>>>>> converter available to convert from type: null to the required type:
>>>>>>> java.io.InputStream with value null
>>>>>>> at
>>>>>>> org.apache.camel.impl.converter.DefaultTypeConverter.mandatoryConvertTo(DefaultTypeConverter.java:126)
>>>>>>> at
>>>>>>> org.apache.camel.impl.MessageSupport.getMandatoryBody(MessageSupport.java:101)
>>>>>>> ... 31 more
>>>>>>> ============
>>>>>>> Now, I can remedy the situation by a try-catch block or an
>>>>>>> interceptor,
>>>>>>> but
>>>>>>> in my opinion this should be considered a bug in camel and it should
>>>>>>> silently ignore if there was nothing to fetch from the ftp.
>>>>>>>
>>>>>>> Bests,
>>>>>>> Farhad Shiraz
>>>>>>>
>>>>>>> --
>>>>>>> View this message in context:
>>>>>>> http://old.nabble.com/camel-casts-exception-when-there-is-nothing-to-fetch-from-the-ftp-endpoint-tp26541447p26541447.html
>>>>>>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Claus Ibsen
>>>>>> Apache Camel Committer
>>>>>>
>>>>>> Author of Camel in Action: http://www.manning.com/ibsen/
>>>>>> Open Source Integration: http://fusesource.com
>>>>>> Blog: http://davsclaus.blogspot.com/
>>>>>> Twitter: http://twitter.com/davsclaus
>>>>>>
>>>>>>
>>>>>
>>>>> --
>>>>> View this message in context:
>>>>> http://old.nabble.com/camel-casts-exception-when-there-is-nothing-to-fetch-from-the-ftp-endpoint-tp26541447p26541650.html
>>>>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Claus Ibsen
>>>> Apache Camel Committer
>>>>
>>>> Author of Camel in Action: http://www.manning.com/ibsen/
>>>> Open Source Integration: http://fusesource.com
>>>> Blog: http://davsclaus.blogspot.com/
>>>> Twitter: http://twitter.com/davsclaus
>>>>
>>>>
>>>
>>> --
>>> View this message in context:
>>> http://old.nabble.com/camel-casts-exception-when-there-is-nothing-to-fetch-from-the-ftp-endpoint-tp26541447p26541877.html
>>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>>
>>>
>>
>>
>>
>> --
>> Claus Ibsen
>> Apache Camel Committer
>>
>> Author of Camel in Action: http://www.manning.com/ibsen/
>> Open Source Integration: http://fusesource.com
>> Blog: http://davsclaus.blogspot.com/
>> Twitter: http://twitter.com/davsclaus
>>
>>
>
> --
> View this message in context: http://old.nabble.com/camel-casts-exception-when-there-is-nothing-to-fetch-from-the-ftp-endpoint-tp26541447p26542125.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>
>
--
Claus Ibsen
Apache Camel Committer
Author of Camel in Action: http://www.manning.com/ibsen/
Open Source Integration: http://fusesource.com
Blog: http://davsclaus.blogspot.com/
Twitter: http://twitter.com/davsclaus
Re: camel casts exception when there is nothing to fetch from the
ftp endpoint
Posted by shirazi <me...@farhad.eu>.
Sure. I have attached the file.
Actually I use from->to components for transfering files and I guess it
copies the file, rather than moving.
Thanks,
Farhad S.
http://old.nabble.com/file/p26542125/Nio.java Nio.java
Claus Ibsen-2 wrote:
>
> On Fri, Nov 27, 2009 at 1:37 PM, shirazi <me...@farhad.eu> wrote:
>>
>> Wau... thanks for that. I don't understand how I could miss it, cause i
>> searched a lot for such a feature.
>>
>> Another thing that I have experienced in 2.0, is that the "file"
>> component
>> is unable to move/copy files larger than 2Gigs, such large files are cut
>> around 2 Gigs limit. Now, I'm not sure if it has anything to do with the
>> OS
>> or the filesystems or what. I have seen it both in Linux (debian) and Mac
>> OS
>> X. And I don't understand either, why the file component does not use the
>> NIO api or at least we don't have an option to tell it use the NIO api
>> instead.
>> Anyway, for now I've solved the problem by using a smiple component
>> instead
>> of the "file" comp. to move/copy files using the NIO api. It is much
>> faster
>> and it has no problem with very large files.
>>
>
> For move it uses java.io.File rename and are you saying its not
> optimized for the underlying OS?
> I would assume that it could move a 2gb file in no time if the file is
> on the same share.
>
> Can you share your code that you are using?
>
>
>> Bests,
>> Farhad S.
>>
>>
>> Claus Ibsen-2 wrote:
>>>
>>> On Fri, Nov 27, 2009 at 1:12 PM, shirazi <me...@farhad.eu> wrote:
>>>>
>>>> Hi,
>>>> Ok, I guess I should rather provide an AggregationStrategy.
>>>> The reason why I use a timer is that when I use the ftp endpoint, it
>>>> starts
>>>> to fetch everything that is on the ftp in the first call and that's not
>>>> what
>>>> i want, because I'm processing files over 3 Gigs and there are many of
>>>> them.
>>>> Then I figured that by using a timer, i could have ftp to download one
>>>> file
>>>> at a time!!
>>>> What I basically need is to somehow tell the ftp, how many files I want
>>>> to
>>>> fetch at a time. This is probably doable now in camel-2.1 by using the
>>>> "ThrottlingInflightRoutePolicy", which i havn't tried yet.
>>>> Do you have any suggestions?
>>>
>>> Yeah the FTP component extends the File component so you got all the
>>> options from the file component as well
>>> http://camel.apache.org/file2.html
>>>
>>> And there is a maxMessagesPerPoll you can use to tell how many files
>>> you want at most to poll. For example = 1 in your use.
>>>
>>> And that option is avail in 2.0 also.
>>>
>>> In fact all these are as they are batch polling consumers
>>> http://camel.apache.org/batch-consumer.html
>>>
>>>
>>>
>>>>
>>>> Bests,
>>>> Farhad S.
>>>>
>>>>
>>>>
>>>> Claus Ibsen-2 wrote:
>>>>>
>>>>> Hi
>>>>>
>>>>> BTW why are you using a timer -> ftp when the ftp have a build in
>>>>> timer?
>>>>>
>>>>>
>>>>>
>>>>> On Fri, Nov 27, 2009 at 12:55 PM, shirazi <me...@farhad.eu> wrote:
>>>>>>
>>>>>> Hi,
>>>>>> Please consider the following route:
>>>>>>
>>>>>> from("timer://someTrigger?delay=30000&fixedRate=true&period=300000").
>>>>>> pollEnrich("ftp://cmoreFtp?binary=true&localWorkDirectory=/tmp&idempotent=true&idempotentRepository=#caStreams",
>>>>>> 10000).
>>>>>> to("file:///tmp/vw/cmore?tempPrefix=.inprogress");
>>>>>>
>>>>>> The pollEnrich cast a NullPointerException in camel-2.0, if there is
>>>>>> nothing
>>>>>> to fetch from the ftp. I got rid of the exception by a
>>>>>> "try-catch(NullpointerExcpetion.class).stop()" around the route. Now
>>>>>> in
>>>>>> camel-2.1-SNAPSHOT, I get the following exception on the "to"
>>>>>> endpoint:
>>>>>> =================
>>>>>> 12:37:06,899 INFO [Tracer] >>> -->
>>>>>> file:///tmp/vw/cmore?tempPrefix=.inprogress_, Pattern:InOnly,
>>>>>> Headers:{firedTime=Fri Nov 27 12:36:56 CET 2009},
>>>>>> Exception:org.apache.camel.component.file.GenericFileOperationFailedException:
>>>>>> Cannot store file:
>>>>>> /tmp/vw/cmore/.inprogress_02216f38-8ffc-4aec-b638-27229d857075
>>>>>> 12:37:06,899 INFO [Tracer] >>> -->
>>>>>> file:///tmp/vw/cmore?tempPrefix=.inprogress_, Pattern:InOnly,
>>>>>> Headers:{firedTime=Fri Nov 27 12:36:56 CET 2009},
>>>>>> Exception:org.apache.camel.component.file.GenericFileOperationFailedException:
>>>>>> Cannot store file:
>>>>>> /tmp/vw/cmore/.inprogress_02216f38-8ffc-4aec-b638-27229d857075
>>>>>> 12:37:06,899 ERROR [TimerConsumer] Cannot store file:
>>>>>> /tmp/vw/cmore/.inprogress_02216f38-8ffc-4aec-b638-27229d857075
>>>>>> org.apache.camel.component.file.GenericFileOperationFailedException:
>>>>>> Cannot
>>>>>> store file:
>>>>>> /tmp/vw/cmore/.inprogress_02216f38-8ffc-4aec-b638-27229d857075
>>>>>> at
>>>>>> org.apache.camel.component.file.FileOperations.storeFile(FileOperations.java:204)
>>>>>> at
>>>>>> org.apache.camel.component.file.GenericFileProducer.writeFile(GenericFileProducer.java:184)
>>>>>> at
>>>>>> org.apache.camel.component.file.GenericFileProducer.processExchange(GenericFileProducer.java:126)
>>>>>> at
>>>>>> org.apache.camel.component.file.GenericFileProducer.process(GenericFileProducer.java:57)
>>>>>> at
>>>>>> org.apache.camel.processor.SendProcessor$1.doInProducer(SendProcessor.java:97)
>>>>>> at
>>>>>> org.apache.camel.processor.SendProcessor$1.doInProducer(SendProcessor.java:95)
>>>>>> at
>>>>>> org.apache.camel.impl.ProducerCache.doInProducer(ProducerCache.java:146)
>>>>>> at
>>>>>> org.apache.camel.processor.SendProcessor.doProcess(SendProcessor.java:94)
>>>>>> at
>>>>>> org.apache.camel.processor.SendProcessor.process(SendProcessor.java:82)
>>>>>> at
>>>>>> org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:67)
>>>>>> at
>>>>>> org.apache.camel.processor.DelegateProcessor.processNext(DelegateProcessor.java:53)
>>>>>> at
>>>>>> org.apache.camel.processor.DelegateProcessor.proceed(DelegateProcessor.java:82)
>>>>>> at
>>>>>> org.apache.camel.processor.interceptor.TraceInterceptor.process(TraceInterceptor.java:162)
>>>>>> at
>>>>>> org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:67)
>>>>>> at
>>>>>> org.apache.camel.processor.DefaultChannel.process(DefaultChannel.java:206)
>>>>>> at
>>>>>> org.apache.camel.processor.Pipeline.process(Pipeline.java:74)
>>>>>> at
>>>>>> org.apache.camel.processor.TryProcessor.process(TryProcessor.java:63)
>>>>>> at
>>>>>> org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:67)
>>>>>> at
>>>>>> org.apache.camel.processor.DelegateProcessor.processNext(DelegateProcessor.java:53)
>>>>>> at
>>>>>> org.apache.camel.processor.DelegateProcessor.proceed(DelegateProcessor.java:82)
>>>>>> at
>>>>>> org.apache.camel.processor.interceptor.TraceInterceptor.process(TraceInterceptor.java:162)
>>>>>> at
>>>>>> org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:67)
>>>>>> at
>>>>>> org.apache.camel.processor.DefaultChannel.process(DefaultChannel.java:206)
>>>>>> at
>>>>>> org.apache.camel.processor.UnitOfWorkProcessor.processNext(UnitOfWorkProcessor.java:54)
>>>>>> at
>>>>>> org.apache.camel.processor.DelegateProcessor.process(DelegateProcessor.java:48)
>>>>>> at
>>>>>> org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:67)
>>>>>> at
>>>>>> org.apache.camel.component.timer.TimerConsumer.sendTimerExchange(TimerConsumer.java:103)
>>>>>> at
>>>>>> org.apache.camel.component.timer.TimerConsumer$1.run(TimerConsumer.java:50)
>>>>>> at java.util.TimerThread.mainLoop(Timer.java:512)
>>>>>> at java.util.TimerThread.run(Timer.java:462)
>>>>>> Caused by: org.apache.camel.InvalidPayloadException: No body
>>>>>> available
>>>>>> of
>>>>>> type: java.io.InputStream on: Message: [Body is null]. Caused by: No
>>>>>> type
>>>>>> converter available to convert from type: null to the required type:
>>>>>> java.io.InputStream with value null on the exchange:
>>>>>> Exchange[Message:
>>>>>> [Body
>>>>>> is null]]
>>>>>> at
>>>>>> org.apache.camel.impl.MessageSupport.getMandatoryBody(MessageSupport.java:103)
>>>>>> at
>>>>>> org.apache.camel.util.ExchangeHelper.getMandatoryInBody(ExchangeHelper.java:116)
>>>>>> at
>>>>>> org.apache.camel.component.file.FileOperations.storeFile(FileOperations.java:198)
>>>>>> ... 29 more
>>>>>> Caused by: org.apache.camel.NoTypeConversionAvailableException: No
>>>>>> type
>>>>>> converter available to convert from type: null to the required type:
>>>>>> java.io.InputStream with value null
>>>>>> at
>>>>>> org.apache.camel.impl.converter.DefaultTypeConverter.mandatoryConvertTo(DefaultTypeConverter.java:126)
>>>>>> at
>>>>>> org.apache.camel.impl.MessageSupport.getMandatoryBody(MessageSupport.java:101)
>>>>>> ... 31 more
>>>>>> ============
>>>>>> Now, I can remedy the situation by a try-catch block or an
>>>>>> interceptor,
>>>>>> but
>>>>>> in my opinion this should be considered a bug in camel and it should
>>>>>> silently ignore if there was nothing to fetch from the ftp.
>>>>>>
>>>>>> Bests,
>>>>>> Farhad Shiraz
>>>>>>
>>>>>> --
>>>>>> View this message in context:
>>>>>> http://old.nabble.com/camel-casts-exception-when-there-is-nothing-to-fetch-from-the-ftp-endpoint-tp26541447p26541447.html
>>>>>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Claus Ibsen
>>>>> Apache Camel Committer
>>>>>
>>>>> Author of Camel in Action: http://www.manning.com/ibsen/
>>>>> Open Source Integration: http://fusesource.com
>>>>> Blog: http://davsclaus.blogspot.com/
>>>>> Twitter: http://twitter.com/davsclaus
>>>>>
>>>>>
>>>>
>>>> --
>>>> View this message in context:
>>>> http://old.nabble.com/camel-casts-exception-when-there-is-nothing-to-fetch-from-the-ftp-endpoint-tp26541447p26541650.html
>>>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>>>
>>>>
>>>
>>>
>>>
>>> --
>>> Claus Ibsen
>>> Apache Camel Committer
>>>
>>> Author of Camel in Action: http://www.manning.com/ibsen/
>>> Open Source Integration: http://fusesource.com
>>> Blog: http://davsclaus.blogspot.com/
>>> Twitter: http://twitter.com/davsclaus
>>>
>>>
>>
>> --
>> View this message in context:
>> http://old.nabble.com/camel-casts-exception-when-there-is-nothing-to-fetch-from-the-ftp-endpoint-tp26541447p26541877.html
>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>
>>
>
>
>
> --
> Claus Ibsen
> Apache Camel Committer
>
> Author of Camel in Action: http://www.manning.com/ibsen/
> Open Source Integration: http://fusesource.com
> Blog: http://davsclaus.blogspot.com/
> Twitter: http://twitter.com/davsclaus
>
>
--
View this message in context: http://old.nabble.com/camel-casts-exception-when-there-is-nothing-to-fetch-from-the-ftp-endpoint-tp26541447p26542125.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: camel casts exception when there is nothing to fetch from the ftp
endpoint
Posted by Claus Ibsen <cl...@gmail.com>.
On Fri, Nov 27, 2009 at 1:37 PM, shirazi <me...@farhad.eu> wrote:
>
> Wau... thanks for that. I don't understand how I could miss it, cause i
> searched a lot for such a feature.
>
> Another thing that I have experienced in 2.0, is that the "file" component
> is unable to move/copy files larger than 2Gigs, such large files are cut
> around 2 Gigs limit. Now, I'm not sure if it has anything to do with the OS
> or the filesystems or what. I have seen it both in Linux (debian) and Mac OS
> X. And I don't understand either, why the file component does not use the
> NIO api or at least we don't have an option to tell it use the NIO api
> instead.
> Anyway, for now I've solved the problem by using a smiple component instead
> of the "file" comp. to move/copy files using the NIO api. It is much faster
> and it has no problem with very large files.
>
For move it uses java.io.File rename and are you saying its not
optimized for the underlying OS?
I would assume that it could move a 2gb file in no time if the file is
on the same share.
Can you share your code that you are using?
> Bests,
> Farhad S.
>
>
> Claus Ibsen-2 wrote:
>>
>> On Fri, Nov 27, 2009 at 1:12 PM, shirazi <me...@farhad.eu> wrote:
>>>
>>> Hi,
>>> Ok, I guess I should rather provide an AggregationStrategy.
>>> The reason why I use a timer is that when I use the ftp endpoint, it
>>> starts
>>> to fetch everything that is on the ftp in the first call and that's not
>>> what
>>> i want, because I'm processing files over 3 Gigs and there are many of
>>> them.
>>> Then I figured that by using a timer, i could have ftp to download one
>>> file
>>> at a time!!
>>> What I basically need is to somehow tell the ftp, how many files I want
>>> to
>>> fetch at a time. This is probably doable now in camel-2.1 by using the
>>> "ThrottlingInflightRoutePolicy", which i havn't tried yet.
>>> Do you have any suggestions?
>>
>> Yeah the FTP component extends the File component so you got all the
>> options from the file component as well
>> http://camel.apache.org/file2.html
>>
>> And there is a maxMessagesPerPoll you can use to tell how many files
>> you want at most to poll. For example = 1 in your use.
>>
>> And that option is avail in 2.0 also.
>>
>> In fact all these are as they are batch polling consumers
>> http://camel.apache.org/batch-consumer.html
>>
>>
>>
>>>
>>> Bests,
>>> Farhad S.
>>>
>>>
>>>
>>> Claus Ibsen-2 wrote:
>>>>
>>>> Hi
>>>>
>>>> BTW why are you using a timer -> ftp when the ftp have a build in timer?
>>>>
>>>>
>>>>
>>>> On Fri, Nov 27, 2009 at 12:55 PM, shirazi <me...@farhad.eu> wrote:
>>>>>
>>>>> Hi,
>>>>> Please consider the following route:
>>>>>
>>>>> from("timer://someTrigger?delay=30000&fixedRate=true&period=300000").
>>>>> pollEnrich("ftp://cmoreFtp?binary=true&localWorkDirectory=/tmp&idempotent=true&idempotentRepository=#caStreams",
>>>>> 10000).
>>>>> to("file:///tmp/vw/cmore?tempPrefix=.inprogress");
>>>>>
>>>>> The pollEnrich cast a NullPointerException in camel-2.0, if there is
>>>>> nothing
>>>>> to fetch from the ftp. I got rid of the exception by a
>>>>> "try-catch(NullpointerExcpetion.class).stop()" around the route. Now in
>>>>> camel-2.1-SNAPSHOT, I get the following exception on the "to" endpoint:
>>>>> =================
>>>>> 12:37:06,899 INFO [Tracer] >>> -->
>>>>> file:///tmp/vw/cmore?tempPrefix=.inprogress_, Pattern:InOnly,
>>>>> Headers:{firedTime=Fri Nov 27 12:36:56 CET 2009},
>>>>> Exception:org.apache.camel.component.file.GenericFileOperationFailedException:
>>>>> Cannot store file:
>>>>> /tmp/vw/cmore/.inprogress_02216f38-8ffc-4aec-b638-27229d857075
>>>>> 12:37:06,899 INFO [Tracer] >>> -->
>>>>> file:///tmp/vw/cmore?tempPrefix=.inprogress_, Pattern:InOnly,
>>>>> Headers:{firedTime=Fri Nov 27 12:36:56 CET 2009},
>>>>> Exception:org.apache.camel.component.file.GenericFileOperationFailedException:
>>>>> Cannot store file:
>>>>> /tmp/vw/cmore/.inprogress_02216f38-8ffc-4aec-b638-27229d857075
>>>>> 12:37:06,899 ERROR [TimerConsumer] Cannot store file:
>>>>> /tmp/vw/cmore/.inprogress_02216f38-8ffc-4aec-b638-27229d857075
>>>>> org.apache.camel.component.file.GenericFileOperationFailedException:
>>>>> Cannot
>>>>> store file:
>>>>> /tmp/vw/cmore/.inprogress_02216f38-8ffc-4aec-b638-27229d857075
>>>>> at
>>>>> org.apache.camel.component.file.FileOperations.storeFile(FileOperations.java:204)
>>>>> at
>>>>> org.apache.camel.component.file.GenericFileProducer.writeFile(GenericFileProducer.java:184)
>>>>> at
>>>>> org.apache.camel.component.file.GenericFileProducer.processExchange(GenericFileProducer.java:126)
>>>>> at
>>>>> org.apache.camel.component.file.GenericFileProducer.process(GenericFileProducer.java:57)
>>>>> at
>>>>> org.apache.camel.processor.SendProcessor$1.doInProducer(SendProcessor.java:97)
>>>>> at
>>>>> org.apache.camel.processor.SendProcessor$1.doInProducer(SendProcessor.java:95)
>>>>> at
>>>>> org.apache.camel.impl.ProducerCache.doInProducer(ProducerCache.java:146)
>>>>> at
>>>>> org.apache.camel.processor.SendProcessor.doProcess(SendProcessor.java:94)
>>>>> at
>>>>> org.apache.camel.processor.SendProcessor.process(SendProcessor.java:82)
>>>>> at
>>>>> org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:67)
>>>>> at
>>>>> org.apache.camel.processor.DelegateProcessor.processNext(DelegateProcessor.java:53)
>>>>> at
>>>>> org.apache.camel.processor.DelegateProcessor.proceed(DelegateProcessor.java:82)
>>>>> at
>>>>> org.apache.camel.processor.interceptor.TraceInterceptor.process(TraceInterceptor.java:162)
>>>>> at
>>>>> org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:67)
>>>>> at
>>>>> org.apache.camel.processor.DefaultChannel.process(DefaultChannel.java:206)
>>>>> at org.apache.camel.processor.Pipeline.process(Pipeline.java:74)
>>>>> at
>>>>> org.apache.camel.processor.TryProcessor.process(TryProcessor.java:63)
>>>>> at
>>>>> org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:67)
>>>>> at
>>>>> org.apache.camel.processor.DelegateProcessor.processNext(DelegateProcessor.java:53)
>>>>> at
>>>>> org.apache.camel.processor.DelegateProcessor.proceed(DelegateProcessor.java:82)
>>>>> at
>>>>> org.apache.camel.processor.interceptor.TraceInterceptor.process(TraceInterceptor.java:162)
>>>>> at
>>>>> org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:67)
>>>>> at
>>>>> org.apache.camel.processor.DefaultChannel.process(DefaultChannel.java:206)
>>>>> at
>>>>> org.apache.camel.processor.UnitOfWorkProcessor.processNext(UnitOfWorkProcessor.java:54)
>>>>> at
>>>>> org.apache.camel.processor.DelegateProcessor.process(DelegateProcessor.java:48)
>>>>> at
>>>>> org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:67)
>>>>> at
>>>>> org.apache.camel.component.timer.TimerConsumer.sendTimerExchange(TimerConsumer.java:103)
>>>>> at
>>>>> org.apache.camel.component.timer.TimerConsumer$1.run(TimerConsumer.java:50)
>>>>> at java.util.TimerThread.mainLoop(Timer.java:512)
>>>>> at java.util.TimerThread.run(Timer.java:462)
>>>>> Caused by: org.apache.camel.InvalidPayloadException: No body available
>>>>> of
>>>>> type: java.io.InputStream on: Message: [Body is null]. Caused by: No
>>>>> type
>>>>> converter available to convert from type: null to the required type:
>>>>> java.io.InputStream with value null on the exchange: Exchange[Message:
>>>>> [Body
>>>>> is null]]
>>>>> at
>>>>> org.apache.camel.impl.MessageSupport.getMandatoryBody(MessageSupport.java:103)
>>>>> at
>>>>> org.apache.camel.util.ExchangeHelper.getMandatoryInBody(ExchangeHelper.java:116)
>>>>> at
>>>>> org.apache.camel.component.file.FileOperations.storeFile(FileOperations.java:198)
>>>>> ... 29 more
>>>>> Caused by: org.apache.camel.NoTypeConversionAvailableException: No type
>>>>> converter available to convert from type: null to the required type:
>>>>> java.io.InputStream with value null
>>>>> at
>>>>> org.apache.camel.impl.converter.DefaultTypeConverter.mandatoryConvertTo(DefaultTypeConverter.java:126)
>>>>> at
>>>>> org.apache.camel.impl.MessageSupport.getMandatoryBody(MessageSupport.java:101)
>>>>> ... 31 more
>>>>> ============
>>>>> Now, I can remedy the situation by a try-catch block or an interceptor,
>>>>> but
>>>>> in my opinion this should be considered a bug in camel and it should
>>>>> silently ignore if there was nothing to fetch from the ftp.
>>>>>
>>>>> Bests,
>>>>> Farhad Shiraz
>>>>>
>>>>> --
>>>>> View this message in context:
>>>>> http://old.nabble.com/camel-casts-exception-when-there-is-nothing-to-fetch-from-the-ftp-endpoint-tp26541447p26541447.html
>>>>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Claus Ibsen
>>>> Apache Camel Committer
>>>>
>>>> Author of Camel in Action: http://www.manning.com/ibsen/
>>>> Open Source Integration: http://fusesource.com
>>>> Blog: http://davsclaus.blogspot.com/
>>>> Twitter: http://twitter.com/davsclaus
>>>>
>>>>
>>>
>>> --
>>> View this message in context:
>>> http://old.nabble.com/camel-casts-exception-when-there-is-nothing-to-fetch-from-the-ftp-endpoint-tp26541447p26541650.html
>>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>>
>>>
>>
>>
>>
>> --
>> Claus Ibsen
>> Apache Camel Committer
>>
>> Author of Camel in Action: http://www.manning.com/ibsen/
>> Open Source Integration: http://fusesource.com
>> Blog: http://davsclaus.blogspot.com/
>> Twitter: http://twitter.com/davsclaus
>>
>>
>
> --
> View this message in context: http://old.nabble.com/camel-casts-exception-when-there-is-nothing-to-fetch-from-the-ftp-endpoint-tp26541447p26541877.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>
>
--
Claus Ibsen
Apache Camel Committer
Author of Camel in Action: http://www.manning.com/ibsen/
Open Source Integration: http://fusesource.com
Blog: http://davsclaus.blogspot.com/
Twitter: http://twitter.com/davsclaus
Re: camel casts exception when there is nothing to fetch from the
ftp endpoint
Posted by shirazi <me...@farhad.eu>.
Wau... thanks for that. I don't understand how I could miss it, cause i
searched a lot for such a feature.
Another thing that I have experienced in 2.0, is that the "file" component
is unable to move/copy files larger than 2Gigs, such large files are cut
around 2 Gigs limit. Now, I'm not sure if it has anything to do with the OS
or the filesystems or what. I have seen it both in Linux (debian) and Mac OS
X. And I don't understand either, why the file component does not use the
NIO api or at least we don't have an option to tell it use the NIO api
instead.
Anyway, for now I've solved the problem by using a smiple component instead
of the "file" comp. to move/copy files using the NIO api. It is much faster
and it has no problem with very large files.
Bests,
Farhad S.
Claus Ibsen-2 wrote:
>
> On Fri, Nov 27, 2009 at 1:12 PM, shirazi <me...@farhad.eu> wrote:
>>
>> Hi,
>> Ok, I guess I should rather provide an AggregationStrategy.
>> The reason why I use a timer is that when I use the ftp endpoint, it
>> starts
>> to fetch everything that is on the ftp in the first call and that's not
>> what
>> i want, because I'm processing files over 3 Gigs and there are many of
>> them.
>> Then I figured that by using a timer, i could have ftp to download one
>> file
>> at a time!!
>> What I basically need is to somehow tell the ftp, how many files I want
>> to
>> fetch at a time. This is probably doable now in camel-2.1 by using the
>> "ThrottlingInflightRoutePolicy", which i havn't tried yet.
>> Do you have any suggestions?
>
> Yeah the FTP component extends the File component so you got all the
> options from the file component as well
> http://camel.apache.org/file2.html
>
> And there is a maxMessagesPerPoll you can use to tell how many files
> you want at most to poll. For example = 1 in your use.
>
> And that option is avail in 2.0 also.
>
> In fact all these are as they are batch polling consumers
> http://camel.apache.org/batch-consumer.html
>
>
>
>>
>> Bests,
>> Farhad S.
>>
>>
>>
>> Claus Ibsen-2 wrote:
>>>
>>> Hi
>>>
>>> BTW why are you using a timer -> ftp when the ftp have a build in timer?
>>>
>>>
>>>
>>> On Fri, Nov 27, 2009 at 12:55 PM, shirazi <me...@farhad.eu> wrote:
>>>>
>>>> Hi,
>>>> Please consider the following route:
>>>>
>>>> from("timer://someTrigger?delay=30000&fixedRate=true&period=300000").
>>>> pollEnrich("ftp://cmoreFtp?binary=true&localWorkDirectory=/tmp&idempotent=true&idempotentRepository=#caStreams",
>>>> 10000).
>>>> to("file:///tmp/vw/cmore?tempPrefix=.inprogress");
>>>>
>>>> The pollEnrich cast a NullPointerException in camel-2.0, if there is
>>>> nothing
>>>> to fetch from the ftp. I got rid of the exception by a
>>>> "try-catch(NullpointerExcpetion.class).stop()" around the route. Now in
>>>> camel-2.1-SNAPSHOT, I get the following exception on the "to" endpoint:
>>>> =================
>>>> 12:37:06,899 INFO [Tracer] >>> -->
>>>> file:///tmp/vw/cmore?tempPrefix=.inprogress_, Pattern:InOnly,
>>>> Headers:{firedTime=Fri Nov 27 12:36:56 CET 2009},
>>>> Exception:org.apache.camel.component.file.GenericFileOperationFailedException:
>>>> Cannot store file:
>>>> /tmp/vw/cmore/.inprogress_02216f38-8ffc-4aec-b638-27229d857075
>>>> 12:37:06,899 INFO [Tracer] >>> -->
>>>> file:///tmp/vw/cmore?tempPrefix=.inprogress_, Pattern:InOnly,
>>>> Headers:{firedTime=Fri Nov 27 12:36:56 CET 2009},
>>>> Exception:org.apache.camel.component.file.GenericFileOperationFailedException:
>>>> Cannot store file:
>>>> /tmp/vw/cmore/.inprogress_02216f38-8ffc-4aec-b638-27229d857075
>>>> 12:37:06,899 ERROR [TimerConsumer] Cannot store file:
>>>> /tmp/vw/cmore/.inprogress_02216f38-8ffc-4aec-b638-27229d857075
>>>> org.apache.camel.component.file.GenericFileOperationFailedException:
>>>> Cannot
>>>> store file:
>>>> /tmp/vw/cmore/.inprogress_02216f38-8ffc-4aec-b638-27229d857075
>>>> at
>>>> org.apache.camel.component.file.FileOperations.storeFile(FileOperations.java:204)
>>>> at
>>>> org.apache.camel.component.file.GenericFileProducer.writeFile(GenericFileProducer.java:184)
>>>> at
>>>> org.apache.camel.component.file.GenericFileProducer.processExchange(GenericFileProducer.java:126)
>>>> at
>>>> org.apache.camel.component.file.GenericFileProducer.process(GenericFileProducer.java:57)
>>>> at
>>>> org.apache.camel.processor.SendProcessor$1.doInProducer(SendProcessor.java:97)
>>>> at
>>>> org.apache.camel.processor.SendProcessor$1.doInProducer(SendProcessor.java:95)
>>>> at
>>>> org.apache.camel.impl.ProducerCache.doInProducer(ProducerCache.java:146)
>>>> at
>>>> org.apache.camel.processor.SendProcessor.doProcess(SendProcessor.java:94)
>>>> at
>>>> org.apache.camel.processor.SendProcessor.process(SendProcessor.java:82)
>>>> at
>>>> org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:67)
>>>> at
>>>> org.apache.camel.processor.DelegateProcessor.processNext(DelegateProcessor.java:53)
>>>> at
>>>> org.apache.camel.processor.DelegateProcessor.proceed(DelegateProcessor.java:82)
>>>> at
>>>> org.apache.camel.processor.interceptor.TraceInterceptor.process(TraceInterceptor.java:162)
>>>> at
>>>> org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:67)
>>>> at
>>>> org.apache.camel.processor.DefaultChannel.process(DefaultChannel.java:206)
>>>> at org.apache.camel.processor.Pipeline.process(Pipeline.java:74)
>>>> at
>>>> org.apache.camel.processor.TryProcessor.process(TryProcessor.java:63)
>>>> at
>>>> org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:67)
>>>> at
>>>> org.apache.camel.processor.DelegateProcessor.processNext(DelegateProcessor.java:53)
>>>> at
>>>> org.apache.camel.processor.DelegateProcessor.proceed(DelegateProcessor.java:82)
>>>> at
>>>> org.apache.camel.processor.interceptor.TraceInterceptor.process(TraceInterceptor.java:162)
>>>> at
>>>> org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:67)
>>>> at
>>>> org.apache.camel.processor.DefaultChannel.process(DefaultChannel.java:206)
>>>> at
>>>> org.apache.camel.processor.UnitOfWorkProcessor.processNext(UnitOfWorkProcessor.java:54)
>>>> at
>>>> org.apache.camel.processor.DelegateProcessor.process(DelegateProcessor.java:48)
>>>> at
>>>> org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:67)
>>>> at
>>>> org.apache.camel.component.timer.TimerConsumer.sendTimerExchange(TimerConsumer.java:103)
>>>> at
>>>> org.apache.camel.component.timer.TimerConsumer$1.run(TimerConsumer.java:50)
>>>> at java.util.TimerThread.mainLoop(Timer.java:512)
>>>> at java.util.TimerThread.run(Timer.java:462)
>>>> Caused by: org.apache.camel.InvalidPayloadException: No body available
>>>> of
>>>> type: java.io.InputStream on: Message: [Body is null]. Caused by: No
>>>> type
>>>> converter available to convert from type: null to the required type:
>>>> java.io.InputStream with value null on the exchange: Exchange[Message:
>>>> [Body
>>>> is null]]
>>>> at
>>>> org.apache.camel.impl.MessageSupport.getMandatoryBody(MessageSupport.java:103)
>>>> at
>>>> org.apache.camel.util.ExchangeHelper.getMandatoryInBody(ExchangeHelper.java:116)
>>>> at
>>>> org.apache.camel.component.file.FileOperations.storeFile(FileOperations.java:198)
>>>> ... 29 more
>>>> Caused by: org.apache.camel.NoTypeConversionAvailableException: No type
>>>> converter available to convert from type: null to the required type:
>>>> java.io.InputStream with value null
>>>> at
>>>> org.apache.camel.impl.converter.DefaultTypeConverter.mandatoryConvertTo(DefaultTypeConverter.java:126)
>>>> at
>>>> org.apache.camel.impl.MessageSupport.getMandatoryBody(MessageSupport.java:101)
>>>> ... 31 more
>>>> ============
>>>> Now, I can remedy the situation by a try-catch block or an interceptor,
>>>> but
>>>> in my opinion this should be considered a bug in camel and it should
>>>> silently ignore if there was nothing to fetch from the ftp.
>>>>
>>>> Bests,
>>>> Farhad Shiraz
>>>>
>>>> --
>>>> View this message in context:
>>>> http://old.nabble.com/camel-casts-exception-when-there-is-nothing-to-fetch-from-the-ftp-endpoint-tp26541447p26541447.html
>>>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>>>
>>>>
>>>
>>>
>>>
>>> --
>>> Claus Ibsen
>>> Apache Camel Committer
>>>
>>> Author of Camel in Action: http://www.manning.com/ibsen/
>>> Open Source Integration: http://fusesource.com
>>> Blog: http://davsclaus.blogspot.com/
>>> Twitter: http://twitter.com/davsclaus
>>>
>>>
>>
>> --
>> View this message in context:
>> http://old.nabble.com/camel-casts-exception-when-there-is-nothing-to-fetch-from-the-ftp-endpoint-tp26541447p26541650.html
>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>
>>
>
>
>
> --
> Claus Ibsen
> Apache Camel Committer
>
> Author of Camel in Action: http://www.manning.com/ibsen/
> Open Source Integration: http://fusesource.com
> Blog: http://davsclaus.blogspot.com/
> Twitter: http://twitter.com/davsclaus
>
>
--
View this message in context: http://old.nabble.com/camel-casts-exception-when-there-is-nothing-to-fetch-from-the-ftp-endpoint-tp26541447p26541877.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: camel casts exception when there is nothing to fetch from the ftp
endpoint
Posted by Claus Ibsen <cl...@gmail.com>.
On Fri, Nov 27, 2009 at 1:12 PM, shirazi <me...@farhad.eu> wrote:
>
> Hi,
> Ok, I guess I should rather provide an AggregationStrategy.
> The reason why I use a timer is that when I use the ftp endpoint, it starts
> to fetch everything that is on the ftp in the first call and that's not what
> i want, because I'm processing files over 3 Gigs and there are many of them.
> Then I figured that by using a timer, i could have ftp to download one file
> at a time!!
> What I basically need is to somehow tell the ftp, how many files I want to
> fetch at a time. This is probably doable now in camel-2.1 by using the
> "ThrottlingInflightRoutePolicy", which i havn't tried yet.
> Do you have any suggestions?
Yeah the FTP component extends the File component so you got all the
options from the file component as well
http://camel.apache.org/file2.html
And there is a maxMessagesPerPoll you can use to tell how many files
you want at most to poll. For example = 1 in your use.
And that option is avail in 2.0 also.
In fact all these are as they are batch polling consumers
http://camel.apache.org/batch-consumer.html
>
> Bests,
> Farhad S.
>
>
>
> Claus Ibsen-2 wrote:
>>
>> Hi
>>
>> BTW why are you using a timer -> ftp when the ftp have a build in timer?
>>
>>
>>
>> On Fri, Nov 27, 2009 at 12:55 PM, shirazi <me...@farhad.eu> wrote:
>>>
>>> Hi,
>>> Please consider the following route:
>>>
>>> from("timer://someTrigger?delay=30000&fixedRate=true&period=300000").
>>> pollEnrich("ftp://cmoreFtp?binary=true&localWorkDirectory=/tmp&idempotent=true&idempotentRepository=#caStreams",
>>> 10000).
>>> to("file:///tmp/vw/cmore?tempPrefix=.inprogress");
>>>
>>> The pollEnrich cast a NullPointerException in camel-2.0, if there is
>>> nothing
>>> to fetch from the ftp. I got rid of the exception by a
>>> "try-catch(NullpointerExcpetion.class).stop()" around the route. Now in
>>> camel-2.1-SNAPSHOT, I get the following exception on the "to" endpoint:
>>> =================
>>> 12:37:06,899 INFO [Tracer] >>> -->
>>> file:///tmp/vw/cmore?tempPrefix=.inprogress_, Pattern:InOnly,
>>> Headers:{firedTime=Fri Nov 27 12:36:56 CET 2009},
>>> Exception:org.apache.camel.component.file.GenericFileOperationFailedException:
>>> Cannot store file:
>>> /tmp/vw/cmore/.inprogress_02216f38-8ffc-4aec-b638-27229d857075
>>> 12:37:06,899 INFO [Tracer] >>> -->
>>> file:///tmp/vw/cmore?tempPrefix=.inprogress_, Pattern:InOnly,
>>> Headers:{firedTime=Fri Nov 27 12:36:56 CET 2009},
>>> Exception:org.apache.camel.component.file.GenericFileOperationFailedException:
>>> Cannot store file:
>>> /tmp/vw/cmore/.inprogress_02216f38-8ffc-4aec-b638-27229d857075
>>> 12:37:06,899 ERROR [TimerConsumer] Cannot store file:
>>> /tmp/vw/cmore/.inprogress_02216f38-8ffc-4aec-b638-27229d857075
>>> org.apache.camel.component.file.GenericFileOperationFailedException:
>>> Cannot
>>> store file:
>>> /tmp/vw/cmore/.inprogress_02216f38-8ffc-4aec-b638-27229d857075
>>> at
>>> org.apache.camel.component.file.FileOperations.storeFile(FileOperations.java:204)
>>> at
>>> org.apache.camel.component.file.GenericFileProducer.writeFile(GenericFileProducer.java:184)
>>> at
>>> org.apache.camel.component.file.GenericFileProducer.processExchange(GenericFileProducer.java:126)
>>> at
>>> org.apache.camel.component.file.GenericFileProducer.process(GenericFileProducer.java:57)
>>> at
>>> org.apache.camel.processor.SendProcessor$1.doInProducer(SendProcessor.java:97)
>>> at
>>> org.apache.camel.processor.SendProcessor$1.doInProducer(SendProcessor.java:95)
>>> at
>>> org.apache.camel.impl.ProducerCache.doInProducer(ProducerCache.java:146)
>>> at
>>> org.apache.camel.processor.SendProcessor.doProcess(SendProcessor.java:94)
>>> at
>>> org.apache.camel.processor.SendProcessor.process(SendProcessor.java:82)
>>> at
>>> org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:67)
>>> at
>>> org.apache.camel.processor.DelegateProcessor.processNext(DelegateProcessor.java:53)
>>> at
>>> org.apache.camel.processor.DelegateProcessor.proceed(DelegateProcessor.java:82)
>>> at
>>> org.apache.camel.processor.interceptor.TraceInterceptor.process(TraceInterceptor.java:162)
>>> at
>>> org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:67)
>>> at
>>> org.apache.camel.processor.DefaultChannel.process(DefaultChannel.java:206)
>>> at org.apache.camel.processor.Pipeline.process(Pipeline.java:74)
>>> at
>>> org.apache.camel.processor.TryProcessor.process(TryProcessor.java:63)
>>> at
>>> org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:67)
>>> at
>>> org.apache.camel.processor.DelegateProcessor.processNext(DelegateProcessor.java:53)
>>> at
>>> org.apache.camel.processor.DelegateProcessor.proceed(DelegateProcessor.java:82)
>>> at
>>> org.apache.camel.processor.interceptor.TraceInterceptor.process(TraceInterceptor.java:162)
>>> at
>>> org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:67)
>>> at
>>> org.apache.camel.processor.DefaultChannel.process(DefaultChannel.java:206)
>>> at
>>> org.apache.camel.processor.UnitOfWorkProcessor.processNext(UnitOfWorkProcessor.java:54)
>>> at
>>> org.apache.camel.processor.DelegateProcessor.process(DelegateProcessor.java:48)
>>> at
>>> org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:67)
>>> at
>>> org.apache.camel.component.timer.TimerConsumer.sendTimerExchange(TimerConsumer.java:103)
>>> at
>>> org.apache.camel.component.timer.TimerConsumer$1.run(TimerConsumer.java:50)
>>> at java.util.TimerThread.mainLoop(Timer.java:512)
>>> at java.util.TimerThread.run(Timer.java:462)
>>> Caused by: org.apache.camel.InvalidPayloadException: No body available of
>>> type: java.io.InputStream on: Message: [Body is null]. Caused by: No type
>>> converter available to convert from type: null to the required type:
>>> java.io.InputStream with value null on the exchange: Exchange[Message:
>>> [Body
>>> is null]]
>>> at
>>> org.apache.camel.impl.MessageSupport.getMandatoryBody(MessageSupport.java:103)
>>> at
>>> org.apache.camel.util.ExchangeHelper.getMandatoryInBody(ExchangeHelper.java:116)
>>> at
>>> org.apache.camel.component.file.FileOperations.storeFile(FileOperations.java:198)
>>> ... 29 more
>>> Caused by: org.apache.camel.NoTypeConversionAvailableException: No type
>>> converter available to convert from type: null to the required type:
>>> java.io.InputStream with value null
>>> at
>>> org.apache.camel.impl.converter.DefaultTypeConverter.mandatoryConvertTo(DefaultTypeConverter.java:126)
>>> at
>>> org.apache.camel.impl.MessageSupport.getMandatoryBody(MessageSupport.java:101)
>>> ... 31 more
>>> ============
>>> Now, I can remedy the situation by a try-catch block or an interceptor,
>>> but
>>> in my opinion this should be considered a bug in camel and it should
>>> silently ignore if there was nothing to fetch from the ftp.
>>>
>>> Bests,
>>> Farhad Shiraz
>>>
>>> --
>>> View this message in context:
>>> http://old.nabble.com/camel-casts-exception-when-there-is-nothing-to-fetch-from-the-ftp-endpoint-tp26541447p26541447.html
>>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>>
>>>
>>
>>
>>
>> --
>> Claus Ibsen
>> Apache Camel Committer
>>
>> Author of Camel in Action: http://www.manning.com/ibsen/
>> Open Source Integration: http://fusesource.com
>> Blog: http://davsclaus.blogspot.com/
>> Twitter: http://twitter.com/davsclaus
>>
>>
>
> --
> View this message in context: http://old.nabble.com/camel-casts-exception-when-there-is-nothing-to-fetch-from-the-ftp-endpoint-tp26541447p26541650.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>
>
--
Claus Ibsen
Apache Camel Committer
Author of Camel in Action: http://www.manning.com/ibsen/
Open Source Integration: http://fusesource.com
Blog: http://davsclaus.blogspot.com/
Twitter: http://twitter.com/davsclaus
Re: camel casts exception when there is nothing to fetch from the
ftp endpoint
Posted by shirazi <me...@farhad.eu>.
Hi,
Ok, I guess I should rather provide an AggregationStrategy.
The reason why I use a timer is that when I use the ftp endpoint, it starts
to fetch everything that is on the ftp in the first call and that's not what
i want, because I'm processing files over 3 Gigs and there are many of them.
Then I figured that by using a timer, i could have ftp to download one file
at a time!!
What I basically need is to somehow tell the ftp, how many files I want to
fetch at a time. This is probably doable now in camel-2.1 by using the
"ThrottlingInflightRoutePolicy", which i havn't tried yet.
Do you have any suggestions?
Bests,
Farhad S.
Claus Ibsen-2 wrote:
>
> Hi
>
> BTW why are you using a timer -> ftp when the ftp have a build in timer?
>
>
>
> On Fri, Nov 27, 2009 at 12:55 PM, shirazi <me...@farhad.eu> wrote:
>>
>> Hi,
>> Please consider the following route:
>>
>> from("timer://someTrigger?delay=30000&fixedRate=true&period=300000").
>> pollEnrich("ftp://cmoreFtp?binary=true&localWorkDirectory=/tmp&idempotent=true&idempotentRepository=#caStreams",
>> 10000).
>> to("file:///tmp/vw/cmore?tempPrefix=.inprogress");
>>
>> The pollEnrich cast a NullPointerException in camel-2.0, if there is
>> nothing
>> to fetch from the ftp. I got rid of the exception by a
>> "try-catch(NullpointerExcpetion.class).stop()" around the route. Now in
>> camel-2.1-SNAPSHOT, I get the following exception on the "to" endpoint:
>> =================
>> 12:37:06,899 INFO [Tracer] >>> -->
>> file:///tmp/vw/cmore?tempPrefix=.inprogress_, Pattern:InOnly,
>> Headers:{firedTime=Fri Nov 27 12:36:56 CET 2009},
>> Exception:org.apache.camel.component.file.GenericFileOperationFailedException:
>> Cannot store file:
>> /tmp/vw/cmore/.inprogress_02216f38-8ffc-4aec-b638-27229d857075
>> 12:37:06,899 INFO [Tracer] >>> -->
>> file:///tmp/vw/cmore?tempPrefix=.inprogress_, Pattern:InOnly,
>> Headers:{firedTime=Fri Nov 27 12:36:56 CET 2009},
>> Exception:org.apache.camel.component.file.GenericFileOperationFailedException:
>> Cannot store file:
>> /tmp/vw/cmore/.inprogress_02216f38-8ffc-4aec-b638-27229d857075
>> 12:37:06,899 ERROR [TimerConsumer] Cannot store file:
>> /tmp/vw/cmore/.inprogress_02216f38-8ffc-4aec-b638-27229d857075
>> org.apache.camel.component.file.GenericFileOperationFailedException:
>> Cannot
>> store file:
>> /tmp/vw/cmore/.inprogress_02216f38-8ffc-4aec-b638-27229d857075
>> at
>> org.apache.camel.component.file.FileOperations.storeFile(FileOperations.java:204)
>> at
>> org.apache.camel.component.file.GenericFileProducer.writeFile(GenericFileProducer.java:184)
>> at
>> org.apache.camel.component.file.GenericFileProducer.processExchange(GenericFileProducer.java:126)
>> at
>> org.apache.camel.component.file.GenericFileProducer.process(GenericFileProducer.java:57)
>> at
>> org.apache.camel.processor.SendProcessor$1.doInProducer(SendProcessor.java:97)
>> at
>> org.apache.camel.processor.SendProcessor$1.doInProducer(SendProcessor.java:95)
>> at
>> org.apache.camel.impl.ProducerCache.doInProducer(ProducerCache.java:146)
>> at
>> org.apache.camel.processor.SendProcessor.doProcess(SendProcessor.java:94)
>> at
>> org.apache.camel.processor.SendProcessor.process(SendProcessor.java:82)
>> at
>> org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:67)
>> at
>> org.apache.camel.processor.DelegateProcessor.processNext(DelegateProcessor.java:53)
>> at
>> org.apache.camel.processor.DelegateProcessor.proceed(DelegateProcessor.java:82)
>> at
>> org.apache.camel.processor.interceptor.TraceInterceptor.process(TraceInterceptor.java:162)
>> at
>> org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:67)
>> at
>> org.apache.camel.processor.DefaultChannel.process(DefaultChannel.java:206)
>> at org.apache.camel.processor.Pipeline.process(Pipeline.java:74)
>> at
>> org.apache.camel.processor.TryProcessor.process(TryProcessor.java:63)
>> at
>> org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:67)
>> at
>> org.apache.camel.processor.DelegateProcessor.processNext(DelegateProcessor.java:53)
>> at
>> org.apache.camel.processor.DelegateProcessor.proceed(DelegateProcessor.java:82)
>> at
>> org.apache.camel.processor.interceptor.TraceInterceptor.process(TraceInterceptor.java:162)
>> at
>> org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:67)
>> at
>> org.apache.camel.processor.DefaultChannel.process(DefaultChannel.java:206)
>> at
>> org.apache.camel.processor.UnitOfWorkProcessor.processNext(UnitOfWorkProcessor.java:54)
>> at
>> org.apache.camel.processor.DelegateProcessor.process(DelegateProcessor.java:48)
>> at
>> org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:67)
>> at
>> org.apache.camel.component.timer.TimerConsumer.sendTimerExchange(TimerConsumer.java:103)
>> at
>> org.apache.camel.component.timer.TimerConsumer$1.run(TimerConsumer.java:50)
>> at java.util.TimerThread.mainLoop(Timer.java:512)
>> at java.util.TimerThread.run(Timer.java:462)
>> Caused by: org.apache.camel.InvalidPayloadException: No body available of
>> type: java.io.InputStream on: Message: [Body is null]. Caused by: No type
>> converter available to convert from type: null to the required type:
>> java.io.InputStream with value null on the exchange: Exchange[Message:
>> [Body
>> is null]]
>> at
>> org.apache.camel.impl.MessageSupport.getMandatoryBody(MessageSupport.java:103)
>> at
>> org.apache.camel.util.ExchangeHelper.getMandatoryInBody(ExchangeHelper.java:116)
>> at
>> org.apache.camel.component.file.FileOperations.storeFile(FileOperations.java:198)
>> ... 29 more
>> Caused by: org.apache.camel.NoTypeConversionAvailableException: No type
>> converter available to convert from type: null to the required type:
>> java.io.InputStream with value null
>> at
>> org.apache.camel.impl.converter.DefaultTypeConverter.mandatoryConvertTo(DefaultTypeConverter.java:126)
>> at
>> org.apache.camel.impl.MessageSupport.getMandatoryBody(MessageSupport.java:101)
>> ... 31 more
>> ============
>> Now, I can remedy the situation by a try-catch block or an interceptor,
>> but
>> in my opinion this should be considered a bug in camel and it should
>> silently ignore if there was nothing to fetch from the ftp.
>>
>> Bests,
>> Farhad Shiraz
>>
>> --
>> View this message in context:
>> http://old.nabble.com/camel-casts-exception-when-there-is-nothing-to-fetch-from-the-ftp-endpoint-tp26541447p26541447.html
>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>
>>
>
>
>
> --
> Claus Ibsen
> Apache Camel Committer
>
> Author of Camel in Action: http://www.manning.com/ibsen/
> Open Source Integration: http://fusesource.com
> Blog: http://davsclaus.blogspot.com/
> Twitter: http://twitter.com/davsclaus
>
>
--
View this message in context: http://old.nabble.com/camel-casts-exception-when-there-is-nothing-to-fetch-from-the-ftp-endpoint-tp26541447p26541650.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: camel casts exception when there is nothing to fetch from the ftp
endpoint
Posted by Claus Ibsen <cl...@gmail.com>.
Hi
BTW why are you using a timer -> ftp when the ftp have a build in timer?
On Fri, Nov 27, 2009 at 12:55 PM, shirazi <me...@farhad.eu> wrote:
>
> Hi,
> Please consider the following route:
>
> from("timer://someTrigger?delay=30000&fixedRate=true&period=300000").
> pollEnrich("ftp://cmoreFtp?binary=true&localWorkDirectory=/tmp&idempotent=true&idempotentRepository=#caStreams",
> 10000).
> to("file:///tmp/vw/cmore?tempPrefix=.inprogress");
>
> The pollEnrich cast a NullPointerException in camel-2.0, if there is nothing
> to fetch from the ftp. I got rid of the exception by a
> "try-catch(NullpointerExcpetion.class).stop()" around the route. Now in
> camel-2.1-SNAPSHOT, I get the following exception on the "to" endpoint:
> =================
> 12:37:06,899 INFO [Tracer] >>> -->
> file:///tmp/vw/cmore?tempPrefix=.inprogress_, Pattern:InOnly,
> Headers:{firedTime=Fri Nov 27 12:36:56 CET 2009},
> Exception:org.apache.camel.component.file.GenericFileOperationFailedException:
> Cannot store file:
> /tmp/vw/cmore/.inprogress_02216f38-8ffc-4aec-b638-27229d857075
> 12:37:06,899 INFO [Tracer] >>> -->
> file:///tmp/vw/cmore?tempPrefix=.inprogress_, Pattern:InOnly,
> Headers:{firedTime=Fri Nov 27 12:36:56 CET 2009},
> Exception:org.apache.camel.component.file.GenericFileOperationFailedException:
> Cannot store file:
> /tmp/vw/cmore/.inprogress_02216f38-8ffc-4aec-b638-27229d857075
> 12:37:06,899 ERROR [TimerConsumer] Cannot store file:
> /tmp/vw/cmore/.inprogress_02216f38-8ffc-4aec-b638-27229d857075
> org.apache.camel.component.file.GenericFileOperationFailedException: Cannot
> store file: /tmp/vw/cmore/.inprogress_02216f38-8ffc-4aec-b638-27229d857075
> at
> org.apache.camel.component.file.FileOperations.storeFile(FileOperations.java:204)
> at
> org.apache.camel.component.file.GenericFileProducer.writeFile(GenericFileProducer.java:184)
> at
> org.apache.camel.component.file.GenericFileProducer.processExchange(GenericFileProducer.java:126)
> at
> org.apache.camel.component.file.GenericFileProducer.process(GenericFileProducer.java:57)
> at
> org.apache.camel.processor.SendProcessor$1.doInProducer(SendProcessor.java:97)
> at
> org.apache.camel.processor.SendProcessor$1.doInProducer(SendProcessor.java:95)
> at org.apache.camel.impl.ProducerCache.doInProducer(ProducerCache.java:146)
> at
> org.apache.camel.processor.SendProcessor.doProcess(SendProcessor.java:94)
> at org.apache.camel.processor.SendProcessor.process(SendProcessor.java:82)
> at
> org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:67)
> at
> org.apache.camel.processor.DelegateProcessor.processNext(DelegateProcessor.java:53)
> at
> org.apache.camel.processor.DelegateProcessor.proceed(DelegateProcessor.java:82)
> at
> org.apache.camel.processor.interceptor.TraceInterceptor.process(TraceInterceptor.java:162)
> at
> org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:67)
> at
> org.apache.camel.processor.DefaultChannel.process(DefaultChannel.java:206)
> at org.apache.camel.processor.Pipeline.process(Pipeline.java:74)
> at org.apache.camel.processor.TryProcessor.process(TryProcessor.java:63)
> at
> org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:67)
> at
> org.apache.camel.processor.DelegateProcessor.processNext(DelegateProcessor.java:53)
> at
> org.apache.camel.processor.DelegateProcessor.proceed(DelegateProcessor.java:82)
> at
> org.apache.camel.processor.interceptor.TraceInterceptor.process(TraceInterceptor.java:162)
> at
> org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:67)
> at
> org.apache.camel.processor.DefaultChannel.process(DefaultChannel.java:206)
> at
> org.apache.camel.processor.UnitOfWorkProcessor.processNext(UnitOfWorkProcessor.java:54)
> at
> org.apache.camel.processor.DelegateProcessor.process(DelegateProcessor.java:48)
> at
> org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:67)
> at
> org.apache.camel.component.timer.TimerConsumer.sendTimerExchange(TimerConsumer.java:103)
> at
> org.apache.camel.component.timer.TimerConsumer$1.run(TimerConsumer.java:50)
> at java.util.TimerThread.mainLoop(Timer.java:512)
> at java.util.TimerThread.run(Timer.java:462)
> Caused by: org.apache.camel.InvalidPayloadException: No body available of
> type: java.io.InputStream on: Message: [Body is null]. Caused by: No type
> converter available to convert from type: null to the required type:
> java.io.InputStream with value null on the exchange: Exchange[Message: [Body
> is null]]
> at
> org.apache.camel.impl.MessageSupport.getMandatoryBody(MessageSupport.java:103)
> at
> org.apache.camel.util.ExchangeHelper.getMandatoryInBody(ExchangeHelper.java:116)
> at
> org.apache.camel.component.file.FileOperations.storeFile(FileOperations.java:198)
> ... 29 more
> Caused by: org.apache.camel.NoTypeConversionAvailableException: No type
> converter available to convert from type: null to the required type:
> java.io.InputStream with value null
> at
> org.apache.camel.impl.converter.DefaultTypeConverter.mandatoryConvertTo(DefaultTypeConverter.java:126)
> at
> org.apache.camel.impl.MessageSupport.getMandatoryBody(MessageSupport.java:101)
> ... 31 more
> ============
> Now, I can remedy the situation by a try-catch block or an interceptor, but
> in my opinion this should be considered a bug in camel and it should
> silently ignore if there was nothing to fetch from the ftp.
>
> Bests,
> Farhad Shiraz
>
> --
> View this message in context: http://old.nabble.com/camel-casts-exception-when-there-is-nothing-to-fetch-from-the-ftp-endpoint-tp26541447p26541447.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>
>
--
Claus Ibsen
Apache Camel Committer
Author of Camel in Action: http://www.manning.com/ibsen/
Open Source Integration: http://fusesource.com
Blog: http://davsclaus.blogspot.com/
Twitter: http://twitter.com/davsclaus