You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@camel.apache.org by Rolf Sommer Poulsen <rs...@bankdata.dk.INVALID> on 2025/03/06 06:58:32 UTC
Upper file limit in SMB protocol ?
Hi
I was wondering if there is a upper bound on the file size the camel-smb component I capable of handling. I have set up a route, that streamdownload a file from a windows server using the smb-component, and it works fine with normal sized files, however when testing on a very large file (~8gb) I get and error about an array that has grown to large even before my route is being called.
That made me wonder if there is a upper bound on the size of the files that the smb-component can handle and if that size is around INTEGER.MAX_VALUE bytes ?
Venlig hilsen / Best regards
Rolf Sommer Poulsen
SV: Upper file limit in SMB protocol ?
Posted by Rolf Sommer Poulsen <rs...@bankdata.dk.INVALID>.
Brilliant. Thank you very much Claus.
Venlig hilsen / Best regards
Rolf Sommer Poulsen
Senior Software Developer,
FRI Integration
+45 6012 5444 | +45 7924 2811
rsp@bankdata.dk
BANKDATA | Development Centre Silkeborg | Stagehøjvej 30 | 8600 ;Silkeborg
-----Oprindelig meddelelse-----
Fra: Claus Ibsen <cl...@gmail.com>
Sendt: 6. marts 2025 09:28
Til: users@camel.apache.org
Emne: Re: Upper file limit in SMB protocol ?
[Du får ikke ofte mails fra claus.ibsen@gmail.com. Få mere at vide om, hvorfor dette er vigtigt, på https://aka.ms/LearnAboutSenderIdentification ]
Hi
Ah okay read about stream caching
https://camel.apache.org/manual/stream-caching.html
You can turn that off or enable its spool to disk option
On Thu, Mar 6, 2025 at 8:17 AM Rolf Sommer Poulsen <rs...@bankdata.dk.invalid>
wrote:
> Hi
>
> I am currently using the 4.10.1 release and get the following stacktrace:
>
> java.lang.OutOfMemoryError: Required array length 2147483639 + 9 is
> too large
> at
> java.base/jdk.internal.util.ArraysSupport.hugeLength(ArraysSupport.jav
> a:649)
> ~[na:na]
> at
> java.base/jdk.internal.util.ArraysSupport.newLength(ArraysSupport.java
> :642)
> ~[na:na]
> at
> java.base/java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutput
> Stream.java:100)
> ~[na:na]
> at
> java.base/java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.ja
> va:130)
> ~[na:na]
> at
> org.apache.camel.converter.stream.CachedOutputStream.write(CachedOutpu
> tStream.java:108)
> ~[camel-support-4.10.1.jar:4.10.1]
> at
> java.base/java.io.InputStream.transferTo(InputStream.java:783)
> ~[na:na]
> at org.apache.camel.util.IOHelper.copy(IOHelper.java:165)
> ~[camel-util-4.10.1.jar:4.10.1]
> at
> org.apache.camel.util.IOHelper.copyAndCloseInput(IOHelper.java:283)
> ~[camel-util-4.10.1.jar:4.10.1]
> at
> org.apache.camel.converter.stream.StreamCacheConverter.convertToStream
> Cache(StreamCacheConverter.java:54)
> ~[camel-support-4.10.1.jar:4.10.1]
> at
> org.apache.camel.converter.stream.StreamCacheBulkConverterLoader.doCon
> vertTo(StreamCacheBulkConverterLoader.java:83)
> ~[camel-support-4.10.1.jar:4.10.1]
> at
> org.apache.camel.converter.stream.StreamCacheBulkConverterLoader.conve
> rtTo(StreamCacheBulkConverterLoader.java:56)
> ~[camel-support-4.10.1.jar:4.10.1]
> at
> org.apache.camel.spi.BulkTypeConverters.convertTo(BulkTypeConverters.j
> ava:122)
> ~[camel-api-4.10.1.jar:4.10.1]
> at
> org.apache.camel.component.file.GenericFileConverter.convertTo(Generic
> FileConverter.java:106)
> ~[camel-file-4.10.1.jar:4.10.1]
> at
> org.apache.camel.component.file.GenericFileConverterLoader.lambda$regi
> sterFallbackConverters$4(GenericFileConverterLoader.java:63)
> ~[camel-file-4.10.1.jar:4.10.1]
> at
> org.apache.camel.support.SimpleTypeConverter.convertTo(SimpleTypeConve
> rter.java:101)
> ~[camel-support-4.10.1.jar:4.10.1]
> at
> org.apache.camel.impl.converter.CoreTypeConverterRegistry.doConvert(Co
> reTypeConverterRegistry.java:463)
> ~[camel-base-4.10.1.jar:4.10.1]
> at
> org.apache.camel.impl.converter.CoreTypeConverterRegistry.tryFallback(
> CoreTypeConverterRegistry.java:432)
> ~[camel-base-4.10.1.jar:4.10.1]
> at
> org.apache.camel.impl.converter.CoreTypeConverterRegistry.doConvertTo(
> CoreTypeConverterRegistry.java:381)
> ~[camel-base-4.10.1.jar:4.10.1]
> at
> org.apache.camel.impl.converter.CoreTypeConverterRegistry.doConvertToA
> ndStat(CoreTypeConverterRegistry.java:269)
> ~[camel-base-4.10.1.jar:4.10.1]
> at
> org.apache.camel.impl.converter.CoreTypeConverterRegistry.convertTo(Co
> reTypeConverterRegistry.java:164)
> ~[camel-base-4.10.1.jar:4.10.1]
> at
> org.apache.camel.impl.engine.DefaultStreamCachingStrategy.doCache(Defa
> ultStreamCachingStrategy.java:278)
> ~[camel-base-engine-4.10.1.jar:4.10.1]
> at
> org.apache.camel.impl.engine.DefaultStreamCachingStrategy.cache(Defaul
> tStreamCachingStrategy.java:255)
> ~[camel-base-engine-4.10.1.jar:4.10.1]
> at
> org.apache.camel.impl.engine.StreamCachingHelper.tryStreamCache(Stream
> CachingHelper.java:67) ~[camel-base-engine-4.10.1.jar:4.10.1]
> at
> org.apache.camel.impl.engine.StreamCachingHelper.convertToStreamCache(
> StreamCachingHelper.java:54) ~[camel-base-engine-4.10.1.jar:4.10.1]
> at
> org.apache.camel.impl.engine.CamelInternalProcessor$StreamCachingAdvic
> e.before(CamelInternalProcessor.java:1010)
> ~[camel-base-engine-4.10.1.jar:4.10.1]
> at
> org.apache.camel.impl.engine.CamelInternalProcessor$StreamCachingAdvic
> e.before(CamelInternalProcessor.java:1000)
> ~[camel-base-engine-4.10.1.jar:4.10.1]
> at
> org.apache.camel.impl.engine.CamelInternalProcessor.process(CamelInter
> nalProcessor.java:310) ~[camel-base-engine-4.10.1.jar:4.10.1]
> at
> org.apache.camel.processor.Pipeline$PipelineTask.run(Pipeline.java:102
> ) ~[camel-core-processor-4.10.1.jar:4.10.1]
> at
> org.apache.camel.impl.engine.DefaultReactiveExecutor$Worker.doRun(Defa
> ultReactiveExecutor.java:199) ~[camel-base-engine-4.10.1.jar:4.10.1]
> at
> org.apache.camel.impl.engine.DefaultReactiveExecutor$Worker.executeRea
> ctiveWork(DefaultReactiveExecutor.java:189)
> ~[camel-base-engine-4.10.1.jar:4.10.1]
> at
> org.apache.camel.impl.engine.DefaultReactiveExecutor$Worker.tryExecute
> ReactiveWork(DefaultReactiveExecutor.java:166)
> ~[camel-base-engine-4.10.1.jar:4.10.1]
> at
> org.apache.camel.impl.engine.DefaultReactiveExecutor$Worker.schedule(D
> efaultReactiveExecutor.java:148)
> ~[camel-base-engine-4.10.1.jar:4.10.1]
> at
> org.apache.camel.impl.engine.DefaultReactiveExecutor.scheduleMain(Defa
> ultReactiveExecutor.java:59) ~[camel-base-engine-4.10.1.jar:4.10.1]
> at
> org.apache.camel.processor.Pipeline.process(Pipeline.java:163)
> ~[camel-core-processor-4.10.1.jar:4.10.1]
> at
> org.apache.camel.impl.engine.CamelInternalProcessor.processNonTransact
> ed(CamelInternalProcessor.java:347)
> ~[camel-base-engine-4.10.1.jar:4.10.1]
> at
> org.apache.camel.impl.engine.CamelInternalProcessor.process(CamelInter
> nalProcessor.java:323) ~[camel-base-engine-4.10.1.jar:4.10.1]
> at
> org.apache.camel.component.file.GenericFileConsumer.processExchange(Ge
> nericFileConsumer.java:459)
> ~[camel-file-4.10.1.jar:4.10.1]
> at
> org.apache.camel.component.smb.SmbConsumer.processExchange(SmbConsumer
> .java:309)
> ~[camel-smb-4.10.1.jar:4.10.1]
> at
> org.apache.camel.component.file.GenericFileConsumer.processBatch(Gener
> icFileConsumer.java:245)
> ~[camel-file-4.10.1.jar:4.10.1]
> at
> org.apache.camel.component.file.GenericFileConsumer.poll(GenericFileCo
> nsumer.java:206)
> ~[camel-file-4.10.1.jar:4.10.1]
> at
> org.apache.camel.support.ScheduledPollConsumer.doRun(ScheduledPollCons
> umer.java:208)
> ~[camel-support-4.10.1.jar:4.10.1]
> at
> org.apache.camel.support.ScheduledPollConsumer.run(ScheduledPollConsum
> er.java:119)
> ~[camel-support-4.10.1.jar:4.10.1]
> at
> org.springframework.scheduling.support.DelegatingErrorHandlingRunnable
> .run(DelegatingErrorHandlingRunnable.java:54)
> ~[spring-context-6.2.2.jar:6.2.2]
> at
> org.springframework.scheduling.concurrent.ReschedulingRunnable.run(Res
> chedulingRunnable.java:96)
> ~[spring-context-6.2.2.jar:6.2.2]
> at
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executor
> s.java:539)
> ~[na:na]
> at
> java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[na:na]
> at
> java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFu
> tureTask.run(ScheduledThreadPoolExecutor.java:304)
> ~[na:na]
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool
> Executor.java:1136)
> ~[na:na]
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo
> lExecutor.java:635)
> ~[na:na]
> at java.base/java.lang.Thread.run(Thread.java:840) ~[na:na]
>
>
> Venlig hilsen / Best regards
>
> Rolf Sommer Poulsen
> Senior Software Developer,
> FRI Integration
> +45 6012 5444 | +45 7924 2811
> rsp@bankdata.dk
> BANKDATA | Development Centre Silkeborg | Stagehøjvej 30 | 8600
> ;Silkeborg -----Oprindelig meddelelse-----
> Fra: Claus Ibsen <cl...@gmail.com>
> Sendt: 6. marts 2025 08:09
> Til: users@camel.apache.org
> Emne: Re: Upper file limit in SMB protocol ?
>
> [Du får ikke ofte mails fra claus.ibsen@gmail.com. Få mere at vide om,
> hvorfor dette er vigtigt, på
> https://aka.ms/LearnAboutSenderIdentification
> ]
>
> Hi
>
> I am not aware of a limit. But can you try with latest 4.10.1 release
> and also if you see any exception then report the stacktrace
>
> On Thu, Mar 6, 2025 at 8:00 AM Rolf Sommer Poulsen
> <rsp@bankdata.dk.invalid
> >
> wrote:
>
> > Hi
> >
> > I was wondering if there is a upper bound on the file size the
> > camel-smb component I capable of handling. I have set up a route,
> > that streamdownload a file from a windows server using the
> > smb-component, and it works fine with normal sized files, however
> > when testing on a very large file (~8gb) I get and error about an
> > array that has grown to large even before my route is being called.
> >
> > That made me wonder if there is a upper bound on the size of the
> > files that the smb-component can handle and if that size is around
> > INTEGER.MAX_VALUE bytes ?
> >
> >
> > Venlig hilsen / Best regards
> >
> > Rolf Sommer Poulsen
> >
> >
> >
> >
>
> --
> Claus Ibsen
> -----------------
> @davsclaus
> Camel in Action 2:
> https://www/.
> manning.com%2Fibsen2&data=05%7C02%7Crsp%40bankdata.dk%7Cec9462e067c845
> efda5d08dd5c88fbcd%7C1177168698fc4f1b8e0076c52159e8a6%7C0%7C0%7C638768
> 465582935182%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwL
> jAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%
> 7C&sdata=vQOPORJJ%2BZnXv6262%2FTVHbpoMFQrM%2FlWeCYh%2B%2Fsc30k%3D&rese
> rved=0
>
--
Claus Ibsen
-----------------
@davsclaus
Camel in Action 2: https://www.manning.com/ibsen2
Re: Upper file limit in SMB protocol ?
Posted by Claus Ibsen <cl...@gmail.com>.
Hi
Ah okay read about stream caching
https://camel.apache.org/manual/stream-caching.html
You can turn that off or enable its spool to disk option
On Thu, Mar 6, 2025 at 8:17 AM Rolf Sommer Poulsen <rs...@bankdata.dk.invalid>
wrote:
> Hi
>
> I am currently using the 4.10.1 release and get the following stacktrace:
>
> java.lang.OutOfMemoryError: Required array length 2147483639 + 9 is too
> large
> at
> java.base/jdk.internal.util.ArraysSupport.hugeLength(ArraysSupport.java:649)
> ~[na:na]
> at
> java.base/jdk.internal.util.ArraysSupport.newLength(ArraysSupport.java:642)
> ~[na:na]
> at java.base/java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:100)
> ~[na:na]
> at java.base/java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:130)
> ~[na:na]
> at
> org.apache.camel.converter.stream.CachedOutputStream.write(CachedOutputStream.java:108)
> ~[camel-support-4.10.1.jar:4.10.1]
> at java.base/java.io.InputStream.transferTo(InputStream.java:783)
> ~[na:na]
> at org.apache.camel.util.IOHelper.copy(IOHelper.java:165)
> ~[camel-util-4.10.1.jar:4.10.1]
> at
> org.apache.camel.util.IOHelper.copyAndCloseInput(IOHelper.java:283)
> ~[camel-util-4.10.1.jar:4.10.1]
> at
> org.apache.camel.converter.stream.StreamCacheConverter.convertToStreamCache(StreamCacheConverter.java:54)
> ~[camel-support-4.10.1.jar:4.10.1]
> at
> org.apache.camel.converter.stream.StreamCacheBulkConverterLoader.doConvertTo(StreamCacheBulkConverterLoader.java:83)
> ~[camel-support-4.10.1.jar:4.10.1]
> at
> org.apache.camel.converter.stream.StreamCacheBulkConverterLoader.convertTo(StreamCacheBulkConverterLoader.java:56)
> ~[camel-support-4.10.1.jar:4.10.1]
> at
> org.apache.camel.spi.BulkTypeConverters.convertTo(BulkTypeConverters.java:122)
> ~[camel-api-4.10.1.jar:4.10.1]
> at
> org.apache.camel.component.file.GenericFileConverter.convertTo(GenericFileConverter.java:106)
> ~[camel-file-4.10.1.jar:4.10.1]
> at
> org.apache.camel.component.file.GenericFileConverterLoader.lambda$registerFallbackConverters$4(GenericFileConverterLoader.java:63)
> ~[camel-file-4.10.1.jar:4.10.1]
> at
> org.apache.camel.support.SimpleTypeConverter.convertTo(SimpleTypeConverter.java:101)
> ~[camel-support-4.10.1.jar:4.10.1]
> at
> org.apache.camel.impl.converter.CoreTypeConverterRegistry.doConvert(CoreTypeConverterRegistry.java:463)
> ~[camel-base-4.10.1.jar:4.10.1]
> at
> org.apache.camel.impl.converter.CoreTypeConverterRegistry.tryFallback(CoreTypeConverterRegistry.java:432)
> ~[camel-base-4.10.1.jar:4.10.1]
> at
> org.apache.camel.impl.converter.CoreTypeConverterRegistry.doConvertTo(CoreTypeConverterRegistry.java:381)
> ~[camel-base-4.10.1.jar:4.10.1]
> at
> org.apache.camel.impl.converter.CoreTypeConverterRegistry.doConvertToAndStat(CoreTypeConverterRegistry.java:269)
> ~[camel-base-4.10.1.jar:4.10.1]
> at
> org.apache.camel.impl.converter.CoreTypeConverterRegistry.convertTo(CoreTypeConverterRegistry.java:164)
> ~[camel-base-4.10.1.jar:4.10.1]
> at
> org.apache.camel.impl.engine.DefaultStreamCachingStrategy.doCache(DefaultStreamCachingStrategy.java:278)
> ~[camel-base-engine-4.10.1.jar:4.10.1]
> at
> org.apache.camel.impl.engine.DefaultStreamCachingStrategy.cache(DefaultStreamCachingStrategy.java:255)
> ~[camel-base-engine-4.10.1.jar:4.10.1]
> at
> org.apache.camel.impl.engine.StreamCachingHelper.tryStreamCache(StreamCachingHelper.java:67)
> ~[camel-base-engine-4.10.1.jar:4.10.1]
> at
> org.apache.camel.impl.engine.StreamCachingHelper.convertToStreamCache(StreamCachingHelper.java:54)
> ~[camel-base-engine-4.10.1.jar:4.10.1]
> at
> org.apache.camel.impl.engine.CamelInternalProcessor$StreamCachingAdvice.before(CamelInternalProcessor.java:1010)
> ~[camel-base-engine-4.10.1.jar:4.10.1]
> at
> org.apache.camel.impl.engine.CamelInternalProcessor$StreamCachingAdvice.before(CamelInternalProcessor.java:1000)
> ~[camel-base-engine-4.10.1.jar:4.10.1]
> at
> org.apache.camel.impl.engine.CamelInternalProcessor.process(CamelInternalProcessor.java:310)
> ~[camel-base-engine-4.10.1.jar:4.10.1]
> at
> org.apache.camel.processor.Pipeline$PipelineTask.run(Pipeline.java:102)
> ~[camel-core-processor-4.10.1.jar:4.10.1]
> at
> org.apache.camel.impl.engine.DefaultReactiveExecutor$Worker.doRun(DefaultReactiveExecutor.java:199)
> ~[camel-base-engine-4.10.1.jar:4.10.1]
> at
> org.apache.camel.impl.engine.DefaultReactiveExecutor$Worker.executeReactiveWork(DefaultReactiveExecutor.java:189)
> ~[camel-base-engine-4.10.1.jar:4.10.1]
> at
> org.apache.camel.impl.engine.DefaultReactiveExecutor$Worker.tryExecuteReactiveWork(DefaultReactiveExecutor.java:166)
> ~[camel-base-engine-4.10.1.jar:4.10.1]
> at
> org.apache.camel.impl.engine.DefaultReactiveExecutor$Worker.schedule(DefaultReactiveExecutor.java:148)
> ~[camel-base-engine-4.10.1.jar:4.10.1]
> at
> org.apache.camel.impl.engine.DefaultReactiveExecutor.scheduleMain(DefaultReactiveExecutor.java:59)
> ~[camel-base-engine-4.10.1.jar:4.10.1]
> at org.apache.camel.processor.Pipeline.process(Pipeline.java:163)
> ~[camel-core-processor-4.10.1.jar:4.10.1]
> at
> org.apache.camel.impl.engine.CamelInternalProcessor.processNonTransacted(CamelInternalProcessor.java:347)
> ~[camel-base-engine-4.10.1.jar:4.10.1]
> at
> org.apache.camel.impl.engine.CamelInternalProcessor.process(CamelInternalProcessor.java:323)
> ~[camel-base-engine-4.10.1.jar:4.10.1]
> at
> org.apache.camel.component.file.GenericFileConsumer.processExchange(GenericFileConsumer.java:459)
> ~[camel-file-4.10.1.jar:4.10.1]
> at
> org.apache.camel.component.smb.SmbConsumer.processExchange(SmbConsumer.java:309)
> ~[camel-smb-4.10.1.jar:4.10.1]
> at
> org.apache.camel.component.file.GenericFileConsumer.processBatch(GenericFileConsumer.java:245)
> ~[camel-file-4.10.1.jar:4.10.1]
> at
> org.apache.camel.component.file.GenericFileConsumer.poll(GenericFileConsumer.java:206)
> ~[camel-file-4.10.1.jar:4.10.1]
> at
> org.apache.camel.support.ScheduledPollConsumer.doRun(ScheduledPollConsumer.java:208)
> ~[camel-support-4.10.1.jar:4.10.1]
> at
> org.apache.camel.support.ScheduledPollConsumer.run(ScheduledPollConsumer.java:119)
> ~[camel-support-4.10.1.jar:4.10.1]
> at
> org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)
> ~[spring-context-6.2.2.jar:6.2.2]
> at
> org.springframework.scheduling.concurrent.ReschedulingRunnable.run(ReschedulingRunnable.java:96)
> ~[spring-context-6.2.2.jar:6.2.2]
> at
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
> ~[na:na]
> at
> java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[na:na]
> at
> java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
> ~[na:na]
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
> ~[na:na]
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
> ~[na:na]
> at java.base/java.lang.Thread.run(Thread.java:840) ~[na:na]
>
>
> Venlig hilsen / Best regards
>
> Rolf Sommer Poulsen
> Senior Software Developer,
> FRI Integration
> +45 6012 5444 | +45 7924 2811
> rsp@bankdata.dk
> BANKDATA | Development Centre Silkeborg | Stagehøjvej 30 | 8600
> ;Silkeborg
> -----Oprindelig meddelelse-----
> Fra: Claus Ibsen <cl...@gmail.com>
> Sendt: 6. marts 2025 08:09
> Til: users@camel.apache.org
> Emne: Re: Upper file limit in SMB protocol ?
>
> [Du får ikke ofte mails fra claus.ibsen@gmail.com. Få mere at vide om,
> hvorfor dette er vigtigt, på https://aka.ms/LearnAboutSenderIdentification
> ]
>
> Hi
>
> I am not aware of a limit. But can you try with latest 4.10.1 release and
> also if you see any exception then report the stacktrace
>
> On Thu, Mar 6, 2025 at 8:00 AM Rolf Sommer Poulsen <rsp@bankdata.dk.invalid
> >
> wrote:
>
> > Hi
> >
> > I was wondering if there is a upper bound on the file size the
> > camel-smb component I capable of handling. I have set up a route, that
> > streamdownload a file from a windows server using the smb-component,
> > and it works fine with normal sized files, however when testing on a
> > very large file (~8gb) I get and error about an array that has grown
> > to large even before my route is being called.
> >
> > That made me wonder if there is a upper bound on the size of the files
> > that the smb-component can handle and if that size is around
> > INTEGER.MAX_VALUE bytes ?
> >
> >
> > Venlig hilsen / Best regards
> >
> > Rolf Sommer Poulsen
> >
> >
> >
> >
>
> --
> Claus Ibsen
> -----------------
> @davsclaus
> Camel in Action 2: https://www.manning.com/ibsen2
>
--
Claus Ibsen
-----------------
@davsclaus
Camel in Action 2: https://www.manning.com/ibsen2
SV: Upper file limit in SMB protocol ?
Posted by Rolf Sommer Poulsen <rs...@bankdata.dk.INVALID>.
I guess the limit arises when using the bytearraystream. Is it possible to use some kind of buffered reader when one enables streamdownload ?
Venlig hilsen / Best regards
Rolf Sommer Poulsen
Senior Software Developer,
FRI Integration
+45 6012 5444 | +45 7924 2811
rsp@bankdata.dk
BANKDATA | Development Centre Silkeborg | Stagehøjvej 30 | 8600 ;Silkeborg
-----Oprindelig meddelelse-----
Fra: rsp@bankdata.dk <rs...@bankdata.dk> På vegne af Rolf Sommer Poulsen
Sendt: 6. marts 2025 08:16
Til: users@camel.apache.org
Emne: SV: Upper file limit in SMB protocol ?
[Du får ikke ofte mails fra rsp@bankdata.dk.invalid. Få mere at vide om, hvorfor dette er vigtigt, på https://aka.ms/LearnAboutSenderIdentification ]
Hi
I am currently using the 4.10.1 release and get the following stacktrace:
java.lang.OutOfMemoryError: Required array length 2147483639 + 9 is too large
at java.base/jdk.internal.util.ArraysSupport.hugeLength(ArraysSupport.java:649) ~[na:na]
at java.base/jdk.internal.util.ArraysSupport.newLength(ArraysSupport.java:642) ~[na:na]
at java.base/java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:100) ~[na:na]
at java.base/java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:130) ~[na:na]
at org.apache.camel.converter.stream.CachedOutputStream.write(CachedOutputStream.java:108) ~[camel-support-4.10.1.jar:4.10.1]
at java.base/java.io.InputStream.transferTo(InputStream.java:783) ~[na:na]
at org.apache.camel.util.IOHelper.copy(IOHelper.java:165) ~[camel-util-4.10.1.jar:4.10.1]
at org.apache.camel.util.IOHelper.copyAndCloseInput(IOHelper.java:283) ~[camel-util-4.10.1.jar:4.10.1]
at org.apache.camel.converter.stream.StreamCacheConverter.convertToStreamCache(StreamCacheConverter.java:54) ~[camel-support-4.10.1.jar:4.10.1]
at org.apache.camel.converter.stream.StreamCacheBulkConverterLoader.doConvertTo(StreamCacheBulkConverterLoader.java:83) ~[camel-support-4.10.1.jar:4.10.1]
at org.apache.camel.converter.stream.StreamCacheBulkConverterLoader.convertTo(StreamCacheBulkConverterLoader.java:56) ~[camel-support-4.10.1.jar:4.10.1]
at org.apache.camel.spi.BulkTypeConverters.convertTo(BulkTypeConverters.java:122) ~[camel-api-4.10.1.jar:4.10.1]
at org.apache.camel.component.file.GenericFileConverter.convertTo(GenericFileConverter.java:106) ~[camel-file-4.10.1.jar:4.10.1]
at org.apache.camel.component.file.GenericFileConverterLoader.lambda$registerFallbackConverters$4(GenericFileConverterLoader.java:63) ~[camel-file-4.10.1.jar:4.10.1]
at org.apache.camel.support.SimpleTypeConverter.convertTo(SimpleTypeConverter.java:101) ~[camel-support-4.10.1.jar:4.10.1]
at org.apache.camel.impl.converter.CoreTypeConverterRegistry.doConvert(CoreTypeConverterRegistry.java:463) ~[camel-base-4.10.1.jar:4.10.1]
at org.apache.camel.impl.converter.CoreTypeConverterRegistry.tryFallback(CoreTypeConverterRegistry.java:432) ~[camel-base-4.10.1.jar:4.10.1]
at org.apache.camel.impl.converter.CoreTypeConverterRegistry.doConvertTo(CoreTypeConverterRegistry.java:381) ~[camel-base-4.10.1.jar:4.10.1]
at org.apache.camel.impl.converter.CoreTypeConverterRegistry.doConvertToAndStat(CoreTypeConverterRegistry.java:269) ~[camel-base-4.10.1.jar:4.10.1]
at org.apache.camel.impl.converter.CoreTypeConverterRegistry.convertTo(CoreTypeConverterRegistry.java:164) ~[camel-base-4.10.1.jar:4.10.1]
at org.apache.camel.impl.engine.DefaultStreamCachingStrategy.doCache(DefaultStreamCachingStrategy.java:278) ~[camel-base-engine-4.10.1.jar:4.10.1]
at org.apache.camel.impl.engine.DefaultStreamCachingStrategy.cache(DefaultStreamCachingStrategy.java:255) ~[camel-base-engine-4.10.1.jar:4.10.1]
at org.apache.camel.impl.engine.StreamCachingHelper.tryStreamCache(StreamCachingHelper.java:67) ~[camel-base-engine-4.10.1.jar:4.10.1]
at org.apache.camel.impl.engine.StreamCachingHelper.convertToStreamCache(StreamCachingHelper.java:54) ~[camel-base-engine-4.10.1.jar:4.10.1]
at org.apache.camel.impl.engine.CamelInternalProcessor$StreamCachingAdvice.before(CamelInternalProcessor.java:1010) ~[camel-base-engine-4.10.1.jar:4.10.1]
at org.apache.camel.impl.engine.CamelInternalProcessor$StreamCachingAdvice.before(CamelInternalProcessor.java:1000) ~[camel-base-engine-4.10.1.jar:4.10.1]
at org.apache.camel.impl.engine.CamelInternalProcessor.process(CamelInternalProcessor.java:310) ~[camel-base-engine-4.10.1.jar:4.10.1]
at org.apache.camel.processor.Pipeline$PipelineTask.run(Pipeline.java:102) ~[camel-core-processor-4.10.1.jar:4.10.1]
at org.apache.camel.impl.engine.DefaultReactiveExecutor$Worker.doRun(DefaultReactiveExecutor.java:199) ~[camel-base-engine-4.10.1.jar:4.10.1]
at org.apache.camel.impl.engine.DefaultReactiveExecutor$Worker.executeReactiveWork(DefaultReactiveExecutor.java:189) ~[camel-base-engine-4.10.1.jar:4.10.1]
at org.apache.camel.impl.engine.DefaultReactiveExecutor$Worker.tryExecuteReactiveWork(DefaultReactiveExecutor.java:166) ~[camel-base-engine-4.10.1.jar:4.10.1]
at org.apache.camel.impl.engine.DefaultReactiveExecutor$Worker.schedule(DefaultReactiveExecutor.java:148) ~[camel-base-engine-4.10.1.jar:4.10.1]
at org.apache.camel.impl.engine.DefaultReactiveExecutor.scheduleMain(DefaultReactiveExecutor.java:59) ~[camel-base-engine-4.10.1.jar:4.10.1]
at org.apache.camel.processor.Pipeline.process(Pipeline.java:163) ~[camel-core-processor-4.10.1.jar:4.10.1]
at org.apache.camel.impl.engine.CamelInternalProcessor.processNonTransacted(CamelInternalProcessor.java:347) ~[camel-base-engine-4.10.1.jar:4.10.1]
at org.apache.camel.impl.engine.CamelInternalProcessor.process(CamelInternalProcessor.java:323) ~[camel-base-engine-4.10.1.jar:4.10.1]
at org.apache.camel.component.file.GenericFileConsumer.processExchange(GenericFileConsumer.java:459) ~[camel-file-4.10.1.jar:4.10.1]
at org.apache.camel.component.smb.SmbConsumer.processExchange(SmbConsumer.java:309) ~[camel-smb-4.10.1.jar:4.10.1]
at org.apache.camel.component.file.GenericFileConsumer.processBatch(GenericFileConsumer.java:245) ~[camel-file-4.10.1.jar:4.10.1]
at org.apache.camel.component.file.GenericFileConsumer.poll(GenericFileConsumer.java:206) ~[camel-file-4.10.1.jar:4.10.1]
at org.apache.camel.support.ScheduledPollConsumer.doRun(ScheduledPollConsumer.java:208) ~[camel-support-4.10.1.jar:4.10.1]
at org.apache.camel.support.ScheduledPollConsumer.run(ScheduledPollConsumer.java:119) ~[camel-support-4.10.1.jar:4.10.1]
at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) ~[spring-context-6.2.2.jar:6.2.2]
at org.springframework.scheduling.concurrent.ReschedulingRunnable.run(ReschedulingRunnable.java:96) ~[spring-context-6.2.2.jar:6.2.2]
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) ~[na:na]
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[na:na]
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) ~[na:na]
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[na:na]
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[na:na]
at java.base/java.lang.Thread.run(Thread.java:840) ~[na:na]
Venlig hilsen / Best regards
Rolf Sommer Poulsen
Senior Software Developer,
FRI Integration
+45 6012 5444 | +45 7924 2811
rsp@bankdata.dk
BANKDATA | Development Centre Silkeborg | Stagehøjvej 30 | 8600 ;Silkeborg -----Oprindelig meddelelse-----
Fra: Claus Ibsen <cl...@gmail.com>
Sendt: 6. marts 2025 08:09
Til: users@camel.apache.org
Emne: Re: Upper file limit in SMB protocol ?
[Du får ikke ofte mails fra claus.ibsen@gmail.com. Få mere at vide om, hvorfor dette er vigtigt, på https://aka.ms/LearnAboutSenderIdentification ]
Hi
I am not aware of a limit. But can you try with latest 4.10.1 release and also if you see any exception then report the stacktrace
On Thu, Mar 6, 2025 at 8:00 AM Rolf Sommer Poulsen <rs...@bankdata.dk.invalid>
wrote:
> Hi
>
> I was wondering if there is a upper bound on the file size the
> camel-smb component I capable of handling. I have set up a route, that
> streamdownload a file from a windows server using the smb-component,
> and it works fine with normal sized files, however when testing on a
> very large file (~8gb) I get and error about an array that has grown
> to large even before my route is being called.
>
> That made me wonder if there is a upper bound on the size of the files
> that the smb-component can handle and if that size is around
> INTEGER.MAX_VALUE bytes ?
>
>
> Venlig hilsen / Best regards
>
> Rolf Sommer Poulsen
>
>
>
>
--
Claus Ibsen
-----------------
@davsclaus
Camel in Action 2: https://www.manning.com/ibsen2
SV: Upper file limit in SMB protocol ?
Posted by Rolf Sommer Poulsen <rs...@bankdata.dk.INVALID>.
Hi
I am currently using the 4.10.1 release and get the following stacktrace:
java.lang.OutOfMemoryError: Required array length 2147483639 + 9 is too large
at java.base/jdk.internal.util.ArraysSupport.hugeLength(ArraysSupport.java:649) ~[na:na]
at java.base/jdk.internal.util.ArraysSupport.newLength(ArraysSupport.java:642) ~[na:na]
at java.base/java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:100) ~[na:na]
at java.base/java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:130) ~[na:na]
at org.apache.camel.converter.stream.CachedOutputStream.write(CachedOutputStream.java:108) ~[camel-support-4.10.1.jar:4.10.1]
at java.base/java.io.InputStream.transferTo(InputStream.java:783) ~[na:na]
at org.apache.camel.util.IOHelper.copy(IOHelper.java:165) ~[camel-util-4.10.1.jar:4.10.1]
at org.apache.camel.util.IOHelper.copyAndCloseInput(IOHelper.java:283) ~[camel-util-4.10.1.jar:4.10.1]
at org.apache.camel.converter.stream.StreamCacheConverter.convertToStreamCache(StreamCacheConverter.java:54) ~[camel-support-4.10.1.jar:4.10.1]
at org.apache.camel.converter.stream.StreamCacheBulkConverterLoader.doConvertTo(StreamCacheBulkConverterLoader.java:83) ~[camel-support-4.10.1.jar:4.10.1]
at org.apache.camel.converter.stream.StreamCacheBulkConverterLoader.convertTo(StreamCacheBulkConverterLoader.java:56) ~[camel-support-4.10.1.jar:4.10.1]
at org.apache.camel.spi.BulkTypeConverters.convertTo(BulkTypeConverters.java:122) ~[camel-api-4.10.1.jar:4.10.1]
at org.apache.camel.component.file.GenericFileConverter.convertTo(GenericFileConverter.java:106) ~[camel-file-4.10.1.jar:4.10.1]
at org.apache.camel.component.file.GenericFileConverterLoader.lambda$registerFallbackConverters$4(GenericFileConverterLoader.java:63) ~[camel-file-4.10.1.jar:4.10.1]
at org.apache.camel.support.SimpleTypeConverter.convertTo(SimpleTypeConverter.java:101) ~[camel-support-4.10.1.jar:4.10.1]
at org.apache.camel.impl.converter.CoreTypeConverterRegistry.doConvert(CoreTypeConverterRegistry.java:463) ~[camel-base-4.10.1.jar:4.10.1]
at org.apache.camel.impl.converter.CoreTypeConverterRegistry.tryFallback(CoreTypeConverterRegistry.java:432) ~[camel-base-4.10.1.jar:4.10.1]
at org.apache.camel.impl.converter.CoreTypeConverterRegistry.doConvertTo(CoreTypeConverterRegistry.java:381) ~[camel-base-4.10.1.jar:4.10.1]
at org.apache.camel.impl.converter.CoreTypeConverterRegistry.doConvertToAndStat(CoreTypeConverterRegistry.java:269) ~[camel-base-4.10.1.jar:4.10.1]
at org.apache.camel.impl.converter.CoreTypeConverterRegistry.convertTo(CoreTypeConverterRegistry.java:164) ~[camel-base-4.10.1.jar:4.10.1]
at org.apache.camel.impl.engine.DefaultStreamCachingStrategy.doCache(DefaultStreamCachingStrategy.java:278) ~[camel-base-engine-4.10.1.jar:4.10.1]
at org.apache.camel.impl.engine.DefaultStreamCachingStrategy.cache(DefaultStreamCachingStrategy.java:255) ~[camel-base-engine-4.10.1.jar:4.10.1]
at org.apache.camel.impl.engine.StreamCachingHelper.tryStreamCache(StreamCachingHelper.java:67) ~[camel-base-engine-4.10.1.jar:4.10.1]
at org.apache.camel.impl.engine.StreamCachingHelper.convertToStreamCache(StreamCachingHelper.java:54) ~[camel-base-engine-4.10.1.jar:4.10.1]
at org.apache.camel.impl.engine.CamelInternalProcessor$StreamCachingAdvice.before(CamelInternalProcessor.java:1010) ~[camel-base-engine-4.10.1.jar:4.10.1]
at org.apache.camel.impl.engine.CamelInternalProcessor$StreamCachingAdvice.before(CamelInternalProcessor.java:1000) ~[camel-base-engine-4.10.1.jar:4.10.1]
at org.apache.camel.impl.engine.CamelInternalProcessor.process(CamelInternalProcessor.java:310) ~[camel-base-engine-4.10.1.jar:4.10.1]
at org.apache.camel.processor.Pipeline$PipelineTask.run(Pipeline.java:102) ~[camel-core-processor-4.10.1.jar:4.10.1]
at org.apache.camel.impl.engine.DefaultReactiveExecutor$Worker.doRun(DefaultReactiveExecutor.java:199) ~[camel-base-engine-4.10.1.jar:4.10.1]
at org.apache.camel.impl.engine.DefaultReactiveExecutor$Worker.executeReactiveWork(DefaultReactiveExecutor.java:189) ~[camel-base-engine-4.10.1.jar:4.10.1]
at org.apache.camel.impl.engine.DefaultReactiveExecutor$Worker.tryExecuteReactiveWork(DefaultReactiveExecutor.java:166) ~[camel-base-engine-4.10.1.jar:4.10.1]
at org.apache.camel.impl.engine.DefaultReactiveExecutor$Worker.schedule(DefaultReactiveExecutor.java:148) ~[camel-base-engine-4.10.1.jar:4.10.1]
at org.apache.camel.impl.engine.DefaultReactiveExecutor.scheduleMain(DefaultReactiveExecutor.java:59) ~[camel-base-engine-4.10.1.jar:4.10.1]
at org.apache.camel.processor.Pipeline.process(Pipeline.java:163) ~[camel-core-processor-4.10.1.jar:4.10.1]
at org.apache.camel.impl.engine.CamelInternalProcessor.processNonTransacted(CamelInternalProcessor.java:347) ~[camel-base-engine-4.10.1.jar:4.10.1]
at org.apache.camel.impl.engine.CamelInternalProcessor.process(CamelInternalProcessor.java:323) ~[camel-base-engine-4.10.1.jar:4.10.1]
at org.apache.camel.component.file.GenericFileConsumer.processExchange(GenericFileConsumer.java:459) ~[camel-file-4.10.1.jar:4.10.1]
at org.apache.camel.component.smb.SmbConsumer.processExchange(SmbConsumer.java:309) ~[camel-smb-4.10.1.jar:4.10.1]
at org.apache.camel.component.file.GenericFileConsumer.processBatch(GenericFileConsumer.java:245) ~[camel-file-4.10.1.jar:4.10.1]
at org.apache.camel.component.file.GenericFileConsumer.poll(GenericFileConsumer.java:206) ~[camel-file-4.10.1.jar:4.10.1]
at org.apache.camel.support.ScheduledPollConsumer.doRun(ScheduledPollConsumer.java:208) ~[camel-support-4.10.1.jar:4.10.1]
at org.apache.camel.support.ScheduledPollConsumer.run(ScheduledPollConsumer.java:119) ~[camel-support-4.10.1.jar:4.10.1]
at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) ~[spring-context-6.2.2.jar:6.2.2]
at org.springframework.scheduling.concurrent.ReschedulingRunnable.run(ReschedulingRunnable.java:96) ~[spring-context-6.2.2.jar:6.2.2]
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) ~[na:na]
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[na:na]
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) ~[na:na]
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[na:na]
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[na:na]
at java.base/java.lang.Thread.run(Thread.java:840) ~[na:na]
Venlig hilsen / Best regards
Rolf Sommer Poulsen
Senior Software Developer,
FRI Integration
+45 6012 5444 | +45 7924 2811
rsp@bankdata.dk
BANKDATA | Development Centre Silkeborg | Stagehøjvej 30 | 8600 ;Silkeborg
-----Oprindelig meddelelse-----
Fra: Claus Ibsen <cl...@gmail.com>
Sendt: 6. marts 2025 08:09
Til: users@camel.apache.org
Emne: Re: Upper file limit in SMB protocol ?
[Du får ikke ofte mails fra claus.ibsen@gmail.com. Få mere at vide om, hvorfor dette er vigtigt, på https://aka.ms/LearnAboutSenderIdentification ]
Hi
I am not aware of a limit. But can you try with latest 4.10.1 release and also if you see any exception then report the stacktrace
On Thu, Mar 6, 2025 at 8:00 AM Rolf Sommer Poulsen <rs...@bankdata.dk.invalid>
wrote:
> Hi
>
> I was wondering if there is a upper bound on the file size the
> camel-smb component I capable of handling. I have set up a route, that
> streamdownload a file from a windows server using the smb-component,
> and it works fine with normal sized files, however when testing on a
> very large file (~8gb) I get and error about an array that has grown
> to large even before my route is being called.
>
> That made me wonder if there is a upper bound on the size of the files
> that the smb-component can handle and if that size is around
> INTEGER.MAX_VALUE bytes ?
>
>
> Venlig hilsen / Best regards
>
> Rolf Sommer Poulsen
>
>
>
>
--
Claus Ibsen
-----------------
@davsclaus
Camel in Action 2: https://www.manning.com/ibsen2
Re: Upper file limit in SMB protocol ?
Posted by Claus Ibsen <cl...@gmail.com>.
Hi
I am not aware of a limit. But can you try with latest 4.10.1 release and
also if you see any exception then report the stacktrace
On Thu, Mar 6, 2025 at 8:00 AM Rolf Sommer Poulsen <rs...@bankdata.dk.invalid>
wrote:
> Hi
>
> I was wondering if there is a upper bound on the file size the camel-smb
> component I capable of handling. I have set up a route, that streamdownload
> a file from a windows server using the smb-component, and it works fine
> with normal sized files, however when testing on a very large file (~8gb) I
> get and error about an array that has grown to large even before my route
> is being called.
>
> That made me wonder if there is a upper bound on the size of the files
> that the smb-component can handle and if that size is around
> INTEGER.MAX_VALUE bytes ?
>
>
> Venlig hilsen / Best regards
>
> Rolf Sommer Poulsen
>
>
>
>
--
Claus Ibsen
-----------------
@davsclaus
Camel in Action 2: https://www.manning.com/ibsen2