You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@camel.apache.org by Rolf Sommer Poulsen <rs...@bankdata.dk.INVALID> on 2025/03/06 06:58:32 UTC

Upper file limit in SMB protocol ?

Hi

I was wondering if there is a upper bound on the file size the camel-smb component I capable of handling. I have set up a route, that streamdownload a file from a windows server using the smb-component, and it works fine with normal sized files, however when testing on a very large file (~8gb) I get and error about an array that has grown to large even before my route is being called.

That made me wonder if there is a upper bound on the size of the files that the smb-component can handle and if that size is around INTEGER.MAX_VALUE bytes ?


Venlig hilsen / Best regards

Rolf Sommer Poulsen




SV: Upper file limit in SMB protocol ?

Posted by Rolf Sommer Poulsen <rs...@bankdata.dk.INVALID>.
Brilliant. Thank you very much Claus. 


Venlig hilsen / Best regards

Rolf Sommer Poulsen
Senior Software Developer, 
FRI Integration
+45 6012 5444 | +45 7924 2811
rsp@bankdata.dk
BANKDATA | Development Centre Silkeborg |  Stagehøjvej 30 | 8600 ;Silkeborg 
-----Oprindelig meddelelse-----
Fra: Claus Ibsen <cl...@gmail.com> 
Sendt: 6. marts 2025 09:28
Til: users@camel.apache.org
Emne: Re: Upper file limit in SMB protocol ?

[Du får ikke ofte mails fra claus.ibsen@gmail.com. Få mere at vide om, hvorfor dette er vigtigt, på https://aka.ms/LearnAboutSenderIdentification ]

Hi

Ah okay read about stream caching
https://camel.apache.org/manual/stream-caching.html

You can turn that off or enable its spool to disk option

On Thu, Mar 6, 2025 at 8:17 AM Rolf Sommer Poulsen <rs...@bankdata.dk.invalid>
wrote:

> Hi
>
> I am currently using the 4.10.1 release and get the following stacktrace:
>
> java.lang.OutOfMemoryError: Required array length 2147483639 + 9 is 
> too large
>         at
> java.base/jdk.internal.util.ArraysSupport.hugeLength(ArraysSupport.jav
> a:649)
> ~[na:na]
>         at
> java.base/jdk.internal.util.ArraysSupport.newLength(ArraysSupport.java
> :642)
> ~[na:na]
>         at 
> java.base/java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutput
> Stream.java:100)
> ~[na:na]
>         at 
> java.base/java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.ja
> va:130)
> ~[na:na]
>         at
> org.apache.camel.converter.stream.CachedOutputStream.write(CachedOutpu
> tStream.java:108)
> ~[camel-support-4.10.1.jar:4.10.1]
>         at 
> java.base/java.io.InputStream.transferTo(InputStream.java:783)
> ~[na:na]
>         at org.apache.camel.util.IOHelper.copy(IOHelper.java:165)
> ~[camel-util-4.10.1.jar:4.10.1]
>         at
> org.apache.camel.util.IOHelper.copyAndCloseInput(IOHelper.java:283)
> ~[camel-util-4.10.1.jar:4.10.1]
>         at
> org.apache.camel.converter.stream.StreamCacheConverter.convertToStream
> Cache(StreamCacheConverter.java:54)
> ~[camel-support-4.10.1.jar:4.10.1]
>         at
> org.apache.camel.converter.stream.StreamCacheBulkConverterLoader.doCon
> vertTo(StreamCacheBulkConverterLoader.java:83)
> ~[camel-support-4.10.1.jar:4.10.1]
>         at
> org.apache.camel.converter.stream.StreamCacheBulkConverterLoader.conve
> rtTo(StreamCacheBulkConverterLoader.java:56)
> ~[camel-support-4.10.1.jar:4.10.1]
>         at
> org.apache.camel.spi.BulkTypeConverters.convertTo(BulkTypeConverters.j
> ava:122)
> ~[camel-api-4.10.1.jar:4.10.1]
>         at
> org.apache.camel.component.file.GenericFileConverter.convertTo(Generic
> FileConverter.java:106)
> ~[camel-file-4.10.1.jar:4.10.1]
>         at
> org.apache.camel.component.file.GenericFileConverterLoader.lambda$regi
> sterFallbackConverters$4(GenericFileConverterLoader.java:63)
> ~[camel-file-4.10.1.jar:4.10.1]
>         at
> org.apache.camel.support.SimpleTypeConverter.convertTo(SimpleTypeConve
> rter.java:101)
> ~[camel-support-4.10.1.jar:4.10.1]
>         at
> org.apache.camel.impl.converter.CoreTypeConverterRegistry.doConvert(Co
> reTypeConverterRegistry.java:463)
> ~[camel-base-4.10.1.jar:4.10.1]
>         at
> org.apache.camel.impl.converter.CoreTypeConverterRegistry.tryFallback(
> CoreTypeConverterRegistry.java:432)
> ~[camel-base-4.10.1.jar:4.10.1]
>         at
> org.apache.camel.impl.converter.CoreTypeConverterRegistry.doConvertTo(
> CoreTypeConverterRegistry.java:381)
> ~[camel-base-4.10.1.jar:4.10.1]
>         at
> org.apache.camel.impl.converter.CoreTypeConverterRegistry.doConvertToA
> ndStat(CoreTypeConverterRegistry.java:269)
> ~[camel-base-4.10.1.jar:4.10.1]
>         at
> org.apache.camel.impl.converter.CoreTypeConverterRegistry.convertTo(Co
> reTypeConverterRegistry.java:164)
> ~[camel-base-4.10.1.jar:4.10.1]
>         at
> org.apache.camel.impl.engine.DefaultStreamCachingStrategy.doCache(Defa
> ultStreamCachingStrategy.java:278)
> ~[camel-base-engine-4.10.1.jar:4.10.1]
>         at
> org.apache.camel.impl.engine.DefaultStreamCachingStrategy.cache(Defaul
> tStreamCachingStrategy.java:255) 
> ~[camel-base-engine-4.10.1.jar:4.10.1]
>         at
> org.apache.camel.impl.engine.StreamCachingHelper.tryStreamCache(Stream
> CachingHelper.java:67) ~[camel-base-engine-4.10.1.jar:4.10.1]
>         at
> org.apache.camel.impl.engine.StreamCachingHelper.convertToStreamCache(
> StreamCachingHelper.java:54) ~[camel-base-engine-4.10.1.jar:4.10.1]
>         at
> org.apache.camel.impl.engine.CamelInternalProcessor$StreamCachingAdvic
> e.before(CamelInternalProcessor.java:1010)
> ~[camel-base-engine-4.10.1.jar:4.10.1]
>         at
> org.apache.camel.impl.engine.CamelInternalProcessor$StreamCachingAdvic
> e.before(CamelInternalProcessor.java:1000)
> ~[camel-base-engine-4.10.1.jar:4.10.1]
>         at
> org.apache.camel.impl.engine.CamelInternalProcessor.process(CamelInter
> nalProcessor.java:310) ~[camel-base-engine-4.10.1.jar:4.10.1]
>         at
> org.apache.camel.processor.Pipeline$PipelineTask.run(Pipeline.java:102
> ) ~[camel-core-processor-4.10.1.jar:4.10.1]
>         at
> org.apache.camel.impl.engine.DefaultReactiveExecutor$Worker.doRun(Defa
> ultReactiveExecutor.java:199) ~[camel-base-engine-4.10.1.jar:4.10.1]
>         at
> org.apache.camel.impl.engine.DefaultReactiveExecutor$Worker.executeRea
> ctiveWork(DefaultReactiveExecutor.java:189)
> ~[camel-base-engine-4.10.1.jar:4.10.1]
>         at
> org.apache.camel.impl.engine.DefaultReactiveExecutor$Worker.tryExecute
> ReactiveWork(DefaultReactiveExecutor.java:166)
> ~[camel-base-engine-4.10.1.jar:4.10.1]
>         at
> org.apache.camel.impl.engine.DefaultReactiveExecutor$Worker.schedule(D
> efaultReactiveExecutor.java:148) 
> ~[camel-base-engine-4.10.1.jar:4.10.1]
>         at
> org.apache.camel.impl.engine.DefaultReactiveExecutor.scheduleMain(Defa
> ultReactiveExecutor.java:59) ~[camel-base-engine-4.10.1.jar:4.10.1]
>         at 
> org.apache.camel.processor.Pipeline.process(Pipeline.java:163)
> ~[camel-core-processor-4.10.1.jar:4.10.1]
>         at
> org.apache.camel.impl.engine.CamelInternalProcessor.processNonTransact
> ed(CamelInternalProcessor.java:347)
> ~[camel-base-engine-4.10.1.jar:4.10.1]
>         at
> org.apache.camel.impl.engine.CamelInternalProcessor.process(CamelInter
> nalProcessor.java:323) ~[camel-base-engine-4.10.1.jar:4.10.1]
>         at
> org.apache.camel.component.file.GenericFileConsumer.processExchange(Ge
> nericFileConsumer.java:459)
> ~[camel-file-4.10.1.jar:4.10.1]
>         at
> org.apache.camel.component.smb.SmbConsumer.processExchange(SmbConsumer
> .java:309)
> ~[camel-smb-4.10.1.jar:4.10.1]
>         at
> org.apache.camel.component.file.GenericFileConsumer.processBatch(Gener
> icFileConsumer.java:245)
> ~[camel-file-4.10.1.jar:4.10.1]
>         at
> org.apache.camel.component.file.GenericFileConsumer.poll(GenericFileCo
> nsumer.java:206)
> ~[camel-file-4.10.1.jar:4.10.1]
>         at
> org.apache.camel.support.ScheduledPollConsumer.doRun(ScheduledPollCons
> umer.java:208)
> ~[camel-support-4.10.1.jar:4.10.1]
>         at
> org.apache.camel.support.ScheduledPollConsumer.run(ScheduledPollConsum
> er.java:119)
> ~[camel-support-4.10.1.jar:4.10.1]
>         at
> org.springframework.scheduling.support.DelegatingErrorHandlingRunnable
> .run(DelegatingErrorHandlingRunnable.java:54)
> ~[spring-context-6.2.2.jar:6.2.2]
>         at
> org.springframework.scheduling.concurrent.ReschedulingRunnable.run(Res
> chedulingRunnable.java:96)
> ~[spring-context-6.2.2.jar:6.2.2]
>         at
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executor
> s.java:539)
> ~[na:na]
>         at
> java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[na:na]
>         at
> java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFu
> tureTask.run(ScheduledThreadPoolExecutor.java:304)
> ~[na:na]
>         at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool
> Executor.java:1136)
> ~[na:na]
>         at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo
> lExecutor.java:635)
> ~[na:na]
>         at java.base/java.lang.Thread.run(Thread.java:840) ~[na:na]
>
>
> Venlig hilsen / Best regards
>
> Rolf Sommer Poulsen
> Senior Software Developer,
> FRI Integration
> +45 6012 5444 | +45 7924 2811
> rsp@bankdata.dk
> BANKDATA | Development Centre Silkeborg |  Stagehøjvej 30 | 8600 
> ;Silkeborg -----Oprindelig meddelelse-----
> Fra: Claus Ibsen <cl...@gmail.com>
> Sendt: 6. marts 2025 08:09
> Til: users@camel.apache.org
> Emne: Re: Upper file limit in SMB protocol ?
>
> [Du får ikke ofte mails fra claus.ibsen@gmail.com. Få mere at vide om, 
> hvorfor dette er vigtigt, på 
> https://aka.ms/LearnAboutSenderIdentification
> ]
>
> Hi
>
> I am not aware of a limit. But can you try with latest 4.10.1 release 
> and also if you see any exception then report the stacktrace
>
> On Thu, Mar 6, 2025 at 8:00 AM Rolf Sommer Poulsen 
> <rsp@bankdata.dk.invalid
> >
> wrote:
>
> > Hi
> >
> > I was wondering if there is a upper bound on the file size the 
> > camel-smb component I capable of handling. I have set up a route, 
> > that streamdownload a file from a windows server using the 
> > smb-component, and it works fine with normal sized files, however 
> > when testing on a very large file (~8gb) I get and error about an 
> > array that has grown to large even before my route is being called.
> >
> > That made me wonder if there is a upper bound on the size of the 
> > files that the smb-component can handle and if that size is around 
> > INTEGER.MAX_VALUE bytes ?
> >
> >
> > Venlig hilsen / Best regards
> >
> > Rolf Sommer Poulsen
> >
> >
> >
> >
>
> --
> Claus Ibsen
> -----------------
> @davsclaus
> Camel in Action 2: 
> https://www/.
> manning.com%2Fibsen2&data=05%7C02%7Crsp%40bankdata.dk%7Cec9462e067c845
> efda5d08dd5c88fbcd%7C1177168698fc4f1b8e0076c52159e8a6%7C0%7C0%7C638768
> 465582935182%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwL
> jAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%
> 7C&sdata=vQOPORJJ%2BZnXv6262%2FTVHbpoMFQrM%2FlWeCYh%2B%2Fsc30k%3D&rese
> rved=0
>


--
Claus Ibsen
-----------------
@davsclaus
Camel in Action 2: https://www.manning.com/ibsen2

Re: Upper file limit in SMB protocol ?

Posted by Claus Ibsen <cl...@gmail.com>.
Hi

Ah okay read about stream caching
https://camel.apache.org/manual/stream-caching.html

You can turn that off or enable its spool to disk option

On Thu, Mar 6, 2025 at 8:17 AM Rolf Sommer Poulsen <rs...@bankdata.dk.invalid>
wrote:

> Hi
>
> I am currently using the 4.10.1 release and get the following stacktrace:
>
> java.lang.OutOfMemoryError: Required array length 2147483639 + 9 is too
> large
>         at
> java.base/jdk.internal.util.ArraysSupport.hugeLength(ArraysSupport.java:649)
> ~[na:na]
>         at
> java.base/jdk.internal.util.ArraysSupport.newLength(ArraysSupport.java:642)
> ~[na:na]
>         at java.base/java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:100)
> ~[na:na]
>         at java.base/java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:130)
> ~[na:na]
>         at
> org.apache.camel.converter.stream.CachedOutputStream.write(CachedOutputStream.java:108)
> ~[camel-support-4.10.1.jar:4.10.1]
>         at java.base/java.io.InputStream.transferTo(InputStream.java:783)
> ~[na:na]
>         at org.apache.camel.util.IOHelper.copy(IOHelper.java:165)
> ~[camel-util-4.10.1.jar:4.10.1]
>         at
> org.apache.camel.util.IOHelper.copyAndCloseInput(IOHelper.java:283)
> ~[camel-util-4.10.1.jar:4.10.1]
>         at
> org.apache.camel.converter.stream.StreamCacheConverter.convertToStreamCache(StreamCacheConverter.java:54)
> ~[camel-support-4.10.1.jar:4.10.1]
>         at
> org.apache.camel.converter.stream.StreamCacheBulkConverterLoader.doConvertTo(StreamCacheBulkConverterLoader.java:83)
> ~[camel-support-4.10.1.jar:4.10.1]
>         at
> org.apache.camel.converter.stream.StreamCacheBulkConverterLoader.convertTo(StreamCacheBulkConverterLoader.java:56)
> ~[camel-support-4.10.1.jar:4.10.1]
>         at
> org.apache.camel.spi.BulkTypeConverters.convertTo(BulkTypeConverters.java:122)
> ~[camel-api-4.10.1.jar:4.10.1]
>         at
> org.apache.camel.component.file.GenericFileConverter.convertTo(GenericFileConverter.java:106)
> ~[camel-file-4.10.1.jar:4.10.1]
>         at
> org.apache.camel.component.file.GenericFileConverterLoader.lambda$registerFallbackConverters$4(GenericFileConverterLoader.java:63)
> ~[camel-file-4.10.1.jar:4.10.1]
>         at
> org.apache.camel.support.SimpleTypeConverter.convertTo(SimpleTypeConverter.java:101)
> ~[camel-support-4.10.1.jar:4.10.1]
>         at
> org.apache.camel.impl.converter.CoreTypeConverterRegistry.doConvert(CoreTypeConverterRegistry.java:463)
> ~[camel-base-4.10.1.jar:4.10.1]
>         at
> org.apache.camel.impl.converter.CoreTypeConverterRegistry.tryFallback(CoreTypeConverterRegistry.java:432)
> ~[camel-base-4.10.1.jar:4.10.1]
>         at
> org.apache.camel.impl.converter.CoreTypeConverterRegistry.doConvertTo(CoreTypeConverterRegistry.java:381)
> ~[camel-base-4.10.1.jar:4.10.1]
>         at
> org.apache.camel.impl.converter.CoreTypeConverterRegistry.doConvertToAndStat(CoreTypeConverterRegistry.java:269)
> ~[camel-base-4.10.1.jar:4.10.1]
>         at
> org.apache.camel.impl.converter.CoreTypeConverterRegistry.convertTo(CoreTypeConverterRegistry.java:164)
> ~[camel-base-4.10.1.jar:4.10.1]
>         at
> org.apache.camel.impl.engine.DefaultStreamCachingStrategy.doCache(DefaultStreamCachingStrategy.java:278)
> ~[camel-base-engine-4.10.1.jar:4.10.1]
>         at
> org.apache.camel.impl.engine.DefaultStreamCachingStrategy.cache(DefaultStreamCachingStrategy.java:255)
> ~[camel-base-engine-4.10.1.jar:4.10.1]
>         at
> org.apache.camel.impl.engine.StreamCachingHelper.tryStreamCache(StreamCachingHelper.java:67)
> ~[camel-base-engine-4.10.1.jar:4.10.1]
>         at
> org.apache.camel.impl.engine.StreamCachingHelper.convertToStreamCache(StreamCachingHelper.java:54)
> ~[camel-base-engine-4.10.1.jar:4.10.1]
>         at
> org.apache.camel.impl.engine.CamelInternalProcessor$StreamCachingAdvice.before(CamelInternalProcessor.java:1010)
> ~[camel-base-engine-4.10.1.jar:4.10.1]
>         at
> org.apache.camel.impl.engine.CamelInternalProcessor$StreamCachingAdvice.before(CamelInternalProcessor.java:1000)
> ~[camel-base-engine-4.10.1.jar:4.10.1]
>         at
> org.apache.camel.impl.engine.CamelInternalProcessor.process(CamelInternalProcessor.java:310)
> ~[camel-base-engine-4.10.1.jar:4.10.1]
>         at
> org.apache.camel.processor.Pipeline$PipelineTask.run(Pipeline.java:102)
> ~[camel-core-processor-4.10.1.jar:4.10.1]
>         at
> org.apache.camel.impl.engine.DefaultReactiveExecutor$Worker.doRun(DefaultReactiveExecutor.java:199)
> ~[camel-base-engine-4.10.1.jar:4.10.1]
>         at
> org.apache.camel.impl.engine.DefaultReactiveExecutor$Worker.executeReactiveWork(DefaultReactiveExecutor.java:189)
> ~[camel-base-engine-4.10.1.jar:4.10.1]
>         at
> org.apache.camel.impl.engine.DefaultReactiveExecutor$Worker.tryExecuteReactiveWork(DefaultReactiveExecutor.java:166)
> ~[camel-base-engine-4.10.1.jar:4.10.1]
>         at
> org.apache.camel.impl.engine.DefaultReactiveExecutor$Worker.schedule(DefaultReactiveExecutor.java:148)
> ~[camel-base-engine-4.10.1.jar:4.10.1]
>         at
> org.apache.camel.impl.engine.DefaultReactiveExecutor.scheduleMain(DefaultReactiveExecutor.java:59)
> ~[camel-base-engine-4.10.1.jar:4.10.1]
>         at org.apache.camel.processor.Pipeline.process(Pipeline.java:163)
> ~[camel-core-processor-4.10.1.jar:4.10.1]
>         at
> org.apache.camel.impl.engine.CamelInternalProcessor.processNonTransacted(CamelInternalProcessor.java:347)
> ~[camel-base-engine-4.10.1.jar:4.10.1]
>         at
> org.apache.camel.impl.engine.CamelInternalProcessor.process(CamelInternalProcessor.java:323)
> ~[camel-base-engine-4.10.1.jar:4.10.1]
>         at
> org.apache.camel.component.file.GenericFileConsumer.processExchange(GenericFileConsumer.java:459)
> ~[camel-file-4.10.1.jar:4.10.1]
>         at
> org.apache.camel.component.smb.SmbConsumer.processExchange(SmbConsumer.java:309)
> ~[camel-smb-4.10.1.jar:4.10.1]
>         at
> org.apache.camel.component.file.GenericFileConsumer.processBatch(GenericFileConsumer.java:245)
> ~[camel-file-4.10.1.jar:4.10.1]
>         at
> org.apache.camel.component.file.GenericFileConsumer.poll(GenericFileConsumer.java:206)
> ~[camel-file-4.10.1.jar:4.10.1]
>         at
> org.apache.camel.support.ScheduledPollConsumer.doRun(ScheduledPollConsumer.java:208)
> ~[camel-support-4.10.1.jar:4.10.1]
>         at
> org.apache.camel.support.ScheduledPollConsumer.run(ScheduledPollConsumer.java:119)
> ~[camel-support-4.10.1.jar:4.10.1]
>         at
> org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)
> ~[spring-context-6.2.2.jar:6.2.2]
>         at
> org.springframework.scheduling.concurrent.ReschedulingRunnable.run(ReschedulingRunnable.java:96)
> ~[spring-context-6.2.2.jar:6.2.2]
>         at
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
> ~[na:na]
>         at
> java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[na:na]
>         at
> java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
> ~[na:na]
>         at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
> ~[na:na]
>         at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
> ~[na:na]
>         at java.base/java.lang.Thread.run(Thread.java:840) ~[na:na]
>
>
> Venlig hilsen / Best regards
>
> Rolf Sommer Poulsen
> Senior Software Developer,
> FRI Integration
> +45 6012 5444 | +45 7924 2811
> rsp@bankdata.dk
> BANKDATA | Development Centre Silkeborg |  Stagehøjvej 30 | 8600
> ;Silkeborg
> -----Oprindelig meddelelse-----
> Fra: Claus Ibsen <cl...@gmail.com>
> Sendt: 6. marts 2025 08:09
> Til: users@camel.apache.org
> Emne: Re: Upper file limit in SMB protocol ?
>
> [Du får ikke ofte mails fra claus.ibsen@gmail.com. Få mere at vide om,
> hvorfor dette er vigtigt, på https://aka.ms/LearnAboutSenderIdentification
> ]
>
> Hi
>
> I am not aware of a limit. But can you try with latest 4.10.1 release and
> also if you see any exception then report the stacktrace
>
> On Thu, Mar 6, 2025 at 8:00 AM Rolf Sommer Poulsen <rsp@bankdata.dk.invalid
> >
> wrote:
>
> > Hi
> >
> > I was wondering if there is a upper bound on the file size the
> > camel-smb component I capable of handling. I have set up a route, that
> > streamdownload a file from a windows server using the smb-component,
> > and it works fine with normal sized files, however when testing on a
> > very large file (~8gb) I get and error about an array that has grown
> > to large even before my route is being called.
> >
> > That made me wonder if there is a upper bound on the size of the files
> > that the smb-component can handle and if that size is around
> > INTEGER.MAX_VALUE bytes ?
> >
> >
> > Venlig hilsen / Best regards
> >
> > Rolf Sommer Poulsen
> >
> >
> >
> >
>
> --
> Claus Ibsen
> -----------------
> @davsclaus
> Camel in Action 2: https://www.manning.com/ibsen2
>


-- 
Claus Ibsen
-----------------
@davsclaus
Camel in Action 2: https://www.manning.com/ibsen2

SV: Upper file limit in SMB protocol ?

Posted by Rolf Sommer Poulsen <rs...@bankdata.dk.INVALID>.
I guess the limit arises when using the bytearraystream. Is it possible to use some kind of buffered reader when one enables streamdownload ? 


Venlig hilsen / Best regards

Rolf Sommer Poulsen
Senior Software Developer, 
FRI Integration
+45 6012 5444 | +45 7924 2811
rsp@bankdata.dk
BANKDATA | Development Centre Silkeborg |  Stagehøjvej 30 | 8600 ;Silkeborg 
-----Oprindelig meddelelse-----
Fra: rsp@bankdata.dk <rs...@bankdata.dk> På vegne af Rolf Sommer Poulsen
Sendt: 6. marts 2025 08:16
Til: users@camel.apache.org
Emne: SV: Upper file limit in SMB protocol ?

[Du får ikke ofte mails fra rsp@bankdata.dk.invalid. Få mere at vide om, hvorfor dette er vigtigt, på https://aka.ms/LearnAboutSenderIdentification ]

Hi

I am currently using the 4.10.1 release and get the following stacktrace:

java.lang.OutOfMemoryError: Required array length 2147483639 + 9 is too large
        at java.base/jdk.internal.util.ArraysSupport.hugeLength(ArraysSupport.java:649) ~[na:na]
        at java.base/jdk.internal.util.ArraysSupport.newLength(ArraysSupport.java:642) ~[na:na]
        at java.base/java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:100) ~[na:na]
        at java.base/java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:130) ~[na:na]
        at org.apache.camel.converter.stream.CachedOutputStream.write(CachedOutputStream.java:108) ~[camel-support-4.10.1.jar:4.10.1]
        at java.base/java.io.InputStream.transferTo(InputStream.java:783) ~[na:na]
        at org.apache.camel.util.IOHelper.copy(IOHelper.java:165) ~[camel-util-4.10.1.jar:4.10.1]
        at org.apache.camel.util.IOHelper.copyAndCloseInput(IOHelper.java:283) ~[camel-util-4.10.1.jar:4.10.1]
        at org.apache.camel.converter.stream.StreamCacheConverter.convertToStreamCache(StreamCacheConverter.java:54) ~[camel-support-4.10.1.jar:4.10.1]
        at org.apache.camel.converter.stream.StreamCacheBulkConverterLoader.doConvertTo(StreamCacheBulkConverterLoader.java:83) ~[camel-support-4.10.1.jar:4.10.1]
        at org.apache.camel.converter.stream.StreamCacheBulkConverterLoader.convertTo(StreamCacheBulkConverterLoader.java:56) ~[camel-support-4.10.1.jar:4.10.1]
        at org.apache.camel.spi.BulkTypeConverters.convertTo(BulkTypeConverters.java:122) ~[camel-api-4.10.1.jar:4.10.1]
        at org.apache.camel.component.file.GenericFileConverter.convertTo(GenericFileConverter.java:106) ~[camel-file-4.10.1.jar:4.10.1]
        at org.apache.camel.component.file.GenericFileConverterLoader.lambda$registerFallbackConverters$4(GenericFileConverterLoader.java:63) ~[camel-file-4.10.1.jar:4.10.1]
        at org.apache.camel.support.SimpleTypeConverter.convertTo(SimpleTypeConverter.java:101) ~[camel-support-4.10.1.jar:4.10.1]
        at org.apache.camel.impl.converter.CoreTypeConverterRegistry.doConvert(CoreTypeConverterRegistry.java:463) ~[camel-base-4.10.1.jar:4.10.1]
        at org.apache.camel.impl.converter.CoreTypeConverterRegistry.tryFallback(CoreTypeConverterRegistry.java:432) ~[camel-base-4.10.1.jar:4.10.1]
        at org.apache.camel.impl.converter.CoreTypeConverterRegistry.doConvertTo(CoreTypeConverterRegistry.java:381) ~[camel-base-4.10.1.jar:4.10.1]
        at org.apache.camel.impl.converter.CoreTypeConverterRegistry.doConvertToAndStat(CoreTypeConverterRegistry.java:269) ~[camel-base-4.10.1.jar:4.10.1]
        at org.apache.camel.impl.converter.CoreTypeConverterRegistry.convertTo(CoreTypeConverterRegistry.java:164) ~[camel-base-4.10.1.jar:4.10.1]
        at org.apache.camel.impl.engine.DefaultStreamCachingStrategy.doCache(DefaultStreamCachingStrategy.java:278) ~[camel-base-engine-4.10.1.jar:4.10.1]
        at org.apache.camel.impl.engine.DefaultStreamCachingStrategy.cache(DefaultStreamCachingStrategy.java:255) ~[camel-base-engine-4.10.1.jar:4.10.1]
        at org.apache.camel.impl.engine.StreamCachingHelper.tryStreamCache(StreamCachingHelper.java:67) ~[camel-base-engine-4.10.1.jar:4.10.1]
        at org.apache.camel.impl.engine.StreamCachingHelper.convertToStreamCache(StreamCachingHelper.java:54) ~[camel-base-engine-4.10.1.jar:4.10.1]
        at org.apache.camel.impl.engine.CamelInternalProcessor$StreamCachingAdvice.before(CamelInternalProcessor.java:1010) ~[camel-base-engine-4.10.1.jar:4.10.1]
        at org.apache.camel.impl.engine.CamelInternalProcessor$StreamCachingAdvice.before(CamelInternalProcessor.java:1000) ~[camel-base-engine-4.10.1.jar:4.10.1]
        at org.apache.camel.impl.engine.CamelInternalProcessor.process(CamelInternalProcessor.java:310) ~[camel-base-engine-4.10.1.jar:4.10.1]
        at org.apache.camel.processor.Pipeline$PipelineTask.run(Pipeline.java:102) ~[camel-core-processor-4.10.1.jar:4.10.1]
        at org.apache.camel.impl.engine.DefaultReactiveExecutor$Worker.doRun(DefaultReactiveExecutor.java:199) ~[camel-base-engine-4.10.1.jar:4.10.1]
        at org.apache.camel.impl.engine.DefaultReactiveExecutor$Worker.executeReactiveWork(DefaultReactiveExecutor.java:189) ~[camel-base-engine-4.10.1.jar:4.10.1]
        at org.apache.camel.impl.engine.DefaultReactiveExecutor$Worker.tryExecuteReactiveWork(DefaultReactiveExecutor.java:166) ~[camel-base-engine-4.10.1.jar:4.10.1]
        at org.apache.camel.impl.engine.DefaultReactiveExecutor$Worker.schedule(DefaultReactiveExecutor.java:148) ~[camel-base-engine-4.10.1.jar:4.10.1]
        at org.apache.camel.impl.engine.DefaultReactiveExecutor.scheduleMain(DefaultReactiveExecutor.java:59) ~[camel-base-engine-4.10.1.jar:4.10.1]
        at org.apache.camel.processor.Pipeline.process(Pipeline.java:163) ~[camel-core-processor-4.10.1.jar:4.10.1]
        at org.apache.camel.impl.engine.CamelInternalProcessor.processNonTransacted(CamelInternalProcessor.java:347) ~[camel-base-engine-4.10.1.jar:4.10.1]
        at org.apache.camel.impl.engine.CamelInternalProcessor.process(CamelInternalProcessor.java:323) ~[camel-base-engine-4.10.1.jar:4.10.1]
        at org.apache.camel.component.file.GenericFileConsumer.processExchange(GenericFileConsumer.java:459) ~[camel-file-4.10.1.jar:4.10.1]
        at org.apache.camel.component.smb.SmbConsumer.processExchange(SmbConsumer.java:309) ~[camel-smb-4.10.1.jar:4.10.1]
        at org.apache.camel.component.file.GenericFileConsumer.processBatch(GenericFileConsumer.java:245) ~[camel-file-4.10.1.jar:4.10.1]
        at org.apache.camel.component.file.GenericFileConsumer.poll(GenericFileConsumer.java:206) ~[camel-file-4.10.1.jar:4.10.1]
        at org.apache.camel.support.ScheduledPollConsumer.doRun(ScheduledPollConsumer.java:208) ~[camel-support-4.10.1.jar:4.10.1]
        at org.apache.camel.support.ScheduledPollConsumer.run(ScheduledPollConsumer.java:119) ~[camel-support-4.10.1.jar:4.10.1]
        at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) ~[spring-context-6.2.2.jar:6.2.2]
        at org.springframework.scheduling.concurrent.ReschedulingRunnable.run(ReschedulingRunnable.java:96) ~[spring-context-6.2.2.jar:6.2.2]
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) ~[na:na]
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[na:na]
        at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) ~[na:na]
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[na:na]
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[na:na]
        at java.base/java.lang.Thread.run(Thread.java:840) ~[na:na]


Venlig hilsen / Best regards

Rolf Sommer Poulsen
Senior Software Developer,
FRI Integration
+45 6012 5444 | +45 7924 2811
rsp@bankdata.dk
BANKDATA | Development Centre Silkeborg |  Stagehøjvej 30 | 8600 ;Silkeborg -----Oprindelig meddelelse-----
Fra: Claus Ibsen <cl...@gmail.com>
Sendt: 6. marts 2025 08:09
Til: users@camel.apache.org
Emne: Re: Upper file limit in SMB protocol ?

[Du får ikke ofte mails fra claus.ibsen@gmail.com. Få mere at vide om, hvorfor dette er vigtigt, på https://aka.ms/LearnAboutSenderIdentification ]

Hi

I am not aware of a limit. But can you try with latest 4.10.1 release and also if you see any exception then report the stacktrace

On Thu, Mar 6, 2025 at 8:00 AM Rolf Sommer Poulsen <rs...@bankdata.dk.invalid>
wrote:

> Hi
>
> I was wondering if there is a upper bound on the file size the 
> camel-smb component I capable of handling. I have set up a route, that 
> streamdownload a file from a windows server using the smb-component, 
> and it works fine with normal sized files, however when testing on a 
> very large file (~8gb) I get and error about an array that has grown 
> to large even before my route is being called.
>
> That made me wonder if there is a upper bound on the size of the files 
> that the smb-component can handle and if that size is around 
> INTEGER.MAX_VALUE bytes ?
>
>
> Venlig hilsen / Best regards
>
> Rolf Sommer Poulsen
>
>
>
>

--
Claus Ibsen
-----------------
@davsclaus
Camel in Action 2: https://www.manning.com/ibsen2

SV: Upper file limit in SMB protocol ?

Posted by Rolf Sommer Poulsen <rs...@bankdata.dk.INVALID>.
Hi

I am currently using the 4.10.1 release and get the following stacktrace:

java.lang.OutOfMemoryError: Required array length 2147483639 + 9 is too large
	at java.base/jdk.internal.util.ArraysSupport.hugeLength(ArraysSupport.java:649) ~[na:na]
	at java.base/jdk.internal.util.ArraysSupport.newLength(ArraysSupport.java:642) ~[na:na]
	at java.base/java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:100) ~[na:na]
	at java.base/java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:130) ~[na:na]
	at org.apache.camel.converter.stream.CachedOutputStream.write(CachedOutputStream.java:108) ~[camel-support-4.10.1.jar:4.10.1]
	at java.base/java.io.InputStream.transferTo(InputStream.java:783) ~[na:na]
	at org.apache.camel.util.IOHelper.copy(IOHelper.java:165) ~[camel-util-4.10.1.jar:4.10.1]
	at org.apache.camel.util.IOHelper.copyAndCloseInput(IOHelper.java:283) ~[camel-util-4.10.1.jar:4.10.1]
	at org.apache.camel.converter.stream.StreamCacheConverter.convertToStreamCache(StreamCacheConverter.java:54) ~[camel-support-4.10.1.jar:4.10.1]
	at org.apache.camel.converter.stream.StreamCacheBulkConverterLoader.doConvertTo(StreamCacheBulkConverterLoader.java:83) ~[camel-support-4.10.1.jar:4.10.1]
	at org.apache.camel.converter.stream.StreamCacheBulkConverterLoader.convertTo(StreamCacheBulkConverterLoader.java:56) ~[camel-support-4.10.1.jar:4.10.1]
	at org.apache.camel.spi.BulkTypeConverters.convertTo(BulkTypeConverters.java:122) ~[camel-api-4.10.1.jar:4.10.1]
	at org.apache.camel.component.file.GenericFileConverter.convertTo(GenericFileConverter.java:106) ~[camel-file-4.10.1.jar:4.10.1]
	at org.apache.camel.component.file.GenericFileConverterLoader.lambda$registerFallbackConverters$4(GenericFileConverterLoader.java:63) ~[camel-file-4.10.1.jar:4.10.1]
	at org.apache.camel.support.SimpleTypeConverter.convertTo(SimpleTypeConverter.java:101) ~[camel-support-4.10.1.jar:4.10.1]
	at org.apache.camel.impl.converter.CoreTypeConverterRegistry.doConvert(CoreTypeConverterRegistry.java:463) ~[camel-base-4.10.1.jar:4.10.1]
	at org.apache.camel.impl.converter.CoreTypeConverterRegistry.tryFallback(CoreTypeConverterRegistry.java:432) ~[camel-base-4.10.1.jar:4.10.1]
	at org.apache.camel.impl.converter.CoreTypeConverterRegistry.doConvertTo(CoreTypeConverterRegistry.java:381) ~[camel-base-4.10.1.jar:4.10.1]
	at org.apache.camel.impl.converter.CoreTypeConverterRegistry.doConvertToAndStat(CoreTypeConverterRegistry.java:269) ~[camel-base-4.10.1.jar:4.10.1]
	at org.apache.camel.impl.converter.CoreTypeConverterRegistry.convertTo(CoreTypeConverterRegistry.java:164) ~[camel-base-4.10.1.jar:4.10.1]
	at org.apache.camel.impl.engine.DefaultStreamCachingStrategy.doCache(DefaultStreamCachingStrategy.java:278) ~[camel-base-engine-4.10.1.jar:4.10.1]
	at org.apache.camel.impl.engine.DefaultStreamCachingStrategy.cache(DefaultStreamCachingStrategy.java:255) ~[camel-base-engine-4.10.1.jar:4.10.1]
	at org.apache.camel.impl.engine.StreamCachingHelper.tryStreamCache(StreamCachingHelper.java:67) ~[camel-base-engine-4.10.1.jar:4.10.1]
	at org.apache.camel.impl.engine.StreamCachingHelper.convertToStreamCache(StreamCachingHelper.java:54) ~[camel-base-engine-4.10.1.jar:4.10.1]
	at org.apache.camel.impl.engine.CamelInternalProcessor$StreamCachingAdvice.before(CamelInternalProcessor.java:1010) ~[camel-base-engine-4.10.1.jar:4.10.1]
	at org.apache.camel.impl.engine.CamelInternalProcessor$StreamCachingAdvice.before(CamelInternalProcessor.java:1000) ~[camel-base-engine-4.10.1.jar:4.10.1]
	at org.apache.camel.impl.engine.CamelInternalProcessor.process(CamelInternalProcessor.java:310) ~[camel-base-engine-4.10.1.jar:4.10.1]
	at org.apache.camel.processor.Pipeline$PipelineTask.run(Pipeline.java:102) ~[camel-core-processor-4.10.1.jar:4.10.1]
	at org.apache.camel.impl.engine.DefaultReactiveExecutor$Worker.doRun(DefaultReactiveExecutor.java:199) ~[camel-base-engine-4.10.1.jar:4.10.1]
	at org.apache.camel.impl.engine.DefaultReactiveExecutor$Worker.executeReactiveWork(DefaultReactiveExecutor.java:189) ~[camel-base-engine-4.10.1.jar:4.10.1]
	at org.apache.camel.impl.engine.DefaultReactiveExecutor$Worker.tryExecuteReactiveWork(DefaultReactiveExecutor.java:166) ~[camel-base-engine-4.10.1.jar:4.10.1]
	at org.apache.camel.impl.engine.DefaultReactiveExecutor$Worker.schedule(DefaultReactiveExecutor.java:148) ~[camel-base-engine-4.10.1.jar:4.10.1]
	at org.apache.camel.impl.engine.DefaultReactiveExecutor.scheduleMain(DefaultReactiveExecutor.java:59) ~[camel-base-engine-4.10.1.jar:4.10.1]
	at org.apache.camel.processor.Pipeline.process(Pipeline.java:163) ~[camel-core-processor-4.10.1.jar:4.10.1]
	at org.apache.camel.impl.engine.CamelInternalProcessor.processNonTransacted(CamelInternalProcessor.java:347) ~[camel-base-engine-4.10.1.jar:4.10.1]
	at org.apache.camel.impl.engine.CamelInternalProcessor.process(CamelInternalProcessor.java:323) ~[camel-base-engine-4.10.1.jar:4.10.1]
	at org.apache.camel.component.file.GenericFileConsumer.processExchange(GenericFileConsumer.java:459) ~[camel-file-4.10.1.jar:4.10.1]
	at org.apache.camel.component.smb.SmbConsumer.processExchange(SmbConsumer.java:309) ~[camel-smb-4.10.1.jar:4.10.1]
	at org.apache.camel.component.file.GenericFileConsumer.processBatch(GenericFileConsumer.java:245) ~[camel-file-4.10.1.jar:4.10.1]
	at org.apache.camel.component.file.GenericFileConsumer.poll(GenericFileConsumer.java:206) ~[camel-file-4.10.1.jar:4.10.1]
	at org.apache.camel.support.ScheduledPollConsumer.doRun(ScheduledPollConsumer.java:208) ~[camel-support-4.10.1.jar:4.10.1]
	at org.apache.camel.support.ScheduledPollConsumer.run(ScheduledPollConsumer.java:119) ~[camel-support-4.10.1.jar:4.10.1]
	at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) ~[spring-context-6.2.2.jar:6.2.2]
	at org.springframework.scheduling.concurrent.ReschedulingRunnable.run(ReschedulingRunnable.java:96) ~[spring-context-6.2.2.jar:6.2.2]
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) ~[na:na]
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[na:na]
	at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) ~[na:na]
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[na:na]
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[na:na]
	at java.base/java.lang.Thread.run(Thread.java:840) ~[na:na]


Venlig hilsen / Best regards

Rolf Sommer Poulsen
Senior Software Developer, 
FRI Integration
+45 6012 5444 | +45 7924 2811
rsp@bankdata.dk
BANKDATA | Development Centre Silkeborg |  Stagehøjvej 30 | 8600 ;Silkeborg 
-----Oprindelig meddelelse-----
Fra: Claus Ibsen <cl...@gmail.com> 
Sendt: 6. marts 2025 08:09
Til: users@camel.apache.org
Emne: Re: Upper file limit in SMB protocol ?

[Du får ikke ofte mails fra claus.ibsen@gmail.com. Få mere at vide om, hvorfor dette er vigtigt, på https://aka.ms/LearnAboutSenderIdentification ]

Hi

I am not aware of a limit. But can you try with latest 4.10.1 release and also if you see any exception then report the stacktrace

On Thu, Mar 6, 2025 at 8:00 AM Rolf Sommer Poulsen <rs...@bankdata.dk.invalid>
wrote:

> Hi
>
> I was wondering if there is a upper bound on the file size the 
> camel-smb component I capable of handling. I have set up a route, that 
> streamdownload a file from a windows server using the smb-component, 
> and it works fine with normal sized files, however when testing on a 
> very large file (~8gb) I get and error about an array that has grown 
> to large even before my route is being called.
>
> That made me wonder if there is a upper bound on the size of the files 
> that the smb-component can handle and if that size is around 
> INTEGER.MAX_VALUE bytes ?
>
>
> Venlig hilsen / Best regards
>
> Rolf Sommer Poulsen
>
>
>
>

--
Claus Ibsen
-----------------
@davsclaus
Camel in Action 2: https://www.manning.com/ibsen2

Re: Upper file limit in SMB protocol ?

Posted by Claus Ibsen <cl...@gmail.com>.
Hi

I am not aware of a limit. But can you try with latest 4.10.1 release and
also if you see any exception then report the stacktrace

On Thu, Mar 6, 2025 at 8:00 AM Rolf Sommer Poulsen <rs...@bankdata.dk.invalid>
wrote:

> Hi
>
> I was wondering if there is a upper bound on the file size the camel-smb
> component I capable of handling. I have set up a route, that streamdownload
> a file from a windows server using the smb-component, and it works fine
> with normal sized files, however when testing on a very large file (~8gb) I
> get and error about an array that has grown to large even before my route
> is being called.
>
> That made me wonder if there is a upper bound on the size of the files
> that the smb-component can handle and if that size is around
> INTEGER.MAX_VALUE bytes ?
>
>
> Venlig hilsen / Best regards
>
> Rolf Sommer Poulsen
>
>
>
>

-- 
Claus Ibsen
-----------------
@davsclaus
Camel in Action 2: https://www.manning.com/ibsen2