FileConsumer-to-FileProducer bug

[Aaron Crickenberger <aa...@intalgent.com>]

I've been playing around with Camel's FileComponent.  I started with 
camel-example-spring, and modified MyRouteBuilder to look like so:

public class MyRouteBuilder extends RouteBuilder {
    public void configure() {
        from("file:/tmp/test/incoming/").to("file:/tmp/test/outgoing/");
    }
}

Putting a file in the "incoming" directory then results in this:

org.apache.camel.InvalidTypeException: Could not convert value: 
java.io.BufferedInputStream@ce6303 to type: java.nio.ByteBuffer but has 
value: java.io.BufferedInputStream@ce6303 of type: 
java.io.BufferedInputStream on the exchange: Exchange[FileMessage: 
/tmp/test/incoming/foo.lock]
        at 
org.apache.camel.util.ExchangeHelper.convertToMandatoryType(ExchangeHelper.java:91)
        at 
org.apache.camel.component.file.FileProducer.process(FileProducer.java:57)
        at 
org.apache.camel.component.file.FileProducer.process(FileProducer.java:50)
        at 
org.apache.camel.processor.SendProcessor.process(SendProcessor.java:65)
        at 
org.apache.camel.processor.DeadLetterChannel.process(DeadLetterChannel.java:78)
        at 
org.apache.camel.component.file.FileConsumer.pollFile(FileConsumer.java:81)
        at 
org.apache.camel.component.file.FileConsumer.pollFileOrDirectory(FileConsumer.java:51)
        at 
org.apache.camel.component.file.FileConsumer.pollFileOrDirectory(FileConsumer.java:58)
        at 
org.apache.camel.component.file.FileConsumer.poll(FileConsumer.java:45)
        at 
org.apache.camel.impl.ScheduledPollConsumer.run(ScheduledPollConsumer.java:63)
        at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:417)
        at 
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:280)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:135)
        at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:65)
        at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:142)
        at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:166)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
        at java.lang.Thread.run(Thread.java:613)

This extra if block in FileProducer confuses me because it seems like 
there's no InputStream to ByteBuffer conversion:

ByteBuffer payload = exchange.getIn().getBody(ByteBuffer.class);
if (payload == null) {
    InputStream in = ExchangeHelper.getMandatoryInBody(exchange, 
InputStream.class);
    payload = ExchangeHelper.convertToMandatoryType(exchange, 
ByteBuffer.class, in);
}

Adding the following File to ByteBuffer conversion to NIOConverter does 
the trick:

@Converter
public static ByteBuffer toByteBuffer(File file) throws IOException {
    byte[] buf = new byte[(int) file.length()];
    InputStream in = new BufferedInputStream(new FileInputStream(file));
    in.read(buf);
    return ByteBuffer.wrap(buf);
}

Hope this helps!

- aaron

Re: FileConsumer-to-FileProducer bug

[James Strachan <ja...@gmail.com>]

On 8/9/07, Aaron Crickenberger <aa...@intalgent.com> wrote:
> I've been playing around with Camel's FileComponent.  I started with
> camel-example-spring, and modified MyRouteBuilder to look like so:
>
> public class MyRouteBuilder extends RouteBuilder {
>     public void configure() {
>         from("file:/tmp/test/incoming/").to("file:/tmp/test/outgoing/");
>     }
> }
>
> Putting a file in the "incoming" directory then results in this:
>
> org.apache.camel.InvalidTypeException: Could not convert value:
> java.io.BufferedInputStream@ce6303 to type: java.nio.ByteBuffer but has
> value: java.io.BufferedInputStream@ce6303 of type:
> java.io.BufferedInputStream on the exchange: Exchange[FileMessage:
> /tmp/test/incoming/foo.lock]

Great catch! :) This was my fault I think; the fix for CAMEL-90 broke
this. Bad James!

I've raised an issue to track this bug fixes release etc...
https://issues.apache.org/activemq/browse/CAMEL-95

[snip]


> Adding the following File to ByteBuffer conversion to NIOConverter does
> the trick:
>
> @Converter
> public static ByteBuffer toByteBuffer(File file) throws IOException {
>     byte[] buf = new byte[(int) file.length()];
>     InputStream in = new BufferedInputStream(new FileInputStream(file));
>     in.read(buf);
>     return ByteBuffer.wrap(buf);
> }
>
> Hope this helps!

Awesome stuff Aaron!

I've added the test case FileConsumerProducerRouteTest which
reproduces this issue (consuming 2 files from a directory to a new
output directory, then polling this output directory and asserting via
Mocks that the right number of messages are received)...
https://svn.apache.org/repos/asf/activemq/camel/trunk/camel-core/src/test/java/org/apache/camel/component/file/FileConsumerProducerRouteTest.java

Then I've contributed your patch - many thanks!

While reading the code of the FileProducer, I wasn't totally happy
with the implementation; converting a File to a single ByteBuffer by
default; which won't handle massive files very easily; so I've changed
the implementation to just use streams by default (working with 128K
chunks at a time) which will hopefully allow us to deal with huge
files more easily etc.

We could add a further optimisation one day to make FileProcessor be
File-aware and use FileChannel APIs to move an entire file from one
channel to another in a single method call which should be faster.
https://issues.apache.org/activemq/browse/CAMEL-94

Thanks again!

-- 
James
-------
http://macstrac.blogspot.com/