You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@camel.apache.org by Larry Meadors <la...@gmail.com> on 2012/02/03 22:15:46 UTC
File consumer - waiting for it to start
I have what I think is a simple route:
public void configure() throws Exception {
from(sourceUri)
.setHeader(S3Constants.KEY, simple("${header.CamelFileNameOnly}"))
.setHeader(S3Constants.CONTENT_LENGTH, simple("${header.CamelFileLength}"))
.to(destinationUri);
}
It's just taking files from a directory and copying them to an s3 bucket.
What I'm seeing is that when I start it up, it stops immediately, so I
end up doing this:
// create the route - the parameters are the source, then the destination
DirectCopyRoute directCopyRoute = new DirectCopyRoute(arg[0], arg[1]);
// create the camel context and start it
CamelContext camelContext = new DefaultCamelContext();
camelContext.addRoutes(directCopyRoute);
camelContext.start();
// give it time to realize it has work to do
Thread.sleep(1000);
If I remove the Thread.sleep call, it just starts and stops.
Adding that seems to make it work, but I'm wondering why?
I'm guessing the "why" is that it takes a bit of time to create the
list of files to process, and since there is nothing pending to
process, camel figures it's done what it was supposed to, and shuts
down - adding the delay gives the file consumer time to create the
list of files to process, and once it's started, it goes until it's
done.
It that's the case, it leads to a second question: Will 1 second be
long enough to work if the source directory contains a LOT of files?
Normally, the directory will only have a few hundred files, but it may
have a few thousand, and it may be accessing a shared directory over a
network, so the time required to get a directory listing may vary.
Is there a smarter way to do this?
Larry
Re: File consumer - waiting for it to start
Posted by Babak Vahdat <ba...@swissonline.ch>.
You don't have to kill the process by hand using "kill -9" :-)
You can still make use of the way described in the link I sent you already.
Then do consume files with sendEmptyMessageWhenIdle option [1] and then see
how to shutdown your application [2] when *you* detect there're no more
files to consume inside the Processor.
[1] http://camel.apache.org/file2.html
[2] http://camel.apache.org/how-can-i-stop-a-route-from-a-route.html
Babak
--
View this message in context: http://camel.465427.n5.nabble.com/File-consumer-waiting-for-it-to-start-tp5455062p5455909.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: File consumer - waiting for it to start
Posted by Babak Vahdat <ba...@swissonline.ch>.
Just realized that the behavior you see is consistent with the documentation
[1] which clearly states that the onCompletion logic runs *concurrently* and
*not* in the main thread (I did not know that).
While you run your test method shouldCopyFromSourceToDestination() it runs
inside the JVM *main* thread so that while you do:
verify(main).stop();
inside the *main* thread the onCompletion block should have been *already*
run concurrently to make Mockito happy. You make this happen in that you do
sleep a bit inside the *main* thread. If you don't sleep long enough then
the chance is given that by one run your test passes and by another run it
doesn't pass anymore, so that sleeping *long enough* to catch the worst
Thread-Scheduling order & duration of JVM would be a good idea.
[1] http://camel.apache.org/oncompletion.html
Babak
--
View this message in context: http://camel.465427.n5.nabble.com/File-consumer-waiting-for-it-to-start-tp5455062p5456788.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: File consumer - waiting for it to start
Posted by Larry Meadors <la...@gmail.com>.
Same result - works with the sleep; fails without it...which seems odd
- I wonder if it's a result of using the onCompletion() stuff to shut
things down.
I added some logging to see what's happening.
Without the sleep call it looks like this:
[thread #0] INFO 2012-02-04 10:43:16,646: shutting down route
DirectCopyRoute and main (my message)
[main] INFO 2012-02-04 10:43:16,646: Asserting: Endpoint[mock://test]
is satisfied
[main] INFO 2012-02-04 10:43:16,647: checking to see if stop was
called (my message)
[thread #0] INFO 2012-02-04 10:43:16,647: Starting to graceful
shutdown 1 routes (timeout 10 seconds)
[thread #1] INFO 2012-02-04 10:43:16,650: Route: DirectCopyRoute
shutdown complete, was consuming from: Endpoint[direct://test]
[thread #0] INFO 2012-02-04 10:43:16,650: Graceful shutdown of 1
routes completed in 0 seconds
[thread #0] INFO 2012-02-04 10:43:16,651: Route: DirectCopyRoute
stopped, was consuming from: Endpoint[direct://test]
[thread #0] INFO 2012-02-04 10:43:16,657: everything is shut down now
(my message)
With the sleep, it looks like this:
[thread #0] INFO 2012-02-04 10:45:49,111: shutting down route
DirectCopyRoute and main (my message)
[main] INFO 2012-02-04 10:45:49,111: Asserting: Endpoint[mock://test]
is satisfied
[thread #0] INFO 2012-02-04 10:45:49,112: Starting to graceful
shutdown 1 routes (timeout 10 seconds)
[thread #1] INFO 2012-02-04 10:45:49,114: Route: DirectCopyRoute
shutdown complete, was consuming from: Endpoint[direct://test]
[thread #0] INFO 2012-02-04 10:45:49,115: Graceful shutdown of 1
routes completed in 0 seconds
[thread #0] INFO 2012-02-04 10:45:49,116: Route: DirectCopyRoute
stopped, was consuming from: Endpoint[direct://test]
[thread #0] INFO 2012-02-04 10:45:49,122: everything is shut down now
(my message)
[main] INFO 2012-02-04 10:45:49,212: checking to see if stop was
called (my message)
I can live with the sleep in the test - it's only 100ms. :)
Larry
Re: File consumer - waiting for it to start
Posted by Babak Vahdat <ba...@swissonline.ch>.
Make use of template.requestBodyAndHeaders() instead
of template.sendBodyAndHeaders().
Babak
--
View this message in context: http://camel.465427.n5.nabble.com/File-consumer-waiting-for-it-to-start-tp5455062p5456440.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: File consumer - waiting for it to start
Posted by Larry Meadors <la...@gmail.com>.
On Sat, Feb 4, 2012 at 6:49 AM, Babak Vahdat
<ba...@swissonline.ch> wrote:
> Of course the best way to verify it, is to use Camel test toolkit [1] and
> check if it does functionally what you expect it to do.
Thanks - that's a great idea - I've got a unit test for it, but how
can I simulate onCompletion from it? I'm extending CamelTestSupport -
my test method is below.
I'm using mockito, so main is just set to mock(Main.class) in the
setup - and I'm getting a verification failure on the last line of the
test.
I added a log message to the shutdown processor that calls main.stop()
- and I see that in the output, so it's getting called, but I'm
thinking it's getting called *after* the verify() call...actually, I
just tested this with a Thread.sleep(100) before the verify() call,
and that makes it pass.
Is there a more clever way to do this?
---
@Test
public void shouldCopyFromSourceToDestination() throws Exception {
// setup test
MockEndpoint mockEndpoint = getMockEndpoint(MOCK_TEST);
mockEndpoint.expectedMessageCount(1);
// run test
Map<String, Object> map = new HashMap<String, Object>();
map.put("CamelFileNameOnly", "100000000.JPG");
map.put("CamelFileLength", "100");
template.sendBodyAndHeaders(DIRECT_TEST, "test", map);
// verify behavior
assertMockEndpointsSatisfied();
Thread.sleep(100); // <<< fails w/o this
verify(main).stop();
}
---
Larry
Re: File consumer - waiting for it to start
Posted by Babak Vahdat <ba...@swissonline.ch>.
Yeah your code snippet seems reasonable to me and is one possible way to go
for it.
Of course the best way to verify it, is to use Camel test toolkit [1] and
check if it does functionally what you expect it to do.
[1] http://camel.apache.org/testing.html
Babak
--
View this message in context: http://camel.465427.n5.nabble.com/File-consumer-waiting-for-it-to-start-tp5455062p5456174.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: File consumer - waiting for it to start
Posted by Larry Meadors <la...@gmail.com>.
AH!
OK, this makes sense now and is reasonably clean and simple.
I'm not using sendEmptyMessageWhenIdle - I just want to run through
the route one time, so it's not needed in my case.
Just to summarize for the next guy who wants to do this (and to make
sure that I'm really doing this as I'm supposed to), here's what I'm
doing and it seems to be doing exactly what I want it to do.
- create a Main object
- create my route (injecting that main object into the route)
- in the onCompletion() for my route, stop the route and the main
object (this is a one route deal)
- add my route to main
- call main.run()
Look right?
Larry
---
public DirectCopyRoute(String sourceUri, String destinationUri, Main main) {
this.sourceUri = sourceUri;
this.destinationUri = destinationUri;
this.main = main;
}
@Override
public void configure() throws Exception {
from(sourceUri)
.routeId("DirectCopyRoute")
.onCompletion()
.process(new Processor() {
@Override
public void process(Exchange exchange) throws Exception {
CamelContext context = getContext();
context.getInflightRepository().remove(exchange);
context.stopRoute("DirectCopyRoute");
main.stop();
}
})
.end()
.setHeader(S3Constants.KEY, simple("${header.CamelFileNameOnly}"))
.setHeader(S3Constants.CONTENT_LENGTH, simple("${header.CamelFileLength}"))
.log(LoggingLevel.INFO,
"Moving file : " +
simple("${header.CamelFileNameOnly}").getText() + " - " +
simple("${header.CamelFileLength}").getText() + " bytes")
.to(destinationUri);
}
---
Re: File consumer - waiting for it to start
Posted by Babak Vahdat <ba...@swissonline.ch>.
Hi Larry,
I decided to make it real, so following a working example for your usecase:
I did take an already existing Camel example [1] and modified it [2]. You
can run the class either directly inside your IDE or through Maven:
mvn exec:java
-Dexec.mainClass=org.apache.camel.example.spring.MyRouteBuilder
And then watch out how the application shuts down after consuming the 2 xml
files under the data folder.
[1] http://camel.apache.org/spring-example.html
[2]
https://svn.apache.org/repos/asf/camel/trunk/examples/camel-example-spring/src/main/java/org/apache/camel/example/spring/MyRouteBuilder.java
And following the modification I made:
package org.apache.camel.example.spring;
import org.apache.camel.Body;
import org.apache.camel.CamelContext;
import org.apache.camel.Exchange;
import org.apache.camel.Processor;
import org.apache.camel.ProducerTemplate;
import org.apache.camel.builder.RouteBuilder;
import org.apache.camel.spring.Main;
/**
* A simple example router from a file system to an ActiveMQ queue and then
to a
* file system
*
* @version
*/
public class MyRouteBuilder extends RouteBuilder {
static Main main;
/**
* Allow this route to be run as an application
*/
public static void main(String[] args) throws Exception {
main = new Main();
main.run(args);
}
public void configure() {
final CamelContext context = getContext();
final ProducerTemplate template = context.createProducerTemplate();
from("file:src/data?sendEmptyMessageWhenIdle=true").routeId("consumingRoute").process(new
Processor() {
@Override
public void process(Exchange exchange) throws Exception {
if (exchange.getIn().getBody() != null) {
// let's process the file content
template.send("direct:processingRoute", exchange);
} else {
context.getInflightRepository().remove(exchange);
main.stop();
}
}
});
from("direct:processingRoute").bean(new ProcessingBean());
}
public static class ProcessingBean {
public void process(@Body Object body, CamelContext context) {
String fileContent =
context.getTypeConverter().convertTo(String.class, body);
System.out.println("Begin of the file content processing: " +
body);
// do your processing here:
System.out.println(fileContent);
System.out.println("End of the file content processing: " +
body);
}
}
}
--
View this message in context: http://camel.465427.n5.nabble.com/File-consumer-waiting-for-it-to-start-tp5455062p5456068.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: File consumer - waiting for it to start
Posted by Larry Meadors <la...@gmail.com>.
On Fri, Feb 3, 2012 at 5:04 PM, Hadrian Zbarcea <hz...@gmail.com> wrote:
> Babak's link should help. That is useful mostly when you run manually using
> something like camel:run. Another option is to wait for an event after
> context.start() and at the end of your processing trigger the event. That
> way your wait time will be no longer than needed.
Babak is describing something I don't want - I do not want it to start
and then run until I kill it.
Hadrian, I think you're talking about what I want, but I'm not sure
how to do it. :-)
I have a remote process that creates some files in a directory, then
this code is launched after all of the files exist.
I want this process to start up, read and process each file, then exit.
I'm looking for is a way for this route to stay alive until either all
files are processed or the file consumer tells me there are no files
to be processed. It seems like instead it is stopping before it gets a
chance to start.
Is the answer to create a route that waits for a message from the file
processing route that somehow tells camel to shut down?
That seems overly complex...I think I'd prefer adding a long wait time to that.
It'd be perfect if you could somehow tell camel to stay alive until
the route completed at least one time.
Larry
Re: File consumer - waiting for it to start
Posted by Hadrian Zbarcea <hz...@gmail.com>.
Babak's link should help. That is useful mostly when you run manually
using something like camel:run. Another option is to wait for an event
after context.start() and at the end of your processing trigger the
event. That way your wait time will be no longer than needed.
Cheers,
Hadrian
On 02/03/2012 06:47 PM, Babak Vahdat wrote:
> Hi
>
> Starting Camel context (context.start()) does *not* block the main thread.
> However when you do a Thread.sleep() you give a chance to the daemon thread
> to run the file consumer polling inside, eg 1 polling round, and as it's a
> daemon thread it doesn't block the JVM from exiting so that your JVM exits
> at the end!
>
> You can take a look at [1] to see how you can keep your Camel standalone
> application running.
>
> [1]
> http://camel.apache.org/running-camel-standalone-and-have-it-keep-running.html
>
> Babak
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/File-consumer-waiting-for-it-to-start-tp5455062p5455435.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
--
Hadrian Zbarcea
Principal Software Architect
Talend, Inc
http://coders.talend.com/
http://camelbot.blogspot.com/
Re: File consumer - waiting for it to start
Posted by Babak Vahdat <ba...@swissonline.ch>.
Hi
Starting Camel context (context.start()) does *not* block the main thread.
However when you do a Thread.sleep() you give a chance to the daemon thread
to run the file consumer polling inside, eg 1 polling round, and as it's a
daemon thread it doesn't block the JVM from exiting so that your JVM exits
at the end!
You can take a look at [1] to see how you can keep your Camel standalone
application running.
[1]
http://camel.apache.org/running-camel-standalone-and-have-it-keep-running.html
Babak
--
View this message in context: http://camel.465427.n5.nabble.com/File-consumer-waiting-for-it-to-start-tp5455062p5455435.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: File consumer - waiting for it to start
Posted by Babak Vahdat <ba...@swissonline.ch>.
Hi Claus,
Thanks for the offer but thinking a bit more about it I prefer not to
introduce that "hack". The current behaviour
is reasonable and makes sense to me. And the updates you made on the Wiki
makes this *now* clear.
Babak
--
View this message in context: http://camel.465427.n5.nabble.com/File-consumer-waiting-for-it-to-start-tp5455062p5459995.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: File consumer - waiting for it to start
Posted by Claus Ibsen <cl...@gmail.com>.
On Sun, Feb 5, 2012 at 4:28 PM, Babak Vahdat
<ba...@swissonline.ch> wrote:
> Hi Claus,
>
> with the usage of ScheduledBatchPollingConsumer and it's derivates inside
> routes there's an issue while doing context.stopRoute("myCoolRoute") as
> Camel tries to shutdown the route gracefully and as the mentioned class
> does:
>
> if (answer == 0 && isPolling()) {
> // force at least one pending exchange if we are polling as
> there is a little gap
> // in the processBatch method and until an exchange gets
> enlisted as in-flight
> // which happens later, so we need to signal back to the
> shutdown strategy that
> // there is a pending exchange. When we are no longer polling,
> then we will return 0
> LOG.trace("Currently polling so returning 1 as pending
> exchanges");
> answer = 1;
> }
>
> it waits and waits until the (30 seconds) timeout reaches although no *real*
> inflight exchanges are available at all! I realized this behavior as I was
> trying to provide a working example for the user (you see already by this
> thread).
>
> However doing main.stop() (as by the upper example) did *not* behave like
> this!
>
> A possible fix (may be dirty) would be to check that if we already returned
> X times a hard-coded value of 1 then we should better return 0 instead to
> signal there's no real inflight exchange available.
I was first against this as stopping a route from a route was in fact wrong.
And hence why the FAQ is updated etc.
However there is still this problem for people mistakenly using the
old approach.
We could hack as you suggested, but should we add hacks in the code
for bad practice?
If we add the hack, we must ensure that the state is cleared, when you
start the route again,
to avoid the state to be kept around.
Babak, feel free to open a JIRA and work on the hack if you fell like
we need it. But remember
to add code comments why we have this hack in the code. So when we
look at it again in 1+ years
we know why the hack is there.
>
> Babak
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/File-consumer-waiting-for-it-to-start-tp5455062p5458048.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
--
Claus Ibsen
-----------------
FuseSource
Email: cibsen@fusesource.com
Web: http://fusesource.com
Twitter: davsclaus, fusenews
Blog: http://davsclaus.blogspot.com/
Author of Camel in Action: http://www.manning.com/ibsen/
Re: File consumer - waiting for it to start
Posted by Claus Ibsen <cl...@gmail.com>.
On Tue, Feb 7, 2012 at 12:04 AM, Babak Vahdat
<ba...@swissonline.ch> wrote:
> Bilgin,
>
> It has to do with the sync between the Wiki and the Apache Camel site which
> goes wrong [1] from time to time.
>
> [1]
> http://camel.465427.n5.nabble.com/code-snippets-broken-on-confluence-wiki-td5062145.html
>
The dynamic page works
https://cwiki.apache.org/confluence/display/CAMEL/How+can+I+stop+a+route+from+a+route
I will kick a manual export to see if that fixes the static page.
> Babak
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/File-consumer-waiting-for-it-to-start-tp5455062p5461593.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
--
Claus Ibsen
-----------------
FuseSource
Email: cibsen@fusesource.com
Web: http://fusesource.com
Twitter: davsclaus, fusenews
Blog: http://davsclaus.blogspot.com/
Author of Camel in Action: http://www.manning.com/ibsen/
Re: File consumer - waiting for it to start
Posted by Babak Vahdat <ba...@swissonline.ch>.
Bilgin,
It has to do with the sync between the Wiki and the Apache Camel site which
goes wrong [1] from time to time.
[1]
http://camel.465427.n5.nabble.com/code-snippets-broken-on-confluence-wiki-td5062145.html
Babak
--
View this message in context: http://camel.465427.n5.nabble.com/File-consumer-waiting-for-it-to-start-tp5455062p5461593.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: File consumer - waiting for it to start
Posted by Larry Meadors <la...@gmail.com>.
You're the man, Claus. ;-)
I added a "direct:stop" route to my builder with the cleanup and
shutdown code in it.
I changed my route to send a message to it using
"onCompletion().onWhen(header(Exchange.BATCH_COMPLETE).isEqualTo(true))",
and blammo!
It all works exactly how I wanted it to.
Amazing what one line will do. :)
Thanks! Off to the test environment to get it verified by the client. :D
Larry
On Tue, Feb 7, 2012 at 12:19 AM, Claus Ibsen <cl...@gmail.com> wrote:
> On Tue, Feb 7, 2012 at 4:41 AM, Larry Meadors <la...@gmail.com> wrote:
>> Just had a thought - couldn't I use the file component as a batch consumer,
>> then know when it had consumed all of the files and trigger my cleanup code
>> based on that, then have that trigger the shutdown?
>>
>
> Yes the file consumer is a batch consumer, so it enrich the message
> with details.
>
> You can just add add a onWhen predicate to your onCompletion, to only
> trigger when the batch is complete.
> Exchange.BATCH_COMPLETE == true
>
> Something alike
>
> onCompletion().onWhen(header(Exchange.BATCH_COMPLETE).isEqualTo(true))
> .xxxx
> .end();
>
>
>
>
>> Not at my computer at the moment, I'll have to try it in the morning.
>>
>> Larry
>> On Feb 6, 2012 4:22 PM, "Larry Meadors" <la...@gmail.com> wrote:
>>
>>> Wow, this thread is growing and what I thought was a simple task is
>>> turning into a bit of a mess. :-/
>>>
>>> Here's what I'm trying to accomplish:
>>> - I have a directory with files in it
>>> - I want to process every file in that directory
>>> - Once I have processed *all* of the files, I want my app to execute
>>> some cleanup code and then shut down
>>>
>>> The original issue was that when I started up the app, it shut down
>>> immediately and didn't process any files...so I changed my code to use
>>> the Main class to keep it running until it was done.
>>>
>>> That worked, but then it wouldn't *stop* and I couldn't run my cleanup
>>> code.
>>>
>>> It processed all the files, but stayed running...so I added the
>>> shutdown thing in an onCompletion block, thinking (erroneously) that
>>> the onCompletion code would run at the end after ALL files were
>>> processed.
>>>
>>> Now I see that the code in onCompletion() is running after every file
>>> is processed...so it gets one or two files, then shuts down. Which
>>> looks like the expected behavior, now that I'm re-reading that section
>>> in the CIA book.
>>>
>>> Can someone point me to an example that shows how to do what I'm
>>> trying to do here?
>>>
>>> The issue seems to be knowing when I'm done (all the files have been
>>> consumed). Is there a way to do that? Is it documented anywhere?
>>>
>>> Does sendEmptyMessageWhenIdle=true mean that it'll send a null message
>>> after processing all of the files or only if there are no files when
>>> it starts?
>>>
>>> Larry
>>>
>
>
>
> --
> Claus Ibsen
> -----------------
> FuseSource
> Email: cibsen@fusesource.com
> Web: http://fusesource.com
> Twitter: davsclaus, fusenews
> Blog: http://davsclaus.blogspot.com/
> Author of Camel in Action: http://www.manning.com/ibsen/
Re: File consumer - waiting for it to start
Posted by Claus Ibsen <cl...@gmail.com>.
On Tue, Feb 7, 2012 at 4:41 AM, Larry Meadors <la...@gmail.com> wrote:
> Just had a thought - couldn't I use the file component as a batch consumer,
> then know when it had consumed all of the files and trigger my cleanup code
> based on that, then have that trigger the shutdown?
>
Yes the file consumer is a batch consumer, so it enrich the message
with details.
You can just add add a onWhen predicate to your onCompletion, to only
trigger when the batch is complete.
Exchange.BATCH_COMPLETE == true
Something alike
onCompletion().onWhen(header(Exchange.BATCH_COMPLETE).isEqualTo(true))
.xxxx
.end();
> Not at my computer at the moment, I'll have to try it in the morning.
>
> Larry
> On Feb 6, 2012 4:22 PM, "Larry Meadors" <la...@gmail.com> wrote:
>
>> Wow, this thread is growing and what I thought was a simple task is
>> turning into a bit of a mess. :-/
>>
>> Here's what I'm trying to accomplish:
>> - I have a directory with files in it
>> - I want to process every file in that directory
>> - Once I have processed *all* of the files, I want my app to execute
>> some cleanup code and then shut down
>>
>> The original issue was that when I started up the app, it shut down
>> immediately and didn't process any files...so I changed my code to use
>> the Main class to keep it running until it was done.
>>
>> That worked, but then it wouldn't *stop* and I couldn't run my cleanup
>> code.
>>
>> It processed all the files, but stayed running...so I added the
>> shutdown thing in an onCompletion block, thinking (erroneously) that
>> the onCompletion code would run at the end after ALL files were
>> processed.
>>
>> Now I see that the code in onCompletion() is running after every file
>> is processed...so it gets one or two files, then shuts down. Which
>> looks like the expected behavior, now that I'm re-reading that section
>> in the CIA book.
>>
>> Can someone point me to an example that shows how to do what I'm
>> trying to do here?
>>
>> The issue seems to be knowing when I'm done (all the files have been
>> consumed). Is there a way to do that? Is it documented anywhere?
>>
>> Does sendEmptyMessageWhenIdle=true mean that it'll send a null message
>> after processing all of the files or only if there are no files when
>> it starts?
>>
>> Larry
>>
--
Claus Ibsen
-----------------
FuseSource
Email: cibsen@fusesource.com
Web: http://fusesource.com
Twitter: davsclaus, fusenews
Blog: http://davsclaus.blogspot.com/
Author of Camel in Action: http://www.manning.com/ibsen/
Re: File consumer - waiting for it to start
Posted by Larry Meadors <la...@gmail.com>.
Just had a thought - couldn't I use the file component as a batch consumer,
then know when it had consumed all of the files and trigger my cleanup code
based on that, then have that trigger the shutdown?
Not at my computer at the moment, I'll have to try it in the morning.
Larry
On Feb 6, 2012 4:22 PM, "Larry Meadors" <la...@gmail.com> wrote:
> Wow, this thread is growing and what I thought was a simple task is
> turning into a bit of a mess. :-/
>
> Here's what I'm trying to accomplish:
> - I have a directory with files in it
> - I want to process every file in that directory
> - Once I have processed *all* of the files, I want my app to execute
> some cleanup code and then shut down
>
> The original issue was that when I started up the app, it shut down
> immediately and didn't process any files...so I changed my code to use
> the Main class to keep it running until it was done.
>
> That worked, but then it wouldn't *stop* and I couldn't run my cleanup
> code.
>
> It processed all the files, but stayed running...so I added the
> shutdown thing in an onCompletion block, thinking (erroneously) that
> the onCompletion code would run at the end after ALL files were
> processed.
>
> Now I see that the code in onCompletion() is running after every file
> is processed...so it gets one or two files, then shuts down. Which
> looks like the expected behavior, now that I'm re-reading that section
> in the CIA book.
>
> Can someone point me to an example that shows how to do what I'm
> trying to do here?
>
> The issue seems to be knowing when I'm done (all the files have been
> consumed). Is there a way to do that? Is it documented anywhere?
>
> Does sendEmptyMessageWhenIdle=true mean that it'll send a null message
> after processing all of the files or only if there are no files when
> it starts?
>
> Larry
>
Re: File consumer - waiting for it to start
Posted by Babak Vahdat <ba...@swissonline.ch>.
> Can someone point me to an example that shows how to do what I'm
> trying to do here?
Claus has provided 2 new unit-tests based on the discussion on this thread:
https://svn.apache.org/repos/asf/camel/trunk/camel-core/src/test/java/org/apache/camel/issues/StopCamelFromRouteTest.java
https://svn.apache.org/repos/asf/camel/trunk/camel-core/src/test/java/org/apache/camel/issues/StopCamelFromRouteTest.java
> The issue seems to be knowing when I'm done (all the files have been
> consumed). Is there a way to do that? Is it documented anywhere?
You could make use of your own shutdown strategy for this purpose which then
would be done *once* in contrast to onCompletion() clause on your route:
http://camel.apache.org/graceful-shutdown.html
But you could also do your cleanup logic completely decoupled from Camel,
that's as soon as Camel Context has been shut down do whatsoever you want to
do! But if you want to do that logic using Camel then your own shutdown
strategy is a possible option for this purpose.
> Does sendEmptyMessageWhenIdle=true mean that it'll send a null message
> after processing all of the files or only if there are no files when
> it starts?
It will do that as soon as the condition is given! that's no more file to
pickup. This condition could be either given right after your startup (the
directory was empty) or after being up for a while and having processed all
existing files.
Babak
--
View this message in context: http://camel.465427.n5.nabble.com/File-consumer-waiting-for-it-to-start-tp5455062p5462431.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: File consumer - waiting for it to start
Posted by Larry Meadors <la...@gmail.com>.
Wow, this thread is growing and what I thought was a simple task is
turning into a bit of a mess. :-/
Here's what I'm trying to accomplish:
- I have a directory with files in it
- I want to process every file in that directory
- Once I have processed *all* of the files, I want my app to execute
some cleanup code and then shut down
The original issue was that when I started up the app, it shut down
immediately and didn't process any files...so I changed my code to use
the Main class to keep it running until it was done.
That worked, but then it wouldn't *stop* and I couldn't run my cleanup code.
It processed all the files, but stayed running...so I added the
shutdown thing in an onCompletion block, thinking (erroneously) that
the onCompletion code would run at the end after ALL files were
processed.
Now I see that the code in onCompletion() is running after every file
is processed...so it gets one or two files, then shuts down. Which
looks like the expected behavior, now that I'm re-reading that section
in the CIA book.
Can someone point me to an example that shows how to do what I'm
trying to do here?
The issue seems to be knowing when I'm done (all the files have been
consumed). Is there a way to do that? Is it documented anywhere?
Does sendEmptyMessageWhenIdle=true mean that it'll send a null message
after processing all of the files or only if there are no files when
it starts?
Larry
Re: File consumer - waiting for it to start
Posted by Bilgin Ibryam <bi...@gmail.com>.
It looks like there is problem on that page, because instead of source
snippets I see this error message
"Error formatting macro: snippet: java.lang.IndexOutOfBoundsException:
Index: 20, Size: 20 "
No idea where it is coming from, because in edit/preview mode, the
page is rendered fine
Bilgin
On 6 February 2012 06:07, Claus Ibsen <cl...@gmail.com> wrote:
> Hi
>
> I have updated the FAQ with better examples how to stop a route (or
> Camel) from a route
> https://cwiki.apache.org/confluence/display/CAMEL/How+can+I+stop+a+route+from+a+route
>
> The source code for the example will be committed, when ASF SVN again works.
>
> There was in fact more problems with the old FAQ to stop a route using
> the same thread.
> Not only the problem identified by Babak, but you also have problem,
> that when the route is stopped
> it may close resources and whatnot currently in-use, that the current
> message was needed to use
> when it is done. For example a JMS consumer with TX would need to ack
> the message etc.
> A mail message need to set a SEEN flag on the mail etc.
>
> So spinning off a new thread to stop a route is best practice, this is
> also how you would normally stop routes, or the application itself.
>
> On Sun, Feb 5, 2012 at 4:34 PM, Claus Ibsen <cl...@gmail.com> wrote:
>> Hi
>>
>> No we should not do that.
>>
>> People should really not stop a route using the same thread that is
>> process currently messages from a route. But instead let another
>> thread stop the route.
>>
>> There is no good solution for stopping a route with the same thread in
>> mid flight of processing messages on a route. Its like sawing off the
>> branch you are sitting on.
>>
>> I have added a note on the FAQ.
>>
>>
>>
>> On Sun, Feb 5, 2012 at 4:28 PM, Babak Vahdat
>> <ba...@swissonline.ch> wrote:
>>> Hi Claus,
>>>
>>> with the usage of ScheduledBatchPollingConsumer and it's derivates inside
>>> routes there's an issue while doing context.stopRoute("myCoolRoute") as
>>> Camel tries to shutdown the route gracefully and as the mentioned class
>>> does:
>>>
>>> if (answer == 0 && isPolling()) {
>>> // force at least one pending exchange if we are polling as
>>> there is a little gap
>>> // in the processBatch method and until an exchange gets
>>> enlisted as in-flight
>>> // which happens later, so we need to signal back to the
>>> shutdown strategy that
>>> // there is a pending exchange. When we are no longer polling,
>>> then we will return 0
>>> LOG.trace("Currently polling so returning 1 as pending
>>> exchanges");
>>> answer = 1;
>>> }
>>>
>>> it waits and waits until the (30 seconds) timeout reaches although no *real*
>>> inflight exchanges are available at all! I realized this behavior as I was
>>> trying to provide a working example for the user (you see already by this
>>> thread).
>>>
>>> However doing main.stop() (as by the upper example) did *not* behave like
>>> this!
>>>
>>> A possible fix (may be dirty) would be to check that if we already returned
>>> X times a hard-coded value of 1 then we should better return 0 instead to
>>> signal there's no real inflight exchange available.
>>>
>>> Babak
>>>
>>>
>>> --
>>> View this message in context: http://camel.465427.n5.nabble.com/File-consumer-waiting-for-it-to-start-tp5455062p5458048.html
>>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>
>>
>>
>> --
>> Claus Ibsen
>> -----------------
>> FuseSource
>> Email: cibsen@fusesource.com
>> Web: http://fusesource.com
>> Twitter: davsclaus, fusenews
>> Blog: http://davsclaus.blogspot.com/
>> Author of Camel in Action: http://www.manning.com/ibsen/
>
>
>
> --
> Claus Ibsen
> -----------------
> FuseSource
> Email: cibsen@fusesource.com
> Web: http://fusesource.com
> Twitter: davsclaus, fusenews
> Blog: http://davsclaus.blogspot.com/
> Author of Camel in Action: http://www.manning.com/ibsen/
Re: File consumer - waiting for it to start
Posted by Claus Ibsen <cl...@gmail.com>.
Hi
I have updated the FAQ with better examples how to stop a route (or
Camel) from a route
https://cwiki.apache.org/confluence/display/CAMEL/How+can+I+stop+a+route+from+a+route
The source code for the example will be committed, when ASF SVN again works.
There was in fact more problems with the old FAQ to stop a route using
the same thread.
Not only the problem identified by Babak, but you also have problem,
that when the route is stopped
it may close resources and whatnot currently in-use, that the current
message was needed to use
when it is done. For example a JMS consumer with TX would need to ack
the message etc.
A mail message need to set a SEEN flag on the mail etc.
So spinning off a new thread to stop a route is best practice, this is
also how you would normally stop routes, or the application itself.
On Sun, Feb 5, 2012 at 4:34 PM, Claus Ibsen <cl...@gmail.com> wrote:
> Hi
>
> No we should not do that.
>
> People should really not stop a route using the same thread that is
> process currently messages from a route. But instead let another
> thread stop the route.
>
> There is no good solution for stopping a route with the same thread in
> mid flight of processing messages on a route. Its like sawing off the
> branch you are sitting on.
>
> I have added a note on the FAQ.
>
>
>
> On Sun, Feb 5, 2012 at 4:28 PM, Babak Vahdat
> <ba...@swissonline.ch> wrote:
>> Hi Claus,
>>
>> with the usage of ScheduledBatchPollingConsumer and it's derivates inside
>> routes there's an issue while doing context.stopRoute("myCoolRoute") as
>> Camel tries to shutdown the route gracefully and as the mentioned class
>> does:
>>
>> if (answer == 0 && isPolling()) {
>> // force at least one pending exchange if we are polling as
>> there is a little gap
>> // in the processBatch method and until an exchange gets
>> enlisted as in-flight
>> // which happens later, so we need to signal back to the
>> shutdown strategy that
>> // there is a pending exchange. When we are no longer polling,
>> then we will return 0
>> LOG.trace("Currently polling so returning 1 as pending
>> exchanges");
>> answer = 1;
>> }
>>
>> it waits and waits until the (30 seconds) timeout reaches although no *real*
>> inflight exchanges are available at all! I realized this behavior as I was
>> trying to provide a working example for the user (you see already by this
>> thread).
>>
>> However doing main.stop() (as by the upper example) did *not* behave like
>> this!
>>
>> A possible fix (may be dirty) would be to check that if we already returned
>> X times a hard-coded value of 1 then we should better return 0 instead to
>> signal there's no real inflight exchange available.
>>
>> Babak
>>
>>
>> --
>> View this message in context: http://camel.465427.n5.nabble.com/File-consumer-waiting-for-it-to-start-tp5455062p5458048.html
>> Sent from the Camel - Users mailing list archive at Nabble.com.
>
>
>
> --
> Claus Ibsen
> -----------------
> FuseSource
> Email: cibsen@fusesource.com
> Web: http://fusesource.com
> Twitter: davsclaus, fusenews
> Blog: http://davsclaus.blogspot.com/
> Author of Camel in Action: http://www.manning.com/ibsen/
--
Claus Ibsen
-----------------
FuseSource
Email: cibsen@fusesource.com
Web: http://fusesource.com
Twitter: davsclaus, fusenews
Blog: http://davsclaus.blogspot.com/
Author of Camel in Action: http://www.manning.com/ibsen/
Re: File consumer - waiting for it to start
Posted by Claus Ibsen <cl...@gmail.com>.
Hi
No we should not do that.
People should really not stop a route using the same thread that is
process currently messages from a route. But instead let another
thread stop the route.
There is no good solution for stopping a route with the same thread in
mid flight of processing messages on a route. Its like sawing off the
branch you are sitting on.
I have added a note on the FAQ.
On Sun, Feb 5, 2012 at 4:28 PM, Babak Vahdat
<ba...@swissonline.ch> wrote:
> Hi Claus,
>
> with the usage of ScheduledBatchPollingConsumer and it's derivates inside
> routes there's an issue while doing context.stopRoute("myCoolRoute") as
> Camel tries to shutdown the route gracefully and as the mentioned class
> does:
>
> if (answer == 0 && isPolling()) {
> // force at least one pending exchange if we are polling as
> there is a little gap
> // in the processBatch method and until an exchange gets
> enlisted as in-flight
> // which happens later, so we need to signal back to the
> shutdown strategy that
> // there is a pending exchange. When we are no longer polling,
> then we will return 0
> LOG.trace("Currently polling so returning 1 as pending
> exchanges");
> answer = 1;
> }
>
> it waits and waits until the (30 seconds) timeout reaches although no *real*
> inflight exchanges are available at all! I realized this behavior as I was
> trying to provide a working example for the user (you see already by this
> thread).
>
> However doing main.stop() (as by the upper example) did *not* behave like
> this!
>
> A possible fix (may be dirty) would be to check that if we already returned
> X times a hard-coded value of 1 then we should better return 0 instead to
> signal there's no real inflight exchange available.
>
> Babak
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/File-consumer-waiting-for-it-to-start-tp5455062p5458048.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
--
Claus Ibsen
-----------------
FuseSource
Email: cibsen@fusesource.com
Web: http://fusesource.com
Twitter: davsclaus, fusenews
Blog: http://davsclaus.blogspot.com/
Author of Camel in Action: http://www.manning.com/ibsen/
Re: File consumer - waiting for it to start
Posted by Babak Vahdat <ba...@swissonline.ch>.
Hi Claus,
with the usage of ScheduledBatchPollingConsumer and it's derivates inside
routes there's an issue while doing context.stopRoute("myCoolRoute") as
Camel tries to shutdown the route gracefully and as the mentioned class
does:
if (answer == 0 && isPolling()) {
// force at least one pending exchange if we are polling as
there is a little gap
// in the processBatch method and until an exchange gets
enlisted as in-flight
// which happens later, so we need to signal back to the
shutdown strategy that
// there is a pending exchange. When we are no longer polling,
then we will return 0
LOG.trace("Currently polling so returning 1 as pending
exchanges");
answer = 1;
}
it waits and waits until the (30 seconds) timeout reaches although no *real*
inflight exchanges are available at all! I realized this behavior as I was
trying to provide a working example for the user (you see already by this
thread).
However doing main.stop() (as by the upper example) did *not* behave like
this!
A possible fix (may be dirty) would be to check that if we already returned
X times a hard-coded value of 1 then we should better return 0 instead to
signal there's no real inflight exchange available.
Babak
--
View this message in context: http://camel.465427.n5.nabble.com/File-consumer-waiting-for-it-to-start-tp5455062p5458048.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: File consumer - waiting for it to start
Posted by Claus Ibsen <cl...@gmail.com>.
Hi
The FAQ
http://camel.apache.org/how-can-i-stop-a-route-from-a-route.html
We should probably add a note to that given FAQ, that stopping a route
from a route is further complicated if you use the current thread
doing so. Its best practice to just flip a flag or some sort of
signals that the route should be stopped. And then continue routing,
to let Camel do its business.
Then the 3rd part can check the flag, and then commence a route stop.
For example using a CountdownLatch
final CountdownLatch latch = new CountdownLatch(1);
camelContext.start();
// wait for latch before stopping
latch.await();
camelContext.stop();
and then from the route you can countdown the latch when you are done
with the routing.
On Fri, Feb 3, 2012 at 10:15 PM, Larry Meadors <la...@gmail.com> wrote:
> I have what I think is a simple route:
>
>
> public void configure() throws Exception {
> from(sourceUri)
> .setHeader(S3Constants.KEY, simple("${header.CamelFileNameOnly}"))
> .setHeader(S3Constants.CONTENT_LENGTH, simple("${header.CamelFileLength}"))
> .to(destinationUri);
> }
>
>
> It's just taking files from a directory and copying them to an s3 bucket.
>
> What I'm seeing is that when I start it up, it stops immediately, so I
> end up doing this:
>
>
> // create the route - the parameters are the source, then the destination
> DirectCopyRoute directCopyRoute = new DirectCopyRoute(arg[0], arg[1]);
>
> // create the camel context and start it
> CamelContext camelContext = new DefaultCamelContext();
> camelContext.addRoutes(directCopyRoute);
> camelContext.start();
>
> // give it time to realize it has work to do
> Thread.sleep(1000);
>
>
> If I remove the Thread.sleep call, it just starts and stops.
>
> Adding that seems to make it work, but I'm wondering why?
>
> I'm guessing the "why" is that it takes a bit of time to create the
> list of files to process, and since there is nothing pending to
> process, camel figures it's done what it was supposed to, and shuts
> down - adding the delay gives the file consumer time to create the
> list of files to process, and once it's started, it goes until it's
> done.
>
> It that's the case, it leads to a second question: Will 1 second be
> long enough to work if the source directory contains a LOT of files?
>
> Normally, the directory will only have a few hundred files, but it may
> have a few thousand, and it may be accessing a shared directory over a
> network, so the time required to get a directory listing may vary.
>
> Is there a smarter way to do this?
>
> Larry
--
Claus Ibsen
-----------------
FuseSource
Email: cibsen@fusesource.com
Web: http://fusesource.com
Twitter: davsclaus, fusenews
Blog: http://davsclaus.blogspot.com/
Author of Camel in Action: http://www.manning.com/ibsen/