You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@camel.apache.org by Chirag Dewan <ch...@yahoo.in> on 2014/06/19 07:11:08 UTC
A basic query with using Direct Component
Hi All,
I have a route with direct as following :
from("direct:abc").to("hdfs2:xxxxx?splitStrategy=MESSAGE");
I use the route write files in HDFS. I use the producer template to send a OutputStream to direct:
producerTemplate,sendBody("direct:abc",OutputStream);
Now HDFS producer creates an .opened file first and renames it upon writing the file.
So,I will only get the callback from direct,once the file is written and renamed. Is this correct?
Somehow my route has finished but .opene files is still in the HDFS directory,which in turn is polled by a HDFS consumer who fail to poll the .opened file.
BR,
Chirag
Re: A basic query with using Direct Component
Posted by Chirag Dewan <ch...@yahoo.in>.
Got the problem.
The issue was actually with HDFSConsumer-FTPProducer route. HDFS consumer creates a .opened file before reading and FTP distribution was failing,due to which .opened files remains in the HDFS directory. And these files are not picked once the distribution resumes.
Now I can try and use an empty string suffix for the consumer route so that it does not ignore files.
Thanks for the assistance.
Chirag
________________________________
From: Claus Ibsen <cl...@gmail.com>
To: "users@camel.apache.org" <us...@camel.apache.org>
Sent: Thursday, 19 June 2014 12:16 PM
Subject: Re: A basic query with using Direct Component
I am not sure, maybe try setting the openedSuffix to an empty string
to see if that works?
On Thu, Jun 19, 2014 at 8:39 AM, Chirag Dewan <ch...@yahoo.in> wrote:
> Hi Claus,
>
> Thanks for response.
>
> I am using Camel 2.12.1 with camel-hdfs 2.
>
> Yes,that is the problem. Write operation fails and .opened file is still there on HDFS. Is there any option through which I can skip creating .opened file?
>
>
> Chirag
>
>
>
>
>
>
>
>
>
>
> ________________________________
> From: Claus Ibsen <cl...@gmail.com>
> To: "users@camel.apache.org" <us...@camel.apache.org>
> Sent: Thursday, 19 June 2014 11:37 AM
> Subject: Re: A basic query with using Direct Component
>
>
> On Thu, Jun 19, 2014 at 7:11 AM, Chirag Dewan <ch...@yahoo.in> wrote:
>> Hi All,
>>
>> I have a route with direct as following :
>>
>> from("direct:abc").to("hdfs2:xxxxx?splitStrategy=MESSAGE");
>>
>> I use the route write files in HDFS. I use the producer template to send a OutputStream to direct:
>>
>> producerTemplate,sendBody("direct:abc",OutputStream);
>>
>> Now HDFS producer creates an .opened file first and renames it upon writing the file.
>>
>>
>> So,I will only get the callback from direct,once the file is written and renamed. Is this correct?
>>
>
> Yes
>
>>
>> Somehow my route has finished but .opene files is still in the HDFS directory,which in turn is polled by a HDFS consumer who fail to poll the .opened file.
>>
>
> I suspect either the hdfs write failed or something, leaving the
> .opened file orphaned.
>
> And the consumer ought not to poll the .opened file. I would have
> assumed the consumer skips these kind of files.
>
> What version of Camel do you use?
>
>
>
>
>
>>
>> BR,
>>
>> Chirag
>
>
>
> --
> Claus Ibsen
> -----------------
> Red Hat, Inc.
> Email: cibsen@redhat.com
> Twitter: davsclaus
> Blog: http://davsclaus.com
> Author of Camel in Action: http://www.manning.com/ibsen
> hawtio: http://hawt.io/
> fabric8: http://fabric8.io/
--
Claus Ibsen
-----------------
Red Hat, Inc.
Email: cibsen@redhat.com
Twitter: davsclaus
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen
hawtio: http://hawt.io/
fabric8: http://fabric8.io/
Re: A basic query with using Direct Component
Posted by Claus Ibsen <cl...@gmail.com>.
I am not sure, maybe try setting the openedSuffix to an empty string
to see if that works?
On Thu, Jun 19, 2014 at 8:39 AM, Chirag Dewan <ch...@yahoo.in> wrote:
> Hi Claus,
>
> Thanks for response.
>
> I am using Camel 2.12.1 with camel-hdfs 2.
>
> Yes,that is the problem. Write operation fails and .opened file is still there on HDFS. Is there any option through which I can skip creating .opened file?
>
>
> Chirag
>
>
>
>
>
>
>
>
>
>
> ________________________________
> From: Claus Ibsen <cl...@gmail.com>
> To: "users@camel.apache.org" <us...@camel.apache.org>
> Sent: Thursday, 19 June 2014 11:37 AM
> Subject: Re: A basic query with using Direct Component
>
>
> On Thu, Jun 19, 2014 at 7:11 AM, Chirag Dewan <ch...@yahoo.in> wrote:
>> Hi All,
>>
>> I have a route with direct as following :
>>
>> from("direct:abc").to("hdfs2:xxxxx?splitStrategy=MESSAGE");
>>
>> I use the route write files in HDFS. I use the producer template to send a OutputStream to direct:
>>
>> producerTemplate,sendBody("direct:abc",OutputStream);
>>
>> Now HDFS producer creates an .opened file first and renames it upon writing the file.
>>
>>
>> So,I will only get the callback from direct,once the file is written and renamed. Is this correct?
>>
>
> Yes
>
>>
>> Somehow my route has finished but .opene files is still in the HDFS directory,which in turn is polled by a HDFS consumer who fail to poll the .opened file.
>>
>
> I suspect either the hdfs write failed or something, leaving the
> .opened file orphaned.
>
> And the consumer ought not to poll the .opened file. I would have
> assumed the consumer skips these kind of files.
>
> What version of Camel do you use?
>
>
>
>
>
>>
>> BR,
>>
>> Chirag
>
>
>
> --
> Claus Ibsen
> -----------------
> Red Hat, Inc.
> Email: cibsen@redhat.com
> Twitter: davsclaus
> Blog: http://davsclaus.com
> Author of Camel in Action: http://www.manning.com/ibsen
> hawtio: http://hawt.io/
> fabric8: http://fabric8.io/
--
Claus Ibsen
-----------------
Red Hat, Inc.
Email: cibsen@redhat.com
Twitter: davsclaus
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen
hawtio: http://hawt.io/
fabric8: http://fabric8.io/
Re: A basic query with using Direct Component
Posted by Chirag Dewan <ch...@yahoo.in>.
Hi Claus,
Thanks for response.
I am using Camel 2.12.1 with camel-hdfs 2.
Yes,that is the problem. Write operation fails and .opened file is still there on HDFS. Is there any option through which I can skip creating .opened file?
Chirag
________________________________
From: Claus Ibsen <cl...@gmail.com>
To: "users@camel.apache.org" <us...@camel.apache.org>
Sent: Thursday, 19 June 2014 11:37 AM
Subject: Re: A basic query with using Direct Component
On Thu, Jun 19, 2014 at 7:11 AM, Chirag Dewan <ch...@yahoo.in> wrote:
> Hi All,
>
> I have a route with direct as following :
>
> from("direct:abc").to("hdfs2:xxxxx?splitStrategy=MESSAGE");
>
> I use the route write files in HDFS. I use the producer template to send a OutputStream to direct:
>
> producerTemplate,sendBody("direct:abc",OutputStream);
>
> Now HDFS producer creates an .opened file first and renames it upon writing the file.
>
>
> So,I will only get the callback from direct,once the file is written and renamed. Is this correct?
>
Yes
>
> Somehow my route has finished but .opene files is still in the HDFS directory,which in turn is polled by a HDFS consumer who fail to poll the .opened file.
>
I suspect either the hdfs write failed or something, leaving the
.opened file orphaned.
And the consumer ought not to poll the .opened file. I would have
assumed the consumer skips these kind of files.
What version of Camel do you use?
>
> BR,
>
> Chirag
--
Claus Ibsen
-----------------
Red Hat, Inc.
Email: cibsen@redhat.com
Twitter: davsclaus
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen
hawtio: http://hawt.io/
fabric8: http://fabric8.io/
Re: A basic query with using Direct Component
Posted by Claus Ibsen <cl...@gmail.com>.
On Thu, Jun 19, 2014 at 7:11 AM, Chirag Dewan <ch...@yahoo.in> wrote:
> Hi All,
>
> I have a route with direct as following :
>
> from("direct:abc").to("hdfs2:xxxxx?splitStrategy=MESSAGE");
>
> I use the route write files in HDFS. I use the producer template to send a OutputStream to direct:
>
> producerTemplate,sendBody("direct:abc",OutputStream);
>
> Now HDFS producer creates an .opened file first and renames it upon writing the file.
>
>
> So,I will only get the callback from direct,once the file is written and renamed. Is this correct?
>
Yes
>
> Somehow my route has finished but .opene files is still in the HDFS directory,which in turn is polled by a HDFS consumer who fail to poll the .opened file.
>
I suspect either the hdfs write failed or something, leaving the
.opened file orphaned.
And the consumer ought not to poll the .opened file. I would have
assumed the consumer skips these kind of files.
What version of Camel do you use?
>
> BR,
>
> Chirag
--
Claus Ibsen
-----------------
Red Hat, Inc.
Email: cibsen@redhat.com
Twitter: davsclaus
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen
hawtio: http://hawt.io/
fabric8: http://fabric8.io/