You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@camel.apache.org by Ambika Das <da...@outlook.com> on 2017/05/18 06:58:04 UTC
Camel processes only the first 150 rows in CSV
Hi All,
I am developing a aggregator application which is supposed to consume a CSV and generate the message. I have developed it as a Spring boot application. The problem is when I transfer a file it takes only the first 150 record and processes. The file is still there in the folder. If I restart the application the whole file gets processed.
my code looks as follows.
@override
public void configure() {
String end point = "sftp://server:port/dropbox?delay=10000&moveFailed=failed&noop=true&preMove=done"
from(endpoint).to("direct:file_Processor")
}
Regards,
Prasad
Re: Camel processes only the first 150 rows in CSV
Posted by Ambika Das <da...@outlook.com>.
I put the file in the SFTP drop box and started the application. It takes up the first 150 rows for processing.
Any reason why Camel skips rest of the rows?
Regards,
Prasad
________________________________
From: Ambika Das <da...@outlook.com>
Sent: Thursday, May 18, 2017 12:28 PM
To: users@camel.apache.org
Subject: Camel processes only the first 150 rows in CSV
Hi All,
I am developing a aggregator application which is supposed to consume a CSV and generate the message. I have developed it as a Spring boot application. The problem is when I transfer a file it takes only the first 150 record and processes. The file is still there in the folder. If I restart the application the whole file gets processed.
my code looks as follows.
@override
public void configure() {
String end point = "sftp://server:port/dropbox?delay=10000&moveFailed=failed&noop=true&preMove=done"
from(endpoint).to("direct:file_Processor")
}
Regards,
Prasad
Re: Camel processes only the first 150 rows in CSV
Posted by Ambika Das <da...@outlook.com>.
I am using currently readLock and doneFileName, but the issue persists.
sftp://server:port/data?delay=10000&doneFileName=${file:name}.done&move=.done&moveFailed=failed&noop=true&password=xxxxxx&readLock=changed&username=abcd
Regards,
Ambika
________________________________
From: Ambika Das <da...@outlook.com>
Sent: Thursday, May 18, 2017 1:50 PM
To: users@camel.apache.org
Subject: Re: Camel processes only the first 150 rows in CSV
I changed the code and my end point looks like this.
"sftp://server:port/dropbox?delay=10000&doneFileName=done&idempotent=true&moveFailed=failed&noop=true&password=xxxxxx&preMove=done&readLock=true&username=sftpuser1"
It is still stopping at 150 rows.
Regards,
Prasad
________________________________
From: Zoran Regvart <zo...@regvart.com>
Sent: Thursday, May 18, 2017 1:10 PM
To: users@camel.apache.org
Subject: Re: Camel processes only the first 150 rows in CSV
Hi,
you have specified `noop=true` and I guess you didn't configure
`idempotent=true` and `idempotentRepository` (if you wish to preserve
the state of processing between nodes and restarts) -- that keeps the
file as is on the SFTP server.
Also the file might be half-consumed if you're not using
`doneFileName`. It works like this: Camel notices a file on the SFTP
server and starts processing it while the app that is placing the file
is still writing to the file. It could be just that the app that is
writing the file in the first place has a buffer, so on first buffer
flush only first 150 lines are written.
The best course of action is to combine `readLock` and `idempotent*`
settings with `doneFileName`, i.e. the app that is writing the file to
the SFTP server leaves another file (a marker file) that indicates
that it's done writing.
zoran
zoran
On Thu, May 18, 2017 at 8:58 AM, Ambika Das <da...@outlook.com> wrote:
> Hi All,
>
>
> I am developing a aggregator application which is supposed to consume a CSV and generate the message. I have developed it as a Spring boot application. The problem is when I transfer a file it takes only the first 150 record and processes. The file is still there in the folder. If I restart the application the whole file gets processed.
>
>
> my code looks as follows.
>
>
> @override
>
> public void configure() {
>
> String end point = "sftp://server:port/dropbox?delay=10000&moveFailed=failed&noop=true&preMove=done"
>
> from(endpoint).to("direct:file_Processor")
>
>
> }
>
>
> Regards,
>
> Prasad
--
Zoran Regvart
Re: Camel processes only the first 150 rows in CSV
Posted by Ambika Das <da...@outlook.com>.
I changed the code and my end point looks like this.
"sftp://server:port/dropbox?delay=10000&doneFileName=done&idempotent=true&moveFailed=failed&noop=true&password=xxxxxx&preMove=done&readLock=true&username=sftpuser1"
It is still stopping at 150 rows.
Regards,
Prasad
________________________________
From: Zoran Regvart <zo...@regvart.com>
Sent: Thursday, May 18, 2017 1:10 PM
To: users@camel.apache.org
Subject: Re: Camel processes only the first 150 rows in CSV
Hi,
you have specified `noop=true` and I guess you didn't configure
`idempotent=true` and `idempotentRepository` (if you wish to preserve
the state of processing between nodes and restarts) -- that keeps the
file as is on the SFTP server.
Also the file might be half-consumed if you're not using
`doneFileName`. It works like this: Camel notices a file on the SFTP
server and starts processing it while the app that is placing the file
is still writing to the file. It could be just that the app that is
writing the file in the first place has a buffer, so on first buffer
flush only first 150 lines are written.
The best course of action is to combine `readLock` and `idempotent*`
settings with `doneFileName`, i.e. the app that is writing the file to
the SFTP server leaves another file (a marker file) that indicates
that it's done writing.
zoran
zoran
On Thu, May 18, 2017 at 8:58 AM, Ambika Das <da...@outlook.com> wrote:
> Hi All,
>
>
> I am developing a aggregator application which is supposed to consume a CSV and generate the message. I have developed it as a Spring boot application. The problem is when I transfer a file it takes only the first 150 record and processes. The file is still there in the folder. If I restart the application the whole file gets processed.
>
>
> my code looks as follows.
>
>
> @override
>
> public void configure() {
>
> String end point = "sftp://server:port/dropbox?delay=10000&moveFailed=failed&noop=true&preMove=done"
>
> from(endpoint).to("direct:file_Processor")
>
>
> }
>
>
> Regards,
>
> Prasad
--
Zoran Regvart
Re: Camel processes only the first 150 rows in CSV
Posted by Zoran Regvart <zo...@regvart.com>.
Hi,
you have specified `noop=true` and I guess you didn't configure
`idempotent=true` and `idempotentRepository` (if you wish to preserve
the state of processing between nodes and restarts) -- that keeps the
file as is on the SFTP server.
Also the file might be half-consumed if you're not using
`doneFileName`. It works like this: Camel notices a file on the SFTP
server and starts processing it while the app that is placing the file
is still writing to the file. It could be just that the app that is
writing the file in the first place has a buffer, so on first buffer
flush only first 150 lines are written.
The best course of action is to combine `readLock` and `idempotent*`
settings with `doneFileName`, i.e. the app that is writing the file to
the SFTP server leaves another file (a marker file) that indicates
that it's done writing.
zoran
zoran
On Thu, May 18, 2017 at 8:58 AM, Ambika Das <da...@outlook.com> wrote:
> Hi All,
>
>
> I am developing a aggregator application which is supposed to consume a CSV and generate the message. I have developed it as a Spring boot application. The problem is when I transfer a file it takes only the first 150 record and processes. The file is still there in the folder. If I restart the application the whole file gets processed.
>
>
> my code looks as follows.
>
>
> @override
>
> public void configure() {
>
> String end point = "sftp://server:port/dropbox?delay=10000&moveFailed=failed&noop=true&preMove=done"
>
> from(endpoint).to("direct:file_Processor")
>
>
> }
>
>
> Regards,
>
> Prasad
--
Zoran Regvart