You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@camel.apache.org by spurcell <sc...@mastercard.com> on 2016/08/30 17:29:23 UTC
Camel BeanIODataFormat Configure ErrorHandler for BeanIO Reader
I am working on a camel project in which we use the BeanIODataFormat to
unmarshall a CSV file. I want to add a custom Error Handler class
(implements BeanReaderErrorHandler) but cannot figure out how this can be
accomplished.
I have a camel route:
from(INBOUND_PROCESSING_ROUTE)
.routeId("inBoundProcessingRoute")
.process(getProcessor(CREATE_FILE_FROM_DATABASE))
.unmarshal(getCsvUnmarshaller())
.to(FILE_SPLITTER);
which calls:
private DataFormat getCsvUnmarshaller() {
//return new
BeanIODataFormat("beanio-request.xml?beanReaderErrorhandler=ErrorHandler.class",
"transferRequests");
return new BeanIODataFormat("beanio-request.xml",
"transferRequests");
}
The BeanIO Docs say you can add a custom ErrorHandler to the reader, but I
cannot find any examples or help on this. Has anyone done this? I tried
adding to the beanio.xml file, but no support there.
If you need to handle an exception and continue processing, it may be
simpler to register a BeanReaderErrorHandler using the
beanReader.setErrorHandler() method. The BeanReaderErrorHandler interface is
shown below. Any exception thrown by the error handler will be rethrown by
the BeanReader.
Thanks
--
View this message in context: http://camel.465427.n5.nabble.com/Camel-BeanIODataFormat-Configure-ErrorHandler-for-BeanIO-Reader-tp5787005.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Camel BeanIODataFormat Configure ErrorHandler for BeanIO Reader
Posted by spurcell <sc...@mastercard.com>.
I tested some bad files yesterday, and the problems is really when a user l=
eaves out a field, then the Runtime gets caught, but we get a ambigious err=
or such as : Array out of bounds, no row, no column, etc. That is why we we=
re hoping to try grabbing an instance of the ErrorHandler which appears to =
give field name, row, etc.
Here are some stack traces that I could catch upstream:
Unexpected record 'fileHeaderRecord' at line 2
java.lang.IndexOutOfBoundsException: Index: 3, Size: 3
Unexpected record 'paymentData' at line 2 Not very helpful errors
Thanks
--
View this message in context: http://camel.465427.n5.nabble.com/Camel-BeanIODataFormat-Configure-ErrorHandler-for-BeanIO-Reader-tp5787005p5787015.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Camel BeanIODataFormat Configure ErrorHandler for BeanIO Reader
Posted by Brad Johnson <br...@mediadriver.com>.
Then the ignoreUnidentifiedRecords probably wouldn't work for you. There's
another one like it but I don't recall it right now. I'm more used to
doing this in Blueprint so it takes me awhile to make the change from one
to the other. What stack trace are you seeing with this?
Brad
On Tue, Aug 30, 2016 at 12:29 PM, spurcell <sc...@mastercard.com>
wrote:
> I am working on a camel project in which we use the BeanIODataFormat to
> unmarshall a CSV file. I want to add a custom Error Handler class
> (implements BeanReaderErrorHandler) but cannot figure out how this can be
> accomplished.
>
> I have a camel route:
> from(INBOUND_PROCESSING_ROUTE)
> .routeId("inBoundProcessingRoute")
> .process(getProcessor(CREATE_FILE_FROM_DATABASE))
> .unmarshal(getCsvUnmarshaller())
> .to(FILE_SPLITTER);
> which calls:
> private DataFormat getCsvUnmarshaller() {
> //return new
> BeanIODataFormat("beanio-request.xml?beanReaderErrorhandler=
> ErrorHandler.class",
> "transferRequests");
> return new BeanIODataFormat("beanio-request.xml",
> "transferRequests");
> }
>
>
> The BeanIO Docs say you can add a custom ErrorHandler to the reader, but I
> cannot find any examples or help on this. Has anyone done this? I tried
> adding to the beanio.xml file, but no support there.
>
> If you need to handle an exception and continue processing, it may be
> simpler to register a BeanReaderErrorHandler using the
> beanReader.setErrorHandler() method. The BeanReaderErrorHandler interface
> is
> shown below. Any exception thrown by the error handler will be rethrown by
> the BeanReader.
>
> Thanks
>
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.
> com/Camel-BeanIODataFormat-Configure-ErrorHandler-for-
> BeanIO-Reader-tp5787005.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>
Re: Camel BeanIODataFormat Configure ErrorHandler for BeanIO Reader
Posted by Brad Johnson <br...@mediadriver.com>.
By the way, I almost forgot to ask. How big are your input files? If they
aren't enormous you can unmarshal the whole file in one gulp and if
something is not right it will break right there without any processing. If
it succeeds you simply iterate over the now unmarshaled beans.
On Thu, Sep 1, 2016 at 1:03 PM, Brad Johnson <br...@mediadriver.com>
wrote:
> If that works then great. If you want to reject the file you'll likely
> have to run the file twice. The first for validation and the second for
> processing. Remember if you're streaming these and processing them and
> then get a bad record like the one with the missing field, you've already
> processed a good number of records and won't be able to reject the file.
> Either that or you'll have to get into some sort of transaction and
> rollback.
>
> Brad
>
> On Wed, Aug 31, 2016 at 1:50 PM, spurcell <sc...@mastercard.com>
> wrote:
>
>> Hello Brad,
>>
>> Here is an expected file (simplified down for discussion)
>> "FHDR","1","t","20160"
>> "LFHDR","p","4","2016"
>> "TRNSFRREQ","p","2","A","4","U","a"
>> "LFTRLR","p","1","4"
>> "FTRLR","5","4"
>>
>>
>> When a user sends in a bad file (2 FHDR files) I want to reject the file.
>> "FHDR","1","t","20160"
>> "FHDR","1","t","20160"
>> "LFHDR","p","4","2016"
>> "TRNSFRREQ","p","2","A","4","U","a"
>> "LFTRLR","p","1","4"
>> "FTRLR","5","4"
>>
>> If a user sends in a record with a missing field (LFHDR missing the "4"
>> column) I would like to capture that in the reader and reject the file.
>>
>> "FHDR","1","t","20160"
>> "LFHDR","p","2016"
>> "TRNSFRREQ","p","2","A","4","U","a"
>> "LFTRLR","p","1","4"
>> "FTRLR","5","4"
>>
>> I did figure out that using 2.18.0-SNAPSHOT works and one can now insert a
>> Handler so I am debugging that right now.
>>
>> Thanks
>>
>>
>>
>> --
>> View this message in context: http://camel.465427.n5.nabble.
>> com/Camel-BeanIODataFormat-Configure-ErrorHandler-for-BeanIO
>> -Reader-tp5787005p5787071.html
>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>
>
>
Re: Camel BeanIODataFormat Configure ErrorHandler for BeanIO Reader
Posted by Brad Johnson <br...@mediadriver.com>.
If that works then great. If you want to reject the file you'll likely
have to run the file twice. The first for validation and the second for
processing. Remember if you're streaming these and processing them and
then get a bad record like the one with the missing field, you've already
processed a good number of records and won't be able to reject the file.
Either that or you'll have to get into some sort of transaction and
rollback.
Brad
On Wed, Aug 31, 2016 at 1:50 PM, spurcell <sc...@mastercard.com>
wrote:
> Hello Brad,
>
> Here is an expected file (simplified down for discussion)
> "FHDR","1","t","20160"
> "LFHDR","p","4","2016"
> "TRNSFRREQ","p","2","A","4","U","a"
> "LFTRLR","p","1","4"
> "FTRLR","5","4"
>
>
> When a user sends in a bad file (2 FHDR files) I want to reject the file.
> "FHDR","1","t","20160"
> "FHDR","1","t","20160"
> "LFHDR","p","4","2016"
> "TRNSFRREQ","p","2","A","4","U","a"
> "LFTRLR","p","1","4"
> "FTRLR","5","4"
>
> If a user sends in a record with a missing field (LFHDR missing the "4"
> column) I would like to capture that in the reader and reject the file.
>
> "FHDR","1","t","20160"
> "LFHDR","p","2016"
> "TRNSFRREQ","p","2","A","4","U","a"
> "LFTRLR","p","1","4"
> "FTRLR","5","4"
>
> I did figure out that using 2.18.0-SNAPSHOT works and one can now insert a
> Handler so I am debugging that right now.
>
> Thanks
>
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.
> com/Camel-BeanIODataFormat-Configure-ErrorHandler-for-BeanIO-Reader-
> tp5787005p5787071.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>
Re: Camel BeanIODataFormat Configure ErrorHandler for BeanIO Reader
Posted by spurcell <sc...@mastercard.com>.
Hello Brad,
Here is an expected file (simplified down for discussion)
"FHDR","1","t","20160"
"LFHDR","p","4","2016"
"TRNSFRREQ","p","2","A","4","U","a"
"LFTRLR","p","1","4"
"FTRLR","5","4"
When a user sends in a bad file (2 FHDR files) I want to reject the file.
"FHDR","1","t","20160"
"FHDR","1","t","20160"
"LFHDR","p","4","2016"
"TRNSFRREQ","p","2","A","4","U","a"
"LFTRLR","p","1","4"
"FTRLR","5","4"
If a user sends in a record with a missing field (LFHDR missing the "4"
column) I would like to capture that in the reader and reject the file.
"FHDR","1","t","20160"
"LFHDR","p","2016"
"TRNSFRREQ","p","2","A","4","U","a"
"LFTRLR","p","1","4"
"FTRLR","5","4"
I did figure out that using 2.18.0-SNAPSHOT works and one can now insert a
Handler so I am debugging that right now.
Thanks
--
View this message in context: http://camel.465427.n5.nabble.com/Camel-BeanIODataFormat-Configure-ErrorHandler-for-BeanIO-Reader-tp5787005p5787071.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Camel BeanIODataFormat Configure ErrorHandler for BeanIO Reader
Posted by Brad Johnson <br...@mediadriver.com>.
Can you put a few lines of an example of your files up here. Just a few
lines and you can modify the data to obscure anything about the company or
data that you don't want known.
Unexpected record 'fileHeaderRecord' at line 2
java.lang.IndexOutOfBoundsException: Index: 3, Size: 3
Unexpected record 'paymentData' at line 2 Not very helpful errors
There are a few things I can think would cause this but without the mapping
or data can't tell. Whatever you are using to identify a fileHeaderRecord
is failing to disambiguate it from whatever the record on the second line
is. It is still trying to parse the second line as a fileHeaderRecord and
throws up its hands when it tries to parse it.
One error I've never seen before is where it thinks that line two is both a
fileHeaderRecord and a paymentData record line two. If you only expect one
fileHeaderRecord make sure you set the min/max to 1.
On Wed, Aug 31, 2016 at 10:28 AM, Brad Johnson <brad.johnson@mediadriver.com
> wrote:
> This would indicate that you have a record in the file that is
> unexpected. While an error handler could help with that you can tell it to
> ignore records with certain characteristics or if there's a couple of
> header and footer lines just add getter/setters for them along with the
> mappying (or, if you wish, add that to your bean). I made the silly mistake
> early on of believing that "rid" did that. Instead it's short for record
> id.
>
> Does your file have a header in it? It's failing on the parsing due to a
> mapping error.
>
> Brad
>
> On Wed, Aug 31, 2016 at 10:05 AM, Claus Ibsen <cl...@gmail.com>
> wrote:
>
>> Camel 2.18.0 is expected to be released in september 2016.
>>
>> You can build or use SNAPSHOT to try latest code
>> http://camel.apache.org/maven-2-snapshot-repository-in-pom.html
>> http://camel.apache.org/building.html
>>
>> On Wed, Aug 31, 2016 at 4:11 PM, spurcell <sc...@mastercard.com>
>> wrote:
>> > GM Claus-Ibsen-2,
>> >
>> > Thanks for the response ... really appreciated that angle. It looks like
>> > that functionality was just added to MASTER (trunk I assume). We are
>> running
>> > on Camel 2.16.1, and the camel-bean docs show this capability in 2.10,
>> so we
>> > were struggling yesterday to understand.
>> >
>> > As we see this functionality under MASTER I think it will be released
>> under
>> > 2.18. Would you happen to know when this release is going to occur? Our
>> team
>> > would like to start testing and using this if it occurs shortly, as we
>> have
>> > a release date prior to Dec.
>> >
>> > Thanks
>> > Scott
>> >
>> >
>> >
>> > --
>> > View this message in context: http://camel.465427.n5.nabble.
>> com/Camel-BeanIODataFormat-Configure-ErrorHandler-for-BeanIO
>> -Reader-tp5787005p5787051.html
>> > Sent from the Camel - Users mailing list archive at Nabble.com.
>>
>>
>>
>> --
>> Claus Ibsen
>> -----------------
>> http://davsclaus.com @davsclaus
>> Camel in Action 2: https://www.manning.com/ibsen2
>>
>
>
Re: Camel BeanIODataFormat Configure ErrorHandler for BeanIO Reader
Posted by Brad Johnson <br...@mediadriver.com>.
This would indicate that you have a record in the file that is unexpected.
While an error handler could help with that you can tell it to ignore
records with certain characteristics or if there's a couple of header and
footer lines just add getter/setters for them along with the mappying (or,
if you wish, add that to your bean). I made the silly mistake early on of
believing that "rid" did that. Instead it's short for record id.
Does your file have a header in it? It's failing on the parsing due to a
mapping error.
Brad
On Wed, Aug 31, 2016 at 10:05 AM, Claus Ibsen <cl...@gmail.com> wrote:
> Camel 2.18.0 is expected to be released in september 2016.
>
> You can build or use SNAPSHOT to try latest code
> http://camel.apache.org/maven-2-snapshot-repository-in-pom.html
> http://camel.apache.org/building.html
>
> On Wed, Aug 31, 2016 at 4:11 PM, spurcell <sc...@mastercard.com>
> wrote:
> > GM Claus-Ibsen-2,
> >
> > Thanks for the response ... really appreciated that angle. It looks like
> > that functionality was just added to MASTER (trunk I assume). We are
> running
> > on Camel 2.16.1, and the camel-bean docs show this capability in 2.10,
> so we
> > were struggling yesterday to understand.
> >
> > As we see this functionality under MASTER I think it will be released
> under
> > 2.18. Would you happen to know when this release is going to occur? Our
> team
> > would like to start testing and using this if it occurs shortly, as we
> have
> > a release date prior to Dec.
> >
> > Thanks
> > Scott
> >
> >
> >
> > --
> > View this message in context: http://camel.465427.n5.nabble.
> com/Camel-BeanIODataFormat-Configure-ErrorHandler-for-BeanIO-Reader-
> tp5787005p5787051.html
> > Sent from the Camel - Users mailing list archive at Nabble.com.
>
>
>
> --
> Claus Ibsen
> -----------------
> http://davsclaus.com @davsclaus
> Camel in Action 2: https://www.manning.com/ibsen2
>
Re: Camel BeanIODataFormat Configure ErrorHandler for BeanIO Reader
Posted by Claus Ibsen <cl...@gmail.com>.
Camel 2.18.0 is expected to be released in september 2016.
You can build or use SNAPSHOT to try latest code
http://camel.apache.org/maven-2-snapshot-repository-in-pom.html
http://camel.apache.org/building.html
On Wed, Aug 31, 2016 at 4:11 PM, spurcell <sc...@mastercard.com> wrote:
> GM Claus-Ibsen-2,
>
> Thanks for the response ... really appreciated that angle. It looks like
> that functionality was just added to MASTER (trunk I assume). We are running
> on Camel 2.16.1, and the camel-bean docs show this capability in 2.10, so we
> were struggling yesterday to understand.
>
> As we see this functionality under MASTER I think it will be released under
> 2.18. Would you happen to know when this release is going to occur? Our team
> would like to start testing and using this if it occurs shortly, as we have
> a release date prior to Dec.
>
> Thanks
> Scott
>
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/Camel-BeanIODataFormat-Configure-ErrorHandler-for-BeanIO-Reader-tp5787005p5787051.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
--
Claus Ibsen
-----------------
http://davsclaus.com @davsclaus
Camel in Action 2: https://www.manning.com/ibsen2
Re: Camel BeanIODataFormat Configure ErrorHandler for BeanIO Reader
Posted by spurcell <sc...@mastercard.com>.
GM Claus-Ibsen-2,
Thanks for the response ... really appreciated that angle. It looks like
that functionality was just added to MASTER (trunk I assume). We are running
on Camel 2.16.1, and the camel-bean docs show this capability in 2.10, so we
were struggling yesterday to understand.
As we see this functionality under MASTER I think it will be released under
2.18. Would you happen to know when this release is going to occur? Our team
would like to start testing and using this if it occurs shortly, as we have
a release date prior to Dec.
Thanks
Scott
--
View this message in context: http://camel.465427.n5.nabble.com/Camel-BeanIODataFormat-Configure-ErrorHandler-for-BeanIO-Reader-tp5787005p5787051.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Camel BeanIODataFormat Configure ErrorHandler for BeanIO Reader
Posted by Claus Ibsen <cl...@gmail.com>.
A good idea is to look at the unit tests of camel-beanio and how it
tests this functionality.
On Tue, Aug 30, 2016 at 10:35 PM, spurcell <sc...@mastercard.com> wrote:
> Hello Brad,
>
> We have a large project in which we are taking in large csv files. Our team
> wants to create a custom BeanReaderErrorHandler (from:
> http://camel.apache.org/beanio.html) so we could convert errors to our
> return errors and build return files for our customers in our newly created
> application.
>
> If we let the Runtime errors go up the stack, it appears we lose some
> information that our developers wanted to get from the BeanReaderException
> such as getLineNumber(), getFieldText() etc.
>
> This camel is new to me, and I am struggling trying to implement the
> BeanReaderErrorHandler interface
> (http://beanio.org/2.1/docs/reference/index.html#BeanReaderErrorHandler)
> from BeanIO and somehow configure that into the Camel route.
>
> Thank
> Scott
>
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/Camel-BeanIODataFormat-Configure-ErrorHandler-for-BeanIO-Reader-tp5787005p5787013.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
--
Claus Ibsen
-----------------
http://davsclaus.com @davsclaus
Camel in Action 2: https://www.manning.com/ibsen2
Re: Camel BeanIODataFormat Configure ErrorHandler for BeanIO Reader
Posted by spurcell <sc...@mastercard.com>.
Hello Brad,
We have a large project in which we are taking in large csv files. Our team
wants to create a custom BeanReaderErrorHandler (from:
http://camel.apache.org/beanio.html) so we could convert errors to our
return errors and build return files for our customers in our newly created
application.
If we let the Runtime errors go up the stack, it appears we lose some
information that our developers wanted to get from the BeanReaderException
such as getLineNumber(), getFieldText() etc.
This camel is new to me, and I am struggling trying to implement the
BeanReaderErrorHandler interface
(http://beanio.org/2.1/docs/reference/index.html#BeanReaderErrorHandler)
from BeanIO and somehow configure that into the Camel route.
Thank
Scott
--
View this message in context: http://camel.465427.n5.nabble.com/Camel-BeanIODataFormat-Configure-ErrorHandler-for-BeanIO-Reader-tp5787005p5787013.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: Camel BeanIODataFormat Configure ErrorHandler for BeanIO Reader
Posted by Ranx <br...@mediadriver.com>.
Why are you trying to handle the error? There are a couple of settings with
BeanIO that permit you to ignore exceptions or to skip unknown record types.
That may not be what you want though if you are sending the bad things to a
DLQ.
Brad
--
View this message in context: http://camel.465427.n5.nabble.com/Camel-BeanIODataFormat-Configure-ErrorHandler-for-BeanIO-Reader-tp5787005p5787012.html
Sent from the Camel - Users mailing list archive at Nabble.com.