You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by Shawn Weeks <sw...@weeksconsulting.us> on 2019/12/20 17:51:18 UTC

CSV Record Reader - No Quote and No Escape

I’m trying to process some tab delimited text that may contain embedded quotes and  slashes. How do I disable quote and escape in the CSVReader so that they aren’t used. Setting them to empty text doesn’t work and I’m getting the following error because there are quotes and slashes in the data despite setting the format to tab delimited.

Failed to write due to org.apache.nifi.serialization.MalformedRecordException: Error while getting next record. Root cause: java.io.IOException: (line 1050916) invalid char between encapsulated token and delimiter: java.io.IOException: org.apache.nifi.serialization.MalformedRecordException: Error while getting next record. Root cause: java.io.IOException: (line 1050916) invalid char between encapsulated token and delimiter

Re: CSV Record Reader - No Quote and No Escape

Posted by Shawn Weeks <sw...@weeksconsulting.us>.
I’ve created NIFI-6966 and NIFI-6967 with some examples.

Thanks
Shawn Weeks

From: Pierre Villard <pi...@gmail.com>
Reply-To: "users@nifi.apache.org" <us...@nifi.apache.org>
Date: Thursday, December 26, 2019 at 9:33 AM
To: "users@nifi.apache.org" <us...@nifi.apache.org>
Subject: Re: CSV Record Reader - No Quote and No Escape

Hey Shawn,

Could you provide an example of what data you have and how you would like to have it processed? It'll be easier to add a unit test and make a code change if necessary.

Thanks,
Pierre

Le ven. 20 déc. 2019 à 18:51, Shawn Weeks <sw...@weeksconsulting.us>> a écrit :
I’m trying to process some tab delimited text that may contain embedded quotes and  slashes. How do I disable quote and escape in the CSVReader so that they aren’t used. Setting them to empty text doesn’t work and I’m getting the following error because there are quotes and slashes in the data despite setting the format to tab delimited.

Failed to write due to org.apache.nifi.serialization.MalformedRecordException: Error while getting next record. Root cause: java.io.IOException: (line 1050916) invalid char between encapsulated token and delimiter: java.io.IOException: org.apache.nifi.serialization.MalformedRecordException: Error while getting next record. Root cause: java.io.IOException: (line 1050916) invalid char between encapsulated token and delimiter

Re: CSV Record Reader - No Quote and No Escape

Posted by Shawn Weeks <sw...@weeksconsulting.us>.
This example was for a different issue. The examples for CSV Reader are on the JIRA. Sorry.

Thanks
Shawn

From: Shawn Weeks <sw...@weeksconsulting.us>
Reply-To: "users@nifi.apache.org" <us...@nifi.apache.org>
Date: Thursday, December 26, 2019 at 12:28 PM
To: "users@nifi.apache.org" <us...@nifi.apache.org>
Subject: Re: CSV Record Reader - No Quote and No Escape

Attached is an example. Do not try this on a NiFi cluster you care about as you might have to delete your flow.xml.gz file to kill it. I increased my xmx and xms to 4gb for this test.

Thanks
Shawn Weeks

From: Pierre Villard <pi...@gmail.com>
Reply-To: "users@nifi.apache.org" <us...@nifi.apache.org>
Date: Thursday, December 26, 2019 at 9:33 AM
To: "users@nifi.apache.org" <us...@nifi.apache.org>
Subject: Re: CSV Record Reader - No Quote and No Escape

Hey Shawn,

Could you provide an example of what data you have and how you would like to have it processed? It'll be easier to add a unit test and make a code change if necessary.

Thanks,
Pierre

Le ven. 20 déc. 2019 à 18:51, Shawn Weeks <sw...@weeksconsulting.us>> a écrit :
I’m trying to process some tab delimited text that may contain embedded quotes and  slashes. How do I disable quote and escape in the CSVReader so that they aren’t used. Setting them to empty text doesn’t work and I’m getting the following error because there are quotes and slashes in the data despite setting the format to tab delimited.

Failed to write due to org.apache.nifi.serialization.MalformedRecordException: Error while getting next record. Root cause: java.io.IOException: (line 1050916) invalid char between encapsulated token and delimiter: java.io.IOException: org.apache.nifi.serialization.MalformedRecordException: Error while getting next record. Root cause: java.io.IOException: (line 1050916) invalid char between encapsulated token and delimiter

Re: CSV Record Reader - No Quote and No Escape

Posted by Shawn Weeks <sw...@weeksconsulting.us>.
Attached is an example. Do not try this on a NiFi cluster you care about as you might have to delete your flow.xml.gz file to kill it. I increased my xmx and xms to 4gb for this test.

Thanks
Shawn Weeks

From: Pierre Villard <pi...@gmail.com>
Reply-To: "users@nifi.apache.org" <us...@nifi.apache.org>
Date: Thursday, December 26, 2019 at 9:33 AM
To: "users@nifi.apache.org" <us...@nifi.apache.org>
Subject: Re: CSV Record Reader - No Quote and No Escape

Hey Shawn,

Could you provide an example of what data you have and how you would like to have it processed? It'll be easier to add a unit test and make a code change if necessary.

Thanks,
Pierre

Le ven. 20 déc. 2019 à 18:51, Shawn Weeks <sw...@weeksconsulting.us>> a écrit :
I’m trying to process some tab delimited text that may contain embedded quotes and  slashes. How do I disable quote and escape in the CSVReader so that they aren’t used. Setting them to empty text doesn’t work and I’m getting the following error because there are quotes and slashes in the data despite setting the format to tab delimited.

Failed to write due to org.apache.nifi.serialization.MalformedRecordException: Error while getting next record. Root cause: java.io.IOException: (line 1050916) invalid char between encapsulated token and delimiter: java.io.IOException: org.apache.nifi.serialization.MalformedRecordException: Error while getting next record. Root cause: java.io.IOException: (line 1050916) invalid char between encapsulated token and delimiter

Re: CSV Record Reader - No Quote and No Escape

Posted by Pierre Villard <pi...@gmail.com>.
Hey Shawn,

Could you provide an example of what data you have and how you would like
to have it processed? It'll be easier to add a unit test and make a code
change if necessary.

Thanks,
Pierre

Le ven. 20 déc. 2019 à 18:51, Shawn Weeks <sw...@weeksconsulting.us> a
écrit :

> I’m trying to process some tab delimited text that may contain embedded
> quotes and  slashes. How do I disable quote and escape in the CSVReader so
> that they aren’t used. Setting them to empty text doesn’t work and I’m
> getting the following error because there are quotes and slashes in the
> data despite setting the format to tab delimited.
>
>
>
> Failed to write due to
> org.apache.nifi.serialization.MalformedRecordException: Error while getting
> next record. Root cause: java.io.IOException: (line 1050916) invalid char
> between encapsulated token and delimiter: java.io.IOException:
> org.apache.nifi.serialization.MalformedRecordException: Error while getting
> next record. Root cause: java.io.IOException: (line 1050916) invalid char
> between encapsulated token and delimiter
>