You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@commons.apache.org by "Michael Lackhoff (JIRA)" <ji...@apache.org> on 2008/01/04 17:44:34 UTC
[jira] Created: (SANDBOX-206) backslash before quote character
gives an error
backslash before quote character gives an error
-----------------------------------------------
Key: SANDBOX-206
URL: https://issues.apache.org/jira/browse/SANDBOX-206
Project: Commons Sandbox
Issue Type: Bug
Components: CSV
Environment: Windows, SOLR 1.2
Reporter: Michael Lackhoff
A CSV-field with the contents "This is text with a \""quoted"" string" gives the error
"invalid char between encapsualted token end delimiter". If the backslash is not immediately before the double quote, everything is fine.
The same error occurs when the backslash is the last character in the field (directly before the delimiter), like:
"This is a text with a backslash \".
Here the reason might be that the backslash also works as an escape character like in
"This is a field with a \"quoted\" text" (no error, just the quotes in the resulting field)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (SANDBOX-206) backslash before quote character
gives an error
Posted by "Yonik Seeley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SANDBOX-206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Yonik Seeley resolved SANDBOX-206.
----------------------------------
Resolution: Fixed
committed. Thanks for the review Hen!
> backslash before quote character gives an error
> -----------------------------------------------
>
> Key: SANDBOX-206
> URL: https://issues.apache.org/jira/browse/SANDBOX-206
> Project: Commons Sandbox
> Issue Type: Bug
> Components: CSV
> Environment: Windows, SOLR 1.2
> Reporter: Michael Lackhoff
> Attachments: csv_escape.patch
>
>
> A CSV-field with the contents "This is text with a \""quoted"" string" gives the error
> "invalid char between encapsualted token end delimiter". If the backslash is not immediately before the double quote, everything is fine.
> The same error occurs when the backslash is the last character in the field (directly before the delimiter), like:
> "This is a text with a backslash \".
> Here the reason might be that the backslash also works as an escape character like in
> "This is a field with a \"quoted\" text" (no error, just the quotes in the resulting field)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (SANDBOX-206) backslash before quote character
gives an error
Posted by "Yonik Seeley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SANDBOX-206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12555988#action_12555988 ]
Yonik Seeley commented on SANDBOX-206:
--------------------------------------
Thanks for the bug report Michael.
It looks like there are a few issues here:
1) backslash escaping should be able to be disabled
2) backslash escaping currently has bugs that prevent representation of all values
3) perhaps backslash escaping should be off by default since it's not part of the CSV standard (AFAIK?)
> backslash before quote character gives an error
> -----------------------------------------------
>
> Key: SANDBOX-206
> URL: https://issues.apache.org/jira/browse/SANDBOX-206
> Project: Commons Sandbox
> Issue Type: Bug
> Components: CSV
> Environment: Windows, SOLR 1.2
> Reporter: Michael Lackhoff
>
> A CSV-field with the contents "This is text with a \""quoted"" string" gives the error
> "invalid char between encapsualted token end delimiter". If the backslash is not immediately before the double quote, everything is fine.
> The same error occurs when the backslash is the last character in the field (directly before the delimiter), like:
> "This is a text with a backslash \".
> Here the reason might be that the backslash also works as an escape character like in
> "This is a field with a \"quoted\" text" (no error, just the quotes in the resulting field)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (SANDBOX-206) backslash before quote character
gives an error
Posted by "Yonik Seeley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SANDBOX-206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12556053#action_12556053 ]
Yonik Seeley commented on SANDBOX-206:
--------------------------------------
I've done some more investigation of what common CSV producers use:
Excel does not do backslash escaping.
MySQL does backslash escaping to escape separators and or to escape encapsulators if they are used. The escape is used to escape itself.
This is the behavior I think we should mimic (no backslash escaping, but escape consistently if enabled) I haven't been able to find any basis for the current behavior... so if you understand why the current code purposely does what it does, speak up!
> backslash before quote character gives an error
> -----------------------------------------------
>
> Key: SANDBOX-206
> URL: https://issues.apache.org/jira/browse/SANDBOX-206
> Project: Commons Sandbox
> Issue Type: Bug
> Components: CSV
> Environment: Windows, SOLR 1.2
> Reporter: Michael Lackhoff
>
> A CSV-field with the contents "This is text with a \""quoted"" string" gives the error
> "invalid char between encapsualted token end delimiter". If the backslash is not immediately before the double quote, everything is fine.
> The same error occurs when the backslash is the last character in the field (directly before the delimiter), like:
> "This is a text with a backslash \".
> Here the reason might be that the backslash also works as an escape character like in
> "This is a field with a \"quoted\" text" (no error, just the quotes in the resulting field)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (SANDBOX-206) backslash before quote character
gives an error
Posted by "Yonik Seeley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SANDBOX-206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12555992#action_12555992 ]
Yonik Seeley commented on SANDBOX-206:
--------------------------------------
It's interesting... looking at CSVParserTest.testBackslashEscaping(), much of this is deliberate:
{code}
String code =
"one,two,three\n"
+ "on\\\"e,two\n"
+ "on\"e,two\n"
+ "one,\"tw\\\"o\"\n"
+ "one,\"t\\,wo\"\n"
+ "one,two,\"th,ree\"\n"
+ "\"a\\\\\"\n"
+ "a\\,b\n"
+ "\"a\\\\,b\"";
String[][] res = {
{ "one", "two", "three" },
{ "on\\\"e", "two" },
{ "on\"e", "two" },
{ "one", "tw\"o" },
{ "one", "t\\,wo" }, // backslash in quotes only escapes a delimiter (",")
{ "one", "two", "th,ree" },
{ "a\\\\" }, // backslash in quotes only escapes a delimiter (",")
{ "a\\", "b" }, // a backslash must be returnd
{ "a\\\\,b" } // backslash in quotes only escapes a delimiter (",")
};
{code}
Does anyone have knowledge of why this particular escaping mechanism is used?
> backslash before quote character gives an error
> -----------------------------------------------
>
> Key: SANDBOX-206
> URL: https://issues.apache.org/jira/browse/SANDBOX-206
> Project: Commons Sandbox
> Issue Type: Bug
> Components: CSV
> Environment: Windows, SOLR 1.2
> Reporter: Michael Lackhoff
>
> A CSV-field with the contents "This is text with a \""quoted"" string" gives the error
> "invalid char between encapsualted token end delimiter". If the backslash is not immediately before the double quote, everything is fine.
> The same error occurs when the backslash is the last character in the field (directly before the delimiter), like:
> "This is a text with a backslash \".
> Here the reason might be that the backslash also works as an escape character like in
> "This is a field with a \"quoted\" text" (no error, just the quotes in the resulting field)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (SANDBOX-206) backslash before quote character
gives an error
Posted by "Yonik Seeley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SANDBOX-206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12556386#action_12556386 ]
Yonik Seeley commented on SANDBOX-206:
--------------------------------------
The last patch broke whitespace handling around escape chars... I just committed that fix, in addition to an option to not ignore trailing whitespace (both excel and mysql have significant leading and trailing whitespace). The excel strategy was changed to reflect this.
> backslash before quote character gives an error
> -----------------------------------------------
>
> Key: SANDBOX-206
> URL: https://issues.apache.org/jira/browse/SANDBOX-206
> Project: Commons Sandbox
> Issue Type: Bug
> Components: CSV
> Environment: Windows, SOLR 1.2
> Reporter: Michael Lackhoff
> Attachments: csv_escape.patch
>
>
> A CSV-field with the contents "This is text with a \""quoted"" string" gives the error
> "invalid char between encapsualted token end delimiter". If the backslash is not immediately before the double quote, everything is fine.
> The same error occurs when the backslash is the last character in the field (directly before the delimiter), like:
> "This is a text with a backslash \".
> Here the reason might be that the backslash also works as an escape character like in
> "This is a field with a \"quoted\" text" (no error, just the quotes in the resulting field)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (SANDBOX-206) backslash before quote character
gives an error
Posted by "Henri Yandell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SANDBOX-206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12556178#action_12556178 ]
Henri Yandell commented on SANDBOX-206:
---------------------------------------
+1... quick scan of the patch looks good.
> backslash before quote character gives an error
> -----------------------------------------------
>
> Key: SANDBOX-206
> URL: https://issues.apache.org/jira/browse/SANDBOX-206
> Project: Commons Sandbox
> Issue Type: Bug
> Components: CSV
> Environment: Windows, SOLR 1.2
> Reporter: Michael Lackhoff
> Attachments: csv_escape.patch
>
>
> A CSV-field with the contents "This is text with a \""quoted"" string" gives the error
> "invalid char between encapsualted token end delimiter". If the backslash is not immediately before the double quote, everything is fine.
> The same error occurs when the backslash is the last character in the field (directly before the delimiter), like:
> "This is a text with a backslash \".
> Here the reason might be that the backslash also works as an escape character like in
> "This is a field with a \"quoted\" text" (no error, just the quotes in the resulting field)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (SANDBOX-206) backslash before quote character
gives an error
Posted by "Yonik Seeley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SANDBOX-206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Yonik Seeley updated SANDBOX-206:
---------------------------------
Attachment: csv_escape.patch
Patch atteched:
- adds an escape char to the strategy
- disables escaping by default (need to double the encapsulation char)
- when enabled, escaping works as expected (the escape char is expected
to be escaped with itself, escaping other chars results in just that char)
- code simplifications... got rid of skipRead, tokenReady checking in the
innermost loops.
Barring objections or further input, I'll commit soon.
> backslash before quote character gives an error
> -----------------------------------------------
>
> Key: SANDBOX-206
> URL: https://issues.apache.org/jira/browse/SANDBOX-206
> Project: Commons Sandbox
> Issue Type: Bug
> Components: CSV
> Environment: Windows, SOLR 1.2
> Reporter: Michael Lackhoff
> Attachments: csv_escape.patch
>
>
> A CSV-field with the contents "This is text with a \""quoted"" string" gives the error
> "invalid char between encapsualted token end delimiter". If the backslash is not immediately before the double quote, everything is fine.
> The same error occurs when the backslash is the last character in the field (directly before the delimiter), like:
> "This is a text with a backslash \".
> Here the reason might be that the backslash also works as an escape character like in
> "This is a field with a \"quoted\" text" (no error, just the quotes in the resulting field)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Closed: (SANDBOX-206) backslash before quote character gives
an error
Posted by "Henri Yandell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SANDBOX-206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Henri Yandell closed SANDBOX-206.
---------------------------------
> backslash before quote character gives an error
> -----------------------------------------------
>
> Key: SANDBOX-206
> URL: https://issues.apache.org/jira/browse/SANDBOX-206
> Project: Commons Sandbox
> Issue Type: Bug
> Components: CSV
> Environment: Windows, SOLR 1.2
> Reporter: Michael Lackhoff
> Attachments: csv_escape.patch
>
>
> A CSV-field with the contents "This is text with a \""quoted"" string" gives the error
> "invalid char between encapsualted token end delimiter". If the backslash is not immediately before the double quote, everything is fine.
> The same error occurs when the backslash is the last character in the field (directly before the delimiter), like:
> "This is a text with a backslash \".
> Here the reason might be that the backslash also works as an escape character like in
> "This is a field with a \"quoted\" text" (no error, just the quotes in the resulting field)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.