You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by "Josh Patterson (JIRA)" <ji...@apache.org> on 2011/01/06 22:35:47 UTC
[jira] Created: (HIVE-1898) The ESCAPED BY clause does not seem to
pick up newlines in colums and the line terminator cannot be changed
The ESCAPED BY clause does not seem to pick up newlines in colums and the line terminator cannot be changed
-----------------------------------------------------------------------------------------------------------
Key: HIVE-1898
URL: https://issues.apache.org/jira/browse/HIVE-1898
Project: Hive
Issue Type: Bug
Components: Serializers/Deserializers
Affects Versions: 0.5.0
Reporter: Josh Patterson
Priority: Minor
If I want to preserve data in columns which contains a newline (webcrawling for instance) I cannot set the ESCAPED BY clause to escape these out (other characters such as commas escape fine, however). This may be due to the line terminators, which are locked to be newlines, are picked up first, and then fields processed.
This seems to be related to:
"SerDe should escape some special characters"
https://issues.apache.org/jira/browse/HIVE-136
and
"Implement "LINES TERMINATED BY""
https://issues.apache.org/jira/browse/HIVE-302
where at comment: https://issues.apache.org/jira/browse/HIVE-302?focusedCommentId=12793435&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#action_12793435
"This is not fixable currently because the line terminator is determined by LineRecordReader.LineReader which is in the Hadoop land."
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] [Commented] (HIVE-1898) The ESCAPED BY clause does not seem
to pick up newlines in colums and the line terminator cannot be changed
Posted by "Andrew Wilson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13026023#comment-13026023 ]
Andrew Wilson commented on HIVE-1898:
-------------------------------------
Hi Josh,
I am bumping into the same issue, did you find a way to handle this?
Andrew
> The ESCAPED BY clause does not seem to pick up newlines in colums and the line terminator cannot be changed
> -----------------------------------------------------------------------------------------------------------
>
> Key: HIVE-1898
> URL: https://issues.apache.org/jira/browse/HIVE-1898
> Project: Hive
> Issue Type: Bug
> Components: Serializers/Deserializers
> Affects Versions: 0.5.0
> Reporter: Josh Patterson
> Priority: Minor
>
> If I want to preserve data in columns which contains a newline (webcrawling for instance) I cannot set the ESCAPED BY clause to escape these out (other characters such as commas escape fine, however). This may be due to the line terminators, which are locked to be newlines, are picked up first, and then fields processed.
> This seems to be related to:
> "SerDe should escape some special characters"
> https://issues.apache.org/jira/browse/HIVE-136
> and
> "Implement "LINES TERMINATED BY""
> https://issues.apache.org/jira/browse/HIVE-302
> where at comment: https://issues.apache.org/jira/browse/HIVE-302?focusedCommentId=12793435&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#action_12793435
> "This is not fixable currently because the line terminator is determined by LineRecordReader.LineReader which is in the Hadoop land."
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-1898) The ESCAPED BY clause does not seem
to pick up newlines in colums and the line terminator cannot be changed
Posted by "Brian Bloniarz (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13449978#comment-13449978 ]
Brian Bloniarz commented on HIVE-1898:
--------------------------------------
I think Luke is right -- maybe the bug title should be changed to simply say "data with newlines won't work in Text/LazySimpleSerDe tables"?
I haven't tested it, but would STORED AS SEQUENCEFILE tables be immune to this problem?
> The ESCAPED BY clause does not seem to pick up newlines in colums and the line terminator cannot be changed
> -----------------------------------------------------------------------------------------------------------
>
> Key: HIVE-1898
> URL: https://issues.apache.org/jira/browse/HIVE-1898
> Project: Hive
> Issue Type: Bug
> Components: Serializers/Deserializers
> Affects Versions: 0.5.0
> Reporter: Josh Patterson
> Priority: Minor
>
> If I want to preserve data in columns which contains a newline (webcrawling for instance) I cannot set the ESCAPED BY clause to escape these out (other characters such as commas escape fine, however). This may be due to the line terminators, which are locked to be newlines, are picked up first, and then fields processed.
> This seems to be related to:
> "SerDe should escape some special characters"
> https://issues.apache.org/jira/browse/HIVE-136
> and
> "Implement "LINES TERMINATED BY""
> https://issues.apache.org/jira/browse/HIVE-302
> where at comment: https://issues.apache.org/jira/browse/HIVE-302?focusedCommentId=12793435&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#action_12793435
> "This is not fixable currently because the line terminator is determined by LineRecordReader.LineReader which is in the Hadoop land."
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-1898) The ESCAPED BY clause does not seem
to pick up newlines in colums and the line terminator cannot be changed
Posted by "Luke Forehand (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13119581#comment-13119581 ]
Luke Forehand commented on HIVE-1898:
-------------------------------------
I don't think 'ESCAPED BY' will solve your problem. From what I have gathered, 'ESCAPED BY' translates into a SerDe property that indicates the escape character, which is default '\'. It does not allow you to specify which characters you would like to escape.
Looking at the source code that initializes LazySimpleSerDe.SerDeParameters, only the default separator chars (char codes 1, 2, 3) are flagged as needing escaping.
I have data that contains newlines as well, and as a workaround I pass the column through the regexp_replace function like:
regexp_replace(<my column>, "\n", "")
which eliminates the newline chars during deserialization. However I don't like doing this and would appreciate alternative solutions if there are any. Anyone?
> The ESCAPED BY clause does not seem to pick up newlines in colums and the line terminator cannot be changed
> -----------------------------------------------------------------------------------------------------------
>
> Key: HIVE-1898
> URL: https://issues.apache.org/jira/browse/HIVE-1898
> Project: Hive
> Issue Type: Bug
> Components: Serializers/Deserializers
> Affects Versions: 0.5.0
> Reporter: Josh Patterson
> Priority: Minor
>
> If I want to preserve data in columns which contains a newline (webcrawling for instance) I cannot set the ESCAPED BY clause to escape these out (other characters such as commas escape fine, however). This may be due to the line terminators, which are locked to be newlines, are picked up first, and then fields processed.
> This seems to be related to:
> "SerDe should escape some special characters"
> https://issues.apache.org/jira/browse/HIVE-136
> and
> "Implement "LINES TERMINATED BY""
> https://issues.apache.org/jira/browse/HIVE-302
> where at comment: https://issues.apache.org/jira/browse/HIVE-302?focusedCommentId=12793435&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#action_12793435
> "This is not fixable currently because the line terminator is determined by LineRecordReader.LineReader which is in the Hadoop land."
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira