You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Dennis Kubes (JIRA)" <ji...@apache.org> on 2006/08/22 19:07:13 UTC
[jira] Created: (HADOOP-473) TextInputFormat does not correctly
handle all line endings
TextInputFormat does not correctly handle all line endings
----------------------------------------------------------
Key: HADOOP-473
URL: http://issues.apache.org/jira/browse/HADOOP-473
Project: Hadoop
Issue Type: Bug
Components: mapred
Affects Versions: 0.5.0, 0.6.0
Environment: All environments
Reporter: Dennis Kubes
Attachments: text-input-format.patch
The current TextInputFormat readLine method calls break on either a single '\r' or '\n' character. This causes windows formatted text files '\r' '\n' to leave a trailing '\n' character and the next time the readLine method is called on the same input stream it returns a blank string. The patch attached corrects this issue by looking for either single or double character line endings and positions the input stream to the next line. It correctly handles windows, mac, and unix line endings.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HADOOP-473) TextInputFormat does not correctly
handle all line endings
Posted by "Dennis Kubes (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-473?page=all ]
Dennis Kubes updated HADOOP-473:
--------------------------------
Attachment: text-input-format2.patch
Sorry it took me so long. I had the patch a couple of days ago I was just in the middle of testing. Here is an updated patch that removes the read ahead and fixes the same problem of line endings in the getRecordReader method of TextInputFormat. This has been lightly tested.
> TextInputFormat does not correctly handle all line endings
> ----------------------------------------------------------
>
> Key: HADOOP-473
> URL: http://issues.apache.org/jira/browse/HADOOP-473
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.5.0, 0.6.0
> Environment: All environments
> Reporter: Dennis Kubes
> Attachments: text-input-format.patch, text-input-format2.patch
>
>
> The current TextInputFormat readLine method calls break on either a single '\r' or '\n' character. This causes windows formatted text files '\r' '\n' to leave a trailing '\n' character and the next time the readLine method is called on the same input stream it returns a blank string. The patch attached corrects this issue by looking for either single or double character line endings and positions the input stream to the next line. It correctly handles windows, mac, and unix line endings.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-473) TextInputFormat does not correctly
handle all line endings
Posted by "James P. White (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-473?page=comments#action_12429789 ]
James P. White commented on HADOOP-473:
---------------------------------------
The proposed patch does read-ahead/seek-back even when the initial newline character is '\n'.
That is needlessly wasteful. The could should be simplified to:
if (c == '\n')
break;
if (c == '\r') {
long curPos = in.getPos();
char nextC = (char)in.read();
// EOF -1 is also != '\n' so will work with Mac and EOF
if (nextC != '\n')) {
// seek back 1 char because the line ending is only 1 char
in.seek(curPos);
}
break;
}
> TextInputFormat does not correctly handle all line endings
> ----------------------------------------------------------
>
> Key: HADOOP-473
> URL: http://issues.apache.org/jira/browse/HADOOP-473
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.5.0, 0.6.0
> Environment: All environments
> Reporter: Dennis Kubes
> Attachments: text-input-format.patch
>
>
> The current TextInputFormat readLine method calls break on either a single '\r' or '\n' character. This causes windows formatted text files '\r' '\n' to leave a trailing '\n' character and the next time the readLine method is called on the same input stream it returns a blank string. The patch attached corrects this issue by looking for either single or double character line endings and positions the input stream to the next line. It correctly handles windows, mac, and unix line endings.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Resolved: (HADOOP-473) TextInputFormat does not correctly
handle all line endings
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-473?page=all ]
Doug Cutting resolved HADOOP-473.
---------------------------------
Fix Version/s: 0.6.0
Resolution: Fixed
I just committed this. Thanks James & Dennis!
> TextInputFormat does not correctly handle all line endings
> ----------------------------------------------------------
>
> Key: HADOOP-473
> URL: http://issues.apache.org/jira/browse/HADOOP-473
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.5.0, 0.6.0
> Environment: All environments
> Reporter: Dennis Kubes
> Fix For: 0.6.0
>
> Attachments: text-input-format.patch, text-input-format2.patch
>
>
> The current TextInputFormat readLine method calls break on either a single '\r' or '\n' character. This causes windows formatted text files '\r' '\n' to leave a trailing '\n' character and the next time the readLine method is called on the same input stream it returns a blank string. The patch attached corrects this issue by looking for either single or double character line endings and positions the input stream to the next line. It correctly handles windows, mac, and unix line endings.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HADOOP-473) TextInputFormat does not correctly
handle all line endings
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/HADOOP-473?page=comments#action_12430129 ]
Owen O'Malley commented on HADOOP-473:
--------------------------------------
+1 with the read-ahead removal.
> TextInputFormat does not correctly handle all line endings
> ----------------------------------------------------------
>
> Key: HADOOP-473
> URL: http://issues.apache.org/jira/browse/HADOOP-473
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.5.0, 0.6.0
> Environment: All environments
> Reporter: Dennis Kubes
> Attachments: text-input-format.patch
>
>
> The current TextInputFormat readLine method calls break on either a single '\r' or '\n' character. This causes windows formatted text files '\r' '\n' to leave a trailing '\n' character and the next time the readLine method is called on the same input stream it returns a blank string. The patch attached corrects this issue by looking for either single or double character line endings and positions the input stream to the next line. It correctly handles windows, mac, and unix line endings.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira