You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Amareshwari Sriramadasu (JIRA)" <ji...@apache.org> on 2008/07/08 07:56:57 UTC
[jira] Created: (HADOOP-3711) Streaming input is not parsed
properly to find the separator
Streaming input is not parsed properly to find the separator
------------------------------------------------------------
Key: HADOOP-3711
URL: https://issues.apache.org/jira/browse/HADOOP-3711
Project: Hadoop Core
Issue Type: Bug
Components: contrib/streaming
Reporter: Amareshwari Sriramadasu
Assignee: Amareshwari Sriramadasu
Priority: Blocker
Fix For: 0.18.0
Sometimes, finding field separator position in the input line for streaming fails with following exception:
org.apache.hadoop.streaming.PipeMapRed: java.lang.IllegalArgumentException: splitPos must be in the range [0, 17]: 74
at org.apache.hadoop.streaming.UTF8ByteArrayUtils.splitKeyVal(UTF8ByteArrayUtils.java:152)
at org.apache.hadoop.streaming.PipeMapRed.splitKeyVal(PipeMapRed.java:337)
at org.apache.hadoop.streaming.PipeMapRed$MROutputThread.run(PipeMapRed.java:363)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3711) Streaming input is not parsed
properly to find the separator
Posted by "Amareshwari Sriramadasu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Amareshwari Sriramadasu updated HADOOP-3711:
--------------------------------------------
Status: Patch Available (was: Open)
> Streaming input is not parsed properly to find the separator
> ------------------------------------------------------------
>
> Key: HADOOP-3711
> URL: https://issues.apache.org/jira/browse/HADOOP-3711
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/streaming
> Reporter: Amareshwari Sriramadasu
> Assignee: Amareshwari Sriramadasu
> Priority: Blocker
> Fix For: 0.18.0
>
> Attachments: patch-3711.txt
>
>
> Sometimes, finding field separator position in the input line for streaming fails with following exception:
> org.apache.hadoop.streaming.PipeMapRed: java.lang.IllegalArgumentException: splitPos must be in the range [0, 17]: 74
> at org.apache.hadoop.streaming.UTF8ByteArrayUtils.splitKeyVal(UTF8ByteArrayUtils.java:152)
> at org.apache.hadoop.streaming.PipeMapRed.splitKeyVal(PipeMapRed.java:337)
> at org.apache.hadoop.streaming.PipeMapRed$MROutputThread.run(PipeMapRed.java:363)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3711) Streaming input is not parsed
properly to find the separator
Posted by "Amareshwari Sriramadasu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12611464#action_12611464 ]
Amareshwari Sriramadasu commented on HADOOP-3711:
-------------------------------------------------
In org.apache.hadoop.streaming.PipeMapRed.splitKeyVal(), the length passed in the parameter should be passed down to the UTF8ByteArrayUtils.findBytes() routine instead of byte[].length
> Streaming input is not parsed properly to find the separator
> ------------------------------------------------------------
>
> Key: HADOOP-3711
> URL: https://issues.apache.org/jira/browse/HADOOP-3711
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/streaming
> Reporter: Amareshwari Sriramadasu
> Assignee: Amareshwari Sriramadasu
> Priority: Blocker
> Fix For: 0.18.0
>
>
> Sometimes, finding field separator position in the input line for streaming fails with following exception:
> org.apache.hadoop.streaming.PipeMapRed: java.lang.IllegalArgumentException: splitPos must be in the range [0, 17]: 74
> at org.apache.hadoop.streaming.UTF8ByteArrayUtils.splitKeyVal(UTF8ByteArrayUtils.java:152)
> at org.apache.hadoop.streaming.PipeMapRed.splitKeyVal(PipeMapRed.java:337)
> at org.apache.hadoop.streaming.PipeMapRed$MROutputThread.run(PipeMapRed.java:363)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (HADOOP-3711) Streaming input is not
parsed properly to find the separator
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12612271#action_12612271 ]
chris.douglas edited comment on HADOOP-3711 at 7/9/08 1:51 PM:
---------------------------------------------------------------
The patch doesn't apply to 0.18, the test case passes without the change to PipeMapRed, and the code doesn't seem to have the problem this fixes. Does this only apply to trunk?
was (Author: chris.douglas):
This is not in 0.18
> Streaming input is not parsed properly to find the separator
> ------------------------------------------------------------
>
> Key: HADOOP-3711
> URL: https://issues.apache.org/jira/browse/HADOOP-3711
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/streaming
> Reporter: Amareshwari Sriramadasu
> Assignee: Amareshwari Sriramadasu
> Fix For: 0.19.0
>
> Attachments: patch-3711.txt
>
>
> Sometimes, finding field separator position in the input line for streaming fails with following exception:
> org.apache.hadoop.streaming.PipeMapRed: java.lang.IllegalArgumentException: splitPos must be in the range [0, 17]: 74
> at org.apache.hadoop.streaming.UTF8ByteArrayUtils.splitKeyVal(UTF8ByteArrayUtils.java:152)
> at org.apache.hadoop.streaming.PipeMapRed.splitKeyVal(PipeMapRed.java:337)
> at org.apache.hadoop.streaming.PipeMapRed$MROutputThread.run(PipeMapRed.java:363)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3711) Streaming input is not parsed
properly to find the separator
Posted by "Amareshwari Sriramadasu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Amareshwari Sriramadasu updated HADOOP-3711:
--------------------------------------------
Attachment: patch-3711.txt
Here is patch fixing the bug.
I updated the testcase to parse such input.
> Streaming input is not parsed properly to find the separator
> ------------------------------------------------------------
>
> Key: HADOOP-3711
> URL: https://issues.apache.org/jira/browse/HADOOP-3711
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/streaming
> Reporter: Amareshwari Sriramadasu
> Assignee: Amareshwari Sriramadasu
> Priority: Blocker
> Fix For: 0.18.0
>
> Attachments: patch-3711.txt
>
>
> Sometimes, finding field separator position in the input line for streaming fails with following exception:
> org.apache.hadoop.streaming.PipeMapRed: java.lang.IllegalArgumentException: splitPos must be in the range [0, 17]: 74
> at org.apache.hadoop.streaming.UTF8ByteArrayUtils.splitKeyVal(UTF8ByteArrayUtils.java:152)
> at org.apache.hadoop.streaming.PipeMapRed.splitKeyVal(PipeMapRed.java:337)
> at org.apache.hadoop.streaming.PipeMapRed$MROutputThread.run(PipeMapRed.java:363)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3711) Streaming input is not parsed
properly to find the separator
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12611874#action_12611874 ]
Hadoop QA commented on HADOOP-3711:
-----------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12385466/patch-3711.txt
against trunk revision 675054.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2818/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2818/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2818/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2818/console
This message is automatically generated.
> Streaming input is not parsed properly to find the separator
> ------------------------------------------------------------
>
> Key: HADOOP-3711
> URL: https://issues.apache.org/jira/browse/HADOOP-3711
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/streaming
> Reporter: Amareshwari Sriramadasu
> Assignee: Amareshwari Sriramadasu
> Priority: Blocker
> Fix For: 0.18.0
>
> Attachments: patch-3711.txt
>
>
> Sometimes, finding field separator position in the input line for streaming fails with following exception:
> org.apache.hadoop.streaming.PipeMapRed: java.lang.IllegalArgumentException: splitPos must be in the range [0, 17]: 74
> at org.apache.hadoop.streaming.UTF8ByteArrayUtils.splitKeyVal(UTF8ByteArrayUtils.java:152)
> at org.apache.hadoop.streaming.PipeMapRed.splitKeyVal(PipeMapRed.java:337)
> at org.apache.hadoop.streaming.PipeMapRed$MROutputThread.run(PipeMapRed.java:363)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3711) Streaming input is not parsed
properly to find the separator
Posted by "Devaraj Das (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Devaraj Das updated HADOOP-3711:
--------------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
I just committed this. Thanks, Amareshwari!
> Streaming input is not parsed properly to find the separator
> ------------------------------------------------------------
>
> Key: HADOOP-3711
> URL: https://issues.apache.org/jira/browse/HADOOP-3711
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/streaming
> Reporter: Amareshwari Sriramadasu
> Assignee: Amareshwari Sriramadasu
> Fix For: 0.19.0
>
> Attachments: patch-3711.txt
>
>
> Sometimes, finding field separator position in the input line for streaming fails with following exception:
> org.apache.hadoop.streaming.PipeMapRed: java.lang.IllegalArgumentException: splitPos must be in the range [0, 17]: 74
> at org.apache.hadoop.streaming.UTF8ByteArrayUtils.splitKeyVal(UTF8ByteArrayUtils.java:152)
> at org.apache.hadoop.streaming.PipeMapRed.splitKeyVal(PipeMapRed.java:337)
> at org.apache.hadoop.streaming.PipeMapRed$MROutputThread.run(PipeMapRed.java:363)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3711) Streaming input is not parsed
properly to find the separator
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Douglas updated HADOOP-3711:
----------------------------------
Hadoop Flags: [Reviewed]
+1
> Streaming input is not parsed properly to find the separator
> ------------------------------------------------------------
>
> Key: HADOOP-3711
> URL: https://issues.apache.org/jira/browse/HADOOP-3711
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/streaming
> Reporter: Amareshwari Sriramadasu
> Assignee: Amareshwari Sriramadasu
> Priority: Blocker
> Fix For: 0.18.0
>
> Attachments: patch-3711.txt
>
>
> Sometimes, finding field separator position in the input line for streaming fails with following exception:
> org.apache.hadoop.streaming.PipeMapRed: java.lang.IllegalArgumentException: splitPos must be in the range [0, 17]: 74
> at org.apache.hadoop.streaming.UTF8ByteArrayUtils.splitKeyVal(UTF8ByteArrayUtils.java:152)
> at org.apache.hadoop.streaming.PipeMapRed.splitKeyVal(PipeMapRed.java:337)
> at org.apache.hadoop.streaming.PipeMapRed$MROutputThread.run(PipeMapRed.java:363)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3711) Streaming input is not parsed
properly to find the separator
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12624787#action_12624787 ]
Hudson commented on HADOOP-3711:
--------------------------------
Integrated in Hadoop-trunk #581 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/581/])
> Streaming input is not parsed properly to find the separator
> ------------------------------------------------------------
>
> Key: HADOOP-3711
> URL: https://issues.apache.org/jira/browse/HADOOP-3711
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/streaming
> Reporter: Amareshwari Sriramadasu
> Assignee: Amareshwari Sriramadasu
> Fix For: 0.19.0
>
> Attachments: patch-3711.txt
>
>
> Sometimes, finding field separator position in the input line for streaming fails with following exception:
> org.apache.hadoop.streaming.PipeMapRed: java.lang.IllegalArgumentException: splitPos must be in the range [0, 17]: 74
> at org.apache.hadoop.streaming.UTF8ByteArrayUtils.splitKeyVal(UTF8ByteArrayUtils.java:152)
> at org.apache.hadoop.streaming.PipeMapRed.splitKeyVal(PipeMapRed.java:337)
> at org.apache.hadoop.streaming.PipeMapRed$MROutputThread.run(PipeMapRed.java:363)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3711) Streaming input is not parsed
properly to find the separator
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Douglas updated HADOOP-3711:
----------------------------------
Priority: Major (was: Blocker)
Fix Version/s: (was: 0.18.0)
0.19.0
This is not in 0.18
> Streaming input is not parsed properly to find the separator
> ------------------------------------------------------------
>
> Key: HADOOP-3711
> URL: https://issues.apache.org/jira/browse/HADOOP-3711
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/streaming
> Reporter: Amareshwari Sriramadasu
> Assignee: Amareshwari Sriramadasu
> Fix For: 0.19.0
>
> Attachments: patch-3711.txt
>
>
> Sometimes, finding field separator position in the input line for streaming fails with following exception:
> org.apache.hadoop.streaming.PipeMapRed: java.lang.IllegalArgumentException: splitPos must be in the range [0, 17]: 74
> at org.apache.hadoop.streaming.UTF8ByteArrayUtils.splitKeyVal(UTF8ByteArrayUtils.java:152)
> at org.apache.hadoop.streaming.PipeMapRed.splitKeyVal(PipeMapRed.java:337)
> at org.apache.hadoop.streaming.PipeMapRed$MROutputThread.run(PipeMapRed.java:363)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3711) Streaming input is not parsed
properly to find the separator
Posted by "Amareshwari Sriramadasu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12612370#action_12612370 ]
Amareshwari Sriramadasu commented on HADOOP-3711:
-------------------------------------------------
Chris, you are right. The bug is not there in 0.18. It is only on the trunk. It got introduced by HADOOP-3341.
> Streaming input is not parsed properly to find the separator
> ------------------------------------------------------------
>
> Key: HADOOP-3711
> URL: https://issues.apache.org/jira/browse/HADOOP-3711
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/streaming
> Reporter: Amareshwari Sriramadasu
> Assignee: Amareshwari Sriramadasu
> Fix For: 0.19.0
>
> Attachments: patch-3711.txt
>
>
> Sometimes, finding field separator position in the input line for streaming fails with following exception:
> org.apache.hadoop.streaming.PipeMapRed: java.lang.IllegalArgumentException: splitPos must be in the range [0, 17]: 74
> at org.apache.hadoop.streaming.UTF8ByteArrayUtils.splitKeyVal(UTF8ByteArrayUtils.java:152)
> at org.apache.hadoop.streaming.PipeMapRed.splitKeyVal(PipeMapRed.java:337)
> at org.apache.hadoop.streaming.PipeMapRed$MROutputThread.run(PipeMapRed.java:363)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.