You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by "Koji Noguchi (JIRA)" <ji...@apache.org> on 2012/12/03 21:23:58 UTC
[jira] [Created] (PIG-3072) Pig job reporting negative progress
Koji Noguchi created PIG-3072:
---------------------------------
Summary: Pig job reporting negative progress
Key: PIG-3072
URL: https://issues.apache.org/jira/browse/PIG-3072
Project: Pig
Issue Type: Bug
Components: impl
Reporter: Koji Noguchi
Assignee: Koji Noguchi
Priority: Minor
Our users pointed out that their jobs reporting negative progress.
2012-11-02 21:43:11,538 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- -795% complete
...
(due to TFileRecordReader)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (PIG-3072) Pig job reporting negative progress
Posted by "Rohini Palaniswamy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-3072?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Rohini Palaniswamy updated PIG-3072:
------------------------------------
Release Note: (was: Committed to trunk. Thanks Koji.)
Committed to trunk. Thanks Koji.
> Pig job reporting negative progress
> -----------------------------------
>
> Key: PIG-3072
> URL: https://issues.apache.org/jira/browse/PIG-3072
> Project: Pig
> Issue Type: Bug
> Components: impl
> Affects Versions: 0.10.0
> Reporter: Koji Noguchi
> Assignee: Koji Noguchi
> Priority: Minor
> Fix For: 0.12
>
> Attachments: pig-3072-v01.txt, pig-3072-v02.txt, pig-3072-v03.txt, pig-3072-v04.txt
>
>
> Our users pointed out that their jobs reporting negative progress.
> 2012-11-02 21:43:11,538 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - -795% complete
> ...
> (due to TFileRecordReader)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-3072) Pig job reporting negative progress
Posted by "Rohini Palaniswamy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-3072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13509920#comment-13509920 ]
Rohini Palaniswamy commented on PIG-3072:
-----------------------------------------
Koji,
Can you use HadoopShims to create the TaskAttemptContext in your test. The test fails to compile with H23.
{noformat}
[javac] /apache/pig/trunk/test/org/apache/pig/test/TestTmpFileCompression.java:369: org.apache.hadoop.mapreduce.TaskAttemptContext is abstract; cannot be instantiated
[javac] new TaskAttemptContext(conf, new TaskAttemptID()));
{noformat}
> Pig job reporting negative progress
> -----------------------------------
>
> Key: PIG-3072
> URL: https://issues.apache.org/jira/browse/PIG-3072
> Project: Pig
> Issue Type: Bug
> Components: impl
> Affects Versions: 0.10.0
> Reporter: Koji Noguchi
> Assignee: Koji Noguchi
> Priority: Minor
> Fix For: 0.12
>
> Attachments: pig-3072-v01.txt, pig-3072-v02.txt, pig-3072-v03.txt
>
>
> Our users pointed out that their jobs reporting negative progress.
> 2012-11-02 21:43:11,538 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - -795% complete
> ...
> (due to TFileRecordReader)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (PIG-3072) Pig job reporting negative progress
Posted by "Koji Noguchi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-3072?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Koji Noguchi updated PIG-3072:
------------------------------
Attachment: pig-3072-v04.txt
bq. Can you use HadoopShims to create the TaskAttemptContext in your test. The test fails to compile with H23.
Thanks Rohini. Uploading another patch with your suggestion.
Ran both
$ ant clean test -Dtestcase=TestTmpFileCompression
$ ant -Dhadoopversion=23 clean test -Dtestcase=TestTmpFileCompression
> Pig job reporting negative progress
> -----------------------------------
>
> Key: PIG-3072
> URL: https://issues.apache.org/jira/browse/PIG-3072
> Project: Pig
> Issue Type: Bug
> Components: impl
> Affects Versions: 0.10.0
> Reporter: Koji Noguchi
> Assignee: Koji Noguchi
> Priority: Minor
> Fix For: 0.12
>
> Attachments: pig-3072-v01.txt, pig-3072-v02.txt, pig-3072-v03.txt, pig-3072-v04.txt
>
>
> Our users pointed out that their jobs reporting negative progress.
> 2012-11-02 21:43:11,538 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - -795% complete
> ...
> (due to TFileRecordReader)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (PIG-3072) Pig job reporting negative progress
Posted by "Koji Noguchi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-3072?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Koji Noguchi updated PIG-3072:
------------------------------
Attachment: pig-3072-v03.txt
(Minor changes to previous v02 patch.)
* Added comment in TFileRecordReader. "//TFile.Reader reads into buffer so, progress is updated in chunks."
* Printing out progress to stderr.
* Changed loop size from 1000 to 25000 so that at least progress changes once for codec='none'. 66%->99% For 'gz', it immediately jumps to 99% with this small tfile.
* Added one extra check to make sure value is being read by recordReader. (curval == LOOP_SIZE)
> Pig job reporting negative progress
> -----------------------------------
>
> Key: PIG-3072
> URL: https://issues.apache.org/jira/browse/PIG-3072
> Project: Pig
> Issue Type: Bug
> Components: impl
> Affects Versions: 0.10.0
> Reporter: Koji Noguchi
> Assignee: Koji Noguchi
> Priority: Minor
> Fix For: 0.12
>
> Attachments: pig-3072-v01.txt, pig-3072-v02.txt, pig-3072-v03.txt
>
>
> Our users pointed out that their jobs reporting negative progress.
> 2012-11-02 21:43:11,538 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - -795% complete
> ...
> (due to TFileRecordReader)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (PIG-3072) Pig job reporting negative progress
Posted by "Koji Noguchi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-3072?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Koji Noguchi updated PIG-3072:
------------------------------
Status: Patch Available (was: Open)
> Pig job reporting negative progress
> -----------------------------------
>
> Key: PIG-3072
> URL: https://issues.apache.org/jira/browse/PIG-3072
> Project: Pig
> Issue Type: Bug
> Components: impl
> Affects Versions: 0.10.0
> Reporter: Koji Noguchi
> Assignee: Koji Noguchi
> Priority: Minor
> Fix For: 0.12
>
> Attachments: pig-3072-v01.txt, pig-3072-v02.txt
>
>
> Our users pointed out that their jobs reporting negative progress.
> 2012-11-02 21:43:11,538 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - -795% complete
> ...
> (due to TFileRecordReader)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (PIG-3072) Pig job reporting negative progress
Posted by "Koji Noguchi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-3072?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Koji Noguchi updated PIG-3072:
------------------------------
Attachment: pig-3072-v02.txt
bq. Can we create the "basic.tfile" using Util.createTempFileDelOnExit().
That makes sense.
Since Util.createTempFileDelOnExit() creates an empty file and TFileRecordWriter was failing with "File already exists" error, I added an extra delete before passing the path to TFileRecordWriter.
> Pig job reporting negative progress
> -----------------------------------
>
> Key: PIG-3072
> URL: https://issues.apache.org/jira/browse/PIG-3072
> Project: Pig
> Issue Type: Bug
> Components: impl
> Affects Versions: 0.10.0
> Reporter: Koji Noguchi
> Assignee: Koji Noguchi
> Priority: Minor
> Fix For: 0.12
>
> Attachments: pig-3072-v01.txt, pig-3072-v02.txt
>
>
> Our users pointed out that their jobs reporting negative progress.
> 2012-11-02 21:43:11,538 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - -795% complete
> ...
> (due to TFileRecordReader)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (PIG-3072) Pig job reporting negative progress
Posted by "Rohini Palaniswamy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-3072?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Rohini Palaniswamy updated PIG-3072:
------------------------------------
Resolution: Fixed
Release Note: Committed to trunk. Thanks Koji.
Status: Resolved (was: Patch Available)
> Pig job reporting negative progress
> -----------------------------------
>
> Key: PIG-3072
> URL: https://issues.apache.org/jira/browse/PIG-3072
> Project: Pig
> Issue Type: Bug
> Components: impl
> Affects Versions: 0.10.0
> Reporter: Koji Noguchi
> Assignee: Koji Noguchi
> Priority: Minor
> Fix For: 0.12
>
> Attachments: pig-3072-v01.txt, pig-3072-v02.txt, pig-3072-v03.txt, pig-3072-v04.txt
>
>
> Our users pointed out that their jobs reporting negative progress.
> 2012-11-02 21:43:11,538 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - -795% complete
> ...
> (due to TFileRecordReader)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (PIG-3072) Pig job reporting negative progress
Posted by "Rohini Palaniswamy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-3072?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Rohini Palaniswamy updated PIG-3072:
------------------------------------
Affects Version/s: 0.10.0
Fix Version/s: 0.12
Just a minor comment. Can we create the "basic.tfile" using Util.createTempFileDelOnExit(). Tests clash because of hardcoded file paths and it is hard to run unit tests for different versions at the same time. At least for newer tests we should try to avoid using hardcoded paths.
> Pig job reporting negative progress
> -----------------------------------
>
> Key: PIG-3072
> URL: https://issues.apache.org/jira/browse/PIG-3072
> Project: Pig
> Issue Type: Bug
> Components: impl
> Affects Versions: 0.10.0
> Reporter: Koji Noguchi
> Assignee: Koji Noguchi
> Priority: Minor
> Fix For: 0.12
>
> Attachments: pig-3072-v01.txt
>
>
> Our users pointed out that their jobs reporting negative progress.
> 2012-11-02 21:43:11,538 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - -795% complete
> ...
> (due to TFileRecordReader)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-3072) Pig job reporting negative progress
Posted by "Rohini Palaniswamy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-3072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13509345#comment-13509345 ]
Rohini Palaniswamy commented on PIG-3072:
-----------------------------------------
+1. Will commit soon.
> Pig job reporting negative progress
> -----------------------------------
>
> Key: PIG-3072
> URL: https://issues.apache.org/jira/browse/PIG-3072
> Project: Pig
> Issue Type: Bug
> Components: impl
> Affects Versions: 0.10.0
> Reporter: Koji Noguchi
> Assignee: Koji Noguchi
> Priority: Minor
> Fix For: 0.12
>
> Attachments: pig-3072-v01.txt, pig-3072-v02.txt
>
>
> Our users pointed out that their jobs reporting negative progress.
> 2012-11-02 21:43:11,538 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - -795% complete
> ...
> (due to TFileRecordReader)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (PIG-3072) Pig job reporting negative progress
Posted by "Koji Noguchi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-3072?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Koji Noguchi updated PIG-3072:
------------------------------
Attachment: pig-3072-v01.txt
'pos' inside TFileRecordReader was never updated. Using fileIn.getPos() instead for reporting progress.
> Pig job reporting negative progress
> -----------------------------------
>
> Key: PIG-3072
> URL: https://issues.apache.org/jira/browse/PIG-3072
> Project: Pig
> Issue Type: Bug
> Components: impl
> Reporter: Koji Noguchi
> Assignee: Koji Noguchi
> Priority: Minor
> Attachments: pig-3072-v01.txt
>
>
> Our users pointed out that their jobs reporting negative progress.
> 2012-11-02 21:43:11,538 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - -795% complete
> ...
> (due to TFileRecordReader)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira