You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by "Daniel Dai (JIRA)" <ji...@apache.org> on 2010/02/03 20:29:29 UTC
[jira] Created: (PIG-1219) Extra call to the namenode in
WeightedRangePartitioner
Extra call to the namenode in WeightedRangePartitioner
------------------------------------------------------
Key: PIG-1219
URL: https://issues.apache.org/jira/browse/PIG-1219
Project: Pig
Issue Type: Bug
Components: impl
Affects Versions: 0.6.0
Reporter: Daniel Dai
Assignee: Daniel Dai
Fix For: 0.6.0
We call FileLocalizer.openDFSFile in WeightedRangePartitioner to open quantile file. openDFSFile internally will check the existence of the quantile file, which adds burden to hdfs namenode. We shall remove this extra check.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1219) Extra listStatus call to the namenode
in WeightedRangePartitioner
Posted by "Olga Natkovich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12829690#action_12829690 ]
Olga Natkovich commented on PIG-1219:
-------------------------------------
I asked Pradeep to also review the code - just to have another set of eyes since this change is so late in the game and is not streighforward
> Extra listStatus call to the namenode in WeightedRangePartitioner
> -----------------------------------------------------------------
>
> Key: PIG-1219
> URL: https://issues.apache.org/jira/browse/PIG-1219
> Project: Pig
> Issue Type: Bug
> Components: impl
> Affects Versions: 0.6.0
> Reporter: Daniel Dai
> Assignee: Daniel Dai
> Fix For: 0.6.0
>
> Attachments: PIG-1219-1.patch, PIG-1219-2.patch, PIG-1219-3.patch
>
>
> We call FileLocalizer.openDFSFile in WeightedRangePartitioner to open quantile file. openDFSFile internally will check the existence of the quantile file, which adds burden to hdfs namenode. We shall remove this extra check.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1219) Extra listStatus call to the namenode in
WeightedRangePartitioner
Posted by "Daniel Dai (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Daniel Dai updated PIG-1219:
----------------------------
Status: Patch Available (was: Open)
> Extra listStatus call to the namenode in WeightedRangePartitioner
> -----------------------------------------------------------------
>
> Key: PIG-1219
> URL: https://issues.apache.org/jira/browse/PIG-1219
> Project: Pig
> Issue Type: Bug
> Components: impl
> Affects Versions: 0.6.0
> Reporter: Daniel Dai
> Assignee: Daniel Dai
> Fix For: 0.6.0
>
> Attachments: PIG-1219-1.patch
>
>
> We call FileLocalizer.openDFSFile in WeightedRangePartitioner to open quantile file. openDFSFile internally will check the existence of the quantile file, which adds burden to hdfs namenode. We shall remove this extra check.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1219) Extra listStatus call to the namenode
in WeightedRangePartitioner
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12829575#action_12829575 ]
Hadoop QA commented on PIG-1219:
--------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12434747/PIG-1219-2.patch
against trunk revision 906326.
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no tests are needed for this patch.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/191/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/191/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/191/console
This message is automatically generated.
> Extra listStatus call to the namenode in WeightedRangePartitioner
> -----------------------------------------------------------------
>
> Key: PIG-1219
> URL: https://issues.apache.org/jira/browse/PIG-1219
> Project: Pig
> Issue Type: Bug
> Components: impl
> Affects Versions: 0.6.0
> Reporter: Daniel Dai
> Assignee: Daniel Dai
> Fix For: 0.6.0
>
> Attachments: PIG-1219-1.patch, PIG-1219-2.patch
>
>
> We call FileLocalizer.openDFSFile in WeightedRangePartitioner to open quantile file. openDFSFile internally will check the existence of the quantile file, which adds burden to hdfs namenode. We shall remove this extra check.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1219) Extra listStatus call to the namenode in
WeightedRangePartitioner
Posted by "Daniel Dai (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Daniel Dai updated PIG-1219:
----------------------------
Status: Patch Available (was: Open)
> Extra listStatus call to the namenode in WeightedRangePartitioner
> -----------------------------------------------------------------
>
> Key: PIG-1219
> URL: https://issues.apache.org/jira/browse/PIG-1219
> Project: Pig
> Issue Type: Bug
> Components: impl
> Affects Versions: 0.6.0
> Reporter: Daniel Dai
> Assignee: Daniel Dai
> Fix For: 0.6.0
>
> Attachments: PIG-1219-1.patch, PIG-1219-2.patch
>
>
> We call FileLocalizer.openDFSFile in WeightedRangePartitioner to open quantile file. openDFSFile internally will check the existence of the quantile file, which adds burden to hdfs namenode. We shall remove this extra check.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1219) Extra listStatus call to the namenode in
WeightedRangePartitioner
Posted by "Daniel Dai (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Daniel Dai updated PIG-1219:
----------------------------
Resolution: Won't Fix
Status: Resolved (was: Patch Available)
Will go for distributed cache approach (https://issues.apache.org/jira/browse/PIG-1218). This patch is no longer needed then.
> Extra listStatus call to the namenode in WeightedRangePartitioner
> -----------------------------------------------------------------
>
> Key: PIG-1219
> URL: https://issues.apache.org/jira/browse/PIG-1219
> Project: Pig
> Issue Type: Bug
> Components: impl
> Affects Versions: 0.6.0
> Reporter: Daniel Dai
> Assignee: Daniel Dai
> Fix For: 0.6.0
>
> Attachments: PIG-1219-1.patch, PIG-1219-2.patch, PIG-1219-3.patch
>
>
> We call FileLocalizer.openDFSFile in WeightedRangePartitioner to open quantile file. openDFSFile internally will check the existence of the quantile file, which adds burden to hdfs namenode. We shall remove this extra check.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1219) Extra listStatus call to the namenode in
WeightedRangePartitioner
Posted by "Daniel Dai (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Daniel Dai updated PIG-1219:
----------------------------
Summary: Extra listStatus call to the namenode in WeightedRangePartitioner (was: Extra call to the namenode in WeightedRangePartitioner)
> Extra listStatus call to the namenode in WeightedRangePartitioner
> -----------------------------------------------------------------
>
> Key: PIG-1219
> URL: https://issues.apache.org/jira/browse/PIG-1219
> Project: Pig
> Issue Type: Bug
> Components: impl
> Affects Versions: 0.6.0
> Reporter: Daniel Dai
> Assignee: Daniel Dai
> Fix For: 0.6.0
>
>
> We call FileLocalizer.openDFSFile in WeightedRangePartitioner to open quantile file. openDFSFile internally will check the existence of the quantile file, which adds burden to hdfs namenode. We shall remove this extra check.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1219) Extra listStatus call to the namenode in
WeightedRangePartitioner
Posted by "Daniel Dai (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Daniel Dai updated PIG-1219:
----------------------------
Attachment: PIG-1219-2.patch
Thanks Richard. Post updated patch.
> Extra listStatus call to the namenode in WeightedRangePartitioner
> -----------------------------------------------------------------
>
> Key: PIG-1219
> URL: https://issues.apache.org/jira/browse/PIG-1219
> Project: Pig
> Issue Type: Bug
> Components: impl
> Affects Versions: 0.6.0
> Reporter: Daniel Dai
> Assignee: Daniel Dai
> Fix For: 0.6.0
>
> Attachments: PIG-1219-1.patch, PIG-1219-2.patch
>
>
> We call FileLocalizer.openDFSFile in WeightedRangePartitioner to open quantile file. openDFSFile internally will check the existence of the quantile file, which adds burden to hdfs namenode. We shall remove this extra check.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1219) Extra listStatus call to the namenode in
WeightedRangePartitioner
Posted by "Daniel Dai (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Daniel Dai updated PIG-1219:
----------------------------
Status: Open (was: Patch Available)
> Extra listStatus call to the namenode in WeightedRangePartitioner
> -----------------------------------------------------------------
>
> Key: PIG-1219
> URL: https://issues.apache.org/jira/browse/PIG-1219
> Project: Pig
> Issue Type: Bug
> Components: impl
> Affects Versions: 0.6.0
> Reporter: Daniel Dai
> Assignee: Daniel Dai
> Fix For: 0.6.0
>
> Attachments: PIG-1219-1.patch, PIG-1219-2.patch
>
>
> We call FileLocalizer.openDFSFile in WeightedRangePartitioner to open quantile file. openDFSFile internally will check the existence of the quantile file, which adds burden to hdfs namenode. We shall remove this extra check.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1219) Extra listStatus call to the namenode in
WeightedRangePartitioner
Posted by "Daniel Dai (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Daniel Dai updated PIG-1219:
----------------------------
Attachment: PIG-1219-1.patch
I am still testing with the patch. Attach it first so other committers can review.
> Extra listStatus call to the namenode in WeightedRangePartitioner
> -----------------------------------------------------------------
>
> Key: PIG-1219
> URL: https://issues.apache.org/jira/browse/PIG-1219
> Project: Pig
> Issue Type: Bug
> Components: impl
> Affects Versions: 0.6.0
> Reporter: Daniel Dai
> Assignee: Daniel Dai
> Fix For: 0.6.0
>
> Attachments: PIG-1219-1.patch
>
>
> We call FileLocalizer.openDFSFile in WeightedRangePartitioner to open quantile file. openDFSFile internally will check the existence of the quantile file, which adds burden to hdfs namenode. We shall remove this extra check.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1219) Extra listStatus call to the namenode
in WeightedRangePartitioner
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12829865#action_12829865 ]
Hadoop QA commented on PIG-1219:
--------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12434850/PIG-1219-3.patch
against trunk revision 906326.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/193/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/193/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/193/console
This message is automatically generated.
> Extra listStatus call to the namenode in WeightedRangePartitioner
> -----------------------------------------------------------------
>
> Key: PIG-1219
> URL: https://issues.apache.org/jira/browse/PIG-1219
> Project: Pig
> Issue Type: Bug
> Components: impl
> Affects Versions: 0.6.0
> Reporter: Daniel Dai
> Assignee: Daniel Dai
> Fix For: 0.6.0
>
> Attachments: PIG-1219-1.patch, PIG-1219-2.patch, PIG-1219-3.patch
>
>
> We call FileLocalizer.openDFSFile in WeightedRangePartitioner to open quantile file. openDFSFile internally will check the existence of the quantile file, which adds burden to hdfs namenode. We shall remove this extra check.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1219) Extra listStatus call to the namenode
in WeightedRangePartitioner
Posted by "Richard Ding (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12829344#action_12829344 ]
Richard Ding commented on PIG-1219:
-----------------------------------
Looks good. One comment: JobControlCompiler should handle the case when quantile file doesn't exist.
> Extra listStatus call to the namenode in WeightedRangePartitioner
> -----------------------------------------------------------------
>
> Key: PIG-1219
> URL: https://issues.apache.org/jira/browse/PIG-1219
> Project: Pig
> Issue Type: Bug
> Components: impl
> Affects Versions: 0.6.0
> Reporter: Daniel Dai
> Assignee: Daniel Dai
> Fix For: 0.6.0
>
> Attachments: PIG-1219-1.patch
>
>
> We call FileLocalizer.openDFSFile in WeightedRangePartitioner to open quantile file. openDFSFile internally will check the existence of the quantile file, which adds burden to hdfs namenode. We shall remove this extra check.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1219) Extra listStatus call to the namenode in
WeightedRangePartitioner
Posted by "Daniel Dai (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Daniel Dai updated PIG-1219:
----------------------------
Status: Patch Available (was: Open)
> Extra listStatus call to the namenode in WeightedRangePartitioner
> -----------------------------------------------------------------
>
> Key: PIG-1219
> URL: https://issues.apache.org/jira/browse/PIG-1219
> Project: Pig
> Issue Type: Bug
> Components: impl
> Affects Versions: 0.6.0
> Reporter: Daniel Dai
> Assignee: Daniel Dai
> Fix For: 0.6.0
>
> Attachments: PIG-1219-1.patch, PIG-1219-2.patch, PIG-1219-3.patch
>
>
> We call FileLocalizer.openDFSFile in WeightedRangePartitioner to open quantile file. openDFSFile internally will check the existence of the quantile file, which adds burden to hdfs namenode. We shall remove this extra check.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1219) Extra listStatus call to the namenode in
WeightedRangePartitioner
Posted by "Daniel Dai (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Daniel Dai updated PIG-1219:
----------------------------
Status: Open (was: Patch Available)
> Extra listStatus call to the namenode in WeightedRangePartitioner
> -----------------------------------------------------------------
>
> Key: PIG-1219
> URL: https://issues.apache.org/jira/browse/PIG-1219
> Project: Pig
> Issue Type: Bug
> Components: impl
> Affects Versions: 0.6.0
> Reporter: Daniel Dai
> Assignee: Daniel Dai
> Fix For: 0.6.0
>
> Attachments: PIG-1219-1.patch, PIG-1219-2.patch
>
>
> We call FileLocalizer.openDFSFile in WeightedRangePartitioner to open quantile file. openDFSFile internally will check the existence of the quantile file, which adds burden to hdfs namenode. We shall remove this extra check.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1219) Extra listStatus call to the namenode in
WeightedRangePartitioner
Posted by "Daniel Dai (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Daniel Dai updated PIG-1219:
----------------------------
Attachment: PIG-1219-3.patch
The test failure is because the way we test it, not the core code. We now require the quantile file to be created before we run JobControlCompiler. In our testcase, we invoke the methods of JobControlCompiler directly without actually running the job, so we do not have quantile file when we get into JobControlCompiler. Change testcase to force create the quantile file.
> Extra listStatus call to the namenode in WeightedRangePartitioner
> -----------------------------------------------------------------
>
> Key: PIG-1219
> URL: https://issues.apache.org/jira/browse/PIG-1219
> Project: Pig
> Issue Type: Bug
> Components: impl
> Affects Versions: 0.6.0
> Reporter: Daniel Dai
> Assignee: Daniel Dai
> Fix For: 0.6.0
>
> Attachments: PIG-1219-1.patch, PIG-1219-2.patch, PIG-1219-3.patch
>
>
> We call FileLocalizer.openDFSFile in WeightedRangePartitioner to open quantile file. openDFSFile internally will check the existence of the quantile file, which adds burden to hdfs namenode. We shall remove this extra check.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1219) Extra listStatus call to the namenode
in WeightedRangePartitioner
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12829434#action_12829434 ]
Hadoop QA commented on PIG-1219:
--------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12434733/PIG-1219-1.patch
against trunk revision 905377.
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no tests are needed for this patch.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/190/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/190/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/190/console
This message is automatically generated.
> Extra listStatus call to the namenode in WeightedRangePartitioner
> -----------------------------------------------------------------
>
> Key: PIG-1219
> URL: https://issues.apache.org/jira/browse/PIG-1219
> Project: Pig
> Issue Type: Bug
> Components: impl
> Affects Versions: 0.6.0
> Reporter: Daniel Dai
> Assignee: Daniel Dai
> Fix For: 0.6.0
>
> Attachments: PIG-1219-1.patch, PIG-1219-2.patch
>
>
> We call FileLocalizer.openDFSFile in WeightedRangePartitioner to open quantile file. openDFSFile internally will check the existence of the quantile file, which adds burden to hdfs namenode. We shall remove this extra check.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.