You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by "Namit Jain (JIRA)" <ji...@apache.org> on 2009/04/22 07:15:48 UTC
[jira] Created: (HIVE-440) no way to know number of rows for a
reducer and join keys
no way to know number of rows for a reducer and join keys
---------------------------------------------------------
Key: HIVE-440
URL: https://issues.apache.org/jira/browse/HIVE-440
Project: Hadoop Hive
Issue Type: Wish
Components: Query Processor
Reporter: Namit Jain
Assignee: Namit Jain
It is a good debugging tool to dump number of rows for a reducer, and to emit the number of entries for a join key for a internal table
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-440) no way to know number of rows for a
reducer and join keys
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Namit Jain updated HIVE-440:
----------------------------
Attachment: hive.440.1.patch
> no way to know number of rows for a reducer and join keys
> ---------------------------------------------------------
>
> Key: HIVE-440
> URL: https://issues.apache.org/jira/browse/HIVE-440
> Project: Hadoop Hive
> Issue Type: Wish
> Components: Query Processor
> Reporter: Namit Jain
> Assignee: Namit Jain
> Attachments: hive.440.1.patch
>
>
> It is a good debugging tool to dump number of rows for a reducer, and to emit the number of entries for a join key for a internal table
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-440) no way to know number of rows for a
reducer and join keys
Posted by "Zheng Shao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12701677#action_12701677 ]
Zheng Shao commented on HIVE-440:
---------------------------------
JoinOperator.java
{code}
+ private int getNextSize(int sz) {
+ // A very simple counter to keep track of join entres for a key
+ if ((sz % 128000) == 0)
+ return sz + 128000;
+
+ return 2 * sz;
+ }
+
{code}
EmitInterval is configurable, so there is no way to make sure "(sz % 128000) == 0" will happen. It's better to say "sz >= 128000" I think.
We might want to change the same logic in ExecReducer for precaution.
For ExecReducer, I would prefer to start the counting from 1 instead of 1000. It's 3 additional lines of log, but will give us some insights whether the operator is blocking on the first row, or map-reduce framework never send the first row to our Operators.
> no way to know number of rows for a reducer and join keys
> ---------------------------------------------------------
>
> Key: HIVE-440
> URL: https://issues.apache.org/jira/browse/HIVE-440
> Project: Hadoop Hive
> Issue Type: Wish
> Components: Query Processor
> Reporter: Namit Jain
> Assignee: Namit Jain
> Attachments: hive.440.1.patch
>
>
> It is a good debugging tool to dump number of rows for a reducer, and to emit the number of entries for a join key for a internal table
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Reopened: (HIVE-440) no way to know number of rows for a
reducer and join keys
Posted by "Zheng Shao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Zheng Shao reopened HIVE-440:
-----------------------------
branch 0.3 was always outputing "processed 0 rows" since cntr++ is missing from the patch.
> no way to know number of rows for a reducer and join keys
> ---------------------------------------------------------
>
> Key: HIVE-440
> URL: https://issues.apache.org/jira/browse/HIVE-440
> Project: Hadoop Hive
> Issue Type: Wish
> Components: Query Processor
> Reporter: Namit Jain
> Assignee: Namit Jain
> Fix For: 0.3.1, 0.4.0
>
> Attachments: HIVE-440.br3.1.add.patch, hive.440.1.patch, hive.440.2.patch, hive.440.br3.1.patch
>
>
> It is a good debugging tool to dump number of rows for a reducer, and to emit the number of entries for a join key for a internal table
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-440) no way to know number of rows for a
reducer and join keys
Posted by "Zheng Shao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Zheng Shao updated HIVE-440:
----------------------------
Attachment: HIVE-440.br3.1.add.patch
The HIVE-440.br3.1.patch is missing one line: "cntr++;"
This patch fixed it.
> no way to know number of rows for a reducer and join keys
> ---------------------------------------------------------
>
> Key: HIVE-440
> URL: https://issues.apache.org/jira/browse/HIVE-440
> Project: Hadoop Hive
> Issue Type: Wish
> Components: Query Processor
> Reporter: Namit Jain
> Assignee: Namit Jain
> Fix For: 0.3.1, 0.4.0
>
> Attachments: HIVE-440.br3.1.add.patch, hive.440.1.patch, hive.440.2.patch, hive.440.br3.1.patch
>
>
> It is a good debugging tool to dump number of rows for a reducer, and to emit the number of entries for a join key for a internal table
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-440) no way to know number of rows for a
reducer and join keys
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Namit Jain updated HIVE-440:
----------------------------
Status: Open (was: Patch Available)
> no way to know number of rows for a reducer and join keys
> ---------------------------------------------------------
>
> Key: HIVE-440
> URL: https://issues.apache.org/jira/browse/HIVE-440
> Project: Hadoop Hive
> Issue Type: Wish
> Components: Query Processor
> Reporter: Namit Jain
> Assignee: Namit Jain
> Attachments: hive.440.1.patch, hive.440.2.patch
>
>
> It is a good debugging tool to dump number of rows for a reducer, and to emit the number of entries for a join key for a internal table
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-440) no way to know number of rows for a
reducer and join keys
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Namit Jain updated HIVE-440:
----------------------------
Status: Patch Available (was: Open)
> no way to know number of rows for a reducer and join keys
> ---------------------------------------------------------
>
> Key: HIVE-440
> URL: https://issues.apache.org/jira/browse/HIVE-440
> Project: Hadoop Hive
> Issue Type: Wish
> Components: Query Processor
> Reporter: Namit Jain
> Assignee: Namit Jain
> Attachments: hive.440.1.patch, hive.440.2.patch
>
>
> It is a good debugging tool to dump number of rows for a reducer, and to emit the number of entries for a join key for a internal table
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-440) no way to know number of rows for a
reducer and join keys
Posted by "Carl Steinbach (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Carl Steinbach updated HIVE-440:
--------------------------------
Fix Version/s: (was: 0.3.1)
> no way to know number of rows for a reducer and join keys
> ---------------------------------------------------------
>
> Key: HIVE-440
> URL: https://issues.apache.org/jira/browse/HIVE-440
> Project: Hadoop Hive
> Issue Type: Wish
> Components: Query Processor
> Reporter: Namit Jain
> Assignee: Namit Jain
> Fix For: 0.4.0
>
> Attachments: HIVE-440.br3.1.add.patch, hive.440.1.patch, hive.440.2.patch, hive.440.br3.1.patch
>
>
> It is a good debugging tool to dump number of rows for a reducer, and to emit the number of entries for a join key for a internal table
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-440) no way to know number of rows for a
reducer and join keys
Posted by "Zheng Shao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12701753#action_12701753 ]
Zheng Shao commented on HIVE-440:
---------------------------------
Committed to trunk, but it cannot be applied to branch-0.3 cleanly.
Can you generate a patch for branch 0.3, Namit?
> no way to know number of rows for a reducer and join keys
> ---------------------------------------------------------
>
> Key: HIVE-440
> URL: https://issues.apache.org/jira/browse/HIVE-440
> Project: Hadoop Hive
> Issue Type: Wish
> Components: Query Processor
> Reporter: Namit Jain
> Assignee: Namit Jain
> Attachments: hive.440.1.patch, hive.440.2.patch
>
>
> It is a good debugging tool to dump number of rows for a reducer, and to emit the number of entries for a join key for a internal table
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-440) no way to know number of rows for a
reducer and join keys
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Namit Jain updated HIVE-440:
----------------------------
Attachment: hive.440.2.patch
incorporated Zheng's comments
> no way to know number of rows for a reducer and join keys
> ---------------------------------------------------------
>
> Key: HIVE-440
> URL: https://issues.apache.org/jira/browse/HIVE-440
> Project: Hadoop Hive
> Issue Type: Wish
> Components: Query Processor
> Reporter: Namit Jain
> Assignee: Namit Jain
> Attachments: hive.440.1.patch, hive.440.2.patch
>
>
> It is a good debugging tool to dump number of rows for a reducer, and to emit the number of entries for a join key for a internal table
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (HIVE-440) no way to know number of rows for a
reducer and join keys
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Namit Jain resolved HIVE-440.
-----------------------------
Resolution: Fixed
committed. Thanks Zheng
> no way to know number of rows for a reducer and join keys
> ---------------------------------------------------------
>
> Key: HIVE-440
> URL: https://issues.apache.org/jira/browse/HIVE-440
> Project: Hadoop Hive
> Issue Type: Wish
> Components: Query Processor
> Reporter: Namit Jain
> Assignee: Namit Jain
> Fix For: 0.3.1, 0.4.0
>
> Attachments: HIVE-440.br3.1.add.patch, hive.440.1.patch, hive.440.2.patch, hive.440.br3.1.patch
>
>
> It is a good debugging tool to dump number of rows for a reducer, and to emit the number of entries for a join key for a internal table
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-440) no way to know number of rows for a
reducer and join keys
Posted by "Zheng Shao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Zheng Shao updated HIVE-440:
----------------------------
Resolution: Fixed
Fix Version/s: 0.4.0
0.3.1
Release Note: HIVE-440. Reducer and Join to print out number of rows processed. (Namit Jain via zshao)
Hadoop Flags: [Reviewed]
Status: Resolved (was: Patch Available)
Committed to both trunk and branch-0.3. Thanks Namit!
> no way to know number of rows for a reducer and join keys
> ---------------------------------------------------------
>
> Key: HIVE-440
> URL: https://issues.apache.org/jira/browse/HIVE-440
> Project: Hadoop Hive
> Issue Type: Wish
> Components: Query Processor
> Reporter: Namit Jain
> Assignee: Namit Jain
> Fix For: 0.3.1, 0.4.0
>
> Attachments: hive.440.1.patch, hive.440.2.patch, hive.440.br3.1.patch
>
>
> It is a good debugging tool to dump number of rows for a reducer, and to emit the number of entries for a join key for a internal table
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-440) no way to know number of rows for a
reducer and join keys
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Namit Jain updated HIVE-440:
----------------------------
Attachment: hive.440.br3.1.patch
> no way to know number of rows for a reducer and join keys
> ---------------------------------------------------------
>
> Key: HIVE-440
> URL: https://issues.apache.org/jira/browse/HIVE-440
> Project: Hadoop Hive
> Issue Type: Wish
> Components: Query Processor
> Reporter: Namit Jain
> Assignee: Namit Jain
> Attachments: hive.440.1.patch, hive.440.2.patch, hive.440.br3.1.patch
>
>
> It is a good debugging tool to dump number of rows for a reducer, and to emit the number of entries for a join key for a internal table
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-440) no way to know number of rows for a
reducer and join keys
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Namit Jain updated HIVE-440:
----------------------------
Status: Patch Available (was: Open)
> no way to know number of rows for a reducer and join keys
> ---------------------------------------------------------
>
> Key: HIVE-440
> URL: https://issues.apache.org/jira/browse/HIVE-440
> Project: Hadoop Hive
> Issue Type: Wish
> Components: Query Processor
> Reporter: Namit Jain
> Assignee: Namit Jain
> Attachments: hive.440.1.patch
>
>
> It is a good debugging tool to dump number of rows for a reducer, and to emit the number of entries for a join key for a internal table
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.