You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by "Raghotham Murthy (JIRA)" <ji...@apache.org> on 2009/07/31 10:33:14 UTC
[jira] Created: (HIVE-713) Integrate operator counters into query
plan to view progress of queries
Integrate operator counters into query plan to view progress of queries
-----------------------------------------------------------------------
Key: HIVE-713
URL: https://issues.apache.org/jira/browse/HIVE-713
Project: Hadoop Hive
Issue Type: New Feature
Components: Logging
Reporter: Raghotham Murthy
Assignee: Raghotham Murthy
Annotate the QueryPlan - with multiple stages and each stage(Task) can have an Operator tree - with counters that each Operator/Task updates.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-713) Integrate operator counters into query
plan to view progress of queries
Posted by "Raghotham Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Raghotham Murthy updated HIVE-713:
----------------------------------
Attachment: hive-713.3.patch
Could not get TBinaryProtocol/TMemoryBuffer combination to work across java and php. So, I just implemented toString() in QueryPlan which generates a JSON string. The entire query plan with counters is written out in JSON in the history file everytime there is an update.
With this patch, we should be able to visualize the plan along with the counters.
> Integrate operator counters into query plan to view progress of queries
> -----------------------------------------------------------------------
>
> Key: HIVE-713
> URL: https://issues.apache.org/jira/browse/HIVE-713
> Project: Hadoop Hive
> Issue Type: New Feature
> Components: Logging
> Reporter: Raghotham Murthy
> Assignee: Raghotham Murthy
> Attachments: hive-713.1.donot.patch, hive-713.2.donot.patch, hive-713.2.patch, hive-713.3.patch, hive-713.4.donot.patch, hive-713.5.donot.patch, hive.713.1.good.patch, hive.713.patch.bad
>
>
> Annotate the QueryPlan - with multiple stages and each stage(Task) can have an Operator tree - with counters that each Operator/Task updates.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-713) Integrate operator counters into query
plan to view progress of queries
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12744796#action_12744796 ]
Namit Jain commented on HIVE-713:
---------------------------------
Had a code review with Zheng - filing the following jiras for follow-up tasks
https://issues.apache.org/jira/browse/HIVE-766
https://issues.apache.org/jira/browse/HIVE-767
> Integrate operator counters into query plan to view progress of queries
> -----------------------------------------------------------------------
>
> Key: HIVE-713
> URL: https://issues.apache.org/jira/browse/HIVE-713
> Project: Hadoop Hive
> Issue Type: New Feature
> Components: Logging
> Reporter: Raghotham Murthy
> Assignee: Raghotham Murthy
> Attachments: hive-713.1.donot.patch, hive-713.2.donot.patch, hive-713.2.patch, hive-713.3.patch, hive-713.4.donot.patch, hive-713.5.donot.patch, hive-713.5.patch, hive-713.6.patch, hive.713.1.good.patch, hive.713.7.patch, hive.713.fixdiffs, hive.713.patch.bad, newpatch
>
>
> Annotate the QueryPlan - with multiple stages and each stage(Task) can have an Operator tree - with counters that each Operator/Task updates.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-713) Integrate operator counters into query
plan to view progress of queries
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Namit Jain updated HIVE-713:
----------------------------
Attachment: hive.713.fixdiffs
fixed diffs
> Integrate operator counters into query plan to view progress of queries
> -----------------------------------------------------------------------
>
> Key: HIVE-713
> URL: https://issues.apache.org/jira/browse/HIVE-713
> Project: Hadoop Hive
> Issue Type: New Feature
> Components: Logging
> Reporter: Raghotham Murthy
> Assignee: Raghotham Murthy
> Attachments: hive-713.1.donot.patch, hive-713.2.donot.patch, hive-713.2.patch, hive-713.3.patch, hive-713.4.donot.patch, hive-713.5.donot.patch, hive.713.1.good.patch, hive.713.fixdiffs, hive.713.patch.bad
>
>
> Annotate the QueryPlan - with multiple stages and each stage(Task) can have an Operator tree - with counters that each Operator/Task updates.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-713) Integrate operator counters into query
plan to view progress of queries
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Namit Jain updated HIVE-713:
----------------------------
Attachment: hive.713.1.good.patch
Fixed the bugs - the tests run clean now
Raghu, can you add your stuff and we are good to go
> Integrate operator counters into query plan to view progress of queries
> -----------------------------------------------------------------------
>
> Key: HIVE-713
> URL: https://issues.apache.org/jira/browse/HIVE-713
> Project: Hadoop Hive
> Issue Type: New Feature
> Components: Logging
> Reporter: Raghotham Murthy
> Assignee: Raghotham Murthy
> Attachments: hive-713.1.donot.patch, hive-713.2.donot.patch, hive-713.4.donot.patch, hive-713.5.donot.patch, hive.713.1.good.patch, hive.713.patch.bad
>
>
> Annotate the QueryPlan - with multiple stages and each stage(Task) can have an Operator tree - with counters that each Operator/Task updates.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-713) Integrate operator counters into query
plan to view progress of queries
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Namit Jain updated HIVE-713:
----------------------------
Attachment: hive.713.patch.bad
made some further progress - bad things are not complete -
uploading the patch so that raghu can proceed if he wants to
> Integrate operator counters into query plan to view progress of queries
> -----------------------------------------------------------------------
>
> Key: HIVE-713
> URL: https://issues.apache.org/jira/browse/HIVE-713
> Project: Hadoop Hive
> Issue Type: New Feature
> Components: Logging
> Reporter: Raghotham Murthy
> Assignee: Raghotham Murthy
> Attachments: hive-713.1.donot.patch, hive-713.2.donot.patch, hive.713.patch.bad
>
>
> Annotate the QueryPlan - with multiple stages and each stage(Task) can have an Operator tree - with counters that each Operator/Task updates.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-713) Integrate operator counters into query
plan to view progress of queries
Posted by "Raghotham Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Raghotham Murthy updated HIVE-713:
----------------------------------
Attachment: hive-713.4.donot.patch
added a thrift interface for the queryplan. we can dump out the json version of the query plan potentially using TJsonProtocol. Also added implementation for updateCounters.
Still need to figure out how to represent conditional tasks.
> Integrate operator counters into query plan to view progress of queries
> -----------------------------------------------------------------------
>
> Key: HIVE-713
> URL: https://issues.apache.org/jira/browse/HIVE-713
> Project: Hadoop Hive
> Issue Type: New Feature
> Components: Logging
> Reporter: Raghotham Murthy
> Assignee: Raghotham Murthy
> Attachments: hive-713.1.donot.patch, hive-713.2.donot.patch, hive-713.4.donot.patch, hive.713.patch.bad
>
>
> Annotate the QueryPlan - with multiple stages and each stage(Task) can have an Operator tree - with counters that each Operator/Task updates.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-713) Integrate operator counters into query
plan to view progress of queries
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Namit Jain updated HIVE-713:
----------------------------
Status: Patch Available (was: Open)
fixed the diffs
> Integrate operator counters into query plan to view progress of queries
> -----------------------------------------------------------------------
>
> Key: HIVE-713
> URL: https://issues.apache.org/jira/browse/HIVE-713
> Project: Hadoop Hive
> Issue Type: New Feature
> Components: Logging
> Reporter: Raghotham Murthy
> Assignee: Raghotham Murthy
> Attachments: hive-713.1.donot.patch, hive-713.2.donot.patch, hive-713.2.patch, hive-713.3.patch, hive-713.4.donot.patch, hive-713.5.donot.patch, hive-713.5.patch, hive-713.6.patch, hive.713.1.good.patch, hive.713.7.patch, hive.713.fixdiffs, hive.713.patch.bad, newpatch
>
>
> Annotate the QueryPlan - with multiple stages and each stage(Task) can have an Operator tree - with counters that each Operator/Task updates.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-713) Integrate operator counters into query
plan to view progress of queries
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12744798#action_12744798 ]
Namit Jain commented on HIVE-713:
---------------------------------
and
https://issues.apache.org/jira/browse/HIVE-768
> Integrate operator counters into query plan to view progress of queries
> -----------------------------------------------------------------------
>
> Key: HIVE-713
> URL: https://issues.apache.org/jira/browse/HIVE-713
> Project: Hadoop Hive
> Issue Type: New Feature
> Components: Logging
> Reporter: Raghotham Murthy
> Assignee: Raghotham Murthy
> Attachments: hive-713.1.donot.patch, hive-713.2.donot.patch, hive-713.2.patch, hive-713.3.patch, hive-713.4.donot.patch, hive-713.5.donot.patch, hive-713.5.patch, hive-713.6.patch, hive.713.1.good.patch, hive.713.7.patch, hive.713.fixdiffs, hive.713.patch.bad, newpatch
>
>
> Annotate the QueryPlan - with multiple stages and each stage(Task) can have an Operator tree - with counters that each Operator/Task updates.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-713) Integrate operator counters into query
plan to view progress of queries
Posted by "Raghotham Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Raghotham Murthy updated HIVE-713:
----------------------------------
Attachment: hive-713.5.patch
Added support for map/reduce progress, isDone() and handle ConditionalTask. Havent run all the tests yet.
> Integrate operator counters into query plan to view progress of queries
> -----------------------------------------------------------------------
>
> Key: HIVE-713
> URL: https://issues.apache.org/jira/browse/HIVE-713
> Project: Hadoop Hive
> Issue Type: New Feature
> Components: Logging
> Reporter: Raghotham Murthy
> Assignee: Raghotham Murthy
> Attachments: hive-713.1.donot.patch, hive-713.2.donot.patch, hive-713.2.patch, hive-713.3.patch, hive-713.4.donot.patch, hive-713.5.donot.patch, hive-713.5.patch, hive.713.1.good.patch, hive.713.fixdiffs, hive.713.patch.bad
>
>
> Annotate the QueryPlan - with multiple stages and each stage(Task) can have an Operator tree - with counters that each Operator/Task updates.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-713) Integrate operator counters into query
plan to view progress of queries
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Namit Jain updated HIVE-713:
----------------------------
Attachment: hive.713.8.patch
> Integrate operator counters into query plan to view progress of queries
> -----------------------------------------------------------------------
>
> Key: HIVE-713
> URL: https://issues.apache.org/jira/browse/HIVE-713
> Project: Hadoop Hive
> Issue Type: New Feature
> Components: Logging
> Reporter: Raghotham Murthy
> Assignee: Raghotham Murthy
> Attachments: hive-713.1.donot.patch, hive-713.2.donot.patch, hive-713.2.patch, hive-713.3.patch, hive-713.4.donot.patch, hive-713.5.donot.patch, hive-713.5.patch, hive-713.6.patch, hive.713.1.good.patch, hive.713.7.patch, hive.713.8.patch, hive.713.fixdiffs, hive.713.patch.bad, newpatch
>
>
> Annotate the QueryPlan - with multiple stages and each stage(Task) can have an Operator tree - with counters that each Operator/Task updates.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-713) Integrate operator counters into query
plan to view progress of queries
Posted by "Raghotham Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Raghotham Murthy updated HIVE-713:
----------------------------------
Attachment: (was: hive-713.2.patch)
> Integrate operator counters into query plan to view progress of queries
> -----------------------------------------------------------------------
>
> Key: HIVE-713
> URL: https://issues.apache.org/jira/browse/HIVE-713
> Project: Hadoop Hive
> Issue Type: New Feature
> Components: Logging
> Reporter: Raghotham Murthy
> Assignee: Raghotham Murthy
> Attachments: hive-713.1.donot.patch, hive-713.2.donot.patch, hive-713.2.patch, hive-713.4.donot.patch, hive-713.5.donot.patch, hive.713.1.good.patch, hive.713.patch.bad
>
>
> Annotate the QueryPlan - with multiple stages and each stage(Task) can have an Operator tree - with counters that each Operator/Task updates.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-713) Integrate operator counters into query
plan to view progress of queries
Posted by "Raghotham Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12740197#action_12740197 ]
Raghotham Murthy commented on HIVE-713:
---------------------------------------
I am not very familiar with JMX, but it seems useful in cases where command line tools need to read/modify the internal state of servers. Given that we are only reading stuff from the operators/tasks, would it be simpler to use a logging based approach like chukwa or scribe?
> Integrate operator counters into query plan to view progress of queries
> -----------------------------------------------------------------------
>
> Key: HIVE-713
> URL: https://issues.apache.org/jira/browse/HIVE-713
> Project: Hadoop Hive
> Issue Type: New Feature
> Components: Logging
> Reporter: Raghotham Murthy
> Assignee: Raghotham Murthy
> Attachments: hive-713.1.donot.patch, hive-713.2.donot.patch, hive-713.2.patch, hive-713.3.patch, hive-713.4.donot.patch, hive-713.5.donot.patch, hive.713.1.good.patch, hive.713.patch.bad
>
>
> Annotate the QueryPlan - with multiple stages and each stage(Task) can have an Operator tree - with counters that each Operator/Task updates.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-713) Integrate operator counters into query
plan to view progress of queries
Posted by "Raghotham Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12740238#action_12740238 ]
Raghotham Murthy commented on HIVE-713:
---------------------------------------
Some tests are failing with the latest patch. I am looking into it now.
> Integrate operator counters into query plan to view progress of queries
> -----------------------------------------------------------------------
>
> Key: HIVE-713
> URL: https://issues.apache.org/jira/browse/HIVE-713
> Project: Hadoop Hive
> Issue Type: New Feature
> Components: Logging
> Reporter: Raghotham Murthy
> Assignee: Raghotham Murthy
> Attachments: hive-713.1.donot.patch, hive-713.2.donot.patch, hive-713.2.patch, hive-713.3.patch, hive-713.4.donot.patch, hive-713.5.donot.patch, hive.713.1.good.patch, hive.713.patch.bad
>
>
> Annotate the QueryPlan - with multiple stages and each stage(Task) can have an Operator tree - with counters that each Operator/Task updates.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-713) Integrate operator counters into query
plan to view progress of queries
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Namit Jain updated HIVE-713:
----------------------------
Attachment: newpatch
fixed a lot of problems - however, a couple pf them still remain.
look at nullgroup3.q - for some reason, only the last file is picked - i think most of the other problems is in this category
> Integrate operator counters into query plan to view progress of queries
> -----------------------------------------------------------------------
>
> Key: HIVE-713
> URL: https://issues.apache.org/jira/browse/HIVE-713
> Project: Hadoop Hive
> Issue Type: New Feature
> Components: Logging
> Reporter: Raghotham Murthy
> Assignee: Raghotham Murthy
> Attachments: hive-713.1.donot.patch, hive-713.2.donot.patch, hive-713.2.patch, hive-713.3.patch, hive-713.4.donot.patch, hive-713.5.donot.patch, hive-713.5.patch, hive-713.6.patch, hive.713.1.good.patch, hive.713.fixdiffs, hive.713.patch.bad, newpatch
>
>
> Annotate the QueryPlan - with multiple stages and each stage(Task) can have an Operator tree - with counters that each Operator/Task updates.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-713) Integrate operator counters into query
plan to view progress of queries
Posted by "Raghotham Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Raghotham Murthy updated HIVE-713:
----------------------------------
Attachment: hive-713.2.donot.patch
More changes, but not yet ready to patch. Remaining tasks:
- in each operator, if there are new counters, call addToCounterNames() in initialize
- from SemanticAnalyzer, after plan generation walk all tasks and call assignCounterNameToEnum for each operator
- in each Operator.process increment counters
- in driver add call to return plan, dump queryplan to history file
- in ExecDriver.progress walk all operators and call updateCounters and dump out stats to history file
- in queryplan walk tasks/operators and return plan which includes counters
- in separate class provide command line option to parse history file and print latest plan
> Integrate operator counters into query plan to view progress of queries
> -----------------------------------------------------------------------
>
> Key: HIVE-713
> URL: https://issues.apache.org/jira/browse/HIVE-713
> Project: Hadoop Hive
> Issue Type: New Feature
> Components: Logging
> Reporter: Raghotham Murthy
> Assignee: Raghotham Murthy
> Attachments: hive-713.1.donot.patch, hive-713.2.donot.patch
>
>
> Annotate the QueryPlan - with multiple stages and each stage(Task) can have an Operator tree - with counters that each Operator/Task updates.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-713) Integrate operator counters into query
plan to view progress of queries
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Namit Jain updated HIVE-713:
----------------------------
Attachment: hive.713.7.patch
> Integrate operator counters into query plan to view progress of queries
> -----------------------------------------------------------------------
>
> Key: HIVE-713
> URL: https://issues.apache.org/jira/browse/HIVE-713
> Project: Hadoop Hive
> Issue Type: New Feature
> Components: Logging
> Reporter: Raghotham Murthy
> Assignee: Raghotham Murthy
> Attachments: hive-713.1.donot.patch, hive-713.2.donot.patch, hive-713.2.patch, hive-713.3.patch, hive-713.4.donot.patch, hive-713.5.donot.patch, hive-713.5.patch, hive-713.6.patch, hive.713.1.good.patch, hive.713.7.patch, hive.713.fixdiffs, hive.713.patch.bad, newpatch
>
>
> Annotate the QueryPlan - with multiple stages and each stage(Task) can have an Operator tree - with counters that each Operator/Task updates.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-713) Integrate operator counters into query
plan to view progress of queries
Posted by "Raghotham Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Raghotham Murthy updated HIVE-713:
----------------------------------
Attachment: hive-713.9.patch
Fixed problems which resulted in printing null plans. Also, fixed a bug where childTasks' status was not getting updated if the root task was done.
> Integrate operator counters into query plan to view progress of queries
> -----------------------------------------------------------------------
>
> Key: HIVE-713
> URL: https://issues.apache.org/jira/browse/HIVE-713
> Project: Hadoop Hive
> Issue Type: New Feature
> Components: Logging
> Reporter: Raghotham Murthy
> Assignee: Raghotham Murthy
> Attachments: hive-713.1.donot.patch, hive-713.2.donot.patch, hive-713.2.patch, hive-713.3.patch, hive-713.4.donot.patch, hive-713.5.donot.patch, hive-713.5.patch, hive-713.6.patch, hive-713.9.patch, hive.713.1.good.patch, hive.713.7.patch, hive.713.8.patch, hive.713.fixdiffs, hive.713.patch.bad, newpatch
>
>
> Annotate the QueryPlan - with multiple stages and each stage(Task) can have an Operator tree - with counters that each Operator/Task updates.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-713) Integrate operator counters into query
plan to view progress of queries
Posted by "Edward Capriolo (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12740182#action_12740182 ]
Edward Capriolo commented on HIVE-713:
--------------------------------------
I am looking to get some hive-jmx integeration (HIVE-551). One problem I ran into was dealing with true distributed operation. I am wondered where the overlap in what we are doing is and or how you are tackling this.
1. For hive 551 I am thinking each map/reduce can open a separate JMX port. This is a problem because first multiple mappers might run on the same machine. Secondly, I think i would have to write the location jmx ports to /tmp space so someone can read the file and figure out what to attach to.
My thinking use JMX proxy. We know that each system is running a task-tracker. From a mapper/reduce we can proxy the task tracker and add MX beans named with query-id. I wonder can we get counters/jmx working together somehow?
> Integrate operator counters into query plan to view progress of queries
> -----------------------------------------------------------------------
>
> Key: HIVE-713
> URL: https://issues.apache.org/jira/browse/HIVE-713
> Project: Hadoop Hive
> Issue Type: New Feature
> Components: Logging
> Reporter: Raghotham Murthy
> Assignee: Raghotham Murthy
> Attachments: hive-713.1.donot.patch, hive-713.2.donot.patch, hive-713.2.patch, hive-713.3.patch, hive-713.4.donot.patch, hive-713.5.donot.patch, hive.713.1.good.patch, hive.713.patch.bad
>
>
> Annotate the QueryPlan - with multiple stages and each stage(Task) can have an Operator tree - with counters that each Operator/Task updates.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-713) Integrate operator counters into query
plan to view progress of queries
Posted by "Zheng Shao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Zheng Shao updated HIVE-713:
----------------------------
Resolution: Fixed
Fix Version/s: 0.5.0
Release Note: HIVE-713. Integrate operator counters into query plan to view progress of queries. (Namit Jain and Raghu Murthy via zshao)
Hadoop Flags: [Reviewed]
Status: Resolved (was: Patch Available)
Committed. Thanks Raghu and Namit!
Let's get the follow-up JIRAs done soon.
> Integrate operator counters into query plan to view progress of queries
> -----------------------------------------------------------------------
>
> Key: HIVE-713
> URL: https://issues.apache.org/jira/browse/HIVE-713
> Project: Hadoop Hive
> Issue Type: New Feature
> Components: Logging
> Reporter: Raghotham Murthy
> Assignee: Raghotham Murthy
> Fix For: 0.5.0
>
> Attachments: hive-713.1.donot.patch, hive-713.2.donot.patch, hive-713.2.patch, hive-713.3.patch, hive-713.4.donot.patch, hive-713.5.donot.patch, hive-713.5.patch, hive-713.6.patch, hive-713.9.patch, hive.713.1.good.patch, hive.713.7.patch, hive.713.8.patch, hive.713.fixdiffs, hive.713.patch.bad, newpatch
>
>
> Annotate the QueryPlan - with multiple stages and each stage(Task) can have an Operator tree - with counters that each Operator/Task updates.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-713) Integrate operator counters into query
plan to view progress of queries
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12744808#action_12744808 ]
Namit Jain commented on HIVE-713:
---------------------------------
https://issues.apache.org/jira/browse/HIVE-770
> Integrate operator counters into query plan to view progress of queries
> -----------------------------------------------------------------------
>
> Key: HIVE-713
> URL: https://issues.apache.org/jira/browse/HIVE-713
> Project: Hadoop Hive
> Issue Type: New Feature
> Components: Logging
> Reporter: Raghotham Murthy
> Assignee: Raghotham Murthy
> Attachments: hive-713.1.donot.patch, hive-713.2.donot.patch, hive-713.2.patch, hive-713.3.patch, hive-713.4.donot.patch, hive-713.5.donot.patch, hive-713.5.patch, hive-713.6.patch, hive.713.1.good.patch, hive.713.7.patch, hive.713.fixdiffs, hive.713.patch.bad, newpatch
>
>
> Annotate the QueryPlan - with multiple stages and each stage(Task) can have an Operator tree - with counters that each Operator/Task updates.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-713) Integrate operator counters into query
plan to view progress of queries
Posted by "Raghotham Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Raghotham Murthy updated HIVE-713:
----------------------------------
Attachment: hive-713.6.patch
Fixed stagegraph for ConditionalTask. Added 'started' flag to query, stage, tasks and operators.
> Integrate operator counters into query plan to view progress of queries
> -----------------------------------------------------------------------
>
> Key: HIVE-713
> URL: https://issues.apache.org/jira/browse/HIVE-713
> Project: Hadoop Hive
> Issue Type: New Feature
> Components: Logging
> Reporter: Raghotham Murthy
> Assignee: Raghotham Murthy
> Attachments: hive-713.1.donot.patch, hive-713.2.donot.patch, hive-713.2.patch, hive-713.3.patch, hive-713.4.donot.patch, hive-713.5.donot.patch, hive-713.5.patch, hive-713.6.patch, hive.713.1.good.patch, hive.713.fixdiffs, hive.713.patch.bad
>
>
> Annotate the QueryPlan - with multiple stages and each stage(Task) can have an Operator tree - with counters that each Operator/Task updates.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-713) Integrate operator counters into query
plan to view progress of queries
Posted by "Zheng Shao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12744905#action_12744905 ]
Zheng Shao commented on HIVE-713:
---------------------------------
I think https://issues.apache.org/jira/browse/HIVE-768 should go together with this patch, because otherwise it's going to be a performance hit.
It should be a very simple change as well.
I am OK with all other follow-ups being done after this one is committed.
> Integrate operator counters into query plan to view progress of queries
> -----------------------------------------------------------------------
>
> Key: HIVE-713
> URL: https://issues.apache.org/jira/browse/HIVE-713
> Project: Hadoop Hive
> Issue Type: New Feature
> Components: Logging
> Reporter: Raghotham Murthy
> Assignee: Raghotham Murthy
> Attachments: hive-713.1.donot.patch, hive-713.2.donot.patch, hive-713.2.patch, hive-713.3.patch, hive-713.4.donot.patch, hive-713.5.donot.patch, hive-713.5.patch, hive-713.6.patch, hive.713.1.good.patch, hive.713.7.patch, hive.713.8.patch, hive.713.fixdiffs, hive.713.patch.bad, newpatch
>
>
> Annotate the QueryPlan - with multiple stages and each stage(Task) can have an Operator tree - with counters that each Operator/Task updates.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-713) Integrate operator counters into query
plan to view progress of queries
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12744820#action_12744820 ]
Namit Jain commented on HIVE-713:
---------------------------------
https://issues.apache.org/jira/browse/HIVE-771
> Integrate operator counters into query plan to view progress of queries
> -----------------------------------------------------------------------
>
> Key: HIVE-713
> URL: https://issues.apache.org/jira/browse/HIVE-713
> Project: Hadoop Hive
> Issue Type: New Feature
> Components: Logging
> Reporter: Raghotham Murthy
> Assignee: Raghotham Murthy
> Attachments: hive-713.1.donot.patch, hive-713.2.donot.patch, hive-713.2.patch, hive-713.3.patch, hive-713.4.donot.patch, hive-713.5.donot.patch, hive-713.5.patch, hive-713.6.patch, hive.713.1.good.patch, hive.713.7.patch, hive.713.fixdiffs, hive.713.patch.bad, newpatch
>
>
> Annotate the QueryPlan - with multiple stages and each stage(Task) can have an Operator tree - with counters that each Operator/Task updates.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-713) Integrate operator counters into query
plan to view progress of queries
Posted by "Edward Capriolo (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12740207#action_12740207 ]
Edward Capriolo commented on HIVE-713:
--------------------------------------
True. If we were able to implement JMX we could attach to a mapper/reduce and profile it directly. IE real time how much memory a mapper is using, there are a bunch of standard jmx objects that could be useful. Possibly a logging based approach might be a better way to go. Another technique might be a job completion URL. counters+job completion would be a nice combination.
JMX could be a tricky implementation due to the multi-jvm nature of the hadoop. But if the framework is managing counters for us getting them into JMX should be doable. I will review your code more.
> Integrate operator counters into query plan to view progress of queries
> -----------------------------------------------------------------------
>
> Key: HIVE-713
> URL: https://issues.apache.org/jira/browse/HIVE-713
> Project: Hadoop Hive
> Issue Type: New Feature
> Components: Logging
> Reporter: Raghotham Murthy
> Assignee: Raghotham Murthy
> Attachments: hive-713.1.donot.patch, hive-713.2.donot.patch, hive-713.2.patch, hive-713.3.patch, hive-713.4.donot.patch, hive-713.5.donot.patch, hive.713.1.good.patch, hive.713.patch.bad
>
>
> Annotate the QueryPlan - with multiple stages and each stage(Task) can have an Operator tree - with counters that each Operator/Task updates.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-713) Integrate operator counters into query
plan to view progress of queries
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12737637#action_12737637 ]
Namit Jain commented on HIVE-713:
---------------------------------
One of the follow-ups that needs to be done is that the progress needs to be done by Driver instead of ExecDriver() -
Driver should do the following
task.execute()
task.progress()
task.postProcessing()
This way each task does not have to invoke progress within execute.
> Integrate operator counters into query plan to view progress of queries
> -----------------------------------------------------------------------
>
> Key: HIVE-713
> URL: https://issues.apache.org/jira/browse/HIVE-713
> Project: Hadoop Hive
> Issue Type: New Feature
> Components: Logging
> Reporter: Raghotham Murthy
> Assignee: Raghotham Murthy
> Attachments: hive-713.1.donot.patch
>
>
> Annotate the QueryPlan - with multiple stages and each stage(Task) can have an Operator tree - with counters that each Operator/Task updates.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-713) Integrate operator counters into query
plan to view progress of queries
Posted by "Raghotham Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Raghotham Murthy updated HIVE-713:
----------------------------------
Attachment: hive-713.1.donot.patch
This patch contains most of the changes to the APIs in the ql package. It is only for review. The following need to be done:
* Add the following default counters to Operator: #input rows, # output rows, cputime
* In QueryPlan, fetch the counters from the active tasks and store them locally to be queried later
* Add progress method to MoveTask(?) and CopyTask
* Dump stats to history file in all progress methods
* Dump the query plan so that it can be viewed using tools like graphviz
* Change CliDriver or create a new class which can reconstruct the plan annotated with counters from logs in history file
* Provide a call in HiveServer to retrieve the annotated query plan
> Integrate operator counters into query plan to view progress of queries
> -----------------------------------------------------------------------
>
> Key: HIVE-713
> URL: https://issues.apache.org/jira/browse/HIVE-713
> Project: Hadoop Hive
> Issue Type: New Feature
> Components: Logging
> Reporter: Raghotham Murthy
> Assignee: Raghotham Murthy
> Attachments: hive-713.1.donot.patch
>
>
> Annotate the QueryPlan - with multiple stages and each stage(Task) can have an Operator tree - with counters that each Operator/Task updates.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-713) Integrate operator counters into query
plan to view progress of queries
Posted by "Raghotham Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Raghotham Murthy updated HIVE-713:
----------------------------------
Attachment: hive-713.5.donot.patch
added support for conditional task by allowing disjunctive edges in the graphs.
> Integrate operator counters into query plan to view progress of queries
> -----------------------------------------------------------------------
>
> Key: HIVE-713
> URL: https://issues.apache.org/jira/browse/HIVE-713
> Project: Hadoop Hive
> Issue Type: New Feature
> Components: Logging
> Reporter: Raghotham Murthy
> Assignee: Raghotham Murthy
> Attachments: hive-713.1.donot.patch, hive-713.2.donot.patch, hive-713.4.donot.patch, hive-713.5.donot.patch, hive.713.patch.bad
>
>
> Annotate the QueryPlan - with multiple stages and each stage(Task) can have an Operator tree - with counters that each Operator/Task updates.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-713) Integrate operator counters into query
plan to view progress of queries
Posted by "Raghotham Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Raghotham Murthy updated HIVE-713:
----------------------------------
Attachment: hive-713.2.patch
Added the following:
1. construct thrift query plan from hive structures
2. populate counters from hive structures into query plan - currently the counters include #input rows, #output rows and time taken
3. dump out the entire thrift query plan in progress in binary format (urlencoded)
4. sample php script ql/src/test/php/test_queryplan.php which should decode the urlencoded binary formatted query plan (its not working properly - i am still debugging it)
5. refreshed php and py thrift libraries
> Integrate operator counters into query plan to view progress of queries
> -----------------------------------------------------------------------
>
> Key: HIVE-713
> URL: https://issues.apache.org/jira/browse/HIVE-713
> Project: Hadoop Hive
> Issue Type: New Feature
> Components: Logging
> Reporter: Raghotham Murthy
> Assignee: Raghotham Murthy
> Attachments: hive-713.1.donot.patch, hive-713.2.donot.patch, hive-713.2.patch, hive-713.4.donot.patch, hive-713.5.donot.patch, hive.713.1.good.patch, hive.713.patch.bad
>
>
> Annotate the QueryPlan - with multiple stages and each stage(Task) can have an Operator tree - with counters that each Operator/Task updates.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-713) Integrate operator counters into query
plan to view progress of queries
Posted by "Raghotham Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Raghotham Murthy updated HIVE-713:
----------------------------------
Attachment: hive-713.2.patch
removed a couple of typos in php script (still not working though).
> Integrate operator counters into query plan to view progress of queries
> -----------------------------------------------------------------------
>
> Key: HIVE-713
> URL: https://issues.apache.org/jira/browse/HIVE-713
> Project: Hadoop Hive
> Issue Type: New Feature
> Components: Logging
> Reporter: Raghotham Murthy
> Assignee: Raghotham Murthy
> Attachments: hive-713.1.donot.patch, hive-713.2.donot.patch, hive-713.2.patch, hive-713.4.donot.patch, hive-713.5.donot.patch, hive.713.1.good.patch, hive.713.patch.bad
>
>
> Annotate the QueryPlan - with multiple stages and each stage(Task) can have an Operator tree - with counters that each Operator/Task updates.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.