You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by "Jeremy A. Lucas (JIRA)" <ji...@apache.org> on 2012/10/12 23:31:04 UTC
[jira] [Created] (HIVE-3574) Allow Hive to Submit MapReduce jobs
via the MapReduce API (instead of using Hadoop BIN)
Jeremy A. Lucas created HIVE-3574:
-------------------------------------
Summary: Allow Hive to Submit MapReduce jobs via the MapReduce API (instead of using Hadoop BIN)
Key: HIVE-3574
URL: https://issues.apache.org/jira/browse/HIVE-3574
Project: Hive
Issue Type: Improvement
Components: Query Processor, SQL
Affects Versions: 0.9.0, 0.8.1, 0.8.0, 0.7.1, 0.7.0, 0.6.0, 0.5.0, 0.4.1, 0.4.0, 0.3.0, 0.10.0, 0.9.1
Environment: All environments would be affected by this
Reporter: Jeremy A. Lucas
Priority: Minor
Having Hive to submit generated jobs to an M/R cluster via the MapReduce API would allow for potentially greater compatibility across platforms, in addition to allowing for these jobs to be run easily against pseudo-clusters in tests (think MiniMRCluster).
This kind of change could involve something as simple as using a Hadoop Configuration object with a generic ToolRunner or something similar to run jobs.
Specifically, this kind of change would most likely occur in the execute() method of org.apache.hadoop.hive.ql.exec.MapRedTask (note that this class already has access to a JobConf object as well, which could serve in itself as a Configuration object).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3574) Allow Hive to Submit MapReduce jobs
via the MapReduce API (instead of using Hadoop BIN)
Posted by "Andrew Look (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-3574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13475437#comment-13475437 ]
Andrew Look commented on HIVE-3574:
-----------------------------------
+1. I find it to be a much more robust and flexible approach to rely on what's in the classpath rather than what's installed on the system.
> Allow Hive to Submit MapReduce jobs via the MapReduce API (instead of using Hadoop BIN)
> ---------------------------------------------------------------------------------------
>
> Key: HIVE-3574
> URL: https://issues.apache.org/jira/browse/HIVE-3574
> Project: Hive
> Issue Type: Improvement
> Components: Query Processor, SQL
> Affects Versions: 0.3.0, 0.4.0, 0.4.1, 0.5.0, 0.6.0, 0.7.0, 0.7.1, 0.8.0, 0.8.1, 0.9.0, 0.10.0, 0.9.1
> Environment: All environments would be affected by this
> Reporter: Jeremy A. Lucas
> Priority: Minor
> Labels: feature, test
>
> The current behavior of the MapRedTask is to start a process that invokes the "hadoop jar" command, passing each additional jobconf property as an argument to this Hadoop CLI.
> Having Hive to submit generated jobs to an M/R cluster via the MapReduce API would allow for potentially greater compatibility across platforms, in addition to allowing for these jobs to be run easily against pseudo-clusters in tests (think MiniMRCluster).
> This kind of change could involve something as simple as using a Hadoop Configuration object with a generic ToolRunner or something similar to run jobs.
> Specifically, this kind of change would most likely occur in the execute() method of org.apache.hadoop.hive.ql.exec.MapRedTask.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3574) Allow Hive to Submit MapReduce jobs
via the MapReduce API (instead of using Hadoop BIN)
Posted by "Bing Li (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-3574?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Bing Li updated HIVE-3574:
--------------------------
Assignee: Bing Li
> Allow Hive to Submit MapReduce jobs via the MapReduce API (instead of using Hadoop BIN)
> ---------------------------------------------------------------------------------------
>
> Key: HIVE-3574
> URL: https://issues.apache.org/jira/browse/HIVE-3574
> Project: Hive
> Issue Type: Improvement
> Components: Query Processor, SQL
> Affects Versions: 0.3.0, 0.4.0, 0.4.1, 0.5.0, 0.6.0, 0.7.0, 0.7.1, 0.8.0, 0.8.1, 0.9.0, 0.10.0, 0.9.1
> Environment: All environments would be affected by this
> Reporter: Jeremy A. Lucas
> Assignee: Bing Li
> Priority: Minor
> Labels: feature, test
>
> The current behavior of the MapRedTask is to start a process that invokes the "hadoop jar" command, passing each additional jobconf property as an argument to this Hadoop CLI.
> Having Hive to submit generated jobs to an M/R cluster via the MapReduce API would allow for potentially greater compatibility across platforms, in addition to allowing for these jobs to be run easily against pseudo-clusters in tests (think MiniMRCluster).
> This kind of change could involve something as simple as using a Hadoop Configuration object with a generic ToolRunner or something similar to run jobs.
> Specifically, this kind of change would most likely occur in the execute() method of org.apache.hadoop.hive.ql.exec.MapRedTask.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3574) Allow Hive to Submit MapReduce jobs
via the MapReduce API (instead of using Hadoop BIN)
Posted by "Jeremy A. Lucas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-3574?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jeremy A. Lucas updated HIVE-3574:
----------------------------------
Description:
The current behavior of the MapRedTask is to start a process that invokes the "hadoop jar" command, passing each additional jobconf property as an argument to this Hadoop CLI.
Having Hive to submit generated jobs to an M/R cluster via the MapReduce API would allow for potentially greater compatibility across platforms, in addition to allowing for these jobs to be run easily against pseudo-clusters in tests (think MiniMRCluster).
This kind of change could involve something as simple as using a Hadoop Configuration object with a generic ToolRunner or something similar to run jobs.
Specifically, this kind of change would most likely occur in the execute() method of org.apache.hadoop.hive.ql.exec.MapRedTask.
was:
Having Hive to submit generated jobs to an M/R cluster via the MapReduce API would allow for potentially greater compatibility across platforms, in addition to allowing for these jobs to be run easily against pseudo-clusters in tests (think MiniMRCluster).
This kind of change could involve something as simple as using a Hadoop Configuration object with a generic ToolRunner or something similar to run jobs.
Specifically, this kind of change would most likely occur in the execute() method of org.apache.hadoop.hive.ql.exec.MapRedTask.
> Allow Hive to Submit MapReduce jobs via the MapReduce API (instead of using Hadoop BIN)
> ---------------------------------------------------------------------------------------
>
> Key: HIVE-3574
> URL: https://issues.apache.org/jira/browse/HIVE-3574
> Project: Hive
> Issue Type: Improvement
> Components: Query Processor, SQL
> Affects Versions: 0.3.0, 0.4.0, 0.4.1, 0.5.0, 0.6.0, 0.7.0, 0.7.1, 0.8.0, 0.8.1, 0.9.0, 0.10.0, 0.9.1
> Environment: All environments would be affected by this
> Reporter: Jeremy A. Lucas
> Priority: Minor
> Labels: feature, test
>
> The current behavior of the MapRedTask is to start a process that invokes the "hadoop jar" command, passing each additional jobconf property as an argument to this Hadoop CLI.
> Having Hive to submit generated jobs to an M/R cluster via the MapReduce API would allow for potentially greater compatibility across platforms, in addition to allowing for these jobs to be run easily against pseudo-clusters in tests (think MiniMRCluster).
> This kind of change could involve something as simple as using a Hadoop Configuration object with a generic ToolRunner or something similar to run jobs.
> Specifically, this kind of change would most likely occur in the execute() method of org.apache.hadoop.hive.ql.exec.MapRedTask.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3574) Allow Hive to Submit MapReduce jobs
via the MapReduce API (instead of using Hadoop BIN)
Posted by "Jeremy A. Lucas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-3574?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jeremy A. Lucas updated HIVE-3574:
----------------------------------
Description:
Having Hive to submit generated jobs to an M/R cluster via the MapReduce API would allow for potentially greater compatibility across platforms, in addition to allowing for these jobs to be run easily against pseudo-clusters in tests (think MiniMRCluster).
This kind of change could involve something as simple as using a Hadoop Configuration object with a generic ToolRunner or something similar to run jobs.
Specifically, this kind of change would most likely occur in the execute() method of org.apache.hadoop.hive.ql.exec.MapRedTask.
was:
Having Hive to submit generated jobs to an M/R cluster via the MapReduce API would allow for potentially greater compatibility across platforms, in addition to allowing for these jobs to be run easily against pseudo-clusters in tests (think MiniMRCluster).
This kind of change could involve something as simple as using a Hadoop Configuration object with a generic ToolRunner or something similar to run jobs.
Specifically, this kind of change would most likely occur in the execute() method of org.apache.hadoop.hive.ql.exec.MapRedTask (note that this class already has access to a JobConf object as well, which could serve in itself as a Configuration object).
> Allow Hive to Submit MapReduce jobs via the MapReduce API (instead of using Hadoop BIN)
> ---------------------------------------------------------------------------------------
>
> Key: HIVE-3574
> URL: https://issues.apache.org/jira/browse/HIVE-3574
> Project: Hive
> Issue Type: Improvement
> Components: Query Processor, SQL
> Affects Versions: 0.3.0, 0.4.0, 0.4.1, 0.5.0, 0.6.0, 0.7.0, 0.7.1, 0.8.0, 0.8.1, 0.9.0, 0.10.0, 0.9.1
> Environment: All environments would be affected by this
> Reporter: Jeremy A. Lucas
> Priority: Minor
> Labels: feature, test
>
> Having Hive to submit generated jobs to an M/R cluster via the MapReduce API would allow for potentially greater compatibility across platforms, in addition to allowing for these jobs to be run easily against pseudo-clusters in tests (think MiniMRCluster).
> This kind of change could involve something as simple as using a Hadoop Configuration object with a generic ToolRunner or something similar to run jobs.
> Specifically, this kind of change would most likely occur in the execute() method of org.apache.hadoop.hive.ql.exec.MapRedTask.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3574) Allow Hive to Submit MapReduce jobs
via the MapReduce API (instead of using Hadoop BIN)
Posted by "Ashutosh Chauhan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-3574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13475405#comment-13475405 ]
Ashutosh Chauhan commented on HIVE-3574:
----------------------------------------
+1 on the idea. I have also been hit by this. Reliance on hadoop script to launch MR jobs is not cool.
> Allow Hive to Submit MapReduce jobs via the MapReduce API (instead of using Hadoop BIN)
> ---------------------------------------------------------------------------------------
>
> Key: HIVE-3574
> URL: https://issues.apache.org/jira/browse/HIVE-3574
> Project: Hive
> Issue Type: Improvement
> Components: Query Processor, SQL
> Affects Versions: 0.3.0, 0.4.0, 0.4.1, 0.5.0, 0.6.0, 0.7.0, 0.7.1, 0.8.0, 0.8.1, 0.9.0, 0.10.0, 0.9.1
> Environment: All environments would be affected by this
> Reporter: Jeremy A. Lucas
> Priority: Minor
> Labels: feature, test
>
> The current behavior of the MapRedTask is to start a process that invokes the "hadoop jar" command, passing each additional jobconf property as an argument to this Hadoop CLI.
> Having Hive to submit generated jobs to an M/R cluster via the MapReduce API would allow for potentially greater compatibility across platforms, in addition to allowing for these jobs to be run easily against pseudo-clusters in tests (think MiniMRCluster).
> This kind of change could involve something as simple as using a Hadoop Configuration object with a generic ToolRunner or something similar to run jobs.
> Specifically, this kind of change would most likely occur in the execute() method of org.apache.hadoop.hive.ql.exec.MapRedTask.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira