You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by Mohit Sabharwal <mo...@cloudera.com> on 2014/10/23 08:31:07 UTC
Review Request 26988: HIVE-8568 : Add HS2 API to fetch Job IDs for a
given query
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/26988/
-----------------------------------------------------------
Review request for hive.
Bugs: HIVE-8568
https://issues.apache.org/jira/browse/HIVE-8568
Repository: hive-git
Description
-------
HIVE-8568 : Add HS2 API to fetch Job IDs for a given query
This patch adds a Thrift call to fetch Job IDs corresponding to all
running MR tasks. The response includes a list of job IDs and an
enum indicating the execution engine. Throws an exception if no
jobIDs were fetched.
Currently only returns MR job IDs. Support for Tez job IDs is left
as a todo item for a future commit.
The client may need to make the call multiple times since the jobs
may not have started running. Exposed RunningJob associated with
ExecDriver as a public method, so that the Driver can access the
jobIDs corresponding to all running tasks.
Diffs
-----
itests/hive-unit-hadoop2/src/test/java/org/apache/hive/TestThriftGetJobIDs.java PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/Driver.java e25450531a71ef4ae4c6d9ea1788e618189a17cb
ql/src/java/org/apache/hadoop/hive/ql/DriverContext.java c7d3b6652f89cf7b6507f35176962ff3287d112d
ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecDriver.java 4e3df75c614fe7232e670201f2560c7ccd1db41c
ql/src/java/org/apache/hadoop/hive/ql/thrift/JobIDSet.java PRE-CREATION
service/if/TCLIService.thrift 4024bb3f412440fb7533f2e2d8ebc9a7cdc0776d
service/src/gen/thrift/gen-cpp/TCLIService.h 030475b25188c5d2494da4de0bd6edc1ae807eca
service/src/gen/thrift/gen-cpp/TCLIService.cpp 209ce63ae1ffd593de81e8e0a8e73218afe3cd79
service/src/gen/thrift/gen-cpp/TCLIService_server.skeleton.cpp 988bb4c11ddb717f585e0ba2fb4773ec5fff77e6
service/src/gen/thrift/gen-cpp/TCLIService_types.h f32dc3c90caedba86d943a9295a2f246a7b0ec90
service/src/gen/thrift/gen-cpp/TCLIService_types.cpp 326d25b8b7d814f7bbdfab7dde805be4834493dc
service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TCLIService.java 54851b8d513179e3618ee5a974941bb6a72378b6
service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TExecutionEngine.java PRE-CREATION
service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TGetJobIDsReq.java PRE-CREATION
service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TGetJobIDsResp.java PRE-CREATION
service/src/gen/thrift/gen-php/TCLIService.php d2462967c4ee40c46bdeb6c8e24e22e63f3567e3
service/src/gen/thrift/gen-py/TCLIService/TCLIService-remote f6ff43f021524adb1a179595bfdd9260e98bff28
service/src/gen/thrift/gen-py/TCLIService/TCLIService.py ebc65746ccd7e58c0878f426b429274d4b59ed0b
service/src/gen/thrift/gen-py/TCLIService/ttypes.py 6cd64d0386f1e7c73eee7d9868c387c4942a5f9f
service/src/gen/thrift/gen-rb/t_c_l_i_service.rb fd1ca9aa13f3db170caf310ebb1ee1bac9f70b63
service/src/gen/thrift/gen-rb/t_c_l_i_service_types.rb c731544888f7480b0b1af70440ce1697e8597c12
service/src/java/org/apache/hive/service/cli/CLIService.java f5751f1305d7dd4c1f74af5a3a4f94f018b7a38f
service/src/java/org/apache/hive/service/cli/CLIServiceClient.java 3155c238ff688bfea16b0aaeea950599bb659b5b
service/src/java/org/apache/hive/service/cli/EmbeddedCLIServiceClient.java 9cad5be198c063115a8e90c67b1c2fd910ca8bc6
service/src/java/org/apache/hive/service/cli/ICLIService.java c9cc1f4da56f1cd10f6348ea2b9e17e203b87664
service/src/java/org/apache/hive/service/cli/operation/Operation.java acb95cb015395f4a1a9280c3b0c719228e584df7
service/src/java/org/apache/hive/service/cli/operation/OperationManager.java a57b6e5d322ac312636c19633cee44f711b653df
service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java 8cabf7ee2945296774d31925a2bce46a7320d668
service/src/java/org/apache/hive/service/cli/session/HiveSession.java 6359a5b879928e8726017520f9a733d6b11decd4
service/src/java/org/apache/hive/service/cli/session/HiveSessionImpl.java fa28a6b6a4acb61d8b442ed13b0421e1e0f13368
service/src/java/org/apache/hive/service/cli/thrift/ThriftCLIService.java a0a6e183bbd05cd61ba97f187d66b286c145969c
service/src/java/org/apache/hive/service/cli/thrift/ThriftCLIServiceClient.java 1af45398b895cd7616c5627d318422e14b81e734
service/src/test/org/apache/hive/service/cli/thrift/ThriftCLIServiceTest.java 630cfc9124abf7a8871b613b967141d0447eb18e
Diff: https://reviews.apache.org/r/26988/diff/
Testing
-------
Added unit test that issues async execute statements and follows
it up with getJobID calls.
Did not add to ThriftCLIServiceTest since test needs a miniMR cluster
to run jobs that generate jobIDs.
Thanks,
Mohit Sabharwal
Re: Review Request 26988: HIVE-8568 : Add HS2 API to fetch Job IDs
for a given query
Posted by Brock Noland <br...@cloudera.com>.
> On Oct. 26, 2014, 12:32 a.m., Brock Noland wrote:
> > Nice work! This looks great! The only thing I think we need to change is the execution engine level. It's possible for a single HS2 to have both MR and Tez (and Spark) tasks running. For example, in seperate sessions I can set the execution engine and run some queries in MR and some in tez. Thus I think each job id needs a paired execution engine.
Actually I don't think this is true since we are getting the job ids for a specific session not for all sessions....
- Brock
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/26988/#review58542
-----------------------------------------------------------
On Oct. 23, 2014, 6:31 a.m., Mohit Sabharwal wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/26988/
> -----------------------------------------------------------
>
> (Updated Oct. 23, 2014, 6:31 a.m.)
>
>
> Review request for hive.
>
>
> Bugs: HIVE-8568
> https://issues.apache.org/jira/browse/HIVE-8568
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> HIVE-8568 : Add HS2 API to fetch Job IDs for a given query
>
> This patch adds a Thrift call to fetch Job IDs corresponding to all
> running MR tasks. The response includes a list of job IDs and an
> enum indicating the execution engine. Throws an exception if no
> jobIDs were fetched.
>
> Currently only returns MR job IDs. Support for Tez job IDs is left
> as a todo item for a future commit.
>
> The client may need to make the call multiple times since the jobs
> may not have started running. Exposed RunningJob associated with
> ExecDriver as a public method, so that the Driver can access the
> jobIDs corresponding to all running tasks.
>
>
> Diffs
> -----
>
> itests/hive-unit-hadoop2/src/test/java/org/apache/hive/TestThriftGetJobIDs.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/Driver.java e25450531a71ef4ae4c6d9ea1788e618189a17cb
> ql/src/java/org/apache/hadoop/hive/ql/DriverContext.java c7d3b6652f89cf7b6507f35176962ff3287d112d
> ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecDriver.java 4e3df75c614fe7232e670201f2560c7ccd1db41c
> ql/src/java/org/apache/hadoop/hive/ql/thrift/JobIDSet.java PRE-CREATION
> service/if/TCLIService.thrift 4024bb3f412440fb7533f2e2d8ebc9a7cdc0776d
> service/src/gen/thrift/gen-cpp/TCLIService.h 030475b25188c5d2494da4de0bd6edc1ae807eca
> service/src/gen/thrift/gen-cpp/TCLIService.cpp 209ce63ae1ffd593de81e8e0a8e73218afe3cd79
> service/src/gen/thrift/gen-cpp/TCLIService_server.skeleton.cpp 988bb4c11ddb717f585e0ba2fb4773ec5fff77e6
> service/src/gen/thrift/gen-cpp/TCLIService_types.h f32dc3c90caedba86d943a9295a2f246a7b0ec90
> service/src/gen/thrift/gen-cpp/TCLIService_types.cpp 326d25b8b7d814f7bbdfab7dde805be4834493dc
> service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TCLIService.java 54851b8d513179e3618ee5a974941bb6a72378b6
> service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TExecutionEngine.java PRE-CREATION
> service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TGetJobIDsReq.java PRE-CREATION
> service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TGetJobIDsResp.java PRE-CREATION
> service/src/gen/thrift/gen-php/TCLIService.php d2462967c4ee40c46bdeb6c8e24e22e63f3567e3
> service/src/gen/thrift/gen-py/TCLIService/TCLIService-remote f6ff43f021524adb1a179595bfdd9260e98bff28
> service/src/gen/thrift/gen-py/TCLIService/TCLIService.py ebc65746ccd7e58c0878f426b429274d4b59ed0b
> service/src/gen/thrift/gen-py/TCLIService/ttypes.py 6cd64d0386f1e7c73eee7d9868c387c4942a5f9f
> service/src/gen/thrift/gen-rb/t_c_l_i_service.rb fd1ca9aa13f3db170caf310ebb1ee1bac9f70b63
> service/src/gen/thrift/gen-rb/t_c_l_i_service_types.rb c731544888f7480b0b1af70440ce1697e8597c12
> service/src/java/org/apache/hive/service/cli/CLIService.java f5751f1305d7dd4c1f74af5a3a4f94f018b7a38f
> service/src/java/org/apache/hive/service/cli/CLIServiceClient.java 3155c238ff688bfea16b0aaeea950599bb659b5b
> service/src/java/org/apache/hive/service/cli/EmbeddedCLIServiceClient.java 9cad5be198c063115a8e90c67b1c2fd910ca8bc6
> service/src/java/org/apache/hive/service/cli/ICLIService.java c9cc1f4da56f1cd10f6348ea2b9e17e203b87664
> service/src/java/org/apache/hive/service/cli/operation/Operation.java acb95cb015395f4a1a9280c3b0c719228e584df7
> service/src/java/org/apache/hive/service/cli/operation/OperationManager.java a57b6e5d322ac312636c19633cee44f711b653df
> service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java 8cabf7ee2945296774d31925a2bce46a7320d668
> service/src/java/org/apache/hive/service/cli/session/HiveSession.java 6359a5b879928e8726017520f9a733d6b11decd4
> service/src/java/org/apache/hive/service/cli/session/HiveSessionImpl.java fa28a6b6a4acb61d8b442ed13b0421e1e0f13368
> service/src/java/org/apache/hive/service/cli/thrift/ThriftCLIService.java a0a6e183bbd05cd61ba97f187d66b286c145969c
> service/src/java/org/apache/hive/service/cli/thrift/ThriftCLIServiceClient.java 1af45398b895cd7616c5627d318422e14b81e734
> service/src/test/org/apache/hive/service/cli/thrift/ThriftCLIServiceTest.java 630cfc9124abf7a8871b613b967141d0447eb18e
>
> Diff: https://reviews.apache.org/r/26988/diff/
>
>
> Testing
> -------
>
> Added unit test that issues async execute statements and follows
> it up with getJobID calls.
>
> Did not add to ThriftCLIServiceTest since test needs a miniMR cluster
> to run jobs that generate jobIDs.
>
>
> Thanks,
>
> Mohit Sabharwal
>
>
Re: Review Request 26988: HIVE-8568 : Add HS2 API to fetch Job IDs
for a given query
Posted by Brock Noland <br...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/26988/#review58542
-----------------------------------------------------------
Nice work! This looks great! The only thing I think we need to change is the execution engine level. It's possible for a single HS2 to have both MR and Tez (and Spark) tasks running. For example, in seperate sessions I can set the execution engine and run some queries in MR and some in tez. Thus I think each job id needs a paired execution engine.
- Brock Noland
On Oct. 23, 2014, 6:31 a.m., Mohit Sabharwal wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/26988/
> -----------------------------------------------------------
>
> (Updated Oct. 23, 2014, 6:31 a.m.)
>
>
> Review request for hive.
>
>
> Bugs: HIVE-8568
> https://issues.apache.org/jira/browse/HIVE-8568
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> HIVE-8568 : Add HS2 API to fetch Job IDs for a given query
>
> This patch adds a Thrift call to fetch Job IDs corresponding to all
> running MR tasks. The response includes a list of job IDs and an
> enum indicating the execution engine. Throws an exception if no
> jobIDs were fetched.
>
> Currently only returns MR job IDs. Support for Tez job IDs is left
> as a todo item for a future commit.
>
> The client may need to make the call multiple times since the jobs
> may not have started running. Exposed RunningJob associated with
> ExecDriver as a public method, so that the Driver can access the
> jobIDs corresponding to all running tasks.
>
>
> Diffs
> -----
>
> itests/hive-unit-hadoop2/src/test/java/org/apache/hive/TestThriftGetJobIDs.java PRE-CREATION
> ql/src/java/org/apache/hadoop/hive/ql/Driver.java e25450531a71ef4ae4c6d9ea1788e618189a17cb
> ql/src/java/org/apache/hadoop/hive/ql/DriverContext.java c7d3b6652f89cf7b6507f35176962ff3287d112d
> ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecDriver.java 4e3df75c614fe7232e670201f2560c7ccd1db41c
> ql/src/java/org/apache/hadoop/hive/ql/thrift/JobIDSet.java PRE-CREATION
> service/if/TCLIService.thrift 4024bb3f412440fb7533f2e2d8ebc9a7cdc0776d
> service/src/gen/thrift/gen-cpp/TCLIService.h 030475b25188c5d2494da4de0bd6edc1ae807eca
> service/src/gen/thrift/gen-cpp/TCLIService.cpp 209ce63ae1ffd593de81e8e0a8e73218afe3cd79
> service/src/gen/thrift/gen-cpp/TCLIService_server.skeleton.cpp 988bb4c11ddb717f585e0ba2fb4773ec5fff77e6
> service/src/gen/thrift/gen-cpp/TCLIService_types.h f32dc3c90caedba86d943a9295a2f246a7b0ec90
> service/src/gen/thrift/gen-cpp/TCLIService_types.cpp 326d25b8b7d814f7bbdfab7dde805be4834493dc
> service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TCLIService.java 54851b8d513179e3618ee5a974941bb6a72378b6
> service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TExecutionEngine.java PRE-CREATION
> service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TGetJobIDsReq.java PRE-CREATION
> service/src/gen/thrift/gen-javabean/org/apache/hive/service/cli/thrift/TGetJobIDsResp.java PRE-CREATION
> service/src/gen/thrift/gen-php/TCLIService.php d2462967c4ee40c46bdeb6c8e24e22e63f3567e3
> service/src/gen/thrift/gen-py/TCLIService/TCLIService-remote f6ff43f021524adb1a179595bfdd9260e98bff28
> service/src/gen/thrift/gen-py/TCLIService/TCLIService.py ebc65746ccd7e58c0878f426b429274d4b59ed0b
> service/src/gen/thrift/gen-py/TCLIService/ttypes.py 6cd64d0386f1e7c73eee7d9868c387c4942a5f9f
> service/src/gen/thrift/gen-rb/t_c_l_i_service.rb fd1ca9aa13f3db170caf310ebb1ee1bac9f70b63
> service/src/gen/thrift/gen-rb/t_c_l_i_service_types.rb c731544888f7480b0b1af70440ce1697e8597c12
> service/src/java/org/apache/hive/service/cli/CLIService.java f5751f1305d7dd4c1f74af5a3a4f94f018b7a38f
> service/src/java/org/apache/hive/service/cli/CLIServiceClient.java 3155c238ff688bfea16b0aaeea950599bb659b5b
> service/src/java/org/apache/hive/service/cli/EmbeddedCLIServiceClient.java 9cad5be198c063115a8e90c67b1c2fd910ca8bc6
> service/src/java/org/apache/hive/service/cli/ICLIService.java c9cc1f4da56f1cd10f6348ea2b9e17e203b87664
> service/src/java/org/apache/hive/service/cli/operation/Operation.java acb95cb015395f4a1a9280c3b0c719228e584df7
> service/src/java/org/apache/hive/service/cli/operation/OperationManager.java a57b6e5d322ac312636c19633cee44f711b653df
> service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java 8cabf7ee2945296774d31925a2bce46a7320d668
> service/src/java/org/apache/hive/service/cli/session/HiveSession.java 6359a5b879928e8726017520f9a733d6b11decd4
> service/src/java/org/apache/hive/service/cli/session/HiveSessionImpl.java fa28a6b6a4acb61d8b442ed13b0421e1e0f13368
> service/src/java/org/apache/hive/service/cli/thrift/ThriftCLIService.java a0a6e183bbd05cd61ba97f187d66b286c145969c
> service/src/java/org/apache/hive/service/cli/thrift/ThriftCLIServiceClient.java 1af45398b895cd7616c5627d318422e14b81e734
> service/src/test/org/apache/hive/service/cli/thrift/ThriftCLIServiceTest.java 630cfc9124abf7a8871b613b967141d0447eb18e
>
> Diff: https://reviews.apache.org/r/26988/diff/
>
>
> Testing
> -------
>
> Added unit test that issues async execute statements and follows
> it up with getJobID calls.
>
> Did not add to ThriftCLIServiceTest since test needs a miniMR cluster
> to run jobs that generate jobIDs.
>
>
> Thanks,
>
> Mohit Sabharwal
>
>