You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by chengxiang li <ch...@intel.com> on 2014/12/17 07:28:13 UTC
Review Request 29145: HIVE-9094 TimeoutException when trying get
executor count from RSC [Spark Branch]
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/29145/
-----------------------------------------------------------
Review request for hive and Xuefu Zhang.
Bugs: HIVE-9094
https://issues.apache.org/jira/browse/HIVE-9094
Repository: hive-git
Description
-------
RemoteHiveSparkClient::getExecutorCount timeout after 5s as Spark cluster has not launched yet
1. set the timeout value configurable.
2. set default timeout value 60s.
3. enable timeout for get spark job info and get spark stage info.
Diffs
-----
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 22f052a
ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HiveSparkClientFactory.java 5d6a02c
ql/src/java/org/apache/hadoop/hive/ql/exec/spark/RemoteHiveSparkClient.java e1946d5
ql/src/java/org/apache/hadoop/hive/ql/exec/spark/status/impl/RemoteSparkJobStatus.java 6217de4
Diff: https://reviews.apache.org/r/29145/diff/
Testing
-------
Thanks,
chengxiang li
Re: Review Request 29145: HIVE-9094 TimeoutException when trying get
executor count from RSC [Spark Branch]
Posted by Xuefu Zhang <xz...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/29145/#review65489
-----------------------------------------------------------
Ship it!
Ship It!
- Xuefu Zhang
On Dec. 18, 2014, 9:40 a.m., chengxiang li wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/29145/
> -----------------------------------------------------------
>
> (Updated Dec. 18, 2014, 9:40 a.m.)
>
>
> Review request for hive and Xuefu Zhang.
>
>
> Bugs: HIVE-9094
> https://issues.apache.org/jira/browse/HIVE-9094
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> RemoteHiveSparkClient::getExecutorCount timeout after 5s as Spark cluster has not launched yet
> 1. set the timeout value configurable.
> 2. set default timeout value 60s.
> 3. enable timeout for get spark job info and get spark stage info.
>
>
> Diffs
> -----
>
> common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 22f052a
> ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HiveSparkClientFactory.java 5d6a02c
> ql/src/java/org/apache/hadoop/hive/ql/exec/spark/RemoteHiveSparkClient.java 256d0b0
> ql/src/java/org/apache/hadoop/hive/ql/exec/spark/status/impl/RemoteSparkJobStatus.java 1d3a9d8
>
> Diff: https://reviews.apache.org/r/29145/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> chengxiang li
>
>
Re: Review Request 29145: HIVE-9094 TimeoutException when trying get
executor count from RSC [Spark Branch]
Posted by chengxiang li <ch...@intel.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/29145/
-----------------------------------------------------------
(Updated Dec. 18, 2014, 9:40 a.m.)
Review request for hive and Xuefu Zhang.
Changes
-------
update patch, and the setting name/desc.
Bugs: HIVE-9094
https://issues.apache.org/jira/browse/HIVE-9094
Repository: hive-git
Description
-------
RemoteHiveSparkClient::getExecutorCount timeout after 5s as Spark cluster has not launched yet
1. set the timeout value configurable.
2. set default timeout value 60s.
3. enable timeout for get spark job info and get spark stage info.
Diffs (updated)
-----
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 22f052a
ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HiveSparkClientFactory.java 5d6a02c
ql/src/java/org/apache/hadoop/hive/ql/exec/spark/RemoteHiveSparkClient.java 256d0b0
ql/src/java/org/apache/hadoop/hive/ql/exec/spark/status/impl/RemoteSparkJobStatus.java 1d3a9d8
Diff: https://reviews.apache.org/r/29145/diff/
Testing
-------
Thanks,
chengxiang li
Re: Review Request 29145: HIVE-9094 TimeoutException when trying get
executor count from RSC [Spark Branch]
Posted by Xuefu Zhang <xz...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/29145/#review65323
-----------------------------------------------------------
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
<https://reviews.apache.org/r/29145/#comment108440>
If the same timeout is used for multiple rpc calls, then the description here might need to be updated.
- Xuefu Zhang
On Dec. 17, 2014, 6:28 a.m., chengxiang li wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/29145/
> -----------------------------------------------------------
>
> (Updated Dec. 17, 2014, 6:28 a.m.)
>
>
> Review request for hive and Xuefu Zhang.
>
>
> Bugs: HIVE-9094
> https://issues.apache.org/jira/browse/HIVE-9094
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> RemoteHiveSparkClient::getExecutorCount timeout after 5s as Spark cluster has not launched yet
> 1. set the timeout value configurable.
> 2. set default timeout value 60s.
> 3. enable timeout for get spark job info and get spark stage info.
>
>
> Diffs
> -----
>
> common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 22f052a
> ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HiveSparkClientFactory.java 5d6a02c
> ql/src/java/org/apache/hadoop/hive/ql/exec/spark/RemoteHiveSparkClient.java e1946d5
> ql/src/java/org/apache/hadoop/hive/ql/exec/spark/status/impl/RemoteSparkJobStatus.java 6217de4
>
> Diff: https://reviews.apache.org/r/29145/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> chengxiang li
>
>
Re: Review Request 29145: HIVE-9094 TimeoutException when trying get
executor count from RSC [Spark Branch]
Posted by chengxiang li <ch...@intel.com>.
> On Dec. 17, 2014, 7:06 p.m., Marcelo Vanzin wrote:
> > +1 to Xuefu's comments. The config name also looks very generic, since it's only applied to a couple of jobs submitted to the client. But I don't have a good suggestion here.
While getExecutorCount/getJobInfo/getStageInfo, we use JobHandle.get() to wait result, so I use SPARK_CLIENT_FUTURE_TIMEOUT here, which means Hive would use this setting as timeout value while call JobHandle.get(), it seems more reasonable than previous name.
- chengxiang
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/29145/#review65348
-----------------------------------------------------------
On Dec. 17, 2014, 6:28 a.m., chengxiang li wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/29145/
> -----------------------------------------------------------
>
> (Updated Dec. 17, 2014, 6:28 a.m.)
>
>
> Review request for hive and Xuefu Zhang.
>
>
> Bugs: HIVE-9094
> https://issues.apache.org/jira/browse/HIVE-9094
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> RemoteHiveSparkClient::getExecutorCount timeout after 5s as Spark cluster has not launched yet
> 1. set the timeout value configurable.
> 2. set default timeout value 60s.
> 3. enable timeout for get spark job info and get spark stage info.
>
>
> Diffs
> -----
>
> common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 22f052a
> ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HiveSparkClientFactory.java 5d6a02c
> ql/src/java/org/apache/hadoop/hive/ql/exec/spark/RemoteHiveSparkClient.java e1946d5
> ql/src/java/org/apache/hadoop/hive/ql/exec/spark/status/impl/RemoteSparkJobStatus.java 6217de4
>
> Diff: https://reviews.apache.org/r/29145/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> chengxiang li
>
>
Re: Review Request 29145: HIVE-9094 TimeoutException when trying get
executor count from RSC [Spark Branch]
Posted by Marcelo Vanzin <va...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/29145/#review65348
-----------------------------------------------------------
Ship it!
+1 to Xuefu's comments. The config name also looks very generic, since it's only applied to a couple of jobs submitted to the client. But I don't have a good suggestion here.
- Marcelo Vanzin
On Dec. 17, 2014, 6:28 a.m., chengxiang li wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/29145/
> -----------------------------------------------------------
>
> (Updated Dec. 17, 2014, 6:28 a.m.)
>
>
> Review request for hive and Xuefu Zhang.
>
>
> Bugs: HIVE-9094
> https://issues.apache.org/jira/browse/HIVE-9094
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> RemoteHiveSparkClient::getExecutorCount timeout after 5s as Spark cluster has not launched yet
> 1. set the timeout value configurable.
> 2. set default timeout value 60s.
> 3. enable timeout for get spark job info and get spark stage info.
>
>
> Diffs
> -----
>
> common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 22f052a
> ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HiveSparkClientFactory.java 5d6a02c
> ql/src/java/org/apache/hadoop/hive/ql/exec/spark/RemoteHiveSparkClient.java e1946d5
> ql/src/java/org/apache/hadoop/hive/ql/exec/spark/status/impl/RemoteSparkJobStatus.java 6217de4
>
> Diff: https://reviews.apache.org/r/29145/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> chengxiang li
>
>