You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by Jimmy Xiang <jx...@cloudera.com> on 2015/02/26 02:42:13 UTC
Review Request 31452: HIVE-9781 Utilize spark.kryo.classesToRegister
[Spark Branch]
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/31452/
-----------------------------------------------------------
Review request for hive, Brock Noland and Xuefu Zhang.
Bugs: HIVE-9781
https://issues.apache.org/jira/browse/HIVE-9781
Repository: hive-git
Description
-------
Now, Spark RSC can use setting spark.kryo.classesToRegister. Reference checking is still needed since there are loops.
Diffs
-----
ql/src/java/org/apache/hadoop/hive/ql/exec/spark/KryoSerializer.java 76a4fd1
ql/src/java/org/apache/hadoop/hive/ql/exec/spark/RemoteHiveSparkClient.java f1d65ea
Diff: https://reviews.apache.org/r/31452/diff/
Testing
-------
Tested it on a yarn cluster with spark.kryo.classesToRegister set.
Thanks,
Jimmy Xiang
Re: Review Request 31452: HIVE-9781 Utilize
spark.kryo.classesToRegister [Spark Branch]
Posted by Xuefu Zhang <xz...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/31452/#review74370
-----------------------------------------------------------
Ship it!
Ship It!
- Xuefu Zhang
On Feb. 26, 2015, 9:38 p.m., Jimmy Xiang wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/31452/
> -----------------------------------------------------------
>
> (Updated Feb. 26, 2015, 9:38 p.m.)
>
>
> Review request for hive, Brock Noland and Xuefu Zhang.
>
>
> Bugs: HIVE-9781
> https://issues.apache.org/jira/browse/HIVE-9781
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> Now, Spark RSC can use setting spark.kryo.classesToRegister. Reference checking is still needed since there are loops.
>
>
> Diffs
> -----
>
> ql/src/java/org/apache/hadoop/hive/ql/exec/spark/KryoSerializer.java 76a4fd1
> ql/src/java/org/apache/hadoop/hive/ql/exec/spark/RemoteHiveSparkClient.java f1d65ea
>
> Diff: https://reviews.apache.org/r/31452/diff/
>
>
> Testing
> -------
>
> Tested it on a yarn cluster with spark.kryo.classesToRegister set.
>
>
> Thanks,
>
> Jimmy Xiang
>
>
Re: Review Request 31452: HIVE-9781 Utilize
spark.kryo.classesToRegister [Spark Branch]
Posted by Jimmy Xiang <jx...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/31452/
-----------------------------------------------------------
(Updated Feb. 26, 2015, 9:38 p.m.)
Review request for hive, Brock Noland and Xuefu Zhang.
Changes
-------
Instead of using spark.kryo.classesToRegister, we register those used classes, per Xuefu's suggestion.
Bugs: HIVE-9781
https://issues.apache.org/jira/browse/HIVE-9781
Repository: hive-git
Description
-------
Now, Spark RSC can use setting spark.kryo.classesToRegister. Reference checking is still needed since there are loops.
Diffs (updated)
-----
ql/src/java/org/apache/hadoop/hive/ql/exec/spark/KryoSerializer.java 76a4fd1
ql/src/java/org/apache/hadoop/hive/ql/exec/spark/RemoteHiveSparkClient.java f1d65ea
Diff: https://reviews.apache.org/r/31452/diff/
Testing
-------
Tested it on a yarn cluster with spark.kryo.classesToRegister set.
Thanks,
Jimmy Xiang
Re: Review Request 31452: HIVE-9781 Utilize
spark.kryo.classesToRegister [Spark Branch]
Posted by Jimmy Xiang <jx...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/31452/
-----------------------------------------------------------
(Updated Feb. 26, 2015, 5:39 p.m.)
Review request for hive, Brock Noland and Xuefu Zhang.
Bugs: HIVE-9781
https://issues.apache.org/jira/browse/HIVE-9781
Repository: hive-git
Description
-------
Now, Spark RSC can use setting spark.kryo.classesToRegister. Reference checking is still needed since there are loops.
Diffs (updated)
-----
ql/src/java/org/apache/hadoop/hive/ql/exec/spark/KryoSerializer.java 76a4fd1
ql/src/java/org/apache/hadoop/hive/ql/exec/spark/RemoteHiveSparkClient.java f1d65ea
Diff: https://reviews.apache.org/r/31452/diff/
Testing
-------
Tested it on a yarn cluster with spark.kryo.classesToRegister set.
Thanks,
Jimmy Xiang
Re: Review Request 31452: HIVE-9781 Utilize
spark.kryo.classesToRegister [Spark Branch]
Posted by Jimmy Xiang <jx...@cloudera.com>.
> On Feb. 26, 2015, 2:53 p.m., Xuefu Zhang wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/spark/RemoteHiveSparkClient.java, line 152
> > <https://reviews.apache.org/r/31452/diff/1/?file=877054#file877054line152>
> >
> > Could we rename this variable for better readability?
Sure.
- Jimmy
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/31452/#review74273
-----------------------------------------------------------
On Feb. 26, 2015, 1:42 a.m., Jimmy Xiang wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/31452/
> -----------------------------------------------------------
>
> (Updated Feb. 26, 2015, 1:42 a.m.)
>
>
> Review request for hive, Brock Noland and Xuefu Zhang.
>
>
> Bugs: HIVE-9781
> https://issues.apache.org/jira/browse/HIVE-9781
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> Now, Spark RSC can use setting spark.kryo.classesToRegister. Reference checking is still needed since there are loops.
>
>
> Diffs
> -----
>
> ql/src/java/org/apache/hadoop/hive/ql/exec/spark/KryoSerializer.java 76a4fd1
> ql/src/java/org/apache/hadoop/hive/ql/exec/spark/RemoteHiveSparkClient.java f1d65ea
>
> Diff: https://reviews.apache.org/r/31452/diff/
>
>
> Testing
> -------
>
> Tested it on a yarn cluster with spark.kryo.classesToRegister set.
>
>
> Thanks,
>
> Jimmy Xiang
>
>
Re: Review Request 31452: HIVE-9781 Utilize
spark.kryo.classesToRegister [Spark Branch]
Posted by Xuefu Zhang <xz...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/31452/#review74273
-----------------------------------------------------------
ql/src/java/org/apache/hadoop/hive/ql/exec/spark/RemoteHiveSparkClient.java
<https://reviews.apache.org/r/31452/#comment120817>
Could we rename this variable for better readability?
- Xuefu Zhang
On Feb. 26, 2015, 1:42 a.m., Jimmy Xiang wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/31452/
> -----------------------------------------------------------
>
> (Updated Feb. 26, 2015, 1:42 a.m.)
>
>
> Review request for hive, Brock Noland and Xuefu Zhang.
>
>
> Bugs: HIVE-9781
> https://issues.apache.org/jira/browse/HIVE-9781
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> Now, Spark RSC can use setting spark.kryo.classesToRegister. Reference checking is still needed since there are loops.
>
>
> Diffs
> -----
>
> ql/src/java/org/apache/hadoop/hive/ql/exec/spark/KryoSerializer.java 76a4fd1
> ql/src/java/org/apache/hadoop/hive/ql/exec/spark/RemoteHiveSparkClient.java f1d65ea
>
> Diff: https://reviews.apache.org/r/31452/diff/
>
>
> Testing
> -------
>
> Tested it on a yarn cluster with spark.kryo.classesToRegister set.
>
>
> Thanks,
>
> Jimmy Xiang
>
>