You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@kudu.apache.org by "yangz (Code Review)" <ge...@cloudera.org> on 2019/02/02 15:10:19 UTC
[kudu-CR] KUDU-2672: Spark write to kudu, too many machines write to one tserver.
yangz has uploaded this change for review. ( http://gerrit.cloudera.org:8080/12341
Change subject: KUDU-2672: Spark write to kudu, too many machines write to one tserver.
......................................................................
KUDU-2672: Spark write to kudu, too many machines write to one tserver.
1. Change kudu flush mode to MANUAL mode.
2. add a optional repartition step for the write.
Change-Id: Ic6e0df2d43e93dc60d0b5c68daf91eb75b8d3bc7
---
M java/kudu-client/src/main/java/org/apache/kudu/client/AsyncKuduClient.java
M java/kudu-client/src/main/java/org/apache/kudu/client/KeyEncoder.java
M java/kudu-client/src/main/java/org/apache/kudu/client/RemoteTablet.java
M java/kudu-client/src/main/java/org/apache/kudu/client/TableLocationsCache.java
M java/kudu-spark/src/main/scala/org/apache/kudu/spark/kudu/KuduContext.scala
5 files changed, 270 insertions(+), 56 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/41/12341/1
--
To view, visit http://gerrit.cloudera.org:8080/12341
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: Ic6e0df2d43e93dc60d0b5c68daf91eb75b8d3bc7
Gerrit-Change-Number: 12341
Gerrit-PatchSet: 1
Gerrit-Owner: yangz <zh...@gmail.com>
[kudu-CR] KUDU-2672: Spark write to kudu, too many machines write to one tserver.
Posted by "Grant Henke (Code Review)" <ge...@cloudera.org>.
Grant Henke has posted comments on this change. ( http://gerrit.cloudera.org:8080/12341 )
Change subject: KUDU-2672: Spark write to kudu, too many machines write to one tserver.
......................................................................
Patch Set 1:
Also a word of warning. I found a spark bug while working on my patch and needed to fix our use of InternalRow to prevent data corruption/loss. You may want to check if this code has that issue: https://issues.apache.org/jira/browse/SPARK-26880
--
To view, visit http://gerrit.cloudera.org:8080/12341
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic6e0df2d43e93dc60d0b5c68daf91eb75b8d3bc7
Gerrit-Change-Number: 12341
Gerrit-PatchSet: 1
Gerrit-Owner: yangz <zh...@gmail.com>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Thu, 14 Feb 2019 18:20:33 +0000
Gerrit-HasComments: No
[kudu-CR] KUDU-2672: Spark write to kudu, too many machines write to one tserver.
Posted by "yangz (Code Review)" <ge...@cloudera.org>.
yangz has posted comments on this change. ( http://gerrit.cloudera.org:8080/12341 )
Change subject: KUDU-2672: Spark write to kudu, too many machines write to one tserver.
......................................................................
Patch Set 1:
Thanks for reply. I am out of the job now. But Xu Yao at our team will continue it. He will reply for this patch soon.
--
To view, visit http://gerrit.cloudera.org:8080/12341
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic6e0df2d43e93dc60d0b5c68daf91eb75b8d3bc7
Gerrit-Change-Number: 12341
Gerrit-PatchSet: 1
Gerrit-Owner: yangz <zh...@gmail.com>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: yangz <zh...@gmail.com>
Gerrit-Comment-Date: Fri, 01 Mar 2019 04:10:50 +0000
Gerrit-HasComments: No
[kudu-CR] KUDU-2672: Spark write to kudu, too many machines write to one tserver.
Posted by "Grant Henke (Code Review)" <ge...@cloudera.org>.
Grant Henke has abandoned this change. ( http://gerrit.cloudera.org:8080/12341 )
Change subject: KUDU-2672: Spark write to kudu, too many machines write to one tserver.
......................................................................
Abandoned
This was handled in https://gerrit.cloudera.org/#/c/12484/
--
To view, visit http://gerrit.cloudera.org:8080/12341
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: abandon
Gerrit-Change-Id: Ic6e0df2d43e93dc60d0b5c68daf91eb75b8d3bc7
Gerrit-Change-Number: 12341
Gerrit-PatchSet: 1
Gerrit-Owner: yangz <zh...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: yangz <zh...@gmail.com>
[kudu-CR] KUDU-2672: Spark write to kudu, too many machines write to one tserver.
Posted by "Grant Henke (Code Review)" <ge...@cloudera.org>.
Grant Henke has posted comments on this change. ( http://gerrit.cloudera.org:8080/12341 )
Change subject: KUDU-2672: Spark write to kudu, too many machines write to one tserver.
......................................................................
Patch Set 1:
Checking in. Are you interested in updating the patch based on the review feedback? If not, I can address my concerns. Thank you again for the initial patch.
--
To view, visit http://gerrit.cloudera.org:8080/12341
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic6e0df2d43e93dc60d0b5c68daf91eb75b8d3bc7
Gerrit-Change-Number: 12341
Gerrit-PatchSet: 1
Gerrit-Owner: yangz <zh...@gmail.com>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Fri, 08 Feb 2019 21:38:41 +0000
Gerrit-HasComments: No
[kudu-CR] KUDU-2672: Spark write to kudu, too many machines write to one tserver.
Posted by "Grant Henke (Code Review)" <ge...@cloudera.org>.
Grant Henke has posted comments on this change. ( http://gerrit.cloudera.org:8080/12341 )
Change subject: KUDU-2672: Spark write to kudu, too many machines write to one tserver.
......................................................................
Patch Set 1:
I hope you don't mind. Given the KuduPartitioner patch was committed and I haven't received any response on my comments, I implemented the repartitioning part of this patch here: https://gerrit.cloudera.org/#/c/12484/
--
To view, visit http://gerrit.cloudera.org:8080/12341
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic6e0df2d43e93dc60d0b5c68daf91eb75b8d3bc7
Gerrit-Change-Number: 12341
Gerrit-PatchSet: 1
Gerrit-Owner: yangz <zh...@gmail.com>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Thu, 14 Feb 2019 18:18:17 +0000
Gerrit-HasComments: No
[kudu-CR] KUDU-2672: Spark write to kudu, too many machines write to one tserver.
Posted by "Grant Henke (Code Review)" <ge...@cloudera.org>.
Grant Henke has posted comments on this change. ( http://gerrit.cloudera.org:8080/12341 )
Change subject: KUDU-2672: Spark write to kudu, too many machines write to one tserver.
......................................................................
Patch Set 1:
(3 comments)
Thank you for the contribution! I added some high level requests and feedback below.
http://gerrit.cloudera.org:8080/#/c/12341/1/java/kudu-spark/src/main/scala/org/apache/kudu/spark/kudu/KuduContext.scala
File java/kudu-spark/src/main/scala/org/apache/kudu/spark/kudu/KuduContext.scala:
http://gerrit.cloudera.org:8080/#/c/12341/1/java/kudu-spark/src/main/scala/org/apache/kudu/spark/kudu/KuduContext.scala@76
PS1, Line 76: val retryTimes = sc.getConf.getInt("spark.kudu.flush.retry", 3)
Do you mind breaking the flush changes into a separate patch? The two features are related to writing but I think we can have clearer reviews and feedback if they are separate.
http://gerrit.cloudera.org:8080/#/c/12341/1/java/kudu-spark/src/main/scala/org/apache/kudu/spark/kudu/KuduContext.scala@333
PS1, Line 333: private[kudu] def writeRows(
Instead of the "spark.kudu.batch.repartition" config, can we add an option to writeOptions?
The config can then be added to DefaultSource and mapped to be set in the options similar to "kudu.ignoreNull"
http://gerrit.cloudera.org:8080/#/c/12341/1/java/kudu-spark/src/main/scala/org/apache/kudu/spark/kudu/KuduContext.scala@346
PS1, Line 346: val partitioner = new PartitionerByTablet(tabletNum)
Could we implement the Spark partition based on the KuduPartitioner in this patch?:
https://gerrit.cloudera.org/#/c/12275/
This makes it so we don't need to expose and use non-public client APIs that are meant to be internal to the client.
--
To view, visit http://gerrit.cloudera.org:8080/12341
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic6e0df2d43e93dc60d0b5c68daf91eb75b8d3bc7
Gerrit-Change-Number: 12341
Gerrit-PatchSet: 1
Gerrit-Owner: yangz <zh...@gmail.com>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Wed, 06 Feb 2019 20:06:54 +0000
Gerrit-HasComments: Yes
[kudu-CR] KUDU-2672: Spark write to kudu, too many machines write to one tserver.
Posted by "Grant Henke (Code Review)" <ge...@cloudera.org>.
Grant Henke has removed a vote on this change.
Change subject: KUDU-2672: Spark write to kudu, too many machines write to one tserver.
......................................................................
Removed Verified-1 by Kudu Jenkins (120)
--
To view, visit http://gerrit.cloudera.org:8080/12341
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: deleteVote
Gerrit-Change-Id: Ic6e0df2d43e93dc60d0b5c68daf91eb75b8d3bc7
Gerrit-Change-Number: 12341
Gerrit-PatchSet: 1
Gerrit-Owner: yangz <zh...@gmail.com>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)