You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@kudu.apache.org by "Grant Henke (Code Review)" <ge...@cloudera.org> on 2019/05/31 03:47:25 UTC
[kudu-CR] WIP: [KUDU-442] Add Kudu support for Hive
Grant Henke has uploaded this change for review. ( http://gerrit.cloudera.org:8080/13483
Change subject: WIP: [KUDU-442] Add Kudu support for Hive
......................................................................
WIP: [KUDU-442] Add Kudu support for Hive
Adds initial support for a Kudu Hive StorageHandler that
supports reading and upserting data into Kudu tables.
TODO: Support all Kudu predicates.
TODO: Fix insert as select.
TODO: Handle managed tables.
TODO: A lot more testing.
Change-Id: Ifdcc72c4099dc40a7a2efa59c0c62f63902733d4
---
M java/gradle/dependencies.gradle
M java/kudu-client/src/main/java/org/apache/kudu/client/KuduPredicate.java
M java/kudu-client/src/main/java/org/apache/kudu/client/Operation.java
M java/kudu-client/src/test/java/org/apache/kudu/client/TestPartialRow.java
M java/kudu-hive/build.gradle
A java/kudu-hive/src/main/java/org/apache/kudu/hive/KuduHiveSerDe.java
A java/kudu-hive/src/main/java/org/apache/kudu/hive/KuduHiveUtils.java
A java/kudu-hive/src/main/java/org/apache/kudu/hive/KuduInputFormat.java
A java/kudu-hive/src/main/java/org/apache/kudu/hive/KuduOutputFormat.java
A java/kudu-hive/src/main/java/org/apache/kudu/hive/KuduPredicateHandler.java
A java/kudu-hive/src/main/java/org/apache/kudu/hive/KuduStorageHandler.java
A java/kudu-hive/src/main/java/org/apache/kudu/hive/KuduWritable.java
A java/kudu-hive/src/test/java/org/apache/kudu/hive/TestHiveSQL.java
M java/kudu-spark/src/main/scala/org/apache/kudu/spark/kudu/DefaultSource.scala
M java/kudu-test-utils/src/main/java/org/apache/kudu/test/ClientTestUtil.java
15 files changed, 1,395 insertions(+), 55 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/83/13483/1
--
To view, visit http://gerrit.cloudera.org:8080/13483
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: Ifdcc72c4099dc40a7a2efa59c0c62f63902733d4
Gerrit-Change-Number: 13483
Gerrit-PatchSet: 1
Gerrit-Owner: Grant Henke <gr...@apache.org>
[kudu-CR] WIP: [KUDU-442] Add Kudu support for Hive
Posted by "Grant Henke (Code Review)" <ge...@cloudera.org>.
Hello Kudu Jenkins, Greg Solovyev,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/13483
to look at the new patch set (#2).
Change subject: WIP: [KUDU-442] Add Kudu support for Hive
......................................................................
WIP: [KUDU-442] Add Kudu support for Hive
Adds initial support for a Kudu Hive StorageHandler that
supports reading and upserting data into Kudu tables.
TODO: Support all Kudu predicates.
TODO: Handle managed tables.
TODO: A lot more testing.
Change-Id: Ifdcc72c4099dc40a7a2efa59c0c62f63902733d4
---
M java/gradle/dependencies.gradle
M java/kudu-client/src/main/java/org/apache/kudu/client/KuduPredicate.java
M java/kudu-client/src/main/java/org/apache/kudu/client/PartialRow.java
M java/kudu-client/src/test/java/org/apache/kudu/client/TestPartialRow.java
M java/kudu-hive/build.gradle
A java/kudu-hive/src/main/java/org/apache/kudu/hive/KuduHiveSerDe.java
A java/kudu-hive/src/main/java/org/apache/kudu/hive/KuduHiveUtils.java
A java/kudu-hive/src/main/java/org/apache/kudu/hive/KuduInputFormat.java
A java/kudu-hive/src/main/java/org/apache/kudu/hive/KuduOutputFormat.java
A java/kudu-hive/src/main/java/org/apache/kudu/hive/KuduPredicateHandler.java
A java/kudu-hive/src/main/java/org/apache/kudu/hive/KuduStorageHandler.java
A java/kudu-hive/src/main/java/org/apache/kudu/hive/KuduWritable.java
A java/kudu-hive/src/test/java/org/apache/kudu/hive/TestHiveSQL.java
M java/kudu-spark/src/main/scala/org/apache/kudu/spark/kudu/DefaultSource.scala
M java/kudu-test-utils/src/main/java/org/apache/kudu/test/ClientTestUtil.java
15 files changed, 1,548 insertions(+), 53 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/83/13483/2
--
To view, visit http://gerrit.cloudera.org:8080/13483
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ifdcc72c4099dc40a7a2efa59c0c62f63902733d4
Gerrit-Change-Number: 13483
Gerrit-PatchSet: 2
Gerrit-Owner: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Greg Solovyev <gs...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
[kudu-CR] WIP: [KUDU-442] Add Kudu support for Hive
Posted by "Grant Henke (Code Review)" <ge...@cloudera.org>.
Grant Henke has abandoned this change. ( http://gerrit.cloudera.org:8080/13483 )
Change subject: WIP: [KUDU-442] Add Kudu support for Hive
......................................................................
Abandoned
This is being contributed to Apache Hive here: https://issues.apache.org/jira/browse/HIVE-12971
--
To view, visit http://gerrit.cloudera.org:8080/13483
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: abandon
Gerrit-Change-Id: Ifdcc72c4099dc40a7a2efa59c0c62f63902733d4
Gerrit-Change-Number: 13483
Gerrit-PatchSet: 2
Gerrit-Owner: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Greg Solovyev <gs...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
[kudu-CR] WIP: [KUDU-442] Add Kudu support for Hive
Posted by "Greg Solovyev (Code Review)" <ge...@cloudera.org>.
Greg Solovyev has posted comments on this change. ( http://gerrit.cloudera.org:8080/13483 )
Change subject: WIP: [KUDU-442] Add Kudu support for Hive
......................................................................
Patch Set 1:
(4 comments)
http://gerrit.cloudera.org:8080/#/c/13483/1/java/kudu-client/src/main/java/org/apache/kudu/client/Operation.java
File java/kudu-client/src/main/java/org/apache/kudu/client/Operation.java:
http://gerrit.cloudera.org:8080/#/c/13483/1/java/kudu-client/src/main/java/org/apache/kudu/client/Operation.java@249
PS1, Line 249: // Preconditions.checkArgument(row.getSchema() == table.getSchema(),
Why is this being commented out and what are the effects of this on Kudu java client?
http://gerrit.cloudera.org:8080/#/c/13483/1/java/kudu-hive/src/main/java/org/apache/kudu/hive/KuduHiveUtils.java
File java/kudu-hive/src/main/java/org/apache/kudu/hive/KuduHiveUtils.java:
http://gerrit.cloudera.org:8080/#/c/13483/1/java/kudu-hive/src/main/java/org/apache/kudu/hive/KuduHiveUtils.java@88
PS1, Line 88: LOG.debug("Not importing credentials for service " + service +
perhaps this would be slightly more elegant and more informative:
LOG.debug("Not importing credentials for service {} (expecting service {})", tok.getService(), service)
http://gerrit.cloudera.org:8080/#/c/13483/1/java/kudu-hive/src/main/java/org/apache/kudu/hive/KuduHiveUtils.java@92
PS1, Line 92: LOG.debug("Importing credentials for service " + service);
Use format instead of string concatenation?
http://gerrit.cloudera.org:8080/#/c/13483/1/java/kudu-hive/src/main/java/org/apache/kudu/hive/KuduInputFormat.java
File java/kudu-hive/src/main/java/org/apache/kudu/hive/KuduInputFormat.java:
http://gerrit.cloudera.org:8080/#/c/13483/1/java/kudu-hive/src/main/java/org/apache/kudu/hive/KuduInputFormat.java@94
PS1, Line 94: List<String> locations = new ArrayList<>(token.getTablet().getReplicas().size());
nit: I am not sure that using ArrayList<String> instead of simple String[] is reasonable here. Looks like the only advantage is using List.add method without having to keep a counter, while 'locations' has to be converted to String[] anyway when passed to KuduInputSplit
--
To view, visit http://gerrit.cloudera.org:8080/13483
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ifdcc72c4099dc40a7a2efa59c0c62f63902733d4
Gerrit-Change-Number: 13483
Gerrit-PatchSet: 1
Gerrit-Owner: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Greg Solovyev <gs...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Fri, 31 May 2019 20:04:58 +0000
Gerrit-HasComments: Yes