You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2020/03/13 02:51:54 UTC
[GitHub] [incubator-iceberg] jerryshao opened a new pull request #841:
Upgrade Hive to 2.3.6 for Iceberg
jerryshao opened a new pull request #841: Upgrade Hive to 2.3.6 for Iceberg
URL: https://github.com/apache/incubator-iceberg/pull/841
This PR address the issue of #832 , upgrading Hive version from 1.2.1 to 2.3.6. The main reason to this upgrade is to make future work (Hive Iceberg support) easy to land to Iceberg codebase.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [incubator-iceberg] rdblue commented on issue #841: Upgrade Hive to
2.3.6 for Iceberg
Posted by GitBox <gi...@apache.org>.
rdblue commented on issue #841: Upgrade Hive to 2.3.6 for Iceberg
URL: https://github.com/apache/incubator-iceberg/pull/841#issuecomment-600163797
Merging this. Thanks for the quick fixes, @jerryshao!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [incubator-iceberg] rdblue commented on a change in pull request
#841: Upgrade Hive to 2.3.6 for Iceberg
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #841: Upgrade Hive to 2.3.6 for Iceberg
URL: https://github.com/apache/incubator-iceberg/pull/841#discussion_r392333374
##########
File path: hive/src/main/java/org/apache/iceberg/hive/HiveTableOperations.java
##########
@@ -175,7 +190,12 @@ protected void doCommit(TableMetadata base, TableMetadata metadata) {
EnvironmentContext envContext = new EnvironmentContext(
ImmutableMap.of(StatsSetupConst.DO_NOT_UPDATE_STATS, StatsSetupConst.TRUE)
);
- client.alter_table(database, tableName, tbl, envContext);
+
+ try {
+ alterTableMethod.invoke(client, database, tableName, tbl, envContext);
+ } catch (Exception e) {
+ throw new RuntimeMetaException(e, "Failed to alter table for %s.%s", database, tableName);
+ }
Review comment:
If you use `DynMethods` then you'd call the method like this:
```java
try {
clients.run(client -> ALTER_TABLE.invoke(client, database, tableName, tbl, envContext));
} catch (TException e) {
throw new RuntimeMetaException(e, "Failed to alter table");
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
throw new RuntimeException("Interrupted in call to alterTable", e);
}
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [incubator-iceberg] rdblue commented on a change in pull request
#841: Upgrade Hive to 2.3.6 for Iceberg
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #841: Upgrade Hive to 2.3.6 for Iceberg
URL: https://github.com/apache/incubator-iceberg/pull/841#discussion_r393176119
##########
File path: versions.lock
##########
@@ -37,34 +37,36 @@ com.sun.jersey:jersey-json:1.9 (5 constraints: 945f2f90)
com.sun.jersey:jersey-server:1.9 (4 constraints: ef373c01)
com.sun.jersey.contribs:jersey-guice:1.9 (4 constraints: 65529ed7)
com.sun.xml.bind:jaxb-impl:2.2.3-1 (1 constraints: 330c2404)
+com.tdunning:json:1.8 (1 constraints: 910b72db)
com.thoughtworks.paranamer:paranamer:2.8 (3 constraints: 742d4cb1)
com.twitter:chill-java:0.9.3 (2 constraints: a716716f)
com.twitter:chill_2.11:0.9.3 (2 constraints: 121b92c3)
-com.twitter:parquet-hadoop-bundle:1.6.0 (3 constraints: 7c262424)
+com.twitter:parquet-hadoop-bundle:1.6.0 (2 constraints: 061b4d93)
com.univocity:univocity-parsers:2.7.3 (1 constraints: c40ccb27)
+com.zaxxer:HikariCP:2.5.1 (1 constraints: 390d7120)
Review comment:
Do we know what this is and whether it is needed?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [incubator-iceberg] jerryshao commented on a change in pull request
#841: Upgrade Hive to 2.3.6 for Iceberg
Posted by GitBox <gi...@apache.org>.
jerryshao commented on a change in pull request #841: Upgrade Hive to 2.3.6 for Iceberg
URL: https://github.com/apache/incubator-iceberg/pull/841#discussion_r392766876
##########
File path: versions.lock
##########
@@ -1,31 +1,39 @@
# Run ./gradlew --write-locks to regenerate this file
-ant:ant:1.6.5 (1 constraints: bb0a41d8)
-antlr:antlr:2.7.7 (2 constraints: 36167e02)
+ant:ant:1.6.5 (2 constraints: 2d1539e1)
aopalliance:aopalliance:1.0 (1 constraints: 170a83ac)
-asm:asm:3.1 (2 constraints: 4f19c3c6)
+asm:asm:3.1 (3 constraints: 251fd7ad)
+asm:asm-commons:3.1 (1 constraints: 9c0f1f7a)
+asm:asm-tree:3.1 (1 constraints: 2307035c)
+ch.qos.logback:logback-classic:1.0.9 (3 constraints: d3258a88)
+ch.qos.logback:logback-core:1.0.9 (4 constraints: dd32435a)
+co.cask.tephra:tephra-api:0.6.0 (3 constraints: 0828ded1)
+co.cask.tephra:tephra-core:0.6.0 (2 constraints: 831cd90d)
+co.cask.tephra:tephra-hbase-compat-1.0:0.6.0 (1 constraints: 370d6920)
com.carrotsearch:hppc:0.7.2 (1 constraints: f70cda14)
com.clearspring.analytics:stream:2.7.0 (1 constraints: 1a0dd136)
com.esotericsoftware:kryo-shaded:4.0.2 (2 constraints: b71345a6)
com.esotericsoftware:minlog:1.3.0 (1 constraints: 670e7c4f)
com.fasterxml.jackson.core:jackson-annotations:2.10.2 (5 constraints: 4155160f)
com.fasterxml.jackson.core:jackson-core:2.10.2 (6 constraints: bb52b302)
-com.fasterxml.jackson.core:jackson-databind:2.10.2 (10 constraints: 8a92f4c4)
+com.fasterxml.jackson.core:jackson-databind:2.10.2 (11 constraints: 7c9eca9a)
com.fasterxml.jackson.module:jackson-module-paranamer:2.10.2 (1 constraints: 03162c16)
com.fasterxml.jackson.module:jackson-module-scala_2.11:2.10.2 (1 constraints: 7f0da251)
com.github.ben-manes.caffeine:caffeine:2.7.0 (1 constraints: 0b050a36)
+com.github.joshelser:dropwizard-metrics-hadoop-metrics2-reporter:0.1.2 (1 constraints: e90b08f3)
com.github.luben:zstd-jni:1.3.2-2 (1 constraints: 760d7c51)
com.github.stephenc.findbugs:findbugs-annotations:1.3.9-1 (1 constraints: 6d05ab40)
-com.google.code.findbugs:jsr305:3.0.2 (9 constraints: d276cf3c)
-com.google.code.gson:gson:2.2.4 (1 constraints: 8c0d3f2f)
+com.google.code.findbugs:jsr305:3.0.2 (11 constraints: 2290ea31)
+com.google.code.gson:gson:2.2.4 (5 constraints: 3b40b359)
Review comment:
This is the dependency graph for jsr305
```
com.google.code.findbugs:jsr305:3.0.2
com.google.guava:guava -> 3.0.2
org.apache.arrow:arrow-memory -> 3.0.2
org.apache.hadoop:hadoop-common -> 3.0.0
org.apache.hive:hive-serde -> 3.0.0
org.apache.spark:spark-core_2.11 -> 1.3.9
org.apache.spark:spark-hive_2.11 -> 1.3.9
org.apache.spark:spark-network-common_2.11 -> 1.3.9
org.apache.spark:spark-unsafe_2.11 -> 1.3.9
```
This is for `gson`.
```
com.google.code.gson:gson:2.2.4
org.apache.hadoop:hadoop-common -> 2.2.4
org.apache.hive:hive-exec -> 2.2.4
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [incubator-iceberg] rdblue commented on a change in pull request
#841: Upgrade Hive to 2.3.6 for Iceberg
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #841: Upgrade Hive to 2.3.6 for Iceberg
URL: https://github.com/apache/incubator-iceberg/pull/841#discussion_r392335722
##########
File path: versions.lock
##########
@@ -1,31 +1,39 @@
# Run ./gradlew --write-locks to regenerate this file
-ant:ant:1.6.5 (1 constraints: bb0a41d8)
-antlr:antlr:2.7.7 (2 constraints: 36167e02)
+ant:ant:1.6.5 (2 constraints: 2d1539e1)
aopalliance:aopalliance:1.0 (1 constraints: 170a83ac)
-asm:asm:3.1 (2 constraints: 4f19c3c6)
+asm:asm:3.1 (3 constraints: 251fd7ad)
+asm:asm-commons:3.1 (1 constraints: 9c0f1f7a)
+asm:asm-tree:3.1 (1 constraints: 2307035c)
+ch.qos.logback:logback-classic:1.0.9 (3 constraints: d3258a88)
+ch.qos.logback:logback-core:1.0.9 (4 constraints: dd32435a)
+co.cask.tephra:tephra-api:0.6.0 (3 constraints: 0828ded1)
+co.cask.tephra:tephra-core:0.6.0 (2 constraints: 831cd90d)
+co.cask.tephra:tephra-hbase-compat-1.0:0.6.0 (1 constraints: 370d6920)
com.carrotsearch:hppc:0.7.2 (1 constraints: f70cda14)
com.clearspring.analytics:stream:2.7.0 (1 constraints: 1a0dd136)
com.esotericsoftware:kryo-shaded:4.0.2 (2 constraints: b71345a6)
com.esotericsoftware:minlog:1.3.0 (1 constraints: 670e7c4f)
com.fasterxml.jackson.core:jackson-annotations:2.10.2 (5 constraints: 4155160f)
com.fasterxml.jackson.core:jackson-core:2.10.2 (6 constraints: bb52b302)
-com.fasterxml.jackson.core:jackson-databind:2.10.2 (10 constraints: 8a92f4c4)
+com.fasterxml.jackson.core:jackson-databind:2.10.2 (11 constraints: 7c9eca9a)
com.fasterxml.jackson.module:jackson-module-paranamer:2.10.2 (1 constraints: 03162c16)
com.fasterxml.jackson.module:jackson-module-scala_2.11:2.10.2 (1 constraints: 7f0da251)
com.github.ben-manes.caffeine:caffeine:2.7.0 (1 constraints: 0b050a36)
+com.github.joshelser:dropwizard-metrics-hadoop-metrics2-reporter:0.1.2 (1 constraints: e90b08f3)
com.github.luben:zstd-jni:1.3.2-2 (1 constraints: 760d7c51)
com.github.stephenc.findbugs:findbugs-annotations:1.3.9-1 (1 constraints: 6d05ab40)
-com.google.code.findbugs:jsr305:3.0.2 (9 constraints: d276cf3c)
-com.google.code.gson:gson:2.2.4 (1 constraints: 8c0d3f2f)
+com.google.code.findbugs:jsr305:3.0.2 (11 constraints: 2290ea31)
+com.google.code.gson:gson:2.2.4 (5 constraints: 3b40b359)
Review comment:
Is this needed?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [incubator-iceberg] rdblue commented on a change in pull request
#841: Upgrade Hive to 2.3.6 for Iceberg
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #841: Upgrade Hive to 2.3.6 for Iceberg
URL: https://github.com/apache/incubator-iceberg/pull/841#discussion_r392336915
##########
File path: versions.lock
##########
@@ -37,90 +45,98 @@ com.sun.jersey:jersey-json:1.9 (5 constraints: 945f2f90)
com.sun.jersey:jersey-server:1.9 (4 constraints: ef373c01)
com.sun.jersey.contribs:jersey-guice:1.9 (4 constraints: 65529ed7)
com.sun.xml.bind:jaxb-impl:2.2.3-1 (1 constraints: 330c2404)
+com.tdunning:json:1.8 (1 constraints: 910b72db)
com.thoughtworks.paranamer:paranamer:2.8 (3 constraints: 742d4cb1)
com.twitter:chill-java:0.9.3 (2 constraints: a716716f)
com.twitter:chill_2.11:0.9.3 (2 constraints: 121b92c3)
-com.twitter:parquet-hadoop-bundle:1.6.0 (3 constraints: 7c262424)
+com.twitter:parquet-hadoop-bundle:1.6.0 (2 constraints: 061b4d93)
com.univocity:univocity-parsers:2.7.3 (1 constraints: c40ccb27)
+com.zaxxer:HikariCP:2.5.1 (1 constraints: 390d7120)
commons-beanutils:commons-beanutils:1.7.0 (1 constraints: da0e635f)
commons-beanutils:commons-beanutils-core:1.8.0 (1 constraints: 1d134124)
-commons-cli:commons-cli:1.2 (8 constraints: 9467c282)
-commons-codec:commons-codec:1.10 (16 constraints: 42d1423c)
-commons-collections:commons-collections:3.2.2 (4 constraints: 42476934)
+commons-cli:commons-cli:1.2 (9 constraints: f874366c)
+commons-codec:commons-codec:1.10 (17 constraints: a8de3870)
+commons-collections:commons-collections:3.2.2 (3 constraints: e73a8e36)
commons-configuration:commons-configuration:1.6 (1 constraints: 2d0d5c14)
commons-daemon:commons-daemon:1.0.13 (1 constraints: d50c811c)
commons-dbcp:commons-dbcp:1.4 (3 constraints: 9029e0e4)
commons-digester:commons-digester:1.8 (1 constraints: bf1228fe)
-commons-el:commons-el:1.0 (1 constraints: fb077074)
+commons-el:commons-el:1.0 (2 constraints: ad11e7f0)
commons-httpclient:commons-httpclient:3.1 (4 constraints: e52cc77f)
commons-io:commons-io:2.4 (6 constraints: 4a568049)
-commons-lang:commons-lang:2.6 (19 constraints: 5f0d34d8)
-commons-logging:commons-logging:1.2 (27 constraints: f2ab390b)
+commons-lang:commons-lang:2.6 (20 constraints: 401f63f3)
+commons-logging:commons-logging:1.2 (20 constraints: 424b646e)
commons-net:commons-net:3.1 (3 constraints: 3d222e61)
commons-pool:commons-pool:1.6 (4 constraints: e336ab5e)
dk.brics.automaton:automaton:1.11-8 (1 constraints: 92088a8d)
hsqldb:hsqldb:1.8.0.10 (1 constraints: f008499f)
io.airlift:aircompressor:0.15 (1 constraints: 0e0aa4b2)
-io.dropwizard.metrics:metrics-core:3.1.5 (6 constraints: 865ea0ba)
+io.dropwizard.metrics:metrics-core:3.1.5 (9 constraints: be90b1ee)
io.dropwizard.metrics:metrics-graphite:3.1.5 (1 constraints: 1a0dc936)
-io.dropwizard.metrics:metrics-json:3.1.5 (1 constraints: 1a0dc936)
-io.dropwizard.metrics:metrics-jvm:3.1.5 (1 constraints: 1a0dc936)
+io.dropwizard.metrics:metrics-json:3.1.5 (2 constraints: 03195c12)
+io.dropwizard.metrics:metrics-jvm:3.1.5 (2 constraints: 03195c12)
io.netty:netty:3.9.9.Final (9 constraints: 9eb0396d)
io.netty:netty-all:4.1.17.Final (3 constraints: d2312526)
io.netty:netty-buffer:4.1.27.Final (1 constraints: 4a0fee77)
-javax.activation:activation:1.1.1 (1 constraints: 140dbb36)
+it.unimi.dsi:fastutil:6.5.6 (1 constraints: 910b3ce5)
+javax.activation:activation:1.1.1 (3 constraints: b02331a0)
javax.annotation:javax.annotation-api:1.3.2 (3 constraints: 55341c48)
javax.inject:javax.inject:1 (4 constraints: 852d0c1a)
javax.jdo:jdo-api:3.0.1 (2 constraints: 4c1dcc1a)
+javax.mail:mail:1.4.1 (1 constraints: fc0fe399)
javax.servlet:javax.servlet-api:3.1.0 (1 constraints: 150dc436)
-javax.servlet:servlet-api:2.5 (7 constraints: c87e5aa1)
+javax.servlet:jsp-api:2.0 (1 constraints: 0b0aa0a7)
+javax.servlet:servlet-api:2.5 (9 constraints: f991a6d2)
javax.servlet.jsp:jsp-api:2.1 (1 constraints: 290d5a14)
javax.transaction:jta:1.1 (1 constraints: 9f07d96b)
+javax.transaction:transaction-api:1.1 (1 constraints: 0a0b64c9)
javax.validation:validation-api:1.1.0.Final (1 constraints: 13133130)
javax.ws.rs:javax.ws.rs-api:2.0.1 (5 constraints: 6e649355)
javax.xml.bind:jaxb-api:2.2.11 (6 constraints: a069fd48)
-javolution:javolution:5.5.1 (1 constraints: f00d1a43)
-jline:jline:2.12 (3 constraints: 98208776)
-joda-time:joda-time:2.9.9 (4 constraints: c125a2fb)
-log4j:apache-log4j-extras:1.2.17 (4 constraints: 3f36b1af)
-log4j:log4j:1.2.17 (12 constraints: 22ab5529)
+javolution:javolution:5.5.1 (2 constraints: 2b1b2b82)
+jline:jline:2.12 (3 constraints: 7c21b2cb)
+joda-time:joda-time:2.9.9 (4 constraints: 2326d336)
+log4j:apache-log4j-extras:1.2.17 (1 constraints: 200e1d51)
+log4j:log4j:1.2.17 (8 constraints: e7772b11)
net.hydromatic:eigenbase-properties:1.1.5 (1 constraints: 5f0daf2c)
net.java.dev.jets3t:jets3t:0.9.0 (2 constraints: ec152b22)
net.razorvine:pyrolite:4.13 (1 constraints: eb0cb829)
net.sf.kosmosfs:kfs:0.3 (1 constraints: fd077074)
net.sf.opencsv:opencsv:2.3 (2 constraints: a218daa5)
net.sf.py4j:py4j:0.10.7 (1 constraints: 490d0044)
org.antlr:ST4:4.0.4 (3 constraints: 5521e4e4)
-org.antlr:antlr-runtime:3.4 (6 constraints: b84229a1)
+org.antlr:antlr-runtime:3.5.2 (6 constraints: 7a43035f)
org.antlr:antlr4-runtime:4.7 (1 constraints: 7a0e125f)
-org.antlr:stringtemplate:3.2.1 (1 constraints: c10a3bc6)
-org.apache.ant:ant:1.9.1 (3 constraints: a721ed14)
+org.apache.ant:ant:1.9.1 (3 constraints: 262660e7)
org.apache.ant:ant-launcher:1.9.1 (1 constraints: 69082485)
org.apache.arrow:arrow-format:0.14.1 (1 constraints: 240df421)
org.apache.arrow:arrow-memory:0.14.1 (1 constraints: 240df421)
org.apache.arrow:arrow-vector:0.14.1 (2 constraints: 2012a545)
org.apache.avro:avro:1.9.2 (4 constraints: 3e2e68f4)
org.apache.avro:avro-ipc:1.8.2 (1 constraints: f90b5bf4)
org.apache.avro:avro-mapred:1.8.2 (2 constraints: 3a1a4787)
-org.apache.calcite:calcite-avatica:1.2.0-incubating (4 constraints: a044b922)
-org.apache.calcite:calcite-core:1.2.0-incubating (2 constraints: bd20f965)
+org.apache.calcite:calcite-avatica:1.2.0-incubating (3 constraints: 4b35b263)
+org.apache.calcite:calcite-core:1.2.0-incubating (1 constraints: 68119fdf)
org.apache.calcite:calcite-linq4j:1.2.0-incubating (1 constraints: ac1147d8)
-org.apache.commons:commons-compress:1.19 (6 constraints: fa4af584)
+org.apache.commons:commons-compress:1.19 (6 constraints: 464a0c7f)
org.apache.commons:commons-crypto:1.0.0 (2 constraints: 3a1e5fbf)
-org.apache.commons:commons-lang3:3.9 (4 constraints: c52f877f)
+org.apache.commons:commons-lang3:3.9 (5 constraints: 503b94b4)
org.apache.commons:commons-math3:3.4.1 (2 constraints: a11af290)
org.apache.curator:curator-client:2.7.1 (2 constraints: 6a1d2734)
-org.apache.curator:curator-framework:2.7.1 (4 constraints: 4937d02c)
+org.apache.curator:curator-framework:2.7.1 (4 constraints: 4d37382d)
org.apache.curator:curator-recipes:2.7.1 (2 constraints: a61acc91)
org.apache.derby:derby:10.12.1.1 (3 constraints: 9f2cb182)
org.apache.directory.api:api-asn1-api:1.0.0-M20 (1 constraints: 3d163b13)
org.apache.directory.api:api-util:1.0.0-M20 (1 constraints: 3d163b13)
org.apache.directory.server:apacheds-i18n:2.0.0-M15 (1 constraints: 42164713)
org.apache.directory.server:apacheds-kerberos-codec:2.0.0-M15 (1 constraints: 8f0d3b45)
+org.apache.geronimo.specs:geronimo-annotation_1.0_spec:1.1.1 (1 constraints: f90fda99)
+org.apache.geronimo.specs:geronimo-jaspic_1.0_spec:1.0 (1 constraints: 990f187a)
+org.apache.geronimo.specs:geronimo-jta_1.1_spec:1.1.1 (1 constraints: f90fda99)
Review comment:
What are these? Are they needed?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [incubator-iceberg] rdblue commented on a change in pull request
#841: Upgrade Hive to 2.3.6 for Iceberg
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #841: Upgrade Hive to 2.3.6 for Iceberg
URL: https://github.com/apache/incubator-iceberg/pull/841#discussion_r392337433
##########
File path: versions.lock
##########
@@ -170,24 +188,33 @@ org.apache.spark:spark-sketch_2.11:2.4.4 (2 constraints: 981bd4f5)
org.apache.spark:spark-sql_2.11:2.4.4 (1 constraints: 1e0d0037)
org.apache.spark:spark-tags_2.11:2.4.4 (8 constraints: 036fa69d)
org.apache.spark:spark-unsafe_2.11:2.4.4 (2 constraints: f11bc213)
-org.apache.thrift:libfb303:0.9.3 (3 constraints: 6725fac0)
-org.apache.thrift:libthrift:0.9.3 (5 constraints: 71415452)
+org.apache.thrift:libfb303:0.9.3 (3 constraints: 27289d07)
+org.apache.thrift:libthrift:0.9.3 (7 constraints: c75ac7de)
+org.apache.twill:twill-api:0.6.0-incubating (2 constraints: e422b4e1)
+org.apache.twill:twill-common:0.6.0-incubating (4 constraints: 3d467991)
+org.apache.twill:twill-core:0.6.0-incubating (1 constraints: d70f9d7c)
+org.apache.twill:twill-discovery-api:0.6.0-incubating (3 constraints: 25345d4c)
+org.apache.twill:twill-discovery-core:0.6.0-incubating (2 constraints: 332039f9)
+org.apache.twill:twill-zookeeper:0.6.0-incubating (3 constraints: 94341288)
Review comment:
Can we remove twill?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [incubator-iceberg] jerryshao commented on a change in pull request
#841: Upgrade Hive to 2.3.6 for Iceberg
Posted by GitBox <gi...@apache.org>.
jerryshao commented on a change in pull request #841: Upgrade Hive to 2.3.6 for Iceberg
URL: https://github.com/apache/incubator-iceberg/pull/841#discussion_r393614374
##########
File path: versions.lock
##########
@@ -37,34 +37,36 @@ com.sun.jersey:jersey-json:1.9 (5 constraints: 945f2f90)
com.sun.jersey:jersey-server:1.9 (4 constraints: ef373c01)
com.sun.jersey.contribs:jersey-guice:1.9 (4 constraints: 65529ed7)
com.sun.xml.bind:jaxb-impl:2.2.3-1 (1 constraints: 330c2404)
+com.tdunning:json:1.8 (1 constraints: 910b72db)
com.thoughtworks.paranamer:paranamer:2.8 (3 constraints: 742d4cb1)
com.twitter:chill-java:0.9.3 (2 constraints: a716716f)
com.twitter:chill_2.11:0.9.3 (2 constraints: 121b92c3)
-com.twitter:parquet-hadoop-bundle:1.6.0 (3 constraints: 7c262424)
+com.twitter:parquet-hadoop-bundle:1.6.0 (2 constraints: 061b4d93)
com.univocity:univocity-parsers:2.7.3 (1 constraints: c40ccb27)
+com.zaxxer:HikariCP:2.5.1 (1 constraints: 390d7120)
Review comment:
Seems not necessary to add, I've already excluded this dependency.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [incubator-iceberg] rdblue merged pull request #841: Upgrade Hive
to 2.3.6 for Iceberg
Posted by GitBox <gi...@apache.org>.
rdblue merged pull request #841: Upgrade Hive to 2.3.6 for Iceberg
URL: https://github.com/apache/incubator-iceberg/pull/841
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [incubator-iceberg] rdblue commented on a change in pull request
#841: Upgrade Hive to 2.3.6 for Iceberg
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #841: Upgrade Hive to 2.3.6 for Iceberg
URL: https://github.com/apache/incubator-iceberg/pull/841#discussion_r392334513
##########
File path: hive/src/main/java/org/apache/iceberg/hive/HiveTableOperations.java
##########
@@ -175,7 +190,12 @@ protected void doCommit(TableMetadata base, TableMetadata metadata) {
EnvironmentContext envContext = new EnvironmentContext(
ImmutableMap.of(StatsSetupConst.DO_NOT_UPDATE_STATS, StatsSetupConst.TRUE)
);
- client.alter_table(database, tableName, tbl, envContext);
+
+ try {
+ alterTableMethod.invoke(client, database, tableName, tbl, envContext);
+ } catch (Exception e) {
Review comment:
Try to avoid catching `Exception` so that this doesn't swallow unexpected errors.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [incubator-iceberg] rdblue commented on a change in pull request
#841: Upgrade Hive to 2.3.6 for Iceberg
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #841: Upgrade Hive to 2.3.6 for Iceberg
URL: https://github.com/apache/incubator-iceberg/pull/841#discussion_r393176440
##########
File path: versions.lock
##########
@@ -37,34 +37,36 @@ com.sun.jersey:jersey-json:1.9 (5 constraints: 945f2f90)
com.sun.jersey:jersey-server:1.9 (4 constraints: ef373c01)
com.sun.jersey.contribs:jersey-guice:1.9 (4 constraints: 65529ed7)
com.sun.xml.bind:jaxb-impl:2.2.3-1 (1 constraints: 330c2404)
+com.tdunning:json:1.8 (1 constraints: 910b72db)
Review comment:
Another JSON library! Can you check whether we need it?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [incubator-iceberg] rdblue commented on a change in pull request
#841: Upgrade Hive to 2.3.6 for Iceberg
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #841: Upgrade Hive to 2.3.6 for Iceberg
URL: https://github.com/apache/incubator-iceberg/pull/841#discussion_r392335663
##########
File path: versions.lock
##########
@@ -1,31 +1,39 @@
# Run ./gradlew --write-locks to regenerate this file
-ant:ant:1.6.5 (1 constraints: bb0a41d8)
-antlr:antlr:2.7.7 (2 constraints: 36167e02)
+ant:ant:1.6.5 (2 constraints: 2d1539e1)
aopalliance:aopalliance:1.0 (1 constraints: 170a83ac)
-asm:asm:3.1 (2 constraints: 4f19c3c6)
+asm:asm:3.1 (3 constraints: 251fd7ad)
+asm:asm-commons:3.1 (1 constraints: 9c0f1f7a)
+asm:asm-tree:3.1 (1 constraints: 2307035c)
+ch.qos.logback:logback-classic:1.0.9 (3 constraints: d3258a88)
+ch.qos.logback:logback-core:1.0.9 (4 constraints: dd32435a)
+co.cask.tephra:tephra-api:0.6.0 (3 constraints: 0828ded1)
+co.cask.tephra:tephra-core:0.6.0 (2 constraints: 831cd90d)
+co.cask.tephra:tephra-hbase-compat-1.0:0.6.0 (1 constraints: 370d6920)
com.carrotsearch:hppc:0.7.2 (1 constraints: f70cda14)
com.clearspring.analytics:stream:2.7.0 (1 constraints: 1a0dd136)
com.esotericsoftware:kryo-shaded:4.0.2 (2 constraints: b71345a6)
com.esotericsoftware:minlog:1.3.0 (1 constraints: 670e7c4f)
com.fasterxml.jackson.core:jackson-annotations:2.10.2 (5 constraints: 4155160f)
com.fasterxml.jackson.core:jackson-core:2.10.2 (6 constraints: bb52b302)
-com.fasterxml.jackson.core:jackson-databind:2.10.2 (10 constraints: 8a92f4c4)
+com.fasterxml.jackson.core:jackson-databind:2.10.2 (11 constraints: 7c9eca9a)
com.fasterxml.jackson.module:jackson-module-paranamer:2.10.2 (1 constraints: 03162c16)
com.fasterxml.jackson.module:jackson-module-scala_2.11:2.10.2 (1 constraints: 7f0da251)
com.github.ben-manes.caffeine:caffeine:2.7.0 (1 constraints: 0b050a36)
+com.github.joshelser:dropwizard-metrics-hadoop-metrics2-reporter:0.1.2 (1 constraints: e90b08f3)
com.github.luben:zstd-jni:1.3.2-2 (1 constraints: 760d7c51)
com.github.stephenc.findbugs:findbugs-annotations:1.3.9-1 (1 constraints: 6d05ab40)
-com.google.code.findbugs:jsr305:3.0.2 (9 constraints: d276cf3c)
-com.google.code.gson:gson:2.2.4 (1 constraints: 8c0d3f2f)
+com.google.code.findbugs:jsr305:3.0.2 (11 constraints: 2290ea31)
Review comment:
This dependency must be excluded because it has a questionable license. The stephenc.findbugs version above is a drop-in replacement.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [incubator-iceberg] jerryshao commented on issue #841: Upgrade Hive
to 2.3.6 for Iceberg
Posted by GitBox <gi...@apache.org>.
jerryshao commented on issue #841: Upgrade Hive to 2.3.6 for Iceberg
URL: https://github.com/apache/incubator-iceberg/pull/841#issuecomment-599304008
Sure, I will revisit all the dependencies and try to remove unnecessary ones as possible as I can.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [incubator-iceberg] jerryshao commented on a change in pull request
#841: Upgrade Hive to 2.3.6 for Iceberg
Posted by GitBox <gi...@apache.org>.
jerryshao commented on a change in pull request #841: Upgrade Hive to 2.3.6 for Iceberg
URL: https://github.com/apache/incubator-iceberg/pull/841#discussion_r392767732
##########
File path: versions.lock
##########
@@ -170,24 +188,33 @@ org.apache.spark:spark-sketch_2.11:2.4.4 (2 constraints: 981bd4f5)
org.apache.spark:spark-sql_2.11:2.4.4 (1 constraints: 1e0d0037)
org.apache.spark:spark-tags_2.11:2.4.4 (8 constraints: 036fa69d)
org.apache.spark:spark-unsafe_2.11:2.4.4 (2 constraints: f11bc213)
-org.apache.thrift:libfb303:0.9.3 (3 constraints: 6725fac0)
-org.apache.thrift:libthrift:0.9.3 (5 constraints: 71415452)
+org.apache.thrift:libfb303:0.9.3 (3 constraints: 27289d07)
+org.apache.thrift:libthrift:0.9.3 (7 constraints: c75ac7de)
+org.apache.twill:twill-api:0.6.0-incubating (2 constraints: e422b4e1)
+org.apache.twill:twill-common:0.6.0-incubating (4 constraints: 3d467991)
+org.apache.twill:twill-core:0.6.0-incubating (1 constraints: d70f9d7c)
+org.apache.twill:twill-discovery-api:0.6.0-incubating (3 constraints: 25345d4c)
+org.apache.twill:twill-discovery-core:0.6.0-incubating (2 constraints: 332039f9)
+org.apache.twill:twill-zookeeper:0.6.0-incubating (3 constraints: 94341288)
org.apache.xbean:xbean-asm6-shaded:4.8 (2 constraints: 2419a30f)
org.apache.yetus:audience-annotations:0.11.0 (1 constraints: c40eb364)
-org.apache.zookeeper:zookeeper:3.4.6 (11 constraints: 18a71f48)
+org.apache.zookeeper:zookeeper:3.4.6 (12 constraints: 54b50652)
org.checkerframework:checker-qual:2.8.1 (2 constraints: 1a1a3944)
org.codehaus.jackson:jackson-core-asl:1.9.13 (11 constraints: 8091cd06)
org.codehaus.jackson:jackson-jaxrs:1.9.13 (2 constraints: 821bca9d)
org.codehaus.jackson:jackson-mapper-asl:1.9.13 (11 constraints: e18d325f)
org.codehaus.jackson:jackson-xc:1.9.13 (2 constraints: 821bca9d)
-org.codehaus.janino:commons-compiler:3.0.9 (3 constraints: 0a2837cc)
-org.codehaus.janino:janino:3.0.9 (2 constraints: 3f1c6304)
+org.codehaus.janino:commons-compiler:3.0.9 (2 constraints: a41a546f)
+org.codehaus.janino:janino:3.0.9 (1 constraints: d90e817c)
org.codehaus.jettison:jettison:1.1 (4 constraints: a84e24a9)
org.codehaus.mojo:animal-sniffer-annotations:1.17 (1 constraints: ed09d8aa)
-org.datanucleus:datanucleus-api-jdo:3.2.6 (2 constraints: 5a1d241c)
-org.datanucleus:datanucleus-core:3.2.10 (5 constraints: 4a4401e0)
-org.datanucleus:datanucleus-rdbms:3.2.9 (2 constraints: 601db41c)
+org.datanucleus:datanucleus-api-jdo:4.2.4 (2 constraints: 591df91b)
+org.datanucleus:datanucleus-core:4.1.17 (5 constraints: 584455e8)
+org.datanucleus:datanucleus-rdbms:4.1.19 (2 constraints: 911dec32)
+org.datanucleus:javax.jdo:3.2.0-m3 (1 constraints: 030ea249)
org.eclipse.jdt:core:3.1.1 (1 constraints: b40a38d8)
+org.eclipse.jetty.aggregate:jetty-all:7.6.0.v20120127 (1 constraints: 260ea577)
+org.eclipse.jetty.orbit:javax.servlet:3.0.0.v201112011016 (1 constraints: dd0e53b1)
Review comment:
Seems not necessary, they're introduced by hive-common, I will remove them.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [incubator-iceberg] rdblue commented on issue #841: Upgrade Hive to
2.3.6 for Iceberg
Posted by GitBox <gi...@apache.org>.
rdblue commented on issue #841: Upgrade Hive to 2.3.6 for Iceberg
URL: https://github.com/apache/incubator-iceberg/pull/841#issuecomment-598809775
Thanks for working on this, @jerryshao! I think the code part is close -- I'd just prefer to use the `DynMethods` builder to avoid try/catch and custom reflect code.
We also need to take a close look at all the dependency additions and remove whatever we can. This pulls in a lot of new dependencies and most of them are probably unnecessary. Those cause a lot of work to maintain our binary LICENSE and NOTICE files and can cause runtime conflicts.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [incubator-iceberg] rdblue commented on a change in pull request
#841: Upgrade Hive to 2.3.6 for Iceberg
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #841: Upgrade Hive to 2.3.6 for Iceberg
URL: https://github.com/apache/incubator-iceberg/pull/841#discussion_r393175128
##########
File path: hive/src/main/java/org/apache/iceberg/hive/HiveTableOperations.java
##########
@@ -173,9 +181,9 @@ protected void doCommit(TableMetadata base, TableMetadata metadata) {
if (base != null) {
metaClients.run(client -> {
EnvironmentContext envContext = new EnvironmentContext(
- ImmutableMap.of(StatsSetupConst.DO_NOT_UPDATE_STATS, StatsSetupConst.TRUE)
- );
- client.alter_table(database, tableName, tbl, envContext);
+ ImmutableMap.of(StatsSetupConst.DO_NOT_UPDATE_STATS, StatsSetupConst.TRUE));
Review comment:
Nit: this line didn't need to change.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [incubator-iceberg] jerryshao commented on a change in pull request
#841: Upgrade Hive to 2.3.6 for Iceberg
Posted by GitBox <gi...@apache.org>.
jerryshao commented on a change in pull request #841: Upgrade Hive to 2.3.6 for Iceberg
URL: https://github.com/apache/incubator-iceberg/pull/841#discussion_r392763469
##########
File path: hive/src/main/java/org/apache/iceberg/hive/HiveTableOperations.java
##########
@@ -175,7 +190,12 @@ protected void doCommit(TableMetadata base, TableMetadata metadata) {
EnvironmentContext envContext = new EnvironmentContext(
ImmutableMap.of(StatsSetupConst.DO_NOT_UPDATE_STATS, StatsSetupConst.TRUE)
);
- client.alter_table(database, tableName, tbl, envContext);
+
+ try {
+ alterTableMethod.invoke(client, database, tableName, tbl, envContext);
+ } catch (Exception e) {
+ throw new RuntimeMetaException(e, "Failed to alter table for %s.%s", database, tableName);
+ }
Review comment:
With this Iceberg will complain about "CyclomaticComplexity" issue, I think it is probably fine to let the existing exceptional handler to handle the exceptions.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [incubator-iceberg] jerryshao commented on a change in pull request
#841: Upgrade Hive to 2.3.6 for Iceberg
Posted by GitBox <gi...@apache.org>.
jerryshao commented on a change in pull request #841: Upgrade Hive to 2.3.6 for Iceberg
URL: https://github.com/apache/incubator-iceberg/pull/841#discussion_r393554452
##########
File path: versions.lock
##########
@@ -37,34 +37,36 @@ com.sun.jersey:jersey-json:1.9 (5 constraints: 945f2f90)
com.sun.jersey:jersey-server:1.9 (4 constraints: ef373c01)
com.sun.jersey.contribs:jersey-guice:1.9 (4 constraints: 65529ed7)
com.sun.xml.bind:jaxb-impl:2.2.3-1 (1 constraints: 330c2404)
+com.tdunning:json:1.8 (1 constraints: 910b72db)
Review comment:
Introduced by hive-common. Seems not necessary, I will remove this.
```
com.tdunning:json:1.8
\--- org.apache.hive:hive-common:2.3.6
\--- org.apache.hive:hive-serde:2.3.6
\--- org.apache.hive:hive-metastore:2.3.6
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [incubator-iceberg] jerryshao commented on a change in pull request
#841: Upgrade Hive to 2.3.6 for Iceberg
Posted by GitBox <gi...@apache.org>.
jerryshao commented on a change in pull request #841: Upgrade Hive to 2.3.6 for Iceberg
URL: https://github.com/apache/incubator-iceberg/pull/841#discussion_r393554719
##########
File path: versions.lock
##########
@@ -73,44 +75,45 @@ javax.annotation:javax.annotation-api:1.3.2 (3 constraints: 55341c48)
javax.inject:javax.inject:1 (4 constraints: 852d0c1a)
javax.jdo:jdo-api:3.0.1 (2 constraints: 4c1dcc1a)
javax.servlet:javax.servlet-api:3.1.0 (1 constraints: 150dc436)
-javax.servlet:servlet-api:2.5 (7 constraints: c87e5aa1)
+javax.servlet:jsp-api:2.0 (1 constraints: 0b0aa0a7)
+javax.servlet:servlet-api:2.5 (9 constraints: f991a6d2)
javax.servlet.jsp:jsp-api:2.1 (1 constraints: 290d5a14)
javax.transaction:jta:1.1 (1 constraints: 9f07d96b)
+javax.transaction:transaction-api:1.1 (1 constraints: 0a0b64c9)
Review comment:
Introduced by datanucleus, seems not necessary, I will remove it.
```
\--- org.datanucleus:javax.jdo:3.2.0-m3
\--- org.apache.hive:hive-metastore:2.3.6
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [incubator-iceberg] rdblue commented on a change in pull request
#841: Upgrade Hive to 2.3.6 for Iceberg
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #841: Upgrade Hive to 2.3.6 for Iceberg
URL: https://github.com/apache/incubator-iceberg/pull/841#discussion_r393802858
##########
File path: versions.lock
##########
@@ -170,24 +188,33 @@ org.apache.spark:spark-sketch_2.11:2.4.4 (2 constraints: 981bd4f5)
org.apache.spark:spark-sql_2.11:2.4.4 (1 constraints: 1e0d0037)
org.apache.spark:spark-tags_2.11:2.4.4 (8 constraints: 036fa69d)
org.apache.spark:spark-unsafe_2.11:2.4.4 (2 constraints: f11bc213)
-org.apache.thrift:libfb303:0.9.3 (3 constraints: 6725fac0)
-org.apache.thrift:libthrift:0.9.3 (5 constraints: 71415452)
+org.apache.thrift:libfb303:0.9.3 (3 constraints: 27289d07)
+org.apache.thrift:libthrift:0.9.3 (7 constraints: c75ac7de)
+org.apache.twill:twill-api:0.6.0-incubating (2 constraints: e422b4e1)
+org.apache.twill:twill-common:0.6.0-incubating (4 constraints: 3d467991)
+org.apache.twill:twill-core:0.6.0-incubating (1 constraints: d70f9d7c)
+org.apache.twill:twill-discovery-api:0.6.0-incubating (3 constraints: 25345d4c)
+org.apache.twill:twill-discovery-core:0.6.0-incubating (2 constraints: 332039f9)
+org.apache.twill:twill-zookeeper:0.6.0-incubating (3 constraints: 94341288)
org.apache.xbean:xbean-asm6-shaded:4.8 (2 constraints: 2419a30f)
org.apache.yetus:audience-annotations:0.11.0 (1 constraints: c40eb364)
-org.apache.zookeeper:zookeeper:3.4.6 (11 constraints: 18a71f48)
+org.apache.zookeeper:zookeeper:3.4.6 (12 constraints: 54b50652)
org.checkerframework:checker-qual:2.8.1 (2 constraints: 1a1a3944)
org.codehaus.jackson:jackson-core-asl:1.9.13 (11 constraints: 8091cd06)
org.codehaus.jackson:jackson-jaxrs:1.9.13 (2 constraints: 821bca9d)
org.codehaus.jackson:jackson-mapper-asl:1.9.13 (11 constraints: e18d325f)
org.codehaus.jackson:jackson-xc:1.9.13 (2 constraints: 821bca9d)
-org.codehaus.janino:commons-compiler:3.0.9 (3 constraints: 0a2837cc)
-org.codehaus.janino:janino:3.0.9 (2 constraints: 3f1c6304)
+org.codehaus.janino:commons-compiler:3.0.9 (2 constraints: a41a546f)
+org.codehaus.janino:janino:3.0.9 (1 constraints: d90e817c)
org.codehaus.jettison:jettison:1.1 (4 constraints: a84e24a9)
org.codehaus.mojo:animal-sniffer-annotations:1.17 (1 constraints: ed09d8aa)
-org.datanucleus:datanucleus-api-jdo:3.2.6 (2 constraints: 5a1d241c)
-org.datanucleus:datanucleus-core:3.2.10 (5 constraints: 4a4401e0)
-org.datanucleus:datanucleus-rdbms:3.2.9 (2 constraints: 601db41c)
+org.datanucleus:datanucleus-api-jdo:4.2.4 (2 constraints: 591df91b)
+org.datanucleus:datanucleus-core:4.1.17 (5 constraints: 584455e8)
+org.datanucleus:datanucleus-rdbms:4.1.19 (2 constraints: 911dec32)
Review comment:
I see, this is because this is for the `iceberg-spark` module and not `iceberg-spark-runtime`. I was thinking this was for the runtime, which excludes the dependency. But you're right that we need to run unit tests with these versions because we are pulling in Hive 1.2.1-spark2. Thanks!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [incubator-iceberg] rdblue commented on a change in pull request
#841: Upgrade Hive to 2.3.6 for Iceberg
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #841: Upgrade Hive to 2.3.6 for Iceberg
URL: https://github.com/apache/incubator-iceberg/pull/841#discussion_r392337866
##########
File path: versions.lock
##########
@@ -170,24 +188,33 @@ org.apache.spark:spark-sketch_2.11:2.4.4 (2 constraints: 981bd4f5)
org.apache.spark:spark-sql_2.11:2.4.4 (1 constraints: 1e0d0037)
org.apache.spark:spark-tags_2.11:2.4.4 (8 constraints: 036fa69d)
org.apache.spark:spark-unsafe_2.11:2.4.4 (2 constraints: f11bc213)
-org.apache.thrift:libfb303:0.9.3 (3 constraints: 6725fac0)
-org.apache.thrift:libthrift:0.9.3 (5 constraints: 71415452)
+org.apache.thrift:libfb303:0.9.3 (3 constraints: 27289d07)
+org.apache.thrift:libthrift:0.9.3 (7 constraints: c75ac7de)
+org.apache.twill:twill-api:0.6.0-incubating (2 constraints: e422b4e1)
+org.apache.twill:twill-common:0.6.0-incubating (4 constraints: 3d467991)
+org.apache.twill:twill-core:0.6.0-incubating (1 constraints: d70f9d7c)
+org.apache.twill:twill-discovery-api:0.6.0-incubating (3 constraints: 25345d4c)
+org.apache.twill:twill-discovery-core:0.6.0-incubating (2 constraints: 332039f9)
+org.apache.twill:twill-zookeeper:0.6.0-incubating (3 constraints: 94341288)
org.apache.xbean:xbean-asm6-shaded:4.8 (2 constraints: 2419a30f)
org.apache.yetus:audience-annotations:0.11.0 (1 constraints: c40eb364)
-org.apache.zookeeper:zookeeper:3.4.6 (11 constraints: 18a71f48)
+org.apache.zookeeper:zookeeper:3.4.6 (12 constraints: 54b50652)
org.checkerframework:checker-qual:2.8.1 (2 constraints: 1a1a3944)
org.codehaus.jackson:jackson-core-asl:1.9.13 (11 constraints: 8091cd06)
org.codehaus.jackson:jackson-jaxrs:1.9.13 (2 constraints: 821bca9d)
org.codehaus.jackson:jackson-mapper-asl:1.9.13 (11 constraints: e18d325f)
org.codehaus.jackson:jackson-xc:1.9.13 (2 constraints: 821bca9d)
-org.codehaus.janino:commons-compiler:3.0.9 (3 constraints: 0a2837cc)
-org.codehaus.janino:janino:3.0.9 (2 constraints: 3f1c6304)
+org.codehaus.janino:commons-compiler:3.0.9 (2 constraints: a41a546f)
+org.codehaus.janino:janino:3.0.9 (1 constraints: d90e817c)
org.codehaus.jettison:jettison:1.1 (4 constraints: a84e24a9)
org.codehaus.mojo:animal-sniffer-annotations:1.17 (1 constraints: ed09d8aa)
-org.datanucleus:datanucleus-api-jdo:3.2.6 (2 constraints: 5a1d241c)
-org.datanucleus:datanucleus-core:3.2.10 (5 constraints: 4a4401e0)
-org.datanucleus:datanucleus-rdbms:3.2.9 (2 constraints: 601db41c)
+org.datanucleus:datanucleus-api-jdo:4.2.4 (2 constraints: 591df91b)
+org.datanucleus:datanucleus-core:4.1.17 (5 constraints: 584455e8)
+org.datanucleus:datanucleus-rdbms:4.1.19 (2 constraints: 911dec32)
Review comment:
These versions don't match the versions that were specified by `force` in `build.gradle`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [incubator-iceberg] rdblue commented on a change in pull request
#841: Upgrade Hive to 2.3.6 for Iceberg
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #841: Upgrade Hive to 2.3.6 for Iceberg
URL: https://github.com/apache/incubator-iceberg/pull/841#discussion_r392336039
##########
File path: versions.lock
##########
@@ -1,31 +1,39 @@
# Run ./gradlew --write-locks to regenerate this file
-ant:ant:1.6.5 (1 constraints: bb0a41d8)
-antlr:antlr:2.7.7 (2 constraints: 36167e02)
+ant:ant:1.6.5 (2 constraints: 2d1539e1)
aopalliance:aopalliance:1.0 (1 constraints: 170a83ac)
-asm:asm:3.1 (2 constraints: 4f19c3c6)
+asm:asm:3.1 (3 constraints: 251fd7ad)
+asm:asm-commons:3.1 (1 constraints: 9c0f1f7a)
+asm:asm-tree:3.1 (1 constraints: 2307035c)
Review comment:
Can we remove asm? This is another common cause of runtime conflicts.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [incubator-iceberg] jerryshao commented on a change in pull request
#841: Upgrade Hive to 2.3.6 for Iceberg
Posted by GitBox <gi...@apache.org>.
jerryshao commented on a change in pull request #841: Upgrade Hive to 2.3.6 for Iceberg
URL: https://github.com/apache/incubator-iceberg/pull/841#discussion_r392766547
##########
File path: versions.lock
##########
@@ -1,31 +1,39 @@
# Run ./gradlew --write-locks to regenerate this file
-ant:ant:1.6.5 (1 constraints: bb0a41d8)
-antlr:antlr:2.7.7 (2 constraints: 36167e02)
+ant:ant:1.6.5 (2 constraints: 2d1539e1)
aopalliance:aopalliance:1.0 (1 constraints: 170a83ac)
-asm:asm:3.1 (2 constraints: 4f19c3c6)
+asm:asm:3.1 (3 constraints: 251fd7ad)
+asm:asm-commons:3.1 (1 constraints: 9c0f1f7a)
+asm:asm-tree:3.1 (1 constraints: 2307035c)
+ch.qos.logback:logback-classic:1.0.9 (3 constraints: d3258a88)
+ch.qos.logback:logback-core:1.0.9 (4 constraints: dd32435a)
+co.cask.tephra:tephra-api:0.6.0 (3 constraints: 0828ded1)
+co.cask.tephra:tephra-core:0.6.0 (2 constraints: 831cd90d)
+co.cask.tephra:tephra-hbase-compat-1.0:0.6.0 (1 constraints: 370d6920)
com.carrotsearch:hppc:0.7.2 (1 constraints: f70cda14)
com.clearspring.analytics:stream:2.7.0 (1 constraints: 1a0dd136)
com.esotericsoftware:kryo-shaded:4.0.2 (2 constraints: b71345a6)
com.esotericsoftware:minlog:1.3.0 (1 constraints: 670e7c4f)
com.fasterxml.jackson.core:jackson-annotations:2.10.2 (5 constraints: 4155160f)
com.fasterxml.jackson.core:jackson-core:2.10.2 (6 constraints: bb52b302)
-com.fasterxml.jackson.core:jackson-databind:2.10.2 (10 constraints: 8a92f4c4)
+com.fasterxml.jackson.core:jackson-databind:2.10.2 (11 constraints: 7c9eca9a)
com.fasterxml.jackson.module:jackson-module-paranamer:2.10.2 (1 constraints: 03162c16)
com.fasterxml.jackson.module:jackson-module-scala_2.11:2.10.2 (1 constraints: 7f0da251)
com.github.ben-manes.caffeine:caffeine:2.7.0 (1 constraints: 0b050a36)
+com.github.joshelser:dropwizard-metrics-hadoop-metrics2-reporter:0.1.2 (1 constraints: e90b08f3)
com.github.luben:zstd-jni:1.3.2-2 (1 constraints: 760d7c51)
com.github.stephenc.findbugs:findbugs-annotations:1.3.9-1 (1 constraints: 6d05ab40)
-com.google.code.findbugs:jsr305:3.0.2 (9 constraints: d276cf3c)
-com.google.code.gson:gson:2.2.4 (1 constraints: 8c0d3f2f)
+com.google.code.findbugs:jsr305:3.0.2 (11 constraints: 2290ea31)
+com.google.code.gson:gson:2.2.4 (5 constraints: 3b40b359)
Review comment:
This above two has been existed, and is depended by several jars, should we remove them all?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [incubator-iceberg] rdblue commented on a change in pull request
#841: Upgrade Hive to 2.3.6 for Iceberg
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #841: Upgrade Hive to 2.3.6 for Iceberg
URL: https://github.com/apache/incubator-iceberg/pull/841#discussion_r393175895
##########
File path: versions.lock
##########
@@ -73,44 +75,45 @@ javax.annotation:javax.annotation-api:1.3.2 (3 constraints: 55341c48)
javax.inject:javax.inject:1 (4 constraints: 852d0c1a)
javax.jdo:jdo-api:3.0.1 (2 constraints: 4c1dcc1a)
javax.servlet:javax.servlet-api:3.1.0 (1 constraints: 150dc436)
-javax.servlet:servlet-api:2.5 (7 constraints: c87e5aa1)
+javax.servlet:jsp-api:2.0 (1 constraints: 0b0aa0a7)
+javax.servlet:servlet-api:2.5 (9 constraints: f991a6d2)
javax.servlet.jsp:jsp-api:2.1 (1 constraints: 290d5a14)
javax.transaction:jta:1.1 (1 constraints: 9f07d96b)
+javax.transaction:transaction-api:1.1 (1 constraints: 0a0b64c9)
Review comment:
Is this needed?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [incubator-iceberg] rdblue commented on a change in pull request
#841: Upgrade Hive to 2.3.6 for Iceberg
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #841: Upgrade Hive to 2.3.6 for Iceberg
URL: https://github.com/apache/incubator-iceberg/pull/841#discussion_r392336626
##########
File path: versions.lock
##########
@@ -37,90 +45,98 @@ com.sun.jersey:jersey-json:1.9 (5 constraints: 945f2f90)
com.sun.jersey:jersey-server:1.9 (4 constraints: ef373c01)
com.sun.jersey.contribs:jersey-guice:1.9 (4 constraints: 65529ed7)
com.sun.xml.bind:jaxb-impl:2.2.3-1 (1 constraints: 330c2404)
+com.tdunning:json:1.8 (1 constraints: 910b72db)
com.thoughtworks.paranamer:paranamer:2.8 (3 constraints: 742d4cb1)
com.twitter:chill-java:0.9.3 (2 constraints: a716716f)
com.twitter:chill_2.11:0.9.3 (2 constraints: 121b92c3)
-com.twitter:parquet-hadoop-bundle:1.6.0 (3 constraints: 7c262424)
+com.twitter:parquet-hadoop-bundle:1.6.0 (2 constraints: 061b4d93)
com.univocity:univocity-parsers:2.7.3 (1 constraints: c40ccb27)
+com.zaxxer:HikariCP:2.5.1 (1 constraints: 390d7120)
commons-beanutils:commons-beanutils:1.7.0 (1 constraints: da0e635f)
commons-beanutils:commons-beanutils-core:1.8.0 (1 constraints: 1d134124)
-commons-cli:commons-cli:1.2 (8 constraints: 9467c282)
-commons-codec:commons-codec:1.10 (16 constraints: 42d1423c)
-commons-collections:commons-collections:3.2.2 (4 constraints: 42476934)
+commons-cli:commons-cli:1.2 (9 constraints: f874366c)
+commons-codec:commons-codec:1.10 (17 constraints: a8de3870)
+commons-collections:commons-collections:3.2.2 (3 constraints: e73a8e36)
commons-configuration:commons-configuration:1.6 (1 constraints: 2d0d5c14)
commons-daemon:commons-daemon:1.0.13 (1 constraints: d50c811c)
commons-dbcp:commons-dbcp:1.4 (3 constraints: 9029e0e4)
commons-digester:commons-digester:1.8 (1 constraints: bf1228fe)
-commons-el:commons-el:1.0 (1 constraints: fb077074)
+commons-el:commons-el:1.0 (2 constraints: ad11e7f0)
commons-httpclient:commons-httpclient:3.1 (4 constraints: e52cc77f)
commons-io:commons-io:2.4 (6 constraints: 4a568049)
-commons-lang:commons-lang:2.6 (19 constraints: 5f0d34d8)
-commons-logging:commons-logging:1.2 (27 constraints: f2ab390b)
+commons-lang:commons-lang:2.6 (20 constraints: 401f63f3)
+commons-logging:commons-logging:1.2 (20 constraints: 424b646e)
commons-net:commons-net:3.1 (3 constraints: 3d222e61)
commons-pool:commons-pool:1.6 (4 constraints: e336ab5e)
dk.brics.automaton:automaton:1.11-8 (1 constraints: 92088a8d)
hsqldb:hsqldb:1.8.0.10 (1 constraints: f008499f)
io.airlift:aircompressor:0.15 (1 constraints: 0e0aa4b2)
-io.dropwizard.metrics:metrics-core:3.1.5 (6 constraints: 865ea0ba)
+io.dropwizard.metrics:metrics-core:3.1.5 (9 constraints: be90b1ee)
io.dropwizard.metrics:metrics-graphite:3.1.5 (1 constraints: 1a0dc936)
-io.dropwizard.metrics:metrics-json:3.1.5 (1 constraints: 1a0dc936)
-io.dropwizard.metrics:metrics-jvm:3.1.5 (1 constraints: 1a0dc936)
+io.dropwizard.metrics:metrics-json:3.1.5 (2 constraints: 03195c12)
+io.dropwizard.metrics:metrics-jvm:3.1.5 (2 constraints: 03195c12)
io.netty:netty:3.9.9.Final (9 constraints: 9eb0396d)
io.netty:netty-all:4.1.17.Final (3 constraints: d2312526)
io.netty:netty-buffer:4.1.27.Final (1 constraints: 4a0fee77)
-javax.activation:activation:1.1.1 (1 constraints: 140dbb36)
+it.unimi.dsi:fastutil:6.5.6 (1 constraints: 910b3ce5)
Review comment:
Is this needed?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [incubator-iceberg] jerryshao commented on a change in pull request
#841: Upgrade Hive to 2.3.6 for Iceberg
Posted by GitBox <gi...@apache.org>.
jerryshao commented on a change in pull request #841: Upgrade Hive to 2.3.6 for Iceberg
URL: https://github.com/apache/incubator-iceberg/pull/841#discussion_r393556681
##########
File path: versions.lock
##########
@@ -37,34 +37,36 @@ com.sun.jersey:jersey-json:1.9 (5 constraints: 945f2f90)
com.sun.jersey:jersey-server:1.9 (4 constraints: ef373c01)
com.sun.jersey.contribs:jersey-guice:1.9 (4 constraints: 65529ed7)
com.sun.xml.bind:jaxb-impl:2.2.3-1 (1 constraints: 330c2404)
+com.tdunning:json:1.8 (1 constraints: 910b72db)
com.thoughtworks.paranamer:paranamer:2.8 (3 constraints: 742d4cb1)
com.twitter:chill-java:0.9.3 (2 constraints: a716716f)
com.twitter:chill_2.11:0.9.3 (2 constraints: 121b92c3)
-com.twitter:parquet-hadoop-bundle:1.6.0 (3 constraints: 7c262424)
+com.twitter:parquet-hadoop-bundle:1.6.0 (2 constraints: 061b4d93)
com.univocity:univocity-parsers:2.7.3 (1 constraints: c40ccb27)
+com.zaxxer:HikariCP:2.5.1 (1 constraints: 390d7120)
Review comment:
Let me take a try if it still can be worked after removing this dependencies.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [incubator-iceberg] jerryshao commented on a change in pull request
#841: Upgrade Hive to 2.3.6 for Iceberg
Posted by GitBox <gi...@apache.org>.
jerryshao commented on a change in pull request #841: Upgrade Hive to 2.3.6 for Iceberg
URL: https://github.com/apache/incubator-iceberg/pull/841#discussion_r392754613
##########
File path: versions.lock
##########
@@ -170,24 +188,33 @@ org.apache.spark:spark-sketch_2.11:2.4.4 (2 constraints: 981bd4f5)
org.apache.spark:spark-sql_2.11:2.4.4 (1 constraints: 1e0d0037)
org.apache.spark:spark-tags_2.11:2.4.4 (8 constraints: 036fa69d)
org.apache.spark:spark-unsafe_2.11:2.4.4 (2 constraints: f11bc213)
-org.apache.thrift:libfb303:0.9.3 (3 constraints: 6725fac0)
-org.apache.thrift:libthrift:0.9.3 (5 constraints: 71415452)
+org.apache.thrift:libfb303:0.9.3 (3 constraints: 27289d07)
+org.apache.thrift:libthrift:0.9.3 (7 constraints: c75ac7de)
+org.apache.twill:twill-api:0.6.0-incubating (2 constraints: e422b4e1)
+org.apache.twill:twill-common:0.6.0-incubating (4 constraints: 3d467991)
+org.apache.twill:twill-core:0.6.0-incubating (1 constraints: d70f9d7c)
+org.apache.twill:twill-discovery-api:0.6.0-incubating (3 constraints: 25345d4c)
+org.apache.twill:twill-discovery-core:0.6.0-incubating (2 constraints: 332039f9)
+org.apache.twill:twill-zookeeper:0.6.0-incubating (3 constraints: 94341288)
org.apache.xbean:xbean-asm6-shaded:4.8 (2 constraints: 2419a30f)
org.apache.yetus:audience-annotations:0.11.0 (1 constraints: c40eb364)
-org.apache.zookeeper:zookeeper:3.4.6 (11 constraints: 18a71f48)
+org.apache.zookeeper:zookeeper:3.4.6 (12 constraints: 54b50652)
org.checkerframework:checker-qual:2.8.1 (2 constraints: 1a1a3944)
org.codehaus.jackson:jackson-core-asl:1.9.13 (11 constraints: 8091cd06)
org.codehaus.jackson:jackson-jaxrs:1.9.13 (2 constraints: 821bca9d)
org.codehaus.jackson:jackson-mapper-asl:1.9.13 (11 constraints: e18d325f)
org.codehaus.jackson:jackson-xc:1.9.13 (2 constraints: 821bca9d)
-org.codehaus.janino:commons-compiler:3.0.9 (3 constraints: 0a2837cc)
-org.codehaus.janino:janino:3.0.9 (2 constraints: 3f1c6304)
+org.codehaus.janino:commons-compiler:3.0.9 (2 constraints: a41a546f)
+org.codehaus.janino:janino:3.0.9 (1 constraints: d90e817c)
org.codehaus.jettison:jettison:1.1 (4 constraints: a84e24a9)
org.codehaus.mojo:animal-sniffer-annotations:1.17 (1 constraints: ed09d8aa)
-org.datanucleus:datanucleus-api-jdo:3.2.6 (2 constraints: 5a1d241c)
-org.datanucleus:datanucleus-core:3.2.10 (5 constraints: 4a4401e0)
-org.datanucleus:datanucleus-rdbms:3.2.9 (2 constraints: 601db41c)
+org.datanucleus:datanucleus-api-jdo:4.2.4 (2 constraints: 591df91b)
+org.datanucleus:datanucleus-core:4.1.17 (5 constraints: 584455e8)
+org.datanucleus:datanucleus-rdbms:4.1.19 (2 constraints: 911dec32)
Review comment:
These versions are introduced by Hive 2.3.6 metastore in iceberg-hive module. The versions `force`d in iceberg-spark module is introduced transiently by Hive 1.2.1. The problem is that Hive 2.3.6 can only work with datanucleus 4.x, and Hive 1.2.1 (spark) can only work with datanucleus 3.x, that's why I enforced the version to 3.x in spark module.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [incubator-iceberg] rdblue commented on a change in pull request
#841: Upgrade Hive to 2.3.6 for Iceberg
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #841: Upgrade Hive to 2.3.6 for Iceberg
URL: https://github.com/apache/incubator-iceberg/pull/841#discussion_r392337355
##########
File path: versions.lock
##########
@@ -156,6 +173,7 @@ org.apache.parquet:parquet-common:1.11.0 (2 constraints: 4c1e7785)
org.apache.parquet:parquet-encoding:1.11.0 (1 constraints: ca0efb64)
org.apache.parquet:parquet-format-structures:1.11.0 (3 constraints: 6e2bc836)
org.apache.parquet:parquet-hadoop:1.11.0 (2 constraints: de1ac7b3)
+org.apache.parquet:parquet-hadoop-bundle:1.8.1 (1 constraints: 7a0bc7e4)
Review comment:
This is the wrong version of Parquet. Can we remove this since we only need the metastore components?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [incubator-iceberg] jerryshao commented on a change in pull request
#841: Upgrade Hive to 2.3.6 for Iceberg
Posted by GitBox <gi...@apache.org>.
jerryshao commented on a change in pull request #841: Upgrade Hive to 2.3.6 for Iceberg
URL: https://github.com/apache/incubator-iceberg/pull/841#discussion_r392023600
##########
File path: hive/src/main/java/org/apache/iceberg/hive/HiveTableOperations.java
##########
@@ -83,6 +86,18 @@ protected HiveTableOperations(Configuration conf, HiveClientPool metaClients, St
this.tableName = table;
this.lockAcquireTimeout =
conf.getLong(HIVE_ACQUIRE_LOCK_STATE_TIMEOUT_MS, HIVE_ACQUIRE_LOCK_STATE_TIMEOUT_MS_DEFAULT);
+ try {
+ this.alterTableMethod = HiveMetaStoreClient.class.getMethod("alter_table_with_environmentContext",
+ String.class, String.class, Table.class, EnvironmentContext.class);
+ } catch (NoSuchMethodException e) {
+ try {
+ this.alterTableMethod = HiveMetaStoreClient.class.getMethod("alter_table",
+ String.class, String.class, Table.class, EnvironmentContext.class);
+ } catch (NoSuchMethodException inner) {
+ throw new RuntimeMetaException(inner, "Fail to find method alter_table_with_environmentContext and " +
+ "alter_table with right signature");
+ }
Review comment:
Use reflection to handle the API difference between Hive 1.2.1 and 2.3.6.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [incubator-iceberg] rdblue commented on a change in pull request
#841: Upgrade Hive to 2.3.6 for Iceberg
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #841: Upgrade Hive to 2.3.6 for Iceberg
URL: https://github.com/apache/incubator-iceberg/pull/841#discussion_r393350269
##########
File path: versions.lock
##########
@@ -37,34 +37,36 @@ com.sun.jersey:jersey-json:1.9 (5 constraints: 945f2f90)
com.sun.jersey:jersey-server:1.9 (4 constraints: ef373c01)
com.sun.jersey.contribs:jersey-guice:1.9 (4 constraints: 65529ed7)
com.sun.xml.bind:jaxb-impl:2.2.3-1 (1 constraints: 330c2404)
+com.tdunning:json:1.8 (1 constraints: 910b72db)
com.thoughtworks.paranamer:paranamer:2.8 (3 constraints: 742d4cb1)
com.twitter:chill-java:0.9.3 (2 constraints: a716716f)
com.twitter:chill_2.11:0.9.3 (2 constraints: 121b92c3)
-com.twitter:parquet-hadoop-bundle:1.6.0 (3 constraints: 7c262424)
+com.twitter:parquet-hadoop-bundle:1.6.0 (2 constraints: 061b4d93)
com.univocity:univocity-parsers:2.7.3 (1 constraints: c40ccb27)
+com.zaxxer:HikariCP:2.5.1 (1 constraints: 390d7120)
Review comment:
Looks like this is a JDBC connection pool. Since we are using the thrift API, we probably don't need it in the Spark runtime module, but it's probably necessary for running tests that use a built-in metastore.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [incubator-iceberg] rdblue commented on a change in pull request
#841: Upgrade Hive to 2.3.6 for Iceberg
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #841: Upgrade Hive to 2.3.6 for Iceberg
URL: https://github.com/apache/incubator-iceberg/pull/841#discussion_r393174186
##########
File path: versions.lock
##########
@@ -170,24 +188,33 @@ org.apache.spark:spark-sketch_2.11:2.4.4 (2 constraints: 981bd4f5)
org.apache.spark:spark-sql_2.11:2.4.4 (1 constraints: 1e0d0037)
org.apache.spark:spark-tags_2.11:2.4.4 (8 constraints: 036fa69d)
org.apache.spark:spark-unsafe_2.11:2.4.4 (2 constraints: f11bc213)
-org.apache.thrift:libfb303:0.9.3 (3 constraints: 6725fac0)
-org.apache.thrift:libthrift:0.9.3 (5 constraints: 71415452)
+org.apache.thrift:libfb303:0.9.3 (3 constraints: 27289d07)
+org.apache.thrift:libthrift:0.9.3 (7 constraints: c75ac7de)
+org.apache.twill:twill-api:0.6.0-incubating (2 constraints: e422b4e1)
+org.apache.twill:twill-common:0.6.0-incubating (4 constraints: 3d467991)
+org.apache.twill:twill-core:0.6.0-incubating (1 constraints: d70f9d7c)
+org.apache.twill:twill-discovery-api:0.6.0-incubating (3 constraints: 25345d4c)
+org.apache.twill:twill-discovery-core:0.6.0-incubating (2 constraints: 332039f9)
+org.apache.twill:twill-zookeeper:0.6.0-incubating (3 constraints: 94341288)
org.apache.xbean:xbean-asm6-shaded:4.8 (2 constraints: 2419a30f)
org.apache.yetus:audience-annotations:0.11.0 (1 constraints: c40eb364)
-org.apache.zookeeper:zookeeper:3.4.6 (11 constraints: 18a71f48)
+org.apache.zookeeper:zookeeper:3.4.6 (12 constraints: 54b50652)
org.checkerframework:checker-qual:2.8.1 (2 constraints: 1a1a3944)
org.codehaus.jackson:jackson-core-asl:1.9.13 (11 constraints: 8091cd06)
org.codehaus.jackson:jackson-jaxrs:1.9.13 (2 constraints: 821bca9d)
org.codehaus.jackson:jackson-mapper-asl:1.9.13 (11 constraints: e18d325f)
org.codehaus.jackson:jackson-xc:1.9.13 (2 constraints: 821bca9d)
-org.codehaus.janino:commons-compiler:3.0.9 (3 constraints: 0a2837cc)
-org.codehaus.janino:janino:3.0.9 (2 constraints: 3f1c6304)
+org.codehaus.janino:commons-compiler:3.0.9 (2 constraints: a41a546f)
+org.codehaus.janino:janino:3.0.9 (1 constraints: d90e817c)
org.codehaus.jettison:jettison:1.1 (4 constraints: a84e24a9)
org.codehaus.mojo:animal-sniffer-annotations:1.17 (1 constraints: ed09d8aa)
-org.datanucleus:datanucleus-api-jdo:3.2.6 (2 constraints: 5a1d241c)
-org.datanucleus:datanucleus-core:3.2.10 (5 constraints: 4a4401e0)
-org.datanucleus:datanucleus-rdbms:3.2.9 (2 constraints: 601db41c)
+org.datanucleus:datanucleus-api-jdo:4.2.4 (2 constraints: 591df91b)
+org.datanucleus:datanucleus-core:4.1.17 (5 constraints: 584455e8)
+org.datanucleus:datanucleus-rdbms:4.1.19 (2 constraints: 911dec32)
Review comment:
Thanks for the clarification. Why do we need to force the versions if we are relying on the version of Hive built into Spark? Shouldn't we just exclude all of Hive and its transitive dependencies?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [incubator-iceberg] rdblue commented on a change in pull request
#841: Upgrade Hive to 2.3.6 for Iceberg
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #841: Upgrade Hive to 2.3.6 for Iceberg
URL: https://github.com/apache/incubator-iceberg/pull/841#discussion_r393801741
##########
File path: versions.lock
##########
@@ -1,31 +1,39 @@
# Run ./gradlew --write-locks to regenerate this file
-ant:ant:1.6.5 (1 constraints: bb0a41d8)
-antlr:antlr:2.7.7 (2 constraints: 36167e02)
+ant:ant:1.6.5 (2 constraints: 2d1539e1)
aopalliance:aopalliance:1.0 (1 constraints: 170a83ac)
-asm:asm:3.1 (2 constraints: 4f19c3c6)
+asm:asm:3.1 (3 constraints: 251fd7ad)
+asm:asm-commons:3.1 (1 constraints: 9c0f1f7a)
+asm:asm-tree:3.1 (1 constraints: 2307035c)
+ch.qos.logback:logback-classic:1.0.9 (3 constraints: d3258a88)
+ch.qos.logback:logback-core:1.0.9 (4 constraints: dd32435a)
+co.cask.tephra:tephra-api:0.6.0 (3 constraints: 0828ded1)
+co.cask.tephra:tephra-core:0.6.0 (2 constraints: 831cd90d)
+co.cask.tephra:tephra-hbase-compat-1.0:0.6.0 (1 constraints: 370d6920)
com.carrotsearch:hppc:0.7.2 (1 constraints: f70cda14)
com.clearspring.analytics:stream:2.7.0 (1 constraints: 1a0dd136)
com.esotericsoftware:kryo-shaded:4.0.2 (2 constraints: b71345a6)
com.esotericsoftware:minlog:1.3.0 (1 constraints: 670e7c4f)
com.fasterxml.jackson.core:jackson-annotations:2.10.2 (5 constraints: 4155160f)
com.fasterxml.jackson.core:jackson-core:2.10.2 (6 constraints: bb52b302)
-com.fasterxml.jackson.core:jackson-databind:2.10.2 (10 constraints: 8a92f4c4)
+com.fasterxml.jackson.core:jackson-databind:2.10.2 (11 constraints: 7c9eca9a)
com.fasterxml.jackson.module:jackson-module-paranamer:2.10.2 (1 constraints: 03162c16)
com.fasterxml.jackson.module:jackson-module-scala_2.11:2.10.2 (1 constraints: 7f0da251)
com.github.ben-manes.caffeine:caffeine:2.7.0 (1 constraints: 0b050a36)
+com.github.joshelser:dropwizard-metrics-hadoop-metrics2-reporter:0.1.2 (1 constraints: e90b08f3)
com.github.luben:zstd-jni:1.3.2-2 (1 constraints: 760d7c51)
com.github.stephenc.findbugs:findbugs-annotations:1.3.9-1 (1 constraints: 6d05ab40)
-com.google.code.findbugs:jsr305:3.0.2 (9 constraints: d276cf3c)
-com.google.code.gson:gson:2.2.4 (1 constraints: 8c0d3f2f)
+com.google.code.findbugs:jsr305:3.0.2 (11 constraints: 2290ea31)
+com.google.code.gson:gson:2.2.4 (5 constraints: 3b40b359)
Review comment:
If it was present before, it should be okay.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [incubator-iceberg] rdblue commented on a change in pull request
#841: Upgrade Hive to 2.3.6 for Iceberg
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #841: Upgrade Hive to 2.3.6 for Iceberg
URL: https://github.com/apache/incubator-iceberg/pull/841#discussion_r392335102
##########
File path: versions.lock
##########
@@ -1,31 +1,39 @@
# Run ./gradlew --write-locks to regenerate this file
-ant:ant:1.6.5 (1 constraints: bb0a41d8)
-antlr:antlr:2.7.7 (2 constraints: 36167e02)
+ant:ant:1.6.5 (2 constraints: 2d1539e1)
aopalliance:aopalliance:1.0 (1 constraints: 170a83ac)
-asm:asm:3.1 (2 constraints: 4f19c3c6)
+asm:asm:3.1 (3 constraints: 251fd7ad)
+asm:asm-commons:3.1 (1 constraints: 9c0f1f7a)
+asm:asm-tree:3.1 (1 constraints: 2307035c)
+ch.qos.logback:logback-classic:1.0.9 (3 constraints: d3258a88)
+ch.qos.logback:logback-core:1.0.9 (4 constraints: dd32435a)
+co.cask.tephra:tephra-api:0.6.0 (3 constraints: 0828ded1)
+co.cask.tephra:tephra-core:0.6.0 (2 constraints: 831cd90d)
+co.cask.tephra:tephra-hbase-compat-1.0:0.6.0 (1 constraints: 370d6920)
Review comment:
Do we need this HBase dependency? What is tephra used for?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [incubator-iceberg] rdblue commented on a change in pull request
#841: Upgrade Hive to 2.3.6 for Iceberg
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #841: Upgrade Hive to 2.3.6 for Iceberg
URL: https://github.com/apache/incubator-iceberg/pull/841#discussion_r392333011
##########
File path: hive/src/main/java/org/apache/iceberg/hive/HiveTableOperations.java
##########
@@ -83,6 +86,18 @@ protected HiveTableOperations(Configuration conf, HiveClientPool metaClients, St
this.tableName = table;
this.lockAcquireTimeout =
conf.getLong(HIVE_ACQUIRE_LOCK_STATE_TIMEOUT_MS, HIVE_ACQUIRE_LOCK_STATE_TIMEOUT_MS_DEFAULT);
+ try {
+ this.alterTableMethod = HiveMetaStoreClient.class.getMethod("alter_table_with_environmentContext",
+ String.class, String.class, Table.class, EnvironmentContext.class);
+ } catch (NoSuchMethodException e) {
+ try {
+ this.alterTableMethod = HiveMetaStoreClient.class.getMethod("alter_table",
+ String.class, String.class, Table.class, EnvironmentContext.class);
+ } catch (NoSuchMethodException inner) {
+ throw new RuntimeMetaException(inner, "Fail to find method alter_table_with_environmentContext and " +
+ "alter_table with right signature");
+ }
Review comment:
Can you update this to use the reflection utilities from `iceberg-common`? Those handle exceptions and use a builder pattern so it's concise and easy to maintain across versions:
```java
private static final DynMethods.UnboundMethod ALTER_TABLE = DynMethods
.builder("alter_table")
.impl(HiveMetaStoreClient.class, "alter_table_with_environmentContext",
String.class, String.class, Table.class, EnvironmentContext.class)
.impl(HiveMetaStoreClient.class, "alter_table",
String.class, String.class, Table.class, EnvironmentContext.class)
.build();
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [incubator-iceberg] rdblue commented on a change in pull request
#841: Upgrade Hive to 2.3.6 for Iceberg
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #841: Upgrade Hive to 2.3.6 for Iceberg
URL: https://github.com/apache/incubator-iceberg/pull/841#discussion_r392333374
##########
File path: hive/src/main/java/org/apache/iceberg/hive/HiveTableOperations.java
##########
@@ -175,7 +190,12 @@ protected void doCommit(TableMetadata base, TableMetadata metadata) {
EnvironmentContext envContext = new EnvironmentContext(
ImmutableMap.of(StatsSetupConst.DO_NOT_UPDATE_STATS, StatsSetupConst.TRUE)
);
- client.alter_table(database, tableName, tbl, envContext);
+
+ try {
+ alterTableMethod.invoke(client, database, tableName, tbl, envContext);
+ } catch (Exception e) {
+ throw new RuntimeMetaException(e, "Failed to alter table for %s.%s", database, tableName);
+ }
Review comment:
If you use `DynMethods` then you'd call the method like this:
```java
try {
clients.run(client -> ALTER_TABLE.invoke(client, null, null, null, null));
} catch (TException e) {
throw new RuntimeMetaException(e, "Failed to alter table");
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
throw new RuntimeException("Interrupted in call to alterTable", e);
}
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [incubator-iceberg] rdblue edited a comment on issue #841: Upgrade
Hive to 2.3.6 for Iceberg
Posted by GitBox <gi...@apache.org>.
rdblue edited a comment on issue #841: Upgrade Hive to 2.3.6 for Iceberg
URL: https://github.com/apache/incubator-iceberg/pull/841#issuecomment-598809775
Thanks for working on this, @jerryshao! I think the code part is close -- I'd just prefer to use the `DynMethods` builder to avoid try/catch and custom reflect code.
We also need to take a close look at all the dependency additions and remove whatever we can. This pulls in a lot of new dependencies and most of them are probably unnecessary. Those cause a lot of work to maintain our binary LICENSE and NOTICE files and can cause conflicts.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [incubator-iceberg] jerryshao commented on a change in pull request
#841: Upgrade Hive to 2.3.6 for Iceberg
Posted by GitBox <gi...@apache.org>.
jerryshao commented on a change in pull request #841: Upgrade Hive to 2.3.6 for Iceberg
URL: https://github.com/apache/incubator-iceberg/pull/841#discussion_r392765019
##########
File path: versions.lock
##########
@@ -1,31 +1,39 @@
# Run ./gradlew --write-locks to regenerate this file
-ant:ant:1.6.5 (1 constraints: bb0a41d8)
-antlr:antlr:2.7.7 (2 constraints: 36167e02)
+ant:ant:1.6.5 (2 constraints: 2d1539e1)
aopalliance:aopalliance:1.0 (1 constraints: 170a83ac)
-asm:asm:3.1 (2 constraints: 4f19c3c6)
+asm:asm:3.1 (3 constraints: 251fd7ad)
+asm:asm-commons:3.1 (1 constraints: 9c0f1f7a)
+asm:asm-tree:3.1 (1 constraints: 2307035c)
+ch.qos.logback:logback-classic:1.0.9 (3 constraints: d3258a88)
+ch.qos.logback:logback-core:1.0.9 (4 constraints: dd32435a)
+co.cask.tephra:tephra-api:0.6.0 (3 constraints: 0828ded1)
+co.cask.tephra:tephra-core:0.6.0 (2 constraints: 831cd90d)
+co.cask.tephra:tephra-hbase-compat-1.0:0.6.0 (1 constraints: 370d6920)
Review comment:
tephra is a "globally consistent transactions on top of Apache HBase.", introduced by Hive Metastore, I don't think it is needed, will remove it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [incubator-iceberg] jerryshao commented on a change in pull request
#841: Upgrade Hive to 2.3.6 for Iceberg
Posted by GitBox <gi...@apache.org>.
jerryshao commented on a change in pull request #841: Upgrade Hive to 2.3.6 for Iceberg
URL: https://github.com/apache/incubator-iceberg/pull/841#discussion_r393542421
##########
File path: versions.lock
##########
@@ -170,24 +188,33 @@ org.apache.spark:spark-sketch_2.11:2.4.4 (2 constraints: 981bd4f5)
org.apache.spark:spark-sql_2.11:2.4.4 (1 constraints: 1e0d0037)
org.apache.spark:spark-tags_2.11:2.4.4 (8 constraints: 036fa69d)
org.apache.spark:spark-unsafe_2.11:2.4.4 (2 constraints: f11bc213)
-org.apache.thrift:libfb303:0.9.3 (3 constraints: 6725fac0)
-org.apache.thrift:libthrift:0.9.3 (5 constraints: 71415452)
+org.apache.thrift:libfb303:0.9.3 (3 constraints: 27289d07)
+org.apache.thrift:libthrift:0.9.3 (7 constraints: c75ac7de)
+org.apache.twill:twill-api:0.6.0-incubating (2 constraints: e422b4e1)
+org.apache.twill:twill-common:0.6.0-incubating (4 constraints: 3d467991)
+org.apache.twill:twill-core:0.6.0-incubating (1 constraints: d70f9d7c)
+org.apache.twill:twill-discovery-api:0.6.0-incubating (3 constraints: 25345d4c)
+org.apache.twill:twill-discovery-core:0.6.0-incubating (2 constraints: 332039f9)
+org.apache.twill:twill-zookeeper:0.6.0-incubating (3 constraints: 94341288)
org.apache.xbean:xbean-asm6-shaded:4.8 (2 constraints: 2419a30f)
org.apache.yetus:audience-annotations:0.11.0 (1 constraints: c40eb364)
-org.apache.zookeeper:zookeeper:3.4.6 (11 constraints: 18a71f48)
+org.apache.zookeeper:zookeeper:3.4.6 (12 constraints: 54b50652)
org.checkerframework:checker-qual:2.8.1 (2 constraints: 1a1a3944)
org.codehaus.jackson:jackson-core-asl:1.9.13 (11 constraints: 8091cd06)
org.codehaus.jackson:jackson-jaxrs:1.9.13 (2 constraints: 821bca9d)
org.codehaus.jackson:jackson-mapper-asl:1.9.13 (11 constraints: e18d325f)
org.codehaus.jackson:jackson-xc:1.9.13 (2 constraints: 821bca9d)
-org.codehaus.janino:commons-compiler:3.0.9 (3 constraints: 0a2837cc)
-org.codehaus.janino:janino:3.0.9 (2 constraints: 3f1c6304)
+org.codehaus.janino:commons-compiler:3.0.9 (2 constraints: a41a546f)
+org.codehaus.janino:janino:3.0.9 (1 constraints: d90e817c)
org.codehaus.jettison:jettison:1.1 (4 constraints: a84e24a9)
org.codehaus.mojo:animal-sniffer-annotations:1.17 (1 constraints: ed09d8aa)
-org.datanucleus:datanucleus-api-jdo:3.2.6 (2 constraints: 5a1d241c)
-org.datanucleus:datanucleus-core:3.2.10 (5 constraints: 4a4401e0)
-org.datanucleus:datanucleus-rdbms:3.2.9 (2 constraints: 601db41c)
+org.datanucleus:datanucleus-api-jdo:4.2.4 (2 constraints: 591df91b)
+org.datanucleus:datanucleus-core:4.1.17 (5 constraints: 584455e8)
+org.datanucleus:datanucleus-rdbms:4.1.19 (2 constraints: 911dec32)
Review comment:
I'm not sure if I understand your meaning.
I'm not relying on the Hive version (2.3.6) in iceberg-spark module, instead iceberg-spark module will continue to use Hive (1.2.1-spark2) to run UTs. The datanucleus 4.x jars will only be used in iceberg-hive module. And iceberg-spark module will use the datanucleus 3.x, which is introduced by Hive 1.2.1.
The reason why I enforce the 3.x versions in iceberg-spark module is that Hive 1.2.1 can only work with datanucleus 3.x, also I'm not sure if Hive 2.3.6 can work with Spark 2.4.4, so in iceberg-spark module, I don't include the Hive 2.3.6 jars, still use Hive jars bring in by Spark.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [incubator-iceberg] rdblue commented on a change in pull request
#841: Upgrade Hive to 2.3.6 for Iceberg
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #841: Upgrade Hive to 2.3.6 for Iceberg
URL: https://github.com/apache/incubator-iceberg/pull/841#discussion_r392338021
##########
File path: versions.lock
##########
@@ -170,24 +188,33 @@ org.apache.spark:spark-sketch_2.11:2.4.4 (2 constraints: 981bd4f5)
org.apache.spark:spark-sql_2.11:2.4.4 (1 constraints: 1e0d0037)
org.apache.spark:spark-tags_2.11:2.4.4 (8 constraints: 036fa69d)
org.apache.spark:spark-unsafe_2.11:2.4.4 (2 constraints: f11bc213)
-org.apache.thrift:libfb303:0.9.3 (3 constraints: 6725fac0)
-org.apache.thrift:libthrift:0.9.3 (5 constraints: 71415452)
+org.apache.thrift:libfb303:0.9.3 (3 constraints: 27289d07)
+org.apache.thrift:libthrift:0.9.3 (7 constraints: c75ac7de)
+org.apache.twill:twill-api:0.6.0-incubating (2 constraints: e422b4e1)
+org.apache.twill:twill-common:0.6.0-incubating (4 constraints: 3d467991)
+org.apache.twill:twill-core:0.6.0-incubating (1 constraints: d70f9d7c)
+org.apache.twill:twill-discovery-api:0.6.0-incubating (3 constraints: 25345d4c)
+org.apache.twill:twill-discovery-core:0.6.0-incubating (2 constraints: 332039f9)
+org.apache.twill:twill-zookeeper:0.6.0-incubating (3 constraints: 94341288)
org.apache.xbean:xbean-asm6-shaded:4.8 (2 constraints: 2419a30f)
org.apache.yetus:audience-annotations:0.11.0 (1 constraints: c40eb364)
-org.apache.zookeeper:zookeeper:3.4.6 (11 constraints: 18a71f48)
+org.apache.zookeeper:zookeeper:3.4.6 (12 constraints: 54b50652)
org.checkerframework:checker-qual:2.8.1 (2 constraints: 1a1a3944)
org.codehaus.jackson:jackson-core-asl:1.9.13 (11 constraints: 8091cd06)
org.codehaus.jackson:jackson-jaxrs:1.9.13 (2 constraints: 821bca9d)
org.codehaus.jackson:jackson-mapper-asl:1.9.13 (11 constraints: e18d325f)
org.codehaus.jackson:jackson-xc:1.9.13 (2 constraints: 821bca9d)
-org.codehaus.janino:commons-compiler:3.0.9 (3 constraints: 0a2837cc)
-org.codehaus.janino:janino:3.0.9 (2 constraints: 3f1c6304)
+org.codehaus.janino:commons-compiler:3.0.9 (2 constraints: a41a546f)
+org.codehaus.janino:janino:3.0.9 (1 constraints: d90e817c)
org.codehaus.jettison:jettison:1.1 (4 constraints: a84e24a9)
org.codehaus.mojo:animal-sniffer-annotations:1.17 (1 constraints: ed09d8aa)
-org.datanucleus:datanucleus-api-jdo:3.2.6 (2 constraints: 5a1d241c)
-org.datanucleus:datanucleus-core:3.2.10 (5 constraints: 4a4401e0)
-org.datanucleus:datanucleus-rdbms:3.2.9 (2 constraints: 601db41c)
+org.datanucleus:datanucleus-api-jdo:4.2.4 (2 constraints: 591df91b)
+org.datanucleus:datanucleus-core:4.1.17 (5 constraints: 584455e8)
+org.datanucleus:datanucleus-rdbms:4.1.19 (2 constraints: 911dec32)
+org.datanucleus:javax.jdo:3.2.0-m3 (1 constraints: 030ea249)
org.eclipse.jdt:core:3.1.1 (1 constraints: b40a38d8)
+org.eclipse.jetty.aggregate:jetty-all:7.6.0.v20120127 (1 constraints: 260ea577)
+org.eclipse.jetty.orbit:javax.servlet:3.0.0.v201112011016 (1 constraints: dd0e53b1)
Review comment:
Are these servlet classes needed?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [incubator-iceberg] rdblue commented on a change in pull request
#841: Upgrade Hive to 2.3.6 for Iceberg
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #841: Upgrade Hive to 2.3.6 for Iceberg
URL: https://github.com/apache/incubator-iceberg/pull/841#discussion_r392327461
##########
File path: hive/src/main/java/org/apache/iceberg/hive/HiveTableOperations.java
##########
@@ -75,6 +77,7 @@
private final long lockAcquireTimeout;
private FileIO fileIO;
+ private Method alterTableMethod;
Review comment:
Since the version of Hive is unlikely to change, can we make this a `static` field?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org