You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2022/03/01 03:30:38 UTC
[GitHub] [hudi] pan3793 edited a comment on issue #4793: [SUPPORT] Integration test broken after upgrading from 0.10.0 to 0.10.1
pan3793 edited a comment on issue #4793:
URL: https://github.com/apache/hudi/issues/4793#issuecomment-1054965094
@YannByron
> what other hudi jars you put in the spark env
<details>
<summary>Dependency Tree</summary>
```
(apache-kyuubi) ➜ apache-kyuubi git:(hudi) mvn -pl :kyuubi-spark-sql-engine_2.12 dependency:tree
[INFO] Scanning for projects...
[INFO]
[INFO] -----------< org.apache.kyuubi:kyuubi-spark-sql-engine_2.12 >-----------
[INFO] Building Kyuubi Project Engine Spark SQL 1.5.0-SNAPSHOT
[INFO] --------------------------------[ jar ]---------------------------------
[INFO]
[INFO] --- maven-dependency-plugin:3.1.1:tree (default-cli) @ kyuubi-spark-sql-engine_2.12 ---
[INFO] org.apache.kyuubi:kyuubi-spark-sql-engine_2.12:jar:1.5.0-SNAPSHOT
[INFO] +- org.apache.kyuubi:kyuubi-common_2.12:jar:1.5.0-SNAPSHOT:compile
[INFO] | +- org.scala-lang:scala-library:jar:2.12.15:compile
[INFO] | +- org.slf4j:slf4j-api:jar:1.7.35:compile
[INFO] | +- org.slf4j:jcl-over-slf4j:jar:1.7.35:compile
[INFO] | +- org.apache.logging.log4j:log4j-slf4j-impl:jar:2.17.1:compile
[INFO] | +- org.apache.logging.log4j:log4j-api:jar:2.17.1:compile
[INFO] | +- org.apache.logging.log4j:log4j-core:jar:2.17.1:compile
[INFO] | +- org.apache.logging.log4j:log4j-1.2-api:jar:2.17.1:compile
[INFO] | +- commons-codec:commons-codec:jar:1.15:compile
[INFO] | +- org.apache.commons:commons-lang3:jar:3.10:compile
[INFO] | +- com.google.guava:guava:jar:30.1-jre:compile
[INFO] | | \- com.google.guava:failureaccess:jar:1.0.1:compile
[INFO] | +- org.apache.thrift:libfb303:jar:0.9.3:compile
[INFO] | +- org.apache.thrift:libthrift:jar:0.16.0:compile
[INFO] | | \- org.apache.httpcomponents:httpcore:jar:4.4.15:compile
[INFO] | +- org.apache.hive:hive-service-rpc:jar:2.3.9:compile
[INFO] | +- jakarta.xml.bind:jakarta.xml.bind-api:jar:2.3.2:compile
[INFO] | | \- jakarta.activation:jakarta.activation-api:jar:1.2.1:compile
[INFO] | +- com.fasterxml.jackson.module:jackson-module-scala_2.12:jar:2.13.1:compile
[INFO] | | \- com.thoughtworks.paranamer:paranamer:jar:2.8:compile
[INFO] | \- com.fasterxml.jackson.core:jackson-databind:jar:2.13.1:compile
[INFO] +- org.apache.kyuubi:kyuubi-ha_2.12:jar:1.5.0-SNAPSHOT:compile
[INFO] | +- org.apache.curator:curator-framework:jar:2.12.0:compile
[INFO] | | \- org.apache.curator:curator-client:jar:2.12.0:compile
[INFO] | +- org.apache.curator:curator-recipes:jar:2.12.0:compile
[INFO] | \- org.apache.zookeeper:zookeeper:jar:3.4.14:compile
[INFO] +- org.apache.spark:spark-sql_2.12:jar:3.1.2:provided
[INFO] | +- com.univocity:univocity-parsers:jar:2.9.1:provided
[INFO] | +- org.apache.spark:spark-sketch_2.12:jar:3.1.2:provided
[INFO] | +- org.apache.spark:spark-core_2.12:jar:3.1.2:provided
[INFO] | | +- com.twitter:chill_2.12:jar:0.9.5:provided
[INFO] | | +- com.twitter:chill-java:jar:0.9.5:provided
[INFO] | | +- org.apache.spark:spark-launcher_2.12:jar:3.1.2:provided
[INFO] | | +- org.apache.spark:spark-kvstore_2.12:jar:3.1.2:provided
[INFO] | | | \- org.fusesource.leveldbjni:leveldbjni-all:jar:1.8:provided
[INFO] | | +- org.apache.spark:spark-network-common_2.12:jar:3.1.2:provided
[INFO] | | +- org.apache.spark:spark-network-shuffle_2.12:jar:3.1.2:provided
[INFO] | | +- org.apache.spark:spark-unsafe_2.12:jar:3.1.2:provided
[INFO] | | +- javax.activation:activation:jar:1.1.1:provided
[INFO] | | +- jakarta.servlet:jakarta.servlet-api:jar:4.0.4:provided
[INFO] | | +- org.apache.commons:commons-math3:jar:3.4.1:provided
[INFO] | | +- org.apache.commons:commons-text:jar:1.6:provided
[INFO] | | +- com.ning:compress-lzf:jar:1.0.3:provided
[INFO] | | +- org.xerial.snappy:snappy-java:jar:1.1.8.2:provided
[INFO] | | +- org.lz4:lz4-java:jar:1.7.1:provided
[INFO] | | +- com.github.luben:zstd-jni:jar:1.4.8-1:provided
[INFO] | | +- org.roaringbitmap:RoaringBitmap:jar:0.9.0:provided
[INFO] | | | \- org.roaringbitmap:shims:jar:0.9.0:provided
[INFO] | | +- commons-net:commons-net:jar:3.1:provided
[INFO] | | +- org.json4s:json4s-jackson_2.12:jar:3.7.0-M5:provided
[INFO] | | | \- org.json4s:json4s-core_2.12:jar:3.7.0-M5:provided
[INFO] | | | +- org.json4s:json4s-ast_2.12:jar:3.7.0-M5:provided
[INFO] | | | \- org.json4s:json4s-scalap_2.12:jar:3.7.0-M5:provided
[INFO] | | +- org.glassfish.jersey.core:jersey-client:jar:2.30:provided
[INFO] | | | +- jakarta.ws.rs:jakarta.ws.rs-api:jar:2.1.6:provided
[INFO] | | | \- org.glassfish.hk2.external:jakarta.inject:jar:2.6.1:provided
[INFO] | | +- org.glassfish.jersey.core:jersey-common:jar:2.34:provided
[INFO] | | | +- jakarta.annotation:jakarta.annotation-api:jar:1.3.5:provided
[INFO] | | | \- org.glassfish.hk2:osgi-resource-locator:jar:1.0.3:provided
[INFO] | | +- org.glassfish.jersey.core:jersey-server:jar:2.34:provided
[INFO] | | | \- jakarta.validation:jakarta.validation-api:jar:2.0.2:provided
[INFO] | | +- org.glassfish.jersey.containers:jersey-container-servlet:jar:2.30:provided
[INFO] | | +- org.glassfish.jersey.containers:jersey-container-servlet-core:jar:2.34:provided
[INFO] | | +- org.glassfish.jersey.inject:jersey-hk2:jar:2.34:provided
[INFO] | | | +- org.glassfish.hk2:hk2-locator:jar:2.6.1:provided
[INFO] | | | | +- org.glassfish.hk2.external:aopalliance-repackaged:jar:2.6.1:provided
[INFO] | | | | +- org.glassfish.hk2:hk2-api:jar:2.6.1:provided
[INFO] | | | | \- org.glassfish.hk2:hk2-utils:jar:2.6.1:provided
[INFO] | | | \- org.javassist:javassist:jar:3.25.0-GA:provided
[INFO] | | +- io.netty:netty-all:jar:4.1.73.Final:provided
[INFO] | | | +- io.netty:netty-buffer:jar:4.1.73.Final:provided
[INFO] | | | +- io.netty:netty-codec:jar:4.1.73.Final:provided
[INFO] | | | +- io.netty:netty-common:jar:4.1.73.Final:provided
[INFO] | | | +- io.netty:netty-handler:jar:4.1.73.Final:provided
[INFO] | | | +- io.netty:netty-tcnative-classes:jar:2.0.46.Final:provided
[INFO] | | | +- io.netty:netty-resolver:jar:4.1.73.Final:provided
[INFO] | | | +- io.netty:netty-transport:jar:4.1.73.Final:provided
[INFO] | | | +- io.netty:netty-transport-classes-epoll:jar:4.1.73.Final:provided
[INFO] | | | +- io.netty:netty-transport-native-unix-common:jar:4.1.73.Final:provided
[INFO] | | | +- io.netty:netty-transport-classes-kqueue:jar:4.1.73.Final:provided
[INFO] | | | +- io.netty:netty-transport-native-epoll:jar:linux-x86_64:4.1.73.Final:provided
[INFO] | | | +- io.netty:netty-transport-native-epoll:jar:linux-aarch_64:4.1.73.Final:provided
[INFO] | | | +- io.netty:netty-transport-native-kqueue:jar:osx-x86_64:4.1.73.Final:provided
[INFO] | | | \- io.netty:netty-transport-native-kqueue:jar:osx-aarch_64:4.1.73.Final:provided
[INFO] | | +- com.clearspring.analytics:stream:jar:2.9.6:provided
[INFO] | | +- io.dropwizard.metrics:metrics-core:jar:4.2.8:provided
[INFO] | | +- io.dropwizard.metrics:metrics-jvm:jar:4.2.8:provided
[INFO] | | +- io.dropwizard.metrics:metrics-json:jar:4.2.8:provided
[INFO] | | +- io.dropwizard.metrics:metrics-graphite:jar:4.1.1:provided
[INFO] | | +- io.dropwizard.metrics:metrics-jmx:jar:4.2.8:provided
[INFO] | | +- org.apache.ivy:ivy:jar:2.4.0:provided
[INFO] | | +- oro:oro:jar:2.0.8:provided
[INFO] | | +- net.razorvine:pyrolite:jar:4.30:provided
[INFO] | | +- net.sf.py4j:py4j:jar:0.10.9:provided
[INFO] | | \- org.apache.commons:commons-crypto:jar:1.1.0:provided
[INFO] | +- org.apache.spark:spark-catalyst_2.12:jar:3.1.2:provided
[INFO] | | +- org.scala-lang.modules:scala-parser-combinators_2.12:jar:1.1.2:provided
[INFO] | | +- org.codehaus.janino:janino:jar:3.0.16:provided
[INFO] | | +- org.codehaus.janino:commons-compiler:jar:3.0.16:provided
[INFO] | | +- org.antlr:antlr4-runtime:jar:4.8-1:provided
[INFO] | | +- org.apache.arrow:arrow-vector:jar:2.0.0:provided
[INFO] | | | +- org.apache.arrow:arrow-format:jar:2.0.0:provided
[INFO] | | | +- org.apache.arrow:arrow-memory-core:jar:2.0.0:provided
[INFO] | | | \- com.google.flatbuffers:flatbuffers-java:jar:1.9.0:provided
[INFO] | | \- org.apache.arrow:arrow-memory-netty:jar:2.0.0:provided
[INFO] | +- org.apache.spark:spark-tags_2.12:jar:3.1.2:provided
[INFO] | +- org.apache.orc:orc-core:jar:1.5.12:provided
[INFO] | | +- org.apache.orc:orc-shims:jar:1.5.12:provided
[INFO] | | +- com.google.protobuf:protobuf-java:jar:2.5.0:provided
[INFO] | | +- commons-lang:commons-lang:jar:2.6:provided
[INFO] | | +- io.airlift:aircompressor:jar:0.10:provided
[INFO] | | \- org.threeten:threeten-extra:jar:1.5.0:provided
[INFO] | +- org.apache.orc:orc-mapreduce:jar:1.5.12:provided
[INFO] | +- org.apache.hive:hive-storage-api:jar:2.7.2:provided
[INFO] | +- org.apache.parquet:parquet-column:jar:1.10.1:provided
[INFO] | | +- org.apache.parquet:parquet-common:jar:1.10.1:provided
[INFO] | | \- org.apache.parquet:parquet-encoding:jar:1.10.1:provided
[INFO] | +- org.apache.parquet:parquet-hadoop:jar:1.10.1:provided
[INFO] | | +- org.apache.parquet:parquet-jackson:jar:1.10.1:provided
[INFO] | | \- org.codehaus.jackson:jackson-core-asl:jar:1.9.13:provided
[INFO] | +- org.apache.xbean:xbean-asm7-shaded:jar:4.15:provided
[INFO] | \- org.spark-project.spark:unused:jar:1.0.0:provided
[INFO] +- org.apache.spark:spark-repl_2.12:jar:3.1.2:provided
[INFO] | \- org.apache.spark:spark-mllib_2.12:jar:3.1.2:provided
[INFO] | +- org.apache.spark:spark-streaming_2.12:jar:3.1.2:provided
[INFO] | +- org.apache.spark:spark-graphx_2.12:jar:3.1.2:provided
[INFO] | | +- com.github.fommil.netlib:core:jar:1.1.2:provided
[INFO] | | \- net.sourceforge.f2j:arpack_combined_all:jar:0.1:provided
[INFO] | +- org.apache.spark:spark-mllib-local_2.12:jar:3.1.2:provided
[INFO] | +- org.scalanlp:breeze_2.12:jar:1.0:provided
[INFO] | | +- org.scalanlp:breeze-macros_2.12:jar:1.0:provided
[INFO] | | +- com.github.wendykierp:JTransforms:jar:3.1:provided
[INFO] | | | \- pl.edu.icm:JLargeArrays:jar:1.5:provided
[INFO] | | +- com.chuusai:shapeless_2.12:jar:2.3.3:provided
[INFO] | | | \- org.typelevel:macro-compat_2.12:jar:1.1.1:provided
[INFO] | | +- org.typelevel:spire_2.12:jar:0.17.0-M1:provided
[INFO] | | | +- org.typelevel:spire-macros_2.12:jar:0.17.0-M1:provided
[INFO] | | | +- org.typelevel:spire-platform_2.12:jar:0.17.0-M1:provided
[INFO] | | | +- org.typelevel:spire-util_2.12:jar:0.17.0-M1:provided
[INFO] | | | +- org.typelevel:machinist_2.12:jar:0.6.8:provided
[INFO] | | | \- org.typelevel:algebra_2.12:jar:2.0.0-M2:provided
[INFO] | | | \- org.typelevel:cats-kernel_2.12:jar:2.0.0-M4:provided
[INFO] | | \- org.scala-lang.modules:scala-collection-compat_2.12:jar:2.1.1:provided
[INFO] | \- org.glassfish.jaxb:jaxb-runtime:jar:2.3.2:provided
[INFO] | \- com.sun.istack:istack-commons-runtime:jar:3.0.8:provided
[INFO] +- org.scala-lang:scala-compiler:jar:2.12.15:provided
[INFO] | \- org.scala-lang.modules:scala-xml_2.12:jar:1.0.6:provided
[INFO] +- org.scala-lang:scala-reflect:jar:2.12.15:provided
[INFO] +- org.apache.hadoop:hadoop-client-api:jar:3.3.1:provided
[INFO] +- org.apache.kyuubi:kyuubi-common_2.12:test-jar:tests:1.5.0-SNAPSHOT:test
[INFO] +- commons-collections:commons-collections:jar:3.2.2:test
[INFO] +- commons-io:commons-io:jar:2.8.0:test
[INFO] +- org.apache.spark:spark-hive_2.12:jar:3.1.2:test
[INFO] | +- org.apache.hive:hive-common:jar:2.3.7:test
[INFO] | | +- commons-cli:commons-cli:jar:1.2:test
[INFO] | | +- jline:jline:jar:0.9.94:test
[INFO] | | +- org.apache.commons:commons-compress:jar:1.9:provided
[INFO] | | +- com.tdunning:json:jar:1.8:test
[INFO] | | \- com.github.joshelser:dropwizard-metrics-hadoop-metrics2-reporter:jar:0.1.2:test
[INFO] | +- org.apache.hive:hive-exec:jar:core:2.3.7:test
[INFO] | | +- org.apache.hive:hive-vector-code-gen:jar:2.3.7:test
[INFO] | | | \- org.apache.velocity:velocity:jar:1.5:test
[INFO] | | +- org.antlr:antlr-runtime:jar:3.5.2:test
[INFO] | | +- org.antlr:ST4:jar:4.0.4:test
[INFO] | | +- com.google.code.gson:gson:jar:2.2.4:test
[INFO] | | \- stax:stax-api:jar:1.0.1:test
[INFO] | +- org.apache.hive:hive-metastore:jar:2.3.7:test
[INFO] | | +- javolution:javolution:jar:5.5.1:test
[INFO] | | +- com.jolbox:bonecp:jar:0.8.0.RELEASE:test
[INFO] | | +- com.zaxxer:HikariCP:jar:2.5.1:test
[INFO] | | +- org.datanucleus:datanucleus-api-jdo:jar:4.2.4:test
[INFO] | | +- org.datanucleus:datanucleus-rdbms:jar:4.1.19:test
[INFO] | | +- commons-pool:commons-pool:jar:1.5.4:test
[INFO] | | +- commons-dbcp:commons-dbcp:jar:1.4:test
[INFO] | | +- javax.jdo:jdo-api:jar:3.0.1:test
[INFO] | | | \- javax.transaction:jta:jar:1.1:test
[INFO] | | \- org.datanucleus:javax.jdo:jar:3.2.0-m3:test
[INFO] | | \- javax.transaction:transaction-api:jar:1.1:test
[INFO] | +- org.apache.hive:hive-serde:jar:2.3.7:test
[INFO] | | \- net.sf.opencsv:opencsv:jar:2.3:provided
[INFO] | +- org.apache.hive:hive-shims:jar:2.3.7:test
[INFO] | | +- org.apache.hive.shims:hive-shims-common:jar:2.3.7:test
[INFO] | | +- org.apache.hive.shims:hive-shims-0.23:jar:2.3.7:test
[INFO] | | \- org.apache.hive.shims:hive-shims-scheduler:jar:2.3.7:test
[INFO] | +- org.apache.hive:hive-llap-common:jar:2.3.7:test
[INFO] | +- org.apache.hive:hive-llap-client:jar:2.3.7:test
[INFO] | +- org.apache.avro:avro:jar:1.8.2:provided
[INFO] | | \- org.tukaani:xz:jar:1.5:provided
[INFO] | +- org.apache.avro:avro-mapred:jar:hadoop2:1.8.2:provided
[INFO] | | \- org.apache.avro:avro-ipc:jar:1.8.2:provided
[INFO] | +- commons-httpclient:commons-httpclient:jar:3.1:test
[INFO] | | \- commons-logging:commons-logging:jar:1.0.4:compile
[INFO] | +- org.apache.httpcomponents:httpclient:jar:4.5.6:compile
[INFO] | +- org.codehaus.jackson:jackson-mapper-asl:jar:1.9.13:provided
[INFO] | +- joda-time:joda-time:jar:2.10.5:test
[INFO] | +- org.jodd:jodd-core:jar:3.5.2:test
[INFO] | +- com.google.code.findbugs:jsr305:jar:3.0.2:compile
[INFO] | +- org.datanucleus:datanucleus-core:jar:4.1.17:test
[INFO] | \- org.apache.derby:derby:jar:10.12.1.1:test
[INFO] +- org.apache.kyuubi:kyuubi-hive-jdbc-shaded:jar:1.5.0-SNAPSHOT:test
[INFO] +- org.apache.hadoop:hadoop-client-runtime:jar:3.3.1:test
[INFO] | \- org.apache.htrace:htrace-core4:jar:4.1.0-incubating:test
[INFO] +- org.slf4j:jul-to-slf4j:jar:1.7.35:test
[INFO] +- org.apache.iceberg:iceberg-spark3-runtime:jar:0.13.0:test
[INFO] +- org.apache.spark:spark-avro_2.12:jar:3.1.2:test
[INFO] +- org.apache.hudi:hudi-spark-common_2.12:jar:0.10.1:test
[INFO] | +- org.apache.hudi:hudi-client-common:jar:0.10.1:test
[INFO] | | \- com.github.davidmoten:hilbert-curve:jar:0.2.2:test
[INFO] | | \- com.github.davidmoten:guava-mini:jar:0.1.3:test
[INFO] | +- org.apache.hudi:hudi-spark-client:jar:0.10.1:test
[INFO] | +- org.apache.hudi:hudi-common:jar:0.10.1:test
[INFO] | | +- org.apache.httpcomponents:fluent-hc:jar:4.4.1:test
[INFO] | | +- org.rocksdb:rocksdbjni:jar:5.17.2:test
[INFO] | | \- com.esotericsoftware:kryo-shaded:jar:4.0.2:provided
[INFO] | | +- com.esotericsoftware:minlog:jar:1.3.0:provided
[INFO] | | \- org.objenesis:objenesis:jar:2.5.1:provided
[INFO] | \- org.apache.hudi:hudi-hive-sync:jar:0.10.1:test
[INFO] | \- com.beust:jcommander:jar:1.72:test
[INFO] +- org.apache.hudi:hudi-spark3_2.12:jar:0.10.1:test
[INFO] | +- com.fasterxml.jackson.core:jackson-annotations:jar:2.13.1:compile
[INFO] | \- com.fasterxml.jackson.core:jackson-core:jar:2.13.1:compile
[INFO] +- org.apache.parquet:parquet-avro:jar:1.10.1:test
[INFO] | +- org.apache.parquet:parquet-format:jar:2.4.0:provided
[INFO] | \- it.unimi.dsi:fastutil:jar:7.0.13:test
[INFO] +- org.apache.hudi:hudi-spark_2.12:jar:0.10.1:test
[INFO] | +- org.apache.hudi:hudi-hadoop-mr:jar:0.10.1:test
[INFO] | \- org.apache.hudi:hudi-sync-common:jar:0.10.1:test
[INFO] +- io.delta:delta-core_2.12:jar:1.0.1:test
[INFO] +- org.apache.kyuubi:kyuubi-zookeeper_2.12:jar:1.5.0-SNAPSHOT:test
[INFO] \- org.scalatest:scalatest_2.12:jar:3.2.9:test
[INFO] +- org.scalatest:scalatest-core_2.12:jar:3.2.9:test
[INFO] | +- org.scalatest:scalatest-compatible:jar:3.2.9:test
[INFO] | \- org.scalactic:scalactic_2.12:jar:3.2.9:test
[INFO] +- org.scalatest:scalatest-featurespec_2.12:jar:3.2.9:test
[INFO] +- org.scalatest:scalatest-flatspec_2.12:jar:3.2.9:test
[INFO] +- org.scalatest:scalatest-freespec_2.12:jar:3.2.9:test
[INFO] +- org.scalatest:scalatest-funsuite_2.12:jar:3.2.9:test
[INFO] +- org.scalatest:scalatest-funspec_2.12:jar:3.2.9:test
[INFO] +- org.scalatest:scalatest-propspec_2.12:jar:3.2.9:test
[INFO] +- org.scalatest:scalatest-refspec_2.12:jar:3.2.9:test
[INFO] +- org.scalatest:scalatest-wordspec_2.12:jar:3.2.9:test
[INFO] +- org.scalatest:scalatest-diagrams_2.12:jar:3.2.9:test
[INFO] +- org.scalatest:scalatest-matchers-core_2.12:jar:3.2.9:test
[INFO] +- org.scalatest:scalatest-shouldmatchers_2.12:jar:3.2.9:test
[INFO] \- org.scalatest:scalatest-mustmatchers_2.12:jar:3.2.9:test
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 1.702 s
[INFO] Finished at: 2022-03-01T11:24:46+08:00
[INFO] ------------------------------------------------------------------------
```
</details>
> is there the same problem if you use spark-sql instead of kyuubi
Basically, the component `kyuubi-spark-sql-engine` is just a Spark application which start a thrift server in Spark driver process and exposes interface which compatible with HiveServer2 thrift protocol. In this case, it just play a role forward the user SQL to `spark.sql(xxx)`, I don't the result will different with direct run SQL using `spark.sql(xxx)`, but let me try.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org