You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "Tadas Sugintas (Jira)" <ji...@apache.org> on 2019/11/12 07:31:00 UTC
[jira] [Commented] (HUDI-238) Make separate release for hudi
spark/scala based packages for scala 2.12
[ https://issues.apache.org/jira/browse/HUDI-238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16972139#comment-16972139 ]
Tadas Sugintas commented on HUDI-238:
-------------------------------------
As commented on [https://github.com/apache/incubator-hudi/pull/1005,] I'd like contribute to this.
Without knowing about this ticket, I ran into the same issues with `KafkaOffsetGen` as mentioned above. I didn't crack it fully, but I think it could be done, replacing `org.apache.spark.streaming.kafka.KafkaCluster` with `org.apache.kafka.clients.consumer.KafkaConsumer`.
> Make separate release for hudi spark/scala based packages for scala 2.12
> -------------------------------------------------------------------------
>
> Key: HUDI-238
> URL: https://issues.apache.org/jira/browse/HUDI-238
> Project: Apache Hudi (incubating)
> Issue Type: Improvement
> Components: asf-migration
> Reporter: Balaji Varadarajan
> Assignee: Balaji Varadarajan
> Priority: Major
> Fix For: 0.5.1
>
>
> [https://github.com/apache/incubator-hudi/issues/881#issuecomment-528700749]
> Suspects:
> h3. Hudi utilities package
> bringing in spark-streaming-kafka-0.8*
> {code:java}
> [INFO] Scanning for projects...
> [INFO]
> [INFO] -------------------< org.apache.hudi:hudi-utilities >-------------------
> [INFO] Building hudi-utilities 0.5.0-SNAPSHOT
> [INFO] --------------------------------[ jar ]---------------------------------
> [INFO]
> [INFO] --- maven-dependency-plugin:3.1.1:tree (default-cli) @ hudi-utilities ---
> [INFO] org.apache.hudi:hudi-utilities:jar:0.5.0-SNAPSHOT
> [INFO] ...
> [INFO] +- org.apache.hudi:hudi-client:jar:0.5.0-SNAPSHOT:compile
> ...
> [INFO]
> [INFO] +- org.apache.hudi:hudi-spark:jar:0.5.0-SNAPSHOT:compile
> [INFO] | \- org.scala-lang:scala-library:jar:2.11.8:compile
> [INFO] +- log4j:log4j:jar:1.2.17:compile
> ...
> [INFO] +- org.apache.spark:spark-core_2.11:jar:2.1.0:provided
> [INFO] | +- org.apache.avro:avro-mapred:jar:hadoop2:1.7.7:provided
> [INFO] | | +- org.apache.avro:avro-ipc:jar:1.7.7:provided
> [INFO] | | \- org.apache.avro:avro-ipc:jar:tests:1.7.7:provided
> [INFO] | +- com.twitter:chill_2.11:jar:0.8.0:provided
> [INFO] | +- com.twitter:chill-java:jar:0.8.0:provided
> [INFO] | +- org.apache.xbean:xbean-asm5-shaded:jar:4.4:provided
> [INFO] | +- org.apache.spark:spark-launcher_2.11:jar:2.1.0:provided
> [INFO] | +- org.apache.spark:spark-network-common_2.11:jar:2.1.0:provided
> [INFO] | +- org.apache.spark:spark-network-shuffle_2.11:jar:2.1.0:provided
> [INFO] | +- org.apache.spark:spark-unsafe_2.11:jar:2.1.0:provided
> [INFO] | +- net.java.dev.jets3t:jets3t:jar:0.7.1:provided
> [INFO] | +- org.apache.curator:curator-recipes:jar:2.4.0:provided
> [INFO] | +- org.apache.commons:commons-lang3:jar:3.5:provided
> [INFO] | +- org.apache.commons:commons-math3:jar:3.4.1:provided
> [INFO] | +- com.google.code.findbugs:jsr305:jar:1.3.9:provided
> [INFO] | +- org.slf4j:slf4j-api:jar:1.7.16:compile
> [INFO] | +- org.slf4j:jul-to-slf4j:jar:1.7.16:provided
> [INFO] | +- org.slf4j:jcl-over-slf4j:jar:1.7.16:provided
> [INFO] | +- org.slf4j:slf4j-log4j12:jar:1.7.16:compile
> [INFO] | +- com.ning:compress-lzf:jar:1.0.3:provided
> [INFO] | +- org.xerial.snappy:snappy-java:jar:1.1.2.6:compile
> [INFO] | +- net.jpountz.lz4:lz4:jar:1.3.0:compile
> [INFO] | +- org.roaringbitmap:RoaringBitmap:jar:0.5.11:provided
> [INFO] | +- commons-net:commons-net:jar:2.2:provided
> ....
> [INFO] +- org.apache.spark:spark-sql_2.11:jar:2.1.0:provided
> [INFO] | +- com.univocity:univocity-parsers:jar:2.2.1:provided
> [INFO] | +- org.apache.spark:spark-sketch_2.11:jar:2.1.0:provided
> [INFO] | \- org.apache.spark:spark-catalyst_2.11:jar:2.1.0:provided
> [INFO] | +- org.codehaus.janino:janino:jar:3.0.0:provided
> [INFO] | +- org.codehaus.janino:commons-compiler:jar:3.0.0:provided
> [INFO] | \- org.antlr:antlr4-runtime:jar:4.5.3:provided
> [INFO] +- com.databricks:spark-avro_2.11:jar:4.0.0:provided
> [INFO] +- org.apache.spark:spark-streaming_2.11:jar:2.1.0:compile
> [INFO] +- org.apache.spark:spark-streaming-kafka-0-8_2.11:jar:2.1.0:compile
> [INFO] | \- org.apache.kafka:kafka_2.11:jar:0.8.2.1:compile
> [INFO] | +- org.scala-lang.modules:scala-xml_2.11:jar:1.0.2:compile
> [INFO] | +- org.scala-lang.modules:scala-parser-combinators_2.11:jar:1.0.2:compile
> [INFO] | \- org.apache.kafka:kafka-clients:jar:0.8.2.1:compile
> [INFO] +- io.dropwizard.metrics:metrics-core:jar:4.0.2:compile
> [INFO] +- org.antlr:stringtemplate:jar:4.0.2:compile
> [INFO] | \- org.antlr:antlr-runtime:jar:3.3:compile
> [INFO] +- com.beust:jcommander:jar:1.72:compile
> [INFO] +- com.twitter:bijection-avro_2.11:jar:0.9.2:compile
> [INFO] | \- com.twitter:bijection-core_2.11:jar:0.9.2:compile
> [INFO] +- io.confluent:kafka-avro-serializer:jar:3.0.0:compile
> [INFO] +- io.confluent:common-config:jar:3.0.0:compile
> [INFO] +- io.confluent:common-utils:jar:3.0.0:compile
> [INFO] | \- com.101tec:zkclient:jar:0.5:compile
> [INFO] +- io.confluent:kafka-schema-registry-client:jar:3.0.0:compile
> [INFO] \- org.mockito:mockito-all:jar:1.10.19:test
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 2.211 s
> [INFO] Finished at: 2019-09-12T09:39:32-07:00
> [INFO] ------------------------------------------------------------------------
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)