You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "Pavan792reddy (via GitHub)" <gi...@apache.org> on 2024/05/20 15:02:33 UTC
[I] issue with reading the data using hudi streamer [hudi]
Pavan792reddy opened a new issue, #11263:
URL: https://github.com/apache/hudi/issues/11263
**_Tips before filing an issue_**
- Have you gone through our [FAQs](https://hudi.apache.org/learn/faq/)?
- Join the mailing list to engage in conversations and get faster support at dev-subscribe@hudi.apache.org.
- If you have triaged this as a bug, then file an [issue](https://issues.apache.org/jira/projects/HUDI/issues) directly.
**Describe the problem you faced**
i have tried to load the messages from pulsar into spark using hoodie streamer.but facing unexpected behaviour in the error log
A clear and concise description of the problem.
**To Reproduce**
Steps to reproduce the behavior:
1.
2.
3.
4.
**Expected behavior**
A clear and concise description of what you expected to happen.
**Environment Description**
* Hudi version :
* Spark version :3.3.2
* Hive version :
* Hadoop version :
* Storage (HDFS/S3/GCS..) :GCS
* Running on Docker? (yes/no) :NO
**Additional context**
Add any other context about the problem here.
**Stacktrace**
```Add the stacktrace of the error.```
spark-submit \
--master 'local[*]' \
--deploy-mode client \
--class org.apache.hudi.utilities.streamer.HoodieStreamer \
--packages 'org.apache.hudi:hudi-spark3.4-bundle_2.12:0.14.0,io.streamnative.connectors:pulsar-spark-connector_2.12:3.1.1.4,org.apache.spark:spark-avro_2.12:3.0.1,org.apache.hudi:hudi-utilities-slim-bundle_2.12:0.13.0,org.slf4j:slf4j-api:1.7.32' \
--repositories https://repo.maven.apache.org/maven2 \
--conf 'spark.serializer=org.apache.spark.serializer.KryoSerializer' \
--conf 'spark.sql.catalog.spark_catalog=org.apache.spark.sql.hudi.catalog.HoodieCatalog' \
--conf 'spark.sql.extensions=org.apache.spark.sql.hudi.HoodieSparkSessionExtension' \
/home/pavankumar_reddy/hudi-utilities-slim-bundle_2.12-0.14.0.jar \
--source-class org.apache.hudi.utilities.sources.PulsarSource \
--source-ordering-field ts \
--target-base-path gs://test-hudi-searce-q6/hudi_data/$TOPIC_NAME_STREAM \
--target-table $TOPIC_NAME_STREAM \
--hoodie-conf hoodie.datasource.write.recordkey.field=id \
--hoodie-conf hoodie.datasource.write.partitionpath.field=id \
--table-type COPY_ON_WRITE \
--hoodie-conf hoodie.datasource.write.keygenerator.class=org.apache.hudi.keygen.SimpleKeyGenerator \
--hoodie-conf hoodie.deltastreamer.source.pulsar.topic=persistent://mytenant/mynamespace/$TOPIC_NAME \
--hoodie-conf hoodie.deltastreamer.source.pulsar.offset.autoResetStrategy=EARLIEST \
--hoodie-conf hoodie.deltastreamer.source.pulsar.endpoint.service.url=http://localhost/::6650 \
--hoodie-conf hoodie.deltastreamer.source.pulsar.endpoint.admin.url=http://localhost/::8080
Error:-
**https://repo.maven.apache.org/maven2 added as a remote repository with the name: repo-1
:: loading settings :: url = jar:file:/usr/lib/spark/jars/ivy-2.5.1.jar!/org/apache/ivy/core/settings/ivysettings.xml
Ivy Default Cache set to: /root/.ivy2/cache
The jars for the packages stored in: /root/.ivy2/jars
org.apache.hudi#hudi-spark3.4-bundle_2.12 added as a dependency
io.streamnative.connectors#pulsar-spark-connector_2.12 added as a dependency
:: resolving dependencies :: org.apache.spark#spark-submit-parent-cb5638c3-36ce-4d04-ae3a-fbf707f05694;1.0
confs: [default]
found org.apache.hudi#hudi-spark3.4-bundle_2.12;0.14.0 in central
found io.streamnative.connectors#pulsar-spark-connector_2.12;3.1.1.4 in central
found io.swagger#swagger-annotations;1.5.21 in central
found org.slf4j#slf4j-api;1.7.25 in central
found com.sun.activation#javax.activation;1.2.0 in central
found org.slf4j#jul-to-slf4j;1.7.25 in central
found org.checkerframework#checker-qual;2.0.0 in central
found org.codehaus.mojo#animal-sniffer-annotations;1.14 in central
:: resolution report :: resolve 346ms :: artifacts dl 16ms
:: modules in use:
com.sun.activation#javax.activation;1.2.0 from central in [default]
io.streamnative.connectors#pulsar-spark-connector_2.12;3.1.1.4 from central in [default]
io.swagger#swagger-annotations;1.5.21 from central in [default]
org.apache.hudi#hudi-spark3.4-bundle_2.12;0.14.0 from central in [default]
org.checkerframework#checker-qual;2.0.0 from central in [default]
org.codehaus.mojo#animal-sniffer-annotations;1.14 from central in [default]
org.slf4j#jul-to-slf4j;1.7.25 from central in [default]
org.slf4j#slf4j-api;1.7.25 from central in [default]
---------------------------------------------------------------------
| | modules || artifacts |
| conf | number| search|dwnlded|evicted|| number|dwnlded|
---------------------------------------------------------------------
| default | 8 | 0 | 0 | 0 || 8 | 0 |
---------------------------------------------------------------------
:: retrieving :: org.apache.spark#spark-submit-parent-cb5638c3-36ce-4d04-ae3a-fbf707f05694
confs: [default]
0 artifacts copied, 8 already retrieved (0kB/10ms)
Exception in thread "main" java.lang.NoSuchFieldError: SENSITIVE_CONFIG_KEYS_FILTER
at org.apache.hudi.utilities.streamer.HoodieStreamer.<clinit>(HoodieStreamer.java:110)
at java.base/java.lang.Class.forName0(Native Method)
at java.base/java.lang.Class.forName(Class.java:398)
at org.apache.spark.util.Utils$.classForName(Utils.scala:219)
at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:938)
at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1061)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1070)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)**
root@pulsar-2-hudi-2-trino-m:/home/pavankumar_reddy#
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] issue with reading the data using hudi streamer [hudi]
Posted by "Pavan792reddy (via GitHub)" <gi...@apache.org>.
Pavan792reddy commented on issue #11263:
URL: https://github.com/apache/hudi/issues/11263#issuecomment-2125069736
the messageid is generating from the pulsar topic but it was generating as **__messageId**| not the _MessageId_ ,
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] issue with reading the data using hudi streamer [hudi]
Posted by "codope (via GitHub)" <gi...@apache.org>.
codope closed issue #11263: issue with reading the data using hudi streamer
URL: https://github.com/apache/hudi/issues/11263
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] issue with reading the data using hudi streamer [hudi]
Posted by "Pavan792reddy (via GitHub)" <gi...@apache.org>.
Pavan792reddy commented on issue #11263:
URL: https://github.com/apache/hudi/issues/11263#issuecomment-2124908753
@ad1happy2go i have made all the changes it was working as expected. Now the script was failing with below error
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] issue with reading the data using hudi streamer [hudi]
Posted by "ad1happy2go (via GitHub)" <gi...@apache.org>.
ad1happy2go commented on issue #11263:
URL: https://github.com/apache/hudi/issues/11263#issuecomment-2141946954
Using schema registry fixed this issue. Discussed in this thread - https://apache-hudi.slack.com/archives/C4D716NPQ/p1716384858692059
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] issue with reading the data using hudi streamer [hudi]
Posted by "ad1happy2go (via GitHub)" <gi...@apache.org>.
ad1happy2go commented on issue #11263:
URL: https://github.com/apache/hudi/issues/11263#issuecomment-2120935671
@Pavan792reddy Looks to be the library version conflict between utilities jar and spark bundle. can you check hudi-utilities-slim-bundle_2.12-0.14.0.jar
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org