You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2021/11/20 09:56:45 UTC
[GitHub] [hudi] JB-data opened a new issue #4055: [SUPPORT] Hudi with SqlQueryBasedTransformer fails-> spark error exit 134 or exit 143 in "isEmpty at DeltaSync.java:344" : Container from a bad node: container_yy on host: xxx Exit status: 134
JB-data opened a new issue #4055:
URL: https://github.com/apache/hudi/issues/4055
**Describe the problem you faced**
We are running a spark-submit command with an SQL transformation. The data from the kafka topic is then temorarily registered in hive's default db, and applies a very simple sql to it. We see the job fail quite soon, while without the sql transformation, it succeeds. This is more related to spark settings probably than to hudi. But maybe someone has experience in the context with hudi?
**To Reproduce**
Steps to reproduce the behavior:
spark-submit command with sql transformation to take data from kafka topic, and write slighty modified to another table:
1.
spark-submit --master yarn --deploy-mode cluster --driver-java-options "-Djava.security.auth.login.config=./client.jaas" \
--conf "spark.executor.extraJavaOptions=-Djava.security.auth.login.config=./client.jaas" --conf spark.app.name=myapp \
--conf spark.yarn.maxAppAttempts=4 --conf spark.yarn.am.attemptFailuresValidityInterval=1h --conf spark.yarn.executor.failuresValidityInterval=1h \
--conf spark.task.maxFailures=8 --principal myprinc@myserver --keytab mykeytab \
--files client.jaas,mykeytab,kafka.client.truststore.jks --class org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer \
--jars /opt/cloudera/parcels/CDH-7.2.2-1.cdh7.2.2.p3.7839477/lib/hive/lib/hive-service-3.1.3000.7.2.2.3-1.jar,\
/opt/cloudera/parcels/CDH-7.2.2-1.cdh7.2.2.p3.7839477/lib/hive/lib/hive-jdbc-3.1.3000.7.2.2.3-1.jar,/opt/cloudera/parcels/CDH/lib/hbase/hbase-client.jar abfs://storage-fs@cdpdlmain.dfs.core.windows.net/config/streaming/fr24/hudi-utilities-bundle_2.11-0.6.0.jar \
\
--props abfs://stora/myproperties \
--target-base-path abfs://storagemytransfo/unmanaged/mytable \
--table-type COPY_ON_WRITE \
--source-class org.apache.hudi.utilities.sources.AvroKafkaSource \
--target-table refactored_created_by_hue_smalltransfo \
--schemaprovider-class org.apache.hudi.utilities.schema.FilebasedSchemaProvider \
--source-limit 40000 \
\
--transformer-class org.apache.hudi.utilities.transform.SqlQueryBasedTransformer \
--hoodie-conf hoodie.deltastreamer.transformer.sql='SELECT data[0].id as id FROM <SRC>'
--executor-memory 4G \
--driver-memory 2G \
**Expected behavior**
No error during that stage, correct landing of transformed data in target table.
Environment Description
Hudi version : 0.6.0
Spark version :2.4.5
Hive version :Hive 3.1 on Tez
Hadoop version :3.1.1
Storage (HDFS/S3/GCS..) : abfs (azure blob storage)
Running on Docker? (yes/no) :no
**Additional context**
Add any other context about the problem here.
**Stacktrace**
YARN: Driver stacktrace
```org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 1.0 failed 8 times, most recent failure: Lost task ... ExecutorLostFailure ...Container from a bad node: container_e29_1632244056069_1301_01_000009 on host: xxx Exit status: 134. Diagnostics: 2 WARN consumer.ConsumerConfig: The configuration 'hoodie.deltastreamer.transformer.sql' was supplied but isn't a known config.```
Note: running same job also exits with exit code 143 instead of 134.
SPARK UI:
``` after 1.6 minutes failure in 2nd stage:
isEmpty at DeltaSync.java:344``` + same error as from yarn ( Container from a bad node: container_yy on host: xxx Exit status: 134)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] JB-data edited a comment on issue #4055: [SUPPORT] Hudi with SqlQueryBasedTransformer fails-> spark error exit 134 or exit 143 in "isEmpty at DeltaSync.java:344" : Container from a bad node: container_yy on host: xxx Exit status: 134
Posted by GitBox <gi...@apache.org>.
JB-data edited a comment on issue #4055:
URL: https://github.com/apache/hudi/issues/4055#issuecomment-975241019
@xushiyan
Thanks for reply. Indeed the sql should say FROM < SRC > ... I will correct.
Where exactly should we check the application stacktrace? In both the the yarn application logs as the SPARK UI stacktrace I only find back the fact the container exited with code 134 or 143 (changes ), but not exactly why. The only place where I can infer where this happens is the stage in the spark ui that mentions:
isEmpty at DeltaSync.java:344
so I am assuming it comes from there? Here another part of the yarn applicatino logs:
```
[2021-11-19 13:44:57.189]Container exited with a non-zero exit code 134. Error file: prelaunch.err.
Last 4096 bytes of prelaunch.err :
/bin/bash: line 1: 7907 Aborted LD_LIBRARY_PATH="/opt/cloudera/parcels/CDH-7.2.2-1.cdh7.2.2.p3.7839477/lib/hadoop/../../../CDH-7.2.2-1.cdh7.2.2.p3.7839477/lib/hadoop/lib/native:" /usr/lib/jvm/java/bin/java -server -Xmx12288m '-Djava.security.auth.login.config=./client.jaas' -Djava.io.tmpdir=/hadoopfs/fs1/nodemanager/usercache/myuser/appcache/application_1632244056069_1301/container_e29_1632244056069_1301_01_000009/tmp '-Dspark.driver.port=33466' '-Dspark.network.crypto.enabled=false' '-Dspark.authenticate=false' '-Dspark.shuffle.service.port=7337' '-Dspark.ui.port=0' -Dspark.yarn.app.container.log.dir=/hadoopfs/fs1/nodemanager/log/application_1632244056069_1301/container_e29_1632244056069_1301_01_000009 -XX:OnOutOfMemoryError='kill %p' org.apache.spark.executor.CoarseGrainedExecutorBackend --driver-url spark://CoarseGrainedScheduler@someworker0.:33466 --executor-id 8 --hostname someworkerr1 --cores 1 --app-id application_1632244056069_1301 --user-class-path file:/
hadoopfs/fs1/nodemanager/usercache/myuser/appcache/application_1632244056069_1301/container_e29_1632244056069_1301_01_000009/__app__.jar --user-class-path file:/hadoopfs/fs1/nodemanager/usercache/myuser/appcache/application_1632244056069_1301/container_e29_1632244056069_1301_01_000009/hive-service-3.1.3000.7.2.2.3-1.jar --user-class-path file:/hadoopfs/fs1/nodemanager/usercache/myuser/appcache/application_1632244056069_1301/container_e29_1632244056069_1301_01_000009/hive-jdbc-3.1.3000.7.2.2.3-1.jar --user-class-path file:/hadoopfs/fs1/nodemanager/usercache/myuser/appcache/application_1632244056069_1301/container_e29_1632244056069_1301_01_000009/hbase-client.jar > /hadoopfs/fs1/nodemanager/log/application_1632244056069_1301/container_e29_1632244056069_1301_01_000009/stdout 2> /hadoopfs/fs1/nodemanager/log/application_1632244056069_1301/container_e29_1632244056069_1301_01_000009/stderr
Last 4096 bytes of stderr :
Last 4096 bytes of stderr :
ed but isn't a known config.
21/11/19 13:44:52 WARN consumer.ConsumerConfig: The configuration 'hoodie.datasource.hive_sync.partition_fields' was supplied but isn't a known config.
21/11/19 13:44:52 WARN consumer.ConsumerConfig: The configuration 'hoodie.deltastreamer.schemaprovider.source.schema.file' was supplied but isn't a known config.
21/11/19 13:44:52 WARN consumer.ConsumerConfig: The configuration 'hoodie.deltastreamer.transformer.sql' was supplied but isn't a known config.
21/11/19 13:44:52 WARN consumer.ConsumerConfig: The configuration 'hoodie.datasource.write.recordkey.field' was supplied but isn't a known config.
21/11/19 13:44:52 WARN consumer.ConsumerConfig: The configuration 'hoodie.deltastreamer.source.kafka.topic' was supplied but isn't a known config.
21/11/19 13:44:52 WARN consumer.ConsumerConfig: The configuration 'hoodie.datasource.hive_sync.enable' was supplied but isn't a known config.
21/11/19 13:44:52 WARN consumer.ConsumerConfig: The configuration 'hoodie.datasource.hive_sync.assume_date_partitioning' was supplied but isn't a known config.
21/11/19 13:44:52 WARN consumer.ConsumerConfig: The configuration 'hoodie.datasource.hive_sync.useJdbc' was supplied but isn't a known config.
21/11/19 13:44:52 WARN consumer.ConsumerConfig: The configuration 'hoodie.deltastreamer.schemaprovider.target.schema.file' was supplied but isn't a known config.
21/11/19 13:44:52 WARN consumer.ConsumerConfig: The configuration 'hoodie.datasource.write.partitionpath.field' was supplied but isn't a known config.
21/11/19 13:44:52 WARN consumer.ConsumerConfig: The configuration 'hoodie.datasource.hive_sync.jdbcurl' was supplied but isn't a known config.
21/11/19 13:44:52 WARN consumer.ConsumerConfig: The configuration 'hoodie.datasource.hive_sync.database' was supplied but isn't a known config.
21/11/19 13:44:52 INFO utils.AppInfoParser: Kafka version: 2.4.1.7.1.1.0-565
21/11/19 13:44:52 INFO utils.AppInfoParser: Kafka commitId: 7a66ef499adcd5b9
21/11/19 13:44:52 INFO utils.AppInfoParser: Kafka startTimeMs: 1637329492101
21/11/19 13:44:52 INFO consumer.KafkaConsumer: [Consumer clientId=consumer-spark-executor-hudi-1, groupId=spark-executor-hudi] Subscribed to partition(s): fr24messages-0
21/11/19 13:44:52 INFO executor.CoarseGrainedExecutorBackend: Received tokens of 4197 bytes
21/11/19 13:44:52 INFO deploy.SparkHadoopUtil: Updating delegation tokens for current user.
21/11/19 13:44:52 INFO codegen.CodeGenerator: Code generated in 317.609485 ms
21/11/19 13:44:52 INFO codegen.CodeGenerator: Code generated in 19.735724 ms
21/11/19 13:44:52 INFO kafka010.InternalKafkaConsumer: Initial fetch for spark-executor-hudi fr24messages-0 10254239
21/11/19 13:44:52 INFO consumer.KafkaConsumer: [Consumer clientId=consumer-spark-executor-hudi-1, groupId=spark-executor-hudi] Seeking to offset 10254239 for partition fr24messages-0
21/11/19 13:44:53 INFO executor.CoarseGrainedExecutorBackend: Received tokens of 4197 bytes
21/11/19 13:44:53 INFO deploy.SparkHadoopUtil: Updating delegation tokens for current user.
21/11/19 13:44:53 INFO clients.Metadata: [Consumer clientId=consumer-spark-executor-hudi-1, groupId=spark-executor-hudi] Cluster ID: AFejYh8dRyaRfFU4x5PeMA
21/11/19 13:44:54 INFO executor.CoarseGrainedExecutorBackend: Received tokens of 4197 bytes
21/11/19 13:44:54 INFO deploy.SparkHadoopUtil: Updating delegation tokens for current user.
21/11/19 13:44:54 INFO executor.CoarseGrainedExecutorBackend: Received tokens of 4197 bytes
21/11/19 13:44:54 INFO deploy.SparkHadoopUtil: Updating delegation tokens for current user.
21/11/19 13:44:55 INFO executor.CoarseGrainedExecutorBackend: Received tokens of 4197 bytes
21/11/19 13:44:55 INFO deploy.SparkHadoopUtil: Updating delegation tokens for current user.
21/11/19 13:44:56 INFO codegen.CodeGenerator: Code generated in 208.118952 ms
21/11/19 13:44:56 INFO codegen.CodeGenerator: Code generated in 91.06291 ms
21/11/19 13:44:56 INFO executor.CoarseGrainedExecutorBackend: Received tokens of 4197 bytes
21/11/19 13:44:56 INFO deploy.SparkHadoopUtil: Updating delegation tokens for current user.
.
Driver stacktrace:
at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1931)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1919)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1918)
at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1918)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:953)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:953)
at scala.Option.foreach(Option.scala:257)
at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:953)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:2152)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2101)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2090)
at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:49)
at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:764)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:2082)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:2103)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:2122)
at org.apache.spark.rdd.RDD$$anonfun$take$1.apply(RDD.scala:1409)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
at org.apache.spark.rdd.RDD.withScope(RDD.scala:385)
at org.apache.spark.rdd.RDD.take(RDD.scala:1382)
at org.apache.spark.rdd.RDD$$anonfun$isEmpty$1.apply$mcZ$sp(RDD.scala:1517)
at org.apache.spark.rdd.RDD$$anonfun$isEmpty$1.apply(RDD.scala:1517)
at org.apache.spark.rdd.RDD$$anonfun$isEmpty$1.apply(RDD.scala:1517)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
at org.apache.spark.rdd.RDD.withScope(RDD.scala:385)
at org.apache.spark.rdd.RDD.isEmpty(RDD.scala:1516)
at org.apache.spark.api.java.JavaRDDLike$class.isEmpty(JavaRDDLike.scala:544)
at org.apache.spark.api.java.AbstractJavaRDDLike.isEmpty(JavaRDDLike.scala:45)
at org.apache.hudi.utilities.deltastreamer.DeltaSync.readFromSource(DeltaSync.java:344)
at org.apache.hudi.utilities.deltastreamer.DeltaSync.syncOnce(DeltaSync.java:233)
at org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer.lambda$sync$2(HoodieDeltaStreamer.java:161)
at org.apache.hudi.common.util.Option.ifPresent(Option.java:96)
at org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer.sync(HoodieDeltaStreamer.java:159)
at org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer.main(HoodieDeltaStreamer.java:464)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:665)
....
...
Driver stacktrace:
org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 1.0 failed 8 times, most recent failure: Lost task 0.7 in stage 1.0 (TID 8, myworker1, executor 8): ExecutorLostFailure (executor 8 exited caused by one of the running tasks) Reason: Container from a bad node: container_e29_1632244056069_1301_01_000009 on host: myworker1. Exit status: 134. Diagnostics: 2 WARN consumer.ConsumerConfig: The configuration 'hoodie.deltastreamer.transformer.sql' was supplied but isn't a known config.
21/11/19 13:44:52 WARN consumer.ConsumerConfig: The configuration 'hoodie.datasource.write.recordkey.field' was supplied but isn't a known config.
21/11/19 13:44:52 WARN consumer.ConsumerConfig: The configuration 'hoodie.deltastreamer.source.kafka.topic' was supplied but isn't a known config.
21/11/19 13:44:52 WARN consumer.ConsumerConfig: The configuration 'hoodie.datasource.hive_sync.enable' was supplied but isn't a known config.
21/11/19 13:44:52 WARN consumer.ConsumerConfig: The configuration 'hoodie.datasource.hive_sync.assume_date_partitioning' was supplied but isn't a known config.
21/11/19 13:44:52 WARN consumer.ConsumerConfig: The configuration 'hoodie.datasource.hive_sync.useJdbc' was supplied but isn't a known config.
21/11/19 13:44:52 WARN consumer.ConsumerConfig: The configuration 'hoodie.deltastreamer.schemaprovider.target.schema.file' was supplied but isn't a known config.
21/11/19 13:44:52 WARN consumer.ConsumerConfig: The configuration 'hoodie.datasource.write.partitionpath.field' was supplied but isn't a known config.
21/11/19 13:44:52 WARN consumer.ConsumerConfig: The configuration 'hoodie.datasource.hive_sync.jdbcurl' was supplied but isn't a known config.
21/11/19 13:44:52 WARN consumer.ConsumerConfig: The configuration 'hoodie.datasource.hive_sync.database' was supplied but isn't a known config.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] guanlisheng commented on issue #4055: [SUPPORT] Hudi with SqlQueryBasedTransformer fails-> spark error exit 134 or exit 143 in "isEmpty at DeltaSync.java:344" : Container from a bad node: container_yy on host: xxx Exit status: 134
Posted by GitBox <gi...@apache.org>.
guanlisheng commented on issue #4055:
URL: https://github.com/apache/hudi/issues/4055#issuecomment-989456516
hey @nsivabalan , thanks for the reply.
very similar and the only difference is the dataset operation vs a spark SQL.
additional clue is that the transformer works well with HoodieIncrSource and such incident happens with JsonKafkaSource.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan closed issue #4055: [SUPPORT] Hudi with SqlQueryBasedTransformer fails-> spark error exit 134 or exit 143 in "isEmpty at DeltaSync.java:344" : Container from a bad node: container_yy on host: xxx Exit status: 134
Posted by GitBox <gi...@apache.org>.
nsivabalan closed issue #4055:
URL: https://github.com/apache/hudi/issues/4055
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan edited a comment on issue #4055: [SUPPORT] Hudi with SqlQueryBasedTransformer fails-> spark error exit 134 or exit 143 in "isEmpty at DeltaSync.java:344" : Container from a bad node: container_yy on host: xxx Exit status: 134
Posted by GitBox <gi...@apache.org>.
nsivabalan edited a comment on issue #4055:
URL: https://github.com/apache/hudi/issues/4055#issuecomment-1018552453
@JB-data : Do you have any updates for us. Can you confirm that target schema you set is aligned with the output from after transformation?
Can you also enable debug logs(for hudi) when you run your deltastreamer job and share w/ us.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #4055: [SUPPORT] Hudi with SqlQueryBasedTransformer fails-> spark error exit 134 or exit 143 in "isEmpty at DeltaSync.java:344" : Container from a bad node: container_yy on host: xxx Exit status: 134
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #4055:
URL: https://github.com/apache/hudi/issues/4055#issuecomment-1018552453
@JB-data : Do you have any updates for us. Can you confirm that target schema you set is aligned with the output from after transformation?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan closed issue #4055: [SUPPORT] Hudi with SqlQueryBasedTransformer fails-> spark error exit 134 or exit 143 in "isEmpty at DeltaSync.java:344" : Container from a bad node: container_yy on host: xxx Exit status: 134
Posted by GitBox <gi...@apache.org>.
nsivabalan closed issue #4055:
URL: https://github.com/apache/hudi/issues/4055
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] JB-data commented on issue #4055: [SUPPORT] Hudi with SqlQueryBasedTransformer fails-> spark error exit 134 or exit 143 in "isEmpty at DeltaSync.java:344" : Container from a bad node: container_yy on host: xxx Exit status: 134
Posted by GitBox <gi...@apache.org>.
JB-data commented on issue #4055:
URL: https://github.com/apache/hudi/issues/4055#issuecomment-975241019
@xushiyan
Thanks for reply. Indeed the sql should say FROM sometable... I will correct.
Where exactly should we check the application stacktrace? In both the the yarn application logs as the SPARK UI stacktrace I only find back the fact the container exited with code 134 or 143 (changes ), but not exactly why. The only place where I can infer where this happens is the stage in the spark ui that mentions:
isEmpty at DeltaSync.java:344
so I am assuming it comes from there?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] guanlisheng edited a comment on issue #4055: [SUPPORT] Hudi with SqlQueryBasedTransformer fails-> spark error exit 134 or exit 143 in "isEmpty at DeltaSync.java:344" : Container from a bad node: container_yy on host: xxx Exit status: 134
Posted by GitBox <gi...@apache.org>.
guanlisheng edited a comment on issue #4055:
URL: https://github.com/apache/hudi/issues/4055#issuecomment-989456516
hey @nsivabalan , thanks for the reply.
very similar and the only difference is the dataset operation vs a spark SQL.
an additional clue is that the transformer works well with HoodieIncrSource and such an issue is happening with JsonKafkaSource.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #4055: [SUPPORT] Hudi with SqlQueryBasedTransformer fails-> spark error exit 134 or exit 143 in "isEmpty at DeltaSync.java:344" : Container from a bad node: container_yy on host: xxx Exit status: 134
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #4055:
URL: https://github.com/apache/hudi/issues/4055#issuecomment-1039647858
@JB-data : Will close the ticket due to no activity. we can't do much w/o further logs from your end.
But feel free to re-open if you are still looking for assistance. would be happy to help.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] JB-data commented on issue #4055: [SUPPORT] Hudi with SqlQueryBasedTransformer fails-> spark error exit 134 or exit 143 in "isEmpty at DeltaSync.java:344" : Container from a bad node: container_yy on host: xxx Exit status: 134
Posted by GitBox <gi...@apache.org>.
JB-data commented on issue #4055:
URL: https://github.com/apache/hudi/issues/4055#issuecomment-975592693
The error comes from :
https://github.com/apache/hudi/blob/release-0.6.0/hudi-utilities/src/main/java/org/apache/hudi/utilities/deltastreamer/DeltaSync.java line 344 : the check if avroRDDOptional.get().isEmpty() . I wonder if it could be due to a wrong schema provided, as avroRddOptional is initialized in line 311 and this requires the schema to be fetched.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] guanlisheng commented on issue #4055: [SUPPORT] Hudi with SqlQueryBasedTransformer fails-> spark error exit 134 or exit 143 in "isEmpty at DeltaSync.java:344" : Container from a bad node: container_yy on host: xxx Exit status: 134
Posted by GitBox <gi...@apache.org>.
guanlisheng commented on issue #4055:
URL: https://github.com/apache/hudi/issues/4055#issuecomment-988592379
Hi there,
I have an identical issue when enabling my customized transformer class in Hudi 7.0 on EMR. the transformer class is performing `mapPartitions` operation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] JB-data edited a comment on issue #4055: [SUPPORT] Hudi with SqlQueryBasedTransformer fails-> spark error exit 134 or exit 143 in "isEmpty at DeltaSync.java:344" : Container from a bad node: container_yy on host: xxx Exit status: 134
Posted by GitBox <gi...@apache.org>.
JB-data edited a comment on issue #4055:
URL: https://github.com/apache/hudi/issues/4055#issuecomment-975241019
@xushiyan
Thanks for reply. Indeed the sql should say FROM < SRC > ... I will correct.
Where exactly should we check the application stacktrace? In both the the yarn application logs as the SPARK UI stacktrace I only find back the fact the container exited with code 134 or 143 (changes ), but not exactly why. The only place where I can infer where this happens is the stage in the spark ui that mentions:
isEmpty at DeltaSync.java:344
so I am assuming it comes from there?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xushiyan commented on issue #4055: [SUPPORT] Hudi with SqlQueryBasedTransformer fails-> spark error exit 134 or exit 143 in "isEmpty at DeltaSync.java:344" : Container from a bad node: container_yy on host: xxx Exit status: 134
Posted by GitBox <gi...@apache.org>.
xushiyan commented on issue #4055:
URL: https://github.com/apache/hudi/issues/4055#issuecomment-974699184
@JB-data
> --hoodie-conf hoodie.deltastreamer.transformer.sql='SELECT data[0].id as id FROM '
Is this actually valid SQL?
The error you showed is not really the root cause. Can you track down to the application stacktrace and see which part of code threw exceptions?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #4055: [SUPPORT] Hudi with SqlQueryBasedTransformer fails-> spark error exit 134 or exit 143 in "isEmpty at DeltaSync.java:344" : Container from a bad node: container_yy on host: xxx Exit status: 134
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #4055:
URL: https://github.com/apache/hudi/issues/4055#issuecomment-989123733
@guanlisheng : Are you facing the same issue as author or is it diff?
If its diff, then here is my response.
Transformer is an interface and I see you have mentioned you are using a custom transformer. So, I guess we might have to look into the custom transformer and see whats happening.
```
@PublicAPIMethod(maturity = ApiMaturityLevel.STABLE)
Dataset<Row> apply(JavaSparkContext jsc, SparkSession sparkSession, Dataset<Row> rowDataset, TypedProperties properties);
```
Hudi just calls into this method for transformation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #4055: [SUPPORT] Hudi with SqlQueryBasedTransformer fails-> spark error exit 134 or exit 143 in "isEmpty at DeltaSync.java:344" : Container from a bad node: container_yy on host: xxx Exit status: 134
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #4055:
URL: https://github.com/apache/hudi/issues/4055#issuecomment-989459731
@guanlisheng : Can you please file a new github issue with all details and CC me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] JB-data edited a comment on issue #4055: [SUPPORT] Hudi with SqlQueryBasedTransformer fails-> spark error exit 134 or exit 143 in "isEmpty at DeltaSync.java:344" : Container from a bad node: container_yy on host: xxx Exit status: 134
Posted by GitBox <gi...@apache.org>.
JB-data edited a comment on issue #4055:
URL: https://github.com/apache/hudi/issues/4055#issuecomment-975241019
@xushiyan
Thanks for reply. Indeed the query in my ticket was rong- the sql really says FROM < SRC > ... I just corrected (so the job ran with correct statement).
Where exactly should we check the application stacktrace? In both the the yarn application logs as the SPARK UI stacktrace I only find back the fact the container exited with code 134 or 143 (changes ), but not exactly why. The only place where I can infer where this happens is the stage in the spark ui that mentions:
isEmpty at DeltaSync.java:344
so I am assuming it comes from there? Here another part of the yarn applicatino logs:
```
[2021-11-19 13:44:57.189]Container exited with a non-zero exit code 134. Error file: prelaunch.err.
Last 4096 bytes of prelaunch.err :
/bin/bash: line 1: 7907 Aborted LD_LIBRARY_PATH="/opt/cloudera/parcels/CDH-7.2.2-1.cdh7.2.2.p3.7839477/lib/hadoop/../../../CDH-7.2.2-1.cdh7.2.2.p3.7839477/lib/hadoop/lib/native:" /usr/lib/jvm/java/bin/java -server -Xmx12288m '-Djava.security.auth.login.config=./client.jaas' -Djava.io.tmpdir=/hadoopfs/fs1/nodemanager/usercache/myuser/appcache/application_1632244056069_1301/container_e29_1632244056069_1301_01_000009/tmp '-Dspark.driver.port=33466' '-Dspark.network.crypto.enabled=false' '-Dspark.authenticate=false' '-Dspark.shuffle.service.port=7337' '-Dspark.ui.port=0' -Dspark.yarn.app.container.log.dir=/hadoopfs/fs1/nodemanager/log/application_1632244056069_1301/container_e29_1632244056069_1301_01_000009 -XX:OnOutOfMemoryError='kill %p' org.apache.spark.executor.CoarseGrainedExecutorBackend --driver-url spark://CoarseGrainedScheduler@someworker0.:33466 --executor-id 8 --hostname someworkerr1 --cores 1 --app-id application_1632244056069_1301 --user-class-path file:/
hadoopfs/fs1/nodemanager/usercache/myuser/appcache/application_1632244056069_1301/container_e29_1632244056069_1301_01_000009/__app__.jar --user-class-path file:/hadoopfs/fs1/nodemanager/usercache/myuser/appcache/application_1632244056069_1301/container_e29_1632244056069_1301_01_000009/hive-service-3.1.3000.7.2.2.3-1.jar --user-class-path file:/hadoopfs/fs1/nodemanager/usercache/myuser/appcache/application_1632244056069_1301/container_e29_1632244056069_1301_01_000009/hive-jdbc-3.1.3000.7.2.2.3-1.jar --user-class-path file:/hadoopfs/fs1/nodemanager/usercache/myuser/appcache/application_1632244056069_1301/container_e29_1632244056069_1301_01_000009/hbase-client.jar > /hadoopfs/fs1/nodemanager/log/application_1632244056069_1301/container_e29_1632244056069_1301_01_000009/stdout 2> /hadoopfs/fs1/nodemanager/log/application_1632244056069_1301/container_e29_1632244056069_1301_01_000009/stderr
Last 4096 bytes of stderr :
Last 4096 bytes of stderr :
ed but isn't a known config.
21/11/19 13:44:52 WARN consumer.ConsumerConfig: The configuration 'hoodie.datasource.hive_sync.partition_fields' was supplied but isn't a known config.
21/11/19 13:44:52 WARN consumer.ConsumerConfig: The configuration 'hoodie.deltastreamer.schemaprovider.source.schema.file' was supplied but isn't a known config.
21/11/19 13:44:52 WARN consumer.ConsumerConfig: The configuration 'hoodie.deltastreamer.transformer.sql' was supplied but isn't a known config.
21/11/19 13:44:52 WARN consumer.ConsumerConfig: The configuration 'hoodie.datasource.write.recordkey.field' was supplied but isn't a known config.
21/11/19 13:44:52 WARN consumer.ConsumerConfig: The configuration 'hoodie.deltastreamer.source.kafka.topic' was supplied but isn't a known config.
21/11/19 13:44:52 WARN consumer.ConsumerConfig: The configuration 'hoodie.datasource.hive_sync.enable' was supplied but isn't a known config.
21/11/19 13:44:52 WARN consumer.ConsumerConfig: The configuration 'hoodie.datasource.hive_sync.assume_date_partitioning' was supplied but isn't a known config.
21/11/19 13:44:52 WARN consumer.ConsumerConfig: The configuration 'hoodie.datasource.hive_sync.useJdbc' was supplied but isn't a known config.
21/11/19 13:44:52 WARN consumer.ConsumerConfig: The configuration 'hoodie.deltastreamer.schemaprovider.target.schema.file' was supplied but isn't a known config.
21/11/19 13:44:52 WARN consumer.ConsumerConfig: The configuration 'hoodie.datasource.write.partitionpath.field' was supplied but isn't a known config.
21/11/19 13:44:52 WARN consumer.ConsumerConfig: The configuration 'hoodie.datasource.hive_sync.jdbcurl' was supplied but isn't a known config.
21/11/19 13:44:52 WARN consumer.ConsumerConfig: The configuration 'hoodie.datasource.hive_sync.database' was supplied but isn't a known config.
21/11/19 13:44:52 INFO utils.AppInfoParser: Kafka version: 2.4.1.7.1.1.0-565
21/11/19 13:44:52 INFO utils.AppInfoParser: Kafka commitId: 7a66ef499adcd5b9
21/11/19 13:44:52 INFO utils.AppInfoParser: Kafka startTimeMs: 1637329492101
21/11/19 13:44:52 INFO consumer.KafkaConsumer: [Consumer clientId=consumer-spark-executor-hudi-1, groupId=spark-executor-hudi] Subscribed to partition(s): fr24messages-0
21/11/19 13:44:52 INFO executor.CoarseGrainedExecutorBackend: Received tokens of 4197 bytes
21/11/19 13:44:52 INFO deploy.SparkHadoopUtil: Updating delegation tokens for current user.
21/11/19 13:44:52 INFO codegen.CodeGenerator: Code generated in 317.609485 ms
21/11/19 13:44:52 INFO codegen.CodeGenerator: Code generated in 19.735724 ms
21/11/19 13:44:52 INFO kafka010.InternalKafkaConsumer: Initial fetch for spark-executor-hudi fr24messages-0 10254239
21/11/19 13:44:52 INFO consumer.KafkaConsumer: [Consumer clientId=consumer-spark-executor-hudi-1, groupId=spark-executor-hudi] Seeking to offset 10254239 for partition fr24messages-0
21/11/19 13:44:53 INFO executor.CoarseGrainedExecutorBackend: Received tokens of 4197 bytes
21/11/19 13:44:53 INFO deploy.SparkHadoopUtil: Updating delegation tokens for current user.
21/11/19 13:44:53 INFO clients.Metadata: [Consumer clientId=consumer-spark-executor-hudi-1, groupId=spark-executor-hudi] Cluster ID: AFejYh8dRyaRfFU4x5PeMA
21/11/19 13:44:54 INFO executor.CoarseGrainedExecutorBackend: Received tokens of 4197 bytes
21/11/19 13:44:54 INFO deploy.SparkHadoopUtil: Updating delegation tokens for current user.
21/11/19 13:44:54 INFO executor.CoarseGrainedExecutorBackend: Received tokens of 4197 bytes
21/11/19 13:44:54 INFO deploy.SparkHadoopUtil: Updating delegation tokens for current user.
21/11/19 13:44:55 INFO executor.CoarseGrainedExecutorBackend: Received tokens of 4197 bytes
21/11/19 13:44:55 INFO deploy.SparkHadoopUtil: Updating delegation tokens for current user.
21/11/19 13:44:56 INFO codegen.CodeGenerator: Code generated in 208.118952 ms
21/11/19 13:44:56 INFO codegen.CodeGenerator: Code generated in 91.06291 ms
21/11/19 13:44:56 INFO executor.CoarseGrainedExecutorBackend: Received tokens of 4197 bytes
21/11/19 13:44:56 INFO deploy.SparkHadoopUtil: Updating delegation tokens for current user.
.
Driver stacktrace:
at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1931)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1919)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1918)
at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1918)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:953)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:953)
at scala.Option.foreach(Option.scala:257)
at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:953)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:2152)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2101)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2090)
at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:49)
at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:764)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:2082)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:2103)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:2122)
at org.apache.spark.rdd.RDD$$anonfun$take$1.apply(RDD.scala:1409)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
at org.apache.spark.rdd.RDD.withScope(RDD.scala:385)
at org.apache.spark.rdd.RDD.take(RDD.scala:1382)
at org.apache.spark.rdd.RDD$$anonfun$isEmpty$1.apply$mcZ$sp(RDD.scala:1517)
at org.apache.spark.rdd.RDD$$anonfun$isEmpty$1.apply(RDD.scala:1517)
at org.apache.spark.rdd.RDD$$anonfun$isEmpty$1.apply(RDD.scala:1517)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
at org.apache.spark.rdd.RDD.withScope(RDD.scala:385)
at org.apache.spark.rdd.RDD.isEmpty(RDD.scala:1516)
at org.apache.spark.api.java.JavaRDDLike$class.isEmpty(JavaRDDLike.scala:544)
at org.apache.spark.api.java.AbstractJavaRDDLike.isEmpty(JavaRDDLike.scala:45)
at org.apache.hudi.utilities.deltastreamer.DeltaSync.readFromSource(DeltaSync.java:344)
at org.apache.hudi.utilities.deltastreamer.DeltaSync.syncOnce(DeltaSync.java:233)
at org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer.lambda$sync$2(HoodieDeltaStreamer.java:161)
at org.apache.hudi.common.util.Option.ifPresent(Option.java:96)
at org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer.sync(HoodieDeltaStreamer.java:159)
at org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer.main(HoodieDeltaStreamer.java:464)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:665)
....
...
Driver stacktrace:
org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 1.0 failed 8 times, most recent failure: Lost task 0.7 in stage 1.0 (TID 8, myworker1, executor 8): ExecutorLostFailure (executor 8 exited caused by one of the running tasks) Reason: Container from a bad node: container_e29_1632244056069_1301_01_000009 on host: myworker1. Exit status: 134. Diagnostics: 2 WARN consumer.ConsumerConfig: The configuration 'hoodie.deltastreamer.transformer.sql' was supplied but isn't a known config.
21/11/19 13:44:52 WARN consumer.ConsumerConfig: The configuration 'hoodie.datasource.write.recordkey.field' was supplied but isn't a known config.
21/11/19 13:44:52 WARN consumer.ConsumerConfig: The configuration 'hoodie.deltastreamer.source.kafka.topic' was supplied but isn't a known config.
21/11/19 13:44:52 WARN consumer.ConsumerConfig: The configuration 'hoodie.datasource.hive_sync.enable' was supplied but isn't a known config.
21/11/19 13:44:52 WARN consumer.ConsumerConfig: The configuration 'hoodie.datasource.hive_sync.assume_date_partitioning' was supplied but isn't a known config.
21/11/19 13:44:52 WARN consumer.ConsumerConfig: The configuration 'hoodie.datasource.hive_sync.useJdbc' was supplied but isn't a known config.
21/11/19 13:44:52 WARN consumer.ConsumerConfig: The configuration 'hoodie.deltastreamer.schemaprovider.target.schema.file' was supplied but isn't a known config.
21/11/19 13:44:52 WARN consumer.ConsumerConfig: The configuration 'hoodie.datasource.write.partitionpath.field' was supplied but isn't a known config.
21/11/19 13:44:52 WARN consumer.ConsumerConfig: The configuration 'hoodie.datasource.hive_sync.jdbcurl' was supplied but isn't a known config.
21/11/19 13:44:52 WARN consumer.ConsumerConfig: The configuration 'hoodie.datasource.hive_sync.database' was supplied but isn't a known config.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #4055: [SUPPORT] Hudi with SqlQueryBasedTransformer fails-> spark error exit 134 or exit 143 in "isEmpty at DeltaSync.java:344" : Container from a bad node: container_yy on host: xxx Exit status: 134
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #4055:
URL: https://github.com/apache/hudi/issues/4055#issuecomment-1018553318
@guanlisheng : I see your issue is slightly different, as you claim that you could able to get it working w/ one source, but not w/ jsonKafkaSource. Can you please file a new github issue w/ all details. we can take a look.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] JB-data edited a comment on issue #4055: [SUPPORT] Hudi with SqlQueryBasedTransformer fails-> spark error exit 134 or exit 143 in "isEmpty at DeltaSync.java:344" : Container from a bad node: container_yy on host: xxx Exit status: 134
Posted by GitBox <gi...@apache.org>.
JB-data edited a comment on issue #4055:
URL: https://github.com/apache/hudi/issues/4055#issuecomment-975241019
@xushiyan
Thanks for reply. Indeed the query in my ticket was wrong- the sql really says FROM < SRC > ... I just corrected (so the job always has run with correct statement).
Where exactly should we check the application stacktrace? In both the the yarn application logs as the SPARK UI stacktrace I only find back the fact the container exited with code 134 or 143 (changes ), but not exactly why. The only place where I can infer where this happens is the stage in the spark ui that mentions:
isEmpty at DeltaSync.java:344
so I am assuming it comes from there? Here another part of the yarn applicatino logs:
```
[2021-11-19 13:44:57.189]Container exited with a non-zero exit code 134. Error file: prelaunch.err.
Last 4096 bytes of prelaunch.err :
/bin/bash: line 1: 7907 Aborted LD_LIBRARY_PATH="/opt/cloudera/parcels/CDH-7.2.2-1.cdh7.2.2.p3.7839477/lib/hadoop/../../../CDH-7.2.2-1.cdh7.2.2.p3.7839477/lib/hadoop/lib/native:" /usr/lib/jvm/java/bin/java -server -Xmx12288m '-Djava.security.auth.login.config=./client.jaas' -Djava.io.tmpdir=/hadoopfs/fs1/nodemanager/usercache/myuser/appcache/application_1632244056069_1301/container_e29_1632244056069_1301_01_000009/tmp '-Dspark.driver.port=33466' '-Dspark.network.crypto.enabled=false' '-Dspark.authenticate=false' '-Dspark.shuffle.service.port=7337' '-Dspark.ui.port=0' -Dspark.yarn.app.container.log.dir=/hadoopfs/fs1/nodemanager/log/application_1632244056069_1301/container_e29_1632244056069_1301_01_000009 -XX:OnOutOfMemoryError='kill %p' org.apache.spark.executor.CoarseGrainedExecutorBackend --driver-url spark://CoarseGrainedScheduler@someworker0.:33466 --executor-id 8 --hostname someworkerr1 --cores 1 --app-id application_1632244056069_1301 --user-class-path file:/
hadoopfs/fs1/nodemanager/usercache/myuser/appcache/application_1632244056069_1301/container_e29_1632244056069_1301_01_000009/__app__.jar --user-class-path file:/hadoopfs/fs1/nodemanager/usercache/myuser/appcache/application_1632244056069_1301/container_e29_1632244056069_1301_01_000009/hive-service-3.1.3000.7.2.2.3-1.jar --user-class-path file:/hadoopfs/fs1/nodemanager/usercache/myuser/appcache/application_1632244056069_1301/container_e29_1632244056069_1301_01_000009/hive-jdbc-3.1.3000.7.2.2.3-1.jar --user-class-path file:/hadoopfs/fs1/nodemanager/usercache/myuser/appcache/application_1632244056069_1301/container_e29_1632244056069_1301_01_000009/hbase-client.jar > /hadoopfs/fs1/nodemanager/log/application_1632244056069_1301/container_e29_1632244056069_1301_01_000009/stdout 2> /hadoopfs/fs1/nodemanager/log/application_1632244056069_1301/container_e29_1632244056069_1301_01_000009/stderr
Last 4096 bytes of stderr :
Last 4096 bytes of stderr :
ed but isn't a known config.
21/11/19 13:44:52 WARN consumer.ConsumerConfig: The configuration 'hoodie.datasource.hive_sync.partition_fields' was supplied but isn't a known config.
21/11/19 13:44:52 WARN consumer.ConsumerConfig: The configuration 'hoodie.deltastreamer.schemaprovider.source.schema.file' was supplied but isn't a known config.
21/11/19 13:44:52 WARN consumer.ConsumerConfig: The configuration 'hoodie.deltastreamer.transformer.sql' was supplied but isn't a known config.
21/11/19 13:44:52 WARN consumer.ConsumerConfig: The configuration 'hoodie.datasource.write.recordkey.field' was supplied but isn't a known config.
21/11/19 13:44:52 WARN consumer.ConsumerConfig: The configuration 'hoodie.deltastreamer.source.kafka.topic' was supplied but isn't a known config.
21/11/19 13:44:52 WARN consumer.ConsumerConfig: The configuration 'hoodie.datasource.hive_sync.enable' was supplied but isn't a known config.
21/11/19 13:44:52 WARN consumer.ConsumerConfig: The configuration 'hoodie.datasource.hive_sync.assume_date_partitioning' was supplied but isn't a known config.
21/11/19 13:44:52 WARN consumer.ConsumerConfig: The configuration 'hoodie.datasource.hive_sync.useJdbc' was supplied but isn't a known config.
21/11/19 13:44:52 WARN consumer.ConsumerConfig: The configuration 'hoodie.deltastreamer.schemaprovider.target.schema.file' was supplied but isn't a known config.
21/11/19 13:44:52 WARN consumer.ConsumerConfig: The configuration 'hoodie.datasource.write.partitionpath.field' was supplied but isn't a known config.
21/11/19 13:44:52 WARN consumer.ConsumerConfig: The configuration 'hoodie.datasource.hive_sync.jdbcurl' was supplied but isn't a known config.
21/11/19 13:44:52 WARN consumer.ConsumerConfig: The configuration 'hoodie.datasource.hive_sync.database' was supplied but isn't a known config.
21/11/19 13:44:52 INFO utils.AppInfoParser: Kafka version: 2.4.1.7.1.1.0-565
21/11/19 13:44:52 INFO utils.AppInfoParser: Kafka commitId: 7a66ef499adcd5b9
21/11/19 13:44:52 INFO utils.AppInfoParser: Kafka startTimeMs: 1637329492101
21/11/19 13:44:52 INFO consumer.KafkaConsumer: [Consumer clientId=consumer-spark-executor-hudi-1, groupId=spark-executor-hudi] Subscribed to partition(s): fr24messages-0
21/11/19 13:44:52 INFO executor.CoarseGrainedExecutorBackend: Received tokens of 4197 bytes
21/11/19 13:44:52 INFO deploy.SparkHadoopUtil: Updating delegation tokens for current user.
21/11/19 13:44:52 INFO codegen.CodeGenerator: Code generated in 317.609485 ms
21/11/19 13:44:52 INFO codegen.CodeGenerator: Code generated in 19.735724 ms
21/11/19 13:44:52 INFO kafka010.InternalKafkaConsumer: Initial fetch for spark-executor-hudi fr24messages-0 10254239
21/11/19 13:44:52 INFO consumer.KafkaConsumer: [Consumer clientId=consumer-spark-executor-hudi-1, groupId=spark-executor-hudi] Seeking to offset 10254239 for partition fr24messages-0
21/11/19 13:44:53 INFO executor.CoarseGrainedExecutorBackend: Received tokens of 4197 bytes
21/11/19 13:44:53 INFO deploy.SparkHadoopUtil: Updating delegation tokens for current user.
21/11/19 13:44:53 INFO clients.Metadata: [Consumer clientId=consumer-spark-executor-hudi-1, groupId=spark-executor-hudi] Cluster ID: AFejYh8dRyaRfFU4x5PeMA
21/11/19 13:44:54 INFO executor.CoarseGrainedExecutorBackend: Received tokens of 4197 bytes
21/11/19 13:44:54 INFO deploy.SparkHadoopUtil: Updating delegation tokens for current user.
21/11/19 13:44:54 INFO executor.CoarseGrainedExecutorBackend: Received tokens of 4197 bytes
21/11/19 13:44:54 INFO deploy.SparkHadoopUtil: Updating delegation tokens for current user.
21/11/19 13:44:55 INFO executor.CoarseGrainedExecutorBackend: Received tokens of 4197 bytes
21/11/19 13:44:55 INFO deploy.SparkHadoopUtil: Updating delegation tokens for current user.
21/11/19 13:44:56 INFO codegen.CodeGenerator: Code generated in 208.118952 ms
21/11/19 13:44:56 INFO codegen.CodeGenerator: Code generated in 91.06291 ms
21/11/19 13:44:56 INFO executor.CoarseGrainedExecutorBackend: Received tokens of 4197 bytes
21/11/19 13:44:56 INFO deploy.SparkHadoopUtil: Updating delegation tokens for current user.
.
Driver stacktrace:
at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1931)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1919)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1918)
at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1918)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:953)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:953)
at scala.Option.foreach(Option.scala:257)
at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:953)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:2152)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2101)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2090)
at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:49)
at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:764)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:2082)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:2103)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:2122)
at org.apache.spark.rdd.RDD$$anonfun$take$1.apply(RDD.scala:1409)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
at org.apache.spark.rdd.RDD.withScope(RDD.scala:385)
at org.apache.spark.rdd.RDD.take(RDD.scala:1382)
at org.apache.spark.rdd.RDD$$anonfun$isEmpty$1.apply$mcZ$sp(RDD.scala:1517)
at org.apache.spark.rdd.RDD$$anonfun$isEmpty$1.apply(RDD.scala:1517)
at org.apache.spark.rdd.RDD$$anonfun$isEmpty$1.apply(RDD.scala:1517)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
at org.apache.spark.rdd.RDD.withScope(RDD.scala:385)
at org.apache.spark.rdd.RDD.isEmpty(RDD.scala:1516)
at org.apache.spark.api.java.JavaRDDLike$class.isEmpty(JavaRDDLike.scala:544)
at org.apache.spark.api.java.AbstractJavaRDDLike.isEmpty(JavaRDDLike.scala:45)
at org.apache.hudi.utilities.deltastreamer.DeltaSync.readFromSource(DeltaSync.java:344)
at org.apache.hudi.utilities.deltastreamer.DeltaSync.syncOnce(DeltaSync.java:233)
at org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer.lambda$sync$2(HoodieDeltaStreamer.java:161)
at org.apache.hudi.common.util.Option.ifPresent(Option.java:96)
at org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer.sync(HoodieDeltaStreamer.java:159)
at org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer.main(HoodieDeltaStreamer.java:464)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:665)
....
...
Driver stacktrace:
org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 1.0 failed 8 times, most recent failure: Lost task 0.7 in stage 1.0 (TID 8, myworker1, executor 8): ExecutorLostFailure (executor 8 exited caused by one of the running tasks) Reason: Container from a bad node: container_e29_1632244056069_1301_01_000009 on host: myworker1. Exit status: 134. Diagnostics: 2 WARN consumer.ConsumerConfig: The configuration 'hoodie.deltastreamer.transformer.sql' was supplied but isn't a known config.
21/11/19 13:44:52 WARN consumer.ConsumerConfig: The configuration 'hoodie.datasource.write.recordkey.field' was supplied but isn't a known config.
21/11/19 13:44:52 WARN consumer.ConsumerConfig: The configuration 'hoodie.deltastreamer.source.kafka.topic' was supplied but isn't a known config.
21/11/19 13:44:52 WARN consumer.ConsumerConfig: The configuration 'hoodie.datasource.hive_sync.enable' was supplied but isn't a known config.
21/11/19 13:44:52 WARN consumer.ConsumerConfig: The configuration 'hoodie.datasource.hive_sync.assume_date_partitioning' was supplied but isn't a known config.
21/11/19 13:44:52 WARN consumer.ConsumerConfig: The configuration 'hoodie.datasource.hive_sync.useJdbc' was supplied but isn't a known config.
21/11/19 13:44:52 WARN consumer.ConsumerConfig: The configuration 'hoodie.deltastreamer.schemaprovider.target.schema.file' was supplied but isn't a known config.
21/11/19 13:44:52 WARN consumer.ConsumerConfig: The configuration 'hoodie.datasource.write.partitionpath.field' was supplied but isn't a known config.
21/11/19 13:44:52 WARN consumer.ConsumerConfig: The configuration 'hoodie.datasource.hive_sync.jdbcurl' was supplied but isn't a known config.
21/11/19 13:44:52 WARN consumer.ConsumerConfig: The configuration 'hoodie.datasource.hive_sync.database' was supplied but isn't a known config.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xushiyan commented on issue #4055: [SUPPORT] Hudi with SqlQueryBasedTransformer fails-> spark error exit 134 or exit 143 in "isEmpty at DeltaSync.java:344" : Container from a bad node: container_yy on host: xxx Exit status: 134
Posted by GitBox <gi...@apache.org>.
xushiyan commented on issue #4055:
URL: https://github.com/apache/hudi/issues/4055#issuecomment-974699184
@JB-data
> --hoodie-conf hoodie.deltastreamer.transformer.sql='SELECT data[0].id as id FROM '
Is this actually valid SQL?
The error you showed is not really the root cause. Can you track down to the application stacktrace and see which part of code threw exceptions?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #4055: [SUPPORT] Hudi with SqlQueryBasedTransformer fails-> spark error exit 134 or exit 143 in "isEmpty at DeltaSync.java:344" : Container from a bad node: container_yy on host: xxx Exit status: 134
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #4055:
URL: https://github.com/apache/hudi/issues/4055#issuecomment-989122167
yeah. probably schema issue. @JB-data : Can you confirm that the target schema you set in your configs matches the output from sql transformer?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #4055: [SUPPORT] Hudi with SqlQueryBasedTransformer fails-> spark error exit 134 or exit 143 in "isEmpty at DeltaSync.java:344" : Container from a bad node: container_yy on host: xxx Exit status: 134
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #4055:
URL: https://github.com/apache/hudi/issues/4055#issuecomment-1039647858
@JB-data : Will close the ticket due to no activity. we can't do much w/o further logs from your end.
But feel free to re-open if you are still looking for assistance. would be happy to help.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] guanlisheng commented on issue #4055: [SUPPORT] Hudi with SqlQueryBasedTransformer fails-> spark error exit 134 or exit 143 in "isEmpty at DeltaSync.java:344" : Container from a bad node: container_yy on host: xxx Exit status: 134
Posted by GitBox <gi...@apache.org>.
guanlisheng commented on issue #4055:
URL: https://github.com/apache/hudi/issues/4055#issuecomment-989975900
hey @nsivabalan
did not file one as I think it might be redundant with the one. and we share the same stack trace
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org