You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2020/10/27 07:46:27 UTC
[GitHub] [hudi] liujinhui1994 opened a new issue #2209: [SUPPORT] Deltastreamer uses parquetDFSSource OOM
liujinhui1994 opened a new issue #2209:
URL: https://github.com/apache/hudi/issues/2209
**_Tips before filing an issue_**
- Have you gone through our [FAQs](https://cwiki.apache.org/confluence/display/HUDI/FAQ)?
- Join the mailing list to engage in conversations and get faster support at dev-subscribe@hudi.apache.org.
- If you have triaged this as a bug, then file an [issue](https://issues.apache.org/jira/projects/HUDI/issues) directly.
**Describe the problem you faced**
A clear and concise description of the problem.
**To Reproduce**
Steps to reproduce the behavior:
hoodie.deltastreamer.source.dfs.root
There are only two parquet files in this configuration, the size is about 2.5MB
This is my start command
spark-submit --jars hdfs://nameservice1/data_lake/jars/hive-service-2.1.1.jar,hdfs://nameservice1/data_lake/jars/hive-jdbc-2.1.1.jar \
--master yarn \
--name t3_science.t_short_url \
--driver-memory 2G \
--num-executors 1 \
--executor-memory 2G \
--executor-cores 1 \
--deploy-mode cluster \
--conf spark.driver.userClassPathFirst=true \
--conf spark.dynamicAllocation.enabled=false \
--conf spark.yarn.submit.waitAppCompletion=false \
--class org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer hdfs://nameservice1/data_lake/jars/data_lake.jar \
--schemaprovider-class org.apache.hudi.utilities.schema.FilebasedSchemaProvider \
--source-class org.apache.hudi.utilities.sources.ParquetDFSSource \
--target-base-path hdfs://nameservice1/data_lake/data/full/t3_science.t_short_url/20201032 \
--op BULK_INSERT \
--target-table t3_science.t_short_url_4 \
--table-type COPY_ON_WRITE \
--source-limit 5242880 \
--props hdfs://nameservice1/data_lake/config/full/t3_science.t_short_url/dfs-source.properties \
**Stacktrace**
20/10/27 15:33:29 INFO util.ShutdownHookManager: Shutdown hook called
20/10/27 15:33:29 INFO executor.Executor: Not reporting error to driver during JVM shutdown.
20/10/27 15:33:29 ERROR util.SparkUncaughtExceptionHandler: [Container in shutdown] Uncaught exception in thread Thread[Executor task launch worker for task 1,5,main]
java.lang.OutOfMemoryError: Java heap space
at java.lang.StringCoding.decode(StringCoding.java:215)
at java.lang.String.<init>(String.java:463)
at java.lang.String.<init>(String.java:515)
at org.apache.spark.unsafe.types.UTF8String.toString(UTF8String.java:1214)
at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificSafeProjection.createExternalRow_0_0$(Unknown Source)
at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificSafeProjection.apply(Unknown Source)
at org.apache.spark.sql.catalyst.encoders.ExpressionEncoder.fromRow(ExpressionEncoder.scala:302)
at org.apache.hudi.AvroConversionUtils$$anonfun$1.apply(AvroConversionUtils.scala:44)
at org.apache.hudi.AvroConversionUtils$$anonfun$1.apply(AvroConversionUtils.scala:44)
at scala.collection.Iterator$$anon$11.next(Iterator.scala:410)
at scala.collection.Iterator$$anon$11.next(Iterator.scala:410)
at scala.collection.Iterator$$anon$10.next(Iterator.scala:394)
at scala.collection.Iterator$class.foreach(Iterator.scala:891)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1334)
at scala.collection.generic.Growable$class.$plus$plus$eq(Growable.scala:59)
at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:104)
at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:48)
at scala.collection.TraversableOnce$class.to(TraversableOnce.scala:310)
at scala.collection.AbstractIterator.to(Iterator.scala:1334)
at scala.collection.TraversableOnce$class.toBuffer(TraversableOnce.scala:302)
at scala.collection.AbstractIterator.toBuffer(Iterator.scala:1334)
at scala.collection.TraversableOnce$class.toArray(TraversableOnce.scala:289)
at scala.collection.AbstractIterator.toArray(Iterator.scala:1334)
at org.apache.spark.rdd.RDD$$anonfun$take$1$$anonfun$29.apply(RDD.scala:1364)
at org.apache.spark.rdd.RDD$$anonfun$take$1$$anonfun$29.apply(RDD.scala:1364)
at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:2121)
at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:2121)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
at org.apache.spark.scheduler.Task.run(Task.scala:121)
at org.apache.spark.executor.Executor$TaskRunner$$anonfun$11.apply(Executor.scala:407)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1408)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:413)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] youngyangp commented on issue #2209: [SUPPORT] Deltastreamer uses parquetDFSSource OOM
Posted by GitBox <gi...@apache.org>.
youngyangp commented on issue #2209:
URL: https://github.com/apache/hudi/issues/2209#issuecomment-773795510
怎么解决的
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] liujinhui1994 closed issue #2209: [SUPPORT] Deltastreamer uses parquetDFSSource OOM
Posted by GitBox <gi...@apache.org>.
liujinhui1994 closed issue #2209:
URL: https://github.com/apache/hudi/issues/2209
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] youngyangp commented on issue #2209: [SUPPORT] Deltastreamer uses parquetDFSSource OOM
Posted by GitBox <gi...@apache.org>.
youngyangp commented on issue #2209:
URL: https://github.com/apache/hudi/issues/2209#issuecomment-773795510
怎么解决的
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org