You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "andreagarcia20 (via GitHub)" <gi...@apache.org> on 2023/03/27 15:18:47 UTC
[GitHub] [hudi] andreagarcia20 opened a new issue, #8302: [SUPPORT] Hudi Job fails when trying to upsert/insert data due to Executor Process Lost
andreagarcia20 opened a new issue, #8302:
URL: https://github.com/apache/hudi/issues/8302
**Problem Description**
Hi team,
We are trying to construct a Hudi application to run daily (as a batch job) to incrementally update data when new information arrives. This is our first time working with Hudi and some issues appear when doing experiments during the deployment. Currently, we have an input data of 11GB divided into 627 parquet files and we are working with two scenarios:
- 100% of inserts
- 100% of updates
Every time we try to run a complete execution (inserts or updates) we obtain an error of Executor Process Lost until all the executors die and the job fails. The failure always occurs on "Building Workload profile" stage (more info below)
We are using AWS EMR with 1 master node (m5.2xlarge) and 4 instances (r4.8xlarge)
**Hudi Options**
hudiOptions = {
"hoodie.table.name": "F5",
"hoodie.datasource.write.table.type": "MERGE_ON_READ",
"hoodie.datasource.write.recordkey.field": "hour,cups",
"hoodie.datasource.write.partitionpath.field": "subsystem,year,month,day",
"hoodie.datasource.write.precombine.field": "dedup",
"hoodie.datasource.write.hive_style_partitioning": "true",
"hoodie.datasource.write.drop.partition.columns": "true",
"hooddie.compact.inline": "true",
"hoodie.datasource.compaction.async.enable": "false",
"hoodie.compact.inline.max.delta.commits": 1,
"hoodie.cleaner.policy": "KEEP_LATEST_FILE_VERSIONS",
"hoodie.cleaner.fileversions.retained": 1,
"hoodie.datasource.write.keygenerator.class": "org.apache.hudi.keygen.ComplexKeyGenerator",
"hoodie.datasource.hive_sync.partition_extractor_class": "org.apache.hudi.hive.MultiPartKeysValueExtractor",
"hoodie.datasource.write.payload.class": "org.apache.hudi.common.model.DefaultHoodieRecordPayload",
"hoodie.index.type": "BLOOM",
"hoodie.bloom.index.filter.type": "DYNAMIC_V0",
"hoodie.upsert.shuffle.parallelism": 500,
"hoodie.metadata.enable": "true",
"hoodie.metadata.index.column.stats.enable": "true",
"hoodie.enable.data.skipping": "true"
}
To write the data we always use the upsert mode: _.option('hoodie.datasource.write.operation', 'upsert')_
**Spark-submit**
--jars /usr/lib/hudi/hudi-spark-bundle.jar
--conf spark.serializer=org.apache.spark.serializer.KryoSerializer
--conf spark.sql.catalog.spark_catalog=org.apache.spark.sql.hudi.catalog.HoodieCatalog
--conf spark.sql.extensions=org.apache.spark.sql.hudi.HoodieSparkSessionExtension
--conf spark.emr.maximizeResourceAllocation=true
**Environment Description**
* Hudi version : 0.12.2-amzn-0
* Spark version: 3.3.1
* Hive version : 3.1.3
* Hadoop version : Amazon 3.3.3
* Storage : S3
* Running on Docker? (yes/no) : No
**Additional context**
We have been playing with both spark and hudi configurations (memory, number of executors/cores, parallelism...) but nothing seems to work for us. Any suggestion or improvement in our configuration is welcome.
**SparkUI information**
<img width="929" alt="image" src="https://user-images.githubusercontent.com/129081554/227980692-aa9830fe-a28b-4f8d-9dc7-5ae094772838.png">
<img width="944" alt="image" src="https://user-images.githubusercontent.com/129081554/227980959-af418b5e-4e75-4c14-bbfe-7451d37ef94e.png">
<img width="931" alt="image" src="https://user-images.githubusercontent.com/129081554/227981100-a6d9053a-c687-497b-83e3-79db87eeb266.png">
<img width="929" alt="image" src="https://user-images.githubusercontent.com/129081554/227981324-b95fc73c-6eb9-4333-97f4-065a5a072987.png">
<img width="935" alt="image" src="https://user-images.githubusercontent.com/129081554/227981497-1390b612-d5ad-48e6-8d60-5540e6ddfa60.png">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] ad1happy2go commented on issue #8302: [SUPPORT] Hudi Job fails when trying to upsert/insert data due to Executor Process Lost
Posted by "ad1happy2go (via GitHub)" <gi...@apache.org>.
ad1happy2go commented on issue #8302:
URL: https://github.com/apache/hudi/issues/8302#issuecomment-1493595411
@andreagarcia20 This step will be also do the index lookup which is resulting in lots of spill to disk and ultimately failing the jobs.
Can you try to use SIMPLE index and see if you are facing the same issue.
Also is the key very randomly distributed across the data set.
Also check if there is a skew in the dataset as lots of disk spilling is happening so lots of data going into few partitions. Check the amount of data being written by successful tasks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] andreagarcia20 commented on issue #8302: [SUPPORT] Hudi Job fails when trying to upsert/insert data due to Executor Process Lost
Posted by "andreagarcia20 (via GitHub)" <gi...@apache.org>.
andreagarcia20 commented on issue #8302:
URL: https://github.com/apache/hudi/issues/8302#issuecomment-1486413916
Here you have one example:
[stderr.txt](https://github.com/apache/hudi/files/11086956/stderr.txt)
[stdout.txt](https://github.com/apache/hudi/files/11086957/stdout.txt)
Thank you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] ad1happy2go commented on issue #8302: [SUPPORT] Hudi Job fails when trying to upsert/insert data due to Executor Process Lost
Posted by "ad1happy2go (via GitHub)" <gi...@apache.org>.
ad1happy2go commented on issue #8302:
URL: https://github.com/apache/hudi/issues/8302#issuecomment-1569770974
@andreagarcia20 I tried to reproduce but unable to hit this issue. Were you able to resolve this issue? If yes, Please let us know the resolution.
If not can you try to reproduce this using the mockup data and share us.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Hudi Job fails when trying to upsert/insert data due to Executor Process Lost [hudi]
Posted by "Gatsby-Lee (via GitHub)" <gi...@apache.org>.
Gatsby-Lee commented on issue #8302:
URL: https://github.com/apache/hudi/issues/8302#issuecomment-2264073823
I guess I am hitting the similar issue as well.
not OOM, but high GC time
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] andreagarcia20 commented on issue #8302: [SUPPORT] Hudi Job fails when trying to upsert/insert data due to Executor Process Lost
Posted by "andreagarcia20 (via GitHub)" <gi...@apache.org>.
andreagarcia20 commented on issue #8302:
URL: https://github.com/apache/hudi/issues/8302#issuecomment-1486362145
The error of the executors when failing is: _ExecutorLostFailure (executor 8 exited caused by one of the running tasks) Reason: Executor Process Lost_
<img width="932" alt="image" src="https://user-images.githubusercontent.com/129081554/228160566-84ae057c-d7cd-4cd2-a543-98bce1b33e80.png">
We are using 32 executor-cores and the executor-memory is by default 230051M (we have tried to increase and decrease these parameters but we do not know exactly how this affects our workload)
<img width="691" alt="image" src="https://user-images.githubusercontent.com/129081554/228161407-bfe0af97-eac3-443a-b614-3ebcb113b118.png">
We also have an spark.default.parallelism equal to cores x instances x 2 (in this case 256)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] ad1happy2go commented on issue #8302: [SUPPORT] Hudi Job fails when trying to upsert/insert data due to Executor Process Lost
Posted by "ad1happy2go (via GitHub)" <gi...@apache.org>.
ad1happy2go commented on issue #8302:
URL: https://github.com/apache/hudi/issues/8302#issuecomment-1486326376
@andreagarcia20 Can you check why we are getting failures in the Building Workload profile step. Also let us know what executor-memory and executor-cores you are trying
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] ad1happy2go commented on issue #8302: [SUPPORT] Hudi Job fails when trying to upsert/insert data due to Executor Process Lost
Posted by "ad1happy2go (via GitHub)" <gi...@apache.org>.
ad1happy2go commented on issue #8302:
URL: https://github.com/apache/hudi/issues/8302#issuecomment-1486372317
Can you send the stdout and stderr file on one of the failed task.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] andreagarcia20 commented on issue #8302: [SUPPORT] Hudi Job fails when trying to upsert/insert data due to Executor Process Lost
Posted by "andreagarcia20 (via GitHub)" <gi...@apache.org>.
andreagarcia20 commented on issue #8302:
URL: https://github.com/apache/hudi/issues/8302#issuecomment-1503094445
Hi @ad1happy2go! Thanks for the reply.
Can you give us more detail about the nature of our key? We do not understand why our key is randomly distributed.
In our case, we receive everyday data from the last 12 months at the hourly level, having updates distributed over this period. So, the unique way to identify a record is by having the complete timestamp (year-month-day-hour) plus the 'id'. How should we construct our key and partitions to make our job more convenient?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] ad1happy2go commented on issue #8302: [SUPPORT] Hudi Job fails when trying to upsert/insert data due to Executor Process Lost
Posted by "ad1happy2go (via GitHub)" <gi...@apache.org>.
ad1happy2go commented on issue #8302:
URL: https://github.com/apache/hudi/issues/8302#issuecomment-1531696669
@andreagarcia20 Sorry for the delayed response. Keys looks okay as they will not be randomly distributed. Did you tried to use SIMPLE index?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org