You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "imuntyan (via GitHub)" <gi...@apache.org> on 2023/02/01 19:00:09 UTC
[GitHub] [hudi] imuntyan opened a new issue, #7820: [SUPPORT] Errors are thrown when upserting a record with cleaner service enabled
imuntyan opened a new issue, #7820:
URL: https://github.com/apache/hudi/issues/7820
Enabling the hudi cleaner service (sync or async) throws an error when trying to upsert a record in spark append mode.
**To Reproduce**
I am running the following script in the Jupyter notebook:
```python
from numpy import random
hudi_mode, spark_mode = 'upsert', 'append'
insert_hudi_path = "s3://[redacted]/hudi/test01"
data = [
{"id": f'id-{random.randint(1000000000)}', "text": f'text-{random.randint(1000000000)}'}
]
df = spark.createDataFrame(data)
table_name = 'table_test01'
primary_key="id"
precombine="text"
hudi_options = {
'hoodie.table.name': table_name,
'hoodie.datasource.write.operation': hudi_mode,
'hoodie.datasource.write.recordkey.field': primary_key,
'hoodie.datasource.write.precombine.field': precombine,
'hoodie.metadata.enable': True,
'hoodie.clean.automatic': True,
# 'hoodie.clean.async': True,
'hoodie.cleaner.commits.retained': 1,
'hoodie.datasource.write.keygenerator.class': 'org.apache.hudi.keygen.NonpartitionedKeyGenerator'
}
df.write.format('hudi') \
.options(**hudi_options) \
.mode(spark_mode) \
.save(insert_hudi_path)
```
When the S3 location is empty this script executes fine and creates the data in S3. When executing it again, it throws the errors attached below. The errors are thrown for both sync and async cleaner mode (the async mode throws the errors on the third run though).
The errors are not returned when the following configuration is commented out:
```
# 'hoodie.clean.automatic': True,
# 'hoodie.clean.async': True,
# 'hoodie.cleaner.commits.retained': 1,
```
**Expected behavior**
No errors.
**Environment Description**
* Hudi version : 0.12.1-amzn-0-SNAPSHOT
* Spark version : 3.3.0
* Hive version : 3.1.3
* Hadoop version : 3.3.3
* AWS EMR version: emr-6.9.0
* Storage (HDFS/S3/GCS..) : S3
* Running on Docker? (yes/no) : EMR on EC2
**Stacktrace**
[emr-errors.txt](https://github.com/apache/hudi/files/10560680/emr-errors.txt)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xushiyan commented on issue #7820: [SUPPORT] Errors are thrown when upserting a record with cleaner service enabled
Posted by "xushiyan (via GitHub)" <gi...@apache.org>.
xushiyan commented on issue #7820:
URL: https://github.com/apache/hudi/issues/7820#issuecomment-1428882771
> Update 2: We believe the issue is related to having the CSE (client-side encryption) on the cluster and does not appear when the encryption is not enabled.
@imuntyan Any further finding on this direction? would you be able to confirm the root cause?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] josh-mantovani commented on issue #7820: [SUPPORT] Errors are thrown when upserting a record with cleaner service enabled
Posted by "josh-mantovani (via GitHub)" <gi...@apache.org>.
josh-mantovani commented on issue #7820:
URL: https://github.com/apache/hudi/issues/7820#issuecomment-1440218403
> > Update 2: We believe the issue is related to having the CSE (client-side encryption) on the cluster and does not appear when the encryption is not enabled.
>
>
>
> @imuntyan Any further finding on this direction? would you be able to confirm the root cause?
Hi @xushiyan, do you need anything else from us at this point to replicate this issue?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] umehrot2 commented on issue #7820: [SUPPORT] Errors are thrown when upserting a record with cleaner service enabled
Posted by "umehrot2 (via GitHub)" <gi...@apache.org>.
umehrot2 commented on issue #7820:
URL: https://github.com/apache/hudi/issues/7820#issuecomment-1451174235
I will close this issue out as this is not related to Hudi.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] imuntyan commented on issue #7820: [SUPPORT] Errors are thrown when upserting a record with cleaner service enabled
Posted by "imuntyan (via GitHub)" <gi...@apache.org>.
imuntyan commented on issue #7820:
URL: https://github.com/apache/hudi/issues/7820#issuecomment-1421401075
Update: I also see the `java.util.ConcurrentModificationException` when logging into the master node and using HUDI CLI to run
```
cleans run --sparkMaster local --hoodieConfigs hoodie.cleaner.policy=KEEP_LATEST_COMMITS hoodie.cleaner.commits.retained=3 hoodie.cleaner.parallelism=200
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #7820: [SUPPORT] Errors are thrown when upserting a record with cleaner service enabled
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on issue #7820:
URL: https://github.com/apache/hudi/issues/7820#issuecomment-1418500594
I can try and reproduce the issue. Can you try bumping the cleaner commits retained to 5 and see if you are seeing the issue still. Generally its not advisable to set the value of this config to 1. your readers might run into issues.
More details are covered [here](https://hudi.apache.org/blog/2021/06/10/employing-right-configurations-for-hudi-cleaner/#deeper-dive-into-hudi-cleaner) on how to deduce the right value for this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] umehrot2 closed issue #7820: [SUPPORT] Errors are thrown when upserting a record with cleaner service enabled
Posted by "umehrot2 (via GitHub)" <gi...@apache.org>.
umehrot2 closed issue #7820: [SUPPORT] Errors are thrown when upserting a record with cleaner service enabled
URL: https://github.com/apache/hudi/issues/7820
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] imuntyan commented on issue #7820: [SUPPORT] Errors are thrown when upserting a record with cleaner service enabled
Posted by "imuntyan (via GitHub)" <gi...@apache.org>.
imuntyan commented on issue #7820:
URL: https://github.com/apache/hudi/issues/7820#issuecomment-1419423226
I repeated the same test for `'hoodie.cleaner.commits.retained': 5`, the error appeared on the 6th run. Attached the 6th run logs
[hudi_sync_commits5_run6_driver_stderr.txt](https://github.com/apache/hudi/files/10665478/hudi_sync_commits5_run6_driver_stderr.txt)
[hudi_sync_commits5_run6_driver_stdout.txt](https://github.com/apache/hudi/files/10665479/hudi_sync_commits5_run6_driver_stdout.txt)
[hudi_sync_commits5_run6_exec1_stderr.txt](https://github.com/apache/hudi/files/10665480/hudi_sync_commits5_run6_exec1_stderr.txt)
[hudi_sync_commits5_run6_exec1_stdout.txt](https://github.com/apache/hudi/files/10665481/hudi_sync_commits5_run6_exec1_stdout.txt)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] josh-mantovani commented on issue #7820: [SUPPORT] Errors are thrown when upserting a record with cleaner service enabled
Posted by "josh-mantovani (via GitHub)" <gi...@apache.org>.
josh-mantovani commented on issue #7820:
URL: https://github.com/apache/hudi/issues/7820#issuecomment-1428921763
> > Update 2: We believe the issue is related to having the CSE (client-side encryption) on the cluster and does not appear when the encryption is not enabled.
>
>
>
> @imuntyan Any further finding on this direction? would you be able to confirm the root cause?
Hi @xushiyan , I'm tagging in for Igor here as we work on the same team. After much investigation, we've discovered that the issue only exists when CSE is enabled AND we're using the non partitioned key generator in Hudi. If we disable CSE in EMR OR use another key generator, the issue goes away. It appears as if there may be a bug in either the CSE implementation in EMR or the non partitioned key generator implementation in Hudi - I'm unable to determine which.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] umehrot2 commented on issue #7820: [SUPPORT] Errors are thrown when upserting a record with cleaner service enabled
Posted by "umehrot2 (via GitHub)" <gi...@apache.org>.
umehrot2 commented on issue #7820:
URL: https://github.com/apache/hudi/issues/7820#issuecomment-1451173914
@imuntyan @josh-mantovani I am from Amazon EMR Hudi team. We have reproduced the issue and have been able to identify that this is an issue with EmrFS when client side encryption is enabled, and not a bug with Hudi as such. We have forwarded your ticket to the EmrFS team, and will update you once they investigate and able to narrow it down. I suggest to continue working with AWS support and thank you for your patience.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org