You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "Limess (via GitHub)" <gi...@apache.org> on 2023/03/08 11:08:39 UTC
[GitHub] [hudi] Limess opened a new issue, #8129: [SUPPORT] hoodie.write.lock.dynamodb.partition_key not set automatically
Limess opened a new issue, #8129:
URL: https://github.com/apache/hudi/issues/8129
**Describe the problem you faced**
When using `org.apache.hudi.aws.transaction.lock.DynamoDBBasedLockProvider` as a lock provider for Hudi, the documentation suggests that `hoodie.write.lock.dynamodb.partition_key` will by default be set to the table name if not set.
https://hudi.apache.org/docs/0.12.2/configurations/
> For DynamoDB based lock provider, the partition key for the DynamoDB lock table. Each Hudi dataset should has it's unique key so concurrent writers could refer to the same partition key. By default we use the Hudi table name specified to be the partition key
However this behaviour doesn't seem to work. When `hoodie.write.lock.dynamodb.partition_key` is not specified I see the following error:
```
Caused by: org.apache.hudi.exception.HoodieException: Unable to instantiate class org.apache.hudi.aws.transaction.lock.DynamoDBBasedLockProvider
```
due to `java.lang.IllegalArgumentException: Property hoodie.write.lock.dynamodb.partition_key not found`
**To Reproduce**
Steps to reproduce the behavior:
1. Create a Hudi job with
* `hoodie.write.lock.provider=org.apache.hudi.aws.transaction.lock.DynamoDBBasedLockProvider`
* `hoodie.write.lock.dynamodb.table=test_dynamodb_table`
* `hoodie.write.lock.dynamodb.region=us-east-1`,
* `hoodie.write.lock.dynamodb.endpoint_url=dynamodb.us-east-1.amazonaws.com`
* `hoodie.write.lock.dynamodb.billing_mode=PAY_PER_REQUEST`
* `hoodie.table.name=test_table`
Do not set `hoodie.write.lock.dynamodb.partition_key`
3. Observe that it fails with above exception
**Expected behavior**
I'd expect `hoodie.write.lock.dynamodb.partition_key` to use `hoodie.table.name` here.
**Environment Description**
* Hudi version : 0.12.2
* Spark version : 3.3.1
* Hive version : N/A
* Hadoop version : N/A
* Storage (HDFS/S3/GCS..) : S3
* Running on Docker? (yes/no) : No
**Additional context**
AWS EMR 6.10.0
**Stacktrace**
```
Caused by: org.apache.hudi.exception.HoodieException: Unable to instantiate class org.apache.hudi.aws.transaction.lock.DynamoDBBasedLockProvider at org.apache.hudi.internal.DataSourceInternalWriterHelper.commit(DataSourceInternalWriterHelper.java:87) at org.apache.hudi.spark3.internal.HoodieDataSourceInternalBatchWrite.commit(HoodieDataSourceInternalBatchWrite.java:93) at org.apache.spark.sql.execution.datasources.v2.V2TableWriteExec.writeWithV2(WriteToDataSourceV2Exec.scala:392) ... 88 more Suppressed: org.apache.hudi.exception.HoodieException: Unable to instantiate class org.apache.hudi.aws.transaction.lock.DynamoDBBasedLockProvider at org.apache.hudi.common.util.ReflectionUtils.loadClass(ReflectionUtils.java:91) at org.apache.hudi.common.util.ReflectionUtils.loadClass(ReflectionUtils.java:118) at org.apache.hudi.client.transaction.lock.LockManager.getLockProvider(LockManager.java:116) at org.apache.hudi.client.transaction.lock.LockManager.lock(LockManager.java:67) at org.a
pache.hudi.client.transaction.TransactionManager.beginTransaction(TransactionManager.java:53) at org.apache.hudi.client.BaseHoodieWriteClient.initTable(BaseHoodieWriteClient.java:1473) at org.apache.hudi.client.BaseHoodieWriteClient.initTable(BaseHoodieWriteClient.java:1508) at org.apache.hudi.client.BaseHoodieWriteClient.rollback(BaseHoodieWriteClient.java:750) at org.apache.hudi.internal.DataSourceInternalWriterHelper.abort(DataSourceInternalWriterHelper.java:95) at org.apache.hudi.spark3.internal.HoodieDataSourceInternalBatchWrite.abort(HoodieDataSourceInternalBatchWrite.java:98) at org.apache.spark.sql.execution.datasources.v2.V2TableWriteExec.writeWithV2(WriteToDataSourceV2Exec.scala:399) ... 88 more Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl
.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hudi.common.util.ReflectionUtils.loadClass(ReflectionUtils.java:89) ... 98 more Caused by: java.lang.IllegalArgumentException: Property hoodie.write.lock.dynamodb.partition_key not found at org.apache.hudi.common.config.TypedProperties.checkKey(TypedProperties.java:67) at org.apache.hudi.common.config.TypedProperties.getString(TypedProperties.java:72) at org.apache.hudi.aws.transaction.lock.DynamoDBBasedLockProvider.checkRequiredProps(DynamoDBBasedLockProvider.java:206) at org.apache.hudi.aws.transaction.lock.DynamoDBBasedLockProvider.<init>(DynamoDBBasedLockProvider.java:81) at org.apache.hudi.aws.transaction.lock.DynamoDBBasedLockProvider.<init>(DynamoDBBasedLockProvider.java:77) ... 103 more
--
Caused by: org.apache.hudi.exception.HoodieException: Unable to instantiate class org.apache.hudi.aws.transaction.lock.DynamoDBBasedLockProvider at org.apache.hudi.common.util.ReflectionUtils.loadClass(ReflectionUtils.java:91) at org.apache.hudi.common.util.ReflectionUtils.loadClass(ReflectionUtils.java:118) at org.apache.hudi.client.transaction.lock.LockManager.getLockProvider(LockManager.java:116) at org.apache.hudi.client.transaction.lock.LockManager.lock(LockManager.java:67) at org.apache.hudi.client.transaction.TransactionManager.beginTransaction(TransactionManager.java:53) at org.apache.hudi.client.BaseHoodieWriteClient.commitStats(BaseHoodieWriteClient.java:239) at org.apache.hudi.client.BaseHoodieWriteClient.commitStats(BaseHoodieWriteClient.java:223) at org.apache.hudi.internal.DataSourceInternalWriterHelper.commit(DataSourceInternalWriterHelper.java:84) ... 90 more
Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hudi.common.util.ReflectionUtils.loadClass(ReflectionUtils.java:89) ... 97 more
Caused by: java.lang.IllegalArgumentException: Property hoodie.write.lock.dynamodb.partition_key not found at org.apache.hudi.common.config.TypedProperties.checkKey(TypedProperties.java:67) at org.apache.hudi.common.config.TypedProperties.getString(TypedProperties.java:72) at org.apache.hudi.aws.transaction.lock.DynamoDBBasedLockProvider.checkRequiredProps(DynamoDBBasedLockProvider.java:206) at org.apache.hudi.aws.transaction.lock.DynamoDBBasedLockProvider.<init>(DynamoDBBasedLockProvider.java:81) at org.apache.hudi.aws.transaction.lock.DynamoDBBasedLockProvider.<init>(DynamoDBBasedLockProvider.java:77) ... 102 more
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] Limess commented on issue #8129: [SUPPORT] hoodie.write.lock.dynamodb.partition_key not set automatically
Posted by "Limess (via GitHub)" <gi...@apache.org>.
Limess commented on issue #8129:
URL: https://github.com/apache/hudi/issues/8129#issuecomment-1465144297
Yes we're setting `hoodie.table.name` in our config when running into this issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xushiyan closed issue #8129: [SUPPORT] hoodie.write.lock.dynamodb.partition_key not set automatically
Posted by "xushiyan (via GitHub)" <gi...@apache.org>.
xushiyan closed issue #8129: [SUPPORT] hoodie.write.lock.dynamodb.partition_key not set automatically
URL: https://github.com/apache/hudi/issues/8129
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] ad1happy2go commented on issue #8129: [SUPPORT] hoodie.write.lock.dynamodb.partition_key not set automatically
Posted by "ad1happy2go (via GitHub)" <gi...@apache.org>.
ad1happy2go commented on issue #8129:
URL: https://github.com/apache/hudi/issues/8129#issuecomment-1561091060
@Limess I was able to test dynamos with 0.12 and 0.13 versions of hudi with EMR and Glue (3.0 and 4.0) with the following configurations.
I did face this error when I have not provided the proper dynamo db access to IAM role.
```
'hoodie.write.lock.dynamodb.endpoint_url': 'dynamodb.us-east-2.amazonaws.com',
'hoodie.write.lock.provider': 'org.apache.hudi.aws.transaction.lock.DynamoDBBasedLockProvider',
'hoodie.write.lock.dynamodb.table': 'aditya_github_issues_testing',
'hoodie.write.lock.dynamodb.partition_key': 'tablename',
'hoodie.write.lock.dynamodb.region': 'us-east-2',
'hoodie.write.concurrency.mode':'optimistic_concurrency_control',
'hoodie.cleaner.policy.failed.writes':'LAZY'
```
Let me know if you still face this error.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #8129: [SUPPORT] hoodie.write.lock.dynamodb.partition_key not set automatically
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on issue #8129:
URL: https://github.com/apache/hudi/issues/8129#issuecomment-1465085384
@Limess :
Are you setting "hoodie.table.name" in your write configs? this is our infer fuction
```
public static final ConfigProperty<String> DYNAMODB_LOCK_PARTITION_KEY = ConfigProperty
.key(DYNAMODB_BASED_LOCK_PROPERTY_PREFIX + "partition_key")
.noDefaultValue()
.sinceVersion("0.10.0")
.withInferFunction(cfg -> {
if (cfg.contains(HoodieTableConfig.NAME)) {
return Option.of(cfg.getString(HoodieTableConfig.NAME));
}
return Option.empty();
})
.withDocumentation("For DynamoDB based lock provider, the partition key for the DynamoDB lock table. "
+ "Each Hudi dataset should has it's unique key so concurrent writers could refer to the same partition key."
+ " By default we use the Hudi table name specified to be the partition key");
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] AmareshB commented on issue #8129: [SUPPORT] hoodie.write.lock.dynamodb.partition_key not set automatically
Posted by "AmareshB (via GitHub)" <gi...@apache.org>.
AmareshB commented on issue #8129:
URL: https://github.com/apache/hudi/issues/8129#issuecomment-1624564941
Is this fix part of any release yet?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] ad1happy2go commented on issue #8129: [SUPPORT] hoodie.write.lock.dynamodb.partition_key not set automatically
Posted by "ad1happy2go (via GitHub)" <gi...@apache.org>.
ad1happy2go commented on issue #8129:
URL: https://github.com/apache/hudi/issues/8129#issuecomment-1597584502
@Limess
The fix is merged now https://github.com/apache/hudi/pull/8868
I validated the fix and it's working fine now without any partition key. Using all defaults and infer functions defined in the Config class.
Closing the issue. Please reopen in case you see any more issues. Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] ad1happy2go commented on issue #8129: [SUPPORT] hoodie.write.lock.dynamodb.partition_key not set automatically
Posted by "ad1happy2go (via GitHub)" <gi...@apache.org>.
ad1happy2go commented on issue #8129:
URL: https://github.com/apache/hudi/issues/8129#issuecomment-1566150996
@Limess Able to reproduce this. If we don't set `hoodie.write.lock.dynamodb.partition_key` then the job is failing with error @Limess is facing.
Created jira for the same - https://issues.apache.org/jira/browse/HUDI-6278
Code to reproduce - https://gist.github.com/ad1happy2go/fbb00da466929412282d6dc233f68da8
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org