You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2022/10/28 03:17:57 UTC
[GitHub] [hudi] mandnhdaiyudfaio opened a new issue, #7081: [SUPPORT] optimistic_concurrency_control
mandnhdaiyudfaio opened a new issue, #7081:
URL: https://github.com/apache/hudi/issues/7081
I need to use the concurrent write control function (occ lock) of hudi.
Official documents（ https://hudi.apache.org/docs/concurrency_control ）The demo provided is as follows:
inputDF.write.format("hudi")
.options(getQuickstartWriteConfigs)
.option(PRECOMBINE_FIELD_OPT_KEY, "ts")
.option("hoodie.cleaner.policy.failed.writes", "LAZY")
.option("hoodie.write.concurrency.mode", "optimistic_concurrency_control")
.option("hoodie.write.lock.zookeeper.url", "zookeeper")
.option("hoodie.write.lock.zookeeper.port", "2181")
.option("hoodie.write.lock.zookeeper.lock_key", "test_table")
.option("hoodie.write.lock.zookeeper.base_path", "/test")
.option(RECORDKEY_FIELD_OPT_KEY, "uuid")
.option(PARTITIONPATH_FIELD_OPT_KEY, "partitionpath")
.option(TABLE_NAME, tableName)
.mode(Overwrite)
.save(basePath)
According to the demo of the official website document, I will translate hoodie.cleaner.policy.failed.writes, hoodie.write.contract.mode, hoodie.write.lock.zookeeper.url, hoodie.write.lock.zookeeper.port, hoodie.write.lock.zookeeper.lock_ key、hoodie.write.lock.zookeeper. base_ The five parameters, path, are written into the options parameter.
My demo is as follows:
spark. sql("create table da (id int, ts bigint, actid int, day string)
using hudi tblproperties(type = 'cow', primaryKey = 'id,actid', preCombineField = 'ts')
options (hoodie.write.lock.zookeeper.url = 'node4,node6,node7',
hoodie.write.lock.zookeeper.port = '2181',
hoodie.write.lock.zookeeper.lock_key = 'da',
hoodie.write.lock.zookeeper.base_path = '/hudi/lock/table',
hoodie.write.concurrency.mode = 'optimistic_concurrency_control',
hoodie.cleaner.policy.failed.writes = 'LAZY')
partitioned by (day) location '/user/hudi/warehouse/ods_ dev/da'");
Question:
After the da table is created successfully, the data can be written and read normally. However, I went to the zoo keeper and found that the zoo keeper failed to create the node:/hudi/lock/table.
Even if the official website's demo writing method inputDF. write. format ("hudi")... save (basePath) is used directly, data can be successfully written to the hudi table, but it is also not found that zookeeper successfully creates nodes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #7081: [SUPPORT] optimistic_concurrency_control
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #7081:
URL: https://github.com/apache/hudi/issues/7081#issuecomment-1295674980
did you set
```
hoodie.write.lock.provider=org.apache.hudi.client.transaction.lock.ZookeeperBasedLockProvider
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codope commented on issue #7081: [SUPPORT] optimistic_concurrency_control
Posted by GitBox <gi...@apache.org>.
codope commented on issue #7081:
URL: https://github.com/apache/hudi/issues/7081#issuecomment-1348737794
Closing due to inactivity. Please reopen if you're still facing the issue after suggested config.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #7081: [SUPPORT] optimistic_concurrency_control
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #7081:
URL: https://github.com/apache/hudi/issues/7081#issuecomment-1305144034
@mandnhdaiyudfaio : do you have any updates for us.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codope closed issue #7081: [SUPPORT] optimistic_concurrency_control
Posted by GitBox <gi...@apache.org>.
codope closed issue #7081: [SUPPORT] optimistic_concurrency_control
URL: https://github.com/apache/hudi/issues/7081
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on issue #7081: [SUPPORT] optimistic_concurrency_control
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #7081:
URL: https://github.com/apache/hudi/issues/7081#issuecomment-1299563321
@mandnhdaiyudfaio : gentle ping. let us know after you give it a try w/ the lock provider config.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org