You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2022/09/22 20:59:03 UTC

[GitHub] [hudi] yihua opened a new pull request, #6752: [HUDI-4010] Improve docs on DynamoDB-based lock provider

yihua opened a new pull request, #6752:
URL: https://github.com/apache/hudi/pull/6752

   ### Change Logs
   
   This PR improves docs on DynamoDB-based lock provider to avoid confusion on setting the configs.
   
   ### Impact
   
   **Risk level: none**
   
   The website can be built and visualized with `npm start`.
   
   ### Contributor's checklist
   
   - [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
   - [ ] Change Logs and Impact were stated clearly
   - [ ] Adequate tests were added if applicable
   - [ ] CI passed
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] yihua commented on pull request #6752: [HUDI-4010] Improve docs on DynamoDB-based lock provider

Posted by GitBox <gi...@apache.org>.
yihua commented on PR #6752:
URL: https://github.com/apache/hudi/pull/6752#issuecomment-1255664623

   cc @umehrot2 @rahil-c @CTTY 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] yihua commented on a diff in pull request #6752: [HUDI-4010] Improve docs on DynamoDB-based lock provider

Posted by GitBox <gi...@apache.org>.
yihua commented on code in PR #6752:
URL: https://github.com/apache/hudi/pull/6752#discussion_r979130372


##########
website/docs/concurrency_control.md:
##########
@@ -71,16 +71,26 @@ hoodie.write.lock.hivemetastore.table
 
 **`Amazon DynamoDB`** based lock provider
 
-Amazon DynamoDB based lock provides a simple way to support multi writing across different clusters
+Amazon DynamoDB based lock provides a simple way to support multi writing across different clusters.  You can refer to the
+[DynamoDB based Locks Configurations](https://hudi.apache.org/docs/configurations#DynamoDB-based-Locks-Configurations)
+section for the details of each related configuration knob.
 
 ```
 hoodie.write.lock.provider=org.apache.hudi.aws.transaction.lock.DynamoDBBasedLockProvider
-hoodie.write.lock.dynamodb.table
-hoodie.write.lock.dynamodb.partition_key
-hoodie.write.lock.dynamodb.region
-hoodie.write.lock.dynamodb.endpoint_url
-hoodie.write.lock.dynamodb.billing_mode
-```
+hoodie.write.lock.dynamodb.table (required)
+hoodie.write.lock.dynamodb.partition_key (optional)
+hoodie.write.lock.dynamodb.region (optional)
+hoodie.write.lock.dynamodb.endpoint_url (optional)
+hoodie.write.lock.dynamodb.billing_mode (optional)
+```
+
+When using the DynamoDB-based lock provider, the name of the DynamoDB table acting as the lock table for Hudi is
+specified by the config `hoodie.write.lock.dynamodb.table`. This DynamoDB table is automatically created by Hudi, so you
+don't have to create the table yourself. If you want to use an existing DynamoDB table, make sure that the `key` column

Review Comment:
   Makes sense.  Fixed.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] yihua commented on a diff in pull request #6752: [HUDI-4010] Improve docs on DynamoDB-based lock provider

Posted by GitBox <gi...@apache.org>.
yihua commented on code in PR #6752:
URL: https://github.com/apache/hudi/pull/6752#discussion_r979130710


##########
website/docs/concurrency_control.md:
##########
@@ -71,16 +71,26 @@ hoodie.write.lock.hivemetastore.table
 
 **`Amazon DynamoDB`** based lock provider
 
-Amazon DynamoDB based lock provides a simple way to support multi writing across different clusters
+Amazon DynamoDB based lock provides a simple way to support multi writing across different clusters.  You can refer to the
+[DynamoDB based Locks Configurations](https://hudi.apache.org/docs/configurations#DynamoDB-based-Locks-Configurations)
+section for the details of each related configuration knob.
 
 ```
 hoodie.write.lock.provider=org.apache.hudi.aws.transaction.lock.DynamoDBBasedLockProvider
-hoodie.write.lock.dynamodb.table
-hoodie.write.lock.dynamodb.partition_key
-hoodie.write.lock.dynamodb.region
-hoodie.write.lock.dynamodb.endpoint_url
-hoodie.write.lock.dynamodb.billing_mode
-```
+hoodie.write.lock.dynamodb.table (required)
+hoodie.write.lock.dynamodb.partition_key (optional)
+hoodie.write.lock.dynamodb.region (optional)
+hoodie.write.lock.dynamodb.endpoint_url (optional)
+hoodie.write.lock.dynamodb.billing_mode (optional)

Review Comment:
   Good catch!  They should be applied to versioned_docs as well.  Fixed.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] vamshigv commented on a diff in pull request #6752: [HUDI-4010] Improve docs on DynamoDB-based lock provider

Posted by GitBox <gi...@apache.org>.
vamshigv commented on code in PR #6752:
URL: https://github.com/apache/hudi/pull/6752#discussion_r978193381


##########
website/docs/concurrency_control.md:
##########
@@ -71,16 +71,26 @@ hoodie.write.lock.hivemetastore.table
 
 **`Amazon DynamoDB`** based lock provider
 
-Amazon DynamoDB based lock provides a simple way to support multi writing across different clusters
+Amazon DynamoDB based lock provides a simple way to support multi writing across different clusters.  You can refer to the
+[DynamoDB based Locks Configurations](https://hudi.apache.org/docs/configurations#DynamoDB-based-Locks-Configurations)
+section for the details of each related configuration knob.
 
 ```
 hoodie.write.lock.provider=org.apache.hudi.aws.transaction.lock.DynamoDBBasedLockProvider
-hoodie.write.lock.dynamodb.table
-hoodie.write.lock.dynamodb.partition_key
-hoodie.write.lock.dynamodb.region
-hoodie.write.lock.dynamodb.endpoint_url
-hoodie.write.lock.dynamodb.billing_mode
-```
+hoodie.write.lock.dynamodb.table (required)
+hoodie.write.lock.dynamodb.partition_key (optional)
+hoodie.write.lock.dynamodb.region (optional)
+hoodie.write.lock.dynamodb.endpoint_url (optional)
+hoodie.write.lock.dynamodb.billing_mode (optional)
+```
+
+When using the DynamoDB-based lock provider, the name of the DynamoDB table acting as the lock table for Hudi is
+specified by the config `hoodie.write.lock.dynamodb.table`. This DynamoDB table is automatically created by Hudi, so you
+don't have to create the table yourself. If you want to use an existing DynamoDB table, make sure that the `key` column
+is present in the table, used as the partition key.  The config `hoodie.write.lock.dynamodb.partition_key` specifies the

Review Comment:
   may be we can remove `used as the partition key` as you explain in the next sentence and partition key here has a different meaning than what we use it for generally ?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] yihua merged pull request #6752: [HUDI-4010] Improve docs on DynamoDB-based lock provider

Posted by GitBox <gi...@apache.org>.
yihua merged PR #6752:
URL: https://github.com/apache/hudi/pull/6752


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] vamshigv commented on a diff in pull request #6752: [HUDI-4010] Improve docs on DynamoDB-based lock provider

Posted by GitBox <gi...@apache.org>.
vamshigv commented on code in PR #6752:
URL: https://github.com/apache/hudi/pull/6752#discussion_r978192289


##########
website/docs/concurrency_control.md:
##########
@@ -71,16 +71,26 @@ hoodie.write.lock.hivemetastore.table
 
 **`Amazon DynamoDB`** based lock provider
 
-Amazon DynamoDB based lock provides a simple way to support multi writing across different clusters
+Amazon DynamoDB based lock provides a simple way to support multi writing across different clusters.  You can refer to the
+[DynamoDB based Locks Configurations](https://hudi.apache.org/docs/configurations#DynamoDB-based-Locks-Configurations)
+section for the details of each related configuration knob.
 
 ```
 hoodie.write.lock.provider=org.apache.hudi.aws.transaction.lock.DynamoDBBasedLockProvider
-hoodie.write.lock.dynamodb.table
-hoodie.write.lock.dynamodb.partition_key
-hoodie.write.lock.dynamodb.region
-hoodie.write.lock.dynamodb.endpoint_url
-hoodie.write.lock.dynamodb.billing_mode
-```
+hoodie.write.lock.dynamodb.table (required)
+hoodie.write.lock.dynamodb.partition_key (optional)
+hoodie.write.lock.dynamodb.region (optional)
+hoodie.write.lock.dynamodb.endpoint_url (optional)
+hoodie.write.lock.dynamodb.billing_mode (optional)
+```
+
+When using the DynamoDB-based lock provider, the name of the DynamoDB table acting as the lock table for Hudi is
+specified by the config `hoodie.write.lock.dynamodb.table`. This DynamoDB table is automatically created by Hudi, so you
+don't have to create the table yourself. If you want to use an existing DynamoDB table, make sure that the `key` column

Review Comment:
   nit : may be `column with name "key"` ?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] CTTY commented on a diff in pull request #6752: [HUDI-4010] Improve docs on DynamoDB-based lock provider

Posted by GitBox <gi...@apache.org>.
CTTY commented on code in PR #6752:
URL: https://github.com/apache/hudi/pull/6752#discussion_r978236529


##########
website/docs/concurrency_control.md:
##########
@@ -71,16 +71,26 @@ hoodie.write.lock.hivemetastore.table
 
 **`Amazon DynamoDB`** based lock provider
 
-Amazon DynamoDB based lock provides a simple way to support multi writing across different clusters
+Amazon DynamoDB based lock provides a simple way to support multi writing across different clusters.  You can refer to the
+[DynamoDB based Locks Configurations](https://hudi.apache.org/docs/configurations#DynamoDB-based-Locks-Configurations)
+section for the details of each related configuration knob.
 
 ```
 hoodie.write.lock.provider=org.apache.hudi.aws.transaction.lock.DynamoDBBasedLockProvider
-hoodie.write.lock.dynamodb.table
-hoodie.write.lock.dynamodb.partition_key
-hoodie.write.lock.dynamodb.region
-hoodie.write.lock.dynamodb.endpoint_url
-hoodie.write.lock.dynamodb.billing_mode
-```
+hoodie.write.lock.dynamodb.table (required)
+hoodie.write.lock.dynamodb.partition_key (optional)
+hoodie.write.lock.dynamodb.region (optional)
+hoodie.write.lock.dynamodb.endpoint_url (optional)
+hoodie.write.lock.dynamodb.billing_mode (optional)

Review Comment:
   Why no mark configs versioned_docs with `required/optional` as well?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [hudi] yihua commented on a diff in pull request #6752: [HUDI-4010] Improve docs on DynamoDB-based lock provider

Posted by GitBox <gi...@apache.org>.
yihua commented on code in PR #6752:
URL: https://github.com/apache/hudi/pull/6752#discussion_r979130605


##########
website/docs/concurrency_control.md:
##########
@@ -71,16 +71,26 @@ hoodie.write.lock.hivemetastore.table
 
 **`Amazon DynamoDB`** based lock provider
 
-Amazon DynamoDB based lock provides a simple way to support multi writing across different clusters
+Amazon DynamoDB based lock provides a simple way to support multi writing across different clusters.  You can refer to the
+[DynamoDB based Locks Configurations](https://hudi.apache.org/docs/configurations#DynamoDB-based-Locks-Configurations)
+section for the details of each related configuration knob.
 
 ```
 hoodie.write.lock.provider=org.apache.hudi.aws.transaction.lock.DynamoDBBasedLockProvider
-hoodie.write.lock.dynamodb.table
-hoodie.write.lock.dynamodb.partition_key
-hoodie.write.lock.dynamodb.region
-hoodie.write.lock.dynamodb.endpoint_url
-hoodie.write.lock.dynamodb.billing_mode
-```
+hoodie.write.lock.dynamodb.table (required)
+hoodie.write.lock.dynamodb.partition_key (optional)
+hoodie.write.lock.dynamodb.region (optional)
+hoodie.write.lock.dynamodb.endpoint_url (optional)
+hoodie.write.lock.dynamodb.billing_mode (optional)
+```
+
+When using the DynamoDB-based lock provider, the name of the DynamoDB table acting as the lock table for Hudi is
+specified by the config `hoodie.write.lock.dynamodb.table`. This DynamoDB table is automatically created by Hudi, so you
+don't have to create the table yourself. If you want to use an existing DynamoDB table, make sure that the `key` column
+is present in the table, used as the partition key.  The config `hoodie.write.lock.dynamodb.partition_key` specifies the

Review Comment:
   Basically, I want to point out that the `key` column should be the partition key of the DynamoDB table when user creates it manually.  I rephrased it to be clear.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org