You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2022/08/31 08:27:52 UTC

[GitHub] [hudi] xxWSHxx opened a new issue, #6551: [SUPPORT] Hudi only support one table one HoodieJavaWriteClient

xxWSHxx opened a new issue, #6551:
URL: https://github.com/apache/hudi/issues/6551

   **Describe the problem you faced**
   
   The tableName needs to be specified when HoodieJavaWriteClient is created, so that each table needs to create a corresponding HoodieJavaWriteClient. When there are a lot of tables in the database that need to be updated, a lot of HoodieJavaWriteClient needs to be created
   
   Steps to reproduce the behavior:
   
   `  HoodieWriteConfig cfg = HoodieWriteConfig.newBuilder().withPath(tablePath)
           .withSchema(PERSON_SCHEMA).withParallelism(2, 2)
           .withDeleteParallelism(2).forTable(tableName)
               .withIndexConfig(indexConfig)
           .withArchivalConfig(HoodieArchivalConfig.newBuilder().archiveCommitsWith(20, 30).build()).build();`
   
      ` HoodieJavaWriteClient<HoodieAvroPayload> client =
           new HoodieJavaWriteClient<>(new HoodieJavaEngineContext(hadoopConf), cfg);`
   
   **Expected behavior**
   HoodieJavaWriteClient supports operating all tables in the database instead of only one table
   
   **Environment Description**
   
   * Hudi version : 0.12.0
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] xxWSHxx commented on issue #6551: [SUPPORT] Hudi only support one table one HoodieJavaWriteClient

Posted by GitBox <gi...@apache.org>.
xxWSHxx commented on issue #6551:
URL: https://github.com/apache/hudi/issues/6551#issuecomment-1233787954

   > Similar to the jdbc connection, a database can use only one connection. The main purpose is that if each table has a client, many clients need to be created when many tables need to be updated, which will consume memory resources
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] codope commented on issue #6551: [SUPPORT] Hudi only support one table one HoodieJavaWriteClient

Posted by GitBox <gi...@apache.org>.
codope commented on issue #6551:
URL: https://github.com/apache/hudi/issues/6551#issuecomment-1233763768

   @xxWSHxx IMO, a write client per table makes sense as tables could have different configs for different purposes. Could add a little bit more detail about your use case, why would you want a single write client per database? If that's the case how would you specify different configs for the tables in the database?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] xushiyan commented on issue #6551: [SUPPORT] Hudi only support one table one HoodieJavaWriteClient

Posted by GitBox <gi...@apache.org>.
xushiyan commented on issue #6551:
URL: https://github.com/apache/hudi/issues/6551#issuecomment-1296214803

   The existing design is all meant for interacting with tables. To handle interactions at database level, you could come up with some model say HoodieJavaWriteClientPool. But this is a broader design problem, which is more suitable to be raised in dev email discussion. Feel free to start a `[DISCUSS] ` email thread with dev@hudi.apache.org


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] xushiyan closed issue #6551: [SUPPORT] Hudi only support one table one HoodieJavaWriteClient

Posted by GitBox <gi...@apache.org>.
xushiyan closed issue #6551: [SUPPORT] Hudi only support one table one HoodieJavaWriteClient
URL: https://github.com/apache/hudi/issues/6551


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] codope commented on issue #6551: [SUPPORT] Hudi only support one table one HoodieJavaWriteClient

Posted by GitBox <gi...@apache.org>.
codope commented on issue #6551:
URL: https://github.com/apache/hudi/issues/6551#issuecomment-1234482376

   We need to decouple configs from the client and use fluent builder APIs to build the client. 
   @xxWSHxx I would suggest creating a JIRA and put some ideas, and let's discuss there.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org