You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2022/04/23 03:49:58 UTC

[GitHub] [hudi] hehuiyuan opened a new pull request, #5405: [HUDI-3951]support generan parameter 'sink.parallelism' for flink-hudi

hehuiyuan opened a new pull request, #5405:
URL: https://github.com/apache/hudi/pull/5405

   
   ## What is the purpose of the pull request
   
   
   support generan parameter 'sink.parallelism' for flink-hudi
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot commented on pull request #5405: [HUDI-3951]support general parameter 'sink.parallelism' for flink-hudi

Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5405:
URL: https://github.com/apache/hudi/pull/5405#issuecomment-1107353850

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "b7cdd5fe9294a251e2a8ac97eb57f3e4c265a0e5",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "b7cdd5fe9294a251e2a8ac97eb57f3e4c265a0e5",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * b7cdd5fe9294a251e2a8ac97eb57f3e4c265a0e5 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot commented on pull request #5405: [HUDI-3951]support general parameter 'sink.parallelism' for flink-hudi

Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5405:
URL: https://github.com/apache/hudi/pull/5405#issuecomment-1107376295

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "b7cdd5fe9294a251e2a8ac97eb57f3e4c265a0e5",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8248",
       "triggerID" : "b7cdd5fe9294a251e2a8ac97eb57f3e4c265a0e5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d0059712698d0c54aa951095b66c115a38d8b04a",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "d0059712698d0c54aa951095b66c115a38d8b04a",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * b7cdd5fe9294a251e2a8ac97eb57f3e4c265a0e5 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8248) 
   * d0059712698d0c54aa951095b66c115a38d8b04a UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot commented on pull request #5405: [HUDI-3951]support general parameter 'sink.parallelism' for flink-hudi

Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5405:
URL: https://github.com/apache/hudi/pull/5405#issuecomment-1107377270

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "b7cdd5fe9294a251e2a8ac97eb57f3e4c265a0e5",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8248",
       "triggerID" : "b7cdd5fe9294a251e2a8ac97eb57f3e4c265a0e5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d0059712698d0c54aa951095b66c115a38d8b04a",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8252",
       "triggerID" : "d0059712698d0c54aa951095b66c115a38d8b04a",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * b7cdd5fe9294a251e2a8ac97eb57f3e4c265a0e5 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8248) 
   * d0059712698d0c54aa951095b66c115a38d8b04a Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8252) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] xushiyan commented on pull request #5405: [HUDI-3951]support general parameter 'sink.parallelism' for flink-hudi

Posted by GitBox <gi...@apache.org>.
xushiyan commented on PR #5405:
URL: https://github.com/apache/hudi/pull/5405#issuecomment-1107842324

   @XuQianJin-Stars when merging the PR, can you please make sure JIRA id is added to the message title? for easy referencing. thanks!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hehuiyuan commented on pull request #5405: [HUDI-3951]support general parameter 'sink.parallelism' for flink-hudi

Posted by GitBox <gi...@apache.org>.
hehuiyuan commented on PR #5405:
URL: https://github.com/apache/hudi/pull/5405#issuecomment-1107401464

   @hudi-bot run azure


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot commented on pull request #5405: [HUDI-3951]support general parameter 'sink.parallelism' for flink-hudi

Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5405:
URL: https://github.com/apache/hudi/pull/5405#issuecomment-1107397629

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "b7cdd5fe9294a251e2a8ac97eb57f3e4c265a0e5",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8248",
       "triggerID" : "b7cdd5fe9294a251e2a8ac97eb57f3e4c265a0e5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d0059712698d0c54aa951095b66c115a38d8b04a",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8252",
       "triggerID" : "d0059712698d0c54aa951095b66c115a38d8b04a",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * d0059712698d0c54aa951095b66c115a38d8b04a Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8252) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] XuQianJin-Stars commented on a diff in pull request #5405: [HUDI-3951]support general parameter 'sink.parallelism' for flink-hudi

Posted by GitBox <gi...@apache.org>.
XuQianJin-Stars commented on code in PR #5405:
URL: https://github.com/apache/hudi/pull/5405#discussion_r857245216


##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/configuration/FlinkOptions.java:
##########
@@ -232,6 +233,8 @@ private FlinkOptions() {
   // ------------------------------------------------------------------------
   //  Write Options
   // ------------------------------------------------------------------------
+  public static final ConfigOption<Integer> SINK_PARALLELISM = FactoryUtil.SINK_PARALLELISM;
+
   public static final ConfigOption<String> TABLE_NAME = ConfigOptions

Review Comment:
   > Hello, it seems that we want to add a default parallelism to the sink, but there is already parallelism param named `write.tasks`, can we rename this param to `write.default_tasks` or something ? And we should add test in `TestHoodieTableFactory`, before we make agreement, can you revert this PR first ? Thanks ~
   
   well, let me revert this PR first, Then come to an agreement, sorry @hehuiyuan  my mistake.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] XuQianJin-Stars merged pull request #5405: [HUDI-3951]support general parameter 'sink.parallelism' for flink-hudi

Posted by GitBox <gi...@apache.org>.
XuQianJin-Stars merged PR #5405:
URL: https://github.com/apache/hudi/pull/5405


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] danny0405 commented on a diff in pull request #5405: [HUDI-3951]support general parameter 'sink.parallelism' for flink-hudi

Posted by GitBox <gi...@apache.org>.
danny0405 commented on code in PR #5405:
URL: https://github.com/apache/hudi/pull/5405#discussion_r857212822


##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/configuration/FlinkOptions.java:
##########
@@ -232,6 +233,8 @@ private FlinkOptions() {
   // ------------------------------------------------------------------------
   //  Write Options
   // ------------------------------------------------------------------------
+  public static final ConfigOption<Integer> SINK_PARALLELISM = FactoryUtil.SINK_PARALLELISM;
+
   public static final ConfigOption<String> TABLE_NAME = ConfigOptions

Review Comment:
   Hello, it seems that we want to add a default parallelism to the sink, but there is already parallelism param named `write.tasks`, can we rename this param to `write.default_tasks` or something ? And we should add test in `TestHoodieTableFactory`, before we make agreement, can you revert this PR first ? Thanks ~



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hehuiyuan commented on pull request #5405: [HUDI-3951]support general parameter 'sink.parallelism' for flink-hudi

Posted by GitBox <gi...@apache.org>.
hehuiyuan commented on PR #5405:
URL: https://github.com/apache/hudi/pull/5405#issuecomment-1107692109

   Hi @yanghua , who can look at this pr?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hehuiyuan commented on a diff in pull request #5405: [HUDI-3951]support general parameter 'sink.parallelism' for flink-hudi

Posted by GitBox <gi...@apache.org>.
hehuiyuan commented on code in PR #5405:
URL: https://github.com/apache/hudi/pull/5405#discussion_r857559082


##########
hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/configuration/FlinkOptions.java:
##########
@@ -232,6 +233,8 @@ private FlinkOptions() {
   // ------------------------------------------------------------------------
   //  Write Options
   // ------------------------------------------------------------------------
+  public static final ConfigOption<Integer> SINK_PARALLELISM = FactoryUtil.SINK_PARALLELISM;
+
   public static final ConfigOption<String> TABLE_NAME = ConfigOptions

Review Comment:
   @danny0405 ok.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot commented on pull request #5405: [HUDI-3951]support general parameter 'sink.parallelism' for flink-hudi

Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5405:
URL: https://github.com/apache/hudi/pull/5405#issuecomment-1107401757

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "b7cdd5fe9294a251e2a8ac97eb57f3e4c265a0e5",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8248",
       "triggerID" : "b7cdd5fe9294a251e2a8ac97eb57f3e4c265a0e5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d0059712698d0c54aa951095b66c115a38d8b04a",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8252",
       "triggerID" : "d0059712698d0c54aa951095b66c115a38d8b04a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d0059712698d0c54aa951095b66c115a38d8b04a",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8256",
       "triggerID" : "1107401464",
       "triggerType" : "MANUAL"
     } ]
   }-->
   ## CI report:
   
   * d0059712698d0c54aa951095b66c115a38d8b04a Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8252) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8256) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot commented on pull request #5405: [HUDI-3951]support general parameter 'sink.parallelism' for flink-hudi

Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5405:
URL: https://github.com/apache/hudi/pull/5405#issuecomment-1107442153

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "b7cdd5fe9294a251e2a8ac97eb57f3e4c265a0e5",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8248",
       "triggerID" : "b7cdd5fe9294a251e2a8ac97eb57f3e4c265a0e5",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d0059712698d0c54aa951095b66c115a38d8b04a",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8252",
       "triggerID" : "d0059712698d0c54aa951095b66c115a38d8b04a",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d0059712698d0c54aa951095b66c115a38d8b04a",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8256",
       "triggerID" : "1107401464",
       "triggerType" : "MANUAL"
     } ]
   }-->
   ## CI report:
   
   * d0059712698d0c54aa951095b66c115a38d8b04a Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8252) Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8256) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] XuQianJin-Stars commented on pull request #5405: [HUDI-3951]support general parameter 'sink.parallelism' for flink-hudi

Posted by GitBox <gi...@apache.org>.
XuQianJin-Stars commented on PR #5405:
URL: https://github.com/apache/hudi/pull/5405#issuecomment-1108050883

   > 
   
   https://issues.apache.org/jira/browse/HUDI-3951 @hehuiyuan add this message to What is the purpose of the pull request and jira's Template.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] hudi-bot commented on pull request #5405: [HUDI-3951]support general parameter 'sink.parallelism' for flink-hudi

Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #5405:
URL: https://github.com/apache/hudi/pull/5405#issuecomment-1107354233

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "b7cdd5fe9294a251e2a8ac97eb57f3e4c265a0e5",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8248",
       "triggerID" : "b7cdd5fe9294a251e2a8ac97eb57f3e4c265a0e5",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * b7cdd5fe9294a251e2a8ac97eb57f3e4c265a0e5 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=8248) 
   
   <details>
   <summary>Bot commands</summary>
     @hudi-bot supports the following commands:
   
    - `@hudi-bot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org