You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "FangYongs (via GitHub)" <gi...@apache.org> on 2023/02/11 06:41:15 UTC
[GitHub] [flink-table-store] FangYongs opened a new pull request, #522: [FLINK-30979] Support shuffling data by partition
FangYongs opened a new pull request, #522:
URL: https://github.com/apache/flink-table-store/pull/522
Currently sink operator in flink will shuffle data by bucket id, which cause data skew when there is only 1 bucket with multiple partitions in the table. This PR aims to support shuffling data by bucket id and partition when `sink.shuffle-by-partition.enable` is set.
The main changes are
1. Added config `sink.shuffle-by-partition.enable` to support shuffling data by partition
2. Added `PartitionComputer` to get partition from row data
3. Added shuffling data by partition in `BucketStreamPartitioner`
The main tests are
1. Added `FileStoreShuffleBucketTest` to shuffle data by bucket and partition
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-table-store] JingsongLi merged pull request #522: [FLINK-30979] Support shuffling data by partition
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi merged PR #522:
URL: https://github.com/apache/flink-table-store/pull/522
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-table-store] JingsongLi commented on a diff in pull request #522: [FLINK-30979] Support shuffling data by partition
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #522:
URL: https://github.com/apache/flink-table-store/pull/522#discussion_r1104117049
##########
docs/layouts/shortcodes/generated/flink_connector_configuration.html:
##########
@@ -20,6 +20,12 @@
<td>Integer</td>
<td>Define a custom parallelism for the scan source. By default, if this option is not defined, the planner will derive the parallelism for each statement individually by also considering the global configuration.</td>
</tr>
+ <tr>
+ <td><h5>sink-store.shuffle-by-partition.enable</h5></td>
Review Comment:
Maybe we can document this in `Writing Tables` page?
##########
docs/layouts/shortcodes/generated/flink_connector_configuration.html:
##########
@@ -20,6 +20,12 @@
<td>Integer</td>
<td>Define a custom parallelism for the scan source. By default, if this option is not defined, the planner will derive the parallelism for each statement individually by also considering the global configuration.</td>
</tr>
+ <tr>
+ <td><h5>sink-store.shuffle-by-partition.enable</h5></td>
Review Comment:
just `sink.partition-shuffle`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-table-store] FangYongs commented on pull request #522: [FLINK-30979] Support shuffling data by partition
Posted by "FangYongs (via GitHub)" <gi...@apache.org>.
FangYongs commented on PR #522:
URL: https://github.com/apache/flink-table-store/pull/522#issuecomment-1427781073
The failed test case is a known problem and I have created an issue https://issues.apache.org/jira/browse/FLINK-31039
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-table-store] JingsongLi commented on a diff in pull request #522: [FLINK-30979] Support shuffling data by partition
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #522:
URL: https://github.com/apache/flink-table-store/pull/522#discussion_r1103626550
##########
flink-table-store-connector/src/main/java/org/apache/flink/table/store/connector/FlinkConnectorOptions.java:
##########
@@ -69,6 +69,13 @@ public class FlinkConnectorOptions {
public static final ConfigOption<Integer> SINK_PARALLELISM = FactoryUtil.SINK_PARALLELISM;
+ public static final ConfigOption<Boolean> SINK_SHUFFLE_BY_PARTITION =
+ ConfigOptions.key("sink.shuffle-by-partition.enable")
Review Comment:
If the name is the same as that of Flink, will it lead to another useless partition shuffle? See Flink `StreamPhysicalSinkRule`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-table-store] FangYongs commented on a diff in pull request #522: [FLINK-30979] Support shuffling data by partition
Posted by "FangYongs (via GitHub)" <gi...@apache.org>.
FangYongs commented on code in PR #522:
URL: https://github.com/apache/flink-table-store/pull/522#discussion_r1103924487
##########
flink-table-store-connector/src/main/java/org/apache/flink/table/store/connector/FlinkConnectorOptions.java:
##########
@@ -69,6 +69,13 @@ public class FlinkConnectorOptions {
public static final ConfigOption<Integer> SINK_PARALLELISM = FactoryUtil.SINK_PARALLELISM;
+ public static final ConfigOption<Boolean> SINK_SHUFFLE_BY_PARTITION =
+ ConfigOptions.key("sink.shuffle-by-partition.enable")
Review Comment:
Thanks @JingsongLi I have rebased from master and rename the config name
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [flink-table-store] FangYongs commented on pull request #522: [FLINK-30979] Support shuffling data by partition
Posted by "FangYongs (via GitHub)" <gi...@apache.org>.
FangYongs commented on PR #522:
URL: https://github.com/apache/flink-table-store/pull/522#issuecomment-1427696362
@JingsongLi DONE
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org