You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "yunfengzhou-hub (via GitHub)" <gi...@apache.org> on 2025/04/17 06:58:25 UTC
[PR] [doc] Add document for lookup custom shuffle [flink]
yunfengzhou-hub opened a new pull request, #26471:
URL: https://github.com/apache/flink/pull/26471
## What is the purpose of the change
This PR adds a document section to the lookup custom shuffle feature introduced in #25729.
## Brief change log
Adds a document section to the lookup custom shuffle feature introduced in #25729.
## Verifying this change
This is a document change that does not require program correctness verification.
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): no
- The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no
- The serializers: no
- The runtime per-record code paths (performance sensitive): no
- Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no
- The S3 file system connector: no
## Documentation
- Does this pull request introduce a new feature? no
- If yes, how is the feature documented? docs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-35652][doc] Add document for lookup custom shuffle [flink]
Posted by "reswqa (via GitHub)" <gi...@apache.org>.
reswqa commented on code in PR #26471:
URL: https://github.com/apache/flink/pull/26471#discussion_r2048512158
##########
docs/content/docs/dev/table/sql/queries/hints.md:
##########
@@ -464,6 +472,23 @@ If the lookup source only has one capability, then the 'async' mode option can b
LOOKUP('table'='Customers', 'retry-predicate'='lookup_miss', 'retry-strategy'='fixed_delay', 'fixed-delay'='10s','max-attempts'='3')
```
+#### 4. Enable Custom Data Distribution
Review Comment:
Please also add this part for Chinese doc.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-35652][doc] Add document for lookup custom shuffle [flink]
Posted by "RocMarshal (via GitHub)" <gi...@apache.org>.
RocMarshal commented on code in PR #26471:
URL: https://github.com/apache/flink/pull/26471#discussion_r2049051794
##########
docs/content.zh/docs/dev/table/sql/queries/hints.md:
##########
@@ -445,6 +453,19 @@ LOOKUP('table'='Customers', 'async'='false', 'retry-predicate'='lookup_miss', 'r
LOOKUP('table'='Customers', 'retry-predicate'='lookup_miss', 'retry-strategy'='fixed_delay', 'fixed-delay'='10s','max-attempts'='3')
```
+#### 4. 启用自定义数据分布
+
+在默认情况下，Lookup Join的输入流数据分布是随机的，因此数据源可能无法有效利用缓存来加速查找。 用户可以通过如下方式启用
Review Comment:
```suggestion
在默认情况下，Lookup Join 的输入流数据分布是随机的，因此数据源可能无法有效利用缓存来加速查找。 用户可以通过如下方式启用
```
##########
docs/content.zh/docs/dev/table/sql/queries/hints.md:
##########
@@ -445,6 +453,19 @@ LOOKUP('table'='Customers', 'async'='false', 'retry-predicate'='lookup_miss', 'r
LOOKUP('table'='Customers', 'retry-predicate'='lookup_miss', 'retry-strategy'='fixed_delay', 'fixed-delay'='10s','max-attempts'='3')
```
+#### 4. 启用自定义数据分布
+
+在默认情况下，Lookup Join的输入流数据分布是随机的，因此数据源可能无法有效利用缓存来加速查找。 用户可以通过如下方式启用
+自定义数据分发，使数据源能够自行决定输入数据的分布，并利用这一先验知识来优化其缓存和查找策略。
+
+```sql
+LOOKUP('table'='Customers', 'shuffle'='true')
+```
+
+为了充分利用这个优化，目标 Lookup Source 应该提供对自定义数据分发能力的支持。连接器开发人员可以通过让
+LookupTableSource 子类实现 SupportsLookupCustomShuffle 接口来支持这种能力。即使Source尚未提供这种能力，用户依
Review Comment:
```suggestion
LookupTableSource 子类实现 SupportsLookupCustomShuffle 接口来支持这种能力。即使 Source 尚未提供这种能力，用户依
```
##########
docs/content.zh/docs/dev/table/sql/queries/hints.md:
##########
@@ -359,6 +359,14 @@ LOOKUP 联接提示允许用户建议 Flink 优化器:
<td>N/A</td>
<td>固定延迟策略的最大重试次数</td>
</tr>
+<tr>
+ <td>shuffle</td>
+ <td>shuffle</td>
+ <td>N</td>
+ <td>boolean</td>
+ <td>false</td>
+ <td>是否开启自定义数据分发功能。此功能允许Lookup Source自行决定数据分布方式并依此对数据查询逻辑做相应优化</td>
Review Comment:
```suggestion
<td>是否开启自定义数据分发功能。此功能允许 Lookup Source 自行决定数据分布方式并依此对数据查询逻辑做相应优化</td>
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-35652][doc] Add document for lookup custom shuffle [flink]
Posted by "yunfengzhou-hub (via GitHub)" <gi...@apache.org>.
yunfengzhou-hub commented on PR #26471:
URL: https://github.com/apache/flink/pull/26471#issuecomment-2811984015
Hi @reswqa, could you please take a look at this PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-35652][doc] Add document for lookup custom shuffle [flink]
Posted by "flinkbot (via GitHub)" <gi...@apache.org>.
flinkbot commented on PR #26471:
URL: https://github.com/apache/flink/pull/26471#issuecomment-2811991179
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "0e8809485e0ebe258cece3adfc4f59c9081f4147",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0e8809485e0ebe258cece3adfc4f59c9081f4147",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 0e8809485e0ebe258cece3adfc4f59c9081f4147 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-35652][doc] Add document for lookup custom shuffle [flink]
Posted by "reswqa (via GitHub)" <gi...@apache.org>.
reswqa merged PR #26471:
URL: https://github.com/apache/flink/pull/26471
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [FLINK-35652][doc] Add document for lookup custom shuffle [flink]
Posted by "yunfengzhou-hub (via GitHub)" <gi...@apache.org>.
yunfengzhou-hub commented on PR #26471:
URL: https://github.com/apache/flink/pull/26471#issuecomment-2814275006
Thanks for the comments @reswqa @RocMarshal. I have updated the PR according to the comments.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org