You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@fluss.apache.org by "SML0127 (via GitHub)" <gi...@apache.org> on 2025/08/15 11:21:13 UTC
[PR] Apply partition filter to lake in flink source [fluss]
SML0127 opened a new pull request, #1549:
URL: https://github.com/apache/fluss/pull/1549
<!--
*Thank you very much for contributing to Fluss - we are happy that you want to help us improve Fluss. To help the community review your contribution in the best possible way, please go through the checklist below, which will get the contribution into a shape in which it can be best reviewed.*
## Contribution Checklist
- Make sure that the pull request corresponds to a [GitHub issue](https://github.com/alibaba/fluss/issues). Exceptions are made for typos in JavaDoc or documentation files, which need no issue.
- Name the pull request in the format "[component] Title of the pull request", where *[component]* should be replaced by the name of the component being changed. Typically, this corresponds to the component label assigned to the issue (e.g., [kv], [log], [client], [flink]). Skip *[component]* if you are unsure about which is the best component.
- Fill out the template below to describe the changes contributed by the pull request. That will give reviewers the context they need to do the review.
- Make sure that the change passes the automated tests, i.e., `mvn clean verify` passes.
- Each pull request should address only one issue, not mix up code from multiple issues.
**(The sections below can be removed for hotfixes or typos)**
-->
### Purpose
<!-- Linking this pull request to the issue -->
Linked issue: close https://github.com/apache/fluss/issues/1529
<!-- What is the purpose of the change -->
### Brief change log
<!-- Please describe the changes made in this pull request and explain how they address the issue -->
- push down partition filter from flink to lake source to prune
- copy convertion logic from https://github.com/apache/fluss/pull/1539
### Tests
<!-- List UT and IT cases to verify this change -->
### API and Format
<!-- Does this change affect API or storage format -->
### Documentation
<!-- Does this change introduce a new feature -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Apply partition filter to lake in flink source [fluss]
Posted by "SML0127 (via GitHub)" <gi...@apache.org>.
SML0127 commented on PR #1549:
URL: https://github.com/apache/fluss/pull/1549#issuecomment-3204048886
@luoyuxia Thank you for the commit. I reviewed the commit you added and now I understand what the issue was. Once the all tests pass, I think it'll be ready to be merged without any problem.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Apply partition filter to lake in flink source [fluss]
Posted by "luoyuxia (via GitHub)" <gi...@apache.org>.
luoyuxia merged PR #1549:
URL: https://github.com/apache/fluss/pull/1549
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Apply partition filter to lake in flink source [fluss]
Posted by "SML0127 (via GitHub)" <gi...@apache.org>.
SML0127 commented on code in PR #1549:
URL: https://github.com/apache/fluss/pull/1549#discussion_r2285575170
##########
fluss-flink/fluss-flink-common/src/main/java/com/alibaba/fluss/flink/source/FlinkTableSource.java:
##########
@@ -449,11 +455,57 @@ && hasPrimaryKey()
getPartitionKeyTypes(),
acceptedFilters,
remainingFilters,
- FLINK_INTERNAL_VALUE);
+ FLUSS_INTERNAL_VALUE);
+
// partitions are filtered by string representations, convert the equals to string first
- fieldEquals = stringifyFieldEquals(fieldEquals);
+ partitionFilters = stringifyFieldEquals(fieldEquals);
+
+ // lake source is not null
+ if (lakeSource != null) {
+ // and exist field equals, push down to lake source
+ if (!fieldEquals.isEmpty()) {
+
+ // convert flink row type to fluss row type
+ RowType flussRowType = FlinkConversions.toFlussRowType(tableOutputType);
+ List<Predicate> lakePredicates = new ArrayList<>();
+
+ for (FieldEqual fieldEqual : fieldEquals) {
+ int idx = fieldEqual.fieldIndex;
+ String fieldName = tableOutputType.getFieldNames().get(idx);
+ com.alibaba.fluss.types.DataType flussDataType =
+ flussRowType.getTypeAt(idx);
+
+ // convert to Paimon literal for partition
+ Object literal =
+ toPaimonLiteralForPartition(flussDataType, fieldEqual.equalValue);
+
+ if (literal == null) {
+ continue;
+ }
+
+ lakePredicates.add(
+ new LeafPredicate(
+ Equal.INSTANCE,
+ flussDataType,
+ idx,
+ fieldName,
+ Collections.singletonList(literal)));
+ }
+
+ if (!lakePredicates.isEmpty()) {
+ final LakeSource.FilterPushDownResult filterPushDownResult =
+ lakeSource.withFilters(lakePredicates);
+ if (filterPushDownResult.acceptedPredicates().size()
+ != lakePredicates.size()) {
+ LOG.warn("Some partition filters are not accepted by the lake source");
Review Comment:
Thank you! It’s a point I hadn’t considered at all. I’ve updated it to pass no accepted filters back to Flink as you suggested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Apply partition filter to lake in flink source [fluss]
Posted by "SML0127 (via GitHub)" <gi...@apache.org>.
SML0127 commented on code in PR #1549:
URL: https://github.com/apache/fluss/pull/1549#discussion_r2285946271
##########
fluss-lake/fluss-lake-paimon/src/test/java/com/alibaba/fluss/lake/paimon/flink/FlinkUnionReadLogTableITCase.java:
##########
@@ -95,11 +95,37 @@ void testReadLogTableFullType(boolean isPartitioned) throws Exception {
.map(row -> Row.of(row.getField(1)))
.collect(Collectors.toList());
assertThat(actual).containsExactlyInAnyOrderElementsOf(expected);
+
+ if (isPartitioned) {
+ // get first partition
+ String partition = waitUntilPartitions(t1).values().iterator().next();
+ String sqlWithPartitionFilter =
+ "select * FROM " + tableName + " WHERE p = '" + partition + "'";
+
+ String plan = batchTEnv.explainSql(sqlWithPartitionFilter);
+
+ // check if the plan contains partition filter
+ assertThat(plan)
+ .contains("TableSourceScan(")
+ .contains("filter=[=(p, _UTF-16LE'" + partition + "'");
+
+ List<Row> expectedFiltered =
+ writtenRows.stream()
+ .filter(r -> partition.equals(r.getField(15)))
+ .collect(Collectors.toList());
+
+ List<Row> actualFiltered =
+ CollectionUtil.iteratorToList(
+ batchTEnv.executeSql(sqlWithPartitionFilter).collect());
+
+ assertThat(actualFiltered).containsExactlyInAnyOrderElementsOf(expectedFiltered);
+ }
Review Comment:
assertion error output
```
java.lang.AssertionError:
Expecting actual:
[+I[true, 100, 200, 30, 400, 500.1, 600.0, another_string_0, 9.00, 1000, 2023-10-25T12:01:13.400Z, 2023-10-25T12:01:13.400007Z, 2023-10-25T12:01:13.501, 2023-10-25T12:01:13.501008, [5, 6, 7, 8], 2025],
+I[true, 100, 200, 30, 400, 500.1, 600.0, another_string_1, 9.00, 1000, 2023-10-25T12:01:13.400Z, 2023-10-25T12:01:13.400007Z, 2023-10-25T12:01:13.501, 2023-10-25T12:01:13.501008, [5, 6, 7, 8], 2025],
+I[true, 100, 200, 30, 400, 500.1, 600.0, another_string_2, 9.00, 1000, 2023-10-25T12:01:13.400Z, 2023-10-25T12:01:13.400007Z, 2023-10-25T12:01:13.501, 2023-10-25T12:01:13.501008, [5, 6, 7, 8], 2025],
+I[true, 100, 200, 30, 400, 500.1, 600.0, another_string_3, 9.00, 1000, 2023-10-25T12:01:13.400Z, 2023-10-25T12:01:13.400007Z, 2023-10-25T12:01:13.501, 2023-10-25T12:01:13.501008, [5, 6, 7, 8], 2025],
+I[true, 100, 200, 30, 400, 500.1, 600.0, another_string_4, 9.00, 1000, 2023-10-25T12:01:13.400Z, 2023-10-25T12:01:13.400007Z, 2023-10-25T12:01:13.501, 2023-10-25T12:01:13.501008, [5, 6, 7, 8], 2025],
+I[true, 100, 200, 30, 400, 500.1, 600.0, another_string_5, 9.00, 1000, 2023-10-25T12:01:13.400Z, 2023-10-25T12:01:13.400007Z, 2023-10-25T12:01:13.501, 2023-10-25T12:01:13.501008, [5, 6, 7, 8], 2025],
+I[true, 100, 200, 30, 400, 500.1, 600.0, another_string_6, 9.00, 1000, 2023-10-25T12:01:13.400Z, 2023-10-25T12:01:13.400007Z, 2023-10-25T12:01:13.501, 2023-10-25T12:01:13.501008, [5, 6, 7, 8], 2025],
+I[true, 100, 200, 30, 400, 500.1, 600.0, another_string_7, 9.00, 1000, 2023-10-25T12:01:13.400Z, 2023-10-25T12:01:13.400007Z, 2023-10-25T12:01:13.501, 2023-10-25T12:01:13.501008, [5, 6, 7, 8], 2025],
+I[true, 100, 200, 30, 400, 500.1, 600.0, another_string_8, 9.00, 1000, 2023-10-25T12:01:13.400Z, 2023-10-25T12:01:13.400007Z, 2023-10-25T12:01:13.501, 2023-10-25T12:01:13.501008, [5, 6, 7, 8], 2025],
+I[true, 100, 200, 30, 400, 500.1, 600.0, another_string_9, 9.00, 1000, 2023-10-25T12:01:13.400Z, 2023-10-25T12:01:13.400007Z, 2023-10-25T12:01:13.501, 2023-10-25T12:01:13.501008, [5, 6, 7, 8], 2025],
+I[true, 100, 200, 30, 400, 500.1, 600.0, another_string_0, 9.00, 1000, 2023-10-25T12:01:13.400Z, 2023-10-25T12:01:13.400007Z, 2023-10-25T12:01:13.501, 2023-10-25T12:01:13.501008, [5, 6, 7, 8], 2025],
+I[true, 100, 200, 30, 400, 500.1, 600.0, another_string_1, 9.00, 1000, 2023-10-25T12:01:13.400Z, 2023-10-25T12:01:13.400007Z, 2023-10-25T12:01:13.501, 2023-10-25T12:01:13.501008, [5, 6, 7, 8], 2025],
+I[true, 100, 200, 30, 400, 500.1, 600.0, another_string_2, 9.00, 1000, 2023-10-25T12:01:13.400Z, 2023-10-25T12:01:13.400007Z, 2023-10-25T12:01:13.501, 2023-10-25T12:01:13.501008, [5, 6, 7, 8], 2025],
+I[true, 100, 200, 30, 400, 500.1, 600.0, another_string_3, 9.00, 1000, 2023-10-25T12:01:13.400Z, 2023-10-25T12:01:13.400007Z, 2023-10-25T12:01:13.501, 2023-10-25T12:01:13.501008, [5, 6, 7, 8], 2025],
+I[true, 100, 200, 30, 400, 500.1, 600.0, another_string_4, 9.00, 1000, 2023-10-25T12:01:13.400Z, 2023-10-25T12:01:13.400007Z, 2023-10-25T12:01:13.501, 2023-10-25T12:01:13.501008, [5, 6, 7, 8], 2025],
+I[true, 100, 200, 30, 400, 500.1, 600.0, another_string_5, 9.00, 1000, 2023-10-25T12:01:13.400Z, 2023-10-25T12:01:13.400007Z, 2023-10-25T12:01:13.501, 2023-10-25T12:01:13.501008, [5, 6, 7, 8], 2025],
+I[true, 100, 200, 30, 400, 500.1, 600.0, another_string_6, 9.00, 1000, 2023-10-25T12:01:13.400Z, 2023-10-25T12:01:13.400007Z, 2023-10-25T12:01:13.501, 2023-10-25T12:01:13.501008, [5, 6, 7, 8], 2025],
+I[true, 100, 200, 30, 400, 500.1, 600.0, another_string_7, 9.00, 1000, 2023-10-25T12:01:13.400Z, 2023-10-25T12:01:13.400007Z, 2023-10-25T12:01:13.501, 2023-10-25T12:01:13.501008, [5, 6, 7, 8], 2025],
+I[true, 100, 200, 30, 400, 500.1, 600.0, another_string_8, 9.00, 1000, 2023-10-25T12:01:13.400Z, 2023-10-25T12:01:13.400007Z, 2023-10-25T12:01:13.501, 2023-10-25T12:01:13.501008, [5, 6, 7, 8], 2025],
+I[true, 100, 200, 30, 400, 500.1, 600.0, another_string_9, 9.00, 1000, 2023-10-25T12:01:13.400Z, 2023-10-25T12:01:13.400007Z, 2023-10-25T12:01:13.501, 2023-10-25T12:01:13.501008, [5, 6, 7, 8], 2025],
+I[true, 100, 200, 30, 400, 500.1, 600.0, another_string_0, 9.00, 1000, 2023-10-25T12:01:13.400Z, 2023-10-25T12:01:13.400007Z, 2023-10-25T12:01:13.501, 2023-10-25T12:01:13.501008, [5, 6, 7, 8], 2025],
+I[true, 100, 200, 30, 400, 500.1, 600.0, another_string_1, 9.00, 1000, 2023-10-25T12:01:13.400Z, 2023-10-25T12:01:13.400007Z, 2023-10-25T12:01:13.501, 2023-10-25T12:01:13.501008, [5, 6, 7, 8], 2025],
+I[true, 100, 200, 30, 400, 500.1, 600.0, another_string_2, 9.00, 1000, 2023-10-25T12:01:13.400Z, 2023-10-25T12:01:13.400007Z, 2023-10-25T12:01:13.501, 2023-10-25T12:01:13.501008, [5, 6, 7, 8], 2025],
+I[true, 100, 200, 30, 400, 500.1, 600.0, another_string_3, 9.00, 1000, 2023-10-25T12:01:13.400Z, 2023-10-25T12:01:13.400007Z, 2023-10-25T12:01:13.501, 2023-10-25T12:01:13.501008, [5, 6, 7, 8], 2025],
+I[true, 100, 200, 30, 400, 500.1, 600.0, another_string_4, 9.00, 1000, 2023-10-25T12:01:13.400Z, 2023-10-25T12:01:13.400007Z, 2023-10-25T12:01:13.501, 2023-10-25T12:01:13.501008, [5, 6, 7, 8], 2025],
+I[true, 100, 200, 30, 400, 500.1, 600.0, another_string_5, 9.00, 1000, 2023-10-25T12:01:13.400Z, 2023-10-25T12:01:13.400007Z, 2023-10-25T12:01:13.501, 2023-10-25T12:01:13.501008, [5, 6, 7, 8], 2025],
+I[true, 100, 200, 30, 400, 500.1, 600.0, another_string_6, 9.00, 1000, 2023-10-25T12:01:13.400Z, 2023-10-25T12:01:13.400007Z, 2023-10-25T12:01:13.501, 2023-10-25T12:01:13.501008, [5, 6, 7, 8], 2025],
+I[true, 100, 200, 30, 400, 500.1, 600.0, another_string_7, 9.00, 1000, 2023-10-25T12:01:13.400Z, 2023-10-25T12:01:13.400007Z, 2023-10-25T12:01:13.501, 2023-10-25T12:01:13.501008, [5, 6, 7, 8], 2025],
+I[true, 100, 200, 30, 400, 500.1, 600.0, another_string_8, 9.00, 1000, 2023-10-25T12:01:13.400Z, 2023-10-25T12:01:13.400007Z, 2023-10-25T12:01:13.501, 2023-10-25T12:01:13.501008, [5, 6, 7, 8], 2025],
+I[true, 100, 200, 30, 400, 500.1, 600.0, another_string_9, 9.00, 1000, 2023-10-25T12:01:13.400Z, 2023-10-25T12:01:13.400007Z, 2023-10-25T12:01:13.501, 2023-10-25T12:01:13.501008, [5, 6, 7, 8], 2025],
+I[true, 100, 200, 30, 400, 500.1, 600.0, another_string_0, 9.00, 1000, 2023-10-25T12:01:13.400Z, 2023-10-25T12:01:13.400007Z, 2023-10-25T12:01:13.501, 2023-10-25T12:01:13.501008, [5, 6, 7, 8], 2025],
+I[true, 100, 200, 30, 400, 500.1, 600.0, another_string_1, 9.00, 1000, 2023-10-25T12:01:13.400Z, 2023-10-25T12:01:13.400007Z, 2023-10-25T12:01:13.501, 2023-10-25T12:01:13.501008, [5, 6, 7, 8], 2025],
+I[true, 100, 200, 30, 400, 500.1, 600.0, another_string_2, 9.00, 1000, 2023-10-25T12:01:13.400Z, 2023-10-25T12:01:13.400007Z, 2023-10-25T12:01:13.501, 2023-10-25T12:01:13.501008, [5, 6, 7, 8], 2025],
+I[true, 100, 200, 30, 400, 500.1, 600.0, another_string_0, 9.00, 1000, 2023-10-25T12:01:13.400Z, 2023-10-25T12:01:13.400007Z, 2023-10-25T12:01:13.501, 2023-10-25T12:01:13.501008, [5, 6, 7, 8], 2025],
+I[true, 100, 200, 30, 400, 500.1, 600.0, another_string_1, 9.00, 1000, 2023-10-25T12:01:13.400Z, 2023-10-25T12:01:13.400007Z, 2023-10-25T12:01:13.501, 2023-10-25T12:01:13.501008, [5, 6, 7, 8], 2025],
+I[true, 100, 200, 30, 400, 500.1, 600.0, another_string_2, 9.00, 1000, 2023-10-25T12:01:13.400Z, 2023-10-25T12:01:13.400007Z, 2023-10-25T12:01:13.501, 2023-10-25T12:01:13.501008, [5, 6, 7, 8], 2025],
+I[true, 100, 200, 30, 400, 500.1, 600.0, another_string_3, 9.00, 1000, 2023-10-25T12:01:13.400Z, 2023-10-25T12:01:13.400007Z, 2023-10-25T12:01:13.501, 2023-10-25T12:01:13.501008, [5, 6, 7, 8], 2025],
+I[true, 100, 200, 30, 400, 500.1, 600.0, another_string_4, 9.00, 1000, 2023-10-25T12:01:13.400Z, 2023-10-25T12:01:13.400007Z, 2023-10-25T12:01:13.501, 2023-10-25T12:01:13.501008, [5, 6, 7, 8], 2025],
+I[true, 100, 200, 30, 400, 500.1, 600.0, another_string_5, 9.00, 1000, 2023-10-25T12:01:13.400Z, 2023-10-25T12:01:13.400007Z, 2023-10-25T12:01:13.501, 2023-10-25T12:01:13.501008, [5, 6, 7, 8], 2025],
+I[true, 100, 200, 30, 400, 500.1, 600.0, another_string_6, 9.00, 1000, 2023-10-25T12:01:13.400Z, 2023-10-25T12:01:13.400007Z, 2023-10-25T12:01:13.501, 2023-10-25T12:01:13.501008, [5, 6, 7, 8], 2025],
+I[true, 100, 200, 30, 400, 500.1, 600.0, another_string_7, 9.00, 1000, 2023-10-25T12:01:13.400Z, 2023-10-25T12:01:13.400007Z, 2023-10-25T12:01:13.501, 2023-10-25T12:01:13.501008, [5, 6, 7, 8], 2025],
+I[true, 100, 200, 30, 400, 500.1, 600.0, another_string_8, 9.00, 1000, 2023-10-25T12:01:13.400Z, 2023-10-25T12:01:13.400007Z, 2023-10-25T12:01:13.501, 2023-10-25T12:01:13.501008, [5, 6, 7, 8], 2025],
+I[true, 100, 200, 30, 400, 500.1, 600.0, another_string_9, 9.00, 1000, 2023-10-25T12:01:13.400Z, 2023-10-25T12:01:13.400007Z, 2023-10-25T12:01:13.501, 2023-10-25T12:01:13.501008, [5, 6, 7, 8], 2025],
+I[true, 100, 200, 30, 400, 500.1, 600.0, another_string_0, 9.00, 1000, 2023-10-25T12:01:13.400Z, 2023-10-25T12:01:13.400007Z, 2023-10-25T12:01:13.501, 2023-10-25T12:01:13.501008, [5, 6, 7, 8], 2025],
+I[true, 100, 200, 30, 400, 500.1, 600.0, another_string_1, 9.00, 1000, 2023-10-25T12:01:13.400Z, 2023-10-25T12:01:13.400007Z, 2023-10-25T12:01:13.501, 2023-10-25T12:01:13.501008, [5, 6, 7, 8], 2025],
+I[true, 100, 200, 30, 400, 500.1, 600.0, another_string_2, 9.00, 1000, 2023-10-25T12:01:13.400Z, 2023-10-25T12:01:13.400007Z, 2023-10-25T12:01:13.501, 2023-10-25T12:01:13.501008, [5, 6, 7, 8], 2025],
+I[true, 100, 200, 30, 400, 500.1, 600.0, another_string_3, 9.00, 1000, 2023-10-25T12:01:13.400Z, 2023-10-25T12:01:13.400007Z, 2023-10-25T12:01:13.501, 2023-10-25T12:01:13.501008, [5, 6, 7, 8], 2025],
+I[true, 100, 200, 30, 400, 500.1, 600.0, another_string_4, 9.00, 1000, 2023-10-25T12:01:13.400Z, 2023-10-25T12:01:13.400007Z, 2023-10-25T12:01:13.501, 2023-10-25T12:01:13.501008, [5, 6, 7, 8], 2025],
+I[true, 100, 200, 30, 400, 500.1, 600.0, another_string_5, 9.00, 1000, 2023-10-25T12:01:13.400Z, 2023-10-25T12:01:13.400007Z, 2023-10-25T12:01:13.501, 2023-10-25T12:01:13.501008, [5, 6, 7, 8], 2025],
+I[true, 100, 200, 30, 400, 500.1, 600.0, another_string_6, 9.00, 1000, 2023-10-25T12:01:13.400Z, 2023-10-25T12:01:13.400007Z, 2023-10-25T12:01:13.501, 2023-10-25T12:01:13.501008, [5, 6, 7, 8], 2025],
+I[true, 100, 200, 30, 400, 500.1, 600.0, another_string_7, 9.00, 1000, 2023-10-25T12:01:13.400Z, 2023-10-25T12:01:13.400007Z, 2023-10-25T12:01:13.501, 2023-10-25T12:01:13.501008, [5, 6, 7, 8], 2025],
+I[true, 100, 200, 30, 400, 500.1, 600.0, another_string_8, 9.00, 1000, 2023-10-25T12:01:13.400Z, 2023-10-25T12:01:13.400007Z, 2023-10-25T12:01:13.501, 2023-10-25T12:01:13.501008, [5, 6, 7, 8], 2025],
+I[true, 100, 200, 30, 400, 500.1, 600.0, another_string_9, 9.00, 1000, 2023-10-25T12:01:13.400Z, 2023-10-25T12:01:13.400007Z, 2023-10-25T12:01:13.501, 2023-10-25T12:01:13.501008, [5, 6, 7, 8], 2025],
+I[true, 100, 200, 30, 400, 500.1, 600.0, another_string_0, 9.00, 1000, 2023-10-25T12:01:13.400Z, 2023-10-25T12:01:13.400007Z, 2023-10-25T12:01:13.501, 2023-10-25T12:01:13.501008, [5, 6, 7, 8], 2025],
+I[true, 100, 200, 30, 400, 500.1, 600.0, another_string_1, 9.00, 1000, 2023-10-25T12:01:13.400Z, 2023-10-25T12:01:13.400007Z, 2023-10-25T12:01:13.501, 2023-10-25T12:01:13.501008, [5, 6, 7, 8], 2025],
+I[true, 100, 200, 30, 400, 500.1, 600.0, another_string_2, 9.00, 1000, 2023-10-25T12:01:13.400Z, 2023-10-25T12:01:13.400007Z, 2023-10-25T12:01:13.501, 2023-10-25T12:01:13.501008, [5, 6, 7, 8], 2025],
+I[true, 100, 200, 30, 400, 500.1, 600.0, another_string_3, 9.00, 1000, 2023-10-25T12:01:13.400Z, 2023-10-25T12:01:13.400007Z, 2023-10-25T12:01:13.501, 2023-10-25T12:01:13.501008, [5, 6, 7, 8], 2025],
+I[true, 100, 200, 30, 400, 500.1, 600.0, another_string_4, 9.00, 1000, 2023-10-25T12:01:13.400Z, 2023-10-25T12:01:13.400007Z, 2023-10-25T12:01:13.501, 2023-10-25T12:01:13.501008, [5, 6, 7, 8], 2025],
+I[true, 100, 200, 30, 400, 500.1, 600.0, another_string_5, 9.00, 1000, 2023-10-25T12:01:13.400Z, 2023-10-25T12:01:13.400007Z, 2023-10-25T12:01:13.501, 2023-10-25T12:01:13.501008, [5, 6, 7, 8], 2025],
+I[true, 100, 200, 30, 400, 500.1, 600.0, another_string_6, 9.00, 1000, 2023-10-25T12:01:13.400Z, 2023-10-25T12:01:13.400007Z, 2023-10-25T12:01:13.501, 2023-10-25T12:01:13.501008, [5, 6, 7, 8], 2025],
+I[true, 100, 200, 30, 400, 500.1, 600.0, another_string_7, 9.00, 1000, 2023-10-25T12:01:13.400Z, 2023-10-25T12:01:13.400007Z, 2023-10-25T12:01:13.501, 2023-10-25T12:01:13.501008, [5, 6, 7, 8], 2025],
+I[true, 100, 200, 30, 400, 500.1, 600.0, another_string_8, 9.00, 1000, 2023-10-25T12:01:13.400Z, 2023-10-25T12:01:13.400007Z, 2023-10-25T12:01:13.501, 2023-10-25T12:01:13.501008, [5, 6, 7, 8], 2025],
+I[true, 100, 200, 30, 400, 500.1, 600.0, another_string_9, 9.00, 1000, 2023-10-25T12:01:13.400Z, 2023-10-25T12:01:13.400007Z, 2023-10-25T12:01:13.501, 2023-10-25T12:01:13.501008, [5, 6, 7, 8], 2025],
+I[true, 100, 200, 30, 400, 500.1, 600.0, another_string_0, 9.00, 1000, 2023-10-25T12:01:13.400Z, 2023-10-25T12:01:13.400007Z, 2023-10-25T12:01:13.501, 2023-10-25T12:01:13.501008, [5, 6, 7, 8], 2025],
+I[true, 100, 200, 30, 400, 500.1, 600.0, another_string_1, 9.00, 1000, 2023-10-25T12:01:13.400Z, 2023-10-25T12:01:13.400007Z, 2023-10-25T12:01:13.501, 2023-10-25T12:01:13.501008, [5, 6, 7, 8], 2025],
+I[true, 100, 200, 30, 400, 500.1, 600.0, another_string_2, 9.00, 1000, 2023-10-25T12:01:13.400Z, 2023-10-25T12:01:13.400007Z, 2023-10-25T12:01:13.501, 2023-10-25T12:01:13.501008, [5, 6, 7, 8], 2025]]
to contain exactly in any order:
[+I[true, 100, 200, 30, 400, 500.1, 600.0, another_string_0, 9.00, 1000, 2023-10-25T12:01:13.400Z, 2023-10-25T12:01:13.400007Z, 2023-10-25T12:01:13.501, 2023-10-25T12:01:13.501008, [5, 6, 7, 8], 2025],
+I[true, 100, 200, 30, 400, 500.1, 600.0, another_string_1, 9.00, 1000, 2023-10-25T12:01:13.400Z, 2023-10-25T12:01:13.400007Z, 2023-10-25T12:01:13.501, 2023-10-25T12:01:13.501008, [5, 6, 7, 8], 2025],
+I[true, 100, 200, 30, 400, 500.1, 600.0, another_string_2, 9.00, 1000, 2023-10-25T12:01:13.400Z, 2023-10-25T12:01:13.400007Z, 2023-10-25T12:01:13.501, 2023-10-25T12:01:13.501008, [5, 6, 7, 8], 2025],
+I[true, 100, 200, 30, 400, 500.1, 600.0, another_string_3, 9.00, 1000, 2023-10-25T12:01:13.400Z, 2023-10-25T12:01:13.400007Z, 2023-10-25T12:01:13.501, 2023-10-25T12:01:13.501008, [5, 6, 7, 8], 2025],
+I[true, 100, 200, 30, 400, 500.1, 600.0, another_string_4, 9.00, 1000, 2023-10-25T12:01:13.400Z, 2023-10-25T12:01:13.400007Z, 2023-10-25T12:01:13.501, 2023-10-25T12:01:13.501008, [5, 6, 7, 8], 2025],
+I[true, 100, 200, 30, 400, 500.1, 600.0, another_string_5, 9.00, 1000, 2023-10-25T12:01:13.400Z, 2023-10-25T12:01:13.400007Z, 2023-10-25T12:01:13.501, 2023-10-25T12:01:13.501008, [5, 6, 7, 8], 2025],
+I[true, 100, 200, 30, 400, 500.1, 600.0, another_string_6, 9.00, 1000, 2023-10-25T12:01:13.400Z, 2023-10-25T12:01:13.400007Z, 2023-10-25T12:01:13.501, 2023-10-25T12:01:13.501008, [5, 6, 7, 8], 2025],
+I[true, 100, 200, 30, 400, 500.1, 600.0, another_string_7, 9.00, 1000, 2023-10-25T12:01:13.400Z, 2023-10-25T12:01:13.400007Z, 2023-10-25T12:01:13.501, 2023-10-25T12:01:13.501008, [5, 6, 7, 8], 2025],
+I[true, 100, 200, 30, 400, 500.1, 600.0, another_string_8, 9.00, 1000, 2023-10-25T12:01:13.400Z, 2023-10-25T12:01:13.400007Z, 2023-10-25T12:01:13.501, 2023-10-25T12:01:13.501008, [5, 6, 7, 8], 2025],
+I[true, 100, 200, 30, 400, 500.1, 600.0, another_string_9, 9.00, 1000, 2023-10-25T12:01:13.400Z, 2023-10-25T12:01:13.400007Z, 2023-10-25T12:01:13.501, 2023-10-25T12:01:13.501008, [5, 6, 7, 8], 2025],
+I[true, 100, 200, 30, 400, 500.1, 600.0, another_string_0, 9.00, 1000, 2023-10-25T12:01:13.400Z, 2023-10-25T12:01:13.400007Z, 2023-10-25T12:01:13.501, 2023-10-25T12:01:13.501008, [5, 6, 7, 8], 2025],
+I[true, 100, 200, 30, 400, 500.1, 600.0, another_string_1, 9.00, 1000, 2023-10-25T12:01:13.400Z, 2023-10-25T12:01:13.400007Z, 2023-10-25T12:01:13.501, 2023-10-25T12:01:13.501008, [5, 6, 7, 8], 2025],
+I[true, 100, 200, 30, 400, 500.1, 600.0, another_string_2, 9.00, 1000, 2023-10-25T12:01:13.400Z, 2023-10-25T12:01:13.400007Z, 2023-10-25T12:01:13.501, 2023-10-25T12:01:13.501008, [5, 6, 7, 8], 2025],
+I[true, 100, 200, 30, 400, 500.1, 600.0, another_string_3, 9.00, 1000, 2023-10-25T12:01:13.400Z, 2023-10-25T12:01:13.400007Z, 2023-10-25T12:01:13.501, 2023-10-25T12:01:13.501008, [5, 6, 7, 8], 2025],
+I[true, 100, 200, 30, 400, 500.1, 600.0, another_string_4, 9.00, 1000, 2023-10-25T12:01:13.400Z, 2023-10-25T12:01:13.400007Z, 2023-10-25T12:01:13.501, 2023-10-25T12:01:13.501008, [5, 6, 7, 8], 2025],
+I[true, 100, 200, 30, 400, 500.1, 600.0, another_string_5, 9.00, 1000, 2023-10-25T12:01:13.400Z, 2023-10-25T12:01:13.400007Z, 2023-10-25T12:01:13.501, 2023-10-25T12:01:13.501008, [5, 6, 7, 8], 2025],
+I[true, 100, 200, 30, 400, 500.1, 600.0, another_string_6, 9.00, 1000, 2023-10-25T12:01:13.400Z, 2023-10-25T12:01:13.400007Z, 2023-10-25T12:01:13.501, 2023-10-25T12:01:13.501008, [5, 6, 7, 8], 2025],
+I[true, 100, 200, 30, 400, 500.1, 600.0, another_string_7, 9.00, 1000, 2023-10-25T12:01:13.400Z, 2023-10-25T12:01:13.400007Z, 2023-10-25T12:01:13.501, 2023-10-25T12:01:13.501008, [5, 6, 7, 8], 2025],
+I[true, 100, 200, 30, 400, 500.1, 600.0, another_string_8, 9.00, 1000, 2023-10-25T12:01:13.400Z, 2023-10-25T12:01:13.400007Z, 2023-10-25T12:01:13.501, 2023-10-25T12:01:13.501008, [5, 6, 7, 8], 2025],
+I[true, 100, 200, 30, 400, 500.1, 600.0, another_string_9, 9.00, 1000, 2023-10-25T12:01:13.400Z, 2023-10-25T12:01:13.400007Z, 2023-10-25T12:01:13.501, 2023-10-25T12:01:13.501008, [5, 6, 7, 8], 2025],
+I[true, 100, 200, 30, 400, 500.1, 600.0, another_string_0, 9.00, 1000, 2023-10-25T12:01:13.400Z, 2023-10-25T12:01:13.400007Z, 2023-10-25T12:01:13.501, 2023-10-25T12:01:13.501008, [5, 6, 7, 8], 2025],
+I[true, 100, 200, 30, 400, 500.1, 600.0, another_string_1, 9.00, 1000, 2023-10-25T12:01:13.400Z, 2023-10-25T12:01:13.400007Z, 2023-10-25T12:01:13.501, 2023-10-25T12:01:13.501008, [5, 6, 7, 8], 2025],
+I[true, 100, 200, 30, 400, 500.1, 600.0, another_string_2, 9.00, 1000, 2023-10-25T12:01:13.400Z, 2023-10-25T12:01:13.400007Z, 2023-10-25T12:01:13.501, 2023-10-25T12:01:13.501008, [5, 6, 7, 8], 2025],
+I[true, 100, 200, 30, 400, 500.1, 600.0, another_string_3, 9.00, 1000, 2023-10-25T12:01:13.400Z, 2023-10-25T12:01:13.400007Z, 2023-10-25T12:01:13.501, 2023-10-25T12:01:13.501008, [5, 6, 7, 8], 2025],
+I[true, 100, 200, 30, 400, 500.1, 600.0, another_string_4, 9.00, 1000, 2023-10-25T12:01:13.400Z, 2023-10-25T12:01:13.400007Z, 2023-10-25T12:01:13.501, 2023-10-25T12:01:13.501008, [5, 6, 7, 8], 2025],
+I[true, 100, 200, 30, 400, 500.1, 600.0, another_string_5, 9.00, 1000, 2023-10-25T12:01:13.400Z, 2023-10-25T12:01:13.400007Z, 2023-10-25T12:01:13.501, 2023-10-25T12:01:13.501008, [5, 6, 7, 8], 2025],
+I[true, 100, 200, 30, 400, 500.1, 600.0, another_string_6, 9.00, 1000, 2023-10-25T12:01:13.400Z, 2023-10-25T12:01:13.400007Z, 2023-10-25T12:01:13.501, 2023-10-25T12:01:13.501008, [5, 6, 7, 8], 2025],
+I[true, 100, 200, 30, 400, 500.1, 600.0, another_string_7, 9.00, 1000, 2023-10-25T12:01:13.400Z, 2023-10-25T12:01:13.400007Z, 2023-10-25T12:01:13.501, 2023-10-25T12:01:13.501008, [5, 6, 7, 8], 2025],
+I[true, 100, 200, 30, 400, 500.1, 600.0, another_string_8, 9.00, 1000, 2023-10-25T12:01:13.400Z, 2023-10-25T12:01:13.400007Z, 2023-10-25T12:01:13.501, 2023-10-25T12:01:13.501008, [5, 6, 7, 8], 2025],
+I[true, 100, 200, 30, 400, 500.1, 600.0, another_string_9, 9.00, 1000, 2023-10-25T12:01:13.400Z, 2023-10-25T12:01:13.400007Z, 2023-10-25T12:01:13.501, 2023-10-25T12:01:13.501008, [5, 6, 7, 8], 2025],
+I[true, 100, 200, 30, 400, 500.1, 600.0, another_string_0, 9.00, 1000, 2023-10-25T12:01:13.400Z, 2023-10-25T12:01:13.400007Z, 2023-10-25T12:01:13.501, 2023-10-25T12:01:13.501008, [5, 6, 7, 8], 2025],
+I[true, 100, 200, 30, 400, 500.1, 600.0, another_string_1, 9.00, 1000, 2023-10-25T12:01:13.400Z, 2023-10-25T12:01:13.400007Z, 2023-10-25T12:01:13.501, 2023-10-25T12:01:13.501008, [5, 6, 7, 8], 2025],
+I[true, 100, 200, 30, 400, 500.1, 600.0, another_string_2, 9.00, 1000, 2023-10-25T12:01:13.400Z, 2023-10-25T12:01:13.400007Z, 2023-10-25T12:01:13.501, 2023-10-25T12:01:13.501008, [5, 6, 7, 8], 2025]]
but the following elements were unexpected:
[+I[true, 100, 200, 30, 400, 500.1, 600.0, another_string_0, 9.00, 1000, 2023-10-25T12:01:13.400Z, 2023-10-25T12:01:13.400007Z, 2023-10-25T12:01:13.501, 2023-10-25T12:01:13.501008, [5, 6, 7, 8], 2025],
+I[true, 100, 200, 30, 400, 500.1, 600.0, another_string_1, 9.00, 1000, 2023-10-25T12:01:13.400Z, 2023-10-25T12:01:13.400007Z, 2023-10-25T12:01:13.501, 2023-10-25T12:01:13.501008, [5, 6, 7, 8], 2025],
+I[true, 100, 200, 30, 400, 500.1, 600.0, another_string_2, 9.00, 1000, 2023-10-25T12:01:13.400Z, 2023-10-25T12:01:13.400007Z, 2023-10-25T12:01:13.501, 2023-10-25T12:01:13.501008, [5, 6, 7, 8], 2025],
+I[true, 100, 200, 30, 400, 500.1, 600.0, another_string_3, 9.00, 1000, 2023-10-25T12:01:13.400Z, 2023-10-25T12:01:13.400007Z, 2023-10-25T12:01:13.501, 2023-10-25T12:01:13.501008, [5, 6, 7, 8], 2025],
+I[true, 100, 200, 30, 400, 500.1, 600.0, another_string_4, 9.00, 1000, 2023-10-25T12:01:13.400Z, 2023-10-25T12:01:13.400007Z, 2023-10-25T12:01:13.501, 2023-10-25T12:01:13.501008, [5, 6, 7, 8], 2025],
+I[true, 100, 200, 30, 400, 500.1, 600.0, another_string_5, 9.00, 1000, 2023-10-25T12:01:13.400Z, 2023-10-25T12:01:13.400007Z, 2023-10-25T12:01:13.501, 2023-10-25T12:01:13.501008, [5, 6, 7, 8], 2025],
+I[true, 100, 200, 30, 400, 500.1, 600.0, another_string_6, 9.00, 1000, 2023-10-25T12:01:13.400Z, 2023-10-25T12:01:13.400007Z, 2023-10-25T12:01:13.501, 2023-10-25T12:01:13.501008, [5, 6, 7, 8], 2025],
+I[true, 100, 200, 30, 400, 500.1, 600.0, another_string_7, 9.00, 1000, 2023-10-25T12:01:13.400Z, 2023-10-25T12:01:13.400007Z, 2023-10-25T12:01:13.501, 2023-10-25T12:01:13.501008, [5, 6, 7, 8], 2025],
+I[true, 100, 200, 30, 400, 500.1, 600.0, another_string_8, 9.00, 1000, 2023-10-25T12:01:13.400Z, 2023-10-25T12:01:13.400007Z, 2023-10-25T12:01:13.501, 2023-10-25T12:01:13.501008, [5, 6, 7, 8], 2025],
+I[true, 100, 200, 30, 400, 500.1, 600.0, another_string_9, 9.00, 1000, 2023-10-25T12:01:13.400Z, 2023-10-25T12:01:13.400007Z, 2023-10-25T12:01:13.501, 2023-10-25T12:01:13.501008, [5, 6, 7, 8], 2025],
+I[true, 100, 200, 30, 400, 500.1, 600.0, another_string_0, 9.00, 1000, 2023-10-25T12:01:13.400Z, 2023-10-25T12:01:13.400007Z, 2023-10-25T12:01:13.501, 2023-10-25T12:01:13.501008, [5, 6, 7, 8], 2025],
+I[true, 100, 200, 30, 400, 500.1, 600.0, another_string_1, 9.00, 1000, 2023-10-25T12:01:13.400Z, 2023-10-25T12:01:13.400007Z, 2023-10-25T12:01:13.501, 2023-10-25T12:01:13.501008, [5, 6, 7, 8], 2025],
+I[true, 100, 200, 30, 400, 500.1, 600.0, another_string_2, 9.00, 1000, 2023-10-25T12:01:13.400Z, 2023-10-25T12:01:13.400007Z, 2023-10-25T12:01:13.501, 2023-10-25T12:01:13.501008, [5, 6, 7, 8], 2025],
+I[true, 100, 200, 30, 400, 500.1, 600.0, another_string_3, 9.00, 1000, 2023-10-25T12:01:13.400Z, 2023-10-25T12:01:13.400007Z, 2023-10-25T12:01:13.501, 2023-10-25T12:01:13.501008, [5, 6, 7, 8], 2025],
+I[true, 100, 200, 30, 400, 500.1, 600.0, another_string_4, 9.00, 1000, 2023-10-25T12:01:13.400Z, 2023-10-25T12:01:13.400007Z, 2023-10-25T12:01:13.501, 2023-10-25T12:01:13.501008, [5, 6, 7, 8], 2025],
+I[true, 100, 200, 30, 400, 500.1, 600.0, another_string_5, 9.00, 1000, 2023-10-25T12:01:13.400Z, 2023-10-25T12:01:13.400007Z, 2023-10-25T12:01:13.501, 2023-10-25T12:01:13.501008, [5, 6, 7, 8], 2025],
+I[true, 100, 200, 30, 400, 500.1, 600.0, another_string_6, 9.00, 1000, 2023-10-25T12:01:13.400Z, 2023-10-25T12:01:13.400007Z, 2023-10-25T12:01:13.501, 2023-10-25T12:01:13.501008, [5, 6, 7, 8], 2025],
+I[true, 100, 200, 30, 400, 500.1, 600.0, another_string_7, 9.00, 1000, 2023-10-25T12:01:13.400Z, 2023-10-25T12:01:13.400007Z, 2023-10-25T12:01:13.501, 2023-10-25T12:01:13.501008, [5, 6, 7, 8], 2025],
+I[true, 100, 200, 30, 400, 500.1, 600.0, another_string_8, 9.00, 1000, 2023-10-25T12:01:13.400Z, 2023-10-25T12:01:13.400007Z, 2023-10-25T12:01:13.501, 2023-10-25T12:01:13.501008, [5, 6, 7, 8], 2025],
+I[true, 100, 200, 30, 400, 500.1, 600.0, another_string_9, 9.00, 1000, 2023-10-25T12:01:13.400Z, 2023-10-25T12:01:13.400007Z, 2023-10-25T12:01:13.501, 2023-10-25T12:01:13.501008, [5, 6, 7, 8], 2025],
+I[true, 100, 200, 30, 400, 500.1, 600.0, another_string_0, 9.00, 1000, 2023-10-25T12:01:13.400Z, 2023-10-25T12:01:13.400007Z, 2023-10-25T12:01:13.501, 2023-10-25T12:01:13.501008, [5, 6, 7, 8], 2025],
+I[true, 100, 200, 30, 400, 500.1, 600.0, another_string_1, 9.00, 1000, 2023-10-25T12:01:13.400Z, 2023-10-25T12:01:13.400007Z, 2023-10-25T12:01:13.501, 2023-10-25T12:01:13.501008, [5, 6, 7, 8], 2025],
+I[true, 100, 200, 30, 400, 500.1, 600.0, another_string_2, 9.00, 1000, 2023-10-25T12:01:13.400Z, 2023-10-25T12:01:13.400007Z, 2023-10-25T12:01:13.501, 2023-10-25T12:01:13.501008, [5, 6, 7, 8], 2025],
+I[true, 100, 200, 30, 400, 500.1, 600.0, another_string_3, 9.00, 1000, 2023-10-25T12:01:13.400Z, 2023-10-25T12:01:13.400007Z, 2023-10-25T12:01:13.501, 2023-10-25T12:01:13.501008, [5, 6, 7, 8], 2025],
+I[true, 100, 200, 30, 400, 500.1, 600.0, another_string_4, 9.00, 1000, 2023-10-25T12:01:13.400Z, 2023-10-25T12:01:13.400007Z, 2023-10-25T12:01:13.501, 2023-10-25T12:01:13.501008, [5, 6, 7, 8], 2025],
+I[true, 100, 200, 30, 400, 500.1, 600.0, another_string_5, 9.00, 1000, 2023-10-25T12:01:13.400Z, 2023-10-25T12:01:13.400007Z, 2023-10-25T12:01:13.501, 2023-10-25T12:01:13.501008, [5, 6, 7, 8], 2025],
+I[true, 100, 200, 30, 400, 500.1, 600.0, another_string_6, 9.00, 1000, 2023-10-25T12:01:13.400Z, 2023-10-25T12:01:13.400007Z, 2023-10-25T12:01:13.501, 2023-10-25T12:01:13.501008, [5, 6, 7, 8], 2025],
+I[true, 100, 200, 30, 400, 500.1, 600.0, another_string_7, 9.00, 1000, 2023-10-25T12:01:13.400Z, 2023-10-25T12:01:13.400007Z, 2023-10-25T12:01:13.501, 2023-10-25T12:01:13.501008, [5, 6, 7, 8], 2025],
+I[true, 100, 200, 30, 400, 500.1, 600.0, another_string_8, 9.00, 1000, 2023-10-25T12:01:13.400Z, 2023-10-25T12:01:13.400007Z, 2023-10-25T12:01:13.501, 2023-10-25T12:01:13.501008, [5, 6, 7, 8], 2025],
+I[true, 100, 200, 30, 400, 500.1, 600.0, another_string_9, 9.00, 1000, 2023-10-25T12:01:13.400Z, 2023-10-25T12:01:13.400007Z, 2023-10-25T12:01:13.501, 2023-10-25T12:01:13.501008, [5, 6, 7, 8], 2025],
+I[true, 100, 200, 30, 400, 500.1, 600.0, another_string_0, 9.00, 1000, 2023-10-25T12:01:13.400Z, 2023-10-25T12:01:13.400007Z, 2023-10-25T12:01:13.501, 2023-10-25T12:01:13.501008, [5, 6, 7, 8], 2025],
+I[true, 100, 200, 30, 400, 500.1, 600.0, another_string_1, 9.00, 1000, 2023-10-25T12:01:13.400Z, 2023-10-25T12:01:13.400007Z, 2023-10-25T12:01:13.501, 2023-10-25T12:01:13.501008, [5, 6, 7, 8], 2025],
+I[true, 100, 200, 30, 400, 500.1, 600.0, another_string_2, 9.00, 1000, 2023-10-25T12:01:13.400Z, 2023-10-25T12:01:13.400007Z, 2023-10-25T12:01:13.501, 2023-10-25T12:01:13.501008, [5, 6, 7, 8], 2025]]
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Apply partition filter to lake in flink source [fluss]
Posted by "luoyuxia (via GitHub)" <gi...@apache.org>.
luoyuxia commented on code in PR #1549:
URL: https://github.com/apache/fluss/pull/1549#discussion_r2286797043
##########
fluss-flink/fluss-flink-common/src/test/java/com/alibaba/fluss/flink/source/FlinkTableSourceITCase.java:
##########
@@ -1066,6 +1066,52 @@ void testStreamingReadMultiPartitionPushDown() throws Exception {
assertResultsIgnoreOrder(rowIter, expectedRowValues, true);
}
+ @Test
+ void testStreamingReadMultiPartitionPushDownReverseOrderCondition() throws Exception {
Review Comment:
Let's remove this test method since we already have a test in `FlinkUnionReadLogTableITCase` which is what we really need.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Apply partition filter to lake in flink source [fluss]
Posted by "luoyuxia (via GitHub)" <gi...@apache.org>.
luoyuxia commented on PR #1549:
URL: https://github.com/apache/fluss/pull/1549#issuecomment-3203969886
@SML0127 Thanks for the pr. I append a commit to fix the test failure. It turns a bug in the code base. PTAL. If you have no problem, I'll merge it then.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Apply partition filter to lake in flink source [fluss]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #1549:
URL: https://github.com/apache/fluss/pull/1549#discussion_r2281132677
##########
fluss-flink/fluss-flink-common/src/main/java/com/alibaba/fluss/flink/lake/LakeSplitGenerator.java:
##########
@@ -100,7 +109,73 @@ public List<SourceSplitBase> generateHybridLakeSplits() throws Exception {
.createPlanner(
(LakeSource.PlannerContext) lakeSnapshotInfo::getSnapshotId)
.plan());
+
+ if (lakeSplits.isEmpty()) {
+ return Collections.emptyList();
+ }
+
+ // first, filter lake splits by partition filters
+ if (!partitionFilters.isEmpty()) {
+ List<String> partitionKeys = fileStoreTable.partitionKeys();
+ boolean isSinglePartitionKey = partitionKeys.size() == 1;
+
+ if (isSinglePartitionKey) {
+ // in single key, name == value
+ Set<String> allowedPartitionNames =
+ partitionFilters.stream()
+ .map(fieldEqual -> String.valueOf(fieldEqual.equalValue))
+ .collect(Collectors.toSet());
+ lakeSplits
+ .entrySet()
+ .removeIf(entry -> !allowedPartitionNames.contains(entry.getKey()));
+
+ } else {
+ // multi partition key
+ List<DataField> dataFields = tableInfo.getRowType().getFields();
+ Map<Integer, String> partitionKeyIdxToValueMap = new java.util.HashMap<>();
+
+ // build partition key idx to expected value map
+ for (FieldEqual fieldEqual : partitionFilters) {
+ String fieldName = dataFields.get(fieldEqual.fieldIndex).getName();
+ int partitionKeyIdx = partitionKeys.indexOf(fieldName);
+ if (partitionKeyIdx >= 0) {
+ partitionKeyIdxToValueMap.put(
+ partitionKeyIdx, String.valueOf(fieldEqual.equalValue));
+ }
+ }
+
+ if (!partitionKeyIdxToValueMap.isEmpty()) {
+ lakeSplits
+ .entrySet()
+ .removeIf(
+ entry -> {
+ String partitionName = entry.getKey(); // e.g. "20250815$08"
+ String[] partitionValues =
+ partitionName.split(
+ Pattern.quote(
+ PARTITION_SPEC_SEPARATOR)); // ["20250815", "08"]
+
+ // check if all partition key idx match to expected values
+ for (Map.Entry<Integer, String> mapEntry :
+ partitionKeyIdxToValueMap.entrySet()) {
+ int partitionKeyIdx = mapEntry.getKey();
+ String expectedValue = mapEntry.getValue();
+
+ // idx out of bounds or value not match
+ if (partitionKeyIdx >= partitionValues.length
+ || !expectedValue.equals(
+ partitionValues[partitionKeyIdx])) {
Review Comment:
This bounds check could incorrectly filter out valid partitions. If `partitionKeyIdx` is beyond the partition values array, it suggests a mismatch between the expected partition structure and actual data, which should be handled as an error rather than silently filtering out the partition.
```suggestion
if (partitionKeyIdx >= partitionValues.length) {
throw new IllegalStateException(
String.format(
"Partition key index %d is out of bounds for partition '%s'. Expected partition keys: %s, actual partition values: %s",
partitionKeyIdx,
partitionName,
partitionKeys,
java.util.Arrays.toString(partitionValues)
)
);
}
if (!expectedValue.equals(partitionValues[partitionKeyIdx])) {
```
##########
fluss-flink/fluss-flink-common/src/main/java/com/alibaba/fluss/flink/source/FlinkTableSource.java:
##########
@@ -540,6 +592,18 @@ private int[] getKeyRowProjection() {
return projection;
}
+ @Nullable
+ private Object toPaimonLiteralForPartition(
+ com.alibaba.fluss.types.DataType flussDataType, Object equalValue) {
+ String typeSummary = flussDataType.toString().toUpperCase();
+ if (typeSummary.contains("CHAR") || typeSummary.contains("STRING")) {
Review Comment:
The string-based type checking using `contains()` is fragile and could match unintended types. Consider using proper type checking with `flussDataType instanceof CharType` or `flussDataType instanceof VarCharType` for more robust type detection.
```suggestion
// Use instanceof for robust type checking
if (flussDataType instanceof CharType
|| flussDataType instanceof VarCharType
|| flussDataType instanceof StringType) {
```
##########
fluss-flink/fluss-flink-common/src/main/java/com/alibaba/fluss/flink/lake/LakeSplitGenerator.java:
##########
@@ -100,7 +109,73 @@ public List<SourceSplitBase> generateHybridLakeSplits() throws Exception {
.createPlanner(
(LakeSource.PlannerContext) lakeSnapshotInfo::getSnapshotId)
.plan());
+
+ if (lakeSplits.isEmpty()) {
+ return Collections.emptyList();
+ }
+
+ // first, filter lake splits by partition filters
+ if (!partitionFilters.isEmpty()) {
+ List<String> partitionKeys = fileStoreTable.partitionKeys();
+ boolean isSinglePartitionKey = partitionKeys.size() == 1;
+
+ if (isSinglePartitionKey) {
+ // in single key, name == value
+ Set<String> allowedPartitionNames =
+ partitionFilters.stream()
+ .map(fieldEqual -> String.valueOf(fieldEqual.equalValue))
+ .collect(Collectors.toSet());
+ lakeSplits
+ .entrySet()
+ .removeIf(entry -> !allowedPartitionNames.contains(entry.getKey()));
+
+ } else {
+ // multi partition key
+ List<DataField> dataFields = tableInfo.getRowType().getFields();
+ Map<Integer, String> partitionKeyIdxToValueMap = new java.util.HashMap<>();
+
+ // build partition key idx to expected value map
+ for (FieldEqual fieldEqual : partitionFilters) {
+ String fieldName = dataFields.get(fieldEqual.fieldIndex).getName();
+ int partitionKeyIdx = partitionKeys.indexOf(fieldName);
+ if (partitionKeyIdx >= 0) {
+ partitionKeyIdxToValueMap.put(
+ partitionKeyIdx, String.valueOf(fieldEqual.equalValue));
+ }
+ }
+
+ if (!partitionKeyIdxToValueMap.isEmpty()) {
+ lakeSplits
+ .entrySet()
+ .removeIf(
+ entry -> {
+ String partitionName = entry.getKey(); // e.g. "20250815$08"
+ String[] partitionValues =
+ partitionName.split(
+ Pattern.quote(
+ PARTITION_SPEC_SEPARATOR)); // ["20250815", "08"]
Review Comment:
The `Pattern.quote()` call is executed for every partition entry during filtering. Consider pre-compiling the pattern outside the removeIf lambda to avoid repeated regex compilation overhead.
```suggestion
// Precompile the partition separator pattern to avoid repeated regex compilation
Pattern partitionSeparatorPattern = Pattern.compile(Pattern.quote(PARTITION_SPEC_SEPARATOR));
lakeSplits
.entrySet()
.removeIf(
entry -> {
String partitionName = entry.getKey(); // e.g. "20250815$08"
String[] partitionValues =
partitionSeparatorPattern.split(partitionName); // ["20250815", "08"]
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Apply partition filter to lake in flink source [fluss]
Posted by "SML0127 (via GitHub)" <gi...@apache.org>.
SML0127 commented on code in PR #1549:
URL: https://github.com/apache/fluss/pull/1549#discussion_r2285942988
##########
fluss-lake/fluss-lake-paimon/src/test/java/com/alibaba/fluss/lake/paimon/flink/FlinkUnionReadLogTableITCase.java:
##########
@@ -95,11 +95,37 @@ void testReadLogTableFullType(boolean isPartitioned) throws Exception {
.map(row -> Row.of(row.getField(1)))
.collect(Collectors.toList());
assertThat(actual).containsExactlyInAnyOrderElementsOf(expected);
+
+ if (isPartitioned) {
+ // get first partition
+ String partition = waitUntilPartitions(t1).values().iterator().next();
+ String sqlWithPartitionFilter =
+ "select * FROM " + tableName + " WHERE p = '" + partition + "'";
+
+ String plan = batchTEnv.explainSql(sqlWithPartitionFilter);
+
+ // check if the plan contains partition filter
+ assertThat(plan)
+ .contains("TableSourceScan(")
+ .contains("filter=[=(p, _UTF-16LE'" + partition + "'");
+
+ List<Row> expectedFiltered =
+ writtenRows.stream()
+ .filter(r -> partition.equals(r.getField(15)))
+ .collect(Collectors.toList());
+
+ List<Row> actualFiltered =
+ CollectionUtil.iteratorToList(
+ batchTEnv.executeSql(sqlWithPartitionFilter).collect());
+
+ assertThat(actualFiltered).containsExactlyInAnyOrderElementsOf(expectedFiltered);
+ }
Review Comment:
The newly added test case is failing; I'm debugging it.
```
plan =
== Abstract Syntax Tree ==
LogicalProject(f_boolean=[$0], f_byte=[$1], f_short=[$2], f_int=[$3], f_long=[$4], f_float=[$5], f_double=[$6], f_string=[$7], f_decimal1=[$8], f_decimal2=[$9], f_timestamp_ltz1=[$10], f_timestamp_ltz2=[$11], f_timestamp_ntz1=[$12], f_timestamp_ntz2=[$13], f_binary=[$14], p=[$15])
+- LogicalFilter(condition=[=($15, _UTF-16LE'2025')])
+- LogicalTableScan(table=[[testcatalog, fluss, logTable_partitioned]])
== Optimized Physical Plan ==
Calc(select=[f_boolean, f_byte, f_short, f_int, f_long, f_float, f_double, f_string, f_decimal1, f_decimal2, f_timestamp_ltz1, f_timestamp_ltz2, f_timestamp_ntz1, f_timestamp_ntz2, f_binary, CAST('2025' AS VARCHAR(2147483647)) AS p])
+- TableSourceScan(table=[[testcatalog, fluss, logTable_partitioned, filter=[=(p, _UTF-16LE'2025':VARCHAR(2147483647) CHARACTER SET "UTF-16LE")], project=[f_boolean, f_byte, f_short, f_int, f_long, f_float, f_double, f_string, f_decimal1, f_decimal2, f_timestamp_ltz1, f_timestamp_ltz2, f_timestamp_ntz1, f_timestamp_ntz2, f_binary]]], fields=[f_boolean, f_byte, f_short, f_int, f_long, f_float, f_double, f_string, f_decimal1, f_decimal2, f_timestamp_ltz1, f_timestamp_ltz2, f_timestamp_ntz1, f_timestamp_ntz2, f_binary])
== Optimized Execution Plan ==
Calc(select=[f_boolean, f_byte, f_short, f_int, f_long, f_float, f_double, f_string, f_decimal1, f_decimal2, f_timestamp_ltz1, f_timestamp_ltz2, f_timestamp_ntz1, f_timestamp_ntz2, f_binary, CAST('2025' AS VARCHAR(2147483647)) AS p])
+- TableSourceScan(table=[[testcatalog, fluss, logTable_partitioned, filter=[=(p, _UTF-16LE'2025':VARCHAR(2147483647) CHARACTER SET "UTF-16LE")], project=[f_boolean, f_byte, f_short, f_int, f_long, f_float, f_double, f_string, f_decimal1, f_decimal2, f_timestamp_ltz1, f_timestamp_ltz2, f_timestamp_ntz1, f_timestamp_ntz2, f_binary]]], fields=[f_boolean, f_byte, f_short, f_int, f_long, f_float, f_double, f_string, f_decimal1, f_decimal2, f_timestamp_ltz1, f_timestamp_ltz2, f_timestamp_ntz1, f_timestamp_ntz2, f_binary])
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Apply partition filter to lake in flink source [fluss]
Posted by "luoyuxia (via GitHub)" <gi...@apache.org>.
luoyuxia commented on code in PR #1549:
URL: https://github.com/apache/fluss/pull/1549#discussion_r2281175629
##########
fluss-flink/fluss-flink-common/src/main/java/com/alibaba/fluss/flink/lake/LakeSplitGenerator.java:
##########
@@ -73,13 +79,16 @@ public LakeSplitGenerator(
LakeSource<LakeSplit> lakeSource,
OffsetsInitializer.BucketOffsetsRetriever bucketOffsetsRetriever,
OffsetsInitializer stoppingOffsetInitializer,
- int bucketCount) {
+ int bucketCount,
+ List<FieldEqual> partitionFilters) {
Review Comment:
We don't need to modify `LakeSplitGenerator`, if we already push filter to `LakeSource`. The splits generate by ``LakeSource` should always contain the splits match the partition filter.
##########
fluss-flink/fluss-flink-common/src/main/java/com/alibaba/fluss/flink/source/FlinkTableSource.java:
##########
@@ -449,11 +455,57 @@ && hasPrimaryKey()
getPartitionKeyTypes(),
acceptedFilters,
remainingFilters,
- FLINK_INTERNAL_VALUE);
+ FLUSS_INTERNAL_VALUE);
+
// partitions are filtered by string representations, convert the equals to string first
- fieldEquals = stringifyFieldEquals(fieldEquals);
+ partitionFilters = stringifyFieldEquals(fieldEquals);
+
+ // lake source is not null
+ if (lakeSource != null) {
+ // and exist field equals, push down to lake source
+ if (!fieldEquals.isEmpty()) {
+
+ // convert flink row type to fluss row type
+ RowType flussRowType = FlinkConversions.toFlussRowType(tableOutputType);
+ List<Predicate> lakePredicates = new ArrayList<>();
+
+ for (FieldEqual fieldEqual : fieldEquals) {
+ int idx = fieldEqual.fieldIndex;
+ String fieldName = tableOutputType.getFieldNames().get(idx);
+ com.alibaba.fluss.types.DataType flussDataType =
+ flussRowType.getTypeAt(idx);
+
+ // convert to Paimon literal for partition
+ Object literal =
+ toPaimonLiteralForPartition(flussDataType, fieldEqual.equalValue);
+
+ if (literal == null) {
+ continue;
+ }
+
+ lakePredicates.add(
+ new LeafPredicate(
+ Equal.INSTANCE,
+ flussDataType,
+ idx,
+ fieldName,
+ Collections.singletonList(literal)));
+ }
+
+ if (!lakePredicates.isEmpty()) {
+ final LakeSource.FilterPushDownResult filterPushDownResult =
+ lakeSource.withFilters(lakePredicates);
+ if (filterPushDownResult.acceptedPredicates().size()
+ != lakePredicates.size()) {
+ LOG.warn("Some partition filters are not accepted by the lake source");
Review Comment:
If the lake source can't accept this predicate, we shouldn't consider this filter i accepted by Fluss, otherwise, it'll cause data wrong.
For example, it'll still read the data that doesn't match the predicates, but the flink mistake it as pushed down and won't filter it in Flink side, so that's data wrong
```
return Result.of(Collections.emptyList(), filters);
```
##########
fluss-flink/fluss-flink-common/src/main/java/com/alibaba/fluss/flink/source/FlinkTableSource.java:
##########
@@ -449,11 +455,57 @@ && hasPrimaryKey()
getPartitionKeyTypes(),
acceptedFilters,
remainingFilters,
- FLINK_INTERNAL_VALUE);
+ FLUSS_INTERNAL_VALUE);
+
// partitions are filtered by string representations, convert the equals to string first
- fieldEquals = stringifyFieldEquals(fieldEquals);
+ partitionFilters = stringifyFieldEquals(fieldEquals);
+
+ // lake source is not null
+ if (lakeSource != null) {
+ // and exist field equals, push down to lake source
+ if (!fieldEquals.isEmpty()) {
+
+ // convert flink row type to fluss row type
+ RowType flussRowType = FlinkConversions.toFlussRowType(tableOutputType);
+ List<Predicate> lakePredicates = new ArrayList<>();
+
+ for (FieldEqual fieldEqual : fieldEquals) {
+ int idx = fieldEqual.fieldIndex;
+ String fieldName = tableOutputType.getFieldNames().get(idx);
+ com.alibaba.fluss.types.DataType flussDataType =
+ flussRowType.getTypeAt(idx);
+
+ // convert to Paimon literal for partition
+ Object literal =
Review Comment:
We don't need to convert to paimon literal since fluss literal to paimon literal is already done in `FlussToPaimonPredicateConverter`
##########
fluss-flink/fluss-flink-common/src/main/java/com/alibaba/fluss/flink/source/FlinkTableSource.java:
##########
@@ -540,6 +592,18 @@ private int[] getKeyRowProjection() {
return projection;
}
+ @Nullable
+ private Object toPaimonLiteralForPartition(
Review Comment:
we can remove this.
##########
fluss-flink/fluss-flink-common/src/main/java/com/alibaba/fluss/flink/source/FlinkTableSource.java:
##########
@@ -449,11 +455,57 @@ && hasPrimaryKey()
getPartitionKeyTypes(),
acceptedFilters,
remainingFilters,
- FLINK_INTERNAL_VALUE);
+ FLUSS_INTERNAL_VALUE);
+
// partitions are filtered by string representations, convert the equals to string first
- fieldEquals = stringifyFieldEquals(fieldEquals);
+ partitionFilters = stringifyFieldEquals(fieldEquals);
+
+ // lake source is not null
+ if (lakeSource != null) {
+ // and exist field equals, push down to lake source
+ if (!fieldEquals.isEmpty()) {
+
+ // convert flink row type to fluss row type
+ RowType flussRowType = FlinkConversions.toFlussRowType(tableOutputType);
+ List<Predicate> lakePredicates = new ArrayList<>();
+
+ for (FieldEqual fieldEqual : fieldEquals) {
Review Comment:
nit:
the peiece of code can be simplified to
```
PredicateBuilder predicateBuilder = new PredicateBuilder(flussRowType);
for (FieldEqual fieldEqual : fieldEquals) {
lakePredicates.add(
predicateBuilder.equal(
fieldEqual.fieldIndex, fieldEqual.equalValue));
}
```
##########
fluss-flink/fluss-flink-common/src/main/java/com/alibaba/fluss/flink/source/FlinkTableSource.java:
##########
@@ -449,11 +455,57 @@ && hasPrimaryKey()
getPartitionKeyTypes(),
acceptedFilters,
remainingFilters,
- FLINK_INTERNAL_VALUE);
+ FLUSS_INTERNAL_VALUE);
+
// partitions are filtered by string representations, convert the equals to string first
- fieldEquals = stringifyFieldEquals(fieldEquals);
+ partitionFilters = stringifyFieldEquals(fieldEquals);
+
+ // lake source is not null
+ if (lakeSource != null) {
+ // and exist field equals, push down to lake source
+ if (!fieldEquals.isEmpty()) {
+
+ // convert flink row type to fluss row type
+ RowType flussRowType = FlinkConversions.toFlussRowType(tableOutputType);
+ List<Predicate> lakePredicates = new ArrayList<>();
+
+ for (FieldEqual fieldEqual : fieldEquals) {
+ int idx = fieldEqual.fieldIndex;
+ String fieldName = tableOutputType.getFieldNames().get(idx);
+ com.alibaba.fluss.types.DataType flussDataType =
+ flussRowType.getTypeAt(idx);
+
+ // convert to Paimon literal for partition
+ Object literal =
+ toPaimonLiteralForPartition(flussDataType, fieldEqual.equalValue);
+
+ if (literal == null) {
+ continue;
+ }
+
+ lakePredicates.add(
+ new LeafPredicate(
+ Equal.INSTANCE,
+ flussDataType,
+ idx,
+ fieldName,
+ Collections.singletonList(literal)));
+ }
+
+ if (!lakePredicates.isEmpty()) {
+ final LakeSource.FilterPushDownResult filterPushDownResult =
+ lakeSource.withFilters(lakePredicates);
+ if (filterPushDownResult.acceptedPredicates().size()
+ != lakePredicates.size()) {
+ LOG.warn("Some partition filters are not accepted by the lake source");
+ }
+ }
+ } else {
+ // fill with empty list explicitly, when field equal is not exist
+ lakeSource.withFilters(Collections.emptyList());
Review Comment:
nit: don't need to call this method if no predicate
##########
fluss-flink/fluss-flink-common/src/test/java/com/alibaba/fluss/flink/source/FlinkTableSourceITCase.java:
##########
@@ -1066,6 +1066,52 @@ void testStreamingReadMultiPartitionPushDown() throws Exception {
assertResultsIgnoreOrder(rowIter, expectedRowValues, true);
}
+ @Test
+ void testStreamingReadMultiPartitionPushDownReverseOrderCondition() throws Exception {
Review Comment:
We actually want to verify when a table is lake enable, partition push down can still work, so you need to add test for lake enabled table.
I think you can add a case in test method `FlinkUnionReadLogTableITCase#testReadLogTableFullType`
Add a test case for `select * from xx where p = xx`, where p is the partitioned key
when `isPartitioned` is true,
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org