You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@fluss.apache.org by "platinumhamburg (via GitHub)" <gi...@apache.org> on 2025/08/28 11:33:17 UTC

Re: [PR] [flink]Support partition pushdown in Flink connector [fluss]

platinumhamburg commented on code in PR #420:
URL: https://github.com/apache/fluss/pull/420#discussion_r2307127804


##########
fluss-flink/fluss-flink-common/src/main/java/com/alibaba/fluss/flink/source/enumerator/FlinkSourceEnumerator.java:
##########
@@ -275,30 +303,34 @@ private Set<PartitionInfo> listPartitions() {
 
     /** Apply partition filter. */
     private List<PartitionInfo> applyPartitionFilter(List<PartitionInfo> partitionInfos) {
-        if (!partitionFilters.isEmpty()) {
-            return partitionInfos.stream()
-                    .filter(
-                            partitionInfo -> {
-                                Map<String, String> specMap =
-                                        partitionInfo.getPartitionSpec().getSpecMap();
-                                // use getFields() instead of getFieldNames() to
-                                // avoid collection construction
-                                List<DataField> fields = tableInfo.getRowType().getFields();
-                                for (FieldEqual filter : partitionFilters) {
-                                    String fieldName = fields.get(filter.fieldIndex).getName();
-                                    String partitionValue = specMap.get(fieldName);
-                                    if (partitionValue == null
-                                            || !filter.equalValue
-                                                    .toString()
-                                                    .equals(partitionValue)) {
-                                        return false;
-                                    }
-                                }
-                                return true;
-                            })
-                    .collect(Collectors.toList());
-        }
-        return partitionInfos;
+        if (predicate == null) {
+            return partitionInfos;
+        } else {
+            List<PartitionInfo> filteredPartitionInfos =
+                    partitionInfos.stream()
+                            .filter(
+                                    partitionInfo ->
+                                            predicate.test(
+                                                    convertPartitionInfoToInternalRow(
+                                                            partitionInfo)))
+                            .collect(Collectors.toList());
+            LOG.info(
+                    "Filtered partitions {} for table {} with predicate: {}",
+                    filteredPartitionInfos,
+                    tablePath,
+                    predicate);
+            return filteredPartitionInfos;
+        }
+    }
+
+    private InternalRow convertPartitionInfoToInternalRow(PartitionInfo partitionInfo) {
+        List<String> partitionValues =
+                partitionInfo.getResolvedPartitionSpec().getPartitionValues();
+        GenericRow genericRow = new GenericRow(partitionValues.size());
+        for (int i = 0; i < partitionValues.size(); i++) {
+            genericRow.setField(i, BinaryString.fromString(partitionValues.get(i)));

Review Comment:
   I found that in the latest code, under certain circumstances, the Equal function would trigger a comparison between BinaryString and String, and CompareUtils would throw a type conversion exception。



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@fluss.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org