You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@seatunnel.apache.org by "xiaochen-zhou (via GitHub)" <gi...@apache.org> on 2025/08/01 12:05:42 UTC
[PR] [Improve][Connector-V2] Support like predicate pushdown in paimon [seatunnel]
xiaochen-zhou opened a new pull request, #9653:
URL: https://github.com/apache/seatunnel/pull/9653
### Purpose of this pull request
With the Paimon version upgraded to 1.1.1, we now support full "LIKE" pushdown.
### Does this PR introduce _any_ user-facing change?
no
### How was this patch tested?
Add e2e tests.
### Check list
* [ ] If any new Jar binary package adding in your PR, please add License Notice according
[New License Guide](https://github.com/apache/seatunnel/blob/dev/docs/en/contribution/new-license.md)
* [ ] If necessary, please update the documentation to describe the new feature. https://github.com/apache/seatunnel/tree/dev/docs
* [ ] If you are contributing the connector code, please check that the following files are updated:
1. Update [plugin-mapping.properties](https://github.com/apache/seatunnel/blob/dev/plugin-mapping.properties) and add new connector information in it
2. Update the pom file of [seatunnel-dist](https://github.com/apache/seatunnel/blob/dev/seatunnel-dist/pom.xml)
3. Add ci label in [label-scope-conf](https://github.com/apache/seatunnel/blob/dev/.github/workflows/labeler/label-scope-conf.yml)
4. Add e2e testcase in [seatunnel-e2e](https://github.com/apache/seatunnel/tree/dev/seatunnel-e2e/seatunnel-connector-v2-e2e/)
5. Update connector [plugin_config](https://github.com/apache/seatunnel/blob/dev/config/plugin_config)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Improve][Connector-V2] Support like predicate pushdown in paimon [seatunnel]
Posted by "Hisoka-X (via GitHub)" <gi...@apache.org>.
Hisoka-X commented on PR #9653:
URL: https://github.com/apache/seatunnel/pull/9653#issuecomment-3148983028
cc @hawk9821
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Improve][Connector-V2] Support like predicate pushdown in paimon [seatunnel]
Posted by "Hisoka-X (via GitHub)" <gi...@apache.org>.
Hisoka-X commented on code in PR #9653:
URL: https://github.com/apache/seatunnel/pull/9653#discussion_r2250306595
##########
seatunnel-connectors-v2/connector-paimon/src/main/java/org/apache/seatunnel/connectors/seatunnel/paimon/source/converter/SqlToPaimonPredicateConverter.java:
##########
@@ -253,14 +253,23 @@ private static Predicate parseExpressionToPredicate(
convertValueByPaimonDataType(rowType, column.getColumnName(), rightPredicate);
Pattern BEGIN_PATTERN = Pattern.compile("([^%]+)%");
- Matcher matcher = BEGIN_PATTERN.matcher(rightVal.toString());
- if (matcher.matches()) {
- return builder.startsWith(columnIndex, BinaryString.fromString(matcher.group(1)));
- } else {
- throw new IllegalArgumentException(
- "Unsupported expression type: "
- + expression.getClass().getSimpleName()
- + ", only support like pattern matching with prefix");
+ Matcher beginMatcher = BEGIN_PATTERN.matcher(rightVal.toString());
+ if (beginMatcher.matches()) {
+ return builder.startsWith(
+ columnIndex, BinaryString.fromString(beginMatcher.group(1)));
+ }
+
+ Pattern END_PATTERN = Pattern.compile("%([^%]+)");
+ Matcher endMatcher = END_PATTERN.matcher(rightVal.toString());
+ if (endMatcher.matches()) {
+ return builder.endsWith(columnIndex, BinaryString.fromString(endMatcher.group(1)));
+ }
+
+ Pattern CONTAINS_PATTERN = Pattern.compile("%([^%]+)%");
+ Matcher containsMatcher = CONTAINS_PATTERN.matcher(rightVal.toString());
+ if (containsMatcher.matches()) {
+ return builder.contains(
+ columnIndex, BinaryString.fromString(containsMatcher.group(1)));
}
Review Comment:
we should thrown exception when all patterns not matched.
##########
seatunnel-connectors-v2/connector-paimon/src/main/java/org/apache/seatunnel/connectors/seatunnel/paimon/source/converter/SqlToPaimonPredicateConverter.java:
##########
@@ -253,14 +253,23 @@ private static Predicate parseExpressionToPredicate(
convertValueByPaimonDataType(rowType, column.getColumnName(), rightPredicate);
Pattern BEGIN_PATTERN = Pattern.compile("([^%]+)%");
Review Comment:
we should make sure % is string end.
##########
seatunnel-connectors-v2/connector-paimon/src/main/java/org/apache/seatunnel/connectors/seatunnel/paimon/source/converter/SqlToPaimonPredicateConverter.java:
##########
@@ -253,14 +253,23 @@ private static Predicate parseExpressionToPredicate(
convertValueByPaimonDataType(rowType, column.getColumnName(), rightPredicate);
Pattern BEGIN_PATTERN = Pattern.compile("([^%]+)%");
- Matcher matcher = BEGIN_PATTERN.matcher(rightVal.toString());
- if (matcher.matches()) {
- return builder.startsWith(columnIndex, BinaryString.fromString(matcher.group(1)));
- } else {
- throw new IllegalArgumentException(
- "Unsupported expression type: "
- + expression.getClass().getSimpleName()
- + ", only support like pattern matching with prefix");
+ Matcher beginMatcher = BEGIN_PATTERN.matcher(rightVal.toString());
+ if (beginMatcher.matches()) {
+ return builder.startsWith(
+ columnIndex, BinaryString.fromString(beginMatcher.group(1)));
+ }
+
+ Pattern END_PATTERN = Pattern.compile("%([^%]+)");
Review Comment:
we should make sure the % is string start.
##########
seatunnel-connectors-v2/connector-paimon/src/main/java/org/apache/seatunnel/connectors/seatunnel/paimon/source/converter/SqlToPaimonPredicateConverter.java:
##########
@@ -253,14 +253,23 @@ private static Predicate parseExpressionToPredicate(
convertValueByPaimonDataType(rowType, column.getColumnName(), rightPredicate);
Pattern BEGIN_PATTERN = Pattern.compile("([^%]+)%");
- Matcher matcher = BEGIN_PATTERN.matcher(rightVal.toString());
- if (matcher.matches()) {
- return builder.startsWith(columnIndex, BinaryString.fromString(matcher.group(1)));
- } else {
- throw new IllegalArgumentException(
- "Unsupported expression type: "
- + expression.getClass().getSimpleName()
- + ", only support like pattern matching with prefix");
+ Matcher beginMatcher = BEGIN_PATTERN.matcher(rightVal.toString());
+ if (beginMatcher.matches()) {
+ return builder.startsWith(
+ columnIndex, BinaryString.fromString(beginMatcher.group(1)));
+ }
+
+ Pattern END_PATTERN = Pattern.compile("%([^%]+)");
+ Matcher endMatcher = END_PATTERN.matcher(rightVal.toString());
+ if (endMatcher.matches()) {
+ return builder.endsWith(columnIndex, BinaryString.fromString(endMatcher.group(1)));
+ }
+
+ Pattern CONTAINS_PATTERN = Pattern.compile("%([^%]+)%");
Review Comment:
ditto.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Improve][Connector-V2] Support like predicate pushdown in paimon [seatunnel]
Posted by "xiaochen-zhou (via GitHub)" <gi...@apache.org>.
xiaochen-zhou commented on PR #9653:
URL: https://github.com/apache/seatunnel/pull/9653#issuecomment-3157074464
> @xiaochen-zhou hi,How does he perform for versions lower than 1.1.1 ?
In version 0.7 and its subsequent versions, only the startWith LIKE pattern is supported. In version 1.1.1, the startWith, endWith, and contains LIKE patterns are supported.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Improve][Connector-V2] Support like predicate pushdown in paimon [seatunnel]
Posted by "Hisoka-X (via GitHub)" <gi...@apache.org>.
Hisoka-X merged PR #9653:
URL: https://github.com/apache/seatunnel/pull/9653
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Improve][Connector-V2] Support like predicate pushdown in paimon [seatunnel]
Posted by "Carl-Zhou-CN (via GitHub)" <gi...@apache.org>.
Carl-Zhou-CN commented on PR #9653:
URL: https://github.com/apache/seatunnel/pull/9653#issuecomment-3150362725
@xiaochen-zhou hi,How does he perform for versions lower than 1.1.1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Improve][Connector-V2] Support like predicate pushdown in paimon [seatunnel]
Posted by "hawk9821 (via GitHub)" <gi...@apache.org>.
hawk9821 commented on code in PR #9653:
URL: https://github.com/apache/seatunnel/pull/9653#discussion_r2256440199
##########
seatunnel-connectors-v2/connector-paimon/src/main/java/org/apache/seatunnel/connectors/seatunnel/paimon/source/converter/SqlToPaimonPredicateConverter.java:
##########
@@ -253,14 +253,23 @@ private static Predicate parseExpressionToPredicate(
convertValueByPaimonDataType(rowType, column.getColumnName(), rightPredicate);
Pattern BEGIN_PATTERN = Pattern.compile("([^%]+)%");
- Matcher matcher = BEGIN_PATTERN.matcher(rightVal.toString());
- if (matcher.matches()) {
- return builder.startsWith(columnIndex, BinaryString.fromString(matcher.group(1)));
- } else {
- throw new IllegalArgumentException(
- "Unsupported expression type: "
- + expression.getClass().getSimpleName()
- + ", only support like pattern matching with prefix");
+ Matcher beginMatcher = BEGIN_PATTERN.matcher(rightVal.toString());
+ if (beginMatcher.matches()) {
+ return builder.startsWith(
+ columnIndex, BinaryString.fromString(beginMatcher.group(1)));
+ }
+
+ Pattern END_PATTERN = Pattern.compile("%([^%]+)");
+ Matcher endMatcher = END_PATTERN.matcher(rightVal.toString());
+ if (endMatcher.matches()) {
+ return builder.endsWith(columnIndex, BinaryString.fromString(endMatcher.group(1)));
+ }
+
+ Pattern CONTAINS_PATTERN = Pattern.compile("%([^%]+)%");
+ Matcher containsMatcher = CONTAINS_PATTERN.matcher(rightVal.toString());
+ if (containsMatcher.matches()) {
+ return builder.contains(
+ columnIndex, BinaryString.fromString(containsMatcher.group(1)));
}
Review Comment:
when all of them don't match, an exception should be thrown. [SqlToPaimonPredicateConverter.java#L274](https://github.com/apache/seatunnel/blob/610e7da09ec0adef02b36e1ec6f019f8a1b9945f/seatunnel-connectors-v2/connector-paimon/src/main/java/org/apache/seatunnel/connectors/seatunnel/paimon/source/converter/SqlToPaimonPredicateConverter.java#L274)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@seatunnel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org