You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "zyl891229 (via GitHub)" <gi...@apache.org> on 2024/04/23 03:06:35 UTC
[PR] support trino query paimon file index for map type with map-keys [paimon-trino]
zyl891229 opened a new pull request, #67:
URL: https://github.com/apache/paimon-trino/pull/67
support trino query paimon file index for map type with map-keys
enable map type index as follow
```
CREATE TABLE my_table (age int , jsonmap MAP<STRING,STRING> ) WITH
(
"file-index.bloom-filter.columns" = "jsonmap[a]",
)
```
query map type index as follow inpaimon trino must use fun element_at to extract value
`select * from paimon.default.my_table where age = 30 and element_at(jsonmap, 'a') = '1';`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] support trino query paimon file index for map type with map-keys [paimon-trino]
Posted by "leaves12138 (via GitHub)" <gi...@apache.org>.
leaves12138 merged PR #67:
URL: https://github.com/apache/paimon-trino/pull/67
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] support trino query paimon file index for map type with map-keys [paimon-trino]
Posted by "leaves12138 (via GitHub)" <gi...@apache.org>.
leaves12138 commented on code in PR #67:
URL: https://github.com/apache/paimon-trino/pull/67#discussion_r1578931950
##########
paimon-trino-common/src/main/java/org/apache/paimon/trino/TrinoMetadata.java:
##########
@@ -448,7 +461,37 @@ public Optional<ConstraintApplicationResult<ConnectorTableHandle>> applyFilter(
.intersect(TupleDomain.withColumnDomains(unenforcedDomains));
return Optional.of(
- new ConstraintApplicationResult<>(trinoTableHandle.copy(newFilter), remain, false));
+ new ConstraintApplicationResult<>(
+ trinoTableHandle.copy(newFilter, expressionFilter), remain, false));
+ }
+
+ private static TupleDomain<TrinoColumnHandle> getTrinoColumnHandleTupleDomain(
+ Constraint constraint) {
+ Map<TrinoColumnHandle, Domain> expressionPredicates = new HashMap<>();
Review Comment:
I think we could just declair expressionPredicate. In default: we put expressionPredicates=Collections.emptyMap()
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] support trino query paimon file index for map type with map-keys [paimon-trino]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #67:
URL: https://github.com/apache/paimon-trino/pull/67#discussion_r1579301961
##########
paimon-trino-common/src/main/java/org/apache/paimon/trino/TrinoMetadata.java:
##########
@@ -423,6 +424,10 @@ public Optional<ConstraintApplicationResult<ConnectorTableHandle>> applyFilter(
.getSummary()
Review Comment:
add comment here: what is returned from `getSummary`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] support trino query paimon file index for map type with map-keys [paimon-trino]
Posted by "leaves12138 (via GitHub)" <gi...@apache.org>.
leaves12138 commented on code in PR #67:
URL: https://github.com/apache/paimon-trino/pull/67#discussion_r1578943113
##########
paimon-trino-common/src/main/java/org/apache/paimon/trino/TrinoMetadata.java:
##########
@@ -516,4 +559,83 @@ public Optional<LimitApplicationResult<ConnectorTableHandle>> applyLimit(
return Optional.of(new LimitApplicationResult<>(table, false, false));
}
+
+ private static Map<TrinoColumnHandle, Domain> handleElementAtArguments(
Review Comment:
I think we can't put these expression parser code in a separated class, just left TrinoMetadata clean.
We can make a new class parse from expression to paimon Predicate
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] support trino query paimon file index for map type with map-keys [paimon-trino]
Posted by "leaves12138 (via GitHub)" <gi...@apache.org>.
leaves12138 commented on code in PR #67:
URL: https://github.com/apache/paimon-trino/pull/67#discussion_r1578933144
##########
paimon-trino-common/src/main/java/org/apache/paimon/trino/TrinoMetadata.java:
##########
@@ -448,7 +461,37 @@ public Optional<ConstraintApplicationResult<ConnectorTableHandle>> applyFilter(
.intersect(TupleDomain.withColumnDomains(unenforcedDomains));
return Optional.of(
- new ConstraintApplicationResult<>(trinoTableHandle.copy(newFilter), remain, false));
+ new ConstraintApplicationResult<>(
+ trinoTableHandle.copy(newFilter, expressionFilter), remain, false));
+ }
+
+ private static TupleDomain<TrinoColumnHandle> getTrinoColumnHandleTupleDomain(
+ Constraint constraint) {
+ Map<TrinoColumnHandle, Domain> expressionPredicates = new HashMap<>();
+
+ if (constraint.getExpression() instanceof Call) {
+ Call expression = (Call) constraint.getExpression();
+ Map<String, ColumnHandle> assignments = constraint.getAssignments();
+
+ switch (expression.getFunctionName().getName()) {
+ case "$equal":
Review Comment:
Can we put there in static final string?
private static final String TRINO_EQUAL = "$equal"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] support trino query paimon file index for map type with map-keys [paimon-trino]
Posted by "leaves12138 (via GitHub)" <gi...@apache.org>.
leaves12138 commented on code in PR #67:
URL: https://github.com/apache/paimon-trino/pull/67#discussion_r1578773543
##########
paimon-trino-common/src/main/java/org/apache/paimon/trino/TrinoFilterConverter.java:
##########
@@ -103,10 +107,20 @@ public Optional<Predicate> convert(
TrinoColumnHandle columnHandle = entry.getKey();
Domain domain = entry.getValue();
String field = columnHandle.getColumnName();
+ Optional<Integer> nestedColumn = getNestedColumn(field);
+ if (nestedColumn.isPresent()) {
+ int position = nestedColumn.get();
+ field = field.substring(0, position);
+ }
int index = fieldNames.indexOf(field);
if (index != -1) {
try {
- conjuncts.add(toPredicate(index, columnHandle.getTrinoType(), domain));
+ conjuncts.add(
+ toPredicate(
+ index,
+ columnHandle.getColumnName(),
+ columnHandle.getTrinoType(),
+ domain));
Review Comment:
We should pass the nested field to the predicate. For example, new LeafPredicate(columnName="map_data[key1]")
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] support trino query paimon file index for map type with map-keys [paimon-trino]
Posted by "leaves12138 (via GitHub)" <gi...@apache.org>.
leaves12138 commented on code in PR #67:
URL: https://github.com/apache/paimon-trino/pull/67#discussion_r1578773543
##########
paimon-trino-common/src/main/java/org/apache/paimon/trino/TrinoFilterConverter.java:
##########
@@ -103,10 +107,20 @@ public Optional<Predicate> convert(
TrinoColumnHandle columnHandle = entry.getKey();
Domain domain = entry.getValue();
String field = columnHandle.getColumnName();
+ Optional<Integer> nestedColumn = getNestedColumn(field);
+ if (nestedColumn.isPresent()) {
+ int position = nestedColumn.get();
+ field = field.substring(0, position);
+ }
int index = fieldNames.indexOf(field);
if (index != -1) {
try {
- conjuncts.add(toPredicate(index, columnHandle.getTrinoType(), domain));
+ conjuncts.add(
+ toPredicate(
+ index,
+ columnHandle.getColumnName(),
+ columnHandle.getTrinoType(),
+ domain));
Review Comment:
We should pass the nested field to the predicate. For example, new LeafPredicate(columnName="map_data[key1]")
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] support trino query paimon file index for map type with map-keys [paimon-trino]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #67:
URL: https://github.com/apache/paimon-trino/pull/67#discussion_r1579299238
##########
paimon-trino-common/src/main/java/org/apache/paimon/trino/TrinoExpressionFilterExtract.java:
##########
@@ -0,0 +1,144 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.paimon.trino;
+
+import org.apache.paimon.shade.guava30.com.google.common.collect.ImmutableList;
+import org.apache.paimon.shade.guava30.com.google.common.collect.Maps;
+
+import io.airlift.slice.Slice;
+import io.trino.spi.connector.ColumnHandle;
+import io.trino.spi.connector.Constraint;
+import io.trino.spi.expression.Call;
+import io.trino.spi.expression.Constant;
+import io.trino.spi.expression.Variable;
+import io.trino.spi.predicate.Domain;
+import io.trino.spi.predicate.Range;
+import io.trino.spi.predicate.SortedRangeSet;
+import io.trino.spi.predicate.TupleDomain;
+import io.trino.spi.type.ArrayType;
+import io.trino.spi.type.MapType;
+import io.trino.spi.type.Type;
+
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.stream.Collectors;
+
+import static io.trino.spi.expression.StandardFunctions.AND_FUNCTION_NAME;
+import static io.trino.spi.expression.StandardFunctions.EQUAL_OPERATOR_FUNCTION_NAME;
+import static io.trino.spi.expression.StandardFunctions.IN_PREDICATE_FUNCTION_NAME;
+
+/** Extract Trino Expression filter ( e.g. element_at(jsonmap, 'a') = '1' ) to TrinoColumnHandle. */
+public class TrinoExpressionFilterExtract {
Review Comment:
`TrinoMapKeyFilterExtract`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] support trino query paimon file index for map type with map-keys [paimon-trino]
Posted by "leaves12138 (via GitHub)" <gi...@apache.org>.
leaves12138 commented on code in PR #67:
URL: https://github.com/apache/paimon-trino/pull/67#discussion_r1578931950
##########
paimon-trino-common/src/main/java/org/apache/paimon/trino/TrinoMetadata.java:
##########
@@ -448,7 +461,37 @@ public Optional<ConstraintApplicationResult<ConnectorTableHandle>> applyFilter(
.intersect(TupleDomain.withColumnDomains(unenforcedDomains));
return Optional.of(
- new ConstraintApplicationResult<>(trinoTableHandle.copy(newFilter), remain, false));
+ new ConstraintApplicationResult<>(
+ trinoTableHandle.copy(newFilter, expressionFilter), remain, false));
+ }
+
+ private static TupleDomain<TrinoColumnHandle> getTrinoColumnHandleTupleDomain(
+ Constraint constraint) {
+ Map<TrinoColumnHandle, Domain> expressionPredicates = new HashMap<>();
Review Comment:
I think we could just declare expressionPredicate. In default: we put expressionPredicates=Collections.emptyMap()
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] support trino query paimon file index for map type with map-keys [paimon-trino]
Posted by "leaves12138 (via GitHub)" <gi...@apache.org>.
leaves12138 commented on code in PR #67:
URL: https://github.com/apache/paimon-trino/pull/67#discussion_r1578945371
##########
paimon-trino-common/src/main/java/org/apache/paimon/trino/TrinoMetadata.java:
##########
@@ -516,4 +559,83 @@ public Optional<LimitApplicationResult<ConnectorTableHandle>> applyLimit(
return Optional.of(new LimitApplicationResult<>(table, false, false));
}
+
+ private static Map<TrinoColumnHandle, Domain> handleElementAtArguments(
Review Comment:
And we can add some UT test for this new class if possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] support trino query paimon file index for map type with map-keys [paimon-trino]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #67:
URL: https://github.com/apache/paimon-trino/pull/67#discussion_r1579301961
##########
paimon-trino-common/src/main/java/org/apache/paimon/trino/TrinoMetadata.java:
##########
@@ -423,6 +424,10 @@ public Optional<ConstraintApplicationResult<ConnectorTableHandle>> applyFilter(
.getSummary()
Review Comment:
add comment here: what is returned from `getSummary`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] support trino query paimon file index for map type with map-keys [paimon-trino]
Posted by "leaves12138 (via GitHub)" <gi...@apache.org>.
leaves12138 commented on code in PR #67:
URL: https://github.com/apache/paimon-trino/pull/67#discussion_r1578764292
##########
pom.xml:
##########
@@ -348,6 +348,19 @@ under the License.
</execution>
</executions>
</plugin>
+ <plugin>
+ <groupId>org.apache.maven.plugins</groupId>
+ <artifactId>maven-source-plugin</artifactId>
+ <version>3.2.1</version>
+ <executions>
+ <execution>
+ <id>attach-sources</id>
+ <goals>
+ <goal>jar-no-fork</goal>
Review Comment:
Can you explain why use this goal "jar-no-fork"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] support trino query paimon file index for map type with map-keys [paimon-trino]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #67:
URL: https://github.com/apache/paimon-trino/pull/67#discussion_r1579308475
##########
paimon-trino-common/src/main/java/org/apache/paimon/trino/TrinoExpressionFilterExtract.java:
##########
@@ -0,0 +1,144 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.paimon.trino;
+
+import org.apache.paimon.shade.guava30.com.google.common.collect.ImmutableList;
+import org.apache.paimon.shade.guava30.com.google.common.collect.Maps;
+
+import io.airlift.slice.Slice;
+import io.trino.spi.connector.ColumnHandle;
+import io.trino.spi.connector.Constraint;
+import io.trino.spi.expression.Call;
+import io.trino.spi.expression.Constant;
+import io.trino.spi.expression.Variable;
+import io.trino.spi.predicate.Domain;
+import io.trino.spi.predicate.Range;
+import io.trino.spi.predicate.SortedRangeSet;
+import io.trino.spi.predicate.TupleDomain;
+import io.trino.spi.type.ArrayType;
+import io.trino.spi.type.MapType;
+import io.trino.spi.type.Type;
+
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.stream.Collectors;
+
+import static io.trino.spi.expression.StandardFunctions.AND_FUNCTION_NAME;
+import static io.trino.spi.expression.StandardFunctions.EQUAL_OPERATOR_FUNCTION_NAME;
+import static io.trino.spi.expression.StandardFunctions.IN_PREDICATE_FUNCTION_NAME;
+
+/** Extract Trino Expression filter ( e.g. element_at(jsonmap, 'a') = '1' ) to TrinoColumnHandle. */
+public class TrinoExpressionFilterExtract {
+ public static final String TRINO_MAP_ELEMENT_AT_FUNCTION_NAME = "element_at";
+
+ public static TupleDomain<TrinoColumnHandle> getTrinoColumnHandleForExpressionFilter(
+ Constraint constraint) {
+ Map<TrinoColumnHandle, Domain> expressionPredicates = Collections.emptyMap();
+
+ if (constraint.getExpression() instanceof Call) {
+ Call expression = (Call) constraint.getExpression();
+ Map<String, ColumnHandle> assignments = constraint.getAssignments();
+
+ if (expression.getFunctionName().equals(EQUAL_OPERATOR_FUNCTION_NAME)) {
+ expressionPredicates = handleElementAtArguments(assignments, expression, false);
+ } else if (expression.getFunctionName().equals(IN_PREDICATE_FUNCTION_NAME)) {
+ expressionPredicates = handleElementAtArguments(assignments, expression, true);
+ } else if (expression.getFunctionName().equals(AND_FUNCTION_NAME)) {
+ expressionPredicates = handleAndArguments(assignments, expression);
+ }
+ }
+ return TupleDomain.withColumnDomains(expressionPredicates);
+ }
+
+ private static Map<TrinoColumnHandle, Domain> handleElementAtArguments(
+ Map<String, ColumnHandle> assignments, Call expression, boolean isIn) {
+ Map<TrinoColumnHandle, Domain> expressionPredicates = Maps.newHashMap();
+
+ Call elementAtExpression = (Call) expression.getArguments().get(0);
+
+ if (!elementAtExpression
+ .getFunctionName()
+ .getName()
+ .equals(TRINO_MAP_ELEMENT_AT_FUNCTION_NAME)) {
+ return expressionPredicates;
+ }
+
+ Variable columnExpression = (Variable) elementAtExpression.getArguments().get(0);
+ Constant columnKey = (Constant) elementAtExpression.getArguments().get(1);
+
+ TrinoColumnHandle trinoColumnHandle =
+ (TrinoColumnHandle) assignments.get(columnExpression.getName());
+ Type trinoType = trinoColumnHandle.getTrinoType();
+ if (trinoType instanceof MapType) {
+ String columnName = trinoColumnHandle.getColumnName();
+ String key = ((Slice) columnKey.getValue()).toStringUtf8();
+ Constant elementAtValue = (Constant) expression.getArguments().get(1);
+ expressionPredicates.put(
+ TrinoColumnHandle.of(
+ toMapKey(columnName, key), TrinoTypeUtils.toPaimonType(trinoType)),
+ Domain.create(
+ SortedRangeSet.copyOf(
+ isIn
+ ? ((ArrayType) elementAtValue.getType())
+ .getElementType()
+ : elementAtValue.getType(),
+ isIn
+ ? elementAtValue.getChildren().stream()
+ .map(
+ arguemnt ->
+ Range.equal(
+ arguemnt.getType(),
+ ((Constant) arguemnt)
+ .getValue()))
+ .collect(Collectors.toList())
+ : ImmutableList.of(
+ Range.equal(
+ elementAtValue.getType(),
+ elementAtValue.getValue()))),
+ false));
+ }
+ return expressionPredicates;
+ }
+
+ public static String toMapKey(String mapColumnName, String keyName) {
+ return mapColumnName + "[" + keyName + "]";
Review Comment:
add comment: Follow Paimon API definition.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] support trino query paimon file index for map type with map-keys [paimon-trino]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #67:
URL: https://github.com/apache/paimon-trino/pull/67#discussion_r1579307295
##########
paimon-trino-common/src/main/java/org/apache/paimon/trino/TrinoMetadata.java:
##########
@@ -423,6 +424,10 @@ public Optional<ConstraintApplicationResult<ConnectorTableHandle>> applyFilter(
.getSummary()
.transformKeys(TrinoColumnHandle.class::cast)
.intersect(oldFilter);
+
+ TupleDomain<TrinoColumnHandle> expressionFilter =
+ getTrinoColumnHandleForExpressionFilter(constraint);
Review Comment:
Can we just merge `filter` and `expressionFilter`?
We can introduce a `TrinoFilterExtractor.extract(constraint)` static method. It just return one filter (Which is one `TupleDomain<TrinoColumnHandle>`).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] support trino query paimon file index for map type with map-keys [paimon-trino]
Posted by "leaves12138 (via GitHub)" <gi...@apache.org>.
leaves12138 commented on PR #67:
URL: https://github.com/apache/paimon-trino/pull/67#issuecomment-2095301722
+1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] support trino query paimon file index for map type with map-keys [paimon-trino]
Posted by "leaves12138 (via GitHub)" <gi...@apache.org>.
leaves12138 commented on code in PR #67:
URL: https://github.com/apache/paimon-trino/pull/67#discussion_r1578764763
##########
paimon-trino-common/src/main/java/org/apache/paimon/trino/TrinoFilterConverter.java:
##########
@@ -69,6 +72,7 @@
public class TrinoFilterConverter {
private static final Logger LOG = LoggerFactory.getLogger(TrinoFilterConverter.class);
+ public static final Pattern IS_NESTED = Pattern.compile(".+\\[.+]");
Review Comment:
Please refer to https://github.com/apache/paimon/pull/3221, it does not use regex anymore
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] support trino query paimon file index for map type with map-keys [paimon-trino]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #67:
URL: https://github.com/apache/paimon-trino/pull/67#discussion_r1579299238
##########
paimon-trino-common/src/main/java/org/apache/paimon/trino/TrinoExpressionFilterExtract.java:
##########
@@ -0,0 +1,144 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.paimon.trino;
+
+import org.apache.paimon.shade.guava30.com.google.common.collect.ImmutableList;
+import org.apache.paimon.shade.guava30.com.google.common.collect.Maps;
+
+import io.airlift.slice.Slice;
+import io.trino.spi.connector.ColumnHandle;
+import io.trino.spi.connector.Constraint;
+import io.trino.spi.expression.Call;
+import io.trino.spi.expression.Constant;
+import io.trino.spi.expression.Variable;
+import io.trino.spi.predicate.Domain;
+import io.trino.spi.predicate.Range;
+import io.trino.spi.predicate.SortedRangeSet;
+import io.trino.spi.predicate.TupleDomain;
+import io.trino.spi.type.ArrayType;
+import io.trino.spi.type.MapType;
+import io.trino.spi.type.Type;
+
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.stream.Collectors;
+
+import static io.trino.spi.expression.StandardFunctions.AND_FUNCTION_NAME;
+import static io.trino.spi.expression.StandardFunctions.EQUAL_OPERATOR_FUNCTION_NAME;
+import static io.trino.spi.expression.StandardFunctions.IN_PREDICATE_FUNCTION_NAME;
+
+/** Extract Trino Expression filter ( e.g. element_at(jsonmap, 'a') = '1' ) to TrinoColumnHandle. */
+public class TrinoExpressionFilterExtract {
Review Comment:
`TrinoMapKeyFilterExtract`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] support trino query paimon file index for map type with map-keys [paimon-trino]
Posted by "leaves12138 (via GitHub)" <gi...@apache.org>.
leaves12138 commented on code in PR #67:
URL: https://github.com/apache/paimon-trino/pull/67#discussion_r1578933997
##########
paimon-trino-common/src/main/java/org/apache/paimon/trino/TrinoMetadata.java:
##########
@@ -516,4 +559,83 @@ public Optional<LimitApplicationResult<ConnectorTableHandle>> applyLimit(
return Optional.of(new LimitApplicationResult<>(table, false, false));
}
+
+ private static Map<TrinoColumnHandle, Domain> handleElementAtArguments(
+ Map<String, ColumnHandle> assignments, Call expression, boolean isIn) {
+ Map<TrinoColumnHandle, Domain> expressionPredicates = new HashMap<>();
+
+ Call elementAtExpression = (Call) expression.getArguments().get(0);
+
+ if (!elementAtExpression.getFunctionName().getName().equals("element_at")) {
Review Comment:
ditto, element_at is a special function name, put it in static final String
##########
paimon-trino-common/src/main/java/org/apache/paimon/trino/TrinoMetadata.java:
##########
@@ -448,7 +461,37 @@ public Optional<ConstraintApplicationResult<ConnectorTableHandle>> applyFilter(
.intersect(TupleDomain.withColumnDomains(unenforcedDomains));
return Optional.of(
- new ConstraintApplicationResult<>(trinoTableHandle.copy(newFilter), remain, false));
+ new ConstraintApplicationResult<>(
+ trinoTableHandle.copy(newFilter, expressionFilter), remain, false));
+ }
+
+ private static TupleDomain<TrinoColumnHandle> getTrinoColumnHandleTupleDomain(
+ Constraint constraint) {
+ Map<TrinoColumnHandle, Domain> expressionPredicates = new HashMap<>();
+
+ if (constraint.getExpression() instanceof Call) {
+ Call expression = (Call) constraint.getExpression();
+ Map<String, ColumnHandle> assignments = constraint.getAssignments();
+
+ switch (expression.getFunctionName().getName()) {
+ case "$equal":
+ expressionPredicates = handleElementAtArguments(assignments, expression, false);
+ break;
+
+ case "$in":
Review Comment:
ditto
##########
paimon-trino-common/src/main/java/org/apache/paimon/trino/TrinoMetadata.java:
##########
@@ -448,7 +461,37 @@ public Optional<ConstraintApplicationResult<ConnectorTableHandle>> applyFilter(
.intersect(TupleDomain.withColumnDomains(unenforcedDomains));
return Optional.of(
- new ConstraintApplicationResult<>(trinoTableHandle.copy(newFilter), remain, false));
+ new ConstraintApplicationResult<>(
+ trinoTableHandle.copy(newFilter, expressionFilter), remain, false));
+ }
+
+ private static TupleDomain<TrinoColumnHandle> getTrinoColumnHandleTupleDomain(
+ Constraint constraint) {
+ Map<TrinoColumnHandle, Domain> expressionPredicates = new HashMap<>();
+
+ if (constraint.getExpression() instanceof Call) {
+ Call expression = (Call) constraint.getExpression();
+ Map<String, ColumnHandle> assignments = constraint.getAssignments();
+
+ switch (expression.getFunctionName().getName()) {
+ case "$equal":
+ expressionPredicates = handleElementAtArguments(assignments, expression, false);
+ break;
+
+ case "$in":
+ expressionPredicates = handleElementAtArguments(assignments, expression, true);
+ break;
+
+ case "$and":
Review Comment:
ditto
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org