You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@drill.apache.org by GitBox <gi...@apache.org> on 2022/11/08 22:12:57 UTC
[GitHub] [drill] cgivre opened a new pull request, #2703: DRILL-8354: Add IS_EMPTY Function
cgivre opened a new pull request, #2703:
URL: https://github.com/apache/drill/pull/2703
# [DRILL-8354](https://issues.apache.org/jira/browse/DRILL-8354): Add IS_EMPTY Function
## Description
When analyzing data, there is currently no single function to evaluate whether a given field is empty. With scalar fields, this can be accomplished with the `IS NOT NULL` operator, but with complex fields, this is more challenging as complex fields are never null.
This PR adds a UDF called `IS_EMPTY()` which accepts any type of field and returns true if the field does not contain data.
In the case of scalar fields, if the field is `null` this returns true. In the case of complex fields, which can never be `null`, in the case of lists, the function returns true if the list is empty. In the case of maps, it returns true if all of the map's fields are unpopulated.
## Documentation
See above.
## Testing
Added unit tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@drill.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [drill] jnturton commented on a diff in pull request #2703: DRILL-8354: Add IS_EMPTY Function
Posted by GitBox <gi...@apache.org>.
jnturton commented on code in PR #2703:
URL: https://github.com/apache/drill/pull/2703#discussion_r1017701372
##########
exec/java-exec/src/test/resources/jsoninput/is_empty_tests.json:
##########
@@ -0,0 +1,27 @@
+[
+ {
+ "numeric_col": 1.3,
+ "text_col": "text",
+ "list_col": [
+ "v1", "v2"
+ ],
+ "map_column": {
+ "field1": "value1",
+ "field2": "value2",
+ "nested_map": {
+ "nested_field1": "nested_value1",
+ "nested_field2" : "nested_value2"
+ }
+ }
+ },{
+ "numeric_col": 2.3,
+ "text_col": "",
+ "list_col": [],
+ "map_column": {}
+ },{
+ "numeric_col": null,
+ "text_col": null,
+ "list_col": [],
+ "map_column": {}
+ }
Review Comment:
```suggestion
}, {
"numeric_col": 1
}
```
Can we add an object that is outright missing some queried properties and check that the is_empty unit tests pick those up as being empty?
##########
exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/impl/IsEmptyUtils.java:
##########
@@ -0,0 +1,61 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.drill.exec.expr.fn.impl;
+
+import org.apache.drill.common.types.TypeProtos.MinorType;
+import org.apache.drill.exec.vector.complex.impl.SingleMapReaderImpl;
+import org.apache.drill.exec.vector.complex.reader.FieldReader;
+
+public class IsEmptyUtils {
+
+ /**
+ * This function recursively traverses a Drill map to determine whether the map is empty or not.
+ * @param reader A {@link FieldReader} containing the field in question
+ * @return True if the field contains no data, false if it does.
+ */
+ public static boolean mapIsEmpty(FieldReader reader) {
+
+ if (reader.getType().getMinorType() == MinorType.MAP) {
+ SingleMapReaderImpl mapReader = (SingleMapReaderImpl) reader;
+
+ // If the map reader has no fields returns nothing return true
Review Comment:
```suggestion
// If the map reader has no fields return true
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@drill.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [drill] cgivre merged pull request #2703: DRILL-8354: Add IS_EMPTY Function
Posted by GitBox <gi...@apache.org>.
cgivre merged PR #2703:
URL: https://github.com/apache/drill/pull/2703
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@drill.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [drill] cgivre commented on a diff in pull request #2703: DRILL-8354: Add IS_EMPTY Function
Posted by GitBox <gi...@apache.org>.
cgivre commented on code in PR #2703:
URL: https://github.com/apache/drill/pull/2703#discussion_r1017888203
##########
exec/java-exec/src/main/java/org/apache/drill/exec/expr/fn/impl/IsEmptyUtils.java:
##########
@@ -0,0 +1,61 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.drill.exec.expr.fn.impl;
+
+import org.apache.drill.common.types.TypeProtos.MinorType;
+import org.apache.drill.exec.vector.complex.impl.SingleMapReaderImpl;
+import org.apache.drill.exec.vector.complex.reader.FieldReader;
+
+public class IsEmptyUtils {
+
+ /**
+ * This function recursively traverses a Drill map to determine whether the map is empty or not.
+ * @param reader A {@link FieldReader} containing the field in question
+ * @return True if the field contains no data, false if it does.
+ */
+ public static boolean mapIsEmpty(FieldReader reader) {
+
+ if (reader.getType().getMinorType() == MinorType.MAP) {
+ SingleMapReaderImpl mapReader = (SingleMapReaderImpl) reader;
+
+ // If the map reader has no fields returns nothing return true
Review Comment:
Fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@drill.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [drill] cgivre commented on a diff in pull request #2703: DRILL-8354: Add IS_EMPTY Function
Posted by GitBox <gi...@apache.org>.
cgivre commented on code in PR #2703:
URL: https://github.com/apache/drill/pull/2703#discussion_r1017891449
##########
exec/java-exec/src/test/resources/jsoninput/is_empty_tests.json:
##########
@@ -0,0 +1,27 @@
+[
+ {
+ "numeric_col": 1.3,
+ "text_col": "text",
+ "list_col": [
+ "v1", "v2"
+ ],
+ "map_column": {
+ "field1": "value1",
+ "field2": "value2",
+ "nested_map": {
+ "nested_field1": "nested_value1",
+ "nested_field2" : "nested_value2"
+ }
+ }
+ },{
+ "numeric_col": 2.3,
+ "text_col": "",
+ "list_col": [],
+ "map_column": {}
+ },{
+ "numeric_col": null,
+ "text_col": null,
+ "list_col": [],
+ "map_column": {}
+ }
Review Comment:
@jnturton Done! I addressed your review comments. Thanks for the quick review!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@drill.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org