You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2020/11/09 18:53:09 UTC
[GitHub] [iceberg] RussellSpitzer opened a new pull request #1744: Fix Avro Pruning Bugs with ManifestEntries Table
RussellSpitzer opened a new pull request #1744:
URL: https://github.com/apache/iceberg/pull/1744
Previously we would crash whenever attempting to project only non
data_file columns (or data_file sub columns) from a partitioned
Iceberg table. This would occur because our projection in Manifest Reader
would always require the "data_file" column even if no columns from within
in it were required. This worked on unpartitioned tables because of a second
bug in the column pruning which would not prune a field which contained another field
whose schema was an empty struct regardless of whether it was requested or not. An
empty partition schema would make sure that the data_file would not actually
be pruned even though no columns would actually match the pruning request.
Here we fix both bugs, first by correctly pruning empty structs if they are not
explicitly requested.
Second by chaning the projection for ManifestEntry to ignore the data_file field
if no subfields are requested from it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] RussellSpitzer commented on pull request #1744: Fix Avro Pruning Bugs with ManifestEntries Table
Posted by GitBox <gi...@apache.org>.
RussellSpitzer commented on pull request #1744:
URL: https://github.com/apache/iceberg/pull/1744#issuecomment-810390392
@szehon-ho Let's work together to figure out a solution to this :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on pull request #1744: Fix Avro Pruning Bugs with ManifestEntries Table
Posted by GitBox <gi...@apache.org>.
rdblue commented on pull request #1744:
URL: https://github.com/apache/iceberg/pull/1744#issuecomment-724394053
For reference later, the problem here is that Avro's `BuildAvroProjection` can fail if the expected schema and Avro schema don't match. The problem in `GetProjectedIds` caused an empty `data_file` field to not get projected in the result of `PruneColumns` because no ID was returned to select it. The number of fields in the final Avro projection must match so that all column indexes align between the expected Iceberg schema and the schema passed to the Avro resolver.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] RussellSpitzer commented on a change in pull request #1744: Fix Avro Pruning Bugs with ManifestEntries Table
Posted by GitBox <gi...@apache.org>.
RussellSpitzer commented on a change in pull request #1744:
URL: https://github.com/apache/iceberg/pull/1744#discussion_r521782173
##########
File path: api/src/main/java/org/apache/iceberg/types/GetEmptyStructIds.java
##########
@@ -0,0 +1,76 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.types;
+
+import java.util.List;
+import java.util.Set;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.relocated.com.google.common.collect.Sets;
+
+class GetEmptyStructIds extends TypeUtil.SchemaVisitor<Set<Integer>> {
+ private final Set<Integer> fieldIds = Sets.newHashSet();
+
+ @Override
+ public Set<Integer> schema(Schema schema, Set<Integer> structResult) {
+ return fieldIds;
+ }
+
+ @Override
+ public Set<Integer> struct(Types.StructType struct, List<Set<Integer>> fieldResults) {
+ return fieldIds;
+ }
+
+ private boolean isEmptyStruct(Types.NestedField field) {
+ return field.type().isStructType() && ((Types.StructType) field.type()).fields().isEmpty();
+ }
+
+ /**
+ * We don't know at this point whether the fields contained within a struct are Metadata (and don't exist in the file)
+ * or have been projected in as optional and don't exist in the file. Here we check if all the fields are a
+ * combination of those two states.
+ * @param field a nested field which may be an empty struct
+ * @return true if we need to preserve this struct as it may be needed for its child fields
+ */
+ private boolean isPotentiallyEmpty(Types.NestedField field) {
+ if (!field.type().isStructType()) {
+ return false;
+ }
+ List<Types.NestedField> fields = ((Types.StructType) field.type()).fields();
+ return fields.stream().allMatch(f -> f.fieldId() > Integer.MAX_VALUE - 201 || f.isOptional());
+ }
+
+ @Override
+ public Set<Integer> field(Types.NestedField field, Set<Integer> fieldResult) {
+ if (isEmptyStruct(field) || isPotentiallyEmpty(field)) {
Review comment:
I think that's another alternative. We could pass through the original avro schema to build projection and just fall back to copy field and remove subfields if a record is missing but required.
In my mind "pruning" means removing everything that is unnecessary, but these fields are necessary so fixing pruning seemed more natural to me than building it back in in buildAvroProjection. Although we already do add more fields in the projection code for fields that don't exist at all.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] RussellSpitzer commented on a change in pull request #1744: Fix Avro Pruning Bugs with ManifestEntries Table
Posted by GitBox <gi...@apache.org>.
RussellSpitzer commented on a change in pull request #1744:
URL: https://github.com/apache/iceberg/pull/1744#discussion_r520047139
##########
File path: core/src/main/java/org/apache/iceberg/ManifestEntry.java
##########
@@ -54,7 +54,21 @@ static Schema getSchema(StructType partitionType) {
}
static Schema wrapFileSchema(StructType fileType) {
- return new Schema(STATUS, SNAPSHOT_ID, SEQUENCE_NUMBER, required(DATA_FILE_ID, "data_file", fileType));
+ /*
+ * Used when determining the projection for ManifestReader, because the projection for data_file can be empty
+ * we need to explicitly remove the column if it is empty. If we leave it in the schema we end up pruning it out
+ * (because no columns within it are required) but still require it be present in the projection. This causes
+ * an exception to be raised.
+ *
+ * To fix this we skip requesting the DATA_FILE column when nothing is required from it.
+ *
+ * TODO Fix the projection in ManifestReader itself to not just work with data_file columns
+ */
+ if (!fileType.fields().isEmpty()) {
+ return new Schema(STATUS, SNAPSHOT_ID, SEQUENCE_NUMBER, required(DATA_FILE_ID, "data_file", fileType));
Review comment:
This is the first bug. This schema is used in the Manifest Reader .project call, when the data_file file type requested is empty then the column is ignored. But since it is there in the schema it is still required. This didn't break unpartitioned tables because of the second bug below.
As mentioned in the comment, it would be better if we got the projection schema from a combination of only the columns required, rather than always assuming all 4 structs are required. So this is still not an ideal solution but doing more would require a more significant rewrite of ManifestReader
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] RussellSpitzer commented on a change in pull request #1744: Fix Avro Pruning Bugs with ManifestEntries Table
Posted by GitBox <gi...@apache.org>.
RussellSpitzer commented on a change in pull request #1744:
URL: https://github.com/apache/iceberg/pull/1744#discussion_r520043874
##########
File path: core/src/main/java/org/apache/iceberg/avro/PruneColumns.java
##########
@@ -95,7 +95,7 @@ public Schema record(Schema record, List<String> names, List<Schema> fields) {
if (hasChange) {
return copyRecord(record, filteredFields);
- } else if (filteredFields.size() == record.getFields().size()) {
+ } else if (record.getFields().size() != 0 && filteredFields.size() == record.getFields().size()) {
Review comment:
This is what keeps "data_file" from being pruned on Unpartitioned tables (since partitionType would be an empty struct)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #1744: Fix Avro Pruning Bugs with ManifestEntries Table
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #1744:
URL: https://github.com/apache/iceberg/pull/1744#discussion_r521769125
##########
File path: core/src/main/java/org/apache/iceberg/avro/PruneColumns.java
##########
@@ -90,12 +92,18 @@ public Schema record(Schema record, List<String> names, List<Schema> fields) {
} else if (fieldSchema != null) {
hasChange = true;
filteredFields.add(copyField(field, fieldSchema, fieldId));
+ } else if (emptyStructIds.contains(fieldId)) {
+ // This field does not require any known sub-fields but is required in the projection so keep it without
+ // any of it's subfields
+ hasChange = true;
+ Schema empty = AvroSchemaUtil.removeFields(field);
Review comment:
Why use `removeFields` when this class already has `copyRecord`? Couldn't this just call `copyRecord(field.schema(), ImmutableList.of())`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #1744: Fix Avro Pruning Bugs with ManifestEntries Table
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #1744:
URL: https://github.com/apache/iceberg/pull/1744#discussion_r521770183
##########
File path: api/src/main/java/org/apache/iceberg/types/GetEmptyStructIds.java
##########
@@ -0,0 +1,76 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.types;
+
+import java.util.List;
+import java.util.Set;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.relocated.com.google.common.collect.Sets;
+
+class GetEmptyStructIds extends TypeUtil.SchemaVisitor<Set<Integer>> {
+ private final Set<Integer> fieldIds = Sets.newHashSet();
+
+ @Override
+ public Set<Integer> schema(Schema schema, Set<Integer> structResult) {
+ return fieldIds;
+ }
+
+ @Override
+ public Set<Integer> struct(Types.StructType struct, List<Set<Integer>> fieldResults) {
+ return fieldIds;
+ }
+
+ private boolean isEmptyStruct(Types.NestedField field) {
+ return field.type().isStructType() && ((Types.StructType) field.type()).fields().isEmpty();
+ }
+
+ /**
+ * We don't know at this point whether the fields contained within a struct are Metadata (and don't exist in the file)
+ * or have been projected in as optional and don't exist in the file. Here we check if all the fields are a
+ * combination of those two states.
+ * @param field a nested field which may be an empty struct
+ * @return true if we need to preserve this struct as it may be needed for its child fields
+ */
+ private boolean isPotentiallyEmpty(Types.NestedField field) {
+ if (!field.type().isStructType()) {
+ return false;
+ }
+ List<Types.NestedField> fields = ((Types.StructType) field.type()).fields();
+ return fields.stream().allMatch(f -> f.fieldId() > Integer.MAX_VALUE - 201 || f.isOptional());
+ }
+
+ @Override
+ public Set<Integer> field(Types.NestedField field, Set<Integer> fieldResult) {
+ if (isEmptyStruct(field) || isPotentiallyEmpty(field)) {
Review comment:
The potentially empty handling here seems strange to me. I think it would make more sense for the field to be added or fixed up in `BuildAvroProjection` instead of catching the case so that an empty struct is retained by `PruneColumns`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] RussellSpitzer commented on pull request #1744: Fix Avro Pruning Bugs with ManifestEntries Table
Posted by GitBox <gi...@apache.org>.
RussellSpitzer commented on pull request #1744:
URL: https://github.com/apache/iceberg/pull/1744#issuecomment-886834800
@rdblue + @szehon-ho - @karuppayya reminded me to fix this with our hope that it would fix #2783, While I did fix this PR it doesn't seem to have fixed the other issue. That said, I at least fixed these two issues :) I would appreciate a review so we can at least fix this portion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] RussellSpitzer commented on a change in pull request #1744: Fix Avro Pruning Bugs with ManifestEntries Table
Posted by GitBox <gi...@apache.org>.
RussellSpitzer commented on a change in pull request #1744:
URL: https://github.com/apache/iceberg/pull/1744#discussion_r521782173
##########
File path: api/src/main/java/org/apache/iceberg/types/GetEmptyStructIds.java
##########
@@ -0,0 +1,76 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.types;
+
+import java.util.List;
+import java.util.Set;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.relocated.com.google.common.collect.Sets;
+
+class GetEmptyStructIds extends TypeUtil.SchemaVisitor<Set<Integer>> {
+ private final Set<Integer> fieldIds = Sets.newHashSet();
+
+ @Override
+ public Set<Integer> schema(Schema schema, Set<Integer> structResult) {
+ return fieldIds;
+ }
+
+ @Override
+ public Set<Integer> struct(Types.StructType struct, List<Set<Integer>> fieldResults) {
+ return fieldIds;
+ }
+
+ private boolean isEmptyStruct(Types.NestedField field) {
+ return field.type().isStructType() && ((Types.StructType) field.type()).fields().isEmpty();
+ }
+
+ /**
+ * We don't know at this point whether the fields contained within a struct are Metadata (and don't exist in the file)
+ * or have been projected in as optional and don't exist in the file. Here we check if all the fields are a
+ * combination of those two states.
+ * @param field a nested field which may be an empty struct
+ * @return true if we need to preserve this struct as it may be needed for its child fields
+ */
+ private boolean isPotentiallyEmpty(Types.NestedField field) {
+ if (!field.type().isStructType()) {
+ return false;
+ }
+ List<Types.NestedField> fields = ((Types.StructType) field.type()).fields();
+ return fields.stream().allMatch(f -> f.fieldId() > Integer.MAX_VALUE - 201 || f.isOptional());
+ }
+
+ @Override
+ public Set<Integer> field(Types.NestedField field, Set<Integer> fieldResult) {
+ if (isEmptyStruct(field) || isPotentiallyEmpty(field)) {
Review comment:
I think that's another alternative, we could when doing build avro projection by also passing back through the original AvroSchema and doing the remove fields and copy, when missing.
In my mind "pruning" means removing everything that is unnecessary, but these fields are necessary so fixing pruning seemed more natural to me than building it back in in buildAvroProjection. Although we already do add more fields in the projection code for fields that don't exist at all.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #1744: Fix Avro Pruning Bugs with ManifestEntries Table
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #1744:
URL: https://github.com/apache/iceberg/pull/1744#discussion_r520229507
##########
File path: core/src/main/java/org/apache/iceberg/avro/PruneColumns.java
##########
@@ -95,7 +95,7 @@ public Schema record(Schema record, List<String> names, List<Schema> fields) {
if (hasChange) {
return copyRecord(record, filteredFields);
- } else if (filteredFields.size() == record.getFields().size()) {
+ } else if (record.getFields().size() != 0 && filteredFields.size() == record.getFields().size()) {
Review comment:
I think we need a different operation that is the opposite of `GetProjectedIds`, like `ProjectFromIds`.
`TypeUtil.select` uses this class, `PruneColumns`, but it has behavior like a SQL `SELECT`. If I have a schema `a int, b struct<x double, y double>, c string` and I select `b`, then everything underneath `b` is selected, which is what you'd expect from `SELECT b FROM table`.
If we were to update `GetProjectedIds` with the logic above, then projecting `b struct<>` (which you can't do by naming columns) would actually result in the full struct getting projected because of the logic here that selects all of `b`. This class cannot be used to reconstruct a schema using the result of `GetProjectedIds`.
I think that we also need a `BuildProjection` that does the opposite of `GetProjectedIds` with the update to add empty structs. Then the datum reader could use that logic to prune the Avro schema and get an exact match with the expected schema.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] RussellSpitzer commented on a change in pull request #1744: Fix Avro Pruning Bugs with ManifestEntries Table
Posted by GitBox <gi...@apache.org>.
RussellSpitzer commented on a change in pull request #1744:
URL: https://github.com/apache/iceberg/pull/1744#discussion_r521781466
##########
File path: core/src/main/java/org/apache/iceberg/avro/AvroSchemaUtil.java
##########
@@ -404,4 +405,21 @@ private static String sanitize(char character) {
}
return "_x" + Integer.toHexString(character).toUpperCase();
}
+
+ public static Schema removeFields(Schema.Field field) {
Review comment:
Yep sure, it's not very common
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] RussellSpitzer closed pull request #1744: Fix Avro Pruning Bugs with ManifestEntries Table
Posted by GitBox <gi...@apache.org>.
RussellSpitzer closed pull request #1744:
URL: https://github.com/apache/iceberg/pull/1744
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #1744: Fix Avro Pruning Bugs with ManifestEntries Table
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #1744:
URL: https://github.com/apache/iceberg/pull/1744#discussion_r521770737
##########
File path: core/src/main/java/org/apache/iceberg/avro/AvroSchemaUtil.java
##########
@@ -404,4 +405,21 @@ private static String sanitize(char character) {
}
return "_x" + Integer.toHexString(character).toUpperCase();
}
+
+ public static Schema removeFields(Schema.Field field) {
Review comment:
The implementation here seems very specific to a case and not useful as a generic method. Maybe we should move it into the `PruneColumns` class.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] RussellSpitzer commented on a change in pull request #1744: Fix Avro Pruning Bugs with ManifestEntries Table
Posted by GitBox <gi...@apache.org>.
RussellSpitzer commented on a change in pull request #1744:
URL: https://github.com/apache/iceberg/pull/1744#discussion_r521727789
##########
File path: api/src/main/java/org/apache/iceberg/types/GetEmptyStructIds.java
##########
@@ -0,0 +1,76 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.types;
+
+import java.util.List;
+import java.util.Set;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.relocated.com.google.common.collect.Sets;
+
+class GetEmptyStructIds extends TypeUtil.SchemaVisitor<Set<Integer>> {
+ private final Set<Integer> fieldIds = Sets.newHashSet();
+
+ @Override
+ public Set<Integer> schema(Schema schema, Set<Integer> structResult) {
+ return fieldIds;
+ }
+
+ @Override
+ public Set<Integer> struct(Types.StructType struct, List<Set<Integer>> fieldResults) {
+ return fieldIds;
+ }
+
+ private boolean isEmptyStruct(Types.NestedField field) {
+ return field.type().isStructType() && ((Types.StructType) field.type()).fields().isEmpty();
+ }
+
+ /**
+ * We don't know at this point whether the fields contained within a struct are Metadata (and don't exist in the file)
+ * or have been projected in as optional and don't exist in the file. Here we check if all the fields are a
+ * combination of those two states.
+ * @param field a nested field which may be an empty struct
+ * @return true if we need to preserve this struct as it may be needed for its child fields
+ */
+ private boolean isPotentiallyEmpty(Types.NestedField field) {
+ if (!field.type().isStructType()) {
+ return false;
+ }
+ List<Types.NestedField> fields = ((Types.StructType) field.type()).fields();
+ return fields.stream().allMatch(f -> f.fieldId() > Integer.MAX_VALUE - 201 || f.isOptional());
Review comment:
The magic number here is our block of Reserved and Metadata FieldIds
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] RussellSpitzer commented on a change in pull request #1744: Fix Avro Pruning Bugs with ManifestEntries Table
Posted by GitBox <gi...@apache.org>.
RussellSpitzer commented on a change in pull request #1744:
URL: https://github.com/apache/iceberg/pull/1744#discussion_r521779472
##########
File path: core/src/main/java/org/apache/iceberg/avro/PruneColumns.java
##########
@@ -90,12 +92,18 @@ public Schema record(Schema record, List<String> names, List<Schema> fields) {
} else if (fieldSchema != null) {
hasChange = true;
filteredFields.add(copyField(field, fieldSchema, fieldId));
+ } else if (emptyStructIds.contains(fieldId)) {
+ // This field does not require any known sub-fields but is required in the projection so keep it without
+ // any of it's subfields
+ hasChange = true;
+ Schema empty = AvroSchemaUtil.removeFields(field);
Review comment:
CopyRecord does not work on UnionTypes :(
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] RussellSpitzer commented on a change in pull request #1744: Fix Avro Pruning Bugs with ManifestEntries Table
Posted by GitBox <gi...@apache.org>.
RussellSpitzer commented on a change in pull request #1744:
URL: https://github.com/apache/iceberg/pull/1744#discussion_r520041194
##########
File path: spark/src/test/java/org/apache/iceberg/spark/source/TestIcebergSourceTablesBase.java
##########
@@ -225,7 +300,7 @@ public void testCountEntriesTable() {
@Test
public void testFilesTable() throws Exception {
TableIdentifier tableIdentifier = TableIdentifier.of("db", "files_test");
- Table table = createTable(tableIdentifier, SCHEMA, PartitionSpec.builderFor(SCHEMA).identity("id").build());
Review comment:
Minor test cleanup here to use a constant partition spec like other tests
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] RussellSpitzer commented on a change in pull request #1744: Fix Avro Pruning Bugs with ManifestEntries Table
Posted by GitBox <gi...@apache.org>.
RussellSpitzer commented on a change in pull request #1744:
URL: https://github.com/apache/iceberg/pull/1744#discussion_r520732133
##########
File path: api/src/main/java/org/apache/iceberg/types/GetProjectedIds.java
##########
@@ -39,7 +39,7 @@
@Override
public Set<Integer> field(Types.NestedField field, Set<Integer> fieldResult) {
- if (fieldResult == null) {
+ if (fieldResult == null || isEmptyStruct(field)) {
Review comment:
Removed this change leaving the API behavior exactly the same, now instead we add on empty projections when doing the projection itself.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] RussellSpitzer commented on a change in pull request #1744: Fix Avro Pruning Bugs with ManifestEntries Table
Posted by GitBox <gi...@apache.org>.
RussellSpitzer commented on a change in pull request #1744:
URL: https://github.com/apache/iceberg/pull/1744#discussion_r520185320
##########
File path: api/src/main/java/org/apache/iceberg/types/GetProjectedIds.java
##########
@@ -39,7 +39,7 @@
@Override
public Set<Integer> field(Types.NestedField field, Set<Integer> fieldResult) {
- if (fieldResult == null) {
+ if (fieldResult == null || isEmptyStruct(field)) {
Review comment:
The error here previously, is that an empty struct has no fields, since it has no fields we never would ad an ID representing it (normally this happens by referring to subfields)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] RussellSpitzer commented on pull request #1744: Fix Avro Pruning Bugs with ManifestEntries Table
Posted by GitBox <gi...@apache.org>.
RussellSpitzer commented on pull request #1744:
URL: https://github.com/apache/iceberg/pull/1744#issuecomment-893573295
Yep, Seems close to the original plan we had here, i'll keep the new tests and work on rebuilding the original getId's/Project methods.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] RussellSpitzer commented on pull request #1744: Fix Avro Pruning Bugs with ManifestEntries Table
Posted by GitBox <gi...@apache.org>.
RussellSpitzer commented on pull request #1744:
URL: https://github.com/apache/iceberg/pull/1744#issuecomment-923186892
Fixing with #2953
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #1744: Fix Avro Pruning Bugs with ManifestEntries Table
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #1744:
URL: https://github.com/apache/iceberg/pull/1744#discussion_r521707500
##########
File path: core/src/main/java/org/apache/iceberg/avro/PruneColumns.java
##########
@@ -95,7 +95,7 @@ public Schema record(Schema record, List<String> names, List<Schema> fields) {
if (hasChange) {
return copyRecord(record, filteredFields);
- } else if (filteredFields.size() == record.getFields().size()) {
+ } else if (record.getFields().size() != 0 && filteredFields.size() == record.getFields().size()) {
Review comment:
I think there is a reasonable argument that `BuildAvroProjection` should do what your describe, since it is creating the final schema that will be requested from Avro.
If that's adding back the empty structs, then why change the behavior of `PruneColumns` here?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] RussellSpitzer commented on a change in pull request #1744: Fix Avro Pruning Bugs with ManifestEntries Table
Posted by GitBox <gi...@apache.org>.
RussellSpitzer commented on a change in pull request #1744:
URL: https://github.com/apache/iceberg/pull/1744#discussion_r522218058
##########
File path: api/src/main/java/org/apache/iceberg/types/GetEmptyStructIds.java
##########
@@ -0,0 +1,76 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.types;
+
+import java.util.List;
+import java.util.Set;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.relocated.com.google.common.collect.Sets;
+
+class GetEmptyStructIds extends TypeUtil.SchemaVisitor<Set<Integer>> {
+ private final Set<Integer> fieldIds = Sets.newHashSet();
+
+ @Override
+ public Set<Integer> schema(Schema schema, Set<Integer> structResult) {
+ return fieldIds;
+ }
+
+ @Override
+ public Set<Integer> struct(Types.StructType struct, List<Set<Integer>> fieldResults) {
+ return fieldIds;
+ }
+
+ private boolean isEmptyStruct(Types.NestedField field) {
+ return field.type().isStructType() && ((Types.StructType) field.type()).fields().isEmpty();
+ }
+
+ /**
+ * We don't know at this point whether the fields contained within a struct are Metadata (and don't exist in the file)
+ * or have been projected in as optional and don't exist in the file. Here we check if all the fields are a
+ * combination of those two states.
+ * @param field a nested field which may be an empty struct
+ * @return true if we need to preserve this struct as it may be needed for its child fields
+ */
+ private boolean isPotentiallyEmpty(Types.NestedField field) {
+ if (!field.type().isStructType()) {
+ return false;
+ }
+ List<Types.NestedField> fields = ((Types.StructType) field.type()).fields();
+ return fields.stream().allMatch(f -> f.fieldId() > Integer.MAX_VALUE - 201 || f.isOptional());
+ }
+
+ @Override
+ public Set<Integer> field(Types.NestedField field, Set<Integer> fieldResult) {
+ if (isEmptyStruct(field) || isPotentiallyEmpty(field)) {
Review comment:
Ok so I did a brief look at the buildAvroProjection code to see any possible issues, the main difficulty is that we would need to simultaneously traverse the real avro schema along with the one passed as "pruned". If we don't do this we have no way of determining if a required field is actually in the avro schema or not so we can't do proper correctness checks for whether an expected required field should actually be there.
Other than that I think the implementation should be fairly straight forward, if we find a record that should be there but isn't, we build a new record based on the actual field then call "visit" on that field. We will end up breaking the
Let me see how difficult this would be.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] RussellSpitzer commented on pull request #1744: Fix Avro Pruning Bugs with ManifestEntries Table
Posted by GitBox <gi...@apache.org>.
RussellSpitzer commented on pull request #1744:
URL: https://github.com/apache/iceberg/pull/1744#issuecomment-724437522
Also for reference, the other issue we are dealing with is that "PruneColumns" in the current implementation will always select any record whose sub records are empty structs regardless of pruning Id's.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] RussellSpitzer commented on a change in pull request #1744: Fix Avro Pruning Bugs with ManifestEntries Table
Posted by GitBox <gi...@apache.org>.
RussellSpitzer commented on a change in pull request #1744:
URL: https://github.com/apache/iceberg/pull/1744#discussion_r520042587
##########
File path: core/src/main/java/org/apache/iceberg/avro/PruneColumns.java
##########
@@ -95,7 +95,7 @@ public Schema record(Schema record, List<String> names, List<Schema> fields) {
if (hasChange) {
return copyRecord(record, filteredFields);
- } else if (filteredFields.size() == record.getFields().size()) {
+ } else if (record.getFields().size() != 0 && filteredFields.size() == record.getFields().size()) {
Review comment:
This is the second bug mentioned, If you are pruning
`X : Struct { Y : Struct<>, Z: Int, ..... } `
And don't request "X" or "Y"
Y Has no fields so previously `filteredFields.size() = 0 = record.getFields().size()` which means we return
Y as a required field.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] RussellSpitzer commented on a change in pull request #1744: Fix Avro Pruning Bugs with ManifestEntries Table
Posted by GitBox <gi...@apache.org>.
RussellSpitzer commented on a change in pull request #1744:
URL: https://github.com/apache/iceberg/pull/1744#discussion_r521722895
##########
File path: core/src/main/java/org/apache/iceberg/avro/PruneColumns.java
##########
@@ -90,12 +92,18 @@ public Schema record(Schema record, List<String> names, List<Schema> fields) {
} else if (fieldSchema != null) {
hasChange = true;
filteredFields.add(copyField(field, fieldSchema, fieldId));
+ } else if (emptyStructIds.contains(fieldId)) {
+ // This field does not require any known sub-fields but is required in the projection so keep it without
Review comment:
This is the pruning Side fix to Bug 1, Basically making sure we add the parents of any fields in the projection even if none of their real (in the file) fields are projected.
For example without this fix
```
From (A, B { C } )
Project (A, B { D.Optional })
Returns
(A)
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #1744: Fix Avro Pruning Bugs with ManifestEntries Table
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #1744:
URL: https://github.com/apache/iceberg/pull/1744#discussion_r521763485
##########
File path: core/src/main/java/org/apache/iceberg/avro/PruneColumns.java
##########
@@ -95,7 +95,7 @@ public Schema record(Schema record, List<String> names, List<Schema> fields) {
if (hasChange) {
return copyRecord(record, filteredFields);
- } else if (filteredFields.size() == record.getFields().size()) {
+ } else if (record.getFields().size() != 0 && filteredFields.size() == record.getFields().size()) {
Review comment:
Sorry, I meant to ask this on the block above.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] szehon-ho commented on pull request #1744: Fix Avro Pruning Bugs with ManifestEntries Table
Posted by GitBox <gi...@apache.org>.
szehon-ho commented on pull request #1744:
URL: https://github.com/apache/iceberg/pull/1744#issuecomment-810549378
Yes , as we talked offline, I ended up debugging the same thing as @RussellSpitzer while hitting https://github.com/apache/iceberg/issues/1378 . My thought was that that changing the PruneColumns or BuildAvroProjection behaviour would be a bit involved (and maybe not right), so I took a different approach, which I put up as reference: https://github.com/apache/iceberg/pull/2395
It's a poor workaround of the problem by adding non-empty struct when reading the Metadata (manifest) Entries and All-Entries table, it seems to work for most cases. I just put it as an option in case there is no good other fix.
Of course if there is a proper fix, I'd be interested to follow/ help wherever I can, though I heard from Russell that it is not very easy :)
In any case would love to see a solution to the bug. We were trying to write an analytics job that calculated how many data lands in a table per time period, and thought to aggregate entries table and join with snapshot table (which has timestamp), but it seems this bug prevents any aggregates on entries table, without ugly workarounds.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] RussellSpitzer commented on a change in pull request #1744: Fix Avro Pruning Bugs with ManifestEntries Table
Posted by GitBox <gi...@apache.org>.
RussellSpitzer commented on a change in pull request #1744:
URL: https://github.com/apache/iceberg/pull/1744#discussion_r520273924
##########
File path: core/src/main/java/org/apache/iceberg/avro/PruneColumns.java
##########
@@ -95,7 +95,7 @@ public Schema record(Schema record, List<String> names, List<Schema> fields) {
if (hasChange) {
return copyRecord(record, filteredFields);
- } else if (filteredFields.size() == record.getFields().size()) {
+ } else if (record.getFields().size() != 0 && filteredFields.size() == record.getFields().size()) {
Review comment:
Oh yeah I forgot about that :/ yeah we'll need an inverse sort of thing.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] RussellSpitzer commented on a change in pull request #1744: Fix Avro Pruning Bugs with ManifestEntries Table
Posted by GitBox <gi...@apache.org>.
RussellSpitzer commented on a change in pull request #1744:
URL: https://github.com/apache/iceberg/pull/1744#discussion_r522142694
##########
File path: core/src/test/java/org/apache/iceberg/avro/TestReadProjection.java
##########
@@ -526,4 +527,216 @@ public void testListOfStructsProjection() throws IOException {
Assert.assertNull("Should not project y", projectedP2.get("y"));
Assert.assertEquals("Should project null z", null, projectedP2.get("z"));
}
+
+ @Test
+ public void testEmptyStructProjection() throws Exception {
+ Schema writeSchema = new Schema(
+ Types.NestedField.required(0, "id", Types.LongType.get()),
+ Types.NestedField.optional(3, "location", Types.StructType.of(
+ Types.NestedField.required(1, "lat", Types.FloatType.get()),
+ Types.NestedField.required(2, "long", Types.FloatType.get())
+ ))
+ );
+
+ Record record = new Record(AvroSchemaUtil.convert(writeSchema, "table"));
+ record.put("id", 34L);
+ Record location = new Record(
+ AvroSchemaUtil.fromOption(record.getSchema().getField("location").schema()));
+ location.put("lat", 52.995143f);
+ location.put("long", -1.539054f);
+ record.put("location", location);
+
+ Schema emptyStruct = new Schema(
+ Types.NestedField.required(3, "location", Types.StructType.of())
+ );
+
+ Record projected = writeAndRead("empty_proj", writeSchema, emptyStruct, record);
+ Assert.assertNull("Should not project data", projected.get("data"));
+ Record result = (Record) projected.get("location");
Review comment:
Wouldn't this be the same as checking if "id" is not projected? Maybe that's a better check? I can add that in too
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #1744: Fix Avro Pruning Bugs with ManifestEntries Table
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #1744:
URL: https://github.com/apache/iceberg/pull/1744#discussion_r521767305
##########
File path: core/src/test/java/org/apache/iceberg/avro/TestReadProjection.java
##########
@@ -526,4 +527,216 @@ public void testListOfStructsProjection() throws IOException {
Assert.assertNull("Should not project y", projectedP2.get("y"));
Assert.assertEquals("Should project null z", null, projectedP2.get("z"));
}
+
+ @Test
+ public void testEmptyStructProjection() throws Exception {
+ Schema writeSchema = new Schema(
+ Types.NestedField.required(0, "id", Types.LongType.get()),
+ Types.NestedField.optional(3, "location", Types.StructType.of(
+ Types.NestedField.required(1, "lat", Types.FloatType.get()),
+ Types.NestedField.required(2, "long", Types.FloatType.get())
+ ))
+ );
+
+ Record record = new Record(AvroSchemaUtil.convert(writeSchema, "table"));
+ record.put("id", 34L);
+ Record location = new Record(
+ AvroSchemaUtil.fromOption(record.getSchema().getField("location").schema()));
+ location.put("lat", 52.995143f);
+ location.put("long", -1.539054f);
+ record.put("location", location);
+
+ Schema emptyStruct = new Schema(
+ Types.NestedField.required(3, "location", Types.StructType.of())
+ );
+
+ Record projected = writeAndRead("empty_proj", writeSchema, emptyStruct, record);
+ Assert.assertNull("Should not project data", projected.get("data"));
+ Record result = (Record) projected.get("location");
+ Assert.assertNotNull("Should contain an empty record", result);
+ Assert.assertNull("Should not project lat", result.get("lat"));
+ Assert.assertNull("Should not project long", result.get("long"));
+ }
+
+ @Test
+ public void testEmptyStructRequiredProjection() throws Exception {
+ Schema writeSchema = new Schema(
+ Types.NestedField.required(0, "id", Types.LongType.get()),
+ Types.NestedField.required(3, "location", Types.StructType.of(
+ Types.NestedField.required(1, "lat", Types.FloatType.get()),
+ Types.NestedField.required(2, "long", Types.FloatType.get())
+ ))
+ );
+
+ Record record = new Record(AvroSchemaUtil.convert(writeSchema, "table"));
+ record.put("id", 34L);
+ Record location = new Record(record.getSchema().getField("location").schema());
+ location.put("lat", 52.995143f);
+ location.put("long", -1.539054f);
+ record.put("location", location);
+
+ Schema emptyStruct = new Schema(
+ Types.NestedField.required(3, "location", Types.StructType.of())
+ );
+
+ Record projected = writeAndRead("empty_req_proj", writeSchema, emptyStruct, record);
+ Assert.assertNull("Should not project data", projected.get("data"));
+ Record result = (Record) projected.get("location");
+ Assert.assertNotNull("Should contain an empty record", result);
+ Assert.assertNull("Should not project lat", result.get("lat"));
+ Assert.assertNull("Should not project long", result.get("long"));
+ }
+
+ @Test
+ public void testEmptyNestedStructProjection() throws Exception {
+ Schema writeSchema = new Schema(
+ Types.NestedField.required(0, "id", Types.LongType.get()),
+ Types.NestedField.optional(3, "outer", Types.StructType.of(
+ Types.NestedField.required(1, "lat", Types.FloatType.get()),
+ Types.NestedField.optional(2, "inner", Types.StructType.of(
+ Types.NestedField.required(5, "lon", Types.FloatType.get())
+ )
+ )
+ ))
+ );
+
+ Record record = new Record(AvroSchemaUtil.convert(writeSchema, "table"));
+ record.put("id", 34L);
+ Record outer = new Record(
+ AvroSchemaUtil.fromOption(record.getSchema().getField("outer").schema()));
+ Record inner = new Record(AvroSchemaUtil.fromOption(outer.getSchema().getField("inner").schema()));
+ inner.put("lon", 32.14f);
+ outer.put("lat", 52.995143f);
+ outer.put("inner", inner);
+ record.put("outer", outer);
+
+ Schema emptyStruct = new Schema(
+ Types.NestedField.required(3, "outer", Types.StructType.of(
+ Types.NestedField.required(2, "inner", Types.StructType.of())
+ )));
+
+ Record projected = writeAndRead("nested_empty_proj", writeSchema, emptyStruct, record);
+ Assert.assertNull("Should not project data", projected.get("id"));
+ Record outerResult = (Record) projected.get("outer");
+ Assert.assertNotNull("Should contain the outer record", outerResult);
+ Assert.assertNull("Should not contain lat", outerResult.get("lat"));
+ Record innerResult = (Record) outerResult.get("inner");
+ Assert.assertNotNull("Should contain the inner record", innerResult);
+ Assert.assertNull("Should not contain lon", innerResult.get("lon"));
+ }
+
+ @Test
+ public void testEmptyNestedStructRequiredProjection() throws Exception {
+ Schema writeSchema = new Schema(
+ Types.NestedField.required(0, "id", Types.LongType.get()),
+ Types.NestedField.required(3, "outer", Types.StructType.of(
+ Types.NestedField.required(1, "lat", Types.FloatType.get()),
+ Types.NestedField.required(2, "inner", Types.StructType.of(
+ Types.NestedField.required(5, "lon", Types.FloatType.get())
+ )
+ )
+ ))
+ );
+
+ Record record = new Record(AvroSchemaUtil.convert(writeSchema, "table"));
+ record.put("id", 34L);
+ Record outer = new Record(record.getSchema().getField("outer").schema());
+ Record inner = new Record(outer.getSchema().getField("inner").schema());
+ inner.put("lon", 32.14f);
+ outer.put("lat", 52.995143f);
+ outer.put("inner", inner);
+ record.put("outer", outer);
+
+ Schema emptyStruct = new Schema(
+ Types.NestedField.required(3, "outer", Types.StructType.of(
+ Types.NestedField.required(2, "inner", Types.StructType.of())
+ )));
+
+ Record projected = writeAndRead("nested_empty_req_proj", writeSchema, emptyStruct, record);
+ Assert.assertNull("Should not project data", projected.get("id"));
+ Record outerResult = (Record) projected.get("outer");
+ Assert.assertNotNull("Should contain the outer record", outerResult);
+ Assert.assertNull("Should not contain lat", outerResult.get("lat"));
+ Record innerResult = (Record) outerResult.get("inner");
+ Assert.assertNotNull("Should contain the inner record", innerResult);
+ Assert.assertNull("Should not contain lon", innerResult.get("lon"));
+ }
+
+ @Test
+ public void testMetadataFieldProjection() throws Exception {
+ Schema writeSchema = new Schema(
+ Types.NestedField.required(0, "id", Types.LongType.get()),
+ Types.NestedField.required(3, "outer", Types.StructType.of(
+ Types.NestedField.required(1, "lat", Types.FloatType.get()),
+ Types.NestedField.required(2, "inner", Types.StructType.of(
+ Types.NestedField.required(5, "lon", Types.FloatType.get())
+ )
+ )
+ ))
+ );
+
+ Record record = new Record(AvroSchemaUtil.convert(writeSchema, "table"));
+ record.put("id", 34L);
+ Record outer = new Record(record.getSchema().getField("outer").schema());
+ Record inner = new Record(outer.getSchema().getField("inner").schema());
+ inner.put("lon", 32.14f);
+ outer.put("lat", 52.995143f);
+ outer.put("inner", inner);
+ record.put("outer", outer);
+
+ Schema metadataStruct = new Schema(
+ Types.NestedField.required(3, "outer", Types.StructType.of(
+ Types.NestedField.required(2, "inner", Types.StructType.of(MetadataColumns.ROW_POSITION)
+ ))
+ ));
+
+ Record projected = writeAndRead("metadata_field_proj", writeSchema, metadataStruct, record);
+ Assert.assertNull("Should not project data", projected.get("id"));
+ Record outerResult = (Record) projected.get("outer");
+ Assert.assertNotNull("Should contain the outer record", outerResult);
+ Assert.assertNull("Should not contain lat", outerResult.get("lat"));
+ Record innerResult = (Record) outerResult.get("inner");
+ Assert.assertNotNull("Should contain the inner record", innerResult);
+ Assert.assertNull("Should not contain lon", innerResult.get("lon"));
+ String metaName = MetadataColumns.ROW_POSITION.name() + "_r" + MetadataColumns.ROW_POSITION.fieldId();
+ Assert.assertNotNull("Should contain metadata field", innerResult.get(metaName));
+ }
+
+ @Test
+ public void testNonExistentProjection() throws Exception {
+ Schema writeSchema = new Schema(
+ Types.NestedField.required(0, "id", Types.LongType.get()),
+ Types.NestedField.optional(3, "location", Types.StructType.of(
+ Types.NestedField.required(1, "lat", Types.FloatType.get()),
+ Types.NestedField.required(2, "long", Types.FloatType.get())
+ ))
+ );
+
+ Record record = new Record(AvroSchemaUtil.convert(writeSchema, "table"));
+ record.put("id", 34L);
+ Record location = new Record(
+ AvroSchemaUtil.fromOption(record.getSchema().getField("location").schema()));
+ location.put("lat", 52.995143f);
+ location.put("long", -1.539054f);
+ record.put("location", location);
+
+ Schema emptyStruct = new Schema(
+ Types.NestedField.required(3, "location", Types.StructType.of(
+ Types.NestedField.optional(10000, "foo", Types.StructType.of(
+ Types.NestedField.optional(10001, "bar", Types.IntegerType.get())
+ ))
+ ))
+ );
+
+ Record projected = writeAndRead("non_existant_proj", writeSchema, emptyStruct, record);
+ Assert.assertNull("Should not project data", projected.get("data"));
+ Record result = (Record) projected.get("location");
+ Assert.assertNotNull("Should contain an fake optional record", result);
+ Assert.assertNull("Should not project lat", result.get("lat"));
+ Assert.assertNull("Should not project long", result.get("long"));
Review comment:
I think this also needs to assert that `foo` is null because it is not in the source data. A record that is missing from the source data should be null if optional and result in an exception if required.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] RussellSpitzer commented on a change in pull request #1744: Fix Avro Pruning Bugs with ManifestEntries Table
Posted by GitBox <gi...@apache.org>.
RussellSpitzer commented on a change in pull request #1744:
URL: https://github.com/apache/iceberg/pull/1744#discussion_r521718519
##########
File path: core/src/main/java/org/apache/iceberg/avro/PruneColumns.java
##########
@@ -95,7 +95,7 @@ public Schema record(Schema record, List<String> names, List<Schema> fields) {
if (hasChange) {
return copyRecord(record, filteredFields);
- } else if (filteredFields.size() == record.getFields().size()) {
+ } else if (record.getFields().size() != 0 && filteredFields.size() == record.getFields().size()) {
Review comment:
This line is the change to fix bug 2,
**Example of Bug 2**
```
From (A, B, C, D {foo, bar, baz {}} ) // Starting with D containing a field baz which is empty
Project (A)
Returns
(A, D{ baz {} }) // Includes D even though it was not requested
```
The issue being that a struct which contains no elements naturally will always be included because
`records.filedsize == 0 == filteredFields.size == 0`
Because the field is included, all parent structs which include it are also included in the projection. For example
```
From (A , B{ C { D { E { F{}, G }}}})
Project (A)
Returns
(A, B{ C { D { E { F{}}}}})
```
I think that behavior is incorrect, and this should be pruned out if it isn't needed in the projection
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on pull request #1744: Fix Avro Pruning Bugs with ManifestEntries Table
Posted by GitBox <gi...@apache.org>.
rdblue commented on pull request #1744:
URL: https://github.com/apache/iceberg/pull/1744#issuecomment-890564959
If I remember correctly, the main problem here was a mismatch between the expected schema and the actual schema when reading from Avro files. That was caused by some unexpected behaviors across a few helpers:
1. `TypeUtil.select` (that uses `PruneColumns`) has behavior like SQL projection using IDs. Any field that is selected by ID is completely projected. For example, selecting `{4}` from the schema `3: id bigint, 4: location struct<5: lat double, 6: long double>` is like `SELECT location` and produces `4: location struct<5: lat double, 6 long double>` as the projected schema. As a result, empty structs can't be selected by this method (or in SQL).
2. `TypeUtil.getProjectedIds` (that uses `GetProjectedIds`) is written expecting the current behavior from `TypeUtil.select`. When finding the IDs needed to describe a projection, it will return all leaf field IDs. Because empty structs contain no leaf fields, there are no IDs returned for an empty struct. Reusing the example above, if the requested projection were `4: location struct<>` (no fields) then the result is `{}` rather than `{4}` because `select` would return all fields of `location`.
3. Record types are position based and readers assume that the expected schema will match the read schema, even though the read schema is produced using the file schema, conversions, and calls to the two methods above. An empty struct in the requested schema will be silently removed, producing a mismatch in the field positions. When reading with `id bigint, location struct<>, data string`, the data produced will be `id bigint, data string`, resulting in an `ArrayIndexOutOfBounds` exception or a `ClassCastException` depending on the rest of the projection.
4. There is a bug that sometimes avoids this problem by accidentally projecting empty structs, but this is not reliable because sometimes the empty struct should not be produced or else it introduces the opposite problem: projecting `id bigint, data string` actually produces `id bigint, location struct<>, data string`.
Hopefully that captures the issues here.
First, I think that `select` behavior is correct. This matches the behavior where a list of selected fields should be turned into a list of IDs and completely projected. Selecting a struct should select all of its fields, as expected by `SELECT location`. The problem is that this is that we assume that the output of `getProjectedIds` will be used for `select` and so it can't return any non-leaf field IDs.
To fix this, I propose the following:
* Update the behavior of `getProjectedIds` to return an ID for empty structs. The ID should indicate that the struct must be returned without any fields, unless there are sub-fields selected by a nested field ID.
* Add a `TypeUtil.project` method that is the opposite of `getProjectedIds`. The combination of `getProjectedIds` and `project` should always be the same schema, with empty structs.
* Update all calls to `select` and avoid passing the result of `getProjectedIds` because of the behavior change. In some cases, this will actually fix behavior: `StructProjection.create(Schema base, Schema projection)` should use `project` to preserve empty structs, for example. But other places, like `BaseTableScan.lazyColumnProjection` can avoid calling `getProjectedIds` because it isn't needed (and would change behavior).
* Update most calls that need to produce a schema matching the expected schema to use `project` instead of `select` so that empty structs are preserved.
* Fix the second bug, which is unrelated but also incorrect.
I think that should fix this. To me, that seems like a couple of PRs: one to add `TypeUtil.project` that will produce empty structs instead of fully-selected structs (maybe just a boolean option to configure `PruneColumns`). Then one PR to update the behavior of `getProjectedIds` and fix `select` calls that use the result to correctly call `project` or avoid `getProjectedIds`. And finally, one PR to fix the second bug.
Does that sound reasonable?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] RussellSpitzer commented on pull request #1744: Fix Avro Pruning Bugs with ManifestEntries Table
Posted by GitBox <gi...@apache.org>.
RussellSpitzer commented on pull request #1744:
URL: https://github.com/apache/iceberg/pull/1744#issuecomment-725235420
Automated tests failed on the last run that work locally on my MacBook :/ but oddly enough not the direct schema projection tests, only the metadata table test ... maybe it won't break this time?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] RussellSpitzer commented on a change in pull request #1744: Fix Avro Pruning Bugs with ManifestEntries Table
Posted by GitBox <gi...@apache.org>.
RussellSpitzer commented on a change in pull request #1744:
URL: https://github.com/apache/iceberg/pull/1744#discussion_r521725991
##########
File path: api/src/main/java/org/apache/iceberg/types/GetEmptyStructIds.java
##########
@@ -0,0 +1,76 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iceberg.types;
+
+import java.util.List;
+import java.util.Set;
+import org.apache.iceberg.Schema;
+import org.apache.iceberg.relocated.com.google.common.collect.Sets;
+
+class GetEmptyStructIds extends TypeUtil.SchemaVisitor<Set<Integer>> {
Review comment:
This Visitor is used to find the FieldIDs of any fields which must remain in the projection, but if they have no sub fields, should be just included as empty Structs.
There are 3 Main cases of this
1. The Struct is Empty
2. The Sturct contains required MetadataColumns, these are not real so we cannot properly include their parents in the normal path. Currently the only one of these allowed is MetadataColumns.ROW_POSITION, see https://github.com/apache/iceberg/blob/074323214a8a580f85251d700f3754e2fb6e2758/core/src/main/java/org/apache/iceberg/avro/BuildAvroProjection.java#L98-L102
3. The Struct contains Optional Columns which aren't part of the schema, these are a allowed by the same code block mentioned above, but if their parents are pruned out we will have an error when we attempt to add them back to the projection.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #1744: Fix Avro Pruning Bugs with ManifestEntries Table
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #1744:
URL: https://github.com/apache/iceberg/pull/1744#discussion_r521765745
##########
File path: core/src/test/java/org/apache/iceberg/avro/TestReadProjection.java
##########
@@ -526,4 +527,216 @@ public void testListOfStructsProjection() throws IOException {
Assert.assertNull("Should not project y", projectedP2.get("y"));
Assert.assertEquals("Should project null z", null, projectedP2.get("z"));
}
+
+ @Test
+ public void testEmptyStructProjection() throws Exception {
+ Schema writeSchema = new Schema(
+ Types.NestedField.required(0, "id", Types.LongType.get()),
+ Types.NestedField.optional(3, "location", Types.StructType.of(
+ Types.NestedField.required(1, "lat", Types.FloatType.get()),
+ Types.NestedField.required(2, "long", Types.FloatType.get())
+ ))
+ );
+
+ Record record = new Record(AvroSchemaUtil.convert(writeSchema, "table"));
+ record.put("id", 34L);
+ Record location = new Record(
+ AvroSchemaUtil.fromOption(record.getSchema().getField("location").schema()));
+ location.put("lat", 52.995143f);
+ location.put("long", -1.539054f);
+ record.put("location", location);
+
+ Schema emptyStruct = new Schema(
+ Types.NestedField.required(3, "location", Types.StructType.of())
+ );
+
+ Record projected = writeAndRead("empty_proj", writeSchema, emptyStruct, record);
+ Assert.assertNull("Should not project data", projected.get("data"));
+ Record result = (Record) projected.get("location");
Review comment:
I think we should access the fields that are present by position as well. The expected position for this is 0, so asserting that the same record is returned would be a good check.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #1744: Fix Avro Pruning Bugs with ManifestEntries Table
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #1744:
URL: https://github.com/apache/iceberg/pull/1744#discussion_r521767804
##########
File path: core/src/test/java/org/apache/iceberg/avro/TestReadProjection.java
##########
@@ -526,4 +527,216 @@ public void testListOfStructsProjection() throws IOException {
Assert.assertNull("Should not project y", projectedP2.get("y"));
Assert.assertEquals("Should project null z", null, projectedP2.get("z"));
}
+
+ @Test
+ public void testEmptyStructProjection() throws Exception {
+ Schema writeSchema = new Schema(
+ Types.NestedField.required(0, "id", Types.LongType.get()),
+ Types.NestedField.optional(3, "location", Types.StructType.of(
+ Types.NestedField.required(1, "lat", Types.FloatType.get()),
+ Types.NestedField.required(2, "long", Types.FloatType.get())
+ ))
+ );
+
+ Record record = new Record(AvroSchemaUtil.convert(writeSchema, "table"));
+ record.put("id", 34L);
+ Record location = new Record(
+ AvroSchemaUtil.fromOption(record.getSchema().getField("location").schema()));
+ location.put("lat", 52.995143f);
+ location.put("long", -1.539054f);
+ record.put("location", location);
+
+ Schema emptyStruct = new Schema(
+ Types.NestedField.required(3, "location", Types.StructType.of())
+ );
+
+ Record projected = writeAndRead("empty_proj", writeSchema, emptyStruct, record);
+ Assert.assertNull("Should not project data", projected.get("data"));
+ Record result = (Record) projected.get("location");
+ Assert.assertNotNull("Should contain an empty record", result);
+ Assert.assertNull("Should not project lat", result.get("lat"));
+ Assert.assertNull("Should not project long", result.get("long"));
+ }
+
+ @Test
+ public void testEmptyStructRequiredProjection() throws Exception {
+ Schema writeSchema = new Schema(
+ Types.NestedField.required(0, "id", Types.LongType.get()),
+ Types.NestedField.required(3, "location", Types.StructType.of(
+ Types.NestedField.required(1, "lat", Types.FloatType.get()),
+ Types.NestedField.required(2, "long", Types.FloatType.get())
+ ))
+ );
+
+ Record record = new Record(AvroSchemaUtil.convert(writeSchema, "table"));
+ record.put("id", 34L);
+ Record location = new Record(record.getSchema().getField("location").schema());
+ location.put("lat", 52.995143f);
+ location.put("long", -1.539054f);
+ record.put("location", location);
+
+ Schema emptyStruct = new Schema(
+ Types.NestedField.required(3, "location", Types.StructType.of())
+ );
+
+ Record projected = writeAndRead("empty_req_proj", writeSchema, emptyStruct, record);
+ Assert.assertNull("Should not project data", projected.get("data"));
+ Record result = (Record) projected.get("location");
+ Assert.assertNotNull("Should contain an empty record", result);
+ Assert.assertNull("Should not project lat", result.get("lat"));
+ Assert.assertNull("Should not project long", result.get("long"));
+ }
+
+ @Test
+ public void testEmptyNestedStructProjection() throws Exception {
+ Schema writeSchema = new Schema(
+ Types.NestedField.required(0, "id", Types.LongType.get()),
+ Types.NestedField.optional(3, "outer", Types.StructType.of(
+ Types.NestedField.required(1, "lat", Types.FloatType.get()),
+ Types.NestedField.optional(2, "inner", Types.StructType.of(
+ Types.NestedField.required(5, "lon", Types.FloatType.get())
+ )
+ )
+ ))
+ );
+
+ Record record = new Record(AvroSchemaUtil.convert(writeSchema, "table"));
+ record.put("id", 34L);
+ Record outer = new Record(
+ AvroSchemaUtil.fromOption(record.getSchema().getField("outer").schema()));
+ Record inner = new Record(AvroSchemaUtil.fromOption(outer.getSchema().getField("inner").schema()));
+ inner.put("lon", 32.14f);
+ outer.put("lat", 52.995143f);
+ outer.put("inner", inner);
+ record.put("outer", outer);
+
+ Schema emptyStruct = new Schema(
+ Types.NestedField.required(3, "outer", Types.StructType.of(
+ Types.NestedField.required(2, "inner", Types.StructType.of())
+ )));
+
+ Record projected = writeAndRead("nested_empty_proj", writeSchema, emptyStruct, record);
+ Assert.assertNull("Should not project data", projected.get("id"));
+ Record outerResult = (Record) projected.get("outer");
+ Assert.assertNotNull("Should contain the outer record", outerResult);
+ Assert.assertNull("Should not contain lat", outerResult.get("lat"));
+ Record innerResult = (Record) outerResult.get("inner");
+ Assert.assertNotNull("Should contain the inner record", innerResult);
+ Assert.assertNull("Should not contain lon", innerResult.get("lon"));
+ }
+
+ @Test
+ public void testEmptyNestedStructRequiredProjection() throws Exception {
Review comment:
This case looks right to me.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] RussellSpitzer commented on a change in pull request #1744: Fix Avro Pruning Bugs with ManifestEntries Table
Posted by GitBox <gi...@apache.org>.
RussellSpitzer commented on a change in pull request #1744:
URL: https://github.com/apache/iceberg/pull/1744#discussion_r521718519
##########
File path: core/src/main/java/org/apache/iceberg/avro/PruneColumns.java
##########
@@ -95,7 +95,7 @@ public Schema record(Schema record, List<String> names, List<Schema> fields) {
if (hasChange) {
return copyRecord(record, filteredFields);
- } else if (filteredFields.size() == record.getFields().size()) {
+ } else if (record.getFields().size() != 0 && filteredFields.size() == record.getFields().size()) {
Review comment:
This line is the change to fix bug 2,
**Example of Bug 2**
```
From (A, B, C, D {foo, bar, baz {}} ) // Starting with D containing a field baz which is empty
Project (A)
Returns
(A, D{ baz {} }) // Includes D even though it was not requested
```
The issue being that a struct which contains no elements naturally will always be included because
`records.filedsize == 0 == filteredFields.size == 0`
I think that behavior is incorrect, and this should be pruned out if it isn't needed in the projection
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] szehon-ho edited a comment on pull request #1744: Fix Avro Pruning Bugs with ManifestEntries Table
Posted by GitBox <gi...@apache.org>.
szehon-ho edited a comment on pull request #1744:
URL: https://github.com/apache/iceberg/pull/1744#issuecomment-810549378
Yes , as we talked offline, I ended up debugging the same thing as @RussellSpitzer while hitting https://github.com/apache/iceberg/issues/1378 . My thought was that that changing the PruneColumns or BuildAvroProjection behaviour would be a bit involved, so I took a different approach, which I put up as reference: https://github.com/apache/iceberg/pull/2395
It's a poor workaround of the problem by adding non-empty struct when reading the Metadata (manifest) Entries and All-Entries table, though it seems to work for most cases. I'll have to try some other cases (I guess it's required non-empty structs that get pruned away, if there are other ones in Iceberg metadata tables) I just put it as an option in case there is no good other fix.
Of course if there is a proper fix, I'd be interested to follow/ help wherever I can, though I heard from Russell that it is not very easy :)
In any case would love to see a solution to the bug. We were trying to write an analytics job that calculated how many data lands in a table per time period, and thought to aggregate entries table and join with snapshot table (which has timestamp), but it seems this bug prevents any aggregates on entries table, without ugly workarounds.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] RussellSpitzer commented on pull request #1744: Fix Avro Pruning Bugs with ManifestEntries Table
Posted by GitBox <gi...@apache.org>.
RussellSpitzer commented on pull request #1744:
URL: https://github.com/apache/iceberg/pull/1744#issuecomment-725252841
It was the merge, something in master has broken my tests :/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] szehon-ho edited a comment on pull request #1744: Fix Avro Pruning Bugs with ManifestEntries Table
Posted by GitBox <gi...@apache.org>.
szehon-ho edited a comment on pull request #1744:
URL: https://github.com/apache/iceberg/pull/1744#issuecomment-810549378
Yes , as we talked offline, I ended up debugging the same thing as @RussellSpitzer while hitting https://github.com/apache/iceberg/issues/1378 . My thought was that that changing the PruneColumns or BuildAvroProjection behaviour would be a bit involved, so I took a different approach, which I put up as reference: https://github.com/apache/iceberg/pull/2395
It's a poor workaround of the problem by adding non-empty struct when reading the Metadata (manifest) Entries and All-Entries table, though it seems to work for most cases. (Bug hits nested non-empty required structs that are pruned away, can look for those if there are more) I just put it as an option in case there is no good other fix.
Of course if there is a proper fix, I'd be interested to follow/ help wherever I can, though I heard from Russell that it is not very easy :)
In any case would love to see a solution to the bug. We were trying to write an analytics job that calculated how many data lands in a table per time period, and thought to aggregate entries table and join with snapshot table (which has timestamp), but it seems this bug prevents any aggregates on entries table, without ugly workarounds.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] RussellSpitzer commented on a change in pull request #1744: Fix Avro Pruning Bugs with ManifestEntries Table
Posted by GitBox <gi...@apache.org>.
RussellSpitzer commented on a change in pull request #1744:
URL: https://github.com/apache/iceberg/pull/1744#discussion_r520707087
##########
File path: core/src/main/java/org/apache/iceberg/avro/PruneColumns.java
##########
@@ -95,7 +95,7 @@ public Schema record(Schema record, List<String> names, List<Schema> fields) {
if (hasChange) {
return copyRecord(record, filteredFields);
- } else if (filteredFields.size() == record.getFields().size()) {
+ } else if (record.getFields().size() != 0 && filteredFields.size() == record.getFields().size()) {
Review comment:
I tried to figure out if there was a simpler way to do this and I ended up modifying BuildAvroProjection. In my mind the real problem here was we were telling BuildAvroProjection what columns we wanted, but our pruned schema really only showed us what columns we needed that have data. This only left leaf empty-struct nodes. So during BuildAvroProjection we check for these empty-structs being requested in the expected schema and just add them back.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org