You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2020/05/05 00:04:40 UTC
[GitHub] [incubator-iceberg] rdblue opened a new pull request #1001: Refactor Parquet visitor
rdblue opened a new pull request #1001:
URL: https://github.com/apache/incubator-iceberg/pull/1001
This refactors the Parquet visitor to support #830.
* Adds before and after calls that can be overridden to avoid adding field names
* Makes `fieldNames` private
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [incubator-iceberg] rdblue commented on a change in pull request #1001: Refactor Parquet visitor
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #1001:
URL: https://github.com/apache/incubator-iceberg/pull/1001#discussion_r420225920
##########
File path: parquet/src/main/java/org/apache/iceberg/parquet/ParquetTypeVisitor.java
##########
@@ -123,19 +60,115 @@
}
}
- private static <T> T visitField(Type field, ParquetTypeVisitor<T> visitor) {
- visitor.fieldNames.push(field.getName());
+ private static <T> T visitList(GroupType list, ParquetTypeVisitor<T> visitor) {
Review comment:
This was moved into its own method. The only changes are to use the new before and after methods.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [incubator-iceberg] rdblue commented on a change in pull request #1001: Refactor Parquet visitor
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #1001:
URL: https://github.com/apache/incubator-iceberg/pull/1001#discussion_r420226161
##########
File path: parquet/src/main/java/org/apache/iceberg/parquet/ParquetTypeVisitor.java
##########
@@ -123,19 +60,115 @@
}
}
- private static <T> T visitField(Type field, ParquetTypeVisitor<T> visitor) {
- visitor.fieldNames.push(field.getName());
+ private static <T> T visitList(GroupType list, ParquetTypeVisitor<T> visitor) {
+ Preconditions.checkArgument(!list.isRepetition(Type.Repetition.REPEATED),
+ "Invalid list: top-level group is repeated: %s", list);
+ Preconditions.checkArgument(list.getFieldCount() == 1,
+ "Invalid list: does not contain single repeated field: %s", list);
+
+ GroupType repeatedElement = list.getFields().get(0).asGroupType();
+ Preconditions.checkArgument(repeatedElement.isRepetition(Type.Repetition.REPEATED),
+ "Invalid list: inner group is not repeated");
+ Preconditions.checkArgument(repeatedElement.getFieldCount() <= 1,
+ "Invalid list: repeated group is not a single field: %s", list);
+
+ visitor.beforeRepeatedElement(repeatedElement);
try {
- return visit(field, visitor);
+ T elementResult = null;
+ if (repeatedElement.getFieldCount() > 0) {
+ Type elementField = repeatedElement.getType(0);
+ visitor.beforeElementField(elementField);
+ try {
+ elementResult = visit(elementField, visitor);
+ } finally {
+ visitor.afterElementField(elementField);
+ }
+ }
+
+ return visitor.list(list, elementResult);
+
} finally {
- visitor.fieldNames.pop();
+ visitor.afterRepeatedElement(repeatedElement);
+ }
+ }
+
+ private static <T> T visitMap(GroupType map, ParquetTypeVisitor<T> visitor) {
Review comment:
This was moved to its own method. The only changes are to use the new before and after methods.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org