You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2020/06/19 23:33:36 UTC
[GitHub] [iceberg] rdsr opened a new pull request #1130: Identity Partitioning: Use partitionsMap in InputFormat
rdsr opened a new pull request #1130:
URL: https://github.com/apache/iceberg/pull/1130
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue merged pull request #1130: Identity Partitioning: Use PartitionUtil.constantsMap in InputFormat
Posted by GitBox <gi...@apache.org>.
rdblue merged pull request #1130:
URL: https://github.com/apache/iceberg/pull/1130
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdsr commented on a change in pull request #1130: Identity Partitioning: Use PartitionUtil.constantsMap in InputFormat
Posted by GitBox <gi...@apache.org>.
rdsr commented on a change in pull request #1130:
URL: https://github.com/apache/iceberg/pull/1130#discussion_r443077759
##########
File path: mr/src/main/java/org/apache/iceberg/mr/mapreduce/IcebergInputFormat.java
##########
@@ -348,49 +342,38 @@ public void close() throws IOException {
currentIterator.close();
}
- private static Map<String, Integer> buildNameToPos(Schema expectedSchema) {
- Map<String, Integer> nameToPos = Maps.newHashMap();
- for (int pos = 0; pos < expectedSchema.asStruct().fields().size(); pos++) {
- Types.NestedField field = expectedSchema.asStruct().fields().get(pos);
- nameToPos.put(field.name(), pos);
- }
- return nameToPos;
- }
-
- private CloseableIterator<T> open(FileScanTask currentTask) {
- DataFile file = currentTask.file();
- // schema of rows returned by readers
- PartitionSpec spec = currentTask.spec();
- Set<Integer> idColumns = Sets.intersection(spec.identitySourceIds(), TypeUtil.getProjectedIds(expectedSchema));
- boolean hasJoinedPartitionColumns = !idColumns.isEmpty();
+ private CloseableIterator<T> open(FileScanTask task) {
+ PartitionSpec spec = task.spec();
+ Set<Integer> idColumns = spec.identitySourceIds();
+ Schema partitionSchema = TypeUtil.select(expectedSchema, idColumns);
+ boolean projectsIdentityPartitionColumns = !partitionSchema.columns().isEmpty();
CloseableIterable<T> iterable;
- if (hasJoinedPartitionColumns) {
- Schema readDataSchema = TypeUtil.selectNot(expectedSchema, idColumns);
- Schema identityPartitionSchema = TypeUtil.select(expectedSchema, idColumns);
- iterable = CloseableIterable.transform(open(currentTask, readDataSchema),
- row -> withIdentityPartitionColumns(row, identityPartitionSchema, spec, file.partition()));
+ if (projectsIdentityPartitionColumns) {
+ //TODO: seems like we have to specify a converter to convert the partition values
Review comment:
I was hesitant to use the converter from TableScanIterable as it seems it might only supports Avro. Also we might require different maps for Hive and Pig
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #1130: Identity Partitioning: Use PartitionUtil.constantsMap in InputFormat
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #1130:
URL: https://github.com/apache/iceberg/pull/1130#discussion_r443725492
##########
File path: mr/src/main/java/org/apache/iceberg/mr/mapreduce/IcebergInputFormat.java
##########
@@ -348,49 +342,38 @@ public void close() throws IOException {
currentIterator.close();
}
- private static Map<String, Integer> buildNameToPos(Schema expectedSchema) {
- Map<String, Integer> nameToPos = Maps.newHashMap();
- for (int pos = 0; pos < expectedSchema.asStruct().fields().size(); pos++) {
- Types.NestedField field = expectedSchema.asStruct().fields().get(pos);
- nameToPos.put(field.name(), pos);
- }
- return nameToPos;
- }
-
- private CloseableIterator<T> open(FileScanTask currentTask) {
- DataFile file = currentTask.file();
- // schema of rows returned by readers
- PartitionSpec spec = currentTask.spec();
- Set<Integer> idColumns = Sets.intersection(spec.identitySourceIds(), TypeUtil.getProjectedIds(expectedSchema));
- boolean hasJoinedPartitionColumns = !idColumns.isEmpty();
+ private CloseableIterator<T> open(FileScanTask task) {
+ PartitionSpec spec = task.spec();
+ Set<Integer> idColumns = spec.identitySourceIds();
+ Schema partitionSchema = TypeUtil.select(expectedSchema, idColumns);
+ boolean projectsIdentityPartitionColumns = !partitionSchema.columns().isEmpty();
CloseableIterable<T> iterable;
- if (hasJoinedPartitionColumns) {
- Schema readDataSchema = TypeUtil.selectNot(expectedSchema, idColumns);
- Schema identityPartitionSchema = TypeUtil.select(expectedSchema, idColumns);
- iterable = CloseableIterable.transform(open(currentTask, readDataSchema),
- row -> withIdentityPartitionColumns(row, identityPartitionSchema, spec, file.partition()));
+ if (projectsIdentityPartitionColumns) {
+ //TODO: seems like we have to specify a converter to convert the partition values
Review comment:
Now that the generics for Avro are in core, we can put them in the org.apache.iceberg.data package in iceberg-core. How about a new `Converters` class?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdsr commented on a change in pull request #1130: Identity Partitioning: Use PartitionUtil.constantsMap in InputFormat
Posted by GitBox <gi...@apache.org>.
rdsr commented on a change in pull request #1130:
URL: https://github.com/apache/iceberg/pull/1130#discussion_r443740815
##########
File path: mr/src/main/java/org/apache/iceberg/mr/mapreduce/IcebergInputFormat.java
##########
@@ -348,49 +342,38 @@ public void close() throws IOException {
currentIterator.close();
}
- private static Map<String, Integer> buildNameToPos(Schema expectedSchema) {
- Map<String, Integer> nameToPos = Maps.newHashMap();
- for (int pos = 0; pos < expectedSchema.asStruct().fields().size(); pos++) {
- Types.NestedField field = expectedSchema.asStruct().fields().get(pos);
- nameToPos.put(field.name(), pos);
- }
- return nameToPos;
- }
-
- private CloseableIterator<T> open(FileScanTask currentTask) {
- DataFile file = currentTask.file();
- // schema of rows returned by readers
- PartitionSpec spec = currentTask.spec();
- Set<Integer> idColumns = Sets.intersection(spec.identitySourceIds(), TypeUtil.getProjectedIds(expectedSchema));
- boolean hasJoinedPartitionColumns = !idColumns.isEmpty();
+ private CloseableIterator<T> open(FileScanTask task) {
+ PartitionSpec spec = task.spec();
+ Set<Integer> idColumns = spec.identitySourceIds();
+ Schema partitionSchema = TypeUtil.select(expectedSchema, idColumns);
+ boolean projectsIdentityPartitionColumns = !partitionSchema.columns().isEmpty();
CloseableIterable<T> iterable;
- if (hasJoinedPartitionColumns) {
- Schema readDataSchema = TypeUtil.selectNot(expectedSchema, idColumns);
- Schema identityPartitionSchema = TypeUtil.select(expectedSchema, idColumns);
- iterable = CloseableIterable.transform(open(currentTask, readDataSchema),
- row -> withIdentityPartitionColumns(row, identityPartitionSchema, spec, file.partition()));
+ if (projectsIdentityPartitionColumns) {
+ //TODO: seems like we have to specify a converter to convert the partition values
Review comment:
Fixed
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdsr commented on a change in pull request #1130: Identity Partitioning: Use partitionsMap in InputFormat
Posted by GitBox <gi...@apache.org>.
rdsr commented on a change in pull request #1130:
URL: https://github.com/apache/iceberg/pull/1130#discussion_r443077759
##########
File path: mr/src/main/java/org/apache/iceberg/mr/mapreduce/IcebergInputFormat.java
##########
@@ -348,49 +342,38 @@ public void close() throws IOException {
currentIterator.close();
}
- private static Map<String, Integer> buildNameToPos(Schema expectedSchema) {
- Map<String, Integer> nameToPos = Maps.newHashMap();
- for (int pos = 0; pos < expectedSchema.asStruct().fields().size(); pos++) {
- Types.NestedField field = expectedSchema.asStruct().fields().get(pos);
- nameToPos.put(field.name(), pos);
- }
- return nameToPos;
- }
-
- private CloseableIterator<T> open(FileScanTask currentTask) {
- DataFile file = currentTask.file();
- // schema of rows returned by readers
- PartitionSpec spec = currentTask.spec();
- Set<Integer> idColumns = Sets.intersection(spec.identitySourceIds(), TypeUtil.getProjectedIds(expectedSchema));
- boolean hasJoinedPartitionColumns = !idColumns.isEmpty();
+ private CloseableIterator<T> open(FileScanTask task) {
+ PartitionSpec spec = task.spec();
+ Set<Integer> idColumns = spec.identitySourceIds();
+ Schema partitionSchema = TypeUtil.select(expectedSchema, idColumns);
+ boolean projectsIdentityPartitionColumns = !partitionSchema.columns().isEmpty();
CloseableIterable<T> iterable;
- if (hasJoinedPartitionColumns) {
- Schema readDataSchema = TypeUtil.selectNot(expectedSchema, idColumns);
- Schema identityPartitionSchema = TypeUtil.select(expectedSchema, idColumns);
- iterable = CloseableIterable.transform(open(currentTask, readDataSchema),
- row -> withIdentityPartitionColumns(row, identityPartitionSchema, spec, file.partition()));
+ if (projectsIdentityPartitionColumns) {
+ //TODO: seems like we have to specify a converter to convert the partition values
Review comment:
I was hesitant to use the converter from TableScanIterable as it seems it might only supports Avro.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdsr commented on a change in pull request #1130: Identity Partitioning: Use PartitionUtil.constantsMap in InputFormat
Posted by GitBox <gi...@apache.org>.
rdsr commented on a change in pull request #1130:
URL: https://github.com/apache/iceberg/pull/1130#discussion_r443703625
##########
File path: mr/src/main/java/org/apache/iceberg/mr/mapreduce/IcebergInputFormat.java
##########
@@ -348,49 +342,38 @@ public void close() throws IOException {
currentIterator.close();
}
- private static Map<String, Integer> buildNameToPos(Schema expectedSchema) {
- Map<String, Integer> nameToPos = Maps.newHashMap();
- for (int pos = 0; pos < expectedSchema.asStruct().fields().size(); pos++) {
- Types.NestedField field = expectedSchema.asStruct().fields().get(pos);
- nameToPos.put(field.name(), pos);
- }
- return nameToPos;
- }
-
- private CloseableIterator<T> open(FileScanTask currentTask) {
- DataFile file = currentTask.file();
- // schema of rows returned by readers
- PartitionSpec spec = currentTask.spec();
- Set<Integer> idColumns = Sets.intersection(spec.identitySourceIds(), TypeUtil.getProjectedIds(expectedSchema));
- boolean hasJoinedPartitionColumns = !idColumns.isEmpty();
+ private CloseableIterator<T> open(FileScanTask task) {
+ PartitionSpec spec = task.spec();
+ Set<Integer> idColumns = spec.identitySourceIds();
+ Schema partitionSchema = TypeUtil.select(expectedSchema, idColumns);
+ boolean projectsIdentityPartitionColumns = !partitionSchema.columns().isEmpty();
CloseableIterable<T> iterable;
- if (hasJoinedPartitionColumns) {
- Schema readDataSchema = TypeUtil.selectNot(expectedSchema, idColumns);
- Schema identityPartitionSchema = TypeUtil.select(expectedSchema, idColumns);
- iterable = CloseableIterable.transform(open(currentTask, readDataSchema),
- row -> withIdentityPartitionColumns(row, identityPartitionSchema, spec, file.partition()));
+ if (projectsIdentityPartitionColumns) {
+ //TODO: seems like we have to specify a converter to convert the partition values
Review comment:
Any thoughts on which is the right place to put the converter? Or maybe I can just make the one in TableScanIterable public and use that?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] rdblue commented on a change in pull request #1130: Identity Partitioning: Use PartitionUtil.constantsMap in InputFormat
Posted by GitBox <gi...@apache.org>.
rdblue commented on a change in pull request #1130:
URL: https://github.com/apache/iceberg/pull/1130#discussion_r443701477
##########
File path: mr/src/main/java/org/apache/iceberg/mr/mapreduce/IcebergInputFormat.java
##########
@@ -348,49 +342,38 @@ public void close() throws IOException {
currentIterator.close();
}
- private static Map<String, Integer> buildNameToPos(Schema expectedSchema) {
- Map<String, Integer> nameToPos = Maps.newHashMap();
- for (int pos = 0; pos < expectedSchema.asStruct().fields().size(); pos++) {
- Types.NestedField field = expectedSchema.asStruct().fields().get(pos);
- nameToPos.put(field.name(), pos);
- }
- return nameToPos;
- }
-
- private CloseableIterator<T> open(FileScanTask currentTask) {
- DataFile file = currentTask.file();
- // schema of rows returned by readers
- PartitionSpec spec = currentTask.spec();
- Set<Integer> idColumns = Sets.intersection(spec.identitySourceIds(), TypeUtil.getProjectedIds(expectedSchema));
- boolean hasJoinedPartitionColumns = !idColumns.isEmpty();
+ private CloseableIterator<T> open(FileScanTask task) {
+ PartitionSpec spec = task.spec();
+ Set<Integer> idColumns = spec.identitySourceIds();
+ Schema partitionSchema = TypeUtil.select(expectedSchema, idColumns);
+ boolean projectsIdentityPartitionColumns = !partitionSchema.columns().isEmpty();
CloseableIterable<T> iterable;
- if (hasJoinedPartitionColumns) {
- Schema readDataSchema = TypeUtil.selectNot(expectedSchema, idColumns);
- Schema identityPartitionSchema = TypeUtil.select(expectedSchema, idColumns);
- iterable = CloseableIterable.transform(open(currentTask, readDataSchema),
- row -> withIdentityPartitionColumns(row, identityPartitionSchema, spec, file.partition()));
+ if (projectsIdentityPartitionColumns) {
+ //TODO: seems like we have to specify a converter to convert the partition values
Review comment:
Yes, we should use the one from `TableScanIterable`. That conversion is from internal representations (e.g., long for timestamps) to Iceberg generics (`LocalDateTime` or `OffsetDateTime` for timestamps) so it is the right one to use here. When we add support for Pig or Hive objects, we would need to convert differently for those.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org