You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "stevenzwu (via GitHub)" <gi...@apache.org> on 2024/06/14 18:51:26 UTC
[PR] API: add resultSchema() method to StructTransform [iceberg]
stevenzwu opened a new pull request, #10496:
URL: https://github.com/apache/iceberg/pull/10496
This can be used to build `Comparator` for `SortKey`, which contains the transformed view of the original struct. `SortOrderComparators` only works with the original struct.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API: add resultSchema() method to StructTransform [iceberg]
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #10496:
URL: https://github.com/apache/iceberg/pull/10496#discussion_r1687162470
##########
api/src/main/java/org/apache/iceberg/StructTransform.java:
##########
@@ -125,6 +151,11 @@ public int hashCode() {
return Arrays.hashCode(transformedTuple);
}
+ /** Return the schema of transformed result (a flattened structure) */
+ public Schema resultSchema() {
Review Comment:
Do you mean `List<NestedField>` or `List<Type>`? the former still requires field ids.
For my use case (Flink sink shuffling), we want to use `StructLikeComparator` to compare the `SortKey` values. Hence need the result schema of the `SortKey/StructTransform`.
Admittedly, the field ids and names here are very meaningful. With `List<Type>`, I can construct a schema with artificial field ids and names at the caller side. so it is also workable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API: add resultSchema() method to StructTransform [iceberg]
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #10496:
URL: https://github.com/apache/iceberg/pull/10496#discussion_r1687162470
##########
api/src/main/java/org/apache/iceberg/StructTransform.java:
##########
@@ -125,6 +151,11 @@ public int hashCode() {
return Arrays.hashCode(transformedTuple);
}
+ /** Return the schema of transformed result (a flattened structure) */
+ public Schema resultSchema() {
Review Comment:
Do you mean `List<NestedField>` or `List<Type>`? the former still requires field ids.
For my use case (Flink sink shuffling), we want to use `StructLikeComparator` to compare the `SortKey` values. Hence need the result schema of the `SortKey/StructTransform`.
Admittedly, the field ids and names here aren't very meaningful. With `List<Type>`, I can construct a schema with artificial field ids and names at the caller side. so it is also workable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API: add resultSchema() method to StructTransform [iceberg]
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu closed pull request #10496: API: add resultSchema() method to StructTransform
URL: https://github.com/apache/iceberg/pull/10496
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API: add resultSchema() method to StructTransform [iceberg]
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #10496:
URL: https://github.com/apache/iceberg/pull/10496#discussion_r1687198592
##########
api/src/main/java/org/apache/iceberg/StructTransform.java:
##########
@@ -125,6 +151,11 @@ public int hashCode() {
return Arrays.hashCode(transformedTuple);
}
+ /** Return the schema of transformed result (a flattened structure) */
+ public Schema resultSchema() {
Review Comment:
> Maybe I can just extract List<Type> resultTypes from the List<FieldTransform> fieldTransforms
Yeah, this should be doable. I can construct the schema with artificial field ids and names at the caller side
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API: add resultSchema() method to StructTransform [iceberg]
Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #10496:
URL: https://github.com/apache/iceberg/pull/10496#discussion_r1687148043
##########
api/src/main/java/org/apache/iceberg/StructTransform.java:
##########
@@ -125,6 +151,11 @@ public int hashCode() {
return Arrays.hashCode(transformedTuple);
}
+ /** Return the schema of transformed result (a flattened structure) */
Review Comment:
I would just drop the (a flattened structure) note
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API: add resultSchema() method to StructTransform [iceberg]
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on PR #10496:
URL: https://github.com/apache/iceberg/pull/10496#issuecomment-2176866976
> @stevenzwu do we have a corresponding issue for this PR ? Your PR looks like a good idea to me.
@jbonofre thanks for taking a look. I don't think Github issue is required for every PR in Iceberg. Were you looking for more context than the PR description?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API: add resultSchema() method to StructTransform [iceberg]
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #10496:
URL: https://github.com/apache/iceberg/pull/10496#discussion_r1687198592
##########
api/src/main/java/org/apache/iceberg/StructTransform.java:
##########
@@ -125,6 +151,11 @@ public int hashCode() {
return Arrays.hashCode(transformedTuple);
}
+ /** Return the schema of transformed result (a flattened structure) */
+ public Schema resultSchema() {
Review Comment:
> Maybe I can just extract List<Type> resultTypes from the List<FieldTransform> fieldTransforms
Yeah, this should be doable. I can construct the schema with artificial field ids and names at the caller side. I can move the logic here inside Flink module for `SortOrder`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API: add resultSchema() method to StructTransform [iceberg]
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #10496:
URL: https://github.com/apache/iceberg/pull/10496#discussion_r1687162470
##########
api/src/main/java/org/apache/iceberg/StructTransform.java:
##########
@@ -125,6 +151,11 @@ public int hashCode() {
return Arrays.hashCode(transformedTuple);
}
+ /** Return the schema of transformed result (a flattened structure) */
+ public Schema resultSchema() {
Review Comment:
Do you mean `List<NestedField>` or `List<Type>`? the former still requires field ids.
For my use case (Flink sink shuffling), we want to use `StructLikeComparator` to compare the `SortKey` values. Hence need the result schema of the `SortKey/StructTransform`.
```
this.comparator = Comparators.forType(sortKey.resultSchema().asStruct());
```
Admittedly, the field ids and names here aren't very meaningful. With `List<Type>`, I can construct a schema with artificial field ids and names at the caller side. so it is also workable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API: add resultSchema() method to StructTransform [iceberg]
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #10496:
URL: https://github.com/apache/iceberg/pull/10496#discussion_r1687174087
##########
api/src/main/java/org/apache/iceberg/StructTransform.java:
##########
@@ -125,6 +151,11 @@ public int hashCode() {
return Arrays.hashCode(transformedTuple);
}
+ /** Return the schema of transformed result (a flattened structure) */
+ public Schema resultSchema() {
Review Comment:
following on that logic, let me think more if I really need this change. Maybe I can just extract `List<Type> resultTypes` from the `List<FieldTransform> fieldTransforms`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API: add resultSchema() method to StructTransform [iceberg]
Posted by "jbonofre (via GitHub)" <gi...@apache.org>.
jbonofre commented on PR #10496:
URL: https://github.com/apache/iceberg/pull/10496#issuecomment-2171068998
@stevenzwu do we have a corresponding issue for this PR ? Your PR looks like a good idea to me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API: add resultSchema() method to StructTransform [iceberg]
Posted by "RussellSpitzer (via GitHub)" <gi...@apache.org>.
RussellSpitzer commented on code in PR #10496:
URL: https://github.com/apache/iceberg/pull/10496#discussion_r1687151623
##########
api/src/main/java/org/apache/iceberg/StructTransform.java:
##########
@@ -125,6 +151,11 @@ public int hashCode() {
return Arrays.hashCode(transformedTuple);
}
+ /** Return the schema of transformed result (a flattened structure) */
+ public Schema resultSchema() {
Review Comment:
My only question here is why we want to produce this as a Schema? Wouldn't matching the List<Transforms> fit better here? Then we wouldn't have to create artificial fieldId's. If we mostly need Schema out of this api though that would make sense though.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API: add resultSchema() method to StructTransform [iceberg]
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on code in PR #10496:
URL: https://github.com/apache/iceberg/pull/10496#discussion_r1687198592
##########
api/src/main/java/org/apache/iceberg/StructTransform.java:
##########
@@ -125,6 +151,11 @@ public int hashCode() {
return Arrays.hashCode(transformedTuple);
}
+ /** Return the schema of transformed result (a flattened structure) */
+ public Schema resultSchema() {
Review Comment:
> Maybe I can just extract List<Type> resultTypes from the List<FieldTransform> fieldTransforms
Yeah, this should be doable. I can construct the schema with artificial field ids and names at the caller side. I can move the logic here inside Flink module for `SortOrder`.
I am going to close this PR. we can revisit it if there are more needs surfaced.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API: add resultSchema() method to StructTransform [iceberg]
Posted by "szehon-ho (via GitHub)" <gi...@apache.org>.
szehon-ho commented on code in PR #10496:
URL: https://github.com/apache/iceberg/pull/10496#discussion_r1671297464
##########
api/src/main/java/org/apache/iceberg/StructTransform.java:
##########
@@ -125,6 +148,18 @@ public int hashCode() {
return Arrays.hashCode(transformedTuple);
}
+ /**
+ * The transform result type is useful for building a comparator for the transformed struct (like
+ * {@link SortKey} using {@link Comparators#forType(Types.StructType)}.
+ *
+ * <p>
Review Comment:
Nit: do we need this new paragraph?
##########
api/src/main/java/org/apache/iceberg/StructTransform.java:
##########
@@ -125,6 +148,18 @@ public int hashCode() {
return Arrays.hashCode(transformedTuple);
}
+ /**
+ * The transform result type is useful for building a comparator for the transformed struct (like
Review Comment:
My thought is, I'm not sure we need to mention the use case for this here, as it should be generic method. The code reader can just follow the references to see the use case.
Maybe the return statement in the javadoc suffices ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] API: add resultSchema() method to StructTransform [iceberg]
Posted by "stevenzwu (via GitHub)" <gi...@apache.org>.
stevenzwu commented on PR #10496:
URL: https://github.com/apache/iceberg/pull/10496#issuecomment-2223275495
hmm. the `TestSparkDataFile` failed after this change.
```
Caused by: org.apache.iceberg.exceptions.ValidationException: Invalid schema: multiple fields for name ts: 9 and 9
at org.apache.iceberg.exceptions.ValidationException.check(ValidationException.java:49)
at org.apache.iceberg.types.IndexByName.addField(IndexByName.java:195)
at org.apache.iceberg.types.IndexByName.field(IndexByName.java:161)
at org.apache.iceberg.types.IndexByName.field(IndexByName.java:34)
at org.apache.iceberg.types.TypeUtil.visit(TypeUtil.java:611)
at org.apache.iceberg.types.TypeUtil.indexNameById(TypeUtil.java:173)
at org.apache.iceberg.Schema.lazyIdToName(Schema.java:213)
at org.apache.iceberg.Schema.<init>(Schema.java:142)
at org.apache.iceberg.Schema.<init>(Schema.java:99)
at org.apache.iceberg.Schema.<init>(Schema.java:87)
at org.apache.iceberg.Schema.<init>(Schema.java:83)
at org.apache.iceberg.StructTransform.<init>(StructTransform.java:82)
at org.apache.iceberg.PartitionKey.<init>(PartitionKey.java:37)
at org.apache.iceberg.spark.source.SparkWrite$PartitionedDataWriter.<init>(SparkWrite.java:774)
```
This test class apply two partition spec transformation (identify and hour) on the same field `ts`. Is that a legitimate use case? If it is yes, then we have a problem with this proposed change.
```
private static final PartitionSpec SPEC =
PartitionSpec.builderFor(SCHEMA)
.identity("b")
.bucket("i", 2)
.identity("l")
.identity("f")
.identity("d")
.identity("date")
.hour("ts")
.identity("ts")
.identity("tsntz")
.truncate("s", 2)
.identity("bytes")
.bucket("dec_9_0", 2)
.bucket("dec_11_2", 2)
.bucket("dec_38_10", 2)
.build();
```
cc @rdblue @aokolnychyi @RussellSpitzer @szehon-ho
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org