You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "ahmedabu98 (via GitHub)" <gi...@apache.org> on 2024/05/29 20:36:07 UTC
[PR] Cleanup TableRow of unknowns for good structs [beam]
ahmedabu98 opened a new pull request, #31447:
URL: https://github.com/apache/beam/pull/31447
When converting TableRows to dynamic messages with auto-schema update enabled, we gather any unknown fields that we encounter and store them in a TableRow. When we encounter a STRUCT field, we create a placeholder in our "unknown TableRow" in case we need to populate it with unknown nested fields.
If we don't cleanup this placeholder properly, we end up wrongfully marking the STRUCT field as unknown.
Fixes #30850
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Cleanup TableRow of unknowns for good structs [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #31447:
URL: https://github.com/apache/beam/pull/31447#issuecomment-2138304562
Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support unknown nested repeated STRUCTs [beam]
Posted by "ahmedabu98 (via GitHub)" <gi...@apache.org>.
ahmedabu98 commented on code in PR #31447:
URL: https://github.com/apache/beam/pull/31447#discussion_r1622495897
##########
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/TableRowToStorageApiProto.java:
##########
@@ -524,6 +532,15 @@ public static DynamicMessage messageFromMap(
if (value != null) {
builder.setField(fieldDescriptor, value);
}
+ // For STRUCT fields, we add a placeholder to unknownFields using the getNestedUnknown
+ // supplier (in case we encounter unknown nested fields). If the placeholder comes out
+ // to be empty, we should clean it up
+ if (fieldSchemaInformation.getType().equals(TableFieldSchema.Type.STRUCT)
Review Comment:
FYI this happens here: https://github.com/apache/beam/blob/f410b71c076dc4d7b75fc7b018bbebce8c35782c/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/TableRowToStorageApiProto.java#L1031-L1042
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support unknown repeated STRUCTs [beam]
Posted by "ahmedabu98 (via GitHub)" <gi...@apache.org>.
ahmedabu98 merged PR #31447:
URL: https://github.com/apache/beam/pull/31447
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support unknown nested repeated STRUCTs [beam]
Posted by "liferoad (via GitHub)" <gi...@apache.org>.
liferoad commented on PR #31447:
URL: https://github.com/apache/beam/pull/31447#issuecomment-2142456123
cc @fozzie15
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support unknown nested repeated STRUCTs [beam]
Posted by "ahmedabu98 (via GitHub)" <gi...@apache.org>.
ahmedabu98 commented on code in PR #31447:
URL: https://github.com/apache/beam/pull/31447#discussion_r1622495897
##########
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/TableRowToStorageApiProto.java:
##########
@@ -524,6 +532,15 @@ public static DynamicMessage messageFromMap(
if (value != null) {
builder.setField(fieldDescriptor, value);
}
+ // For STRUCT fields, we add a placeholder to unknownFields using the getNestedUnknown
+ // supplier (in case we encounter unknown nested fields). If the placeholder comes out
+ // to be empty, we should clean it up
+ if (fieldSchemaInformation.getType().equals(TableFieldSchema.Type.STRUCT)
Review Comment:
FYI this happens here: https://github.com/apache/beam/blob/f410b71c076dc4d7b75fc7b018bbebce8c35782c/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/TableRowToStorageApiProto.java#L1031-L1040
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Cleanup TableRow of unknowns for good structs [beam]
Posted by "ahmedabu98 (via GitHub)" <gi...@apache.org>.
ahmedabu98 commented on PR #31447:
URL: https://github.com/apache/beam/pull/31447#issuecomment-2138303160
R: @Abacn
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org