You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "ahmedabu98 (via GitHub)" <gi...@apache.org> on 2023/08/28 20:39:52 UTC
[GitHub] [beam] ahmedabu98 commented on a diff in pull request #28189: Add debug logs for updated schema in BigQueryIO
ahmedabu98 commented on code in PR #28189:
URL: https://github.com/apache/beam/pull/28189#discussion_r1307928791
##########
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/WriteTables.java:
##########
@@ -215,6 +216,7 @@ public void processElement(
firstPaneCreateDisposition,
dynamicDestinations,
destination);
+ LOG.debug("Use TableSchema for {}: {}", destination, tableSchema);
Review Comment:
I like "Fetching tableschema..." because it makes it clear we are getting it from BQ table, not from an input schema.
##########
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/StorageApiWriteUnshardedRecords.java:
##########
@@ -430,8 +430,13 @@ SchemaAndDescriptor getCurrentTableSchema(String stream, @Nullable TableSchema u
WriteStream writeStream =
Preconditions.checkStateNotNull(maybeDatasetService).getWriteStream(streamName);
if (writeStream != null && writeStream.hasTableSchema()) {
- currentSchema.set(writeStream.getTableSchema());
+ TableSchema updatedFromStream = writeStream.getTableSchema();
+ currentSchema.set(updatedFromStream);
updated.set(true);
+ LOG.debug(
+ "Updated currentSchema according to stream {}: {}",
+ streamName,
+ updatedFromStream);
Review Comment:
"Fetched updated schema for table {}: \n\t{}" ?
##########
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/StorageApiWriteUnshardedRecords.java:
##########
@@ -430,8 +430,13 @@ SchemaAndDescriptor getCurrentTableSchema(String stream, @Nullable TableSchema u
WriteStream writeStream =
Preconditions.checkStateNotNull(maybeDatasetService).getWriteStream(streamName);
if (writeStream != null && writeStream.hasTableSchema()) {
- currentSchema.set(writeStream.getTableSchema());
+ TableSchema updatedFromStream = writeStream.getTableSchema();
+ currentSchema.set(updatedFromStream);
updated.set(true);
+ LOG.debug(
+ "Updated currentSchema according to stream {}: {}",
+ streamName,
+ updatedFromStream);
Review Comment:
Also this is in the context of a `DestinationState`, so we can display the table (similar to what you do in StorageApiWritesShardedRecords) using `tableUrn`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org