You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "Abacn (via GitHub)" <gi...@apache.org> on 2024/01/04 21:21:59 UTC
[PR] Allow JSON type in TableSchema for BigQuery FILE_LOAD [beam]
Abacn opened a new pull request, #29923:
URL: https://github.com/apache/beam/pull/29923
Fixes #23025 ; fixes #27448
* Warn instead of fail validation and link to documentation
* Add test for this use case
**Please** add a meaningful description for your change here
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] Mention the appropriate issue in your description (for example: `addresses #123`), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment `fixes #<ISSUE NUMBER>` instead.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://github.com/apache/beam/blob/master/CONTRIBUTING.md#make-the-reviewers-job-easier).
To check the build health, please visit [https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md](https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md)
GitHub Actions Tests Status (on master branch)
------------------------------------------------------------------------------------------------
[](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Go+tests%22+branch%3Amaster+event%3Aschedule)
See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI or the [workflows README](https://github.com/apache/beam/blob/master/.github/workflows/README.md) to see a list of phrases to trigger workflows.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Allow JSON type in TableSchema for BigQuery FILE_LOAD [beam]
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on code in PR #29923:
URL: https://github.com/apache/beam/pull/29923#discussion_r1442268989
##########
sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIOJsonIT.java:
##########
@@ -299,55 +324,37 @@ public void readAndValidateRows(BigQueryIOJsonOptions options) {
return;
}
+ final boolean unescape = options.getWriteMethod() == Write.Method.FILE_LOADS;
+
// Testing countries (straight json)
PCollection<KV<String, String>> countries =
jsonRows.apply(
"Convert countries to KV JSON Strings", ParDo.of(new CountryToKVJsonString()));
- PAssert.that(countries).satisfies(new CompareJsonStrings(getTestData("countries")));
+ PAssert.that(countries).satisfies(new CompareJsonStrings(getTestData("countries"), unescape));
// Testing stats (json in struct)
PCollection<KV<String, String>> stats =
jsonRows.apply("Convert stats to KV JSON Strings", ParDo.of(new StatsToKVJsonString()));
- PAssert.that(stats).satisfies(new CompareJsonStrings(getTestData("stats")));
+ PAssert.that(stats).satisfies(new CompareJsonStrings(getTestData("stats"), unescape));
// Testing cities (json in array of structs)
PCollection<KV<String, String>> cities =
jsonRows.apply("Convert cities to KV JSON Strings", ParDo.of(new CitiesToKVJsonString()));
- PAssert.that(cities).satisfies(new CompareJsonStrings(getTestData("cities")));
+ PAssert.that(cities).satisfies(new CompareJsonStrings(getTestData("cities"), unescape));
// Testing landmarks (json in array)
PCollection<KV<String, String>> landmarks =
jsonRows.apply(
"Convert landmarks to KV JSON Strings", ParDo.of(new LandmarksToKVJsonString()));
- PAssert.that(landmarks).satisfies(new CompareJsonStrings(getTestData("landmarks")));
+ PAssert.that(landmarks).satisfies(new CompareJsonStrings(getTestData("landmarks"), unescape));
p.run().waitUntilFinish();
}
- @Test
Review Comment:
The write tests also read from bigquery table for validation with pipelines, with exactly same pipelines of the read test. So I have consolidated these two read tests into writeandread.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Allow JSON type in TableSchema for BigQuery FILE_LOAD [beam]
Posted by "ahmedabu98 (via GitHub)" <gi...@apache.org>.
ahmedabu98 commented on code in PR #29923:
URL: https://github.com/apache/beam/pull/29923#discussion_r1445294389
##########
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java:
##########
@@ -3575,11 +3573,22 @@ private <DestinationT> WriteResult continueExpandTyped(
!getPropagateSuccessfulStorageApiWrites(),
"withPropagateSuccessfulStorageApiWrites only supported when using storage api writes.");
- // Batch load jobs currently support JSON data insertion only with CSV files
+ // Beam does not yet support Batch load jobs with Avro files
Review Comment:
Remove this comment? I think it's supported for all types now
##########
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java:
##########
@@ -3725,28 +3734,27 @@ private <DestinationT> WriteResult continueExpandTyped(
}
}
- private void validateNoJsonTypeInSchema(JsonElement schema) {
+ private boolean hasJsonTypeInSchema(JsonElement schema) {
JsonElement fields = schema.getAsJsonObject().get("fields");
if (!fields.isJsonArray() || fields.getAsJsonArray().isEmpty()) {
- return;
+ return false;
}
JsonArray fieldArray = fields.getAsJsonArray();
for (int i = 0; i < fieldArray.size(); i++) {
JsonObject field = fieldArray.get(i).getAsJsonObject();
- checkArgument(
- !field.get("type").getAsString().equals("JSON"),
- "Found JSON type in TableSchema. JSON data insertion is currently "
- + "not supported with 'FILE_LOADS' write method. This is supported with the "
- + "other write methods, however. For more information, visit: "
- + "https://cloud.google.com/bigquery/docs/reference/standard-sql/"
- + "json-data#ingest_json_data");
+ if (field.get("type").getAsString().equals("JSON")) {
+ return true;
+ }
if (field.get("type").getAsString().equals("STRUCT")) {
- validateNoJsonTypeInSchema(field);
+ if (hasJsonTypeInSchema(field)) {
+ return true;
+ }
Review Comment:
```suggestion
return hasJsonTypeInSchema(field);
```
##########
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java:
##########
@@ -3575,11 +3573,22 @@ private <DestinationT> WriteResult continueExpandTyped(
!getPropagateSuccessfulStorageApiWrites(),
"withPropagateSuccessfulStorageApiWrites only supported when using storage api writes.");
- // Batch load jobs currently support JSON data insertion only with CSV files
+ // Beam does not yet support Batch load jobs with Avro files
if (getJsonSchema() != null && getJsonSchema().isAccessible()) {
JsonElement schema = JsonParser.parseString(getJsonSchema().get());
- if (!schema.getAsJsonObject().keySet().isEmpty()) {
- validateNoJsonTypeInSchema(schema);
+ if (!schema.getAsJsonObject().keySet().isEmpty() && hasJsonTypeInSchema(schema)) {
+ if (rowWriterFactory.getOutputType() == OutputType.JsonTableRow) {
+ LOG.warn(
+ "Found JSON type in TableSchema for 'FILE_LOADS' write method. \n"
+ + "Make sure the TableSchema field is a parsed JSON to ensure the read as a "
+ + "JSON type. Otherwise it will read as a raw (escaped) string.");
Review Comment:
Does this mean the TableRow value should be a parsed JSON?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Allow JSON type in TableSchema for BigQuery FILE_LOAD [beam]
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on code in PR #29923:
URL: https://github.com/apache/beam/pull/29923#discussion_r1445323162
##########
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java:
##########
@@ -3725,28 +3734,27 @@ private <DestinationT> WriteResult continueExpandTyped(
}
}
- private void validateNoJsonTypeInSchema(JsonElement schema) {
+ private boolean hasJsonTypeInSchema(JsonElement schema) {
JsonElement fields = schema.getAsJsonObject().get("fields");
if (!fields.isJsonArray() || fields.getAsJsonArray().isEmpty()) {
- return;
+ return false;
}
JsonArray fieldArray = fields.getAsJsonArray();
for (int i = 0; i < fieldArray.size(); i++) {
JsonObject field = fieldArray.get(i).getAsJsonObject();
- checkArgument(
- !field.get("type").getAsString().equals("JSON"),
- "Found JSON type in TableSchema. JSON data insertion is currently "
- + "not supported with 'FILE_LOADS' write method. This is supported with the "
- + "other write methods, however. For more information, visit: "
- + "https://cloud.google.com/bigquery/docs/reference/standard-sql/"
- + "json-data#ingest_json_data");
+ if (field.get("type").getAsString().equals("JSON")) {
+ return true;
+ }
if (field.get("type").getAsString().equals("STRUCT")) {
- validateNoJsonTypeInSchema(field);
+ if (hasJsonTypeInSchema(field)) {
+ return true;
+ }
Review Comment:
These two are not the same. ture can short card while false does not
##########
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java:
##########
@@ -3725,28 +3734,27 @@ private <DestinationT> WriteResult continueExpandTyped(
}
}
- private void validateNoJsonTypeInSchema(JsonElement schema) {
+ private boolean hasJsonTypeInSchema(JsonElement schema) {
JsonElement fields = schema.getAsJsonObject().get("fields");
if (!fields.isJsonArray() || fields.getAsJsonArray().isEmpty()) {
- return;
+ return false;
}
JsonArray fieldArray = fields.getAsJsonArray();
for (int i = 0; i < fieldArray.size(); i++) {
JsonObject field = fieldArray.get(i).getAsJsonObject();
- checkArgument(
- !field.get("type").getAsString().equals("JSON"),
- "Found JSON type in TableSchema. JSON data insertion is currently "
- + "not supported with 'FILE_LOADS' write method. This is supported with the "
- + "other write methods, however. For more information, visit: "
- + "https://cloud.google.com/bigquery/docs/reference/standard-sql/"
- + "json-data#ingest_json_data");
+ if (field.get("type").getAsString().equals("JSON")) {
+ return true;
+ }
if (field.get("type").getAsString().equals("STRUCT")) {
- validateNoJsonTypeInSchema(field);
+ if (hasJsonTypeInSchema(field)) {
+ return true;
+ }
Review Comment:
These two are not the same. true can short card while false does not
##########
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java:
##########
@@ -3725,28 +3734,27 @@ private <DestinationT> WriteResult continueExpandTyped(
}
}
- private void validateNoJsonTypeInSchema(JsonElement schema) {
+ private boolean hasJsonTypeInSchema(JsonElement schema) {
JsonElement fields = schema.getAsJsonObject().get("fields");
if (!fields.isJsonArray() || fields.getAsJsonArray().isEmpty()) {
- return;
+ return false;
}
JsonArray fieldArray = fields.getAsJsonArray();
for (int i = 0; i < fieldArray.size(); i++) {
JsonObject field = fieldArray.get(i).getAsJsonObject();
- checkArgument(
- !field.get("type").getAsString().equals("JSON"),
- "Found JSON type in TableSchema. JSON data insertion is currently "
- + "not supported with 'FILE_LOADS' write method. This is supported with the "
- + "other write methods, however. For more information, visit: "
- + "https://cloud.google.com/bigquery/docs/reference/standard-sql/"
- + "json-data#ingest_json_data");
+ if (field.get("type").getAsString().equals("JSON")) {
+ return true;
+ }
if (field.get("type").getAsString().equals("STRUCT")) {
- validateNoJsonTypeInSchema(field);
+ if (hasJsonTypeInSchema(field)) {
+ return true;
+ }
Review Comment:
These two are not the same. true can short cut while false does not
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Allow JSON type in TableSchema for BigQuery FILE_LOAD [beam]
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn merged PR #29923:
URL: https://github.com/apache/beam/pull/29923
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Allow JSON type in TableSchema for BigQuery FILE_LOAD [beam]
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on PR #29923:
URL: https://github.com/apache/beam/pull/29923#issuecomment-1912593600
Failed test FhirIOWriteIT known flaky, not related to the change, merging for now
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Allow JSON type in TableSchema for BigQuery FILE_LOAD [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #29923:
URL: https://github.com/apache/beam/pull/29923#issuecomment-1881363567
Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Allow JSON type in TableSchema for BigQuery FILE_LOAD [beam]
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on code in PR #29923:
URL: https://github.com/apache/beam/pull/29923#discussion_r1446294522
##########
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java:
##########
@@ -3575,11 +3573,22 @@ private <DestinationT> WriteResult continueExpandTyped(
!getPropagateSuccessfulStorageApiWrites(),
"withPropagateSuccessfulStorageApiWrites only supported when using storage api writes.");
- // Batch load jobs currently support JSON data insertion only with CSV files
+ // Beam does not yet support Batch load jobs with Avro files
if (getJsonSchema() != null && getJsonSchema().isAccessible()) {
JsonElement schema = JsonParser.parseString(getJsonSchema().get());
- if (!schema.getAsJsonObject().keySet().isEmpty()) {
- validateNoJsonTypeInSchema(schema);
+ if (!schema.getAsJsonObject().keySet().isEmpty() && hasJsonTypeInSchema(schema)) {
+ if (rowWriterFactory.getOutputType() == OutputType.JsonTableRow) {
+ LOG.warn(
+ "Found JSON type in TableSchema for 'FILE_LOADS' write method. \n"
+ + "Make sure the TableSchema field is a parsed JSON to ensure the read as a "
+ + "JSON type. Otherwise it will read as a raw (escaped) string.");
Review Comment:
ah I see. I meant to TableRow value. It was a typo.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Allow JSON type in TableSchema for BigQuery FILE_LOAD [beam]
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on PR #29923:
URL: https://github.com/apache/beam/pull/29923#issuecomment-1912498836
R: @ahmedabu98 missed tracking this, PTAL
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Allow JSON type in TableSchema for BigQuery FILE_LOAD [beam]
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn closed pull request #29923: Allow JSON type in TableSchema for BigQuery FILE_LOAD
URL: https://github.com/apache/beam/pull/29923
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Allow JSON type in TableSchema for BigQuery FILE_LOAD [beam]
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on code in PR #29923:
URL: https://github.com/apache/beam/pull/29923#discussion_r1445317256
##########
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java:
##########
@@ -3575,11 +3573,22 @@ private <DestinationT> WriteResult continueExpandTyped(
!getPropagateSuccessfulStorageApiWrites(),
"withPropagateSuccessfulStorageApiWrites only supported when using storage api writes.");
- // Batch load jobs currently support JSON data insertion only with CSV files
+ // Beam does not yet support Batch load jobs with Avro files
if (getJsonSchema() != null && getJsonSchema().isAccessible()) {
JsonElement schema = JsonParser.parseString(getJsonSchema().get());
- if (!schema.getAsJsonObject().keySet().isEmpty()) {
- validateNoJsonTypeInSchema(schema);
+ if (!schema.getAsJsonObject().keySet().isEmpty() && hasJsonTypeInSchema(schema)) {
+ if (rowWriterFactory.getOutputType() == OutputType.JsonTableRow) {
+ LOG.warn(
+ "Found JSON type in TableSchema for 'FILE_LOADS' write method. \n"
+ + "Make sure the TableSchema field is a parsed JSON to ensure the read as a "
+ + "JSON type. Otherwise it will read as a raw (escaped) string.");
Review Comment:
Yes. This is documented behavior of BQ FILE_LOAD API
e.g., instead of "someColumn": "[1]", it needs to be "someColumn": [1]. Otherwise it will write as a single string (string is also valid json object)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Allow JSON type in TableSchema for BigQuery FILE_LOAD [beam]
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on code in PR #29923:
URL: https://github.com/apache/beam/pull/29923#discussion_r1445317256
##########
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java:
##########
@@ -3575,11 +3573,22 @@ private <DestinationT> WriteResult continueExpandTyped(
!getPropagateSuccessfulStorageApiWrites(),
"withPropagateSuccessfulStorageApiWrites only supported when using storage api writes.");
- // Batch load jobs currently support JSON data insertion only with CSV files
+ // Beam does not yet support Batch load jobs with Avro files
if (getJsonSchema() != null && getJsonSchema().isAccessible()) {
JsonElement schema = JsonParser.parseString(getJsonSchema().get());
- if (!schema.getAsJsonObject().keySet().isEmpty()) {
- validateNoJsonTypeInSchema(schema);
+ if (!schema.getAsJsonObject().keySet().isEmpty() && hasJsonTypeInSchema(schema)) {
+ if (rowWriterFactory.getOutputType() == OutputType.JsonTableRow) {
+ LOG.warn(
+ "Found JSON type in TableSchema for 'FILE_LOADS' write method. \n"
+ + "Make sure the TableSchema field is a parsed JSON to ensure the read as a "
+ + "JSON type. Otherwise it will read as a raw (escaped) string.");
Review Comment:
Yes. e.g., instead of "someColumn": "[1]", it needs to be "someColumn": [1]. Otherwise it will write as a single string (string is also valid json object)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Allow JSON type in TableSchema for BigQuery FILE_LOAD [beam]
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on PR #29923:
URL: https://github.com/apache/beam/pull/29923#issuecomment-1881361238
R: @ahmedabu98
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Allow JSON type in TableSchema for BigQuery FILE_LOAD [beam]
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on code in PR #29923:
URL: https://github.com/apache/beam/pull/29923#discussion_r1445317966
##########
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java:
##########
@@ -3575,11 +3573,22 @@ private <DestinationT> WriteResult continueExpandTyped(
!getPropagateSuccessfulStorageApiWrites(),
"withPropagateSuccessfulStorageApiWrites only supported when using storage api writes.");
- // Batch load jobs currently support JSON data insertion only with CSV files
+ // Beam does not yet support Batch load jobs with Avro files
Review Comment:
thanks for catching this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Allow JSON type in TableSchema for BigQuery FILE_LOAD [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #29923:
URL: https://github.com/apache/beam/pull/29923#issuecomment-1877883536
Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment `assign set of reviewers`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Allow JSON type in TableSchema for BigQuery FILE_LOAD [beam]
Posted by "ahmedabu98 (via GitHub)" <gi...@apache.org>.
ahmedabu98 commented on code in PR #29923:
URL: https://github.com/apache/beam/pull/29923#discussion_r1446285452
##########
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java:
##########
@@ -3575,11 +3573,22 @@ private <DestinationT> WriteResult continueExpandTyped(
!getPropagateSuccessfulStorageApiWrites(),
"withPropagateSuccessfulStorageApiWrites only supported when using storage api writes.");
- // Batch load jobs currently support JSON data insertion only with CSV files
+ // Beam does not yet support Batch load jobs with Avro files
if (getJsonSchema() != null && getJsonSchema().isAccessible()) {
JsonElement schema = JsonParser.parseString(getJsonSchema().get());
- if (!schema.getAsJsonObject().keySet().isEmpty()) {
- validateNoJsonTypeInSchema(schema);
+ if (!schema.getAsJsonObject().keySet().isEmpty() && hasJsonTypeInSchema(schema)) {
+ if (rowWriterFactory.getOutputType() == OutputType.JsonTableRow) {
+ LOG.warn(
+ "Found JSON type in TableSchema for 'FILE_LOADS' write method. \n"
+ + "Make sure the TableSchema field is a parsed JSON to ensure the read as a "
+ + "JSON type. Otherwise it will read as a raw (escaped) string.");
Review Comment:
I see, can we change it to `Make sure the TableRow value is a parsed JSON...`?
Can we also include a link to the documented behavior?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org