You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2020/04/02 16:55:21 UTC
[GitHub] [beam] kmjung opened a new pull request #11292: [BEAM-9674] Don't
specify selected fields when fetching BigQuery table size
kmjung opened a new pull request #11292: [BEAM-9674] Don't specify selected fields when fetching BigQuery table size
URL: https://github.com/apache/beam/pull/11292
Don't specify the list of selected fields when making a tables.get call to fetch the table size in the BigQueryStorageTableSource in order to avoid "selected fields too long" errors from the server. The selected field list is not used by the tables.get API to compute the table size.
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`).
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier).
Post-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
--- | --- | --- | --- | --- | --- | --- | ---
Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/)
Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/)
Python | [](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Python2_PVR_Flink_Cron/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python35_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/)
XLang | --- | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_XVR_Spark/lastCompletedBuild/)
Pre-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
--- |Java | Python | Go | Website
--- | --- | --- | --- | ---
Non-portable | [](https://builds.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/)
Portable | --- | [](https://builds.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/) | --- | ---
See [.test-infra/jenkins/README](https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md) for trigger phrase, status and link of all Jenkins jobs.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] aaltay commented on issue #11292: [BEAM-9674] Don't specify
selected fields when fetching BigQuery table size
Posted by GitBox <gi...@apache.org>.
aaltay commented on issue #11292: [BEAM-9674] Don't specify selected fields when fetching BigQuery table size
URL: https://github.com/apache/beam/pull/11292#issuecomment-611822518
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] kmjung commented on issue #11292: [BEAM-9674] Don't specify
selected fields when fetching BigQuery table size
Posted by GitBox <gi...@apache.org>.
kmjung commented on issue #11292: [BEAM-9674] Don't specify selected fields when fetching BigQuery table size
URL: https://github.com/apache/beam/pull/11292#issuecomment-607966846
R: @chamikaramj
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] kmjung commented on a change in pull request #11292:
[BEAM-9674] Don't specify selected fields when fetching BigQuery table size
Posted by GitBox <gi...@apache.org>.
kmjung commented on a change in pull request #11292: [BEAM-9674] Don't specify selected fields when fetching BigQuery table size
URL: https://github.com/apache/beam/pull/11292#discussion_r409118981
##########
File path: sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIOStorageReadTest.java
##########
@@ -1368,19 +1368,14 @@ public void testStreamSourceSplitAtFractionFailsWhenParentIsPastSplitPoint() thr
public void testReadFromBigQueryIO() throws Exception {
fakeDatasetService.createDataset("foo.com:project", "dataset", "", "", null);
TableReference tableRef = BigQueryHelpers.parseTableSpec("foo.com:project:dataset.table");
-
- Table table =
- new Table().setTableReference(tableRef).setNumBytes(10L).setSchema(new TableSchema());
-
+ Table table = new Table().setTableReference(tableRef).setNumBytes(10L).setSchema(TABLE_SCHEMA);
fakeDatasetService.createTable(table);
CreateReadSessionRequest expectedCreateReadSessionRequest =
CreateReadSessionRequest.newBuilder()
.setParent("projects/project-id")
.setTableReference(BigQueryHelpers.toTableRefProto(tableRef))
.setRequestedStreams(10)
- .setReadOptions(
Review comment:
I'm not sure that I understand your question here. We used to have e2e tests for BigQueryIO with the read API only for the case where the caller specified all fields as selected fields; with this change, now we have two tests -- one which covers the case where no selected fields are covered (which is effectively the same as the case where all fields are specified), and another which covers the case where only a subset of fields are specified.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] chamikaramj commented on a change in pull request #11292:
[BEAM-9674] Don't specify selected fields when fetching BigQuery table size
Posted by GitBox <gi...@apache.org>.
chamikaramj commented on a change in pull request #11292: [BEAM-9674] Don't specify selected fields when fetching BigQuery table size
URL: https://github.com/apache/beam/pull/11292#discussion_r409111886
##########
File path: sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIOStorageReadTest.java
##########
@@ -1368,19 +1368,14 @@ public void testStreamSourceSplitAtFractionFailsWhenParentIsPastSplitPoint() thr
public void testReadFromBigQueryIO() throws Exception {
fakeDatasetService.createDataset("foo.com:project", "dataset", "", "", null);
TableReference tableRef = BigQueryHelpers.parseTableSpec("foo.com:project:dataset.table");
-
- Table table =
- new Table().setTableReference(tableRef).setNumBytes(10L).setSchema(new TableSchema());
-
+ Table table = new Table().setTableReference(tableRef).setNumBytes(10L).setSchema(TABLE_SCHEMA);
fakeDatasetService.createTable(table);
CreateReadSessionRequest expectedCreateReadSessionRequest =
CreateReadSessionRequest.newBuilder()
.setParent("projects/project-id")
.setTableReference(BigQueryHelpers.toTableRefProto(tableRef))
.setRequestedStreams(10)
- .setReadOptions(
Review comment:
Is this a feature regression for the storage API based read path ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] aaltay commented on issue #11292: [BEAM-9674] Don't specify
selected fields when fetching BigQuery table size
Posted by GitBox <gi...@apache.org>.
aaltay commented on issue #11292: [BEAM-9674] Don't specify selected fields when fetching BigQuery table size
URL: https://github.com/apache/beam/pull/11292#issuecomment-612264607
Run Java PostCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] kmjung commented on a change in pull request #11292:
[BEAM-9674] Don't specify selected fields when fetching BigQuery table size
Posted by GitBox <gi...@apache.org>.
kmjung commented on a change in pull request #11292: [BEAM-9674] Don't specify selected fields when fetching BigQuery table size
URL: https://github.com/apache/beam/pull/11292#discussion_r405002376
##########
File path: sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryServices.java
##########
@@ -101,10 +101,6 @@ JobStatistics dryRunQuery(String projectId, JobConfigurationQuery queryConfig, S
@Nullable
Table getTable(TableReference tableRef) throws InterruptedException, IOException;
- @Nullable
Review comment:
Done.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] kmjung commented on a change in pull request #11292:
[BEAM-9674] Don't specify selected fields when fetching BigQuery table size
Posted by GitBox <gi...@apache.org>.
kmjung commented on a change in pull request #11292: [BEAM-9674] Don't specify selected fields when fetching BigQuery table size
URL: https://github.com/apache/beam/pull/11292#discussion_r409117833
##########
File path: sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryStorageSourceBase.java
##########
@@ -149,11 +151,14 @@
return ImmutableList.of();
}
+ Schema sessionSchema = new Schema.Parser().parse(readSession.getAvroSchema().getSchema());
Review comment:
With this change, we're no longer specifying the list of selected fields to the tables.get call from which the BigQuery schema is taken; as a result, we get the entire table schema back, so we have to trim it on the client side in the case where the client has specified selected fields. The Avro schema is returned as part of the read session and contains only the selected fields, so we use it as the basis for trimming the BigQuery schema.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] kmjung commented on a change in pull request #11292:
[BEAM-9674] Don't specify selected fields when fetching BigQuery table size
Posted by GitBox <gi...@apache.org>.
kmjung commented on a change in pull request #11292: [BEAM-9674] Don't specify selected fields when fetching BigQuery table size
URL: https://github.com/apache/beam/pull/11292#discussion_r404996088
##########
File path: sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryServices.java
##########
@@ -101,10 +101,6 @@ JobStatistics dryRunQuery(String projectId, JobConfigurationQuery queryConfig, S
@Nullable
Table getTable(TableReference tableRef) throws InterruptedException, IOException;
- @Nullable
Review comment:
BigQueryServices is a public interface, although it's meant as a private implementation for BigQuery sources and sinks and is not surfaced at the BigQueryIO level. I can restore this functionality and simply stop calling it from BigQueryStorageStreamSource.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] aaltay commented on issue #11292: [BEAM-9674] Don't specify
selected fields when fetching BigQuery table size
Posted by GitBox <gi...@apache.org>.
aaltay commented on issue #11292: [BEAM-9674] Don't specify selected fields when fetching BigQuery table size
URL: https://github.com/apache/beam/pull/11292#issuecomment-612128691
> @aaltay should this have triggered a re-test by Jenkins? I don't see the results, if so.
It should have. Sometimes it does not. I retried again.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] aaltay commented on issue #11292: [BEAM-9674] Don't specify
selected fields when fetching BigQuery table size
Posted by GitBox <gi...@apache.org>.
aaltay commented on issue #11292: [BEAM-9674] Don't specify selected fields when fetching BigQuery table size
URL: https://github.com/apache/beam/pull/11292#issuecomment-611822669
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] chamikaramj commented on issue #11292: [BEAM-9674] Don't
specify selected fields when fetching BigQuery table size
Posted by GitBox <gi...@apache.org>.
chamikaramj commented on issue #11292: [BEAM-9674] Don't specify selected fields when fetching BigQuery table size
URL: https://github.com/apache/beam/pull/11292#issuecomment-610652147
LGTM. Thanks.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] kmjung commented on issue #11292: [BEAM-9674] Don't specify
selected fields when fetching BigQuery table size
Posted by GitBox <gi...@apache.org>.
kmjung commented on issue #11292: [BEAM-9674] Don't specify selected fields when fetching BigQuery table size
URL: https://github.com/apache/beam/pull/11292#issuecomment-612139263
Run Dataflow ValidatesRunner
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] chamikaramj commented on issue #11292: [BEAM-9674] Don't
specify selected fields when fetching BigQuery table size
Posted by GitBox <gi...@apache.org>.
chamikaramj commented on issue #11292: [BEAM-9674] Don't specify selected fields when fetching BigQuery table size
URL: https://github.com/apache/beam/pull/11292#issuecomment-610652251
Run Dataflow ValidatesRunner
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] kmjung commented on issue #11292: [BEAM-9674] Don't specify
selected fields when fetching BigQuery table size
Posted by GitBox <gi...@apache.org>.
kmjung commented on issue #11292: [BEAM-9674] Don't specify selected fields when fetching BigQuery table size
URL: https://github.com/apache/beam/pull/11292#issuecomment-611136168
Run Dataflow ValidatesRunner
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] aaltay commented on issue #11292: [BEAM-9674] Don't specify
selected fields when fetching BigQuery table size
Posted by GitBox <gi...@apache.org>.
aaltay commented on issue #11292: [BEAM-9674] Don't specify selected fields when fetching BigQuery table size
URL: https://github.com/apache/beam/pull/11292#issuecomment-612264643
Run Dataflow ValidatesRunner
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] chamikaramj commented on a change in pull request #11292:
[BEAM-9674] Don't specify selected fields when fetching BigQuery table size
Posted by GitBox <gi...@apache.org>.
chamikaramj commented on a change in pull request #11292: [BEAM-9674] Don't specify selected fields when fetching BigQuery table size
URL: https://github.com/apache/beam/pull/11292#discussion_r409107026
##########
File path: sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryStorageSourceBase.java
##########
@@ -149,11 +151,14 @@
return ImmutableList.of();
}
+ Schema sessionSchema = new Schema.Parser().parse(readSession.getAvroSchema().getSchema());
Review comment:
Not sure how this trims the schema ? Does readSession.getAvroSchema() somehow has a smaller number of fields than targetTable.getSchema() ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] chamikaramj merged pull request #11292: [BEAM-9674] Don't
specify selected fields when fetching BigQuery table size
Posted by GitBox <gi...@apache.org>.
chamikaramj merged pull request #11292: [BEAM-9674] Don't specify selected fields when fetching BigQuery table size
URL: https://github.com/apache/beam/pull/11292
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] kmjung commented on issue #11292: [BEAM-9674] Don't specify
selected fields when fetching BigQuery table size
Posted by GitBox <gi...@apache.org>.
kmjung commented on issue #11292: [BEAM-9674] Don't specify selected fields when fetching BigQuery table size
URL: https://github.com/apache/beam/pull/11292#issuecomment-611136068
Run Java PostCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] kmjung commented on issue #11292: [BEAM-9674] Don't specify
selected fields when fetching BigQuery table size
Posted by GitBox <gi...@apache.org>.
kmjung commented on issue #11292: [BEAM-9674] Don't specify selected fields when fetching BigQuery table size
URL: https://github.com/apache/beam/pull/11292#issuecomment-612096582
@aaltay should this have triggered a re-test by Jenkins? I don't see the results, if so.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] kmjung commented on issue #11292: [BEAM-9674] Don't specify
selected fields when fetching BigQuery table size
Posted by GitBox <gi...@apache.org>.
kmjung commented on issue #11292: [BEAM-9674] Don't specify selected fields when fetching BigQuery table size
URL: https://github.com/apache/beam/pull/11292#issuecomment-612131076
Thanks!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] aaltay commented on issue #11292: [BEAM-9674] Don't specify
selected fields when fetching BigQuery table size
Posted by GitBox <gi...@apache.org>.
aaltay commented on issue #11292: [BEAM-9674] Don't specify selected fields when fetching BigQuery table size
URL: https://github.com/apache/beam/pull/11292#issuecomment-612264998
Run Java PostCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] chamikaramj commented on issue #11292: [BEAM-9674] Don't
specify selected fields when fetching BigQuery table size
Posted by GitBox <gi...@apache.org>.
chamikaramj commented on issue #11292: [BEAM-9674] Don't specify selected fields when fetching BigQuery table size
URL: https://github.com/apache/beam/pull/11292#issuecomment-610652203
Run Java PostCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] aaltay commented on a change in pull request #11292:
[BEAM-9674] Don't specify selected fields when fetching BigQuery table size
Posted by GitBox <gi...@apache.org>.
aaltay commented on a change in pull request #11292: [BEAM-9674] Don't specify selected fields when fetching BigQuery table size
URL: https://github.com/apache/beam/pull/11292#discussion_r404468600
##########
File path: sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryServices.java
##########
@@ -101,10 +101,6 @@ JobStatistics dryRunQuery(String projectId, JobConfigurationQuery queryConfig, S
@Nullable
Table getTable(TableReference tableRef) throws InterruptedException, IOException;
- @Nullable
Review comment:
Is this removing a public API?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] aaltay commented on a change in pull request #11292:
[BEAM-9674] Don't specify selected fields when fetching BigQuery table size
Posted by GitBox <gi...@apache.org>.
aaltay commented on a change in pull request #11292: [BEAM-9674] Don't specify selected fields when fetching BigQuery table size
URL: https://github.com/apache/beam/pull/11292#discussion_r404468600
##########
File path: sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryServices.java
##########
@@ -101,10 +101,6 @@ JobStatistics dryRunQuery(String projectId, JobConfigurationQuery queryConfig, S
@Nullable
Table getTable(TableReference tableRef) throws InterruptedException, IOException;
- @Nullable
Review comment:
Is this deleted a public API?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] kmjung commented on issue #11292: [BEAM-9674] Don't specify
selected fields when fetching BigQuery table size
Posted by GitBox <gi...@apache.org>.
kmjung commented on issue #11292: [BEAM-9674] Don't specify selected fields when fetching BigQuery table size
URL: https://github.com/apache/beam/pull/11292#issuecomment-610758885
I see the issue that's being exposed by the test failures -- the TableRow converter code assumes that all the fields in the BigQuery table schema are also present in the Avro records, which isn't the case here. We'll need to trim the BigQuery schema on the client side after fetching it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] chamikaramj commented on issue #11292: [BEAM-9674] Don't
specify selected fields when fetching BigQuery table size
Posted by GitBox <gi...@apache.org>.
chamikaramj commented on issue #11292: [BEAM-9674] Don't specify selected fields when fetching BigQuery table size
URL: https://github.com/apache/beam/pull/11292#issuecomment-615280319
Thanks for the clarifications. LGTM.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] kmjung commented on issue #11292: [BEAM-9674] Don't specify
selected fields when fetching BigQuery table size
Posted by GitBox <gi...@apache.org>.
kmjung commented on issue #11292: [BEAM-9674] Don't specify selected fields when fetching BigQuery table size
URL: https://github.com/apache/beam/pull/11292#issuecomment-612139114
Run Java PostCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] aaltay commented on issue #11292: [BEAM-9674] Don't specify
selected fields when fetching BigQuery table size
Posted by GitBox <gi...@apache.org>.
aaltay commented on issue #11292: [BEAM-9674] Don't specify selected fields when fetching BigQuery table size
URL: https://github.com/apache/beam/pull/11292#issuecomment-612128471
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services