You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2020/06/09 16:27:00 UTC
[GitHub] [beam] sclukas77 opened a new pull request #11951: [BEAM-8828] Added BigQueryTableProvider WriteDisposition configuration
sclukas77 opened a new pull request #11951:
URL: https://github.com/apache/beam/pull/11951
Made it possible to set BigQueryIO's writeDisposition in a Beam SQL big query table.
R: @TheNeuralBit
------------------------
Post-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
--- | --- | --- | --- | --- | --- | --- | ---
Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/)
Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/)
Python | [](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Python2_PVR_Flink_Cron/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python35_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/)
XLang | --- | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_XVR_Spark/lastCompletedBuild/)
Pre-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
--- |Java | Python | Go | Website
--- | --- | --- | --- | ---
Non-portable | [](https://builds.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/)
Portable | --- | [](https://builds.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/) | --- | ---
See [.test-infra/jenkins/README](https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md) for trigger phrase, status and link of all Jenkins jobs.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TheNeuralBit commented on a change in pull request #11951: [BEAM-8828] Added BigQueryTableProvider WriteDisposition configuration
Posted by GitBox <gi...@apache.org>.
TheNeuralBit commented on a change in pull request #11951:
URL: https://github.com/apache/beam/pull/11951#discussion_r436984551
##########
File path: sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/bigquery/BigQueryTableProviderTest.java
##########
@@ -91,6 +94,16 @@ public void testSelectExportMethod() {
assertEquals(Method.EXPORT, sqlTable.method);
}
+ @Test
+ public void testSelectWriteDispositionMethod() {
+ Table table =
+ fakeTableWithProperties(
+ "hello", "{ " + WRITE_DISPOSITION_PROPERTY + ": " + "\"" + WriteDisposition.WRITE_APPEND.toString() + "\" }");
+ BigQueryTable sqlTable = (BigQueryTable) provider.buildBeamSqlTable(table);
+
+ assertEquals(WriteDisposition.WRITE_APPEND, sqlTable.writeDisposition);
+ }
+
Review comment:
Could you add a test like this for EMPTY and TRUNCATE as well?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TheNeuralBit commented on pull request #11951: [BEAM-8828] Added BigQueryTableProvider WriteDisposition configuration
Posted by GitBox <gi...@apache.org>.
TheNeuralBit commented on pull request #11951:
URL: https://github.com/apache/beam/pull/11951#issuecomment-641543407
Run SQL PostCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TheNeuralBit merged pull request #11951: [BEAM-8828] Added BigQueryTableProvider WriteDisposition configuration
Posted by GitBox <gi...@apache.org>.
TheNeuralBit merged pull request #11951:
URL: https://github.com/apache/beam/pull/11951
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TheNeuralBit commented on pull request #11951: [BEAM-8828] Added BigQueryTableProvider WriteDisposition configuration
Posted by GitBox <gi...@apache.org>.
TheNeuralBit commented on pull request #11951:
URL: https://github.com/apache/beam/pull/11951#issuecomment-641598227
Looks like SQL PostCommit finished successfully [including the new tests](https://builds.apache.org/job/beam_PostCommit_SQL_PR/321/testReport/org.apache.beam.sdk.extensions.sql.meta.provider.bigquery/BigQueryReadWriteIT/testSQLWriteAndRead_WithWriteDispositionAppend/), just hasn't been updated here yet. Merging now.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TheNeuralBit commented on pull request #11951: [BEAM-8828] Added BigQueryTableProvider WriteDisposition configuration
Posted by GitBox <gi...@apache.org>.
TheNeuralBit commented on pull request #11951:
URL: https://github.com/apache/beam/pull/11951#issuecomment-641540376
Run SQL Postcommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TheNeuralBit commented on pull request #11951: [BEAM-8828] Added BigQueryTableProvider WriteDisposition configuration
Posted by GitBox <gi...@apache.org>.
TheNeuralBit commented on pull request #11951:
URL: https://github.com/apache/beam/pull/11951#issuecomment-640874884
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TheNeuralBit commented on pull request #11951: [BEAM-8828] Added BigQueryTableProvider WriteDisposition configuration
Posted by GitBox <gi...@apache.org>.
TheNeuralBit commented on pull request #11951:
URL: https://github.com/apache/beam/pull/11951#issuecomment-641540296
Retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org