You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2020/07/15 02:04:31 UTC
[GitHub] [beam] robinyqiu opened a new pull request #12261: [BEAM-10490] Support read/write ZetaSQL DATE/TIME types from/to BigQuery
robinyqiu opened a new pull request #12261:
URL: https://github.com/apache/beam/pull/12261
Also fix reading BYTES from BigQuery which is also failing currently.
r: @kennknowles @chamikaramj
Post-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
Lang | SDK | Dataflow | Flink | Samza | Spark | Twister2
--- | --- | --- | --- | --- | --- | ---
Go | [](https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) | ---
Java | [](https://ci-beam.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Twister2/lastCompletedBuild/)
Python | [](https://ci-beam.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python38/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Python2_PVR_Flink_Cron/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python35_VR_Flink/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/) | ---
XLang | [](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Direct/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Spark/lastCompletedBuild/) | ---
Pre-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
--- |Java | Python | Go | Website
--- | --- | --- | --- | ---
Non-portable | [](https://ci-beam.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/)
Portable | --- | [](https://ci-beam.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/) | --- | ---
See [.test-infra/jenkins/README](https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md) for trigger phrase, status and link of all Jenkins jobs.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] chamikaramj commented on a change in pull request #12261: [BEAM-10490] Support read/write ZetaSQL DATE/TIME types from/to BigQuery
Posted by GitBox <gi...@apache.org>.
chamikaramj commented on a change in pull request #12261:
URL: https://github.com/apache/beam/pull/12261#discussion_r456658469
##########
File path: sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryUtilsTest.java
##########
@@ -410,96 +415,49 @@ public void testSubMilliPrecisionRejected() {
assertThrows(
"precision",
IllegalArgumentException.class,
- () ->
- BigQueryUtils.convertAvroFormat(
- Schema.Field.of("dummy", Schema.FieldType.DATETIME).getType(),
- 1000000001L,
- REJECT_OPTIONS));
+ () -> BigQueryUtils.convertAvroFormat(FieldType.DATETIME, 1000000001L, REJECT_OPTIONS));
}
@Test
public void testMilliPrecisionOk() {
long millis = 123456789L;
assertThat(
- BigQueryUtils.convertAvroFormat(
- Schema.Field.of("dummy", Schema.FieldType.DATETIME).getType(),
- millis * 1000,
- REJECT_OPTIONS),
+ BigQueryUtils.convertAvroFormat(FieldType.DATETIME, millis * 1000, REJECT_OPTIONS),
equalTo(new Instant(millis)));
}
@Test
public void testSubMilliPrecisionTruncated() {
long millis = 123456789L;
assertThat(
- BigQueryUtils.convertAvroFormat(
- Schema.Field.of("dummy", Schema.FieldType.DATETIME).getType(),
- millis * 1000 + 123,
- TRUNCATE_OPTIONS),
+ BigQueryUtils.convertAvroFormat(FieldType.DATETIME, millis * 1000 + 123, TRUNCATE_OPTIONS),
equalTo(new Instant(millis)));
}
@Test
- public void testSubMilliPrecisionLogicalTypeRejected() {
Review comment:
Is this test not relavent anymore ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] robinyqiu commented on pull request #12261: [BEAM-10490] Support read/write ZetaSQL DATE/TIME types from/to BigQuery
Posted by GitBox <gi...@apache.org>.
robinyqiu commented on pull request #12261:
URL: https://github.com/apache/beam/pull/12261#issuecomment-659008094
Ah, thanks for catching that. Test added.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] robinyqiu commented on a change in pull request #12261: [BEAM-10490] Support read/write ZetaSQL DATE/TIME types from/to BigQuery
Posted by GitBox <gi...@apache.org>.
robinyqiu commented on a change in pull request #12261:
URL: https://github.com/apache/beam/pull/12261#discussion_r456680548
##########
File path: sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryUtilsTest.java
##########
@@ -410,96 +415,49 @@ public void testSubMilliPrecisionRejected() {
assertThrows(
"precision",
IllegalArgumentException.class,
- () ->
- BigQueryUtils.convertAvroFormat(
- Schema.Field.of("dummy", Schema.FieldType.DATETIME).getType(),
- 1000000001L,
- REJECT_OPTIONS));
+ () -> BigQueryUtils.convertAvroFormat(FieldType.DATETIME, 1000000001L, REJECT_OPTIONS));
}
@Test
public void testMilliPrecisionOk() {
long millis = 123456789L;
assertThat(
- BigQueryUtils.convertAvroFormat(
- Schema.Field.of("dummy", Schema.FieldType.DATETIME).getType(),
- millis * 1000,
- REJECT_OPTIONS),
+ BigQueryUtils.convertAvroFormat(FieldType.DATETIME, millis * 1000, REJECT_OPTIONS),
equalTo(new Instant(millis)));
}
@Test
public void testSubMilliPrecisionTruncated() {
long millis = 123456789L;
assertThat(
- BigQueryUtils.convertAvroFormat(
- Schema.Field.of("dummy", Schema.FieldType.DATETIME).getType(),
- millis * 1000 + 123,
- TRUNCATE_OPTIONS),
+ BigQueryUtils.convertAvroFormat(FieldType.DATETIME, millis * 1000 + 123, TRUNCATE_OPTIONS),
equalTo(new Instant(millis)));
}
@Test
- public void testSubMilliPrecisionLogicalTypeRejected() {
Review comment:
Yes, it's not relavent anymore. ZetaSQL TIME type is now able to support micro-second precision.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] robinyqiu commented on pull request #12261: [BEAM-10490] Support read/write ZetaSQL DATE/TIME types from/to BigQuery
Posted by GitBox <gi...@apache.org>.
robinyqiu commented on pull request #12261:
URL: https://github.com/apache/beam/pull/12261#issuecomment-658919794
Run Java PreCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] robinyqiu merged pull request #12261: [BEAM-10490] Support read/write ZetaSQL DATE/TIME types from/to BigQuery
Posted by GitBox <gi...@apache.org>.
robinyqiu merged pull request #12261:
URL: https://github.com/apache/beam/pull/12261
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] robinyqiu commented on pull request #12261: [BEAM-10490] Support read/write ZetaSQL DATE/TIME types from/to BigQuery
Posted by GitBox <gi...@apache.org>.
robinyqiu commented on pull request #12261:
URL: https://github.com/apache/beam/pull/12261#issuecomment-660305793
Friendly ping
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] robinyqiu commented on pull request #12261: [BEAM-10490] Support read/write ZetaSQL DATE/TIME types from/to BigQuery
Posted by GitBox <gi...@apache.org>.
robinyqiu commented on pull request #12261:
URL: https://github.com/apache/beam/pull/12261#issuecomment-659048335
Run Java PreCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org