You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2020/06/23 04:17:43 UTC
[GitHub] [beam] bamnet opened a new pull request #12061: [BEAM-4634] Add options to allow BigQuery StandardSQL queries.
bamnet opened a new pull request #12061:
URL: https://github.com/apache/beam/pull/12061
Currently, the Go SDK `bigquery` is stuck in LegacySQL mode. This limits the type of queries developers can write the and types of tables they can query. To allow developers to switch to Standard SQL mode, we introduce a `QueryOptions` struct which can be passed. Future work could reuse this struct to also control validation and flattening, mirroring the Python and Java Bigquery IO packages.
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [X] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`).
- [X] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier).
Post-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
Lang | SDK | Dataflow | Flink | Samza | Spark
--- | --- | --- | --- | --- | ---
Go | [](https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/)
Java | [](https://ci-beam.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/)
Python | [](https://ci-beam.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Python2_PVR_Flink_Cron/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python35_VR_Flink/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/)
XLang | --- | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/) | --- | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Spark/lastCompletedBuild/)
Pre-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
--- |Java | Python | Go | Website
--- | --- | --- | --- | ---
Non-portable | [](https://ci-beam.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/)
Portable | --- | [](https://ci-beam.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/) | --- | ---
See [.test-infra/jenkins/README](https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md) for trigger phrase, status and link of all Jenkins jobs.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] lostluck commented on a change in pull request #12061: [BEAM-4634] Add options to allow BigQuery StandardSQL queries.
Posted by GitBox <gi...@apache.org>.
lostluck commented on a change in pull request #12061:
URL: https://github.com/apache/beam/pull/12061#discussion_r444548615
##########
File path: sdks/go/pkg/beam/io/bigqueryio/bigquery.go
##########
@@ -93,18 +93,37 @@ func Read(s beam.Scope, project, table string, t reflect.Type) beam.PCollection
return query(s, project, fmt.Sprintf("SELECT * from [%v]", table), t)
}
+// QueryOptions represents additional options for executing a query.
+type QueryOptions struct {
+ // UseStandardSQL enables BigQuery's Standard SQL dialect when executing a query.
+ UseStandardSQL bool
+}
+
+// UseStandardSQL enables BigQuery's Standard SQL dialect when executing a query.
+func UseStandardSQL(qo *QueryOptions) error {
+ qo.UseStandardSQL = true
+ return nil
+}
Review comment:
Part of the functional options is to have the user level functions return functions. This allows for closures over user configuration for more complicated options later on. Probably doesn't matter in this case, but otherwise:
eg.
```suggestion
func UseStandardSQL() func(qo *QueryOptions) error {
return func(qo *QueryOptions) error
qo.UseStandardSQL = true
return nil
}
}
```
I'd probably drop the "use" so it's just StandardSQL. When there are more of these, the repeated Use prefix will get tiresome and reduce readability.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] lostluck commented on pull request #12061: [BEAM-4634] Add options to allow BigQuery StandardSQL queries.
Posted by GitBox <gi...@apache.org>.
lostluck commented on pull request #12061:
URL: https://github.com/apache/beam/pull/12061#issuecomment-648510341
Congratulations! You're now an Apache Beam contributor.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] lostluck commented on a change in pull request #12061: [BEAM-4634] Add options to allow BigQuery StandardSQL queries.
Posted by GitBox <gi...@apache.org>.
lostluck commented on a change in pull request #12061:
URL: https://github.com/apache/beam/pull/12061#discussion_r444345197
##########
File path: sdks/go/pkg/beam/io/bigqueryio/bigquery.go
##########
@@ -90,21 +90,27 @@ func Read(s beam.Scope, project, table string, t reflect.Type) beam.PCollection
// TODO(herohde) 7/13/2017: using * is probably too inefficient. We could infer
// a focused query from the type.
- return query(s, project, fmt.Sprintf("SELECT * from [%v]", table), t)
+ return query(s, project, fmt.Sprintf("SELECT * from [%v]", table), t, nil)
+}
+
+// QueryOptions represents additional options for executing a query.
+type QueryOptions struct {
Review comment:
I'm less familiar with Big Query and it's option set, but consider switching this to a variadic list of functional options ala https://dave.cheney.net/2014/10/17/functional-options-for-friendly-apis In particular, this avoids a breaking compile change when the (admitedly few) users of this function update their beam.
It doesn't remove the need for a QueryOptions struct to serialize the options, but it does allow the callsite to not have empty options every time.
I don't feel super strongly about this as it's likely this package will likely need updating when we fix module versioning, but it's a suggestion worth discussing.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] lostluck commented on pull request #12061: [BEAM-4634] Add options to allow BigQuery StandardSQL queries.
Posted by GitBox <gi...@apache.org>.
lostluck commented on pull request #12061:
URL: https://github.com/apache/beam/pull/12061#issuecomment-653175129
I'm fine with it either way for the reasons Robert (Bradshaw) describes.
Given that standard is the default for the actual go bigquery API,
defaulting to legacy would be confusing.
The documentation for the Query method should make which dialog it's using
unambiguous.
On Thu, Jul 2, 2020, 11:50 AM Robert Bradshaw <no...@github.com>
wrote:
> (1) is not (yet) a concern for Go, but good (if unfortunate) point about
> (2). Something that could mitigate this is to make it required.
>
> On Thu, Jul 2, 2020 at 11:25 AM Brian Michalski <no...@github.com>
> wrote:
>
> > I'm very new to Beam so don't have strong thoughts here, but two things
> > jump out:
> >
> > 1. It's a breaking change.
> > 2. LegacySQL appears to be the default for Python and Java.
> >
> > I don't know enough about the project to understand if there are too many
> > users for 1 to be feasible (my gut sayz no) or if consistency with other
> > language SDKs 2 is goal or non-goal.
> >
> > —
> > You are receiving this because you commented.
> > Reply to this email directly, view it on GitHub
> > <https://github.com/apache/beam/pull/12061#issuecomment-653157515>, or
> > unsubscribe
> > <
> https://github.com/notifications/unsubscribe-auth/AADWVALZNPX55M6FWLNO3ZLRZTGKBANCNFSM4OFISS3A
> >
> > .
> >
>
> —
> You are receiving this because you modified the open/close state.
> Reply to this email directly, view it on GitHub
> <https://github.com/apache/beam/pull/12061#issuecomment-653167957>, or
> unsubscribe
> <https://github.com/notifications/unsubscribe-auth/ADKDOFONHGA7VPPDOZWCJDDRZTJGDANCNFSM4OFISS3A>
> .
>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] lostluck merged pull request #12061: [BEAM-4634] Add options to allow BigQuery StandardSQL queries.
Posted by GitBox <gi...@apache.org>.
lostluck merged pull request #12061:
URL: https://github.com/apache/beam/pull/12061
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] bamnet commented on a change in pull request #12061: [BEAM-4634] Add options to allow BigQuery StandardSQL queries.
Posted by GitBox <gi...@apache.org>.
bamnet commented on a change in pull request #12061:
URL: https://github.com/apache/beam/pull/12061#discussion_r444565916
##########
File path: sdks/go/pkg/beam/io/bigqueryio/bigquery.go
##########
@@ -93,18 +93,37 @@ func Read(s beam.Scope, project, table string, t reflect.Type) beam.PCollection
return query(s, project, fmt.Sprintf("SELECT * from [%v]", table), t)
}
+// QueryOptions represents additional options for executing a query.
+type QueryOptions struct {
+ // UseStandardSQL enables BigQuery's Standard SQL dialect when executing a query.
+ UseStandardSQL bool
+}
+
+// UseStandardSQL enables BigQuery's Standard SQL dialect when executing a query.
+func UseStandardSQL(qo *QueryOptions) error {
+ qo.UseStandardSQL = true
+ return nil
+}
Review comment:
Done. Swapped to returning a function.
I left the `Use` prefix to match the the Python and Java BigQuery IO packages and base BQ library. Other config options, like flatten results and validation, don't follow this pattern so I expect this to be a one-off (feels like legacy decision from the BigQuery API design). Happy to change tho, LMK!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] robertwb commented on pull request #12061: [BEAM-4634] Add options to allow BigQuery StandardSQL queries.
Posted by GitBox <gi...@apache.org>.
robertwb commented on pull request #12061:
URL: https://github.com/apache/beam/pull/12061#issuecomment-653150195
Given that Go is still experimental, any thoughts on switching the default
to non-legacy?
On Tue, Jun 23, 2020 at 5:31 PM Robert Burke <no...@github.com>
wrote:
> Congratulations! You're now an Apache Beam contributor.
>
> —
> You are receiving this because you are subscribed to this thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/apache/beam/pull/12061#issuecomment-648510341>, or
> unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AADWVANQ5NWWOQJWOGVIGL3RYFCMXANCNFSM4OFISS3A>
> .
>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] lostluck commented on a change in pull request #12061: [BEAM-4634] Add options to allow BigQuery StandardSQL queries.
Posted by GitBox <gi...@apache.org>.
lostluck commented on a change in pull request #12061:
URL: https://github.com/apache/beam/pull/12061#discussion_r444579505
##########
File path: sdks/go/pkg/beam/io/bigqueryio/bigquery.go
##########
@@ -93,18 +93,37 @@ func Read(s beam.Scope, project, table string, t reflect.Type) beam.PCollection
return query(s, project, fmt.Sprintf("SELECT * from [%v]", table), t)
}
+// QueryOptions represents additional options for executing a query.
+type QueryOptions struct {
+ // UseStandardSQL enables BigQuery's Standard SQL dialect when executing a query.
+ UseStandardSQL bool
+}
+
+// UseStandardSQL enables BigQuery's Standard SQL dialect when executing a query.
+func UseStandardSQL(qo *QueryOptions) error {
+ qo.UseStandardSQL = true
+ return nil
+}
Review comment:
Works for me! Consistency with the deeper BigQuery options certainly trumps local verbosity.
Thanks!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] robertwb commented on pull request #12061: [BEAM-4634] Add options to allow BigQuery StandardSQL queries.
Posted by GitBox <gi...@apache.org>.
robertwb commented on pull request #12061:
URL: https://github.com/apache/beam/pull/12061#issuecomment-653167957
(1) is not (yet) a concern for Go, but good (if unfortunate) point about
(2). Something that could mitigate this is to make it required.
On Thu, Jul 2, 2020 at 11:25 AM Brian Michalski <no...@github.com>
wrote:
> I'm very new to Beam so don't have strong thoughts here, but two things
> jump out:
>
> 1. It's a breaking change.
> 2. LegacySQL appears to be the default for Python and Java.
>
> I don't know enough about the project to understand if there are too many
> users for 1 to be feasible (my gut sayz no) or if consistency with other
> language SDKs 2 is goal or non-goal.
>
> —
> You are receiving this because you commented.
> Reply to this email directly, view it on GitHub
> <https://github.com/apache/beam/pull/12061#issuecomment-653157515>, or
> unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AADWVALZNPX55M6FWLNO3ZLRZTGKBANCNFSM4OFISS3A>
> .
>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] bamnet commented on pull request #12061: [BEAM-4634] Add options to allow BigQuery StandardSQL queries.
Posted by GitBox <gi...@apache.org>.
bamnet commented on pull request #12061:
URL: https://github.com/apache/beam/pull/12061#issuecomment-653157515
I'm very new to Beam so don't have strong thoughts here, but two things jump out:
1) It's a breaking change.
2) LegacySQL appears to be the default for Python and Java.
I don't know enough about the project to understand if there are too many users for 1 to be feasible (my gut sayz no) or if consistency with other language SDKs 2 is goal or non-goal.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] bamnet commented on pull request #12061: [BEAM-4634] Add options to allow BigQuery StandardSQL queries.
Posted by GitBox <gi...@apache.org>.
bamnet commented on pull request #12061:
URL: https://github.com/apache/beam/pull/12061#issuecomment-647897629
R: @lostluck
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] bamnet commented on a change in pull request #12061: [BEAM-4634] Add options to allow BigQuery StandardSQL queries.
Posted by GitBox <gi...@apache.org>.
bamnet commented on a change in pull request #12061:
URL: https://github.com/apache/beam/pull/12061#discussion_r444481248
##########
File path: sdks/go/pkg/beam/io/bigqueryio/bigquery.go
##########
@@ -90,21 +90,27 @@ func Read(s beam.Scope, project, table string, t reflect.Type) beam.PCollection
// TODO(herohde) 7/13/2017: using * is probably too inefficient. We could infer
// a focused query from the type.
- return query(s, project, fmt.Sprintf("SELECT * from [%v]", table), t)
+ return query(s, project, fmt.Sprintf("SELECT * from [%v]", table), t, nil)
+}
+
+// QueryOptions represents additional options for executing a query.
+type QueryOptions struct {
Review comment:
Done.
I agree, this is a cleaner design. I couldn't think of the phrase "variadic list of functional options" to save my life.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org