You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2020/03/26 22:44:51 UTC
[GitHub] [beam] udim opened a new pull request #11241: [BEAM-5422] Document
DynamicDestinations.getTable uniqueness requirement
udim opened a new pull request #11241: [BEAM-5422] Document DynamicDestinations.getTable uniqueness requirement
URL: https://github.com/apache/beam/pull/11241
Load job IDs are keyed by table (among other things), but not the
destination. Thus multiple DestinationTs mapping to the same table will
have the same BQ job ID. The first one will succeed, and the rest will
seem to Beam as retries (no-ops because the job has already
started/completed).
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`).
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier).
Post-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
--- | --- | --- | --- | --- | --- | --- | ---
Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/)
Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/)
Python | [](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Python2_PVR_Flink_Cron/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python35_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/)
XLang | --- | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_XVR_Spark/lastCompletedBuild/)
Pre-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
--- |Java | Python | Go | Website
--- | --- | --- | --- | ---
Non-portable | [](https://builds.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/)
Portable | --- | [](https://builds.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/) | --- | ---
See [.test-infra/jenkins/README](https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md) for trigger phrase, status and link of all Jenkins jobs.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] udim merged pull request #11241: [BEAM-5422] Document
DynamicDestinations.getTable uniqueness requirement
Posted by GitBox <gi...@apache.org>.
udim merged pull request #11241: [BEAM-5422] Document DynamicDestinations.getTable uniqueness requirement
URL: https://github.com/apache/beam/pull/11241
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] pabloem commented on a change in pull request #11241:
[BEAM-5422] Document DynamicDestinations.getTable uniqueness requirement
Posted by GitBox <gi...@apache.org>.
pabloem commented on a change in pull request #11241: [BEAM-5422] Document DynamicDestinations.getTable uniqueness requirement
URL: https://github.com/apache/beam/pull/11241#discussion_r405198287
##########
File path: sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/DynamicDestinations.java
##########
@@ -142,7 +142,11 @@ void setSideInputAccessorFromProcessContext(DoFn<?, ?>.ProcessContext context) {
return null;
}
- /** Returns a {@link TableDestination} object for the destination. May not return null. */
+ /**
+ * Returns a {@link TableDestination} object for the destination. May not return null. Return
+ * value needs to be unique to each destination: may not return the same {@link TableDestination}
+ * for different destinations.
Review comment:
Hm I do not think this is necessary in Python. In Python, we expect table names to be returned as the destination, so this case may not occur...
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] udim commented on issue #11241: [BEAM-5422] Document
DynamicDestinations.getTable uniqueness requirement
Posted by GitBox <gi...@apache.org>.
udim commented on issue #11241: [BEAM-5422] Document DynamicDestinations.getTable uniqueness requirement
URL: https://github.com/apache/beam/pull/11241#issuecomment-605318740
Run Java PreCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] chamikaramj commented on issue #11241: [BEAM-5422] Document
DynamicDestinations.getTable uniqueness requirement
Posted by GitBox <gi...@apache.org>.
chamikaramj commented on issue #11241: [BEAM-5422] Document DynamicDestinations.getTable uniqueness requirement
URL: https://github.com/apache/beam/pull/11241#issuecomment-608157058
cc: @jkff @reuvenlax
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] chamikaramj commented on a change in pull request #11241:
[BEAM-5422] Document DynamicDestinations.getTable uniqueness requirement
Posted by GitBox <gi...@apache.org>.
chamikaramj commented on a change in pull request #11241: [BEAM-5422] Document DynamicDestinations.getTable uniqueness requirement
URL: https://github.com/apache/beam/pull/11241#discussion_r402664669
##########
File path: sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/DynamicDestinations.java
##########
@@ -142,7 +142,11 @@ void setSideInputAccessorFromProcessContext(DoFn<?, ?>.ProcessContext context) {
return null;
}
- /** Returns a {@link TableDestination} object for the destination. May not return null. */
+ /**
+ * Returns a {@link TableDestination} object for the destination. May not return null. Return
+ * value needs to be unique to each destination: may not return the same {@link TableDestination}
+ * for different destinations.
Review comment:
Should we update Python SDK as well ?
cc: @pabloem
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] udim commented on issue #11241: [BEAM-5422] Document
DynamicDestinations.getTable uniqueness requirement
Posted by GitBox <gi...@apache.org>.
udim commented on issue #11241: [BEAM-5422] Document DynamicDestinations.getTable uniqueness requirement
URL: https://github.com/apache/beam/pull/11241#issuecomment-606274211
Run Java PreCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] udim commented on issue #11241: [BEAM-5422] Document
DynamicDestinations.getTable uniqueness requirement
Posted by GitBox <gi...@apache.org>.
udim commented on issue #11241: [BEAM-5422] Document DynamicDestinations.getTable uniqueness requirement
URL: https://github.com/apache/beam/pull/11241#issuecomment-604725835
R: @chamikaramj
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] udim commented on a change in pull request #11241:
[BEAM-5422] Document DynamicDestinations.getTable uniqueness requirement
Posted by GitBox <gi...@apache.org>.
udim commented on a change in pull request #11241: [BEAM-5422] Document DynamicDestinations.getTable uniqueness requirement
URL: https://github.com/apache/beam/pull/11241#discussion_r405231575
##########
File path: sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/DynamicDestinations.java
##########
@@ -142,7 +142,11 @@ void setSideInputAccessorFromProcessContext(DoFn<?, ?>.ProcessContext context) {
return null;
}
- /** Returns a {@link TableDestination} object for the destination. May not return null. */
+ /**
+ * Returns a {@link TableDestination} object for the destination. May not return null. Return
+ * value needs to be unique to each destination: may not return the same {@link TableDestination}
+ * for different destinations.
Review comment:
TLDR: Pablo is right.
In Python SDK, a user function translates an element to a TableReference.
In Java SDK, a user DynamicDestinations instance translates an element to a DestinationT, and then to a TableDestination.
Java does a reshuffle on (DestinationT, element) pairs, while Python does it on (TableReference, element) pairs.
(Not sure why Java uses an intermediate DestinationT. Convenience? Better GBK performance? Lower resource use?)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services