You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2020/08/21 17:48:00 UTC
[GitHub] [beam] robinyqiu opened a new pull request #12663: [BEAM-10597] Propagate BigQuery streaming insert throttled time to Dataflow worker in Python SDK
robinyqiu opened a new pull request #12663:
URL: https://github.com/apache/beam/pull/12663
Java side change is in https://github.com/apache/beam/pull/12403
r: @chamikaramj @pabloem
Post-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
Lang | SDK | Dataflow | Flink | Samza | Spark | Twister2
--- | --- | --- | --- | --- | --- | ---
Go | [](https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) | ---
Java | [](https://ci-beam.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Twister2/lastCompletedBuild/)
Python | [](https://ci-beam.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python38/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Python2_PVR_Flink_Cron/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python35_VR_Flink/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/) | ---
XLang | [](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Direct/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Spark/lastCompletedBuild/) | ---
Pre-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
--- |Java | Python | Go | Website
--- | --- | --- | --- | ---
Non-portable | [](https://ci-beam.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PreCommit_PythonDocker_Cron/lastCompletedBuild/) <br>[](https://ci-beam.apache.org/job/beam_PreCommit_PythonDocs_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/)
Portable | --- | [](https://ci-beam.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/) | --- | ---
See [.test-infra/jenkins/README](https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md) for trigger phrase, status and link of all Jenkins jobs.
GitHub Actions Tests Status (on master branch)
------------------------------------------------------------------------------------------------
See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem merged pull request #12663: [BEAM-10597] Propagate BigQuery streaming insert throttled time to Dataflow worker in Python SDK
Posted by GitBox <gi...@apache.org>.
pabloem merged pull request #12663:
URL: https://github.com/apache/beam/pull/12663
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] robinyqiu commented on a change in pull request #12663: [BEAM-10597] Propagate BigQuery streaming insert throttled time to Dataflow worker in Python SDK
Posted by GitBox <gi...@apache.org>.
robinyqiu commented on a change in pull request #12663:
URL: https://github.com/apache/beam/pull/12663#discussion_r474842486
##########
File path: sdks/python/apache_beam/io/gcp/bigquery.py
##########
@@ -1257,6 +1260,7 @@ def _flush_batch(self, destination):
_LOGGER.info(
'Sleeping %s seconds before retrying insertion.', retry_backoff)
time.sleep(retry_backoff)
+ self._throttled_secs.inc(retry_backoff)
Review comment:
In Python, BigQuery does not insert rows in parallel threads and retry within each of them, instead it backs off on all retry-able inserts together.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] rezarokni commented on a change in pull request #12663: [BEAM-10597] Propagate BigQuery streaming insert throttled time to Dataflow worker in Python SDK
Posted by GitBox <gi...@apache.org>.
rezarokni commented on a change in pull request #12663:
URL: https://github.com/apache/beam/pull/12663#discussion_r475953736
##########
File path: sdks/python/apache_beam/io/gcp/bigquery.py
##########
@@ -1257,6 +1260,7 @@ def _flush_batch(self, destination):
_LOGGER.info(
'Sleeping %s seconds before retrying insertion.', retry_backoff)
time.sleep(retry_backoff)
+ self._throttled_secs.inc(retry_backoff)
Review comment:
@pabloem Sorry missed this one...
Yes agree with Metrics for this type of value, as the ever increasing number will just chart as a liner value without the ability to usefully capture the changes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on a change in pull request #12663: [BEAM-10597] Propagate BigQuery streaming insert throttled time to Dataflow worker in Python SDK
Posted by GitBox <gi...@apache.org>.
pabloem commented on a change in pull request #12663:
URL: https://github.com/apache/beam/pull/12663#discussion_r474913739
##########
File path: sdks/python/apache_beam/io/gcp/bigquery.py
##########
@@ -1257,6 +1260,7 @@ def _flush_batch(self, destination):
_LOGGER.info(
'Sleeping %s seconds before retrying insertion.', retry_backoff)
time.sleep(retry_backoff)
+ self._throttled_secs.inc(retry_backoff)
Review comment:
I wonder if we should make this a `Metrics.distribution` type metric? @rezarokni do you have an opinion about this? Since the metric may be exported to monitoring systems, a counter may just show a 'sometimes increasing' chart vs a distribution showing a rate of increase?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] robinyqiu commented on a change in pull request #12663: [BEAM-10597] Propagate BigQuery streaming insert throttled time to Dataflow worker in Python SDK
Posted by GitBox <gi...@apache.org>.
robinyqiu commented on a change in pull request #12663:
URL: https://github.com/apache/beam/pull/12663#discussion_r474929840
##########
File path: sdks/python/apache_beam/io/gcp/bigquery.py
##########
@@ -1257,6 +1260,7 @@ def _flush_batch(self, destination):
_LOGGER.info(
'Sleeping %s seconds before retrying insertion.', retry_backoff)
time.sleep(retry_backoff)
+ self._throttled_secs.inc(retry_backoff)
Review comment:
I think we can consider that as an improvement. Currently the metric is reported to autoscalar, so I simply made it a `counter`, similar to the ones in [GCS](https://github.com/apache/beam/blob/3d0f7dc011bb7bbee4eea9525b82f53855de85f1/sdks/python/apache_beam/io/gcp/gcsio_overrides.py#L37) and [DataStore](https://github.com/apache/beam/blob/master/sdks/python/apache_beam/io/gcp/datastore/v1new/datastoreio.py#L316)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] robinyqiu commented on pull request #12663: [BEAM-10597] Propagate BigQuery streaming insert throttled time to Dataflow worker in Python SDK
Posted by GitBox <gi...@apache.org>.
robinyqiu commented on pull request #12663:
URL: https://github.com/apache/beam/pull/12663#issuecomment-678480397
Run Python PreCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on pull request #12663: [BEAM-10597] Propagate BigQuery streaming insert throttled time to Dataflow worker in Python SDK
Posted by GitBox <gi...@apache.org>.
pabloem commented on pull request #12663:
URL: https://github.com/apache/beam/pull/12663#issuecomment-679393188
@rezarokni can you still share your thoughts about making this a distribution?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org