You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2022/06/04 21:38:47 UTC
[GitHub] [beam] damccorm opened a new issue, #21114: Already Exists: Dataset apache-beam-testing:python_bq_file_loads_NNN
damccorm opened a new issue, #21114:
URL: https://github.com/apache/beam/issues/21114
It looks like there are too many collisions in the dataset name.
There don't seem to be many changes to these tests or the implementation in several months except for the [migration to nosetest to pytest](https://github.com/apache/beam/pull/14859)
We seem to be generating a dataset name based upon the [second at which the test is run with a relatively large random value appended](https://github.com/apache/beam/blob/a1abdf1d16cb6c74fd2460e90e89cb3cb7c81308/sdks/python/apache_beam/io/gcp/bigquery_file_loads_test.py#L737).
Note that this same setup code is copied in multiple locations such as [BQ Write IT](https://github.com/apache/beam/blob/a1abdf1d16cb6c74fd2460e90e89cb3cb7c81308/sdks/python/apache_beam/io/gcp/bigquery_write_it_test.py#L63).
Imported from Jira [BEAM-12766](https://issues.apache.org/jira/browse/BEAM-12766). Original Jira may contain additional context.
Reported by: lcwik.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] damccorm closed issue #21114: Already Exists: Dataset apache-beam-testing:python_bq_file_loads_NNN
Posted by GitBox <gi...@apache.org>.
damccorm closed issue #21114: Already Exists: Dataset apache-beam-testing:python_bq_file_loads_NNN
URL: https://github.com/apache/beam/issues/21114
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] damccorm commented on issue #21114: Already Exists: Dataset apache-beam-testing:python_bq_file_loads_NNN
Posted by GitBox <gi...@apache.org>.
damccorm commented on issue #21114:
URL: https://github.com/apache/beam/issues/21114#issuecomment-1281108179
It doesn't seem like this has caused a failure in the retained jenkins history, and I'm not able to get a repro even starting 26 concurrent runs (via 50 instances of calling `pytest -v apache_beam/io/gcp/bigquery_file_loads_test.py::BigQueryFileLoadsIT --test-pipeline-options='--runner=TestDataflowRunner --project=<my gcp project> --temp_location=gs://<my bucket> --sdk_location=dist/apache-beam-2.43.0.dev0.tar.gz --region=us-central1'`). More than 26 concurrent runs caused jobs to fail because my project was out of quota. I tried this twice I also don't see anything wrong with the logic itself. I'm inclined to close this and reopen if it reoccurs.
> Note that this same setup code is copied in multiple locations such as [BQ Write IT](https://github.com/apache/beam/blob/a1abdf1d16cb6c74fd2460e90e89cb3cb7c81308/sdks/python/apache_beam/io/gcp/bigquery_write_it_test.py#L63).
It is worth noting that these tests have different prefixes, so they won't actually conflict with each other. We'd just need any fix we make to hit all of them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org