You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "riteshghorse (via GitHub)" <gi...@apache.org> on 2024/01/24 14:14:08 UTC
[I] [Bug]: Python PostCommit test suite is failing for BigQueryIO read tests [beam]
riteshghorse opened a new issue, #30095:
URL: https://github.com/apache/beam/issues/30095
### What happened?
PostCommits are continuously failing with failures in `apache_beam/io/gcp/bigquery_read_it_test.py`
Recent run - https://github.com/apache/beam/actions/runs/7635974244/job/20802248794
```
FAILED apache_beam/io/gcp/bigquery_read_it_test.py::ReadTests::test_native_source - apache_beam.runners.dataflow.dataflow_runner.DataflowRuntimeException: Dataflow pipeline failed. State: FAILED, Error:
Traceback (most recent call last):
File "apache_beam/runners/common.py", line 1435, in apache_beam.runners.common.DoFnRunner.process
File "apache_beam/runners/common.py", line 851, in apache_beam.runners.common.PerWindowInvoker.invoke_process
File "apache_beam/runners/common.py", line 997, in apache_beam.runners.common.PerWindowInvoker._invoke_process_per_window
File "/runner/_work/beam/beam/sdks/python/apache_beam/transforms/core.py", line 1961, in <lambda>
wrapper = lambda x, *args, **kwargs: [fn(x, *args, **kwargs)]
File "/runner/_work/beam/beam/sdks/python/apache_beam/testing/util.py", line 192, in _equal
raise BeamAssertException(msg)
apache_beam.testing.util.BeamAssertException: Failed assert: [{'number': 1, 'str': 'abc'}, {'number': 2, 'str': 'def'}, {'number': 3, 'str': '??????'}, {'number': 4, 'str': '????????????'}] == [{'number': 1, 'str': 'abc'}, {'number': 1, 'str': 'abc'}, {'number': 2, 'str': 'def'}, {'number': 2, 'str': 'def'}, {'number': 4, 'str': '????????????'}, {'number': 4, 'str': '????????????'}, {'number': 3, 'str': '??????'}, {'number': 3, 'str': '??????'}], unexpected elements [{'number': 1, 'str': 'abc'}, {'number': 2, 'str': 'def'}, {'number': 4, 'str': '????????????'}, {'number': 3, 'str': '??????'}]
```
### Issue Priority
Priority: 2 (default / most bugs should be filed as P2)
### Issue Components
- [X] Component: Python SDK
- [ ] Component: Java SDK
- [ ] Component: Go SDK
- [ ] Component: Typescript SDK
- [X] Component: IO connector
- [ ] Component: Beam YAML
- [ ] Component: Beam examples
- [ ] Component: Beam playground
- [ ] Component: Beam katas
- [ ] Component: Website
- [ ] Component: Spark Runner
- [ ] Component: Flink Runner
- [ ] Component: Samza Runner
- [ ] Component: Twister2 Runner
- [ ] Component: Hazelcast Jet Runner
- [ ] Component: Google Cloud Dataflow Runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: Python PostCommit test suite is failing for BigQueryIO read tests [beam]
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on issue #30095:
URL: https://github.com/apache/beam/issues/30095#issuecomment-1908595318
Both tests start failing on https://github.com/apache/beam/actions/runs/7590387826, at commit https://github.com/apache/beam/commits/d5a7fc92cdfcba199f817cd0bd0793b16c5e105e/sdks/python
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: Python PostCommit test suite is failing for BigQueryIO read tests [beam]
Posted by "ahmedabu98 (via GitHub)" <gi...@apache.org>.
ahmedabu98 commented on issue #30095:
URL: https://github.com/apache/beam/issues/30095#issuecomment-1908519345
Taking a look at `BigQueryFileLoadsIT.test_batch_copy_jobs_with_no_input_schema` now.
Although looking at the error coming out from `BigQueryFileLoadsIT. test_multiple_destinations_transform`, it looks like a separate issue:
```
E "error": {
E "code": 400,
E "message": "Load configuration must specify at least one source URI",
E "errors": [
E {
E "message": "Load configuration must specify at least one source URI",
E "domain": "global",
E "reason": "invalid"
E }
E ],
E "status": "INVALID_ARGUMENT"
E }
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: Python PostCommit test suite is failing for BigQueryIO read tests [beam]
Posted by "ahmedabu98 (via GitHub)" <gi...@apache.org>.
ahmedabu98 closed issue #30095: [Bug]: Python PostCommit test suite is failing for BigQueryIO read tests
URL: https://github.com/apache/beam/issues/30095
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: Python PostCommit test suite is failing for BigQueryIO read tests [beam]
Posted by "ahmedabu98 (via GitHub)" <gi...@apache.org>.
ahmedabu98 commented on issue #30095:
URL: https://github.com/apache/beam/issues/30095#issuecomment-1908931005
I'm able to fix `test_batch_copy_jobs_with_no_input_schema`
Still figuring out what `test_multiple_destinations_transform` is about..
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: Python PostCommit test suite is failing for BigQueryIO read tests [beam]
Posted by "riteshghorse (via GitHub)" <gi...@apache.org>.
riteshghorse commented on issue #30095:
URL: https://github.com/apache/beam/issues/30095#issuecomment-1908210474
CC: @Abacn
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: Python PostCommit test suite is failing for BigQueryIO read tests [beam]
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on issue #30095:
URL: https://github.com/apache/beam/issues/30095#issuecomment-1908335706
`bigquery_read_it_test.ReadTests::test_native_source` assert error with duplicate rows
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: Python PostCommit test suite is failing for BigQueryIO read tests [beam]
Posted by "ahmedabu98 (via GitHub)" <gi...@apache.org>.
ahmedabu98 commented on issue #30095:
URL: https://github.com/apache/beam/issues/30095#issuecomment-1909055836
Ahh looks like the new test changed attributes in `bigquery_file_loads.py` that apparently don't get reset after the test finishes running:
```
# reduce load job size to induce copy jobs
bqfl._DEFAULT_MAX_FILE_SIZE = 10
bqfl._MAXIMUM_LOAD_SIZE = 20
```
Chronologically, the following test that gets run is `test_multiple_destinations_transform` which gets affected by this and fails.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: Python PostCommit test suite is failing for BigQueryIO read tests [beam]
Posted by "riteshghorse (via GitHub)" <gi...@apache.org>.
riteshghorse commented on issue #30095:
URL: https://github.com/apache/beam/issues/30095#issuecomment-1908233124
<img width="1072" alt="image" src="https://github.com/apache/beam/assets/25881114/047bc1aa-fa0c-4a58-94a1-e4dfdd957df6">
is also failing with error:
```
FAILED apache_beam/io/gcp/bigquery_file_loads_test.py::BigQueryFileLoadsIT::test_batch_copy_jobs_with_no_input_schema - AssertionError:
Expected: (<[<BigqueryFullResultMatcher(Expected data is [(0,), (1,), (2,), (3,), (4,)])>, <BigqueryFullResultMatcher(Expected data is [(5,), (6,), (7,), (8,), (9,)])>]>)
but: <[<BigqueryFullResultMatcher(Expected data is [(0,), (1,), (2,), (3,), (4,)])>, <BigqueryFullResultMatcher(Expected data is [(5,), (6,), (7,), (8,), (9,)])>]> was <DataflowPipelineResult 2024-01-23_22_23_28-9024953742164757457 DONE>
```
which was recently added in #30015
CC: @ahmedabu98
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org