You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "riteshghorse (via GitHub)" <gi...@apache.org> on 2024/01/24 14:14:08 UTC

[I] [Bug]: Python PostCommit test suite is failing for BigQueryIO read tests [beam]

riteshghorse opened a new issue, #30095:
URL: https://github.com/apache/beam/issues/30095

   ### What happened?
   
   PostCommits are continuously failing with failures in `apache_beam/io/gcp/bigquery_read_it_test.py`
   
   Recent run - https://github.com/apache/beam/actions/runs/7635974244/job/20802248794
   
   ```
   FAILED apache_beam/io/gcp/bigquery_read_it_test.py::ReadTests::test_native_source - apache_beam.runners.dataflow.dataflow_runner.DataflowRuntimeException: Dataflow pipeline failed. State: FAILED, Error:
   Traceback (most recent call last):
     File "apache_beam/runners/common.py", line 1435, in apache_beam.runners.common.DoFnRunner.process
     File "apache_beam/runners/common.py", line 851, in apache_beam.runners.common.PerWindowInvoker.invoke_process
     File "apache_beam/runners/common.py", line 997, in apache_beam.runners.common.PerWindowInvoker._invoke_process_per_window
     File "/runner/_work/beam/beam/sdks/python/apache_beam/transforms/core.py", line 1961, in <lambda>
       wrapper = lambda x, *args, **kwargs: [fn(x, *args, **kwargs)]
     File "/runner/_work/beam/beam/sdks/python/apache_beam/testing/util.py", line 192, in _equal
       raise BeamAssertException(msg)
   apache_beam.testing.util.BeamAssertException: Failed assert: [{'number': 1, 'str': 'abc'}, {'number': 2, 'str': 'def'}, {'number': 3, 'str': '??????'}, {'number': 4, 'str': '????????????'}] == [{'number': 1, 'str': 'abc'}, {'number': 1, 'str': 'abc'}, {'number': 2, 'str': 'def'}, {'number': 2, 'str': 'def'}, {'number': 4, 'str': '????????????'}, {'number': 4, 'str': '????????????'}, {'number': 3, 'str': '??????'}, {'number': 3, 'str': '??????'}], unexpected elements [{'number': 1, 'str': 'abc'}, {'number': 2, 'str': 'def'}, {'number': 4, 'str': '????????????'}, {'number': 3, 'str': '??????'}]
   ```
   
   ### Issue Priority
   
   Priority: 2 (default / most bugs should be filed as P2)
   
   ### Issue Components
   
   - [X] Component: Python SDK
   - [ ] Component: Java SDK
   - [ ] Component: Go SDK
   - [ ] Component: Typescript SDK
   - [X] Component: IO connector
   - [ ] Component: Beam YAML
   - [ ] Component: Beam examples
   - [ ] Component: Beam playground
   - [ ] Component: Beam katas
   - [ ] Component: Website
   - [ ] Component: Spark Runner
   - [ ] Component: Flink Runner
   - [ ] Component: Samza Runner
   - [ ] Component: Twister2 Runner
   - [ ] Component: Hazelcast Jet Runner
   - [ ] Component: Google Cloud Dataflow Runner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Bug]: Python PostCommit test suite is failing for BigQueryIO read tests [beam]

Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on issue #30095:
URL: https://github.com/apache/beam/issues/30095#issuecomment-1908595318

   Both tests start failing on https://github.com/apache/beam/actions/runs/7590387826, at commit https://github.com/apache/beam/commits/d5a7fc92cdfcba199f817cd0bd0793b16c5e105e/sdks/python
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Bug]: Python PostCommit test suite is failing for BigQueryIO read tests [beam]

Posted by "ahmedabu98 (via GitHub)" <gi...@apache.org>.
ahmedabu98 commented on issue #30095:
URL: https://github.com/apache/beam/issues/30095#issuecomment-1908519345

   Taking a look at `BigQueryFileLoadsIT.test_batch_copy_jobs_with_no_input_schema` now.
   
   Although looking at the error coming out from `BigQueryFileLoadsIT. test_multiple_destinations_transform`, it looks like a separate issue:
   ```
   E               "error": {
   E                 "code": 400,
   E                 "message": "Load configuration must specify at least one source URI",
   E                 "errors": [
   E                   {
   E                     "message": "Load configuration must specify at least one source URI",
   E                     "domain": "global",
   E                     "reason": "invalid"
   E                   }
   E                 ],
   E                 "status": "INVALID_ARGUMENT"
   E               }
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Bug]: Python PostCommit test suite is failing for BigQueryIO read tests [beam]

Posted by "ahmedabu98 (via GitHub)" <gi...@apache.org>.
ahmedabu98 closed issue #30095: [Bug]: Python PostCommit test suite is failing for BigQueryIO read tests
URL: https://github.com/apache/beam/issues/30095


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Bug]: Python PostCommit test suite is failing for BigQueryIO read tests [beam]

Posted by "ahmedabu98 (via GitHub)" <gi...@apache.org>.
ahmedabu98 commented on issue #30095:
URL: https://github.com/apache/beam/issues/30095#issuecomment-1908931005

   I'm able to fix `test_batch_copy_jobs_with_no_input_schema`
   
   Still figuring out what `test_multiple_destinations_transform` is about..


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Bug]: Python PostCommit test suite is failing for BigQueryIO read tests [beam]

Posted by "riteshghorse (via GitHub)" <gi...@apache.org>.
riteshghorse commented on issue #30095:
URL: https://github.com/apache/beam/issues/30095#issuecomment-1908210474

   CC: @Abacn 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Bug]: Python PostCommit test suite is failing for BigQueryIO read tests [beam]

Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on issue #30095:
URL: https://github.com/apache/beam/issues/30095#issuecomment-1908335706

   `bigquery_read_it_test.ReadTests::test_native_source` assert error with duplicate rows


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Bug]: Python PostCommit test suite is failing for BigQueryIO read tests [beam]

Posted by "ahmedabu98 (via GitHub)" <gi...@apache.org>.
ahmedabu98 commented on issue #30095:
URL: https://github.com/apache/beam/issues/30095#issuecomment-1909055836

   Ahh looks like the new test changed attributes in `bigquery_file_loads.py` that apparently don't get reset after the test finishes running:
   ```
   # reduce load job size to induce copy jobs
       bqfl._DEFAULT_MAX_FILE_SIZE = 10
       bqfl._MAXIMUM_LOAD_SIZE = 20
   ```
   Chronologically, the following test that gets run is `test_multiple_destinations_transform` which gets affected by this and fails.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Bug]: Python PostCommit test suite is failing for BigQueryIO read tests [beam]

Posted by "riteshghorse (via GitHub)" <gi...@apache.org>.
riteshghorse commented on issue #30095:
URL: https://github.com/apache/beam/issues/30095#issuecomment-1908233124

   <img width="1072" alt="image" src="https://github.com/apache/beam/assets/25881114/047bc1aa-fa0c-4a58-94a1-e4dfdd957df6">
    is also failing with error:
   
    ```
    FAILED apache_beam/io/gcp/bigquery_file_loads_test.py::BigQueryFileLoadsIT::test_batch_copy_jobs_with_no_input_schema - AssertionError: 
   Expected: (<[<BigqueryFullResultMatcher(Expected data is [(0,), (1,), (2,), (3,), (4,)])>, <BigqueryFullResultMatcher(Expected data is [(5,), (6,), (7,), (8,), (9,)])>]>)
        but: <[<BigqueryFullResultMatcher(Expected data is [(0,), (1,), (2,), (3,), (4,)])>, <BigqueryFullResultMatcher(Expected data is [(5,), (6,), (7,), (8,), (9,)])>]> was <DataflowPipelineResult 2024-01-23_22_23_28-9024953742164757457 DONE>
    ```
   
    which was recently added in #30015
   
   CC: @ahmedabu98 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org