You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "Abacn (via GitHub)" <gi...@apache.org> on 2023/05/11 15:54:08 UTC

[GitHub] [beam] Abacn opened a new issue, #26673: [Failing Test]: beam_PreCommit_Python_Cron failing TestWriteBigTable.test_write_metrics

Abacn opened a new issue, #26673:
URL: https://github.com/apache/beam/issues/26673

   ### What happened?
   
   Permared since May 10.
   
   Error message:
   
   ```
   apache_beam/io/gcp/bigtableio_test.py:84: 
   _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
   apache_beam/io/gcp/bigtableio_test.py:138: in verify_write_call_metric
       self.assertTrue(
   E   AssertionError: False is not true : Did not find write call metric with status: already_exists
   ```
   
   No code change involved. Also, checkout a snapshot prior to May 10 and run this test locally:
   
   ```
   pytest -v apache_beam/io/gcp/bigtableio_test.py::TestWriteBigTable::test_write_metric
   ```
   
   it fails for the same reason.
   
   Likely due to Bigtable service change
   
   ### Issue Failure
   
   Failure: Test is flaky
   
   ### Issue Priority
   
   Priority: 2 (backlog / disabled test but we think the product is healthy)
   
   ### Issue Components
   
   - [ ] Component: Python SDK
   - [ ] Component: Java SDK
   - [ ] Component: Go SDK
   - [ ] Component: Typescript SDK
   - [ ] Component: IO connector
   - [ ] Component: Beam examples
   - [ ] Component: Beam playground
   - [ ] Component: Beam katas
   - [ ] Component: Website
   - [ ] Component: Spark Runner
   - [ ] Component: Flink Runner
   - [ ] Component: Samza Runner
   - [ ] Component: Twister2 Runner
   - [ ] Component: Hazelcast Jet Runner
   - [ ] Component: Google Cloud Dataflow Runner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] tvalentyn commented on issue #26673: [Failing Test]: beam_PreCommit_Python_Cron failing TestWriteBigTable.test_write_metrics

Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on issue #26673:
URL: https://github.com/apache/beam/issues/26673#issuecomment-1549859852

   Caused by 2.18.1


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] Abacn commented on issue #26673: [Failing Test]: beam_PreCommit_Python_Cron failing TestWriteBigTable.test_write_metrics

Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on issue #26673:
URL: https://github.com/apache/beam/issues/26673#issuecomment-1549926651

   opened #26717


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] tvalentyn commented on issue #26673: [Failing Test]: beam_PreCommit_Python_Cron failing TestWriteBigTable.test_write_metrics

Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on issue #26673:
URL: https://github.com/apache/beam/issues/26673#issuecomment-1549864274

   fwiw we should be able to `git bisect` this to a particular google-cloud-bigquery commit assuming we can install that dep from sources


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] Abacn closed issue #26673: [Failing Test]: beam_PreCommit_Python_Cron failing TestWriteBigTable.test_write_metrics

Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn closed issue #26673: [Failing Test]: beam_PreCommit_Python_Cron failing TestWriteBigTable.test_write_metrics
URL: https://github.com/apache/beam/issues/26673


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] mutianf commented on issue #26673: [Failing Test]: beam_PreCommit_Python_Cron failing TestWriteBigTable.test_write_metrics

Posted by "mutianf (via GitHub)" <gi...@apache.org>.
mutianf commented on issue #26673:
URL: https://github.com/apache/beam/issues/26673#issuecomment-1563322969

   Thanks @Abacn for looking into this. We're not sure why bigtableio.py overrides MutationsBatcher. We'll look into it and work on a fix. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] tvalentyn commented on issue #26673: [Failing Test]: beam_PreCommit_Python_Cron failing TestWriteBigTable.test_write_metrics

Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on issue #26673:
URL: https://github.com/apache/beam/issues/26673#issuecomment-1560105079

   given that we pre install this IO dep in containers, would our users be affected?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] tvalentyn commented on issue #26673: [Failing Test]: beam_PreCommit_Python_Cron failing TestWriteBigTable.test_write_metrics

Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on issue #26673:
URL: https://github.com/apache/beam/issues/26673#issuecomment-1552254341

   thanks a lot, do we need to open an issue in https://github.com/googleapis/python-bigtable/issues?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] Abacn commented on issue #26673: [Failing Test]: beam_PreCommit_Python_Cron failing TestWriteBigTable.test_write_metrics

Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on issue #26673:
URL: https://github.com/apache/beam/issues/26673#issuecomment-1553103854

   Yeah, I planned to do. Also notified bigtable maintainers. I wished to extract a minimum working example showing what previously worked now break, from the unit test, just haven't get into it


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] tvalentyn commented on issue #26673: [Failing Test]: beam_PreCommit_Python_Cron failing TestWriteBigTable.test_write_metrics

Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on issue #26673:
URL: https://github.com/apache/beam/issues/26673#issuecomment-1549856593

   looks like this error reappeared this morning.  


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] Abacn commented on issue #26673: [Failing Test]: beam_PreCommit_Python_Cron failing TestWriteBigTable.test_write_metrics

Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on issue #26673:
URL: https://github.com/apache/beam/issues/26673#issuecomment-1563388359

   Yeah, deriving from client MutationsBatcher sounds not a good idea to me either, especially in Python where class is too flexible


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] Abacn commented on issue #26673: [Failing Test]: beam_PreCommit_Python_Cron failing TestWriteBigTable.test_write_metrics

Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on issue #26673:
URL: https://github.com/apache/beam/issues/26673#issuecomment-1549909710

   @tvalentyn the cause is known due and the commit that got reverted. Reverting that PR the beam test passes. The new tag 2.18.1 is identical to 2.18.0
   
    It contains the two commit:
   
   - Revert "fix: Revert "Feat: Threaded MutationsBatcher" (#773)" (#775)
   - fix: Revert "Feat: Threaded MutationsBatcher" (#773)
   
   Possible some miscommunication


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] Abacn commented on issue #26673: [Failing Test]: beam_PreCommit_Python_Cron failing TestWriteBigTable.test_write_metrics

Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on issue #26673:
URL: https://github.com/apache/beam/issues/26673#issuecomment-1560082398

   The cause is the member MutationsBatcher.rows changed its name to MutationsBatcher._rows in https://github.com/googleapis/python-bigtable/pull/722 , so https://github.com/apache/beam/blob/7539ab3580ef0d1ba3d79283962fe953e3a1a0cd/sdks/python/apache_beam/io/gcp/bigtableio.py#L68
   is always false.
   
   In detail, Beam extends the client `MutationsBatcher` as a `_MutationsBatcher`, which overwrites its member `rows`. In newer version, this member is renamed as private (`_rows`) so iiuc current design would no longer work. Python BigtableIO is broken in google-cloud-bigtable>=2.18.0
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] Abacn commented on issue #26673: [Failing Test]: beam_PreCommit_Python_Cron failing TestWriteBigTable.test_write_metrics

Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on issue #26673:
URL: https://github.com/apache/beam/issues/26673#issuecomment-1544522862

   Found the cause: google-cloud-bigtable==2.18.0 breaks the test:
   
   To reproduce: in sdks/python run command:
   
   ```
   pip install .[gcp]
   pytest -v apache_beam/io/gcp/bigtableio_test.py::TestWriteBigTable::test_write_metrics
   apache_beam/io/gcp/bigtableio_test.py::TestWriteBigTable::test_write_metrics FAILED               [100%]
   ```
   
   ```
   pip install google-cloud-bigtable==2.17.0
   pytest -v apache_beam/io/gcp/bigtableio_test.py::TestWriteBigTable::test_write_metrics
   apache_beam/io/gcp/bigtableio_test.py::TestWriteBigTable::test_write_metrics PASSED               [100%]
   ```
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] Abacn commented on issue #26673: [Failing Test]: beam_PreCommit_Python_Cron failing TestWriteBigTable.test_write_metrics

Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on issue #26673:
URL: https://github.com/apache/beam/issues/26673#issuecomment-1544252403

   Change to P1 because this is a precommit permared


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] Abacn commented on issue #26673: [Failing Test]: beam_PreCommit_Python_Cron failing TestWriteBigTable.test_write_metrics

Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on issue #26673:
URL: https://github.com/apache/beam/issues/26673#issuecomment-1560111929

   Users using our published containers won't get affected (e.g. typical Dataflow use case)
   
   Users build custom container after May 10 will be affected (google-cloud-bigtable 2.18 added to the container)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] tvalentyn commented on issue #26673: [Failing Test]: beam_PreCommit_Python_Cron failing TestWriteBigTable.test_write_metrics

Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on issue #26673:
URL: https://github.com/apache/beam/issues/26673#issuecomment-1560106641

   in other words, does this matter at job submission or only at runtime?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] Abacn commented on issue #26673: [Failing Test]: beam_PreCommit_Python_Cron failing TestWriteBigTable.test_write_metrics

Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on issue #26673:
URL: https://github.com/apache/beam/issues/26673#issuecomment-1544561302

   With google-cloud-bigtable=2.18.0, query runner api monitoring infos returns an empty dict here:
   
   https://github.com/apache/beam/blob/2209da6c084c224462bea947c428942c2f6efcce/sdks/python/apache_beam/io/gcp/bigtableio_test.py#LL116C54-L116C54
   
   WIth google-cloud-bigtable=2.17.0, `MetricsEnvironment.process_wide_container().to_runner_api_monitoring_infos(None)` returns a dict like this:
   
   ```
   {frozenset({('SERVICE', 'BigTable'), 'beam:metric:io:api_request_count:v1', ('INSTANCE_ID', 'python-test-6aa8fa26'), ('STATUS', 'already_exists'), ('TABLE_ID', 'python-test-d77bc18a'), ('METHOD', 'google.bigtable.v2.MutateRows'), ('RESOURCE', '//bigtable.googleapis.com/projects/python-test-8b2a433b/instances/python-test-6aa8fa26/tables/python-test-d77bc18a'), ('BIGTABLE_PROJECT_ID', 'python-test-8b2a433b')}): urn: "beam:metric:io:api_request_count:v1"
   type: "beam:metrics:sum_int64:v1"
   payload: "\002"
   labels {
     key: "BIGTABLE_PROJECT_ID"
     value: "python-test-8b2a433b"
   }
   labels {
     key: "INSTANCE_ID"
     value: "python-test-6aa8fa26"
   }
   labels {
     key: "METHOD"
     value: "google.bigtable.v2.MutateRows"
   }
   labels {
     key: "RESOURCE"
     value: "//bigtable.googleapis.com/projects/python-test-8b2a433b/instances/python-test-6aa8fa26/tables/python-test-d77bc18a"
   }
   labels {
     key: "SERVICE"
     value: "BigTable"
   }
   labels {
     key: "STATUS"
     value: "already_exists"
   }
   labels {
     key: "TABLE_ID"
     value: "python-test-d77bc18a"
   }
   start_time {
     seconds: 1683832726
     nanos: 768536000
   }
   , frozenset({('SERVICE', 'BigTable'), ('STATUS', 'ok'), 'beam:metric:io:api_request_count:v1', ('INSTANCE_ID', 'python-test-6aa8fa26'), ('TABLE_ID', 'python-test-d77bc18a'), ('METHOD', 'google.bigtable.v2.MutateRows'), ('RESOURCE', '//bigtable.googleapis.com/projects/python-test-8b2a433b/instances/python-test-6aa8fa26/tables/python-test-d77bc18a'), ('BIGTABLE_PROJECT_ID', 'python-test-8b2a433b')}): urn: "beam:metric:io:api_request_count:v1"
   type: "beam:metrics:sum_int64:v1"
   payload: "\002"
   labels {
     key: "BIGTABLE_PROJECT_ID"
     value: "python-test-8b2a433b"
   }
   labels {
     key: "INSTANCE_ID"
     value: "python-test-6aa8fa26"
   }
   labels {
     key: "METHOD"
     value: "google.bigtable.v2.MutateRows"
   }
   labels {
     key: "RESOURCE"
     value: "//bigtable.googleapis.com/projects/python-test-8b2a433b/instances/python-test-6aa8fa26/tables/python-test-d77bc18a"
   }
   labels {
     key: "SERVICE"
     value: "BigTable"
   }
   labels {
     key: "STATUS"
     value: "ok"
   }
   labels {
     key: "TABLE_ID"
     value: "python-test-d77bc18a"
   }
   start_time {
     seconds: 1683832726
     nanos: 768579000
   }
   }
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] Abacn commented on issue #26673: [Failing Test]: beam_PreCommit_Python_Cron failing TestWriteBigTable.test_write_metrics

Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on issue #26673:
URL: https://github.com/apache/beam/issues/26673#issuecomment-1545754245

   mitigated by excluding the certain version. Remain open to investigate the root cause (either a client bug or intended client change that can be solved by changing the test).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] Abacn commented on issue #26673: [Failing Test]: beam_PreCommit_Python_Cron failing TestWriteBigTable.test_write_metrics

Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on issue #26673:
URL: https://github.com/apache/beam/issues/26673#issuecomment-1549878080

   For now we should pin `google-cloud-bigtable<2.18.0` for now. I just excluded it as `google-cloud-bigtable!=2.18.0`. I suggested to investigate the cause https://github.com/googleapis/python-bigtable/pull/775#issuecomment-1544791951 but appearently new release still has the same issue


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] mutianf commented on issue #26673: [Failing Test]: beam_PreCommit_Python_Cron failing TestWriteBigTable.test_write_metrics

Posted by "mutianf (via GitHub)" <gi...@apache.org>.
mutianf commented on issue #26673:
URL: https://github.com/apache/beam/issues/26673#issuecomment-1563506156

   I think the _MutationBatcher wrapper was created because of 2 reasons:
   - in batcher client implementation < 2.18.0, if some mutations failed and the RPC succeeded, batcher will not return any errors. _MutationBatcher wrapper addresses this issue so mutations don't silently fail.
   - add a callback for metrics.
   
   The first issue is fixed in the 2.18.0 client. So we shouldn't need to wrap the MutationBatcher anymore. We can get the exception count from https://github.com/googleapis/python-bigtable/blob/main/google/cloud/bigtable/batcher.py#L382 and update the metrics in finishBundle.
   
   I'll work on a fix soon! 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org