[GitHub] [beam] benWize edited a comment on pull request #16154: [WIP][BEAM-12572] Run python examples on multiple runners

[GitBox <gi...@apache.org>]

benWize edited a comment on pull request #16154:
URL: https://github.com/apache/beam/pull/16154#issuecomment-988323116


   Hi @tvalentyn, I've been working in this PR to execute the examples in Direct, Dataflow, Spark, and Flink runners. 
   I added some tests to the examples that weren't running the same pipeline in the existing tests. 
   Some of the tests fail for specific runners, would you help me to review if I have something missing in the config task for each runner, or maybe If there are examples that are expected to fail for a specific runner, and I need to ignore those. 
   
   - [Dataflow](https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Dataflow/3/testReport/): Almost all examples succeed, only [this](https://github.com/apache/beam/blob/5d8190ca6d17aa14e2a6460f48debef715823aef/sdks/python/apache_beam/examples/complete/top_wikipedia_sessions_test.py#L90) example is failing, not sure If is because of how the output file is created, maybe writing to GCS would work.
   - [Spark](https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Spark/2/#showFailuresLink) here multiple tests are failing, most are integration tests that require setUp or tearDown of external resources.
   - [Flink](https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Flink/2/#showFailuresLink) same as Spark, most of the integration tests failing.
   - [Direct](https://ci-beam.apache.org/job/beam_PostCommit_Python_Examples_Direct/3/#showFailuresLink) some integration tests failing too.
   
   I think now most of the examples have a corresponding test that executes it, <del>the only one missing is https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/cookbook/filters.py#L75 because I'm not sure what BQ dataset can I use to test it. </del> (Test added in https://github.com/apache/beam/pull/16154/commits/06d43c2aebb1ebb0ed5910bbba81bfa769285045)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org