You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2020/04/29 16:17:32 UTC
[GitHub] [beam] TheNeuralBit opened a new pull request #11571: [BEAM-2128] Remove PipelineRunner.fromOptions call
TheNeuralBit opened a new pull request #11571:
URL: https://github.com/apache/beam/pull/11571
As far as I can tell this call, added in https://github.com/apache/beam/pull/2795, is no longer needed to make BigQueryTornadoesIT work on the DataflowRunner.
There's a risk that other runners mutate PipelineOptions and depend on this though.
Post-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
--- | --- | --- | --- | --- | --- | --- | ---
Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/)
Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/)
Python | [](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Python2_PVR_Flink_Cron/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python35_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/)
XLang | --- | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_XVR_Spark/lastCompletedBuild/)
Pre-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
--- |Java | Python | Go | Website
--- | --- | --- | --- | ---
Non-portable | [](https://builds.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/)
Portable | --- | [](https://builds.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/) | --- | ---
See [.test-infra/jenkins/README](https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md) for trigger phrase, status and link of all Jenkins jobs.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TheNeuralBit commented on pull request #11571: [BEAM-2128] Remove PipelineRunner.fromOptions call
Posted by GitBox <gi...@apache.org>.
TheNeuralBit commented on pull request #11571:
URL: https://github.com/apache/beam/pull/11571#issuecomment-621416935
oh perfect. Thanks :)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] mxm commented on pull request #11571: [BEAM-2128] Remove PipelineRunner.fromOptions call
Posted by GitBox <gi...@apache.org>.
mxm commented on pull request #11571:
URL: https://github.com/apache/beam/pull/11571#issuecomment-621414298
@TheNeuralBit Exactly. I had just pushed a commit which moves the logic.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TheNeuralBit commented on pull request #11571: [BEAM-2128] Remove PipelineRunner.fromOptions call
Posted by GitBox <gi...@apache.org>.
TheNeuralBit commented on pull request #11571:
URL: https://github.com/apache/beam/pull/11571#issuecomment-621425625
@kennknowles any concerns with the other two commits I pushed to fix unit test failures?
https://github.com/apache/beam/pull/11571/commits/7df26276b0d711fb3f1accaef8f3108960c2aee2 relaxes the test added for [BEAM-3493](https://issues.apache.org/jira/browse/BEAM-3493) to just verify that we throw an exception when `run()` is called, rather than on `create()`.
https://github.com/apache/beam/pull/11571/commits/775bb7d45f04f1ec373bdf5f42c8906b65f9068b ensures that the gs:// scheme is added for `BatchStatefulParDoOverridesTest`, previously this was happening inside PipelineOptions.fromRunner.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] mxm commented on pull request #11571: [BEAM-2128] Remove PipelineRunner.fromOptions call
Posted by GitBox <gi...@apache.org>.
mxm commented on pull request #11571:
URL: https://github.com/apache/beam/pull/11571#issuecomment-621410277
I just had a look as well :)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TheNeuralBit commented on pull request #11571: [BEAM-2128] Remove PipelineRunner.fromOptions call
Posted by GitBox <gi...@apache.org>.
TheNeuralBit commented on pull request #11571:
URL: https://github.com/apache/beam/pull/11571#issuecomment-621315454
R: @kennknowles
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TheNeuralBit commented on pull request #11571: [BEAM-2128] Remove PipelineRunner.fromOptions call
Posted by GitBox <gi...@apache.org>.
TheNeuralBit commented on pull request #11571:
URL: https://github.com/apache/beam/pull/11571#issuecomment-621412967
So it looks like the issue is FlinkRunner modifies the pipeline options here: https://github.com/apache/beam/blob/master/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkRunner.java#L77-L86
I'm proposing moving that logic into or around [`prepareFilesToStageForRemoteClusterExecution`](https://github.com/apache/beam/blob/master/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkPipelineExecutionEnvironment.java#L124).
Do you see any issue with that?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TheNeuralBit commented on pull request #11571: [BEAM-2128] Remove PipelineRunner.fromOptions call
Posted by GitBox <gi...@apache.org>.
TheNeuralBit commented on pull request #11571:
URL: https://github.com/apache/beam/pull/11571#issuecomment-621321032
Run Spark ValidatesRunner
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TheNeuralBit commented on pull request #11571: [BEAM-2128] Remove PipelineRunner.fromOptions call
Posted by GitBox <gi...@apache.org>.
TheNeuralBit commented on pull request #11571:
URL: https://github.com/apache/beam/pull/11571#issuecomment-621315667
Run Flink ValidatesRunner
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TheNeuralBit commented on pull request #11571: [BEAM-2128] Remove PipelineRunner.fromOptions call
Posted by GitBox <gi...@apache.org>.
TheNeuralBit commented on pull request #11571:
URL: https://github.com/apache/beam/pull/11571#issuecomment-621355318
@mxm it looks like the tests you added in https://github.com/apache/beam/pull/7748 depend on the call to `PipelineRunner.fromOptions`, which I'm attempting to remove in this PR. Is there any simple way to fix these failures?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TheNeuralBit commented on pull request #11571: [BEAM-2128] Remove PipelineRunner.fromOptions call
Posted by GitBox <gi...@apache.org>.
TheNeuralBit commented on pull request #11571:
URL: https://github.com/apache/beam/pull/11571#issuecomment-621408411
I think I've figured out how to fix it (or at least how to make these tests pass). I'll push a commit here soon
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TheNeuralBit commented on pull request #11571: [BEAM-2128] Remove PipelineRunner.fromOptions call
Posted by GitBox <gi...@apache.org>.
TheNeuralBit commented on pull request #11571:
URL: https://github.com/apache/beam/pull/11571#issuecomment-621315780
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TheNeuralBit commented on pull request #11571: [BEAM-2128] Remove PipelineRunner.fromOptions call
Posted by GitBox <gi...@apache.org>.
TheNeuralBit commented on pull request #11571:
URL: https://github.com/apache/beam/pull/11571#issuecomment-621321957
Spark failure is a known issue: https://issues.apache.org/jira/browse/BEAM-9836
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org