You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@beam.apache.org by Daniel Chen <da...@gmail.com> on 2019/04/22 22:21:28 UTC
Integration of python/portable runner tests for Samza runner
Hi everyone,
I'm working on improving the validation of the Python portable Samza
runner. For java, we have the gradle task ( :validatesRunner) that runs the
runner validation tests.
I am looking for pointers on how to similarly integrate/enable the
portability and Python tests for the Samza runner.
Any help will be greatly appreciated.
Thanks,
Daniel
Re: Integration of python/portable runner tests for Samza runner
Posted by Xinyu Liu <xi...@gmail.com>.
Thanks for the explanation, Max. We will try it out and let you guys know
if we run into any questions.
Thanks,
Xinyu
On Thu, Apr 25, 2019 at 6:28 AM Maximilian Michels <mx...@apache.org> wrote:
> > - For portable running tests: by looking at the
> > portableValidatesRunnerTask in flink_job_server.gradle, it seems it's
> > the same set of Java tests but using portability framework to validate
> > (submit to job server and run the protable pipeline in a specific
> > runner). Is my understanding correct?
>
> That's correct.
>
> > - For python tests: Looks like flinkValidatesRunner is using LOOPBACK
> > SDK worker type in the tests. Not sure what LOOPBACK does. Is it used
> > for testing? Currently Samza portable runner supports PROCESS worker.
>
> This avoids building and using containers for this test. We had a number
> of issues with Docker on Jenkins and wanted to lower build time for
> PreCommit tests. Loopback means that an embedded Python environment will
> be started which listens on localhost. It's comparable to Java's
> EmbeddedSdkHarness.
>
> -Max
>
> On 24.04.19 20:10, Xinyu Liu wrote:
> > Thanks for the useful pointers! We are looking forward to integrating
> > both Portable and Python-specific tests for Samza runner. A few
> questions:
> >
> > - For portable running tests: by looking at the
> > portableValidatesRunnerTask in flink_job_server.gradle, it seems it's
> > the same set of Java tests but using portability framework to validate
> > (submit to job server and run the protable pipeline in a specific
> > runner). Is my understanding correct?
> >
> > - For python tests: Looks like flinkValidatesRunner is using LOOPBACK
> > SDK worker type in the tests. Not sure what LOOPBACK does. Is it used
> > for testing? Currently Samza portable runner supports PROCESS worker.
> >
> > Thanks,
> > Xinyu
> >
> >
> >
> > On Wed, Apr 24, 2019 at 2:45 AM Maximilian Michels <mxm@apache.org
> > <ma...@apache.org>> wrote:
> >
> > > If you are interested in portable python pipeline validation, I
> > think
> > > fn_api_runner_test would also help.
> >
> > Just to note, Ankur mentioned flinkCompatibilityMatrix, that one uses
> > fn_api_runner_test with some tooling on top to bring up the test
> > cluster.
> >
> > On 23.04.19 19:23, Boyuan Zhang wrote:
> > > Hi Daniel,
> > > If you are interested in portable python pipeline validation, I
> > think
> > > fn_api_runner_test
> > >
> > <
> https://github.com/apache/beam/blob/master/sdks/python/apache_beam/runners/portability/fn_api_runner_test.py
> >
> >
> > > would also help.
> > >
> > > On Tue, Apr 23, 2019 at 10:19 AM Pablo Estrada
> > <pabloem@google.com <ma...@google.com>
> > > <mailto:pabloem@google.com <ma...@google.com>>> wrote:
> > >
> > > This is cool, Daniel : ) Glad to see the Samza runner moving
> > forward.
> > > Best
> > > -P.
> > >
> > > On Tue, Apr 23, 2019 at 2:52 AM Maximilian Michels
> > <mxm@apache.org <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>> wrote:
> > >
> > > Hi Daniel,
> > >
> > > Note that there is also Portable Validates Runner which
> > runs Java
> > > portability tests. I don't know if you have integrated
> > with that
> > > one
> > > already.
> > >
> > > Thanks,
> > > Max
> > >
> > > On 23.04.19 02:28, Ankur Goenka wrote:
> > > > Hi Daniel,
> > > >
> > > > We use flinkCompatibilityMatrix [1] to check the Flink
> > > compatibility
> > > > with python. This is python equivalent to
> validatesRunner
> > > tests in java
> > > > for portable runners.
> > > > I think we can reuse it for Samza Portable runner with
> > minor
> > > refactoring.
> > > >
> > > > [1]
> > > >
> > >
> >
> https://github.com/apache/beam/blob/bdb1a713a120a887e71e85c77879dc4446a58541/sdks/python/build.gradle#L305
> > > >
> > > > On Mon, Apr 22, 2019 at 3:21 PM Daniel Chen
> > > <danxich@gmail.com <ma...@gmail.com>
> > <mailto:danxich@gmail.com <ma...@gmail.com>>
> > > > <mailto:danxich@gmail.com <ma...@gmail.com>
> > <mailto:danxich@gmail.com <ma...@gmail.com>>>> wrote:
> > > >
> > > > Hi everyone,
> > > >
> > > > I'm working on improving the validation of the
> Python
> > > portable Samza
> > > > runner. For java, we have the gradle task (
> > > :validatesRunner) that
> > > > runs the runner validation tests.
> > > > I am looking for pointers on how to similarly
> > > integrate/enable the
> > > > portability and Python tests for the Samza runner.
> > > >
> > > > Any help will be greatly appreciated.
> > > >
> > > > Thanks,
> > > > Daniel
> > > >
> > >
> >
>
Re: Integration of python/portable runner tests for Samza runner
Posted by Maximilian Michels <mx...@apache.org>.
> - For portable running tests: by looking at the
> portableValidatesRunnerTask in flink_job_server.gradle, it seems it's
> the same set of Java tests but using portability framework to validate
> (submit to job server and run the protable pipeline in a specific
> runner). Is my understanding correct?
That's correct.
> - For python tests: Looks like flinkValidatesRunner is using LOOPBACK
> SDK worker type in the tests. Not sure what LOOPBACK does. Is it used
> for testing? Currently Samza portable runner supports PROCESS worker.
This avoids building and using containers for this test. We had a number
of issues with Docker on Jenkins and wanted to lower build time for
PreCommit tests. Loopback means that an embedded Python environment will
be started which listens on localhost. It's comparable to Java's
EmbeddedSdkHarness.
-Max
On 24.04.19 20:10, Xinyu Liu wrote:
> Thanks for the useful pointers! We are looking forward to integrating
> both Portable and Python-specific tests for Samza runner. A few questions:
>
> - For portable running tests: by looking at the
> portableValidatesRunnerTask in flink_job_server.gradle, it seems it's
> the same set of Java tests but using portability framework to validate
> (submit to job server and run the protable pipeline in a specific
> runner). Is my understanding correct?
>
> - For python tests: Looks like flinkValidatesRunner is using LOOPBACK
> SDK worker type in the tests. Not sure what LOOPBACK does. Is it used
> for testing? Currently Samza portable runner supports PROCESS worker.
>
> Thanks,
> Xinyu
>
>
>
> On Wed, Apr 24, 2019 at 2:45 AM Maximilian Michels <mxm@apache.org
> <ma...@apache.org>> wrote:
>
> > If you are interested in portable python pipeline validation, I
> think
> > fn_api_runner_test would also help.
>
> Just to note, Ankur mentioned flinkCompatibilityMatrix, that one uses
> fn_api_runner_test with some tooling on top to bring up the test
> cluster.
>
> On 23.04.19 19:23, Boyuan Zhang wrote:
> > Hi Daniel,
> > If you are interested in portable python pipeline validation, I
> think
> > fn_api_runner_test
> >
> <https://github.com/apache/beam/blob/master/sdks/python/apache_beam/runners/portability/fn_api_runner_test.py>
>
> > would also help.
> >
> > On Tue, Apr 23, 2019 at 10:19 AM Pablo Estrada
> <pabloem@google.com <ma...@google.com>
> > <mailto:pabloem@google.com <ma...@google.com>>> wrote:
> >
> > This is cool, Daniel : ) Glad to see the Samza runner moving
> forward.
> > Best
> > -P.
> >
> > On Tue, Apr 23, 2019 at 2:52 AM Maximilian Michels
> <mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>> wrote:
> >
> > Hi Daniel,
> >
> > Note that there is also Portable Validates Runner which
> runs Java
> > portability tests. I don't know if you have integrated
> with that
> > one
> > already.
> >
> > Thanks,
> > Max
> >
> > On 23.04.19 02:28, Ankur Goenka wrote:
> > > Hi Daniel,
> > >
> > > We use flinkCompatibilityMatrix [1] to check the Flink
> > compatibility
> > > with python. This is python equivalent to validatesRunner
> > tests in java
> > > for portable runners.
> > > I think we can reuse it for Samza Portable runner with
> minor
> > refactoring.
> > >
> > > [1]
> > >
> >
> https://github.com/apache/beam/blob/bdb1a713a120a887e71e85c77879dc4446a58541/sdks/python/build.gradle#L305
> > >
> > > On Mon, Apr 22, 2019 at 3:21 PM Daniel Chen
> > <danxich@gmail.com <ma...@gmail.com>
> <mailto:danxich@gmail.com <ma...@gmail.com>>
> > > <mailto:danxich@gmail.com <ma...@gmail.com>
> <mailto:danxich@gmail.com <ma...@gmail.com>>>> wrote:
> > >
> > > Hi everyone,
> > >
> > > I'm working on improving the validation of the Python
> > portable Samza
> > > runner. For java, we have the gradle task (
> > :validatesRunner) that
> > > runs the runner validation tests.
> > > I am looking for pointers on how to similarly
> > integrate/enable the
> > > portability and Python tests for the Samza runner.
> > >
> > > Any help will be greatly appreciated.
> > >
> > > Thanks,
> > > Daniel
> > >
> >
>
Re: Integration of python/portable runner tests for Samza runner
Posted by Xinyu Liu <xi...@gmail.com>.
Thanks for the useful pointers! We are looking forward to integrating both
Portable and Python-specific tests for Samza runner. A few questions:
- For portable running tests: by looking at the portableValidatesRunnerTask in
flink_job_server.gradle, it seems it's the same set of Java tests but using
portability framework to validate (submit to job server and run the
protable pipeline in a specific runner). Is my understanding correct?
- For python tests: Looks like flinkValidatesRunner is using LOOPBACK SDK
worker type in the tests. Not sure what LOOPBACK does. Is it used for
testing? Currently Samza portable runner supports PROCESS worker.
Thanks,
Xinyu
On Wed, Apr 24, 2019 at 2:45 AM Maximilian Michels <mx...@apache.org> wrote:
> > If you are interested in portable python pipeline validation, I think
> > fn_api_runner_test would also help.
>
> Just to note, Ankur mentioned flinkCompatibilityMatrix, that one uses
> fn_api_runner_test with some tooling on top to bring up the test cluster.
>
> On 23.04.19 19:23, Boyuan Zhang wrote:
> > Hi Daniel,
> > If you are interested in portable python pipeline validation, I think
> > fn_api_runner_test
> > <
> https://github.com/apache/beam/blob/master/sdks/python/apache_beam/runners/portability/fn_api_runner_test.py>
>
> > would also help.
> >
> > On Tue, Apr 23, 2019 at 10:19 AM Pablo Estrada <pabloem@google.com
> > <ma...@google.com>> wrote:
> >
> > This is cool, Daniel : ) Glad to see the Samza runner moving forward.
> > Best
> > -P.
> >
> > On Tue, Apr 23, 2019 at 2:52 AM Maximilian Michels <mxm@apache.org
> > <ma...@apache.org>> wrote:
> >
> > Hi Daniel,
> >
> > Note that there is also Portable Validates Runner which runs Java
> > portability tests. I don't know if you have integrated with that
> > one
> > already.
> >
> > Thanks,
> > Max
> >
> > On 23.04.19 02:28, Ankur Goenka wrote:
> > > Hi Daniel,
> > >
> > > We use flinkCompatibilityMatrix [1] to check the Flink
> > compatibility
> > > with python. This is python equivalent to validatesRunner
> > tests in java
> > > for portable runners.
> > > I think we can reuse it for Samza Portable runner with minor
> > refactoring.
> > >
> > > [1]
> > >
> >
> https://github.com/apache/beam/blob/bdb1a713a120a887e71e85c77879dc4446a58541/sdks/python/build.gradle#L305
> > >
> > > On Mon, Apr 22, 2019 at 3:21 PM Daniel Chen
> > <danxich@gmail.com <ma...@gmail.com>
> > > <mailto:danxich@gmail.com <ma...@gmail.com>>> wrote:
> > >
> > > Hi everyone,
> > >
> > > I'm working on improving the validation of the Python
> > portable Samza
> > > runner. For java, we have the gradle task (
> > :validatesRunner) that
> > > runs the runner validation tests.
> > > I am looking for pointers on how to similarly
> > integrate/enable the
> > > portability and Python tests for the Samza runner.
> > >
> > > Any help will be greatly appreciated.
> > >
> > > Thanks,
> > > Daniel
> > >
> >
>
Re: Integration of python/portable runner tests for Samza runner
Posted by Maximilian Michels <mx...@apache.org>.
> If you are interested in portable python pipeline validation, I think
> fn_api_runner_test would also help.
Just to note, Ankur mentioned flinkCompatibilityMatrix, that one uses
fn_api_runner_test with some tooling on top to bring up the test cluster.
On 23.04.19 19:23, Boyuan Zhang wrote:
> Hi Daniel,
> If you are interested in portable python pipeline validation, I think
> fn_api_runner_test
> <https://github.com/apache/beam/blob/master/sdks/python/apache_beam/runners/portability/fn_api_runner_test.py>
> would also help.
>
> On Tue, Apr 23, 2019 at 10:19 AM Pablo Estrada <pabloem@google.com
> <ma...@google.com>> wrote:
>
> This is cool, Daniel : ) Glad to see the Samza runner moving forward.
> Best
> -P.
>
> On Tue, Apr 23, 2019 at 2:52 AM Maximilian Michels <mxm@apache.org
> <ma...@apache.org>> wrote:
>
> Hi Daniel,
>
> Note that there is also Portable Validates Runner which runs Java
> portability tests. I don't know if you have integrated with that
> one
> already.
>
> Thanks,
> Max
>
> On 23.04.19 02:28, Ankur Goenka wrote:
> > Hi Daniel,
> >
> > We use flinkCompatibilityMatrix [1] to check the Flink
> compatibility
> > with python. This is python equivalent to validatesRunner
> tests in java
> > for portable runners.
> > I think we can reuse it for Samza Portable runner with minor
> refactoring.
> >
> > [1]
> >
> https://github.com/apache/beam/blob/bdb1a713a120a887e71e85c77879dc4446a58541/sdks/python/build.gradle#L305
> >
> > On Mon, Apr 22, 2019 at 3:21 PM Daniel Chen
> <danxich@gmail.com <ma...@gmail.com>
> > <mailto:danxich@gmail.com <ma...@gmail.com>>> wrote:
> >
> > Hi everyone,
> >
> > I'm working on improving the validation of the Python
> portable Samza
> > runner. For java, we have the gradle task (
> :validatesRunner) that
> > runs the runner validation tests.
> > I am looking for pointers on how to similarly
> integrate/enable the
> > portability and Python tests for the Samza runner.
> >
> > Any help will be greatly appreciated.
> >
> > Thanks,
> > Daniel
> >
>
Re: Integration of python/portable runner tests for Samza runner
Posted by Boyuan Zhang <bo...@google.com>.
Hi Daniel,
If you are interested in portable python pipeline validation, I think
fn_api_runner_test
<https://github.com/apache/beam/blob/master/sdks/python/apache_beam/runners/portability/fn_api_runner_test.py>
would also help.
On Tue, Apr 23, 2019 at 10:19 AM Pablo Estrada <pa...@google.com> wrote:
> This is cool, Daniel : ) Glad to see the Samza runner moving forward.
> Best
> -P.
>
> On Tue, Apr 23, 2019 at 2:52 AM Maximilian Michels <mx...@apache.org> wrote:
>
>> Hi Daniel,
>>
>> Note that there is also Portable Validates Runner which runs Java
>> portability tests. I don't know if you have integrated with that one
>> already.
>>
>> Thanks,
>> Max
>>
>> On 23.04.19 02:28, Ankur Goenka wrote:
>> > Hi Daniel,
>> >
>> > We use flinkCompatibilityMatrix [1] to check the Flink compatibility
>> > with python. This is python equivalent to validatesRunner tests in java
>> > for portable runners.
>> > I think we can reuse it for Samza Portable runner with minor
>> refactoring.
>> >
>> > [1]
>> >
>> https://github.com/apache/beam/blob/bdb1a713a120a887e71e85c77879dc4446a58541/sdks/python/build.gradle#L305
>> >
>> > On Mon, Apr 22, 2019 at 3:21 PM Daniel Chen <danxich@gmail.com
>> > <ma...@gmail.com>> wrote:
>> >
>> > Hi everyone,
>> >
>> > I'm working on improving the validation of the Python portable Samza
>> > runner. For java, we have the gradle task ( :validatesRunner) that
>> > runs the runner validation tests.
>> > I am looking for pointers on how to similarly integrate/enable the
>> > portability and Python tests for the Samza runner.
>> >
>> > Any help will be greatly appreciated.
>> >
>> > Thanks,
>> > Daniel
>> >
>>
>
Re: Integration of python/portable runner tests for Samza runner
Posted by Pablo Estrada <pa...@google.com>.
This is cool, Daniel : ) Glad to see the Samza runner moving forward.
Best
-P.
On Tue, Apr 23, 2019 at 2:52 AM Maximilian Michels <mx...@apache.org> wrote:
> Hi Daniel,
>
> Note that there is also Portable Validates Runner which runs Java
> portability tests. I don't know if you have integrated with that one
> already.
>
> Thanks,
> Max
>
> On 23.04.19 02:28, Ankur Goenka wrote:
> > Hi Daniel,
> >
> > We use flinkCompatibilityMatrix [1] to check the Flink compatibility
> > with python. This is python equivalent to validatesRunner tests in java
> > for portable runners.
> > I think we can reuse it for Samza Portable runner with minor refactoring.
> >
> > [1]
> >
> https://github.com/apache/beam/blob/bdb1a713a120a887e71e85c77879dc4446a58541/sdks/python/build.gradle#L305
> >
> > On Mon, Apr 22, 2019 at 3:21 PM Daniel Chen <danxich@gmail.com
> > <ma...@gmail.com>> wrote:
> >
> > Hi everyone,
> >
> > I'm working on improving the validation of the Python portable Samza
> > runner. For java, we have the gradle task ( :validatesRunner) that
> > runs the runner validation tests.
> > I am looking for pointers on how to similarly integrate/enable the
> > portability and Python tests for the Samza runner.
> >
> > Any help will be greatly appreciated.
> >
> > Thanks,
> > Daniel
> >
>
Re: Integration of python/portable runner tests for Samza runner
Posted by Maximilian Michels <mx...@apache.org>.
Hi Daniel,
Note that there is also Portable Validates Runner which runs Java
portability tests. I don't know if you have integrated with that one
already.
Thanks,
Max
On 23.04.19 02:28, Ankur Goenka wrote:
> Hi Daniel,
>
> We use flinkCompatibilityMatrix [1] to check the Flink compatibility
> with python. This is python equivalent to validatesRunner tests in java
> for portable runners.
> I think we can reuse it for Samza Portable runner with minor refactoring.
>
> [1]
> https://github.com/apache/beam/blob/bdb1a713a120a887e71e85c77879dc4446a58541/sdks/python/build.gradle#L305
>
> On Mon, Apr 22, 2019 at 3:21 PM Daniel Chen <danxich@gmail.com
> <ma...@gmail.com>> wrote:
>
> Hi everyone,
>
> I'm working on improving the validation of the Python portable Samza
> runner. For java, we have the gradle task ( :validatesRunner) that
> runs the runner validation tests.
> I am looking for pointers on how to similarly integrate/enable the
> portability and Python tests for the Samza runner.
>
> Any help will be greatly appreciated.
>
> Thanks,
> Daniel
>
Re: Integration of python/portable runner tests for Samza runner
Posted by Ankur Goenka <go...@google.com>.
Hi Daniel,
We use flinkCompatibilityMatrix [1] to check the Flink compatibility with
python. This is python equivalent to validatesRunner tests in java for
portable runners.
I think we can reuse it for Samza Portable runner with minor refactoring.
[1]
https://github.com/apache/beam/blob/bdb1a713a120a887e71e85c77879dc4446a58541/sdks/python/build.gradle#L305
On Mon, Apr 22, 2019 at 3:21 PM Daniel Chen <da...@gmail.com> wrote:
> Hi everyone,
>
> I'm working on improving the validation of the Python portable Samza
> runner. For java, we have the gradle task ( :validatesRunner) that runs the
> runner validation tests.
> I am looking for pointers on how to similarly integrate/enable the
> portability and Python tests for the Samza runner.
>
> Any help will be greatly appreciated.
>
> Thanks,
> Daniel
>