You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@beam.apache.org by on 2018/12/03 22:10:53 UTC

How to kick off a Beam pipeline with PortableRunner in Java?

Hi everyone,

Can someone point me to how to kick off a Beam pipeline using the PortableRunner (w/Flink) in Java? I saw some examples in Python but I haven't been able to find any for Java.

I tried to modify the runner option to use PortableRunner but I get the following error below:
java.lang.IllegalArgumentException: Unknown 'runner' specified 'PortableRunner', supported pipeline runners [DirectRunner, FlinkRunner, SparkRunner, TestFlinkRunner, TestSparkRunner]

For reference, I am on Beam 2.8.0 and the reason I want to try to use the PortableRunner is so I can confirm a comment[1] in BEAM-593 that states that pipeline.run() is no longer a blocking call in the PortableRunner with Flink.

[1] https://issues.apache.org/jira/browse/BEAM-593?focusedCommentId=16618916&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16618916

Re: How to kick off a Beam pipeline with PortableRunner in Java?

Posted by Ruoyun Huang <ru...@google.com>.
np.

I think this dependency should be included as default. Sent a PR to fix
this: https://github.com/apache/beam/pull/7213

On Wed, Dec 5, 2018 at 7:10 AM Sai Inampudi <sa...@gmail.com> wrote:

> That did it, thanks for the help Ruoyun, I did forget to pull in the
> dependency in my pom! bad mistake :)
>
> On 2018/12/05 05:24:29, Ruoyun Huang <ru...@google.com> wrote:
> > This should work. but maybe try adding a section like this to your
> pom.xml
> > file:
> >
> >     <profile>
> >
> >       <id>portable-runner</id>
> >
> >       <activation>
> >
> >         <activeByDefault>true</activeByDefault>
> >
> >       </activation>
> >
> >       <!-- Makes the PortableRunner available when running a pipeline.
> -->
> >
> >       <dependencies>
> >
> >         <dependency>
> >
> >           <groupId>org.apache.beam</groupId>
> >
> >           <artifactId>beam-runners-reference-java</artifactId>
> >
> >           <version>${beam.version}</version>
> >
> >           <scope>runtime</scope>
> >
> >         </dependency>
> >
> >       </dependencies>
> >
> >     </profile>
> >
> >
> > Note that PortableRunner has major features still in development. It
> > currently can only run sample pipeline without file output. There are
> > active efforts on it though.
> >
> >
> >
> >
> >
> > On Tue, Dec 4, 2018 at 2:21 PM Sai Inampudi <sa...@gmail.com>
> wrote:
> >
> > > Thanks for the help Ankur and Ruoyun, appreciate it. I went through the
> > > wiki and I am still facing the same issue as before (where it complains
> > > about the following:
> > > java.lang.IllegalArgumentException: Unknown 'runner' specified
> > > 'PortableRunner', supported pipeline runners [DirectRunner,
> FlinkRunner,
> > > SparkRunner, TestFlinkRunner, TestSparkRunner])
> > >
> > > I am probably doing a naive mistake but I am not sure where, below is
> > > everything I have done thus far:
> > > * Confirmed I have met the prerequisities in the wiki
> > >      * Docker is installed and working without sudo
> > >      * gradle target for the java container works
> > >
> > > * Successfully started the JobServer (for now, I didn't bother with the
> > > flink job server and instead just kicked off the reference job server)
> by
> > > running the following
> > >     * ./gradlew :beam-runners-reference-job-server:run
> > >     * Confirmed that the JobService started at 8099 port
> > >         * Started ReferenceRunnerJobService at localhost:8099
> > >            <============-> 98% EXECUTING [22m 43s]
> > >            > :beam-runners-reference-job-server:run
> > >            > IDLE
> > >
> > > * My code itself is being managed via maven, so I made sure to pull in
> the
> > > latest 2.10.0-SNAPSHOT dependencies as per
> > > https://repository.apache.org/content/repositories/snapshots/
> > >
> > > * The pipeline is kicked off with the following options:
> > >      * --runner=PortableRunner --jobEndpoint=localhost:8099
> > > --streaming=true
> > >      * Running the pipeline results in the following error
> > >            * java.lang.IllegalArgumentException: Unknown 'runner'
> > > specified 'PortableRunner', supported pipeline runners [DirectRunner,
> > > FlinkRunner, SparkRunner, TestFlinkRunner, TestSparkRunner]
> > >
> > >
> > >
> > > I am sure I am missing something basic but I was hoping I could get
> ideas
> > > on what it might be?
> > >
> > >
> >
> > --
> > ================
> > Ruoyun  Huang
> >
>


-- 
================
Ruoyun  Huang

Re: How to kick off a Beam pipeline with PortableRunner in Java?

Posted by Sai Inampudi <sa...@gmail.com>.
That did it, thanks for the help Ruoyun, I did forget to pull in the dependency in my pom! bad mistake :)

On 2018/12/05 05:24:29, Ruoyun Huang <ru...@google.com> wrote: 
> This should work. but maybe try adding a section like this to your pom.xml
> file:
> 
>     <profile>
> 
>       <id>portable-runner</id>
> 
>       <activation>
> 
>         <activeByDefault>true</activeByDefault>
> 
>       </activation>
> 
>       <!-- Makes the PortableRunner available when running a pipeline. -->
> 
>       <dependencies>
> 
>         <dependency>
> 
>           <groupId>org.apache.beam</groupId>
> 
>           <artifactId>beam-runners-reference-java</artifactId>
> 
>           <version>${beam.version}</version>
> 
>           <scope>runtime</scope>
> 
>         </dependency>
> 
>       </dependencies>
> 
>     </profile>
> 
> 
> Note that PortableRunner has major features still in development. It
> currently can only run sample pipeline without file output. There are
> active efforts on it though.
> 
> 
> 
> 
> 
> On Tue, Dec 4, 2018 at 2:21 PM Sai Inampudi <sa...@gmail.com> wrote:
> 
> > Thanks for the help Ankur and Ruoyun, appreciate it. I went through the
> > wiki and I am still facing the same issue as before (where it complains
> > about the following:
> > java.lang.IllegalArgumentException: Unknown 'runner' specified
> > 'PortableRunner', supported pipeline runners [DirectRunner, FlinkRunner,
> > SparkRunner, TestFlinkRunner, TestSparkRunner])
> >
> > I am probably doing a naive mistake but I am not sure where, below is
> > everything I have done thus far:
> > * Confirmed I have met the prerequisities in the wiki
> >      * Docker is installed and working without sudo
> >      * gradle target for the java container works
> >
> > * Successfully started the JobServer (for now, I didn't bother with the
> > flink job server and instead just kicked off the reference job server) by
> > running the following
> >     * ./gradlew :beam-runners-reference-job-server:run
> >     * Confirmed that the JobService started at 8099 port
> >         * Started ReferenceRunnerJobService at localhost:8099
> >            <============-> 98% EXECUTING [22m 43s]
> >            > :beam-runners-reference-job-server:run
> >            > IDLE
> >
> > * My code itself is being managed via maven, so I made sure to pull in the
> > latest 2.10.0-SNAPSHOT dependencies as per
> > https://repository.apache.org/content/repositories/snapshots/
> >
> > * The pipeline is kicked off with the following options:
> >      * --runner=PortableRunner --jobEndpoint=localhost:8099
> > --streaming=true
> >      * Running the pipeline results in the following error
> >            * java.lang.IllegalArgumentException: Unknown 'runner'
> > specified 'PortableRunner', supported pipeline runners [DirectRunner,
> > FlinkRunner, SparkRunner, TestFlinkRunner, TestSparkRunner]
> >
> >
> >
> > I am sure I am missing something basic but I was hoping I could get ideas
> > on what it might be?
> >
> >
> 
> -- 
> ================
> Ruoyun  Huang
> 

Re: How to kick off a Beam pipeline with PortableRunner in Java?

Posted by Ruoyun Huang <ru...@google.com>.
This should work. but maybe try adding a section like this to your pom.xml
file:

    <profile>

      <id>portable-runner</id>

      <activation>

        <activeByDefault>true</activeByDefault>

      </activation>

      <!-- Makes the PortableRunner available when running a pipeline. -->

      <dependencies>

        <dependency>

          <groupId>org.apache.beam</groupId>

          <artifactId>beam-runners-reference-java</artifactId>

          <version>${beam.version}</version>

          <scope>runtime</scope>

        </dependency>

      </dependencies>

    </profile>


Note that PortableRunner has major features still in development. It
currently can only run sample pipeline without file output. There are
active efforts on it though.





On Tue, Dec 4, 2018 at 2:21 PM Sai Inampudi <sa...@gmail.com> wrote:

> Thanks for the help Ankur and Ruoyun, appreciate it. I went through the
> wiki and I am still facing the same issue as before (where it complains
> about the following:
> java.lang.IllegalArgumentException: Unknown 'runner' specified
> 'PortableRunner', supported pipeline runners [DirectRunner, FlinkRunner,
> SparkRunner, TestFlinkRunner, TestSparkRunner])
>
> I am probably doing a naive mistake but I am not sure where, below is
> everything I have done thus far:
> * Confirmed I have met the prerequisities in the wiki
>      * Docker is installed and working without sudo
>      * gradle target for the java container works
>
> * Successfully started the JobServer (for now, I didn't bother with the
> flink job server and instead just kicked off the reference job server) by
> running the following
>     * ./gradlew :beam-runners-reference-job-server:run
>     * Confirmed that the JobService started at 8099 port
>         * Started ReferenceRunnerJobService at localhost:8099
>            <============-> 98% EXECUTING [22m 43s]
>            > :beam-runners-reference-job-server:run
>            > IDLE
>
> * My code itself is being managed via maven, so I made sure to pull in the
> latest 2.10.0-SNAPSHOT dependencies as per
> https://repository.apache.org/content/repositories/snapshots/
>
> * The pipeline is kicked off with the following options:
>      * --runner=PortableRunner --jobEndpoint=localhost:8099
> --streaming=true
>      * Running the pipeline results in the following error
>            * java.lang.IllegalArgumentException: Unknown 'runner'
> specified 'PortableRunner', supported pipeline runners [DirectRunner,
> FlinkRunner, SparkRunner, TestFlinkRunner, TestSparkRunner]
>
>
>
> I am sure I am missing something basic but I was hoping I could get ideas
> on what it might be?
>
>

-- 
================
Ruoyun  Huang

Re: How to kick off a Beam pipeline with PortableRunner in Java?

Posted by Sai Inampudi <sa...@gmail.com>.
Thanks for the help Ankur and Ruoyun, appreciate it. I went through the wiki and I am still facing the same issue as before (where it complains about the following:
java.lang.IllegalArgumentException: Unknown 'runner' specified 'PortableRunner', supported pipeline runners [DirectRunner, FlinkRunner, SparkRunner, TestFlinkRunner, TestSparkRunner])

I am probably doing a naive mistake but I am not sure where, below is everything I have done thus far:
* Confirmed I have met the prerequisities in the wiki
     * Docker is installed and working without sudo
     * gradle target for the java container works

* Successfully started the JobServer (for now, I didn't bother with the flink job server and instead just kicked off the reference job server) by running the following
    * ./gradlew :beam-runners-reference-job-server:run
    * Confirmed that the JobService started at 8099 port
        * Started ReferenceRunnerJobService at localhost:8099
           <============-> 98% EXECUTING [22m 43s]
           > :beam-runners-reference-job-server:run
           > IDLE

* My code itself is being managed via maven, so I made sure to pull in the latest 2.10.0-SNAPSHOT dependencies as per https://repository.apache.org/content/repositories/snapshots/

* The pipeline is kicked off with the following options:
     * --runner=PortableRunner --jobEndpoint=localhost:8099 --streaming=true
     * Running the pipeline results in the following error
           * java.lang.IllegalArgumentException: Unknown 'runner' specified 'PortableRunner', supported pipeline runners [DirectRunner, FlinkRunner, SparkRunner, TestFlinkRunner, TestSparkRunner]



I am sure I am missing something basic but I was hoping I could get ideas on what it might be?


Re: How to kick off a Beam pipeline with PortableRunner in Java?

Posted by Ankur Goenka <go...@google.com>.
Thanks Ruoyun!

For Flink, we use a different job server which you can start using
"./gradlew beam-runners-flink_2.11-job-server:runShadow "
The host:port for this jobserver is localhost:8099

On Mon, Dec 3, 2018 at 2:24 PM Ruoyun Huang <ru...@google.com> wrote:

> Maybe this helps:
> https://cwiki.apache.org/confluence/display/BEAM/Usage+Guide
>
> On Mon, Dec 3, 2018 at 2:10 PM sai.inampudi@gmail.com <
> sai.inampudi@gmail.com> wrote:
>
>> Hi everyone,
>>
>> Can someone point me to how to kick off a Beam pipeline using the
>> PortableRunner (w/Flink) in Java? I saw some examples in Python but I
>> haven't been able to find any for Java.
>>
>> I tried to modify the runner option to use PortableRunner but I get the
>> following error below:
>> java.lang.IllegalArgumentException: Unknown 'runner' specified
>> 'PortableRunner', supported pipeline runners [DirectRunner, FlinkRunner,
>> SparkRunner, TestFlinkRunner, TestSparkRunner]
>>
>> For reference, I am on Beam 2.8.0 and the reason I want to try to use the
>> PortableRunner is so I can confirm a comment[1] in BEAM-593 that states
>> that pipeline.run() is no longer a blocking call in the PortableRunner with
>> Flink.
>>
>> [1]
>> https://issues.apache.org/jira/browse/BEAM-593?focusedCommentId=16618916&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16618916
>>
>
>
> --
> ================
> Ruoyun  Huang
>
>

Re: How to kick off a Beam pipeline with PortableRunner in Java?

Posted by Ruoyun Huang <ru...@google.com>.
Maybe this helps:
https://cwiki.apache.org/confluence/display/BEAM/Usage+Guide

On Mon, Dec 3, 2018 at 2:10 PM sai.inampudi@gmail.com <
sai.inampudi@gmail.com> wrote:

> Hi everyone,
>
> Can someone point me to how to kick off a Beam pipeline using the
> PortableRunner (w/Flink) in Java? I saw some examples in Python but I
> haven't been able to find any for Java.
>
> I tried to modify the runner option to use PortableRunner but I get the
> following error below:
> java.lang.IllegalArgumentException: Unknown 'runner' specified
> 'PortableRunner', supported pipeline runners [DirectRunner, FlinkRunner,
> SparkRunner, TestFlinkRunner, TestSparkRunner]
>
> For reference, I am on Beam 2.8.0 and the reason I want to try to use the
> PortableRunner is so I can confirm a comment[1] in BEAM-593 that states
> that pipeline.run() is no longer a blocking call in the PortableRunner with
> Flink.
>
> [1]
> https://issues.apache.org/jira/browse/BEAM-593?focusedCommentId=16618916&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16618916
>


-- 
================
Ruoyun  Huang