You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@beam.apache.org by Ismaël Mejía <ie...@gmail.com> on 2016/05/19 16:32:44 UTC
Problem with Pipeline In Flink Runner
I was trying to run a local Pipeline we have (basically a batch one we
rewrote from TPC-H) in all the possible runners, for the moment I had
success with the DirectPipelineRunner, InProcessPipelineRunner and
SparkPipelineRunner. However when I executed it in the FlinkPipelineRunner
I got this exception:
java.lang.UnsupportedOperationException: The transform Window.Into() is
currently not supported.
I thought it was probably a configuration error since I remember that in
the flink runner we had to explicitly define if it was a batch or streaming
job, but when I checked the options:
Current Settings:
appName: TestQ08Task
filesToStage: ...
flinkMaster: [auto]
parallelism: 1
runner: class org.apache.beam.runners.flink.FlinkPipelineRunner
stableUniqueNames: WARNING
streaming: false
I noticed I was in batch mode, is there something missing in the options
configuration, or is it that the UnsoppertedOperation is a WIP or not
supported at all by Flink ?
And extra question, this option to define if it is in batch or stream mode
is going to stay for long, can't be this inferred some how ?
Thanks,
-Ismaël
Re: Problem with Pipeline In Flink Runner
Posted by Frances Perry <fj...@google.com>.
Amir -- this sounds like a general Flink usage question not related to
Beam? If so, you might want to try the Flink user's list.
On Thu, May 19, 2016 at 4:22 PM, amir bahmanyari <am...@yahoo.com>
wrote:
> Hi Colleagues,
> Pls let me know this is not related to StreamingMode so I can create a
> separate thread for it.
> I thought it might be relevant.
>
> I am trying to run a Storm app in a flink-1.0.0 Cluster following this
> link.
> Apache Flink 0.10.2 Documentation: Storm Compatibility
> <https://ci.apache.org/projects/flink/flink-docs-release-0.10/apis/storm_compatibility.html>
>
> Apache Flink 0.10.2 Documentation: Storm Compatibility
>
> <https://ci.apache.org/projects/flink/flink-docs-release-0.10/apis/storm_compatibility.html>
>
>
> I have everything i.e. fat jar cleanly built & packaged.
> I have not added any jars to flink's lib folder. Just the default jars
> there.
> Following is the only dependency in pom.xml & everything gets compiled &
> packaged by this dependency.
> <dependency>
> <groupId>org.apache.flink</groupId>
> <artifactId>flink-storm</artifactId>
> <version>0.10.2</version>
> </dependency>
>
> When I try to run it in Flink Cluster, I get a StreamingMode related
> MethodNotFound exception.
> I see from Maven report that flink runtime jar has been included in the
> packaged fat jar.
>
> java.lang.*NoSuchMethodError*:
> org.apache.flink.runtime.minicluster.LocalFlinkMiniCluster.<init>(Lorg/apache/flink/configuration/Configuration;ZLorg/apache/flink/runtime/
> *StreamingMode*;)V
> at
> org.apache.flink.storm.api.FlinkLocalCluster.<init>(FlinkLocalCluster.java:54)
> at com.myco.flinkstorm.SimpleTopology.main(SimpleTopology.java:102)
> I appreciate any idea/suggestions pls.
> Thanks for your help.
> Amir-
> ------------------------------
> *From:* Aljoscha Krettek <al...@apache.org>
> *To:* user@beam.incubator.apache.org
> *Sent:* Thursday, May 19, 2016 10:31 AM
> *Subject:* Re: Problem with Pipeline In Flink Runner
>
> Hi,
> as an addition, there are plans to get rid of the "streaming" flag for all
> runners and determine the mode from the job.
>
> -Aljoscha
>
> On Thu, 19 May 2016 at 18:52 Maximilian Michels <mx...@apache.org> wrote:
>
> Hi Ismaël,
>
> There is a pull request which enables Windows on batch. It will be
> merged soon. In the meantime, please use the streaming execution mode
> of the Flink Runner.
>
> Use setStreaming(true) to enable it on the FlinkPipelineOptions.
>
> Cheers,
> Max
>
> On Thu, May 19, 2016 at 6:32 PM, Ismaël Mejía <ie...@gmail.com> wrote:
> > I was trying to run a local Pipeline we have (basically a batch one we
> > rewrote from TPC-H) in all the possible runners, for the moment I had
> > success with the DirectPipelineRunner, InProcessPipelineRunner and
> > SparkPipelineRunner. However when I executed it in the
> FlinkPipelineRunner I
> > got this exception:
> >
> > java.lang.UnsupportedOperationException: The transform Window.Into() is
> > currently not supported.
> >
> > I thought it was probably a configuration error since I remember that in
> the
> > flink runner we had to explicitly define if it was a batch or streaming
> job,
> > but when I checked the options:
> >
> > Current Settings:
> > appName: TestQ08Task
> > filesToStage: ...
> > flinkMaster: [auto]
> > parallelism: 1
> > runner: class org.apache.beam.runners.flink.FlinkPipelineRunner
> > stableUniqueNames: WARNING
> > streaming: false
> >
> > I noticed I was in batch mode, is there something missing in the options
> > configuration, or is it that the UnsoppertedOperation is a WIP or not
> > supported at all by Flink ?
> >
> > And extra question, this option to define if it is in batch or stream
> mode
> > is going to stay for long, can't be this inferred some how ?
> >
> > Thanks,
> > -Ismaël
> >
>
>
>
>
Re: Problem with Pipeline In Flink Runner
Posted by amir bahmanyari <am...@yahoo.com>.
Hi Colleagues,Pls let me know this is not related to StreamingMode so I can create a separate thread for it.I thought it might be relevant.
I am trying to run a Storm app in a flink-1.0.0 Cluster following this link.Apache Flink 0.10.2 Documentation: Storm Compatibility
|
|
|
| | |
|
|
|
| |
Apache Flink 0.10.2 Documentation: Storm Compatibility
| |
|
|
I have everything i.e. fat jar cleanly built & packaged.I have not added any jars to flink's lib folder. Just the default jars there.Following is the only dependency in pom.xml & everything gets compiled & packaged by this dependency.<dependency> <groupId>org.apache.flink</groupId> <artifactId>flink-storm</artifactId> <version>0.10.2</version> </dependency>
When I try to run it in Flink Cluster, I get a StreamingMode related MethodNotFound exception.
I see from Maven report that flink runtime jar has been included in the packaged fat jar.
java.lang.NoSuchMethodError: org.apache.flink.runtime.minicluster.LocalFlinkMiniCluster.<init>(Lorg/apache/flink/configuration/Configuration;ZLorg/apache/flink/runtime/StreamingMode;)V at org.apache.flink.storm.api.FlinkLocalCluster.<init>(FlinkLocalCluster.java:54) at com.myco.flinkstorm.SimpleTopology.main(SimpleTopology.java:102)I appreciate any idea/suggestions pls.Thanks for your help.
Amir- From: Aljoscha Krettek <al...@apache.org>
To: user@beam.incubator.apache.org
Sent: Thursday, May 19, 2016 10:31 AM
Subject: Re: Problem with Pipeline In Flink Runner
Hi,as an addition, there are plans to get rid of the "streaming" flag for all runners and determine the mode from the job.
-Aljoscha
On Thu, 19 May 2016 at 18:52 Maximilian Michels <mx...@apache.org> wrote:
Hi Ismaël,
There is a pull request which enables Windows on batch. It will be
merged soon. In the meantime, please use the streaming execution mode
of the Flink Runner.
Use setStreaming(true) to enable it on the FlinkPipelineOptions.
Cheers,
Max
On Thu, May 19, 2016 at 6:32 PM, Ismaël Mejía <ie...@gmail.com> wrote:
> I was trying to run a local Pipeline we have (basically a batch one we
> rewrote from TPC-H) in all the possible runners, for the moment I had
> success with the DirectPipelineRunner, InProcessPipelineRunner and
> SparkPipelineRunner. However when I executed it in the FlinkPipelineRunner I
> got this exception:
>
> java.lang.UnsupportedOperationException: The transform Window.Into() is
> currently not supported.
>
> I thought it was probably a configuration error since I remember that in the
> flink runner we had to explicitly define if it was a batch or streaming job,
> but when I checked the options:
>
> Current Settings:
> appName: TestQ08Task
> filesToStage: ...
> flinkMaster: [auto]
> parallelism: 1
> runner: class org.apache.beam.runners.flink.FlinkPipelineRunner
> stableUniqueNames: WARNING
> streaming: false
>
> I noticed I was in batch mode, is there something missing in the options
> configuration, or is it that the UnsoppertedOperation is a WIP or not
> supported at all by Flink ?
>
> And extra question, this option to define if it is in batch or stream mode
> is going to stay for long, can't be this inferred some how ?
>
> Thanks,
> -Ismaël
>
Re: Problem with Pipeline In Flink Runner
Posted by Ismaël Mejía <ie...@gmail.com>.
Hi,
Thanks, that's the way I did it, it was just that I thought for a second
about a more complicated system property of the -D kind, but the normal way
is enough for my current case, thanks.
Ismaël
On Fri, May 20, 2016 at 2:56 PM, Aljoscha Krettek <al...@apache.org>
wrote:
> Hi,
> you should be able to set the runner using command line options, e.g:
> --runner=org.apache.beam.runners.flink.FlinkPipelineRunner
> --streaming=false --parallelism=8
>
> And in the program have something like:
> Options options = PipelineOptionsFactory.fromArgs(args).withValidation();
>
> -Aljoscha
>
>
> On Fri, 20 May 2016 at 13:59 Jean-Baptiste Onofré <jb...@nanthrax.net> wrote:
>
>> You can always do it via System, populating the options, right ?
>>
>> Regards
>> JB
>>
>> On 05/20/2016 11:07 AM, Ismaël Mejía wrote:
>> > Hello again,
>> >
>> > I just executed our pipeline with the changes you just merged, and guess
>> > what ? It works perfectly (in batch mode), amazing job guys. I have
>> just one
>> > extra question, is there a way to configure the pipeline options with
>> system
>> > properties (-D...). e.g. to configure the streaming/batch mode and
>> > avoid to do
>> > this explicitly in the code (to remove the explicit FlinkPipelineOptions
>> > import) ?
>> >
>> > Ismaël
>> >
>> >
>> > On Fri, May 20, 2016 at 10:06 AM, Maximilian Michels <mxm@apache.org
>> > <ma...@apache.org>> wrote:
>> >
>> > Hi Ismael,
>> >
>> > Side inputs are not supported when the Flink Runner runs in
>> streaming
>> > mode. Actually, we just merged changes which allow you to use the
>> > batch mode (streaming set to false) with Windows and side inputs.
>> >
>> > So just checkout the latest Beam master and rebuild. Set the
>> streaming
>> > flag to false again and rerun your example.
>> >
>> > Thanks,
>> > Max
>> >
>> > On Fri, May 20, 2016 at 9:22 AM, Ismaël Mejía <iemejia@gmail.com
>> > <ma...@gmail.com>> wrote:
>> > >
>> > > Hello,
>> > >
>> > > Thanks for your answer, I am really happy to know you are
>> working on
>> > > removing
>> > > this flag for batch/streaming since this is not really Beam-like.
>> > >
>> > > I tried to execute the pipeline again in streaming mode, and I
>> > got this
>> > > exception now:
>> > >
>> > > java.lang.UnsupportedOperationException: The transform
>> > > View.CreatePCollectionView is currently not supported.
>> > >
>> > > Any info about this one ?
>> > >
>> > > Thanks again,
>> > > Ismael
>> > >
>> > >
>> > > On Thu, May 19, 2016 at 7:31 PM, Aljoscha Krettek
>> > <aljoscha@apache.org <ma...@apache.org>>
>> > > wrote:
>> > >>
>> > >> Hi,
>> > >> as an addition, there are plans to get rid of the "streaming"
>> > flag for all
>> > >> runners and determine the mode from the job.
>> > >>
>> > >> -Aljoscha
>> > >>
>> > >> On Thu, 19 May 2016 at 18:52 Maximilian Michels <mxm@apache.org
>> > <ma...@apache.org>> wrote:
>> > >>>
>> > >>> Hi Ismaël,
>> > >>>
>> > >>> There is a pull request which enables Windows on batch. It
>> will be
>> > >>> merged soon. In the meantime, please use the streaming
>> > execution mode
>> > >>> of the Flink Runner.
>> > >>>
>> > >>> Use setStreaming(true) to enable it on the
>> FlinkPipelineOptions.
>> > >>>
>> > >>> Cheers,
>> > >>> Max
>> > >>>
>> > >>> On Thu, May 19, 2016 at 6:32 PM, Ismaël Mejía
>> > <iemejia@gmail.com <ma...@gmail.com>> wrote:
>> > >>> > I was trying to run a local Pipeline we have (basically a
>> > batch one we
>> > >>> > rewrote from TPC-H) in all the possible runners, for the
>> > moment I had
>> > >>> > success with the DirectPipelineRunner,
>> > InProcessPipelineRunner and
>> > >>> > SparkPipelineRunner. However when I executed it in the
>> > >>> > FlinkPipelineRunner I
>> > >>> > got this exception:
>> > >>> >
>> > >>> > java.lang.UnsupportedOperationException: The transform
>> > Window.Into() is
>> > >>> > currently not supported.
>> > >>> >
>> > >>> > I thought it was probably a configuration error since I
>> > remember that
>> > >>> > in the
>> > >>> > flink runner we had to explicitly define if it was a batch or
>> > streaming
>> > >>> > job,
>> > >>> > but when I checked the options:
>> > >>> >
>> > >>> > Current Settings:
>> > >>> > appName: TestQ08Task
>> > >>> > filesToStage: ...
>> > >>> > flinkMaster: [auto]
>> > >>> > parallelism: 1
>> > >>> > runner: class
>> org.apache.beam.runners.flink.FlinkPipelineRunner
>> > >>> > stableUniqueNames: WARNING
>> > >>> > streaming: false
>> > >>> >
>> > >>> > I noticed I was in batch mode, is there something missing in
>> the
>> > >>> > options
>> > >>> > configuration, or is it that the UnsoppertedOperation is a
>> > WIP or not
>> > >>> > supported at all by Flink ?
>> > >>> >
>> > >>> > And extra question, this option to define if it is in batch
>> > or stream
>> > >>> > mode
>> > >>> > is going to stay for long, can't be this inferred some how ?
>> > >>> >
>> > >>> > Thanks,
>> > >>> > -Ismaël
>> > >>> >
>> > >
>> > >
>> >
>> >
>>
>> --
>> Jean-Baptiste Onofré
>> jbonofre@apache.org
>> http://blog.nanthrax.net
>> Talend - http://www.talend.com
>>
>
Re: Problem with Pipeline In Flink Runner
Posted by Aljoscha Krettek <al...@apache.org>.
Hi,
you should be able to set the runner using command line options, e.g:
--runner=org.apache.beam.runners.flink.FlinkPipelineRunner
--streaming=false --parallelism=8
And in the program have something like:
Options options = PipelineOptionsFactory.fromArgs(args).withValidation();
-Aljoscha
On Fri, 20 May 2016 at 13:59 Jean-Baptiste Onofré <jb...@nanthrax.net> wrote:
> You can always do it via System, populating the options, right ?
>
> Regards
> JB
>
> On 05/20/2016 11:07 AM, Ismaël Mejía wrote:
> > Hello again,
> >
> > I just executed our pipeline with the changes you just merged, and guess
> > what ? It works perfectly (in batch mode), amazing job guys. I have just
> one
> > extra question, is there a way to configure the pipeline options with
> system
> > properties (-D...). e.g. to configure the streaming/batch mode and
> > avoid to do
> > this explicitly in the code (to remove the explicit FlinkPipelineOptions
> > import) ?
> >
> > Ismaël
> >
> >
> > On Fri, May 20, 2016 at 10:06 AM, Maximilian Michels <mxm@apache.org
> > <ma...@apache.org>> wrote:
> >
> > Hi Ismael,
> >
> > Side inputs are not supported when the Flink Runner runs in streaming
> > mode. Actually, we just merged changes which allow you to use the
> > batch mode (streaming set to false) with Windows and side inputs.
> >
> > So just checkout the latest Beam master and rebuild. Set the
> streaming
> > flag to false again and rerun your example.
> >
> > Thanks,
> > Max
> >
> > On Fri, May 20, 2016 at 9:22 AM, Ismaël Mejía <iemejia@gmail.com
> > <ma...@gmail.com>> wrote:
> > >
> > > Hello,
> > >
> > > Thanks for your answer, I am really happy to know you are working
> on
> > > removing
> > > this flag for batch/streaming since this is not really Beam-like.
> > >
> > > I tried to execute the pipeline again in streaming mode, and I
> > got this
> > > exception now:
> > >
> > > java.lang.UnsupportedOperationException: The transform
> > > View.CreatePCollectionView is currently not supported.
> > >
> > > Any info about this one ?
> > >
> > > Thanks again,
> > > Ismael
> > >
> > >
> > > On Thu, May 19, 2016 at 7:31 PM, Aljoscha Krettek
> > <aljoscha@apache.org <ma...@apache.org>>
> > > wrote:
> > >>
> > >> Hi,
> > >> as an addition, there are plans to get rid of the "streaming"
> > flag for all
> > >> runners and determine the mode from the job.
> > >>
> > >> -Aljoscha
> > >>
> > >> On Thu, 19 May 2016 at 18:52 Maximilian Michels <mxm@apache.org
> > <ma...@apache.org>> wrote:
> > >>>
> > >>> Hi Ismaël,
> > >>>
> > >>> There is a pull request which enables Windows on batch. It will
> be
> > >>> merged soon. In the meantime, please use the streaming
> > execution mode
> > >>> of the Flink Runner.
> > >>>
> > >>> Use setStreaming(true) to enable it on the FlinkPipelineOptions.
> > >>>
> > >>> Cheers,
> > >>> Max
> > >>>
> > >>> On Thu, May 19, 2016 at 6:32 PM, Ismaël Mejía
> > <iemejia@gmail.com <ma...@gmail.com>> wrote:
> > >>> > I was trying to run a local Pipeline we have (basically a
> > batch one we
> > >>> > rewrote from TPC-H) in all the possible runners, for the
> > moment I had
> > >>> > success with the DirectPipelineRunner,
> > InProcessPipelineRunner and
> > >>> > SparkPipelineRunner. However when I executed it in the
> > >>> > FlinkPipelineRunner I
> > >>> > got this exception:
> > >>> >
> > >>> > java.lang.UnsupportedOperationException: The transform
> > Window.Into() is
> > >>> > currently not supported.
> > >>> >
> > >>> > I thought it was probably a configuration error since I
> > remember that
> > >>> > in the
> > >>> > flink runner we had to explicitly define if it was a batch or
> > streaming
> > >>> > job,
> > >>> > but when I checked the options:
> > >>> >
> > >>> > Current Settings:
> > >>> > appName: TestQ08Task
> > >>> > filesToStage: ...
> > >>> > flinkMaster: [auto]
> > >>> > parallelism: 1
> > >>> > runner: class
> org.apache.beam.runners.flink.FlinkPipelineRunner
> > >>> > stableUniqueNames: WARNING
> > >>> > streaming: false
> > >>> >
> > >>> > I noticed I was in batch mode, is there something missing in
> the
> > >>> > options
> > >>> > configuration, or is it that the UnsoppertedOperation is a
> > WIP or not
> > >>> > supported at all by Flink ?
> > >>> >
> > >>> > And extra question, this option to define if it is in batch
> > or stream
> > >>> > mode
> > >>> > is going to stay for long, can't be this inferred some how ?
> > >>> >
> > >>> > Thanks,
> > >>> > -Ismaël
> > >>> >
> > >
> > >
> >
> >
>
> --
> Jean-Baptiste Onofré
> jbonofre@apache.org
> http://blog.nanthrax.net
> Talend - http://www.talend.com
>
Re: Problem with Pipeline In Flink Runner
Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.
You can always do it via System, populating the options, right ?
Regards
JB
On 05/20/2016 11:07 AM, Isma�l Mej�a wrote:
> Hello again,
>
> I just executed our pipeline with the changes you just merged, and guess
> what ? It works perfectly (in batch mode), amazing job guys. I have just one
> extra question, is there a way to configure the pipeline options with system
> properties (-D...). e.g. to configure the streaming/batch mode and
> avoid to do
> this explicitly in the code (to remove the explicit FlinkPipelineOptions
> import) ?
>
> Isma�l
>
>
> On Fri, May 20, 2016 at 10:06 AM, Maximilian Michels <mxm@apache.org
> <ma...@apache.org>> wrote:
>
> Hi Ismael,
>
> Side inputs are not supported when the Flink Runner runs in streaming
> mode. Actually, we just merged changes which allow you to use the
> batch mode (streaming set to false) with Windows and side inputs.
>
> So just checkout the latest Beam master and rebuild. Set the streaming
> flag to false again and rerun your example.
>
> Thanks,
> Max
>
> On Fri, May 20, 2016 at 9:22 AM, Isma�l Mej�a <iemejia@gmail.com
> <ma...@gmail.com>> wrote:
> >
> > Hello,
> >
> > Thanks for your answer, I am really happy to know you are working on
> > removing
> > this flag for batch/streaming since this is not really Beam-like.
> >
> > I tried to execute the pipeline again in streaming mode, and I
> got this
> > exception now:
> >
> > java.lang.UnsupportedOperationException: The transform
> > View.CreatePCollectionView is currently not supported.
> >
> > Any info about this one ?
> >
> > Thanks again,
> > Ismael
> >
> >
> > On Thu, May 19, 2016 at 7:31 PM, Aljoscha Krettek
> <aljoscha@apache.org <ma...@apache.org>>
> > wrote:
> >>
> >> Hi,
> >> as an addition, there are plans to get rid of the "streaming"
> flag for all
> >> runners and determine the mode from the job.
> >>
> >> -Aljoscha
> >>
> >> On Thu, 19 May 2016 at 18:52 Maximilian Michels <mxm@apache.org
> <ma...@apache.org>> wrote:
> >>>
> >>> Hi Isma�l,
> >>>
> >>> There is a pull request which enables Windows on batch. It will be
> >>> merged soon. In the meantime, please use the streaming
> execution mode
> >>> of the Flink Runner.
> >>>
> >>> Use setStreaming(true) to enable it on the FlinkPipelineOptions.
> >>>
> >>> Cheers,
> >>> Max
> >>>
> >>> On Thu, May 19, 2016 at 6:32 PM, Isma�l Mej�a
> <iemejia@gmail.com <ma...@gmail.com>> wrote:
> >>> > I was trying to run a local Pipeline we have (basically a
> batch one we
> >>> > rewrote from TPC-H) in all the possible runners, for the
> moment I had
> >>> > success with the DirectPipelineRunner,
> InProcessPipelineRunner and
> >>> > SparkPipelineRunner. However when I executed it in the
> >>> > FlinkPipelineRunner I
> >>> > got this exception:
> >>> >
> >>> > java.lang.UnsupportedOperationException: The transform
> Window.Into() is
> >>> > currently not supported.
> >>> >
> >>> > I thought it was probably a configuration error since I
> remember that
> >>> > in the
> >>> > flink runner we had to explicitly define if it was a batch or
> streaming
> >>> > job,
> >>> > but when I checked the options:
> >>> >
> >>> > Current Settings:
> >>> > appName: TestQ08Task
> >>> > filesToStage: ...
> >>> > flinkMaster: [auto]
> >>> > parallelism: 1
> >>> > runner: class org.apache.beam.runners.flink.FlinkPipelineRunner
> >>> > stableUniqueNames: WARNING
> >>> > streaming: false
> >>> >
> >>> > I noticed I was in batch mode, is there something missing in the
> >>> > options
> >>> > configuration, or is it that the UnsoppertedOperation is a
> WIP or not
> >>> > supported at all by Flink ?
> >>> >
> >>> > And extra question, this option to define if it is in batch
> or stream
> >>> > mode
> >>> > is going to stay for long, can't be this inferred some how ?
> >>> >
> >>> > Thanks,
> >>> > -Isma�l
> >>> >
> >
> >
>
>
--
Jean-Baptiste Onofr�
jbonofre@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com
Re: Problem with Pipeline In Flink Runner
Posted by Ismaël Mejía <ie...@gmail.com>.
Hello again,
I just executed our pipeline with the changes you just merged, and guess
what ? It works perfectly (in batch mode), amazing job guys. I have just one
extra question, is there a way to configure the pipeline options with system
properties (-D...). e.g. to configure the streaming/batch mode and avoid
to do
this explicitly in the code (to remove the explicit FlinkPipelineOptions
import) ?
Ismaël
On Fri, May 20, 2016 at 10:06 AM, Maximilian Michels <mx...@apache.org> wrote:
> Hi Ismael,
>
> Side inputs are not supported when the Flink Runner runs in streaming
> mode. Actually, we just merged changes which allow you to use the
> batch mode (streaming set to false) with Windows and side inputs.
>
> So just checkout the latest Beam master and rebuild. Set the streaming
> flag to false again and rerun your example.
>
> Thanks,
> Max
>
> On Fri, May 20, 2016 at 9:22 AM, Ismaël Mejía <ie...@gmail.com> wrote:
> >
> > Hello,
> >
> > Thanks for your answer, I am really happy to know you are working on
> > removing
> > this flag for batch/streaming since this is not really Beam-like.
> >
> > I tried to execute the pipeline again in streaming mode, and I got this
> > exception now:
> >
> > java.lang.UnsupportedOperationException: The transform
> > View.CreatePCollectionView is currently not supported.
> >
> > Any info about this one ?
> >
> > Thanks again,
> > Ismael
> >
> >
> > On Thu, May 19, 2016 at 7:31 PM, Aljoscha Krettek <al...@apache.org>
> > wrote:
> >>
> >> Hi,
> >> as an addition, there are plans to get rid of the "streaming" flag for
> all
> >> runners and determine the mode from the job.
> >>
> >> -Aljoscha
> >>
> >> On Thu, 19 May 2016 at 18:52 Maximilian Michels <mx...@apache.org> wrote:
> >>>
> >>> Hi Ismaël,
> >>>
> >>> There is a pull request which enables Windows on batch. It will be
> >>> merged soon. In the meantime, please use the streaming execution mode
> >>> of the Flink Runner.
> >>>
> >>> Use setStreaming(true) to enable it on the FlinkPipelineOptions.
> >>>
> >>> Cheers,
> >>> Max
> >>>
> >>> On Thu, May 19, 2016 at 6:32 PM, Ismaël Mejía <ie...@gmail.com>
> wrote:
> >>> > I was trying to run a local Pipeline we have (basically a batch one
> we
> >>> > rewrote from TPC-H) in all the possible runners, for the moment I
> had
> >>> > success with the DirectPipelineRunner, InProcessPipelineRunner and
> >>> > SparkPipelineRunner. However when I executed it in the
> >>> > FlinkPipelineRunner I
> >>> > got this exception:
> >>> >
> >>> > java.lang.UnsupportedOperationException: The transform Window.Into()
> is
> >>> > currently not supported.
> >>> >
> >>> > I thought it was probably a configuration error since I remember that
> >>> > in the
> >>> > flink runner we had to explicitly define if it was a batch or
> streaming
> >>> > job,
> >>> > but when I checked the options:
> >>> >
> >>> > Current Settings:
> >>> > appName: TestQ08Task
> >>> > filesToStage: ...
> >>> > flinkMaster: [auto]
> >>> > parallelism: 1
> >>> > runner: class org.apache.beam.runners.flink.FlinkPipelineRunner
> >>> > stableUniqueNames: WARNING
> >>> > streaming: false
> >>> >
> >>> > I noticed I was in batch mode, is there something missing in the
> >>> > options
> >>> > configuration, or is it that the UnsoppertedOperation is a WIP or not
> >>> > supported at all by Flink ?
> >>> >
> >>> > And extra question, this option to define if it is in batch or stream
> >>> > mode
> >>> > is going to stay for long, can't be this inferred some how ?
> >>> >
> >>> > Thanks,
> >>> > -Ismaël
> >>> >
> >
> >
>
Re: Problem with Pipeline In Flink Runner
Posted by Maximilian Michels <mx...@apache.org>.
Hi Ismael,
Side inputs are not supported when the Flink Runner runs in streaming
mode. Actually, we just merged changes which allow you to use the
batch mode (streaming set to false) with Windows and side inputs.
So just checkout the latest Beam master and rebuild. Set the streaming
flag to false again and rerun your example.
Thanks,
Max
On Fri, May 20, 2016 at 9:22 AM, Ismaël Mejía <ie...@gmail.com> wrote:
>
> Hello,
>
> Thanks for your answer, I am really happy to know you are working on
> removing
> this flag for batch/streaming since this is not really Beam-like.
>
> I tried to execute the pipeline again in streaming mode, and I got this
> exception now:
>
> java.lang.UnsupportedOperationException: The transform
> View.CreatePCollectionView is currently not supported.
>
> Any info about this one ?
>
> Thanks again,
> Ismael
>
>
> On Thu, May 19, 2016 at 7:31 PM, Aljoscha Krettek <al...@apache.org>
> wrote:
>>
>> Hi,
>> as an addition, there are plans to get rid of the "streaming" flag for all
>> runners and determine the mode from the job.
>>
>> -Aljoscha
>>
>> On Thu, 19 May 2016 at 18:52 Maximilian Michels <mx...@apache.org> wrote:
>>>
>>> Hi Ismaël,
>>>
>>> There is a pull request which enables Windows on batch. It will be
>>> merged soon. In the meantime, please use the streaming execution mode
>>> of the Flink Runner.
>>>
>>> Use setStreaming(true) to enable it on the FlinkPipelineOptions.
>>>
>>> Cheers,
>>> Max
>>>
>>> On Thu, May 19, 2016 at 6:32 PM, Ismaël Mejía <ie...@gmail.com> wrote:
>>> > I was trying to run a local Pipeline we have (basically a batch one we
>>> > rewrote from TPC-H) in all the possible runners, for the moment I had
>>> > success with the DirectPipelineRunner, InProcessPipelineRunner and
>>> > SparkPipelineRunner. However when I executed it in the
>>> > FlinkPipelineRunner I
>>> > got this exception:
>>> >
>>> > java.lang.UnsupportedOperationException: The transform Window.Into() is
>>> > currently not supported.
>>> >
>>> > I thought it was probably a configuration error since I remember that
>>> > in the
>>> > flink runner we had to explicitly define if it was a batch or streaming
>>> > job,
>>> > but when I checked the options:
>>> >
>>> > Current Settings:
>>> > appName: TestQ08Task
>>> > filesToStage: ...
>>> > flinkMaster: [auto]
>>> > parallelism: 1
>>> > runner: class org.apache.beam.runners.flink.FlinkPipelineRunner
>>> > stableUniqueNames: WARNING
>>> > streaming: false
>>> >
>>> > I noticed I was in batch mode, is there something missing in the
>>> > options
>>> > configuration, or is it that the UnsoppertedOperation is a WIP or not
>>> > supported at all by Flink ?
>>> >
>>> > And extra question, this option to define if it is in batch or stream
>>> > mode
>>> > is going to stay for long, can't be this inferred some how ?
>>> >
>>> > Thanks,
>>> > -Ismaël
>>> >
>
>
Re: Problem with Pipeline In Flink Runner
Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.
AFAIK, the side input is not yet supported in Flink runner in streaming
mode. The Spark runner doesn't support it neither ;)
I think Max is working on it for Flink runner. I will work with Amit on
Spark runner.
Regards
JB
On 05/20/2016 09:22 AM, Isma�l Mej�a wrote:
>
> Hello,
>
> Thanks for your answer, I am really happy to know you are working on
> removing
> this flag for batch/streaming since this is not really Beam-like.
>
> I tried to execute the pipeline again in streaming mode, and I got this
> exception now:
>
> java.lang.UnsupportedOperationException: The transform
> View.CreatePCollectionView is currently not supported.
>
> Any info about this one ?
>
> Thanks again,
> Ismael
>
>
> On Thu, May 19, 2016 at 7:31 PM, Aljoscha Krettek <aljoscha@apache.org
> <ma...@apache.org>> wrote:
>
> Hi,
> as an addition, there are plans to get rid of the "streaming" flag
> for all runners and determine the mode from the job.
>
> -Aljoscha
>
> On Thu, 19 May 2016 at 18:52 Maximilian Michels <mxm@apache.org
> <ma...@apache.org>> wrote:
>
> Hi Isma�l,
>
> There is a pull request which enables Windows on batch. It will be
> merged soon. In the meantime, please use the streaming execution
> mode
> of the Flink Runner.
>
> Use setStreaming(true) to enable it on the FlinkPipelineOptions.
>
> Cheers,
> Max
>
> On Thu, May 19, 2016 at 6:32 PM, Isma�l Mej�a <iemejia@gmail.com
> <ma...@gmail.com>> wrote:
> > I was trying to run a local Pipeline we have (basically a
> batch one we
> > rewrote from TPC-H) in all the possible runners, for the
> moment I had
> > success with the DirectPipelineRunner,
> InProcessPipelineRunner and
> > SparkPipelineRunner. However when I executed it in the
> FlinkPipelineRunner I
> > got this exception:
> >
> > java.lang.UnsupportedOperationException: The transform
> Window.Into() is
> > currently not supported.
> >
> > I thought it was probably a configuration error since I
> remember that in the
> > flink runner we had to explicitly define if it was a batch or
> streaming job,
> > but when I checked the options:
> >
> > Current Settings:
> > appName: TestQ08Task
> > filesToStage: ...
> > flinkMaster: [auto]
> > parallelism: 1
> > runner: class org.apache.beam.runners.flink.FlinkPipelineRunner
> > stableUniqueNames: WARNING
> > streaming: false
> >
> > I noticed I was in batch mode, is there something missing in
> the options
> > configuration, or is it that the UnsoppertedOperation is a
> WIP or not
> > supported at all by Flink ?
> >
> > And extra question, this option to define if it is in batch
> or stream mode
> > is going to stay for long, can't be this inferred some how ?
> >
> > Thanks,
> > -Isma�l
> >
>
>
--
Jean-Baptiste Onofr�
jbonofre@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com
Re: Problem with Pipeline In Flink Runner
Posted by Ismaël Mejía <ie...@gmail.com>.
Hello,
Thanks for your answer, I am really happy to know you are working on
removing
this flag for batch/streaming since this is not really Beam-like.
I tried to execute the pipeline again in streaming mode, and I got this
exception now:
java.lang.UnsupportedOperationException: The transform
View.CreatePCollectionView is currently not supported.
Any info about this one ?
Thanks again,
Ismael
On Thu, May 19, 2016 at 7:31 PM, Aljoscha Krettek <al...@apache.org>
wrote:
> Hi,
> as an addition, there are plans to get rid of the "streaming" flag for all
> runners and determine the mode from the job.
>
> -Aljoscha
>
> On Thu, 19 May 2016 at 18:52 Maximilian Michels <mx...@apache.org> wrote:
>
>> Hi Ismaël,
>>
>> There is a pull request which enables Windows on batch. It will be
>> merged soon. In the meantime, please use the streaming execution mode
>> of the Flink Runner.
>>
>> Use setStreaming(true) to enable it on the FlinkPipelineOptions.
>>
>> Cheers,
>> Max
>>
>> On Thu, May 19, 2016 at 6:32 PM, Ismaël Mejía <ie...@gmail.com> wrote:
>> > I was trying to run a local Pipeline we have (basically a batch one we
>> > rewrote from TPC-H) in all the possible runners, for the moment I had
>> > success with the DirectPipelineRunner, InProcessPipelineRunner and
>> > SparkPipelineRunner. However when I executed it in the
>> FlinkPipelineRunner I
>> > got this exception:
>> >
>> > java.lang.UnsupportedOperationException: The transform Window.Into() is
>> > currently not supported.
>> >
>> > I thought it was probably a configuration error since I remember that
>> in the
>> > flink runner we had to explicitly define if it was a batch or streaming
>> job,
>> > but when I checked the options:
>> >
>> > Current Settings:
>> > appName: TestQ08Task
>> > filesToStage: ...
>> > flinkMaster: [auto]
>> > parallelism: 1
>> > runner: class org.apache.beam.runners.flink.FlinkPipelineRunner
>> > stableUniqueNames: WARNING
>> > streaming: false
>> >
>> > I noticed I was in batch mode, is there something missing in the options
>> > configuration, or is it that the UnsoppertedOperation is a WIP or not
>> > supported at all by Flink ?
>> >
>> > And extra question, this option to define if it is in batch or stream
>> mode
>> > is going to stay for long, can't be this inferred some how ?
>> >
>> > Thanks,
>> > -Ismaël
>> >
>>
>
Re: Problem with Pipeline In Flink Runner
Posted by Aljoscha Krettek <al...@apache.org>.
Hi,
as an addition, there are plans to get rid of the "streaming" flag for all
runners and determine the mode from the job.
-Aljoscha
On Thu, 19 May 2016 at 18:52 Maximilian Michels <mx...@apache.org> wrote:
> Hi Ismaël,
>
> There is a pull request which enables Windows on batch. It will be
> merged soon. In the meantime, please use the streaming execution mode
> of the Flink Runner.
>
> Use setStreaming(true) to enable it on the FlinkPipelineOptions.
>
> Cheers,
> Max
>
> On Thu, May 19, 2016 at 6:32 PM, Ismaël Mejía <ie...@gmail.com> wrote:
> > I was trying to run a local Pipeline we have (basically a batch one we
> > rewrote from TPC-H) in all the possible runners, for the moment I had
> > success with the DirectPipelineRunner, InProcessPipelineRunner and
> > SparkPipelineRunner. However when I executed it in the
> FlinkPipelineRunner I
> > got this exception:
> >
> > java.lang.UnsupportedOperationException: The transform Window.Into() is
> > currently not supported.
> >
> > I thought it was probably a configuration error since I remember that in
> the
> > flink runner we had to explicitly define if it was a batch or streaming
> job,
> > but when I checked the options:
> >
> > Current Settings:
> > appName: TestQ08Task
> > filesToStage: ...
> > flinkMaster: [auto]
> > parallelism: 1
> > runner: class org.apache.beam.runners.flink.FlinkPipelineRunner
> > stableUniqueNames: WARNING
> > streaming: false
> >
> > I noticed I was in batch mode, is there something missing in the options
> > configuration, or is it that the UnsoppertedOperation is a WIP or not
> > supported at all by Flink ?
> >
> > And extra question, this option to define if it is in batch or stream
> mode
> > is going to stay for long, can't be this inferred some how ?
> >
> > Thanks,
> > -Ismaël
> >
>
Re: Problem with Pipeline In Flink Runner
Posted by Maximilian Michels <mx...@apache.org>.
Hi Ismaël,
There is a pull request which enables Windows on batch. It will be
merged soon. In the meantime, please use the streaming execution mode
of the Flink Runner.
Use setStreaming(true) to enable it on the FlinkPipelineOptions.
Cheers,
Max
On Thu, May 19, 2016 at 6:32 PM, Ismaël Mejía <ie...@gmail.com> wrote:
> I was trying to run a local Pipeline we have (basically a batch one we
> rewrote from TPC-H) in all the possible runners, for the moment I had
> success with the DirectPipelineRunner, InProcessPipelineRunner and
> SparkPipelineRunner. However when I executed it in the FlinkPipelineRunner I
> got this exception:
>
> java.lang.UnsupportedOperationException: The transform Window.Into() is
> currently not supported.
>
> I thought it was probably a configuration error since I remember that in the
> flink runner we had to explicitly define if it was a batch or streaming job,
> but when I checked the options:
>
> Current Settings:
> appName: TestQ08Task
> filesToStage: ...
> flinkMaster: [auto]
> parallelism: 1
> runner: class org.apache.beam.runners.flink.FlinkPipelineRunner
> stableUniqueNames: WARNING
> streaming: false
>
> I noticed I was in batch mode, is there something missing in the options
> configuration, or is it that the UnsoppertedOperation is a WIP or not
> supported at all by Flink ?
>
> And extra question, this option to define if it is in batch or stream mode
> is going to stay for long, can't be this inferred some how ?
>
> Thanks,
> -Ismaël
>