You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@beam.apache.org by Will Walters <ww...@yahoo-inc.com> on 2017/06/28 20:39:55 UTC
Recommended Flink Version
Hello,
I've been attempting to run Beam through Flink on a Yarn cluster and have run into trouble with getting a job to submit, partly because of incompatibility between versions. Does anyone know what versions of Beam and Flink I should be using to give myself the best chance of finding compatibility?
Thank you,Will.
Re: Recommended Flink Version
Posted by Aljoscha Krettek <al...@apache.org>.
I just tried it with the Beam Quickstart form the website and a fresh Flink 1.2.1 download. This is the command I used for building the Quickstart:
mvn package -DskipTests -Pflink-runner
(The important bit here is -Pflink-runner which brings in the Flink Runner and its dependencies)
For running I did this:
bin/start-cluster.sh
bin/flink run -c org.apache.beam.examples.WordCount ../word-count-beam/target/word-count-beam-0.1.jar --inputFile=/Users/aljoscha/Downloads/word-count-beam/pom.xml --output=/tmp/counts --runner=FlinkRunner
Best,
Aljoscha
> On 30. Jun 2017, at 01:02, Will Walters <ww...@yahoo-inc.com> wrote:
>
> As for creating a fat jar, I've encountered difficulty with this as well. When attempting to compile (with Maven, using the package command) the Beam master to a jar, it instead creates several separate jars in each subfolder. And without messing with the pom file, it fails because of missing dependencies. I've tried to get around this by altering the pom file to include all possible dependencies, but even after doing this, the job submission will still fail, citing some missing dependency.
>
> Any advice on how to create a working fat jar?
>
> Thanks,
> Will.
>
> On Thursday, June 29, 2017 1:28 AM, Jean-Baptiste Onofré <jb...@nanthrax.net> wrote:
>
>
> Good point, fair enough.
>
> Regards
> JB
>
> On 06/29/2017 10:26 AM, Aljoscha Krettek wrote:
> > I think it’s a bug because if you start a Flink cluster on bare-metal it works, just when it’s started in YARN it doesn’t. And I feel that the way you start your cluster should not affect how you can submit jobs to it.
> >
> > Best,
> > Aljoscha
> >
> >> On 29. Jun 2017, at 10:15, Jean-Baptiste Onofré <jb@nanthrax.net <ma...@nanthrax.net>> wrote:
> >>
> >> Yes, it's the same with the spark runner using bin/spark-submit. From my standpoint, it's not a bug, it's a feature request.
> >>
> >> Regards
> >> JB
> >>
> >> On 06/29/2017 10:12 AM, Aljoscha Krettek wrote:
> >>> I also responded to a separate mail by Will. The problem is that currently we cannot submit a job using the remote client to a Flink cluster that was started on YARN. (It’s a bug or “feature” of how communication with a Flink cluster from a client works.)
> >>> The workaround for that is to use the bin/flink command to submit a Beam fat-jar on a Flink YARN cluster.
> >>> Best,
> >>> Aljoscha
> >>>> On 29. Jun 2017, at 07:23, Jean-Baptiste Onofré <jb@nanthrax.net <ma...@nanthrax.net>> wrote:
> >>>>
> >>>> Hi Will,
> >>>>
> >>>> assuming you are using Beam 2.0.0, the Flink runner uses Flink 1.2.1 by default. So, I would recommend this version or 1.2.x.
> >>>>
> >>>> Regards
> >>>> JB
> >>>>
> >>>> On 06/28/2017 10:39 PM, Will Walters wrote:
> >>>>> Hello,
> >>>>> I've been attempting to run Beam through Flink on a Yarn cluster and have run into trouble with getting a job to submit, partly because of incompatibility between versions. Does anyone know what versions of Beam and Flink I should be using to give myself the best chance of finding compatibility?
> >>>>> Thank you,
> >>>>> Will.
> >>>>
> >>>> --
> >>>> Jean-Baptiste Onofré
> >>>> jbonofre@apache.org <ma...@apache.org>
> >>>> http://blog.nanthrax.net <http://blog.nanthrax.net/>
> >>>> Talend - http://www.talend.com <http://www.talend.com/>
>
> >>
> >> --
> >> Jean-Baptiste Onofré
> >> jbonofre@apache.org <ma...@apache.org>
> >> http://blog.nanthrax.net <http://blog.nanthrax.net/>
> >> Talend - http://www.talend.com <http://www.talend.com/>
> >
>
> --
> Jean-Baptiste Onofré
> jbonofre@apache.org <ma...@apache.org>
> http://blog.nanthrax.net <http://blog.nanthrax.net/>
> Talend - http://www.talend.com <http://www.talend.com/>
>
>
Re: Recommended Flink Version
Posted by Will Walters <ww...@yahoo-inc.com>.
As for creating a fat jar, I've encountered difficulty with this as well. When attempting to compile (with Maven, using the package command) the Beam master to a jar, it instead creates several separate jars in each subfolder. And without messing with the pom file, it fails because of missing dependencies. I've tried to get around this by altering the pom file to include all possible dependencies, but even after doing this, the job submission will still fail, citing some missing dependency.
Any advice on how to create a working fat jar?
Thanks,Will.
On Thursday, June 29, 2017 1:28 AM, Jean-Baptiste Onofré <jb...@nanthrax.net> wrote:
Good point, fair enough.
Regards
JB
On 06/29/2017 10:26 AM, Aljoscha Krettek wrote:
> I think it’s a bug because if you start a Flink cluster on bare-metal it works, just when it’s started in YARN it doesn’t. And I feel that the way you start your cluster should not affect how you can submit jobs to it.
>
> Best,
> Aljoscha
>
>> On 29. Jun 2017, at 10:15, Jean-Baptiste Onofré <jb...@nanthrax.net> wrote:
>>
>> Yes, it's the same with the spark runner using bin/spark-submit. From my standpoint, it's not a bug, it's a feature request.
>>
>> Regards
>> JB
>>
>> On 06/29/2017 10:12 AM, Aljoscha Krettek wrote:
>>> I also responded to a separate mail by Will. The problem is that currently we cannot submit a job using the remote client to a Flink cluster that was started on YARN. (It’s a bug or “feature” of how communication with a Flink cluster from a client works.)
>>> The workaround for that is to use the bin/flink command to submit a Beam fat-jar on a Flink YARN cluster.
>>> Best,
>>> Aljoscha
>>>> On 29. Jun 2017, at 07:23, Jean-Baptiste Onofré <jb...@nanthrax.net> wrote:
>>>>
>>>> Hi Will,
>>>>
>>>> assuming you are using Beam 2.0.0, the Flink runner uses Flink 1.2.1 by default. So, I would recommend this version or 1.2.x.
>>>>
>>>> Regards
>>>> JB
>>>>
>>>> On 06/28/2017 10:39 PM, Will Walters wrote:
>>>>> Hello,
>>>>> I've been attempting to run Beam through Flink on a Yarn cluster and have run into trouble with getting a job to submit, partly because of incompatibility between versions. Does anyone know what versions of Beam and Flink I should be using to give myself the best chance of finding compatibility?
>>>>> Thank you,
>>>>> Will.
>>>>
>>>> --
>>>> Jean-Baptiste Onofré
>>>> jbonofre@apache.org
>>>> http://blog.nanthrax.net
>>>> Talend - http://www.talend.com
>>
>> --
>> Jean-Baptiste Onofré
>> jbonofre@apache.org
>> http://blog.nanthrax.net
>> Talend - http://www.talend.com
>
--
Jean-Baptiste Onofré
jbonofre@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com
Re: Recommended Flink Version
Posted by Will Walters <ww...@yahoo-inc.com>.
I believe so. By default it creates a filepath that can't be read when creating registrars later.
-Will
On Wednesday, July 12, 2017 1:02 AM, Aljoscha Krettek <al...@apache.org> wrote:
So this is a general bug in how Flink constructs the HADOOP_CONF_DIR path?
Best,Aljoscha
On 11. Jul 2017, at 20:14, Will Walters <ww...@yahoo-inc.com> wrote:
I've managed to solve the problem I was having, which was with Flink not properly finding my hdfs registrar. It turns out that in the config.sh file, it runs this line on the global variable $HADOOP_CONF_DIR:
if [ -d "$HADOOP_HOME/etc/hadoop" ]; then # Its Hadoop 2.2+ HADOOP_CONF_DIR="$HADOOP_CONF_DIR:$HADOOP_HOME/etc/hadoop"
This means that HADOOP_CONF_DIR is set to two filepaths concatenated with a colon. In the function which reads in the registrars, it passes this string into the File() constructor, which fails because the string isn't a valid filepath. Commenting out the line above solved the problem, allowing a successful submission.
Thanks for your help!Will
On Thursday, June 29, 2017 1:28 AM, Jean-Baptiste Onofré <jb...@nanthrax.net> wrote:
Good point, fair enough.
Regards
JB
On 06/29/2017 10:26 AM, Aljoscha Krettek wrote:
> I think it’s a bug because if you start a Flink cluster on bare-metal it works, just when it’s started in YARN it doesn’t. And I feel that the way you start your cluster should not affect how you can submit jobs to it.
>
> Best,
> Aljoscha
>
>> On 29. Jun 2017, at 10:15, Jean-Baptiste Onofré <jb...@nanthrax.net> wrote:
>>
>> Yes, it's the same with the spark runner using bin/spark-submit. From my standpoint, it's not a bug, it's a feature request.
>>
>> Regards
>> JB
>>
>> On 06/29/2017 10:12 AM, Aljoscha Krettek wrote:
>>> I also responded to a separate mail by Will. The problem is that currently we cannot submit a job using the remote client to a Flink cluster that was started on YARN. (It’s a bug or “feature” of how communication with a Flink cluster from a client works.)
>>> The workaround for that is to use the bin/flink command to submit a Beam fat-jar on a Flink YARN cluster.
>>> Best,
>>> Aljoscha
>>>> On 29. Jun 2017, at 07:23, Jean-Baptiste Onofré <jb...@nanthrax.net> wrote:
>>>>
>>>> Hi Will,
>>>>
>>>> assuming you are using Beam 2.0.0, the Flink runner uses Flink 1.2.1 by default. So, I would recommend this version or 1.2.x.
>>>>
>>>> Regards
>>>> JB
>>>>
>>>> On 06/28/2017 10:39 PM, Will Walters wrote:
>>>>> Hello,
>>>>> I've been attempting to run Beam through Flink on a Yarn cluster and have run into trouble with getting a job to submit, partly because of incompatibility between versions. Does anyone know what versions of Beam and Flink I should be using to give myself the best chance of finding compatibility?
>>>>> Thank you,
>>>>> Will.
>>>>
>>>> --
>>>> Jean-Baptiste Onofré
>>>> jbonofre@apache.org
>>>> http://blog.nanthrax.net
>>>> Talend - http://www.talend.com
>>
>> --
>> Jean-Baptiste Onofré
>> jbonofre@apache.org
>> http://blog.nanthrax.net
>> Talend - http://www.talend.com
>
--
Jean-Baptiste Onofré
jbonofre@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com
Re: Recommended Flink Version
Posted by Aljoscha Krettek <al...@apache.org>.
So this is a general bug in how Flink constructs the HADOOP_CONF_DIR path?
Best,
Aljoscha
> On 11. Jul 2017, at 20:14, Will Walters <ww...@yahoo-inc.com> wrote:
>
> I've managed to solve the problem I was having, which was with Flink not properly finding my hdfs registrar. It turns out that in the config.sh file, it runs this line on the global variable $HADOOP_CONF_DIR:
>
> if [ -d "$HADOOP_HOME/etc/hadoop" ]; then
> # Its Hadoop 2.2+
> HADOOP_CONF_DIR="$HADOOP_CONF_DIR:$HADOOP_HOME/etc/hadoop"
>
> This means that HADOOP_CONF_DIR is set to two filepaths concatenated with a colon. In the function which reads in the registrars, it passes this string into the File() constructor, which fails because the string isn't a valid filepath. Commenting out the line above solved the problem, allowing a successful submission.
>
> Thanks for your help!
> Will
>
>
> On Thursday, June 29, 2017 1:28 AM, Jean-Baptiste Onofré <jb...@nanthrax.net> wrote:
>
>
> Good point, fair enough.
>
> Regards
> JB
>
> On 06/29/2017 10:26 AM, Aljoscha Krettek wrote:
> > I think it’s a bug because if you start a Flink cluster on bare-metal it works, just when it’s started in YARN it doesn’t. And I feel that the way you start your cluster should not affect how you can submit jobs to it.
> >
> > Best,
> > Aljoscha
> >
> >> On 29. Jun 2017, at 10:15, Jean-Baptiste Onofré <jb@nanthrax.net <ma...@nanthrax.net>> wrote:
> >>
> >> Yes, it's the same with the spark runner using bin/spark-submit. From my standpoint, it's not a bug, it's a feature request.
> >>
> >> Regards
> >> JB
> >>
> >> On 06/29/2017 10:12 AM, Aljoscha Krettek wrote:
> >>> I also responded to a separate mail by Will. The problem is that currently we cannot submit a job using the remote client to a Flink cluster that was started on YARN. (It’s a bug or “feature” of how communication with a Flink cluster from a client works.)
> >>> The workaround for that is to use the bin/flink command to submit a Beam fat-jar on a Flink YARN cluster.
> >>> Best,
> >>> Aljoscha
> >>>> On 29. Jun 2017, at 07:23, Jean-Baptiste Onofré <jb@nanthrax.net <ma...@nanthrax.net>> wrote:
> >>>>
> >>>> Hi Will,
> >>>>
> >>>> assuming you are using Beam 2.0.0, the Flink runner uses Flink 1.2.1 by default. So, I would recommend this version or 1.2.x.
> >>>>
> >>>> Regards
> >>>> JB
> >>>>
> >>>> On 06/28/2017 10:39 PM, Will Walters wrote:
> >>>>> Hello,
> >>>>> I've been attempting to run Beam through Flink on a Yarn cluster and have run into trouble with getting a job to submit, partly because of incompatibility between versions. Does anyone know what versions of Beam and Flink I should be using to give myself the best chance of finding compatibility?
> >>>>> Thank you,
> >>>>> Will.
> >>>>
> >>>> --
> >>>> Jean-Baptiste Onofré
> >>>> jbonofre@apache.org <ma...@apache.org>
> >>>> http://blog.nanthrax.net <http://blog.nanthrax.net/>
> >>>> Talend - http://www.talend.com <http://www.talend.com/>
>
> >>
> >> --
> >> Jean-Baptiste Onofré
> >> jbonofre@apache.org <ma...@apache.org>
> >> http://blog.nanthrax.net <http://blog.nanthrax.net/>
> >> Talend - http://www.talend.com <http://www.talend.com/>
> >
>
> --
> Jean-Baptiste Onofré
> jbonofre@apache.org <ma...@apache.org>
> http://blog.nanthrax.net <http://blog.nanthrax.net/>
> Talend - http://www.talend.com <http://www.talend.com/>
>
>
Re: Recommended Flink Version
Posted by Will Walters <ww...@yahoo-inc.com>.
I've managed to solve the problem I was having, which was with Flink not properly finding my hdfs registrar. It turns out that in the config.sh file, it runs this line on the global variable $HADOOP_CONF_DIR:
if [ -d "$HADOOP_HOME/etc/hadoop" ]; then # Its Hadoop 2.2+ HADOOP_CONF_DIR="$HADOOP_CONF_DIR:$HADOOP_HOME/etc/hadoop"
This means that HADOOP_CONF_DIR is set to two filepaths concatenated with a colon. In the function which reads in the registrars, it passes this string into the File() constructor, which fails because the string isn't a valid filepath. Commenting out the line above solved the problem, allowing a successful submission.
Thanks for your help!Will
On Thursday, June 29, 2017 1:28 AM, Jean-Baptiste Onofré <jb...@nanthrax.net> wrote:
Good point, fair enough.
Regards
JB
On 06/29/2017 10:26 AM, Aljoscha Krettek wrote:
> I think it’s a bug because if you start a Flink cluster on bare-metal it works, just when it’s started in YARN it doesn’t. And I feel that the way you start your cluster should not affect how you can submit jobs to it.
>
> Best,
> Aljoscha
>
>> On 29. Jun 2017, at 10:15, Jean-Baptiste Onofré <jb...@nanthrax.net> wrote:
>>
>> Yes, it's the same with the spark runner using bin/spark-submit. From my standpoint, it's not a bug, it's a feature request.
>>
>> Regards
>> JB
>>
>> On 06/29/2017 10:12 AM, Aljoscha Krettek wrote:
>>> I also responded to a separate mail by Will. The problem is that currently we cannot submit a job using the remote client to a Flink cluster that was started on YARN. (It’s a bug or “feature” of how communication with a Flink cluster from a client works.)
>>> The workaround for that is to use the bin/flink command to submit a Beam fat-jar on a Flink YARN cluster.
>>> Best,
>>> Aljoscha
>>>> On 29. Jun 2017, at 07:23, Jean-Baptiste Onofré <jb...@nanthrax.net> wrote:
>>>>
>>>> Hi Will,
>>>>
>>>> assuming you are using Beam 2.0.0, the Flink runner uses Flink 1.2.1 by default. So, I would recommend this version or 1.2.x.
>>>>
>>>> Regards
>>>> JB
>>>>
>>>> On 06/28/2017 10:39 PM, Will Walters wrote:
>>>>> Hello,
>>>>> I've been attempting to run Beam through Flink on a Yarn cluster and have run into trouble with getting a job to submit, partly because of incompatibility between versions. Does anyone know what versions of Beam and Flink I should be using to give myself the best chance of finding compatibility?
>>>>> Thank you,
>>>>> Will.
>>>>
>>>> --
>>>> Jean-Baptiste Onofré
>>>> jbonofre@apache.org
>>>> http://blog.nanthrax.net
>>>> Talend - http://www.talend.com
>>
>> --
>> Jean-Baptiste Onofré
>> jbonofre@apache.org
>> http://blog.nanthrax.net
>> Talend - http://www.talend.com
>
--
Jean-Baptiste Onofré
jbonofre@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com
Re: Recommended Flink Version
Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.
Good point, fair enough.
Regards
JB
On 06/29/2017 10:26 AM, Aljoscha Krettek wrote:
> I think it’s a bug because if you start a Flink cluster on bare-metal it works, just when it’s started in YARN it doesn’t. And I feel that the way you start your cluster should not affect how you can submit jobs to it.
>
> Best,
> Aljoscha
>
>> On 29. Jun 2017, at 10:15, Jean-Baptiste Onofré <jb...@nanthrax.net> wrote:
>>
>> Yes, it's the same with the spark runner using bin/spark-submit. From my standpoint, it's not a bug, it's a feature request.
>>
>> Regards
>> JB
>>
>> On 06/29/2017 10:12 AM, Aljoscha Krettek wrote:
>>> I also responded to a separate mail by Will. The problem is that currently we cannot submit a job using the remote client to a Flink cluster that was started on YARN. (It’s a bug or “feature” of how communication with a Flink cluster from a client works.)
>>> The workaround for that is to use the bin/flink command to submit a Beam fat-jar on a Flink YARN cluster.
>>> Best,
>>> Aljoscha
>>>> On 29. Jun 2017, at 07:23, Jean-Baptiste Onofré <jb...@nanthrax.net> wrote:
>>>>
>>>> Hi Will,
>>>>
>>>> assuming you are using Beam 2.0.0, the Flink runner uses Flink 1.2.1 by default. So, I would recommend this version or 1.2.x.
>>>>
>>>> Regards
>>>> JB
>>>>
>>>> On 06/28/2017 10:39 PM, Will Walters wrote:
>>>>> Hello,
>>>>> I've been attempting to run Beam through Flink on a Yarn cluster and have run into trouble with getting a job to submit, partly because of incompatibility between versions. Does anyone know what versions of Beam and Flink I should be using to give myself the best chance of finding compatibility?
>>>>> Thank you,
>>>>> Will.
>>>>
>>>> --
>>>> Jean-Baptiste Onofré
>>>> jbonofre@apache.org
>>>> http://blog.nanthrax.net
>>>> Talend - http://www.talend.com
>>
>> --
>> Jean-Baptiste Onofré
>> jbonofre@apache.org
>> http://blog.nanthrax.net
>> Talend - http://www.talend.com
>
--
Jean-Baptiste Onofré
jbonofre@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com
Re: Recommended Flink Version
Posted by Aljoscha Krettek <al...@apache.org>.
I think it’s a bug because if you start a Flink cluster on bare-metal it works, just when it’s started in YARN it doesn’t. And I feel that the way you start your cluster should not affect how you can submit jobs to it.
Best,
Aljoscha
> On 29. Jun 2017, at 10:15, Jean-Baptiste Onofré <jb...@nanthrax.net> wrote:
>
> Yes, it's the same with the spark runner using bin/spark-submit. From my standpoint, it's not a bug, it's a feature request.
>
> Regards
> JB
>
> On 06/29/2017 10:12 AM, Aljoscha Krettek wrote:
>> I also responded to a separate mail by Will. The problem is that currently we cannot submit a job using the remote client to a Flink cluster that was started on YARN. (It’s a bug or “feature” of how communication with a Flink cluster from a client works.)
>> The workaround for that is to use the bin/flink command to submit a Beam fat-jar on a Flink YARN cluster.
>> Best,
>> Aljoscha
>>> On 29. Jun 2017, at 07:23, Jean-Baptiste Onofré <jb...@nanthrax.net> wrote:
>>>
>>> Hi Will,
>>>
>>> assuming you are using Beam 2.0.0, the Flink runner uses Flink 1.2.1 by default. So, I would recommend this version or 1.2.x.
>>>
>>> Regards
>>> JB
>>>
>>> On 06/28/2017 10:39 PM, Will Walters wrote:
>>>> Hello,
>>>> I've been attempting to run Beam through Flink on a Yarn cluster and have run into trouble with getting a job to submit, partly because of incompatibility between versions. Does anyone know what versions of Beam and Flink I should be using to give myself the best chance of finding compatibility?
>>>> Thank you,
>>>> Will.
>>>
>>> --
>>> Jean-Baptiste Onofré
>>> jbonofre@apache.org
>>> http://blog.nanthrax.net
>>> Talend - http://www.talend.com
>
> --
> Jean-Baptiste Onofré
> jbonofre@apache.org
> http://blog.nanthrax.net
> Talend - http://www.talend.com
Re: Recommended Flink Version
Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.
Yes, it's the same with the spark runner using bin/spark-submit. From my
standpoint, it's not a bug, it's a feature request.
Regards
JB
On 06/29/2017 10:12 AM, Aljoscha Krettek wrote:
> I also responded to a separate mail by Will. The problem is that currently we cannot submit a job using the remote client to a Flink cluster that was started on YARN. (It’s a bug or “feature” of how communication with a Flink cluster from a client works.)
>
> The workaround for that is to use the bin/flink command to submit a Beam fat-jar on a Flink YARN cluster.
>
> Best,
> Aljoscha
>
>> On 29. Jun 2017, at 07:23, Jean-Baptiste Onofré <jb...@nanthrax.net> wrote:
>>
>> Hi Will,
>>
>> assuming you are using Beam 2.0.0, the Flink runner uses Flink 1.2.1 by default. So, I would recommend this version or 1.2.x.
>>
>> Regards
>> JB
>>
>> On 06/28/2017 10:39 PM, Will Walters wrote:
>>> Hello,
>>> I've been attempting to run Beam through Flink on a Yarn cluster and have run into trouble with getting a job to submit, partly because of incompatibility between versions. Does anyone know what versions of Beam and Flink I should be using to give myself the best chance of finding compatibility?
>>> Thank you,
>>> Will.
>>
>> --
>> Jean-Baptiste Onofré
>> jbonofre@apache.org
>> http://blog.nanthrax.net
>> Talend - http://www.talend.com
>
--
Jean-Baptiste Onofré
jbonofre@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com
Re: Recommended Flink Version
Posted by Aljoscha Krettek <al...@apache.org>.
I also responded to a separate mail by Will. The problem is that currently we cannot submit a job using the remote client to a Flink cluster that was started on YARN. (It’s a bug or “feature” of how communication with a Flink cluster from a client works.)
The workaround for that is to use the bin/flink command to submit a Beam fat-jar on a Flink YARN cluster.
Best,
Aljoscha
> On 29. Jun 2017, at 07:23, Jean-Baptiste Onofré <jb...@nanthrax.net> wrote:
>
> Hi Will,
>
> assuming you are using Beam 2.0.0, the Flink runner uses Flink 1.2.1 by default. So, I would recommend this version or 1.2.x.
>
> Regards
> JB
>
> On 06/28/2017 10:39 PM, Will Walters wrote:
>> Hello,
>> I've been attempting to run Beam through Flink on a Yarn cluster and have run into trouble with getting a job to submit, partly because of incompatibility between versions. Does anyone know what versions of Beam and Flink I should be using to give myself the best chance of finding compatibility?
>> Thank you,
>> Will.
>
> --
> Jean-Baptiste Onofré
> jbonofre@apache.org
> http://blog.nanthrax.net
> Talend - http://www.talend.com
Re: Recommended Flink Version
Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.
Hi Will,
assuming you are using Beam 2.0.0, the Flink runner uses Flink 1.2.1 by default.
So, I would recommend this version or 1.2.x.
Regards
JB
On 06/28/2017 10:39 PM, Will Walters wrote:
> Hello,
>
> I've been attempting to run Beam through Flink on a Yarn cluster and have run
> into trouble with getting a job to submit, partly because of incompatibility
> between versions. Does anyone know what versions of Beam and Flink I should be
> using to give myself the best chance of finding compatibility?
>
> Thank you,
> Will.
--
Jean-Baptiste Onofré
jbonofre@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com