You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@beam.apache.org by Will Walters <ww...@yahoo-inc.com> on 2017/07/11 18:14:49 UTC
Re: Recommended Flink Version
I've managed to solve the problem I was having, which was with Flink not properly finding my hdfs registrar. It turns out that in the config.sh file, it runs this line on the global variable $HADOOP_CONF_DIR:
if [ -d "$HADOOP_HOME/etc/hadoop" ]; then # Its Hadoop 2.2+ HADOOP_CONF_DIR="$HADOOP_CONF_DIR:$HADOOP_HOME/etc/hadoop"
This means that HADOOP_CONF_DIR is set to two filepaths concatenated with a colon. In the function which reads in the registrars, it passes this string into the File() constructor, which fails because the string isn't a valid filepath. Commenting out the line above solved the problem, allowing a successful submission.
Thanks for your help!Will
On Thursday, June 29, 2017 1:28 AM, Jean-Baptiste Onofré <jb...@nanthrax.net> wrote:
Good point, fair enough.
Regards
JB
On 06/29/2017 10:26 AM, Aljoscha Krettek wrote:
> I think it’s a bug because if you start a Flink cluster on bare-metal it works, just when it’s started in YARN it doesn’t. And I feel that the way you start your cluster should not affect how you can submit jobs to it.
>
> Best,
> Aljoscha
>
>> On 29. Jun 2017, at 10:15, Jean-Baptiste Onofré <jb...@nanthrax.net> wrote:
>>
>> Yes, it's the same with the spark runner using bin/spark-submit. From my standpoint, it's not a bug, it's a feature request.
>>
>> Regards
>> JB
>>
>> On 06/29/2017 10:12 AM, Aljoscha Krettek wrote:
>>> I also responded to a separate mail by Will. The problem is that currently we cannot submit a job using the remote client to a Flink cluster that was started on YARN. (It’s a bug or “feature” of how communication with a Flink cluster from a client works.)
>>> The workaround for that is to use the bin/flink command to submit a Beam fat-jar on a Flink YARN cluster.
>>> Best,
>>> Aljoscha
>>>> On 29. Jun 2017, at 07:23, Jean-Baptiste Onofré <jb...@nanthrax.net> wrote:
>>>>
>>>> Hi Will,
>>>>
>>>> assuming you are using Beam 2.0.0, the Flink runner uses Flink 1.2.1 by default. So, I would recommend this version or 1.2.x.
>>>>
>>>> Regards
>>>> JB
>>>>
>>>> On 06/28/2017 10:39 PM, Will Walters wrote:
>>>>> Hello,
>>>>> I've been attempting to run Beam through Flink on a Yarn cluster and have run into trouble with getting a job to submit, partly because of incompatibility between versions. Does anyone know what versions of Beam and Flink I should be using to give myself the best chance of finding compatibility?
>>>>> Thank you,
>>>>> Will.
>>>>
>>>> --
>>>> Jean-Baptiste Onofré
>>>> jbonofre@apache.org
>>>> http://blog.nanthrax.net
>>>> Talend - http://www.talend.com
>>
>> --
>> Jean-Baptiste Onofré
>> jbonofre@apache.org
>> http://blog.nanthrax.net
>> Talend - http://www.talend.com
>
--
Jean-Baptiste Onofré
jbonofre@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com
Re: Recommended Flink Version
Posted by Will Walters <ww...@yahoo-inc.com>.
I believe so. By default it creates a filepath that can't be read when creating registrars later.
-Will
On Wednesday, July 12, 2017 1:02 AM, Aljoscha Krettek <al...@apache.org> wrote:
So this is a general bug in how Flink constructs the HADOOP_CONF_DIR path?
Best,Aljoscha
On 11. Jul 2017, at 20:14, Will Walters <ww...@yahoo-inc.com> wrote:
I've managed to solve the problem I was having, which was with Flink not properly finding my hdfs registrar. It turns out that in the config.sh file, it runs this line on the global variable $HADOOP_CONF_DIR:
if [ -d "$HADOOP_HOME/etc/hadoop" ]; then # Its Hadoop 2.2+ HADOOP_CONF_DIR="$HADOOP_CONF_DIR:$HADOOP_HOME/etc/hadoop"
This means that HADOOP_CONF_DIR is set to two filepaths concatenated with a colon. In the function which reads in the registrars, it passes this string into the File() constructor, which fails because the string isn't a valid filepath. Commenting out the line above solved the problem, allowing a successful submission.
Thanks for your help!Will
On Thursday, June 29, 2017 1:28 AM, Jean-Baptiste Onofré <jb...@nanthrax.net> wrote:
Good point, fair enough.
Regards
JB
On 06/29/2017 10:26 AM, Aljoscha Krettek wrote:
> I think it’s a bug because if you start a Flink cluster on bare-metal it works, just when it’s started in YARN it doesn’t. And I feel that the way you start your cluster should not affect how you can submit jobs to it.
>
> Best,
> Aljoscha
>
>> On 29. Jun 2017, at 10:15, Jean-Baptiste Onofré <jb...@nanthrax.net> wrote:
>>
>> Yes, it's the same with the spark runner using bin/spark-submit. From my standpoint, it's not a bug, it's a feature request.
>>
>> Regards
>> JB
>>
>> On 06/29/2017 10:12 AM, Aljoscha Krettek wrote:
>>> I also responded to a separate mail by Will. The problem is that currently we cannot submit a job using the remote client to a Flink cluster that was started on YARN. (It’s a bug or “feature” of how communication with a Flink cluster from a client works.)
>>> The workaround for that is to use the bin/flink command to submit a Beam fat-jar on a Flink YARN cluster.
>>> Best,
>>> Aljoscha
>>>> On 29. Jun 2017, at 07:23, Jean-Baptiste Onofré <jb...@nanthrax.net> wrote:
>>>>
>>>> Hi Will,
>>>>
>>>> assuming you are using Beam 2.0.0, the Flink runner uses Flink 1.2.1 by default. So, I would recommend this version or 1.2.x.
>>>>
>>>> Regards
>>>> JB
>>>>
>>>> On 06/28/2017 10:39 PM, Will Walters wrote:
>>>>> Hello,
>>>>> I've been attempting to run Beam through Flink on a Yarn cluster and have run into trouble with getting a job to submit, partly because of incompatibility between versions. Does anyone know what versions of Beam and Flink I should be using to give myself the best chance of finding compatibility?
>>>>> Thank you,
>>>>> Will.
>>>>
>>>> --
>>>> Jean-Baptiste Onofré
>>>> jbonofre@apache.org
>>>> http://blog.nanthrax.net
>>>> Talend - http://www.talend.com
>>
>> --
>> Jean-Baptiste Onofré
>> jbonofre@apache.org
>> http://blog.nanthrax.net
>> Talend - http://www.talend.com
>
--
Jean-Baptiste Onofré
jbonofre@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com
Re: Recommended Flink Version
Posted by Aljoscha Krettek <al...@apache.org>.
So this is a general bug in how Flink constructs the HADOOP_CONF_DIR path?
Best,
Aljoscha
> On 11. Jul 2017, at 20:14, Will Walters <ww...@yahoo-inc.com> wrote:
>
> I've managed to solve the problem I was having, which was with Flink not properly finding my hdfs registrar. It turns out that in the config.sh file, it runs this line on the global variable $HADOOP_CONF_DIR:
>
> if [ -d "$HADOOP_HOME/etc/hadoop" ]; then
> # Its Hadoop 2.2+
> HADOOP_CONF_DIR="$HADOOP_CONF_DIR:$HADOOP_HOME/etc/hadoop"
>
> This means that HADOOP_CONF_DIR is set to two filepaths concatenated with a colon. In the function which reads in the registrars, it passes this string into the File() constructor, which fails because the string isn't a valid filepath. Commenting out the line above solved the problem, allowing a successful submission.
>
> Thanks for your help!
> Will
>
>
> On Thursday, June 29, 2017 1:28 AM, Jean-Baptiste Onofré <jb...@nanthrax.net> wrote:
>
>
> Good point, fair enough.
>
> Regards
> JB
>
> On 06/29/2017 10:26 AM, Aljoscha Krettek wrote:
> > I think it’s a bug because if you start a Flink cluster on bare-metal it works, just when it’s started in YARN it doesn’t. And I feel that the way you start your cluster should not affect how you can submit jobs to it.
> >
> > Best,
> > Aljoscha
> >
> >> On 29. Jun 2017, at 10:15, Jean-Baptiste Onofré <jb@nanthrax.net <ma...@nanthrax.net>> wrote:
> >>
> >> Yes, it's the same with the spark runner using bin/spark-submit. From my standpoint, it's not a bug, it's a feature request.
> >>
> >> Regards
> >> JB
> >>
> >> On 06/29/2017 10:12 AM, Aljoscha Krettek wrote:
> >>> I also responded to a separate mail by Will. The problem is that currently we cannot submit a job using the remote client to a Flink cluster that was started on YARN. (It’s a bug or “feature” of how communication with a Flink cluster from a client works.)
> >>> The workaround for that is to use the bin/flink command to submit a Beam fat-jar on a Flink YARN cluster.
> >>> Best,
> >>> Aljoscha
> >>>> On 29. Jun 2017, at 07:23, Jean-Baptiste Onofré <jb@nanthrax.net <ma...@nanthrax.net>> wrote:
> >>>>
> >>>> Hi Will,
> >>>>
> >>>> assuming you are using Beam 2.0.0, the Flink runner uses Flink 1.2.1 by default. So, I would recommend this version or 1.2.x.
> >>>>
> >>>> Regards
> >>>> JB
> >>>>
> >>>> On 06/28/2017 10:39 PM, Will Walters wrote:
> >>>>> Hello,
> >>>>> I've been attempting to run Beam through Flink on a Yarn cluster and have run into trouble with getting a job to submit, partly because of incompatibility between versions. Does anyone know what versions of Beam and Flink I should be using to give myself the best chance of finding compatibility?
> >>>>> Thank you,
> >>>>> Will.
> >>>>
> >>>> --
> >>>> Jean-Baptiste Onofré
> >>>> jbonofre@apache.org <ma...@apache.org>
> >>>> http://blog.nanthrax.net <http://blog.nanthrax.net/>
> >>>> Talend - http://www.talend.com <http://www.talend.com/>
>
> >>
> >> --
> >> Jean-Baptiste Onofré
> >> jbonofre@apache.org <ma...@apache.org>
> >> http://blog.nanthrax.net <http://blog.nanthrax.net/>
> >> Talend - http://www.talend.com <http://www.talend.com/>
> >
>
> --
> Jean-Baptiste Onofré
> jbonofre@apache.org <ma...@apache.org>
> http://blog.nanthrax.net <http://blog.nanthrax.net/>
> Talend - http://www.talend.com <http://www.talend.com/>
>
>