You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@flink.apache.org by "LINZ, Arnaud" <AL...@bouyguestelecom.fr> on 2019/06/07 09:03:18 UTC

No yarn option in self-built flink version

Hello,

I am trying to build my own flink distribution with proper Cloudera dependencies.
Reading https://ci.apache.org/projects/flink/flink-docs-stable/flinkDev/building.html
I've done :
git clone https://github.com/apache/flink
cd flink
git checkout tags/release-1.8.0
$MAVEN_HOME/bin/mvn clean install -DskipTests -Pvendor-repos -Dhadoop.version=2.6.0-cdh5.16.1 -Dfast
cd flink-dist
$MAVEN_HOME/bin/mvn install -DskipTests -Pvendor-repos -Dhadoop.version=2.6.0-cdh5.16.1

Everything was successful.

However when running using flink-dist/target/flink-1.8.0-bin/flink-1.8.0
Running /bin/flink -h prints no yarn/Hadoop options.
And running
bin/flink run -m yarn-cluster -yn 4 -yjm 1024m -ytm 4096m ./examples/batch/WordCount.jar
Prints :
Could not build the program from JAR file.

Am I missing something?

Best regards,
Arnaud

________________________________

L'intégrité de ce message n'étant pas assurée sur internet, la société expéditrice ne peut être tenue responsable de son contenu ni de ses pièces jointes. Toute utilisation ou diffusion non autorisée est interdite. Si vous n'êtes pas destinataire de ce message, merci de le détruire et d'avertir l'expéditeur.

The integrity of this message cannot be guaranteed on the Internet. The company that sent this message cannot therefore be held liable for its content nor attachments. Any unauthorized use or dissemination is prohibited. If you are not the intended recipient of this message, then please delete it and notify the sender.

Re: No yarn option in self-built flink version

Posted by Ufuk Celebi <uc...@apache.org>.

@Arnaud: Turns out those examples are on purpose. As Chesnay pointed out in
the ticket, there are also cases where you don't necessarily want to bundle
the Hadoop dependency, but still want to set a version like that.


On Wed, Jun 12, 2019 at 9:32 AM Ufuk Celebi <uc...@apache.org> wrote:

> I created https://issues.apache.org/jira/browse/FLINK-12813 for
> this. @Arnaud: Would you be interested in opening a PR with a fix?
>
> – Ufuk
>
>
> On Tue, Jun 11, 2019 at 11:10 AM LINZ, Arnaud <AL...@bouyguestelecom.fr>
> wrote:
>
>> Hello,
>>
>>
>>
>> Thanks a lot, it works. However, may I suggest that you update the
>> documentation page :
>>
>>
>>
>> mvn clean install -DskipTests -Pvendor-repos -Dhadoop.version*=*2.6.0-cdh5.16.1
>>
>> has absolutely no interest if you don’t include hadoop, that’s why I
>> thought that -Pvendor-repos was including the -Pinclude-hadoop stated in
>> the above paragraph…
>>
>>
>>
>> Arnaud
>>
>>
>>
>> *De :* Ufuk Celebi <uc...@apache.org>
>> *Envoyé :* vendredi 7 juin 2019 12:00
>> *À :* LINZ, Arnaud <AL...@bouyguestelecom.fr>
>> *Cc :* user <us...@flink.apache.org>; chesnay@apache.org
>> *Objet :* Re: No yarn option in self-built flink version
>>
>>
>>
>> Hey Arnaud,
>>
>>
>>
>> I think you need to active the Hadoop profile via -Pinclude-hadoop (the
>> default was changed to not include Hadoop as far as I know).
>>
>>
>>
>> For more details, check out:
>>
>>
>> https://ci.apache.org/projects/flink/flink-docs-release-1.8/flinkDev/building.html#packaging-hadoop-into-the-flink-distribution
>>
>>
>>
>>
>>
>> If this does not work for you, I would wait for Chesnay's input (cc'd).
>>
>>
>>
>> – Ufuk
>>
>>
>>
>>
>>
>> On Fri, Jun 7, 2019 at 11:04 AM LINZ, Arnaud <AL...@bouyguestelecom.fr>
>> wrote:
>>
>> Hello,
>>
>> I am trying to build my own flink distribution with proper Cloudera
>> dependencies.
>> Reading
>> https://ci.apache.org/projects/flink/flink-docs-stable/flinkDev/building.html
>> I've done :
>> git clone https://github.com/apache/flink
>> cd flink
>> git checkout tags/release-1.8.0
>> $MAVEN_HOME/bin/mvn clean install -DskipTests -Pvendor-repos
>> -Dhadoop.version=2.6.0-cdh5.16.1 -Dfast
>> cd flink-dist
>> $MAVEN_HOME/bin/mvn install -DskipTests -Pvendor-repos
>> -Dhadoop.version=2.6.0-cdh5.16.1
>>
>> Everything was successful.
>>
>> However when running using flink-dist/target/flink-1.8.0-bin/flink-1.8.0
>> Running /bin/flink -h prints no yarn/Hadoop options.
>> And running
>> bin/flink run -m yarn-cluster -yn 4 -yjm 1024m -ytm 4096m
>> ./examples/batch/WordCount.jar
>> Prints :
>> Could not build the program from JAR file.
>>
>> Am I missing something?
>>
>> Best regards,
>> Arnaud
>>
>>
>> ________________________________
>>
>> L'intégrité de ce message n'étant pas assurée sur internet, la société
>> expéditrice ne peut être tenue responsable de son contenu ni de ses pièces
>> jointes. Toute utilisation ou diffusion non autorisée est interdite. Si
>> vous n'êtes pas destinataire de ce message, merci de le détruire et
>> d'avertir l'expéditeur.
>>
>> The integrity of this message cannot be guaranteed on the Internet. The
>> company that sent this message cannot therefore be held liable for its
>> content nor attachments. Any unauthorized use or dissemination is
>> prohibited. If you are not the intended recipient of this message, then
>> please delete it and notify the sender.
>>
>>

Re: No yarn option in self-built flink version

Posted by Ufuk Celebi <uc...@apache.org>.

I created https://issues.apache.org/jira/browse/FLINK-12813 for
this. @Arnaud: Would you be interested in opening a PR with a fix?

– Ufuk


On Tue, Jun 11, 2019 at 11:10 AM LINZ, Arnaud <AL...@bouyguestelecom.fr>
wrote:

> Hello,
>
>
>
> Thanks a lot, it works. However, may I suggest that you update the
> documentation page :
>
>
>
> mvn clean install -DskipTests -Pvendor-repos -Dhadoop.version*=*2.6.0-cdh5.16.1
>
> has absolutely no interest if you don’t include hadoop, that’s why I
> thought that -Pvendor-repos was including the -Pinclude-hadoop stated in
> the above paragraph…
>
>
>
> Arnaud
>
>
>
> *De :* Ufuk Celebi <uc...@apache.org>
> *Envoyé :* vendredi 7 juin 2019 12:00
> *À :* LINZ, Arnaud <AL...@bouyguestelecom.fr>
> *Cc :* user <us...@flink.apache.org>; chesnay@apache.org
> *Objet :* Re: No yarn option in self-built flink version
>
>
>
> Hey Arnaud,
>
>
>
> I think you need to active the Hadoop profile via -Pinclude-hadoop (the
> default was changed to not include Hadoop as far as I know).
>
>
>
> For more details, check out:
>
>
> https://ci.apache.org/projects/flink/flink-docs-release-1.8/flinkDev/building.html#packaging-hadoop-into-the-flink-distribution
>
>
>
>
>
> If this does not work for you, I would wait for Chesnay's input (cc'd).
>
>
>
> – Ufuk
>
>
>
>
>
> On Fri, Jun 7, 2019 at 11:04 AM LINZ, Arnaud <AL...@bouyguestelecom.fr>
> wrote:
>
> Hello,
>
> I am trying to build my own flink distribution with proper Cloudera
> dependencies.
> Reading
> https://ci.apache.org/projects/flink/flink-docs-stable/flinkDev/building.html
> I've done :
> git clone https://github.com/apache/flink
> cd flink
> git checkout tags/release-1.8.0
> $MAVEN_HOME/bin/mvn clean install -DskipTests -Pvendor-repos
> -Dhadoop.version=2.6.0-cdh5.16.1 -Dfast
> cd flink-dist
> $MAVEN_HOME/bin/mvn install -DskipTests -Pvendor-repos
> -Dhadoop.version=2.6.0-cdh5.16.1
>
> Everything was successful.
>
> However when running using flink-dist/target/flink-1.8.0-bin/flink-1.8.0
> Running /bin/flink -h prints no yarn/Hadoop options.
> And running
> bin/flink run -m yarn-cluster -yn 4 -yjm 1024m -ytm 4096m
> ./examples/batch/WordCount.jar
> Prints :
> Could not build the program from JAR file.
>
> Am I missing something?
>
> Best regards,
> Arnaud
>
>
> ________________________________
>
> L'intégrité de ce message n'étant pas assurée sur internet, la société
> expéditrice ne peut être tenue responsable de son contenu ni de ses pièces
> jointes. Toute utilisation ou diffusion non autorisée est interdite. Si
> vous n'êtes pas destinataire de ce message, merci de le détruire et
> d'avertir l'expéditeur.
>
> The integrity of this message cannot be guaranteed on the Internet. The
> company that sent this message cannot therefore be held liable for its
> content nor attachments. Any unauthorized use or dissemination is
> prohibited. If you are not the intended recipient of this message, then
> please delete it and notify the sender.
>
>

RE: No yarn option in self-built flink version

Posted by "LINZ, Arnaud" <AL...@bouyguestelecom.fr>.

Hello,

Thanks a lot, it works. However, may I suggest that you update the documentation page :

mvn clean install -DskipTests -Pvendor-repos -Dhadoop.version=2.6.0-cdh5.16.1
has absolutely no interest if you don’t include hadoop, that’s why I thought that -Pvendor-repos was including the -Pinclude-hadoop stated in the above paragraph…

Arnaud

De : Ufuk Celebi <uc...@apache.org>
Envoyé : vendredi 7 juin 2019 12:00
À : LINZ, Arnaud <AL...@bouyguestelecom.fr>
Cc : user <us...@flink.apache.org>; chesnay@apache.org
Objet : Re: No yarn option in self-built flink version

Hey Arnaud,

I think you need to active the Hadoop profile via -Pinclude-hadoop (the default was changed to not include Hadoop as far as I know).

For more details, check out:
https://ci.apache.org/projects/flink/flink-docs-release-1.8/flinkDev/building.html#packaging-hadoop-into-the-flink-distribution

If this does not work for you, I would wait for Chesnay's input (cc'd).

– Ufuk

On Fri, Jun 7, 2019 at 11:04 AM LINZ, Arnaud <AL...@bouyguestelecom.fr>> wrote:
Hello,

Everything was successful.

Am I missing something?

Best regards,
Arnaud

________________________________

Re: No yarn option in self-built flink version

Posted by Ufuk Celebi <uc...@apache.org>.

Hey Arnaud,

I think you need to active the Hadoop profile via -Pinclude-hadoop (the
default was changed to not include Hadoop as far as I know).

For more details, check out:
https://ci.apache.org/projects/flink/flink-docs-release-1.8/flinkDev/building.html#packaging-hadoop-into-the-flink-distribution


If this does not work for you, I would wait for Chesnay's input (cc'd).

– Ufuk


On Fri, Jun 7, 2019 at 11:04 AM LINZ, Arnaud <AL...@bouyguestelecom.fr>
wrote:

> Hello,
>
> I am trying to build my own flink distribution with proper Cloudera
> dependencies.
> Reading
> https://ci.apache.org/projects/flink/flink-docs-stable/flinkDev/building.html
> I've done :
> git clone https://github.com/apache/flink
> cd flink
> git checkout tags/release-1.8.0
> $MAVEN_HOME/bin/mvn clean install -DskipTests -Pvendor-repos
> -Dhadoop.version=2.6.0-cdh5.16.1 -Dfast
> cd flink-dist
> $MAVEN_HOME/bin/mvn install -DskipTests -Pvendor-repos
> -Dhadoop.version=2.6.0-cdh5.16.1
>
> Everything was successful.
>
> However when running using flink-dist/target/flink-1.8.0-bin/flink-1.8.0
> Running /bin/flink -h prints no yarn/Hadoop options.
> And running
> bin/flink run -m yarn-cluster -yn 4 -yjm 1024m -ytm 4096m
> ./examples/batch/WordCount.jar
> Prints :
> Could not build the program from JAR file.
>
> Am I missing something?
>
> Best regards,
> Arnaud
>
>
> ________________________________
>
> L'intégrité de ce message n'étant pas assurée sur internet, la société
> expéditrice ne peut être tenue responsable de son contenu ni de ses pièces
> jointes. Toute utilisation ou diffusion non autorisée est interdite. Si
> vous n'êtes pas destinataire de ce message, merci de le détruire et
> d'avertir l'expéditeur.
>
> The integrity of this message cannot be guaranteed on the Internet. The
> company that sent this message cannot therefore be held liable for its
> content nor attachments. Any unauthorized use or dissemination is
> prohibited. If you are not the intended recipient of this message, then
> please delete it and notify the sender.
>