You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@flink.apache.org by Dániel Bali <ba...@gmail.com> on 2015/02/20 13:05:07 UTC

Question about Hadoop profiles in POM

Hi!

I was working on FLINK-1522
<https://issues.apache.org/jira/browse/FLINK-1522> and FLINK-1576
<https://issues.apache.org/jira/browse/FLINK-1576> and started converting
Gelly library examples to work with file inputs. I added a test for
LabelPropagationExample and got failures in Travis when hadoop.profile was
1 (Travis build <https://travis-ci.org/balidani/flink/builds/51440170>).

The error was the following:

[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-compiler-plugin:3.1:testCompile
(default-testCompile) on project flink-gelly: Compilation failure:
Compilation failure:
[ERROR] /home/travis/build/balidani/flink/flink-staging/flink-gelly/src/test/java/org/apache/flink/graph/test/LabelPropagationExampleITCase.java:[21,30]
package com.google.common.base does not exist
[ERROR] /home/travis/build/balidani/flink/flink-staging/flink-gelly/src/test/java/org/apache/flink/graph/test/LabelPropagationExampleITCase.java:[22,28]
package com.google.common.io does not exist
[ERROR] /home/travis/build/balidani/flink/flink-staging/flink-gelly/src/test/java/org/apache/flink/graph/test/LabelPropagationExampleITCase.java:[80,53]
cannot find symbol

Then I looked at flink-tests and saw that there was a `profiles` part in
the POM that adds the dependency to guava for hadoop profile 1. I added
this to the POM of Gelly and it seems to fix the issue. (Travis build
<https://travis-ci.org/balidani/flink/builds/51496196>) (There is a
failure, but the log says it's a timeout)

Is this the correct way to solve this error?

Thanks,

Daniel

Re: Question about Hadoop profiles in POM

Posted by Dániel Bali <ba...@gmail.com>.

Thanks for the answers!

I didn't set the scope to provided, but now I will make that change.

Regards,
Daniel

2015-02-20 14:27 GMT+01:00 Aljoscha Krettek <al...@apache.org>:

> Did you set the scope to provided? As in:
>
> <dependency>
>    <groupId>com.google.guava</groupId>
>    <artifactId>guava</artifactId>
>    <version>${guava.version}</version>
>    <scope>provided</scope>
> </dependency>
>
> Then this should be a good solution, yes.
>
> On Fri, Feb 20, 2015 at 1:05 PM, Dániel Bali <ba...@gmail.com>
> wrote:
> > Hi!
> >
> > I was working on FLINK-1522
> > <https://issues.apache.org/jira/browse/FLINK-1522> and FLINK-1576
> > <https://issues.apache.org/jira/browse/FLINK-1576> and started
> converting
> > Gelly library examples to work with file inputs. I added a test for
> > LabelPropagationExample and got failures in Travis when hadoop.profile
> was
> > 1 (Travis build <https://travis-ci.org/balidani/flink/builds/51440170>).
> >
> > The error was the following:
> >
> > [ERROR] Failed to execute goal
> > org.apache.maven.plugins:maven-compiler-plugin:3.1:testCompile
> > (default-testCompile) on project flink-gelly: Compilation failure:
> > Compilation failure:
> > [ERROR]
> /home/travis/build/balidani/flink/flink-staging/flink-gelly/src/test/java/org/apache/flink/graph/test/LabelPropagationExampleITCase.java:[21,30]
> > package com.google.common.base does not exist
> > [ERROR]
> /home/travis/build/balidani/flink/flink-staging/flink-gelly/src/test/java/org/apache/flink/graph/test/LabelPropagationExampleITCase.java:[22,28]
> > package com.google.common.io does not exist
> > [ERROR]
> /home/travis/build/balidani/flink/flink-staging/flink-gelly/src/test/java/org/apache/flink/graph/test/LabelPropagationExampleITCase.java:[80,53]
> > cannot find symbol
> >
> > Then I looked at flink-tests and saw that there was a `profiles` part in
> > the POM that adds the dependency to guava for hadoop profile 1. I added
> > this to the POM of Gelly and it seems to fix the issue. (Travis build
> > <https://travis-ci.org/balidani/flink/builds/51496196>) (There is a
> > failure, but the log says it's a timeout)
> >
> > Is this the correct way to solve this error?
> >
> > Thanks,
> >
> > Daniel
>

Re: Question about Hadoop profiles in POM

Posted by Aljoscha Krettek <al...@apache.org>.

Did you set the scope to provided? As in:

<dependency>
   <groupId>com.google.guava</groupId>
   <artifactId>guava</artifactId>
   <version>${guava.version}</version>
   <scope>provided</scope>
</dependency>

Then this should be a good solution, yes.

On Fri, Feb 20, 2015 at 1:05 PM, Dániel Bali <ba...@gmail.com> wrote:
> Hi!
>
> I was working on FLINK-1522
> <https://issues.apache.org/jira/browse/FLINK-1522> and FLINK-1576
> <https://issues.apache.org/jira/browse/FLINK-1576> and started converting
> Gelly library examples to work with file inputs. I added a test for
> LabelPropagationExample and got failures in Travis when hadoop.profile was
> 1 (Travis build <https://travis-ci.org/balidani/flink/builds/51440170>).
>
> The error was the following:
>
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-compiler-plugin:3.1:testCompile
> (default-testCompile) on project flink-gelly: Compilation failure:
> Compilation failure:
> [ERROR] /home/travis/build/balidani/flink/flink-staging/flink-gelly/src/test/java/org/apache/flink/graph/test/LabelPropagationExampleITCase.java:[21,30]
> package com.google.common.base does not exist
> [ERROR] /home/travis/build/balidani/flink/flink-staging/flink-gelly/src/test/java/org/apache/flink/graph/test/LabelPropagationExampleITCase.java:[22,28]
> package com.google.common.io does not exist
> [ERROR] /home/travis/build/balidani/flink/flink-staging/flink-gelly/src/test/java/org/apache/flink/graph/test/LabelPropagationExampleITCase.java:[80,53]
> cannot find symbol
>
> Then I looked at flink-tests and saw that there was a `profiles` part in
> the POM that adds the dependency to guava for hadoop profile 1. I added
> this to the POM of Gelly and it seems to fix the issue. (Travis build
> <https://travis-ci.org/balidani/flink/builds/51496196>) (There is a
> failure, but the log says it's a timeout)
>
> Is this the correct way to solve this error?
>
> Thanks,
>
> Daniel

Re: Question about Hadoop profiles in POM

Posted by Robert Metzger <rm...@apache.org>.

Hey Daniel,

welcome to the shady world of shading maven dependencies (and Hadoop not
shading its super old guava version ;) ).
I'll have a look at your branch. I think Aljoscha added the guava
dependency for the test scope of the hadoop1 profile.

He made this change when he moved the HadoopInputFormat support into the
"flink-java" package. It seems that his tests also failed due to some
dependency conflict with hadoop. Maybe can explain the details of the issue.

In general, we use these maven build profiles (
http://maven.apache.org/guides/introduction/introduction-to-profiles.html)
to control which modules of flink we are building and which dependencies we
are adding. For example in the hadoop2 build, we add the "flink-yarn" and
"flink-yarn-tests" modules.
Also, the names of the hadoop dependencies have different names for hd1 and
hd2.

I'll get back to you once I've took a closer look into the issue.

Robert

On Fri, Feb 20, 2015 at 1:05 PM, Dániel Bali <ba...@gmail.com>
wrote:

> Hi!
>
> I was working on FLINK-1522
> <https://issues.apache.org/jira/browse/FLINK-1522> and FLINK-1576
> <https://issues.apache.org/jira/browse/FLINK-1576> and started converting
> Gelly library examples to work with file inputs. I added a test for
> LabelPropagationExample and got failures in Travis when hadoop.profile was
> 1 (Travis build <https://travis-ci.org/balidani/flink/builds/51440170>).
>
> The error was the following:
>
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-compiler-plugin:3.1:testCompile
> (default-testCompile) on project flink-gelly: Compilation failure:
> Compilation failure:
> [ERROR]
> /home/travis/build/balidani/flink/flink-staging/flink-gelly/src/test/java/org/apache/flink/graph/test/LabelPropagationExampleITCase.java:[21,30]
> package com.google.common.base does not exist
> [ERROR]
> /home/travis/build/balidani/flink/flink-staging/flink-gelly/src/test/java/org/apache/flink/graph/test/LabelPropagationExampleITCase.java:[22,28]
> package com.google.common.io does not exist
> [ERROR]
> /home/travis/build/balidani/flink/flink-staging/flink-gelly/src/test/java/org/apache/flink/graph/test/LabelPropagationExampleITCase.java:[80,53]
> cannot find symbol
>
> Then I looked at flink-tests and saw that there was a `profiles` part in
> the POM that adds the dependency to guava for hadoop profile 1. I added
> this to the POM of Gelly and it seems to fix the issue. (Travis build
> <https://travis-ci.org/balidani/flink/builds/51496196>) (There is a
> failure, but the log says it's a timeout)
>
> Is this the correct way to solve this error?
>
> Thanks,
>
> Daniel
>