You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@giraph.apache.org by "Dr. Alessio Arleo" <in...@icloud.com> on 2014/12/04 22:01:30 UTC
Compiling Giraph for Hadoop 2.5.x and 2.6.0 -- SASL_PROPS variable
error
Hello everybody
I am trying to compile Giraph release-1.1 for Hadoop 2.5.x and Hadoop 2.6.0 with Maven profile hadoop_yarn. It works fine up to Hadoop 2.4.1, but when trying with a newer version of Hadoop the following error comes up. I am working with jdk 1.7 and Maven 3.2.1.
[ERROR] COMPILATION ERROR :
[INFO] -------------------------------------------------------------
[ERROR] /home/hadoop/git/giraph/1.1/giraph-core/target/munged/main/org/apache/giraph/comm/netty/SaslNettyClient.java:[84,68] cannot find symbol
symbol: variable SASL_PROPS
location: class org.apache.hadoop.security.SaslRpcServer
[ERROR] /home/hadoop/git/giraph/1.1/giraph-core/target/munged/main/org/apache/giraph/comm/netty/SaslNettyServer.java:[105,62] cannot find symbol
symbol: variable SASL_PROPS
location: class org.apache.hadoop.security.SaslRpcServer
Do you have any suggestions? Any would be much appreciated :)
Kind regards,
Alessio
Re: Compiling Giraph for Hadoop 2.5.x and 2.6.0 -- SASL_PROPS
variable error
Posted by Eli Reisman <ap...@gmail.com>.
This makes sense thanks Eugene! I would advise keeping the YARN profile for
now especially if the fix is this minor. I do feel like the advantages and
opportunities for Giraph once development focuses on a post Hadoop1 world
will be huge. Seems like more companies are making the upgrade every day,
mine included. Now that we''re on YARN I'm hoping to run Giraph again
myself.
Just me though and I can certainly see the other side of the argument too.
On Jan 12, 2015 8:05 AM, "Claudio Martella" <cl...@gmail.com>
wrote:
> I see more and more people getting into this. I guess whether we should
> add the fix to the pure_yarn profile by default, as it feel it's going to
> stay.
> Ideas?
>
> On Sat, Jan 10, 2015 at 7:38 PM, Eugene Koontz <ek...@hiro-tan.org>
> wrote:
>
>> Hi Allesio and Eli,
>>
>> Compiling with "mvn -Phadoop_yarn -Dhadoop.version=2.6.0 clean" will
>> avoid the below "SASL_PROPS" compilation error if you remove the
>> STATIC_SASL_SYMBOL from the munge.symbols of the hadoop_yarn profile as
>> follows:
>>
>> diff --git a/pom.xml b/pom.xml
>> index cf0e1f9..8c2a561 100644
>> --- a/pom.xml
>> +++ b/pom.xml
>> @@ -1194,7 +1194,7 @@ under the License.
>> </modules>
>> <properties>
>>
>> <hadoop.version>SET_HADOOP_VERSION_USING_MVN_DASH_D_OPTION</hadoop.version>
>> - <munge.symbols>PURE_YARN,STATIC_SASL_SYMBOL</munge.symbols>
>> + <munge.symbols>PURE_YARN</munge.symbols>
>> <!-- TODO: add these checks eventually -->
>> <project.enforcer.skip>true</project.enforcer.skip>
>>
>> <giraph.maven.dependency.plugin.skip>true</giraph.maven.dependency.plugin.skip>
>>
>> In other words, when compiling Giraph against newer releases of Hadoop,
>> there is no need for this munge symbol.
>>
>> The distinction between newer and older seems to be release 2.4.0 of
>> Hadoop, as given here:
>>
>> https://issues.apache.org/jira/browse/HADOOP-10221 " Add a plugin to
>> specify SaslProperties for RPC protocol based on connection properties".
>>
>> It seems like we need to add some additional profiles to make the
>> pre-2.4 Hadoop (which requires the munge symbol STATIC_SASL_SYMBOL) and
>> newer (which should not).
>>
>> -Eugene
>>
>>
>> On 1/8/15, 11:13 PM, Eugene Koontz wrote:
>>
>> Hi Alessio,
>>
>> I am able to reproduce your problem:
>>
>> https://gist.github.com/ekoontz/7dbaaf6218abb4fd7832
>>
>> I'll try building Hadoop 2.6.0 and getting Giraph to work with it.
>>
>> -Eugene
>>
>>
>> On 1/8/15, 10:55 AM, Eli Reisman wrote:
>>
>> This looks like a munge symbol that needs to be added to the hadoop_yarn
>> profile in the pom.xml. I'm thinking this is an issue a couple people have
>> been having on 2.5 and 2.6 trying to build the hadoop_yarn profile?
>>
>> On Thu, Dec 4, 2014 at 1:01 PM, Dr. Alessio Arleo <in...@icloud.com>
>> wrote:
>>
>>> Hello everybody
>>>
>>> I am trying to compile Giraph release-1.1 for Hadoop 2.5.x and Hadoop
>>> 2.6.0 with Maven profile hadoop_yarn. It works fine up to Hadoop 2.4.1, but
>>> when trying with a newer version of Hadoop the following error comes up. I
>>> am working with jdk 1.7 and Maven 3.2.1.
>>> ST
>>> [ERROR] COMPILATION ERROR :
>>> [INFO] -------------------------------------------------------------
>>> [ERROR]
>>> /home/hadoop/git/giraph/1.1/giraph-core/target/munged/main/org/apache/giraph/comm/netty/SaslNettyClient.java:[84,68]
>>> cannot find symbol
>>> symbol: variable SASL_PROPS
>>> location: class org.apache.hadoop.security.SaslRpcServer
>>> [ERROR]
>>> /home/hadoop/git/giraph/1.1/giraph-core/target/munged/main/org/apache/giraph/comm/netty/SaslNettyServer.java:[105,62]
>>> cannot find symbol
>>> symbol: variable SASL_PROPS
>>> location: class org.apache.hadoop.security.SaslRpcServer
>>>
>>> Do you have any suggestions? Any would be much appreciated :)
>>>
>>> Kind regards,
>>> Alessio
>>>
>>
>>
>>
>>
>
>
> --
> Claudio Martella
>
>
Re: Compiling Giraph for Hadoop 2.5.x and 2.6.0 -- SASL_PROPS
variable error
Posted by Claudio Martella <cl...@gmail.com>.
I see more and more people getting into this. I guess whether we should add
the fix to the pure_yarn profile by default, as it feel it's going to stay.
Ideas?
On Sat, Jan 10, 2015 at 7:38 PM, Eugene Koontz <ek...@hiro-tan.org> wrote:
> Hi Allesio and Eli,
>
> Compiling with "mvn -Phadoop_yarn -Dhadoop.version=2.6.0 clean" will avoid
> the below "SASL_PROPS" compilation error if you remove the
> STATIC_SASL_SYMBOL from the munge.symbols of the hadoop_yarn profile as
> follows:
>
> diff --git a/pom.xml b/pom.xml
> index cf0e1f9..8c2a561 100644
> --- a/pom.xml
> +++ b/pom.xml
> @@ -1194,7 +1194,7 @@ under the License.
> </modules>
> <properties>
>
> <hadoop.version>SET_HADOOP_VERSION_USING_MVN_DASH_D_OPTION</hadoop.version>
> - <munge.symbols>PURE_YARN,STATIC_SASL_SYMBOL</munge.symbols>
> + <munge.symbols>PURE_YARN</munge.symbols>
> <!-- TODO: add these checks eventually -->
> <project.enforcer.skip>true</project.enforcer.skip>
>
> <giraph.maven.dependency.plugin.skip>true</giraph.maven.dependency.plugin.skip>
>
> In other words, when compiling Giraph against newer releases of Hadoop,
> there is no need for this munge symbol.
>
> The distinction between newer and older seems to be release 2.4.0 of
> Hadoop, as given here:
>
> https://issues.apache.org/jira/browse/HADOOP-10221 " Add a plugin to
> specify SaslProperties for RPC protocol based on connection properties".
>
> It seems like we need to add some additional profiles to make the pre-2.4
> Hadoop (which requires the munge symbol STATIC_SASL_SYMBOL) and newer
> (which should not).
>
> -Eugene
>
>
> On 1/8/15, 11:13 PM, Eugene Koontz wrote:
>
> Hi Alessio,
>
> I am able to reproduce your problem:
>
> https://gist.github.com/ekoontz/7dbaaf6218abb4fd7832
>
> I'll try building Hadoop 2.6.0 and getting Giraph to work with it.
>
> -Eugene
>
>
> On 1/8/15, 10:55 AM, Eli Reisman wrote:
>
> This looks like a munge symbol that needs to be added to the hadoop_yarn
> profile in the pom.xml. I'm thinking this is an issue a couple people have
> been having on 2.5 and 2.6 trying to build the hadoop_yarn profile?
>
> On Thu, Dec 4, 2014 at 1:01 PM, Dr. Alessio Arleo <in...@icloud.com>
> wrote:
>
>> Hello everybody
>>
>> I am trying to compile Giraph release-1.1 for Hadoop 2.5.x and Hadoop
>> 2.6.0 with Maven profile hadoop_yarn. It works fine up to Hadoop 2.4.1, but
>> when trying with a newer version of Hadoop the following error comes up. I
>> am working with jdk 1.7 and Maven 3.2.1.
>> ST
>> [ERROR] COMPILATION ERROR :
>> [INFO] -------------------------------------------------------------
>> [ERROR]
>> /home/hadoop/git/giraph/1.1/giraph-core/target/munged/main/org/apache/giraph/comm/netty/SaslNettyClient.java:[84,68]
>> cannot find symbol
>> symbol: variable SASL_PROPS
>> location: class org.apache.hadoop.security.SaslRpcServer
>> [ERROR]
>> /home/hadoop/git/giraph/1.1/giraph-core/target/munged/main/org/apache/giraph/comm/netty/SaslNettyServer.java:[105,62]
>> cannot find symbol
>> symbol: variable SASL_PROPS
>> location: class org.apache.hadoop.security.SaslRpcServer
>>
>> Do you have any suggestions? Any would be much appreciated :)
>>
>> Kind regards,
>> Alessio
>>
>
>
>
>
--
Claudio Martella
Re: Compiling Giraph for Hadoop 2.5.x and 2.6.0 -- SASL_PROPS variable
error
Posted by Eugene Koontz <ek...@hiro-tan.org>.
Hi Allesio and Eli,
Compiling with "mvn -Phadoop_yarn -Dhadoop.version=2.6.0 clean" will
avoid the below "SASL_PROPS" compilation error if you remove the
STATIC_SASL_SYMBOL from the munge.symbols of the hadoop_yarn profile as
follows:
diff --git a/pom.xml b/pom.xml
index cf0e1f9..8c2a561 100644
--- a/pom.xml
+++ b/pom.xml
@@ -1194,7 +1194,7 @@ under the License.
</modules>
<properties>
<hadoop.version>SET_HADOOP_VERSION_USING_MVN_DASH_D_OPTION</hadoop.version>
- <munge.symbols>PURE_YARN,STATIC_SASL_SYMBOL</munge.symbols>
+ <munge.symbols>PURE_YARN</munge.symbols>
<!-- TODO: add these checks eventually -->
<project.enforcer.skip>true</project.enforcer.skip>
<giraph.maven.dependency.plugin.skip>true</giraph.maven.dependency.plugin.skip>
In other words, when compiling Giraph against newer releases of Hadoop,
there is no need for this munge symbol.
The distinction between newer and older seems to be release 2.4.0 of
Hadoop, as given here:
https://issues.apache.org/jira/browse/HADOOP-10221 " Add a plugin to
specify SaslProperties for RPC protocol based on connection properties".
It seems like we need to add some additional profiles to make the
pre-2.4 Hadoop (which requires the munge symbol STATIC_SASL_SYMBOL) and
newer (which should not).
-Eugene
On 1/8/15, 11:13 PM, Eugene Koontz wrote:
> Hi Alessio,
>
> I am able to reproduce your problem:
>
> https://gist.github.com/ekoontz/7dbaaf6218abb4fd7832
>
> I'll try building Hadoop 2.6.0 and getting Giraph to work with it.
>
> -Eugene
>
>
> On 1/8/15, 10:55 AM, Eli Reisman wrote:
>> This looks like a munge symbol that needs to be added to the
>> hadoop_yarn profile in the pom.xml. I'm thinking this is an issue a
>> couple people have been having on 2.5 and 2.6 trying to build the
>> hadoop_yarn profile?
>>
>> On Thu, Dec 4, 2014 at 1:01 PM, Dr. Alessio Arleo
>> <ingarleo@icloud.com <ma...@icloud.com>> wrote:
>>
>> Hello everybody
>>
>> I am trying to compile Giraph release-1.1 for Hadoop 2.5.x and
>> Hadoop 2.6.0 with Maven profile hadoop_yarn. It works fine up to
>> Hadoop 2.4.1, but when trying with a newer version of Hadoop the
>> following error comes up. I am working with jdk 1.7 and Maven 3.2.1.
>> ST
>> [ERROR] COMPILATION ERROR :
>> [INFO] -------------------------------------------------------------
>> [ERROR]
>> /home/hadoop/git/giraph/1.1/giraph-core/target/munged/main/org/apache/giraph/comm/netty/SaslNettyClient.java:[84,68]
>> cannot find symbol
>> symbol: variable SASL_PROPS
>> location: class org.apache.hadoop.security.SaslRpcServer
>> [ERROR]
>> /home/hadoop/git/giraph/1.1/giraph-core/target/munged/main/org/apache/giraph/comm/netty/SaslNettyServer.java:[105,62]
>> cannot find symbol
>> symbol: variable SASL_PROPS
>> location: class org.apache.hadoop.security.SaslRpcServer
>>
>> Do you have any suggestions? Any would be much appreciated :)
>>
>> Kind regards,
>> Alessio
>>
>>
>
Re: Compiling Giraph for Hadoop 2.5.x and 2.6.0 -- SASL_PROPS variable
error
Posted by Eugene Koontz <ek...@hiro-tan.org>.
Hi Alessio,
I am able to reproduce your problem:
https://gist.github.com/ekoontz/7dbaaf6218abb4fd7832
I'll try building Hadoop 2.6.0 and getting Giraph to work with it.
-Eugene
On 1/8/15, 10:55 AM, Eli Reisman wrote:
> This looks like a munge symbol that needs to be added to the
> hadoop_yarn profile in the pom.xml. I'm thinking this is an issue a
> couple people have been having on 2.5 and 2.6 trying to build the
> hadoop_yarn profile?
>
> On Thu, Dec 4, 2014 at 1:01 PM, Dr. Alessio Arleo <ingarleo@icloud.com
> <ma...@icloud.com>> wrote:
>
> Hello everybody
>
> I am trying to compile Giraph release-1.1 for Hadoop 2.5.x and
> Hadoop 2.6.0 with Maven profile hadoop_yarn. It works fine up to
> Hadoop 2.4.1, but when trying with a newer version of Hadoop the
> following error comes up. I am working with jdk 1.7 and Maven 3.2.1.
>
> [ERROR] COMPILATION ERROR :
> [INFO] -------------------------------------------------------------
> [ERROR]
> /home/hadoop/git/giraph/1.1/giraph-core/target/munged/main/org/apache/giraph/comm/netty/SaslNettyClient.java:[84,68]
> cannot find symbol
> symbol: variable SASL_PROPS
> location: class org.apache.hadoop.security.SaslRpcServer
> [ERROR]
> /home/hadoop/git/giraph/1.1/giraph-core/target/munged/main/org/apache/giraph/comm/netty/SaslNettyServer.java:[105,62]
> cannot find symbol
> symbol: variable SASL_PROPS
> location: class org.apache.hadoop.security.SaslRpcServer
>
> Do you have any suggestions? Any would be much appreciated :)
>
> Kind regards,
> Alessio
>
>
Re: Compiling Giraph for Hadoop 2.5.x and 2.6.0 -- SASL_PROPS
variable error
Posted by Eli Reisman <ap...@gmail.com>.
This looks like a munge symbol that needs to be added to the hadoop_yarn
profile in the pom.xml. I'm thinking this is an issue a couple people have
been having on 2.5 and 2.6 trying to build the hadoop_yarn profile?
On Thu, Dec 4, 2014 at 1:01 PM, Dr. Alessio Arleo <in...@icloud.com>
wrote:
> Hello everybody
>
> I am trying to compile Giraph release-1.1 for Hadoop 2.5.x and Hadoop
> 2.6.0 with Maven profile hadoop_yarn. It works fine up to Hadoop 2.4.1, but
> when trying with a newer version of Hadoop the following error comes up. I
> am working with jdk 1.7 and Maven 3.2.1.
>
> [ERROR] COMPILATION ERROR :
> [INFO] -------------------------------------------------------------
> [ERROR]
> /home/hadoop/git/giraph/1.1/giraph-core/target/munged/main/org/apache/giraph/comm/netty/SaslNettyClient.java:[84,68]
> cannot find symbol
> symbol: variable SASL_PROPS
> location: class org.apache.hadoop.security.SaslRpcServer
> [ERROR]
> /home/hadoop/git/giraph/1.1/giraph-core/target/munged/main/org/apache/giraph/comm/netty/SaslNettyServer.java:[105,62]
> cannot find symbol
> symbol: variable SASL_PROPS
> location: class org.apache.hadoop.security.SaslRpcServer
>
> Do you have any suggestions? Any would be much appreciated :)
>
> Kind regards,
> Alessio
>