You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@giraph.apache.org by Carlo Sartiani <sa...@gmail.com> on 2014/07/04 18:11:05 UTC
Giraph 1.1.0 and Jetty 7
Dear All,
we are trying to run Giraph 1.1.0 RC against a distribution of Hadoop
2.2.0 that uses Jetty 7 in place of Jetty 6. When we launch our
application on our cluster we get the following error:
2014-07-02 08:14:23,307 INFO [main]
org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
org.apache.hadoop.mapreduce.v2.app.job.event.JobFinishEvent$Type for
class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobFinishEventHandler
2014-07-02 08:14:23,551 INFO [main]
org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from
hadoop-metrics2.properties
2014-07-02 08:14:23,619 INFO [main]
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot
period at 10 second(s).
2014-07-02 08:14:23,619 INFO [main]
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: MRAppMaster metrics
system started
2014-07-02 08:14:23,630 INFO [main]
org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Adding job token
for job_1403542528912_0632 to jobTokenSecretManager
2014-07-02 08:14:23,726 INFO [main]
org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Not uberizing
job_1403542528912_0632 because: not enabled; too much RAM;
2014-07-02 08:14:23,742 INFO [main]
org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Input size for job
job_1403542528912_0632 = 0. Number of splits = 1
2014-07-02 08:14:23,742 INFO [main]
org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Number of reduces
for job job_1403542528912_0632 = 0
2014-07-02 08:14:23,742 INFO [main]
org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl:
job_1403542528912_0632Job Transitioned from NEW to INITED
2014-07-02 08:14:23,744 INFO [main]
org.apache.hadoop.mapreduce.v2.app.MRAppMaster: MRAppMaster launching
normal, non-uberized, multi-container job job_1403542528912_0632.
2014-07-02 08:14:23,782 INFO [Socket Reader #1 for port 7041]
org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 7041
2014-07-02 08:14:23,804 INFO [main]
org.apache.hadoop.yarn.factories.impl.pb.RpcServerFactoryPBImpl: Adding
protocol org.apache.hadoop.mapreduce.v2.api.MRClientProtocolPB to the server
2014-07-02 08:14:23,805 INFO [IPC Server Responder]
org.apache.hadoop.ipc.Server: IPC Server Responder: starting
2014-07-02 08:14:23,805 INFO [IPC Server listener on 7041]
org.apache.hadoop.ipc.Server: IPC Server listener on 7041: starting
2014-07-02 08:14:23,805 INFO [main]
org.apache.hadoop.mapreduce.v2.app.client.MRClientService: Instantiated
MRClientService at node0654/10.1.33.14:7041
2014-07-02 08:14:23,858 FATAL [main]
org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Error starting MRAppMaster
java.lang.NoSuchMethodError:
org.apache.hadoop.http.HttpServer.<init>(Ljava/lang/String;Ljava/lang/String;IZLorg/apache/hadoop/conf/Configuration;Lorg/apache/hadoop/security/authorize/AccessControlList;Lorg/mortbay/jetty/Connector;[Ljava/lang/String;)V
at
org.apache.hadoop.yarn.webapp.WebApps$Builder$2.<init>(WebApps.java:222)
at
org.apache.hadoop.yarn.webapp.WebApps$Builder.start(WebApps.java:219)
at
org.apache.hadoop.mapreduce.v2.app.client.MRClientService.serviceStart(MRClientService.java:136)
at
org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStart(MRAppMaster.java:1058)
at
org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$1.run(MRAppMaster.java:1445)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1441)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1374)
The error is caused by the fact that in our distribution of Hadoop the
HTTPServer constructor is expecting a parameter of type
org.eclipse.jetty.Connector, but is receiving a parameter of type
org.mortbay.jetty.Connector. Indeed, one big difference of Jetty 7 wrt
Jetty 6 is the different naming of packages.
Similar errors arise when the application is launched in pure yarn mode
(containers just fail).
We examined the code of Giraph 1.1.0, but we actually did not find any
place where a Collector object is created and/or Jetty is really used.
Do you have any idea on how to solve this issue? Please, observe that
we are forced to use this Hadoop distribution and cannot switch to a
plain Hadoop 2.2.0 distribution with Jetty 6.
Thanks in advance.
Carlo
--
"Humanity is overrated", Gregory House, MD
Re: Giraph 1.1.0 and Jetty 7
Posted by Carlo Sartiani <sa...@gmail.com>.
Thanks for the suggestion. Unfortunately, it does not work. Probably,
Giraph is using a library that generates an org.mortbay.jetty.Connector
in place of a org.eclipse.jetty.Connector one.
Carlo
> Roman Shaposhnik <ma...@shaposhnik.org>
> 7 Jul 2014 00:12
>
> Here's what I'd suggest: take 1.1.0-RC0 source code and manually change
> the version of Jetty in the top level pom.xml. Rebuild everything with
> -Phadoop_2
> and also specify the exact version of your Hadoop with
> -Dhadoop.version=X.Y.Z
> for good measure.
>
> Thanks,
> Roman.
>
> Carlo Sartiani <ma...@gmail.com>
> 4 Jul 2014 18:11
> Dear All,
>
> we are trying to run Giraph 1.1.0 RC against a distribution of Hadoop
> 2.2.0 that uses Jetty 7 in place of Jetty 6. When we launch our
> application on our cluster we get the following error:
>
> 2014-07-02 08:14:23,307 INFO [main]
> org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
> org.apache.hadoop.mapreduce.v2.app.job.event.JobFinishEvent$Type for
> class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobFinishEventHandler
> 2014-07-02 08:14:23,551 INFO [main]
> org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from
> hadoop-metrics2.properties
> 2014-07-02 08:14:23,619 INFO [main]
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot
> period at 10 second(s).
> 2014-07-02 08:14:23,619 INFO [main]
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: MRAppMaster metrics
> system started
> 2014-07-02 08:14:23,630 INFO [main]
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Adding job token
> for job_1403542528912_0632 to jobTokenSecretManager
> 2014-07-02 08:14:23,726 INFO [main]
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Not uberizing
> job_1403542528912_0632 because: not enabled; too much RAM;
> 2014-07-02 08:14:23,742 INFO [main]
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Input size for
> job job_1403542528912_0632 = 0. Number of splits = 1
> 2014-07-02 08:14:23,742 INFO [main]
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Number of reduces
> for job job_1403542528912_0632 = 0
> 2014-07-02 08:14:23,742 INFO [main]
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl:
> job_1403542528912_0632Job Transitioned from NEW to INITED
> 2014-07-02 08:14:23,744 INFO [main]
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: MRAppMaster launching
> normal, non-uberized, multi-container job job_1403542528912_0632.
> 2014-07-02 08:14:23,782 INFO [Socket Reader #1 for port 7041]
> org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 7041
> 2014-07-02 08:14:23,804 INFO [main]
> org.apache.hadoop.yarn.factories.impl.pb.RpcServerFactoryPBImpl:
> Adding protocol org.apache.hadoop.mapreduce.v2.api.MRClientProtocolPB
> to the server
> 2014-07-02 08:14:23,805 INFO [IPC Server Responder]
> org.apache.hadoop.ipc.Server: IPC Server Responder: starting
> 2014-07-02 08:14:23,805 INFO [IPC Server listener on 7041]
> org.apache.hadoop.ipc.Server: IPC Server listener on 7041: starting
> 2014-07-02 08:14:23,805 INFO [main]
> org.apache.hadoop.mapreduce.v2.app.client.MRClientService:
> Instantiated MRClientService at node0654/10.1.33.14:7041
> 2014-07-02 08:14:23,858 FATAL [main]
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Error starting MRAppMaster
> java.lang.NoSuchMethodError:
> org.apache.hadoop.http.HttpServer.<init>(Ljava/lang/String;Ljava/lang/String;IZLorg/apache/hadoop/conf/Configuration;Lorg/apache/hadoop/security/authorize/AccessControlList;Lorg/mortbay/jetty/Connector;[Ljava/lang/String;)V
> at
> org.apache.hadoop.yarn.webapp.WebApps$Builder$2.<init>(WebApps.java:222)
> at
> org.apache.hadoop.yarn.webapp.WebApps$Builder.start(WebApps.java:219)
> at
> org.apache.hadoop.mapreduce.v2.app.client.MRClientService.serviceStart(MRClientService.java:136)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStart(MRAppMaster.java:1058)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$1.run(MRAppMaster.java:1445)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1441)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1374)
>
> The error is caused by the fact that in our distribution of Hadoop the
> HTTPServer constructor is expecting a parameter of type
> org.eclipse.jetty.Connector, but is receiving a parameter of type
> org.mortbay.jetty.Connector. Indeed, one big difference of Jetty 7 wrt
> Jetty 6 is the different naming of packages.
>
> Similar errors arise when the application is launched in pure yarn
> mode (containers just fail).
>
> We examined the code of Giraph 1.1.0, but we actually did not find any
> place where a Collector object is created and/or Jetty is really used.
>
> Do you have any idea on how to solve this issue? Please, observe that
> we are forced to use this Hadoop distribution and cannot switch to a
> plain Hadoop 2.2.0 distribution with Jetty 6.
>
> Thanks in advance.
>
> Carlo
> ------------------------------------------------------------------------
--
"Humanity is overrated", Gregory House, MD
Re: Giraph 1.1.0 and Jetty 7
Posted by Roman Shaposhnik <ro...@shaposhnik.org>.
On Fri, Jul 4, 2014 at 9:11 AM, Carlo Sartiani <sa...@gmail.com> wrote:
> We examined the code of Giraph 1.1.0, but we actually did not find any place
> where a Collector object is created and/or Jetty is really used.
>
> Do you have any idea on how to solve this issue? Please, observe that we
> are forced to use this Hadoop distribution and cannot switch to a plain
> Hadoop 2.2.0 distribution with Jetty 6.
Here's what I'd suggest: take 1.1.0-RC0 source code and manually change
the version of Jetty in the top level pom.xml. Rebuild everything with
-Phadoop_2
and also specify the exact version of your Hadoop with -Dhadoop.version=X.Y.Z
for good measure.
Thanks,
Roman.