You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hama.apache.org by Roman Shapovalov <sh...@graphics.cs.msu.su> on 2013/10/14 17:17:26 UTC
Setting application configuration for Hama Streaming
Hi all,
The k-means example for Hama Streaming contains lines like
peer.config.get("kmeans.max.iterations")
How to set such application-specific parameters from outside?
I tried to use -D and -conf options for "hama pipes", but they were
not recognized.
Thanks,
Roman
Re: Setting application configuration for Hama Streaming
Posted by Martin Illecker <mi...@apache.org>.
Maybe we should make the CLI options order more clear.
If you observe any problems using Hama Pipes don't hesitate to contact us.
Thanks!
Martin
2013/10/15 Roman Shapovalov <sh...@graphics.cs.msu.su>
> Hi Martin,
>
> > Do you have tried this already?
>
> No, I missed that line in the help string.
> It works now, thank you!
>
> Roman
>
> On Tue, Oct 15, 2013 at 11:53 AM, Martin Illecker <mi...@apache.org>
> wrote:
> > Hi Roman,
> >
> > Hama Pipes does support the -conf and -D option, but maybe there is a
> > problem with the Python wrapper.
> >
> > I have explained the CLI usage in Edward's JIRA ticket [1].
> >
> > Both options -conf and -D are genericOptions and must be specified before
> > other pipesOptions.
> >
> > bin/hama pipes [genericOptions] [pipesOptions]
> >
> > e.g.,
> > hama pipes -conf job.xml <otherOptions>
> > hama pipes -D property1=value1 <otherOptions>
> > hama pipes -D kmeans.max.iterations=1000 <otherOptions>
> >
> > Do you have tried this already?
> >
> > Please submit your full command.
> >
> > Thanks!
> >
> > Martin
> >
> > [1] https://issues.apache.org/jira/browse/HAMA-810
> >
> >
> >
> > 2013/10/15 Roman Shapovalov <sh...@graphics.cs.msu.su>
> >
> >> Edward, thanks.
> >>
> >> > Alternatively, I think you can use the program arguments instead of
> >> configuration variables.
> >>
> >> So, for using it within Python wrapper, I should pass the arguments
> >> after the class name in -programArgs and hack the BSPRunner script,
> >> right?
> >>
> >> Regards,
> >> Roman
> >>
> >> On Tue, Oct 15, 2013 at 3:01 AM, Edward J. Yoon <ed...@apache.org>
> >> wrote:
> >> > Hi Roman,
> >> >
> >> > Sorry for inconvenience. I checked current source code[1] and it seems
> >> that there's no way to specify an application configuration variables.
> So,
> >> I filed on JIRA[2].
> >> >
> >> >> peer.config.get("kmeans.max.iterations")
> >> >
> >> > Alternatively, I think you can use the program arguments instead of
> >> configuration variables.
> >> >
> >> > 1.
> >>
> http://svn.apache.org/repos/asf/hama/trunk/core/src/main/java/org/apache/hama/pipes/Submitter.java
> >> > 2. https://issues.apache.org/jira/browse/HAMA-810
> >> >
> >> > --
> >> > Best Regards, Edward J. Yoon
> >> > @eddieyoon
> >> >
> >> > On Oct 15, 2013, at 12:17 AM, Roman Shapovalov <
> >> shapovalov@graphics.cs.msu.su> wrote:
> >> >
> >> >> Hi all,
> >> >>
> >> >> The k-means example for Hama Streaming contains lines like
> >> >> peer.config.get("kmeans.max.iterations")
> >> >>
> >> >> How to set such application-specific parameters from outside?
> >> >> I tried to use -D and -conf options for "hama pipes", but they were
> >> >> not recognized.
> >> >>
> >> >> Thanks,
> >> >> Roman
> >> >
> >>
>
Re: Setting application configuration for Hama Streaming
Posted by Roman Shapovalov <sh...@graphics.cs.msu.su>.
Hi Martin,
> Do you have tried this already?
No, I missed that line in the help string.
It works now, thank you!
Roman
On Tue, Oct 15, 2013 at 11:53 AM, Martin Illecker <mi...@apache.org> wrote:
> Hi Roman,
>
> Hama Pipes does support the -conf and -D option, but maybe there is a
> problem with the Python wrapper.
>
> I have explained the CLI usage in Edward's JIRA ticket [1].
>
> Both options -conf and -D are genericOptions and must be specified before
> other pipesOptions.
>
> bin/hama pipes [genericOptions] [pipesOptions]
>
> e.g.,
> hama pipes -conf job.xml <otherOptions>
> hama pipes -D property1=value1 <otherOptions>
> hama pipes -D kmeans.max.iterations=1000 <otherOptions>
>
> Do you have tried this already?
>
> Please submit your full command.
>
> Thanks!
>
> Martin
>
> [1] https://issues.apache.org/jira/browse/HAMA-810
>
>
>
> 2013/10/15 Roman Shapovalov <sh...@graphics.cs.msu.su>
>
>> Edward, thanks.
>>
>> > Alternatively, I think you can use the program arguments instead of
>> configuration variables.
>>
>> So, for using it within Python wrapper, I should pass the arguments
>> after the class name in -programArgs and hack the BSPRunner script,
>> right?
>>
>> Regards,
>> Roman
>>
>> On Tue, Oct 15, 2013 at 3:01 AM, Edward J. Yoon <ed...@apache.org>
>> wrote:
>> > Hi Roman,
>> >
>> > Sorry for inconvenience. I checked current source code[1] and it seems
>> that there's no way to specify an application configuration variables. So,
>> I filed on JIRA[2].
>> >
>> >> peer.config.get("kmeans.max.iterations")
>> >
>> > Alternatively, I think you can use the program arguments instead of
>> configuration variables.
>> >
>> > 1.
>> http://svn.apache.org/repos/asf/hama/trunk/core/src/main/java/org/apache/hama/pipes/Submitter.java
>> > 2. https://issues.apache.org/jira/browse/HAMA-810
>> >
>> > --
>> > Best Regards, Edward J. Yoon
>> > @eddieyoon
>> >
>> > On Oct 15, 2013, at 12:17 AM, Roman Shapovalov <
>> shapovalov@graphics.cs.msu.su> wrote:
>> >
>> >> Hi all,
>> >>
>> >> The k-means example for Hama Streaming contains lines like
>> >> peer.config.get("kmeans.max.iterations")
>> >>
>> >> How to set such application-specific parameters from outside?
>> >> I tried to use -D and -conf options for "hama pipes", but they were
>> >> not recognized.
>> >>
>> >> Thanks,
>> >> Roman
>> >
>>
Re: Setting application configuration for Hama Streaming
Posted by Martin Illecker <mi...@apache.org>.
Hi Roman,
Hama Pipes does support the -conf and -D option, but maybe there is a
problem with the Python wrapper.
I have explained the CLI usage in Edward's JIRA ticket [1].
Both options -conf and -D are genericOptions and must be specified before
other pipesOptions.
bin/hama pipes [genericOptions] [pipesOptions]
e.g.,
hama pipes -conf job.xml <otherOptions>
hama pipes -D property1=value1 <otherOptions>
hama pipes -D kmeans.max.iterations=1000 <otherOptions>
Do you have tried this already?
Please submit your full command.
Thanks!
Martin
[1] https://issues.apache.org/jira/browse/HAMA-810
2013/10/15 Roman Shapovalov <sh...@graphics.cs.msu.su>
> Edward, thanks.
>
> > Alternatively, I think you can use the program arguments instead of
> configuration variables.
>
> So, for using it within Python wrapper, I should pass the arguments
> after the class name in -programArgs and hack the BSPRunner script,
> right?
>
> Regards,
> Roman
>
> On Tue, Oct 15, 2013 at 3:01 AM, Edward J. Yoon <ed...@apache.org>
> wrote:
> > Hi Roman,
> >
> > Sorry for inconvenience. I checked current source code[1] and it seems
> that there's no way to specify an application configuration variables. So,
> I filed on JIRA[2].
> >
> >> peer.config.get("kmeans.max.iterations")
> >
> > Alternatively, I think you can use the program arguments instead of
> configuration variables.
> >
> > 1.
> http://svn.apache.org/repos/asf/hama/trunk/core/src/main/java/org/apache/hama/pipes/Submitter.java
> > 2. https://issues.apache.org/jira/browse/HAMA-810
> >
> > --
> > Best Regards, Edward J. Yoon
> > @eddieyoon
> >
> > On Oct 15, 2013, at 12:17 AM, Roman Shapovalov <
> shapovalov@graphics.cs.msu.su> wrote:
> >
> >> Hi all,
> >>
> >> The k-means example for Hama Streaming contains lines like
> >> peer.config.get("kmeans.max.iterations")
> >>
> >> How to set such application-specific parameters from outside?
> >> I tried to use -D and -conf options for "hama pipes", but they were
> >> not recognized.
> >>
> >> Thanks,
> >> Roman
> >
>
Re: Setting application configuration for Hama Streaming
Posted by Roman Shapovalov <sh...@graphics.cs.msu.su>.
Edward, thanks.
> Alternatively, I think you can use the program arguments instead of configuration variables.
So, for using it within Python wrapper, I should pass the arguments
after the class name in -programArgs and hack the BSPRunner script,
right?
Regards,
Roman
On Tue, Oct 15, 2013 at 3:01 AM, Edward J. Yoon <ed...@apache.org> wrote:
> Hi Roman,
>
> Sorry for inconvenience. I checked current source code[1] and it seems that there's no way to specify an application configuration variables. So, I filed on JIRA[2].
>
>> peer.config.get("kmeans.max.iterations")
>
> Alternatively, I think you can use the program arguments instead of configuration variables.
>
> 1. http://svn.apache.org/repos/asf/hama/trunk/core/src/main/java/org/apache/hama/pipes/Submitter.java
> 2. https://issues.apache.org/jira/browse/HAMA-810
>
> --
> Best Regards, Edward J. Yoon
> @eddieyoon
>
> On Oct 15, 2013, at 12:17 AM, Roman Shapovalov <sh...@graphics.cs.msu.su> wrote:
>
>> Hi all,
>>
>> The k-means example for Hama Streaming contains lines like
>> peer.config.get("kmeans.max.iterations")
>>
>> How to set such application-specific parameters from outside?
>> I tried to use -D and -conf options for "hama pipes", but they were
>> not recognized.
>>
>> Thanks,
>> Roman
>
Re: Problem with 0.6.3
Posted by ik...@csd.auth.gr.
Well you might find it odd (I did, though) but I restarted the
computer, logged in my windows installation to check something, came
back and the problems are fixed. At least the ones I mentioned...!
I now try with the same input and jar with the following configuration
in hama-site.xml:
<configuration>
<property>
<name>bsp.master.address</name>
<value>192.168.1.2:40000</value>
<description>The address of the bsp master server. Either the
literal string "local" or a host:port for distributed mode
</description>
</property>
<property>
<name>hama.zookeeper.quorum</name>
<value>192.168.1.2</value>
<description>Comma separated list of servers in the ZooKeeper Quorum.
For example, "host1.mydomain.com,host2.mydomain.com,host3.mydomain.com".
By default this is set to localhost for local and pseudo-distributed modes
of operation. For a fully-distributed setup, this should be set to a full
list of ZooKeeper quorum servers. If HAMA_MANAGES_ZK is set in hama-env.sh
this is the list of servers which we will start/stop zookeeper on.
</description>
</property>
<property>
<name>fs.default.name</name>
<value>hdfs://192.168.1.2:54310/</value>
<description>
The name of the default file system. Either the literal string
"local" or a host:port for HDFS.
</description>
</property>
</configuration>
Now it starts 2 jobs, even though I have 1 job and the output is just
the start and end messages. All the vertices all loaded correctly but
nothing is printed from the vertices(I have some System.out.prinln in
compute).
I check the tasklogs and they are:
--- From the first job
13/10/19 19:52:58 INFO ipc.Server: Starting Socket Reader #1 for port 42141
13/10/19 19:52:58 INFO ipc.Server: IPC Server Responder: starting
13/10/19 19:52:58 INFO ipc.Server: IPC Server listener on 42141: starting
13/10/19 19:52:58 INFO ipc.Server: IPC Server handler 0 on 42141: starting
13/10/19 19:52:58 INFO message.HadoopMessageManagerImpl: BSPPeer
address:master port:42141
13/10/19 19:52:58 WARN util.NativeCodeLoader: Unable to load
native-hadoop library for your platform... using builtin-java classes
where applicable
13/10/19 19:52:58 WARN snappy.LoadSnappy: Snappy native library not loaded
13/10/19 19:52:58 INFO sync.ZKSyncClient: Initializing ZK Sync Client
13/10/19 19:52:58 INFO sync.ZooKeeperSyncClientImpl: Start connecting
to Zookeeper! At master/192.168.1.2:42141
13/10/19 19:52:59 INFO ipc.Server: Stopping server on 42141
13/10/19 19:52:59 INFO ipc.Server: IPC Server handler 0 on 42141: exiting
13/10/19 19:52:59 INFO ipc.Server: Stopping IPC Server listener on 42141
13/10/19 19:52:59 INFO ipc.Server: Stopping IPC Server Responder
13/10/19 19:52:59 ERROR bsp.BSPTask: Shutting down ping service.
--- And from the second job.
13/10/19 19:53:04 INFO ipc.Server: Starting Socket Reader #1 for port 37447
13/10/19 19:53:04 INFO ipc.Server: IPC Server Responder: starting
13/10/19 19:53:04 INFO ipc.Server: IPC Server listener on 37447: starting
13/10/19 19:53:04 INFO message.HadoopMessageManagerImpl: BSPPeer
address:master port:37447
13/10/19 19:53:04 INFO ipc.Server: IPC Server handler 0 on 37447: starting
13/10/19 19:53:04 INFO sync.ZKSyncClient: Initializing ZK Sync Client
13/10/19 19:53:04 INFO sync.ZooKeeperSyncClientImpl: Start connecting
to Zookeeper! At master/192.168.1.2:37447
13/10/19 19:53:05 INFO graph.GraphJobRunner: 5 vertices are loaded
into master:37447
13/10/19 19:53:05 INFO ipc.Server: Stopping server on 37447
13/10/19 19:53:05 INFO ipc.Server: IPC Server handler 0 on 37447: exiting
13/10/19 19:53:05 INFO ipc.Server: Stopping IPC Server listener on 37447
13/10/19 19:53:05 INFO ipc.Server: Stopping IPC Server Responder
13/10/19 19:53:05 ERROR bsp.BSPTask: Shutting down ping service.
They are the same with different ports.
I think the normal is to start one job :P
Quoting "Edward J. Yoon" <ed...@apache.org>:
> Please attach your job configuration, vertex reader and partitioner codes.
>
>
>
> On Fri, Oct 18, 2013 at 12:44 AM, <ik...@csd.auth.gr> wrote:
>> Hello there,
>>
>> i just downloaded the version 0.6.3, did the configurations and wanted to
>> run my programs. Previously I was using hama 0.6.0. The examples are working
>> fine but when I give one jar of mine to be run, I have the following issue:
>>
>> - It starts 2 jobs and after the first is completed without doing anything,
>> the second hangs and prints this:
>>
>> attempt_201310172123_0009_000000_0: 13/10/17 21:37:36 INFO ipc.Server:
>> Starting Socket Reader #1 for port 52820
>> attempt_201310172123_0009_000000_0: 13/10/17 21:37:36 INFO ipc.Server: IPC
>> Server Responder: starting
>> attempt_201310172123_0009_000000_0: 13/10/17 21:37:36 INFO ipc.Server: IPC
>> Server listener on 52820: starting
>> attempt_201310172123_0009_000000_0: 13/10/17 21:37:36 INFO ipc.Server: IPC
>> Server handler 0 on 52820: starting
>> attempt_201310172123_0009_000000_0: 13/10/17 21:37:36 INFO
>> message.HadoopMessageManagerImpl: BSPPeer address:master port:52820
>> attempt_201310172123_0009_000000_0: 13/10/17 21:37:36 INFO
>> sync.ZKSyncClient: Initializing ZK Sync Client
>> attempt_201310172123_0009_000000_0: 13/10/17 21:37:36 INFO
>> sync.ZooKeeperSyncClientImpl: Start connecting to Zookeeper! At
>> master/192.168.1.2:52820
>> attempt_201310172123_0009_000000_0: 13/10/17 21:37:37 ERROR bsp.BSPTask:
>> Error running bsp setup and bsp function.
>> attempt_201310172123_0009_000000_0: java.lang.NullPointerException
>> attempt_201310172123_0009_000000_0: at
>> org.apache.hama.graph.GraphJobRunner.loadVertices(GraphJobRunner.java:430)
>> attempt_201310172123_0009_000000_0: at
>> org.apache.hama.graph.GraphJobRunner.setup(GraphJobRunner.java:114)
>> attempt_201310172123_0009_000000_0: at
>> org.apache.hama.bsp.BSPTask.runBSP(BSPTask.java:176)
>> attempt_201310172123_0009_000000_0: at
>> org.apache.hama.bsp.BSPTask.run(BSPTask.java:146)
>> attempt_201310172123_0009_000000_0: at
>> org.apache.hama.bsp.GroomServer$BSPPeerChild.main(GroomServer.java:1246)
>> attempt_201310172123_0009_000000_0: 13/10/17 21:37:37 INFO ipc.Server:
>> Stopping server on 52820
>> attempt_201310172123_0009_000000_0: 13/10/17 21:37:37 INFO ipc.Server: IPC
>> Server handler 0 on 52820: exiting
>> attempt_201310172123_0009_000000_0: 13/10/17 21:37:37 INFO ipc.Server:
>> Stopping IPC Server listener on 52820
>> attempt_201310172123_0009_000000_0: 13/10/17 21:37:37 INFO ipc.Server:
>> Stopping IPC Server Responder
>> attempt_201310172123_0009_000000_0: 13/10/17 21:37:37 ERROR bsp.BSPTask:
>> Shutting down ping service.
>> attempt_201310172123_0009_000000_0: 13/10/17 21:37:37 FATAL bsp.GroomServer:
>> Error running child
>> attempt_201310172123_0009_000000_0: java.lang.NullPointerException
>> attempt_201310172123_0009_000000_0: at
>> org.apache.hama.graph.GraphJobRunner.loadVertices(GraphJobRunner.java:430)
>> attempt_201310172123_0009_000000_0: at
>> org.apache.hama.graph.GraphJobRunner.setup(GraphJobRunner.java:114)
>> attempt_201310172123_0009_000000_0: at
>> org.apache.hama.bsp.BSPTask.runBSP(BSPTask.java:176)
>> attempt_201310172123_0009_000000_0: at
>> org.apache.hama.bsp.BSPTask.run(BSPTask.java:146)
>> attempt_201310172123_0009_000000_0: at
>> org.apache.hama.bsp.GroomServer$BSPPeerChild.main(GroomServer.java:1246)
>> attempt_201310172123_0009_000000_0: java.lang.NullPointerException
>> attempt_201310172123_0009_000000_0: at
>> org.apache.hama.graph.GraphJobRunner.loadVertices(GraphJobRunner.java:430)
>> attempt_201310172123_0009_000000_0: at
>> org.apache.hama.graph.GraphJobRunner.setup(GraphJobRunner.java:114)
>> attempt_201310172123_0009_000000_0: at
>> org.apache.hama.bsp.BSPTask.runBSP(BSPTask.java:176)
>> attempt_201310172123_0009_000000_0: at
>> org.apache.hama.bsp.BSPTask.run(BSPTask.java:146)
>> attempt_201310172123_0009_000000_0: at
>> org.apache.hama.bsp.GroomServer$BSPPeerChild.main(GroomServer.java:1246)
>> 13/10/17 21:38:44 INFO bsp.BSPJobClient: Job failed.
>>
>>
>> From what I see, it has to do something either with the setup and bsp
>> functions or with the reader because there is NullPointerException at
>> loadVertices.
>>
>> I don't think it is the first one cause I am using hama Graph.
>>
>> Any ideas?
>> Thanks
>>
>
>
>
> --
> Best Regards, Edward J. Yoon
> @eddieyoon
Re: Problem with 0.6.3
Posted by "Edward J. Yoon" <ed...@apache.org>.
Please attach your job configuration, vertex reader and partitioner codes.
On Fri, Oct 18, 2013 at 12:44 AM, <ik...@csd.auth.gr> wrote:
> Hello there,
>
> i just downloaded the version 0.6.3, did the configurations and wanted to
> run my programs. Previously I was using hama 0.6.0. The examples are working
> fine but when I give one jar of mine to be run, I have the following issue:
>
> - It starts 2 jobs and after the first is completed without doing anything,
> the second hangs and prints this:
>
> attempt_201310172123_0009_000000_0: 13/10/17 21:37:36 INFO ipc.Server:
> Starting Socket Reader #1 for port 52820
> attempt_201310172123_0009_000000_0: 13/10/17 21:37:36 INFO ipc.Server: IPC
> Server Responder: starting
> attempt_201310172123_0009_000000_0: 13/10/17 21:37:36 INFO ipc.Server: IPC
> Server listener on 52820: starting
> attempt_201310172123_0009_000000_0: 13/10/17 21:37:36 INFO ipc.Server: IPC
> Server handler 0 on 52820: starting
> attempt_201310172123_0009_000000_0: 13/10/17 21:37:36 INFO
> message.HadoopMessageManagerImpl: BSPPeer address:master port:52820
> attempt_201310172123_0009_000000_0: 13/10/17 21:37:36 INFO
> sync.ZKSyncClient: Initializing ZK Sync Client
> attempt_201310172123_0009_000000_0: 13/10/17 21:37:36 INFO
> sync.ZooKeeperSyncClientImpl: Start connecting to Zookeeper! At
> master/192.168.1.2:52820
> attempt_201310172123_0009_000000_0: 13/10/17 21:37:37 ERROR bsp.BSPTask:
> Error running bsp setup and bsp function.
> attempt_201310172123_0009_000000_0: java.lang.NullPointerException
> attempt_201310172123_0009_000000_0: at
> org.apache.hama.graph.GraphJobRunner.loadVertices(GraphJobRunner.java:430)
> attempt_201310172123_0009_000000_0: at
> org.apache.hama.graph.GraphJobRunner.setup(GraphJobRunner.java:114)
> attempt_201310172123_0009_000000_0: at
> org.apache.hama.bsp.BSPTask.runBSP(BSPTask.java:176)
> attempt_201310172123_0009_000000_0: at
> org.apache.hama.bsp.BSPTask.run(BSPTask.java:146)
> attempt_201310172123_0009_000000_0: at
> org.apache.hama.bsp.GroomServer$BSPPeerChild.main(GroomServer.java:1246)
> attempt_201310172123_0009_000000_0: 13/10/17 21:37:37 INFO ipc.Server:
> Stopping server on 52820
> attempt_201310172123_0009_000000_0: 13/10/17 21:37:37 INFO ipc.Server: IPC
> Server handler 0 on 52820: exiting
> attempt_201310172123_0009_000000_0: 13/10/17 21:37:37 INFO ipc.Server:
> Stopping IPC Server listener on 52820
> attempt_201310172123_0009_000000_0: 13/10/17 21:37:37 INFO ipc.Server:
> Stopping IPC Server Responder
> attempt_201310172123_0009_000000_0: 13/10/17 21:37:37 ERROR bsp.BSPTask:
> Shutting down ping service.
> attempt_201310172123_0009_000000_0: 13/10/17 21:37:37 FATAL bsp.GroomServer:
> Error running child
> attempt_201310172123_0009_000000_0: java.lang.NullPointerException
> attempt_201310172123_0009_000000_0: at
> org.apache.hama.graph.GraphJobRunner.loadVertices(GraphJobRunner.java:430)
> attempt_201310172123_0009_000000_0: at
> org.apache.hama.graph.GraphJobRunner.setup(GraphJobRunner.java:114)
> attempt_201310172123_0009_000000_0: at
> org.apache.hama.bsp.BSPTask.runBSP(BSPTask.java:176)
> attempt_201310172123_0009_000000_0: at
> org.apache.hama.bsp.BSPTask.run(BSPTask.java:146)
> attempt_201310172123_0009_000000_0: at
> org.apache.hama.bsp.GroomServer$BSPPeerChild.main(GroomServer.java:1246)
> attempt_201310172123_0009_000000_0: java.lang.NullPointerException
> attempt_201310172123_0009_000000_0: at
> org.apache.hama.graph.GraphJobRunner.loadVertices(GraphJobRunner.java:430)
> attempt_201310172123_0009_000000_0: at
> org.apache.hama.graph.GraphJobRunner.setup(GraphJobRunner.java:114)
> attempt_201310172123_0009_000000_0: at
> org.apache.hama.bsp.BSPTask.runBSP(BSPTask.java:176)
> attempt_201310172123_0009_000000_0: at
> org.apache.hama.bsp.BSPTask.run(BSPTask.java:146)
> attempt_201310172123_0009_000000_0: at
> org.apache.hama.bsp.GroomServer$BSPPeerChild.main(GroomServer.java:1246)
> 13/10/17 21:38:44 INFO bsp.BSPJobClient: Job failed.
>
>
> From what I see, it has to do something either with the setup and bsp
> functions or with the reader because there is NullPointerException at
> loadVertices.
>
> I don't think it is the first one cause I am using hama Graph.
>
> Any ideas?
> Thanks
>
--
Best Regards, Edward J. Yoon
@eddieyoon
Re: Problem with 0.6.3
Posted by ik...@csd.auth.gr.
Well I have a test app where it just reads the input, which is:
1 2 3 4 5
2 1 3
3 1 2 4
4 1 3
5 1
and I just send a message but it doesn't even start. The error is:
13/10/19 18:56:27 ERROR bsp.LocalBSPRunner: Exception during BSP execution!
java.lang.NullPointerException
at org.apache.hama.bsp.PartitioningRunner.bsp(PartitioningRunner.java:207)
at org.apache.hama.bsp.LocalBSPRunner$BSPRunner.run(LocalBSPRunner.java:256)
at org.apache.hama.bsp.LocalBSPRunner$BSPRunner.call(LocalBSPRunner.java:286)
at org.apache.hama.bsp.LocalBSPRunner$BSPRunner.call(LocalBSPRunner.java:211)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
at java.lang.Thread.run(Thread.java:662)
13/10/19 18:56:27 ERROR bsp.LocalBSPRunner: Exception during BSP execution!
java.util.concurrent.ExecutionException: java.lang.NullPointerException
at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
at java.util.concurrent.FutureTask.get(FutureTask.java:83)
at
org.apache.hama.bsp.LocalBSPRunner$ThreadObserver.run(LocalBSPRunner.java:313)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.lang.NullPointerException
at org.apache.hama.bsp.PartitioningRunner.bsp(PartitioningRunner.java:207)
at org.apache.hama.bsp.LocalBSPRunner$BSPRunner.run(LocalBSPRunner.java:256)
at org.apache.hama.bsp.LocalBSPRunner$BSPRunner.call(LocalBSPRunner.java:286)
at org.apache.hama.bsp.LocalBSPRunner$BSPRunner.call(LocalBSPRunner.java:211)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
... 1 more
13/10/19 18:56:30 INFO bsp.BSPJobClient: Current supersteps number: 1
13/10/19 18:56:30 INFO bsp.BSPJobClient: Job failed.
13/10/19 18:56:30 ERROR bsp.BSPJobClient: Error partitioning the input path.
Exception in thread "main" java.io.IOException: Runtime partition
failed for the job.
at org.apache.hama.bsp.BSPJobClient.partition(BSPJobClient.java:465)
at org.apache.hama.bsp.BSPJobClient.submitJobInternal(BSPJobClient.java:333)
at org.apache.hama.bsp.BSPJobClient.submitJob(BSPJobClient.java:293)
at org.apache.hama.bsp.BSPJob.submit(BSPJob.java:228)
at org.apache.hama.graph.GraphJob.submit(GraphJob.java:203)
at org.apache.hama.bsp.BSPJob.waitForCompletion(BSPJob.java:235)
at hamatest.HamaTest.main(HamaTest.java:104)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hama.util.RunJar.main(RunJar.java:146)
What is this NullPointerException? I've seen it when NumBspTasks
wasn't specified, but I have job.setNumBspTasks(1) and still happens.
I am also running in local mode and the only thing specified in
hama-site.xml is the hdfs.
Another issue with the same jar.
When I run it again on local mode and nothing is mentioned in
hama-site.xml, the output is this:
13/10/19 19:12:48 WARN util.NativeCodeLoader: Unable to load
native-hadoop library for your platform... using builtin-java classes
where applicable
13/10/19 19:12:48 INFO bsp.FileInputFormat: Total input paths to process : 1
13/10/19 19:12:48 WARN snappy.LoadSnappy: Snappy native library not loaded
13/10/19 19:12:49 INFO bsp.FileInputFormat: Total input paths to process : 1
13/10/19 19:12:49 INFO bsp.BSPJobClient: Running job: job_localrunner_0001
13/10/19 19:12:49 INFO bsp.LocalBSPRunner: Setting up a new barrier
for 1 tasks!
13/10/19 19:12:52 INFO bsp.BSPJobClient: Current supersteps number: 1
13/10/19 19:12:52 INFO bsp.BSPJobClient: The total number of supersteps: 1
13/10/19 19:12:52 INFO bsp.BSPJobClient: Counters: 6
13/10/19 19:12:52 INFO bsp.BSPJobClient:
org.apache.hama.bsp.JobInProgress$JobCounter
13/10/19 19:12:52 INFO bsp.BSPJobClient: SUPERSTEPS=1
13/10/19 19:12:52 INFO bsp.BSPJobClient: LAUNCHED_TASKS=1
13/10/19 19:12:52 INFO bsp.BSPJobClient:
org.apache.hama.bsp.BSPPeerImpl$PeerCounter
13/10/19 19:12:52 INFO bsp.BSPJobClient: SUPERSTEP_SUM=2
13/10/19 19:12:52 INFO bsp.BSPJobClient: IO_BYTES_READ=35
13/10/19 19:12:52 INFO bsp.BSPJobClient: TIME_IN_SYNC_MS=0
13/10/19 19:12:52 INFO bsp.BSPJobClient: TASK_INPUT_RECORDS=6
Exception in thread "main"
org.apache.hadoop.mapred.InvalidInputException: Input path does not
exist: /tmp/hama-parts/job_localrunner_0001
at org.apache.hama.bsp.FileInputFormat.listStatus(FileInputFormat.java:163)
at
org.apache.hama.bsp.SequenceFileInputFormat.listStatus(SequenceFileInputFormat.java:45)
at org.apache.hama.bsp.FileInputFormat.getSplits(FileInputFormat.java:180)
at org.apache.hama.bsp.BSPJobClient.submitJobInternal(BSPJobClient.java:337)
at org.apache.hama.bsp.BSPJobClient.submitJob(BSPJobClient.java:293)
at org.apache.hama.bsp.BSPJob.submit(BSPJob.java:228)
at org.apache.hama.graph.GraphJob.submit(GraphJob.java:203)
at org.apache.hama.bsp.BSPJob.waitForCompletion(BSPJob.java:235)
at hamatest.HamaTest.main(HamaTest.java:104)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hama.util.RunJar.main(RunJar.java:146)
I don't understand again why this is happening. The input path I have
specified is not "/tmp/hama-parts/job_localrunner_0001". I didn't
repsond immediately cause I wanted to search it more by myself but I
give up.
Hope you can help.
Ilias
Quoting Anastasis Andronidis <an...@hotmail.com>:
> Hey,
>
> I suggest you try the following as a general debug approach:
>
> 1) Can you test your input alone with a simple job that is just
> parsing it? You can be sure at this point that something is not
> wrong inside you code. Like lib dependences with problems, etc.
> 2) Check your job config. You might have keys and values to your
> classes that are not fine represented, so Hama can not parse it.
> 3) Try to run Hama in local mode, just to be sure it's not a network problem.
>
> If you still have problem, can you send a part of your input that
> you just checked that fails? So we can search for what is wrong.
>
> Cheers,
> Anastasis
>
> On 17 Οκτ 2013, at 9:34 μ.μ., Ηλίας Καπουράνης <ik...@csd.auth.gr> wrote:
>
>> Hey,
>>
>> I only have 1 job. I don't know why it starts 2 jobs.
>>
>>
>>
>> Στις 17/10/2013 9:08 μμ, ο/η Anastasis Andronidis έγραψε:
>>> Hi,
>>>
>>> this seams to be a problem with the input. If you say that your
>>> first job produces nothing, then the second job is getting null
>>> input and this is probably why you get this error. I think your
>>> problem might be in your first job.
>>>
>>> Can you separate the jobs and investigate why this is happening?
>>> Or can you create a custom input for your second job and run it
>>> with a specific input to be sure that the problem is there?
>>>
>>> Cheers,
>>> Anastasis
>>>
>>> On 17 Οκτ 2013, at 6:44 μ.μ., ikapoura@csd.auth.gr wrote:
>>>
>>>> Hello there,
>>>>
>>>> i just downloaded the version 0.6.3, did the configurations and
>>>> wanted to run my programs. Previously I was using hama 0.6.0. The
>>>> examples are working fine but when I give one jar of mine to be
>>>> run, I have the following issue:
>>>>
>>>> - It starts 2 jobs and after the first is completed without doing
>>>> anything, the second hangs and prints this:
>>>>
>>>> attempt_201310172123_0009_000000_0: 13/10/17 21:37:36 INFO
>>>> ipc.Server: Starting Socket Reader #1 for port 52820
>>>> attempt_201310172123_0009_000000_0: 13/10/17 21:37:36 INFO
>>>> ipc.Server: IPC Server Responder: starting
>>>> attempt_201310172123_0009_000000_0: 13/10/17 21:37:36 INFO
>>>> ipc.Server: IPC Server listener on 52820: starting
>>>> attempt_201310172123_0009_000000_0: 13/10/17 21:37:36 INFO
>>>> ipc.Server: IPC Server handler 0 on 52820: starting
>>>> attempt_201310172123_0009_000000_0: 13/10/17 21:37:36 INFO
>>>> message.HadoopMessageManagerImpl: BSPPeer address:master
>>>> port:52820
>>>> attempt_201310172123_0009_000000_0: 13/10/17 21:37:36 INFO
>>>> sync.ZKSyncClient: Initializing ZK Sync Client
>>>> attempt_201310172123_0009_000000_0: 13/10/17 21:37:36 INFO
>>>> sync.ZooKeeperSyncClientImpl: Start connecting to Zookeeper! At
>>>> master/192.168.1.2:52820
>>>> attempt_201310172123_0009_000000_0: 13/10/17 21:37:37 ERROR
>>>> bsp.BSPTask: Error running bsp setup and bsp function.
>>>> attempt_201310172123_0009_000000_0: java.lang.NullPointerException
>>>> attempt_201310172123_0009_000000_0: at
>>>> org.apache.hama.graph.GraphJobRunner.loadVertices(GraphJobRunner.java:430)
>>>> attempt_201310172123_0009_000000_0: at
>>>> org.apache.hama.graph.GraphJobRunner.setup(GraphJobRunner.java:114)
>>>> attempt_201310172123_0009_000000_0: at
>>>> org.apache.hama.bsp.BSPTask.runBSP(BSPTask.java:176)
>>>> attempt_201310172123_0009_000000_0: at
>>>> org.apache.hama.bsp.BSPTask.run(BSPTask.java:146)
>>>> attempt_201310172123_0009_000000_0: at
>>>> org.apache.hama.bsp.GroomServer$BSPPeerChild.main(GroomServer.java:1246)
>>>> attempt_201310172123_0009_000000_0: 13/10/17 21:37:37 INFO
>>>> ipc.Server: Stopping server on 52820
>>>> attempt_201310172123_0009_000000_0: 13/10/17 21:37:37 INFO
>>>> ipc.Server: IPC Server handler 0 on 52820: exiting
>>>> attempt_201310172123_0009_000000_0: 13/10/17 21:37:37 INFO
>>>> ipc.Server: Stopping IPC Server listener on 52820
>>>> attempt_201310172123_0009_000000_0: 13/10/17 21:37:37 INFO
>>>> ipc.Server: Stopping IPC Server Responder
>>>> attempt_201310172123_0009_000000_0: 13/10/17 21:37:37 ERROR
>>>> bsp.BSPTask: Shutting down ping service.
>>>> attempt_201310172123_0009_000000_0: 13/10/17 21:37:37 FATAL
>>>> bsp.GroomServer: Error running child
>>>> attempt_201310172123_0009_000000_0: java.lang.NullPointerException
>>>> attempt_201310172123_0009_000000_0: at
>>>> org.apache.hama.graph.GraphJobRunner.loadVertices(GraphJobRunner.java:430)
>>>> attempt_201310172123_0009_000000_0: at
>>>> org.apache.hama.graph.GraphJobRunner.setup(GraphJobRunner.java:114)
>>>> attempt_201310172123_0009_000000_0: at
>>>> org.apache.hama.bsp.BSPTask.runBSP(BSPTask.java:176)
>>>> attempt_201310172123_0009_000000_0: at
>>>> org.apache.hama.bsp.BSPTask.run(BSPTask.java:146)
>>>> attempt_201310172123_0009_000000_0: at
>>>> org.apache.hama.bsp.GroomServer$BSPPeerChild.main(GroomServer.java:1246)
>>>> attempt_201310172123_0009_000000_0: java.lang.NullPointerException
>>>> attempt_201310172123_0009_000000_0: at
>>>> org.apache.hama.graph.GraphJobRunner.loadVertices(GraphJobRunner.java:430)
>>>> attempt_201310172123_0009_000000_0: at
>>>> org.apache.hama.graph.GraphJobRunner.setup(GraphJobRunner.java:114)
>>>> attempt_201310172123_0009_000000_0: at
>>>> org.apache.hama.bsp.BSPTask.runBSP(BSPTask.java:176)
>>>> attempt_201310172123_0009_000000_0: at
>>>> org.apache.hama.bsp.BSPTask.run(BSPTask.java:146)
>>>> attempt_201310172123_0009_000000_0: at
>>>> org.apache.hama.bsp.GroomServer$BSPPeerChild.main(GroomServer.java:1246)
>>>> 13/10/17 21:38:44 INFO bsp.BSPJobClient: Job failed.
>>>>
>>>>
>>>> From what I see, it has to do something either with the setup and
>>>> bsp functions or with the reader because there is
>>>> NullPointerException at loadVertices.
>>>>
>>>> I don't think it is the first one cause I am using hama Graph.
>>>>
>>>> Any ideas?
>>>> Thanks
>>>>
>>>>
>>
>>
Re: Problem with 0.6.3
Posted by Anastasis Andronidis <an...@hotmail.com>.
Hey,
I suggest you try the following as a general debug approach:
1) Can you test your input alone with a simple job that is just parsing it? You can be sure at this point that something is not wrong inside you code. Like lib dependences with problems, etc.
2) Check your job config. You might have keys and values to your classes that are not fine represented, so Hama can not parse it.
3) Try to run Hama in local mode, just to be sure it's not a network problem.
If you still have problem, can you send a part of your input that you just checked that fails? So we can search for what is wrong.
Cheers,
Anastasis
On 17 Οκτ 2013, at 9:34 μ.μ., Ηλίας Καπουράνης <ik...@csd.auth.gr> wrote:
> Hey,
>
> I only have 1 job. I don't know why it starts 2 jobs.
>
>
>
> Στις 17/10/2013 9:08 μμ, ο/η Anastasis Andronidis έγραψε:
>> Hi,
>>
>> this seams to be a problem with the input. If you say that your first job produces nothing, then the second job is getting null input and this is probably why you get this error. I think your problem might be in your first job.
>>
>> Can you separate the jobs and investigate why this is happening? Or can you create a custom input for your second job and run it with a specific input to be sure that the problem is there?
>>
>> Cheers,
>> Anastasis
>>
>> On 17 Οκτ 2013, at 6:44 μ.μ., ikapoura@csd.auth.gr wrote:
>>
>>> Hello there,
>>>
>>> i just downloaded the version 0.6.3, did the configurations and wanted to run my programs. Previously I was using hama 0.6.0. The examples are working fine but when I give one jar of mine to be run, I have the following issue:
>>>
>>> - It starts 2 jobs and after the first is completed without doing anything, the second hangs and prints this:
>>>
>>> attempt_201310172123_0009_000000_0: 13/10/17 21:37:36 INFO ipc.Server: Starting Socket Reader #1 for port 52820
>>> attempt_201310172123_0009_000000_0: 13/10/17 21:37:36 INFO ipc.Server: IPC Server Responder: starting
>>> attempt_201310172123_0009_000000_0: 13/10/17 21:37:36 INFO ipc.Server: IPC Server listener on 52820: starting
>>> attempt_201310172123_0009_000000_0: 13/10/17 21:37:36 INFO ipc.Server: IPC Server handler 0 on 52820: starting
>>> attempt_201310172123_0009_000000_0: 13/10/17 21:37:36 INFO message.HadoopMessageManagerImpl: BSPPeer address:master port:52820
>>> attempt_201310172123_0009_000000_0: 13/10/17 21:37:36 INFO sync.ZKSyncClient: Initializing ZK Sync Client
>>> attempt_201310172123_0009_000000_0: 13/10/17 21:37:36 INFO sync.ZooKeeperSyncClientImpl: Start connecting to Zookeeper! At master/192.168.1.2:52820
>>> attempt_201310172123_0009_000000_0: 13/10/17 21:37:37 ERROR bsp.BSPTask: Error running bsp setup and bsp function.
>>> attempt_201310172123_0009_000000_0: java.lang.NullPointerException
>>> attempt_201310172123_0009_000000_0: at org.apache.hama.graph.GraphJobRunner.loadVertices(GraphJobRunner.java:430)
>>> attempt_201310172123_0009_000000_0: at org.apache.hama.graph.GraphJobRunner.setup(GraphJobRunner.java:114)
>>> attempt_201310172123_0009_000000_0: at org.apache.hama.bsp.BSPTask.runBSP(BSPTask.java:176)
>>> attempt_201310172123_0009_000000_0: at org.apache.hama.bsp.BSPTask.run(BSPTask.java:146)
>>> attempt_201310172123_0009_000000_0: at org.apache.hama.bsp.GroomServer$BSPPeerChild.main(GroomServer.java:1246)
>>> attempt_201310172123_0009_000000_0: 13/10/17 21:37:37 INFO ipc.Server: Stopping server on 52820
>>> attempt_201310172123_0009_000000_0: 13/10/17 21:37:37 INFO ipc.Server: IPC Server handler 0 on 52820: exiting
>>> attempt_201310172123_0009_000000_0: 13/10/17 21:37:37 INFO ipc.Server: Stopping IPC Server listener on 52820
>>> attempt_201310172123_0009_000000_0: 13/10/17 21:37:37 INFO ipc.Server: Stopping IPC Server Responder
>>> attempt_201310172123_0009_000000_0: 13/10/17 21:37:37 ERROR bsp.BSPTask: Shutting down ping service.
>>> attempt_201310172123_0009_000000_0: 13/10/17 21:37:37 FATAL bsp.GroomServer: Error running child
>>> attempt_201310172123_0009_000000_0: java.lang.NullPointerException
>>> attempt_201310172123_0009_000000_0: at org.apache.hama.graph.GraphJobRunner.loadVertices(GraphJobRunner.java:430)
>>> attempt_201310172123_0009_000000_0: at org.apache.hama.graph.GraphJobRunner.setup(GraphJobRunner.java:114)
>>> attempt_201310172123_0009_000000_0: at org.apache.hama.bsp.BSPTask.runBSP(BSPTask.java:176)
>>> attempt_201310172123_0009_000000_0: at org.apache.hama.bsp.BSPTask.run(BSPTask.java:146)
>>> attempt_201310172123_0009_000000_0: at org.apache.hama.bsp.GroomServer$BSPPeerChild.main(GroomServer.java:1246)
>>> attempt_201310172123_0009_000000_0: java.lang.NullPointerException
>>> attempt_201310172123_0009_000000_0: at org.apache.hama.graph.GraphJobRunner.loadVertices(GraphJobRunner.java:430)
>>> attempt_201310172123_0009_000000_0: at org.apache.hama.graph.GraphJobRunner.setup(GraphJobRunner.java:114)
>>> attempt_201310172123_0009_000000_0: at org.apache.hama.bsp.BSPTask.runBSP(BSPTask.java:176)
>>> attempt_201310172123_0009_000000_0: at org.apache.hama.bsp.BSPTask.run(BSPTask.java:146)
>>> attempt_201310172123_0009_000000_0: at org.apache.hama.bsp.GroomServer$BSPPeerChild.main(GroomServer.java:1246)
>>> 13/10/17 21:38:44 INFO bsp.BSPJobClient: Job failed.
>>>
>>>
>>> From what I see, it has to do something either with the setup and bsp functions or with the reader because there is NullPointerException at loadVertices.
>>>
>>> I don't think it is the first one cause I am using hama Graph.
>>>
>>> Any ideas?
>>> Thanks
>>>
>>>
>
>
Re: Problem with 0.6.3
Posted by Ηλίας Καπουράνης <ik...@csd.auth.gr>.
Hey,
I only have 1 job. I don't know why it starts 2 jobs.
Στις 17/10/2013 9:08 μμ, ο/η Anastasis Andronidis έγραψε:
> Hi,
>
> this seams to be a problem with the input. If you say that your first job produces nothing, then the second job is getting null input and this is probably why you get this error. I think your problem might be in your first job.
>
> Can you separate the jobs and investigate why this is happening? Or can you create a custom input for your second job and run it with a specific input to be sure that the problem is there?
>
> Cheers,
> Anastasis
>
> On 17 Οκτ 2013, at 6:44 μ.μ., ikapoura@csd.auth.gr wrote:
>
>> Hello there,
>>
>> i just downloaded the version 0.6.3, did the configurations and wanted to run my programs. Previously I was using hama 0.6.0. The examples are working fine but when I give one jar of mine to be run, I have the following issue:
>>
>> - It starts 2 jobs and after the first is completed without doing anything, the second hangs and prints this:
>>
>> attempt_201310172123_0009_000000_0: 13/10/17 21:37:36 INFO ipc.Server: Starting Socket Reader #1 for port 52820
>> attempt_201310172123_0009_000000_0: 13/10/17 21:37:36 INFO ipc.Server: IPC Server Responder: starting
>> attempt_201310172123_0009_000000_0: 13/10/17 21:37:36 INFO ipc.Server: IPC Server listener on 52820: starting
>> attempt_201310172123_0009_000000_0: 13/10/17 21:37:36 INFO ipc.Server: IPC Server handler 0 on 52820: starting
>> attempt_201310172123_0009_000000_0: 13/10/17 21:37:36 INFO message.HadoopMessageManagerImpl: BSPPeer address:master port:52820
>> attempt_201310172123_0009_000000_0: 13/10/17 21:37:36 INFO sync.ZKSyncClient: Initializing ZK Sync Client
>> attempt_201310172123_0009_000000_0: 13/10/17 21:37:36 INFO sync.ZooKeeperSyncClientImpl: Start connecting to Zookeeper! At master/192.168.1.2:52820
>> attempt_201310172123_0009_000000_0: 13/10/17 21:37:37 ERROR bsp.BSPTask: Error running bsp setup and bsp function.
>> attempt_201310172123_0009_000000_0: java.lang.NullPointerException
>> attempt_201310172123_0009_000000_0: at org.apache.hama.graph.GraphJobRunner.loadVertices(GraphJobRunner.java:430)
>> attempt_201310172123_0009_000000_0: at org.apache.hama.graph.GraphJobRunner.setup(GraphJobRunner.java:114)
>> attempt_201310172123_0009_000000_0: at org.apache.hama.bsp.BSPTask.runBSP(BSPTask.java:176)
>> attempt_201310172123_0009_000000_0: at org.apache.hama.bsp.BSPTask.run(BSPTask.java:146)
>> attempt_201310172123_0009_000000_0: at org.apache.hama.bsp.GroomServer$BSPPeerChild.main(GroomServer.java:1246)
>> attempt_201310172123_0009_000000_0: 13/10/17 21:37:37 INFO ipc.Server: Stopping server on 52820
>> attempt_201310172123_0009_000000_0: 13/10/17 21:37:37 INFO ipc.Server: IPC Server handler 0 on 52820: exiting
>> attempt_201310172123_0009_000000_0: 13/10/17 21:37:37 INFO ipc.Server: Stopping IPC Server listener on 52820
>> attempt_201310172123_0009_000000_0: 13/10/17 21:37:37 INFO ipc.Server: Stopping IPC Server Responder
>> attempt_201310172123_0009_000000_0: 13/10/17 21:37:37 ERROR bsp.BSPTask: Shutting down ping service.
>> attempt_201310172123_0009_000000_0: 13/10/17 21:37:37 FATAL bsp.GroomServer: Error running child
>> attempt_201310172123_0009_000000_0: java.lang.NullPointerException
>> attempt_201310172123_0009_000000_0: at org.apache.hama.graph.GraphJobRunner.loadVertices(GraphJobRunner.java:430)
>> attempt_201310172123_0009_000000_0: at org.apache.hama.graph.GraphJobRunner.setup(GraphJobRunner.java:114)
>> attempt_201310172123_0009_000000_0: at org.apache.hama.bsp.BSPTask.runBSP(BSPTask.java:176)
>> attempt_201310172123_0009_000000_0: at org.apache.hama.bsp.BSPTask.run(BSPTask.java:146)
>> attempt_201310172123_0009_000000_0: at org.apache.hama.bsp.GroomServer$BSPPeerChild.main(GroomServer.java:1246)
>> attempt_201310172123_0009_000000_0: java.lang.NullPointerException
>> attempt_201310172123_0009_000000_0: at org.apache.hama.graph.GraphJobRunner.loadVertices(GraphJobRunner.java:430)
>> attempt_201310172123_0009_000000_0: at org.apache.hama.graph.GraphJobRunner.setup(GraphJobRunner.java:114)
>> attempt_201310172123_0009_000000_0: at org.apache.hama.bsp.BSPTask.runBSP(BSPTask.java:176)
>> attempt_201310172123_0009_000000_0: at org.apache.hama.bsp.BSPTask.run(BSPTask.java:146)
>> attempt_201310172123_0009_000000_0: at org.apache.hama.bsp.GroomServer$BSPPeerChild.main(GroomServer.java:1246)
>> 13/10/17 21:38:44 INFO bsp.BSPJobClient: Job failed.
>>
>>
>> From what I see, it has to do something either with the setup and bsp functions or with the reader because there is NullPointerException at loadVertices.
>>
>> I don't think it is the first one cause I am using hama Graph.
>>
>> Any ideas?
>> Thanks
>>
>>
Re: Problem with 0.6.3
Posted by Anastasis Andronidis <an...@hotmail.com>.
Hi,
this seams to be a problem with the input. If you say that your first job produces nothing, then the second job is getting null input and this is probably why you get this error. I think your problem might be in your first job.
Can you separate the jobs and investigate why this is happening? Or can you create a custom input for your second job and run it with a specific input to be sure that the problem is there?
Cheers,
Anastasis
On 17 Οκτ 2013, at 6:44 μ.μ., ikapoura@csd.auth.gr wrote:
> Hello there,
>
> i just downloaded the version 0.6.3, did the configurations and wanted to run my programs. Previously I was using hama 0.6.0. The examples are working fine but when I give one jar of mine to be run, I have the following issue:
>
> - It starts 2 jobs and after the first is completed without doing anything, the second hangs and prints this:
>
> attempt_201310172123_0009_000000_0: 13/10/17 21:37:36 INFO ipc.Server: Starting Socket Reader #1 for port 52820
> attempt_201310172123_0009_000000_0: 13/10/17 21:37:36 INFO ipc.Server: IPC Server Responder: starting
> attempt_201310172123_0009_000000_0: 13/10/17 21:37:36 INFO ipc.Server: IPC Server listener on 52820: starting
> attempt_201310172123_0009_000000_0: 13/10/17 21:37:36 INFO ipc.Server: IPC Server handler 0 on 52820: starting
> attempt_201310172123_0009_000000_0: 13/10/17 21:37:36 INFO message.HadoopMessageManagerImpl: BSPPeer address:master port:52820
> attempt_201310172123_0009_000000_0: 13/10/17 21:37:36 INFO sync.ZKSyncClient: Initializing ZK Sync Client
> attempt_201310172123_0009_000000_0: 13/10/17 21:37:36 INFO sync.ZooKeeperSyncClientImpl: Start connecting to Zookeeper! At master/192.168.1.2:52820
> attempt_201310172123_0009_000000_0: 13/10/17 21:37:37 ERROR bsp.BSPTask: Error running bsp setup and bsp function.
> attempt_201310172123_0009_000000_0: java.lang.NullPointerException
> attempt_201310172123_0009_000000_0: at org.apache.hama.graph.GraphJobRunner.loadVertices(GraphJobRunner.java:430)
> attempt_201310172123_0009_000000_0: at org.apache.hama.graph.GraphJobRunner.setup(GraphJobRunner.java:114)
> attempt_201310172123_0009_000000_0: at org.apache.hama.bsp.BSPTask.runBSP(BSPTask.java:176)
> attempt_201310172123_0009_000000_0: at org.apache.hama.bsp.BSPTask.run(BSPTask.java:146)
> attempt_201310172123_0009_000000_0: at org.apache.hama.bsp.GroomServer$BSPPeerChild.main(GroomServer.java:1246)
> attempt_201310172123_0009_000000_0: 13/10/17 21:37:37 INFO ipc.Server: Stopping server on 52820
> attempt_201310172123_0009_000000_0: 13/10/17 21:37:37 INFO ipc.Server: IPC Server handler 0 on 52820: exiting
> attempt_201310172123_0009_000000_0: 13/10/17 21:37:37 INFO ipc.Server: Stopping IPC Server listener on 52820
> attempt_201310172123_0009_000000_0: 13/10/17 21:37:37 INFO ipc.Server: Stopping IPC Server Responder
> attempt_201310172123_0009_000000_0: 13/10/17 21:37:37 ERROR bsp.BSPTask: Shutting down ping service.
> attempt_201310172123_0009_000000_0: 13/10/17 21:37:37 FATAL bsp.GroomServer: Error running child
> attempt_201310172123_0009_000000_0: java.lang.NullPointerException
> attempt_201310172123_0009_000000_0: at org.apache.hama.graph.GraphJobRunner.loadVertices(GraphJobRunner.java:430)
> attempt_201310172123_0009_000000_0: at org.apache.hama.graph.GraphJobRunner.setup(GraphJobRunner.java:114)
> attempt_201310172123_0009_000000_0: at org.apache.hama.bsp.BSPTask.runBSP(BSPTask.java:176)
> attempt_201310172123_0009_000000_0: at org.apache.hama.bsp.BSPTask.run(BSPTask.java:146)
> attempt_201310172123_0009_000000_0: at org.apache.hama.bsp.GroomServer$BSPPeerChild.main(GroomServer.java:1246)
> attempt_201310172123_0009_000000_0: java.lang.NullPointerException
> attempt_201310172123_0009_000000_0: at org.apache.hama.graph.GraphJobRunner.loadVertices(GraphJobRunner.java:430)
> attempt_201310172123_0009_000000_0: at org.apache.hama.graph.GraphJobRunner.setup(GraphJobRunner.java:114)
> attempt_201310172123_0009_000000_0: at org.apache.hama.bsp.BSPTask.runBSP(BSPTask.java:176)
> attempt_201310172123_0009_000000_0: at org.apache.hama.bsp.BSPTask.run(BSPTask.java:146)
> attempt_201310172123_0009_000000_0: at org.apache.hama.bsp.GroomServer$BSPPeerChild.main(GroomServer.java:1246)
> 13/10/17 21:38:44 INFO bsp.BSPJobClient: Job failed.
>
>
> From what I see, it has to do something either with the setup and bsp functions or with the reader because there is NullPointerException at loadVertices.
>
> I don't think it is the first one cause I am using hama Graph.
>
> Any ideas?
> Thanks
>
>
Problem with 0.6.3
Posted by ik...@csd.auth.gr.
Hello there,
i just downloaded the version 0.6.3, did the configurations and
wanted to run my programs. Previously I was using hama 0.6.0. The
examples are working fine but when I give one jar of mine to be run, I
have the following issue:
- It starts 2 jobs and after the first is completed without doing
anything, the second hangs and prints this:
attempt_201310172123_0009_000000_0: 13/10/17 21:37:36 INFO ipc.Server:
Starting Socket Reader #1 for port 52820
attempt_201310172123_0009_000000_0: 13/10/17 21:37:36 INFO ipc.Server:
IPC Server Responder: starting
attempt_201310172123_0009_000000_0: 13/10/17 21:37:36 INFO ipc.Server:
IPC Server listener on 52820: starting
attempt_201310172123_0009_000000_0: 13/10/17 21:37:36 INFO ipc.Server:
IPC Server handler 0 on 52820: starting
attempt_201310172123_0009_000000_0: 13/10/17 21:37:36 INFO
message.HadoopMessageManagerImpl: BSPPeer address:master port:52820
attempt_201310172123_0009_000000_0: 13/10/17 21:37:36 INFO
sync.ZKSyncClient: Initializing ZK Sync Client
attempt_201310172123_0009_000000_0: 13/10/17 21:37:36 INFO
sync.ZooKeeperSyncClientImpl: Start connecting to Zookeeper! At
master/192.168.1.2:52820
attempt_201310172123_0009_000000_0: 13/10/17 21:37:37 ERROR
bsp.BSPTask: Error running bsp setup and bsp function.
attempt_201310172123_0009_000000_0: java.lang.NullPointerException
attempt_201310172123_0009_000000_0: at
org.apache.hama.graph.GraphJobRunner.loadVertices(GraphJobRunner.java:430)
attempt_201310172123_0009_000000_0: at
org.apache.hama.graph.GraphJobRunner.setup(GraphJobRunner.java:114)
attempt_201310172123_0009_000000_0: at
org.apache.hama.bsp.BSPTask.runBSP(BSPTask.java:176)
attempt_201310172123_0009_000000_0: at
org.apache.hama.bsp.BSPTask.run(BSPTask.java:146)
attempt_201310172123_0009_000000_0: at
org.apache.hama.bsp.GroomServer$BSPPeerChild.main(GroomServer.java:1246)
attempt_201310172123_0009_000000_0: 13/10/17 21:37:37 INFO ipc.Server:
Stopping server on 52820
attempt_201310172123_0009_000000_0: 13/10/17 21:37:37 INFO ipc.Server:
IPC Server handler 0 on 52820: exiting
attempt_201310172123_0009_000000_0: 13/10/17 21:37:37 INFO ipc.Server:
Stopping IPC Server listener on 52820
attempt_201310172123_0009_000000_0: 13/10/17 21:37:37 INFO ipc.Server:
Stopping IPC Server Responder
attempt_201310172123_0009_000000_0: 13/10/17 21:37:37 ERROR
bsp.BSPTask: Shutting down ping service.
attempt_201310172123_0009_000000_0: 13/10/17 21:37:37 FATAL
bsp.GroomServer: Error running child
attempt_201310172123_0009_000000_0: java.lang.NullPointerException
attempt_201310172123_0009_000000_0: at
org.apache.hama.graph.GraphJobRunner.loadVertices(GraphJobRunner.java:430)
attempt_201310172123_0009_000000_0: at
org.apache.hama.graph.GraphJobRunner.setup(GraphJobRunner.java:114)
attempt_201310172123_0009_000000_0: at
org.apache.hama.bsp.BSPTask.runBSP(BSPTask.java:176)
attempt_201310172123_0009_000000_0: at
org.apache.hama.bsp.BSPTask.run(BSPTask.java:146)
attempt_201310172123_0009_000000_0: at
org.apache.hama.bsp.GroomServer$BSPPeerChild.main(GroomServer.java:1246)
attempt_201310172123_0009_000000_0: java.lang.NullPointerException
attempt_201310172123_0009_000000_0: at
org.apache.hama.graph.GraphJobRunner.loadVertices(GraphJobRunner.java:430)
attempt_201310172123_0009_000000_0: at
org.apache.hama.graph.GraphJobRunner.setup(GraphJobRunner.java:114)
attempt_201310172123_0009_000000_0: at
org.apache.hama.bsp.BSPTask.runBSP(BSPTask.java:176)
attempt_201310172123_0009_000000_0: at
org.apache.hama.bsp.BSPTask.run(BSPTask.java:146)
attempt_201310172123_0009_000000_0: at
org.apache.hama.bsp.GroomServer$BSPPeerChild.main(GroomServer.java:1246)
13/10/17 21:38:44 INFO bsp.BSPJobClient: Job failed.
From what I see, it has to do something either with the setup and bsp
functions or with the reader because there is NullPointerException at
loadVertices.
I don't think it is the first one cause I am using hama Graph.
Any ideas?
Thanks
Re: Setting application configuration for Hama Streaming
Posted by Roman Shapovalov <sh...@graphics.cs.msu.su>.
Edward, thanks.
> Alternatively, I think you can use the program arguments instead of configuration variables.
So, for using it within Python wrapper, I should pass the arguments
after the class name in -programArgs and hack the BSPRunner script,
right?
Regards,
Roman
On Tue, Oct 15, 2013 at 3:01 AM, Edward J. Yoon <ed...@apache.org> wrote:
> Hi Roman,
>
> Sorry for inconvenience. I checked current source code[1] and it seems that there's no way to specify an application configuration variables. So, I filed on JIRA[2].
>
>> peer.config.get("kmeans.max.iterations")
>
> Alternatively, I think you can use the program arguments instead of configuration variables.
>
> 1. http://svn.apache.org/repos/asf/hama/trunk/core/src/main/java/org/apache/hama/pipes/Submitter.java
> 2. https://issues.apache.org/jira/browse/HAMA-810
>
> --
> Best Regards, Edward J. Yoon
> @eddieyoon
>
> On Oct 15, 2013, at 12:17 AM, Roman Shapovalov <sh...@graphics.cs.msu.su> wrote:
>
>> Hi all,
>>
>> The k-means example for Hama Streaming contains lines like
>> peer.config.get("kmeans.max.iterations")
>>
>> How to set such application-specific parameters from outside?
>> I tried to use -D and -conf options for "hama pipes", but they were
>> not recognized.
>>
>> Thanks,
>> Roman
>
Re: Setting application configuration for Hama Streaming
Posted by "Edward J. Yoon" <ed...@apache.org>.
Hi Roman,
Sorry for inconvenience. I checked current source code[1] and it seems that there's no way to specify an application configuration variables. So, I filed on JIRA[2].
> peer.config.get("kmeans.max.iterations")
Alternatively, I think you can use the program arguments instead of configuration variables.
1. http://svn.apache.org/repos/asf/hama/trunk/core/src/main/java/org/apache/hama/pipes/Submitter.java
2. https://issues.apache.org/jira/browse/HAMA-810
--
Best Regards, Edward J. Yoon
@eddieyoon
On Oct 15, 2013, at 12:17 AM, Roman Shapovalov <sh...@graphics.cs.msu.su> wrote:
> Hi all,
>
> The k-means example for Hama Streaming contains lines like
> peer.config.get("kmeans.max.iterations")
>
> How to set such application-specific parameters from outside?
> I tried to use -D and -conf options for "hama pipes", but they were
> not recognized.
>
> Thanks,
> Roman
Re: Setting application configuration for Hama Streaming
Posted by "Edward J. Yoon" <ed...@apache.org>.
Hi Roman,
Sorry for inconvenience. I checked current source code[1] and it seems that there's no way to specify an application configuration variables. So, I filed on JIRA[2].
> peer.config.get("kmeans.max.iterations")
Alternatively, I think you can use the program arguments instead of configuration variables.
1. http://svn.apache.org/repos/asf/hama/trunk/core/src/main/java/org/apache/hama/pipes/Submitter.java
2. https://issues.apache.org/jira/browse/HAMA-810
--
Best Regards, Edward J. Yoon
@eddieyoon
On Oct 15, 2013, at 12:17 AM, Roman Shapovalov <sh...@graphics.cs.msu.su> wrote:
> Hi all,
>
> The k-means example for Hama Streaming contains lines like
> peer.config.get("kmeans.max.iterations")
>
> How to set such application-specific parameters from outside?
> I tried to use -D and -conf options for "hama pipes", but they were
> not recognized.
>
> Thanks,
> Roman