Re: Preconfigured BigTop running Giraph

[Martin Neumann <mn...@spotify.com>]

Hej,

The original idea was to run a "local-mode" giraph for programming and
testing without the need of configuration and the description on the BigTop
project sounded like that was possible.
Since I could not get it to work I just run things on the production
cluster even though its complicated debugging.

If anyone knows how to run Giraph on a single machine "cluster" without
loads setup and config tweaking I'm still interested. (preferred for the
1.1.0 trunk version)

cheers Martin



On Thu, Jan 30, 2014 at 2:23 AM, Eli Reisman <ap...@gmail.com>wrote:

> There is a heap check in there when setting up to launch the AM in the
> client module. It uses information coming back from the RM of your YARN
> cluster. I'm guessing its cluster configs?
>
>
> On Tue, Jan 14, 2014 at 5:24 PM, Roman Shaposhnik <rv...@apache.org> wrote:
>
>> Hi Martin,
>>
>> sorry for the belated reply. I am wondering how
>> did you configure your cluster? Did you use
>> Bigtop's puppet recipies or did you do it by hand?
>>
>> It seems that Giraph is working fine on the toy
>> cluster that I'm deploying with the Bigtop bits.
>> But I'm using Puppet and the topology is really
>> simple.
>>
>> Basically without seeing your hadoop and giraph
>> config files its pretty tough to answer your
>> question in any greater details.
>>
>> Thanks,
>> Roman.
>>
>> On Tue, Jan 7, 2014 at 5:44 AM, Martin Neumann <mn...@spotify.com>
>> wrote:
>> > Hej,
>> >
>> > I installed Giraph from the apache BigTop project and want to try to run
>> > some Giraph jobs locally on my machine. If I understood the website
>> > correctly it should be preconfigured to do so.
>> > But when I run a Giraph Job I get the following exception:
>> >
>> > Exception in thread "main" java.lang.IllegalStateException: Giraph's
>> > estimated cluster heap 2048MB ask is greater than the current available
>> > cluster heap of 0MB. Aborting Job.
>> >
>> >
>> > To me it sounds like a configuration problem, I'm not even sure if it
>> comes
>> > from Giraph or from Yarn. If it is a configuration issue is there help
>> page
>> > that tells you to configure the environment correctly?
>> > Thanks for the help.
>> >
>> >
>> >
>> >
>> >
>> > here the log of the whole execution:
>> >
>> > spotify@spotify-ThinkPad-T430s:~$ giraph
>> > /home/spotify/workspace/GiraphExe/SpotifyConComp.jar
>> > spotifyConnectedComponent.ConComp -eif
>> > spotifyConnectedComponent.ConCompInput -of
>> > spotifyConnectedComponent.ConCompOutput -c
>> > spotifyConnectedComponent.MinTextCombiner -eip
>> > /home/spotify/workspace/GiraphExe/in/sample -op
>> > /home/spotify/workspace/GiraphExe/out/spotifyConnectedComponent.ConComp
>> -w 1
>> > HADOOP_CONF_DIR=/etc/hadoop/conf
>> > SLF4J: Class path contains multiple SLF4J bindings.
>> > SLF4J: Found binding in
>> >
>> [jar:file:/usr/lib/hadoop/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>> > SLF4J: Found binding in
>> >
>> [jar:file:/usr/lib/giraph/lib/slf4j-log4j12-1.7.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
>> > SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
>> > explanation.
>> > 14/01/07 14:40:35 INFO utils.ConfigurationUtils: No vertex input format
>> > specified. Ensure your InputFormat does not require one.
>> > 14/01/07 14:40:35 INFO yarn.GiraphYarnClient: Final output path is:
>> >
>> hdfs://localhost:8020/home/spotify/workspace/GiraphExe/out/spotifyConnectedComponent.ConComp
>> > 14/01/07 14:40:35 INFO service.AbstractService:
>> > Service:org.apache.hadoop.yarn.client.YarnClientImpl is inited.
>> > 14/01/07 14:40:35 INFO service.AbstractService:
>> > Service:org.apache.hadoop.yarn.client.YarnClientImpl is started.
>> > 14/01/07 14:40:35 INFO yarn.GiraphYarnClient: Defaulting per-task heap
>> size
>> > to 1024MB.
>> > Exception in thread "main" java.lang.IllegalStateException: Giraph's
>> > estimated cluster heap 2048MB ask is greater than the current available
>> > cluster heap of 0MB. Aborting Job.
>> >     at
>> >
>> org.apache.giraph.yarn.GiraphYarnClient.checkPerNodeResourcesAvailable(GiraphYarnClient.java:204)
>> >     at
>> > org.apache.giraph.yarn.GiraphYarnClient.run(GiraphYarnClient.java:114)
>> >     at org.apache.giraph.GiraphRunner.run(GiraphRunner.java:96)
>> >     at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
>> >     at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
>> >     at org.apache.giraph.GiraphRunner.main(GiraphRunner.java:126)
>> >     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> >     at
>> >
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>> >     at
>> >
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> >     at java.lang.reflect.Method.invoke(Method.java:606)
>> >     at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
>>
>
>