You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@giraph.apache.org by Mirko Kämpf <mi...@cloudera.com> on 2012/11/10 14:24:01 UTC
Re: Giraph and Hadoop Cloudera CDH4.1.1
Hello all,
I created a now pom.xml file, which allows compiling Giraph with the new
CDH 4.1.2 repository.
I attached it to this email.
But there are still some problems. I can not start the examples, because I
do not have
demo data and I do not jet know, what formats have to be used. Has anybody
some test data files to share for a public tutorial?
Currently I prepare a tutorial on working with Giraph, but until now I had
to find out, that
the wiki content is quite outdated. The ShortestPathExample looks nice, but
without
some hints I can not start. Did I miss some documentation?
If not, I will create it, but therefor I would like to ask for some help.
My plan is, to create some demo data files and some Demo-Apps comparable to
the benchmark classes but based
on real data. I have some correlation obtained from wikipedia user data,
but until now I can not analuse that data with Giraph.
I get the following error ( at the end of this mail ) if I want to start
the PageRankBenchmark.
Has anybody an idea, what I can do to solve this problem?
Thanks a lot!
Best wishes
Mirko
[kamir@master2 trunk]$ *hadoop jar
giraph/target/giraph-0.2-SNAPSHOT-for-hadoop-2.0.0-cdh4.2.1-jar-with-dependencies.jar
org.apache.giraph.benchmark.PageRankBenchmark -s 5 -v -w 5 -V 3 -e
80 *
12/11/10 14:12:35 INFO benchmark.PageRankBenchmark: Using class
org.apache.giraph.benchmark.EdgeListVertexPageRankBenchmark
12/11/10 14:12:36 WARN mapred.JobClient: Use GenericOptionsParser for
parsing the arguments. Applications should implement Tool for the
same.
12/11/10 14:12:36 WARN bsp.BspOutputFormat: checkOutputSpecs:
ImmutableOutputCommiter will not check
anything
12/11/10 14:12:37 INFO mapred.JobClient: Running job:
job_201211101312_0007
12/11/10 14:12:38 INFO mapred.JobClient: map 0% reduce
0%
12/11/10 14:12:49 INFO mapred.JobClient: Task Id :
attempt_201211101312_0007_m_000001_0, Status :
FAILED
*Error:
com.google.common.collect.Lists.newCopyOnWriteArrayList()Ljava/util/concurrent/CopyOnWriteArrayList;
*
attempt_201211101312_0007_m_000001_0: SLF4J: Class path contains multiple
SLF4J
bindings.
attempt_201211101312_0007_m_000001_0: SLF4J: Found binding in
[jar:file:/usr/lib/zookeeper/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
attempt_201211101312_0007_m_000001_0: SLF4J: Found binding in
[jar:file:/media/sda/mapred/local/taskTracker/kamir/jobcache/job_201211101312_0007/jars/job.jar!/org/slf4j/impl/StaticLoggerBinder.class]
attempt_201211101312_0007_m_000001_0: SLF4J: See
http://www.slf4j.org/codes.html#multiple_bindings for an
explanation.
12/11/10 14:12:49 INFO mapred.JobClient: Task Id :
attempt_201211101312_0007_m_000003_0, Status :
FAILED
Error:
com.google.common.collect.Lists.newCopyOnWriteArrayList()Ljava/util/concurrent/CopyOnWriteArrayList;
attempt_201211101312_0007_m_000003_0: SLF4J: Class path contains multiple
SLF4J
bindings.
attempt_201211101312_0007_m_000003_0: SLF4J: Found binding in
[jar:file:/usr/lib/zookeeper/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
attempt_201211101312_0007_m_000003_0: SLF4J: Found binding in
[jar:file:/media/sda/mapred/local/taskTracker/kamir/jobcache/job_201211101312_0007/jars/job.jar!/org/slf4j/impl/StaticLoggerBinder.class]
attempt_201211101312_0007_m_000003_0: SLF4J: See
http://www.slf4j.org/codes.html#multiple_bindings for an
explanation.
12/11/10 14:12:52 INFO mapred.JobClient: Task Id :
attempt_201211101312_0007_m_000002_0, Status :
FAILED
Error:
com.google.common.collect.Lists.newCopyOnWriteArrayList()Ljava/util/concurrent/CopyOnWriteArrayList;
attempt_201211101312_0007_m_000002_0: SLF4J: Class path contains multiple
SLF4J
bindings.
attempt_201211101312_0007_m_000002_0: SLF4J: Found binding in
[jar:file:/usr/lib/zookeeper/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
attempt_201211101312_0007_m_000002_0: SLF4J: Found binding in
[jar:file:/media/sda/mapred/local/taskTracker/kamir/jobcache/job_201211101312_0007/jars/job.jar!/org/slf4j/impl/StaticLoggerBinder.class]
attempt_201211101312_0007_m_000002_0: SLF4J: See
http://www.slf4j.org/codes.html#multiple_bindings for an
explanation.
12/11/10 14:12:53 INFO mapred.JobClient: map 16% reduce
0%
12/11/10 14:12:53 INFO mapred.JobClient: Task Id :
attempt_201211101312_0007_m_000003_1, Status :
FAILED
Error:
com.google.common.collect.Lists.newCopyOnWriteArrayList()Ljava/util/concurrent/CopyOnWriteArrayList;
attempt_201211101312_0007_m_000003_1: SLF4J: Class path contains multiple
SLF4J
bindings.
attempt_201211101312_0007_m_000003_1: SLF4J: Found binding in
[jar:file:/usr/lib/zookeeper/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
attempt_201211101312_0007_m_000003_1: SLF4J: Found binding in
[jar:file:/media/sda/mapred/local/taskTracker/kamir/jobcache/job_201211101312_0007/jars/job.jar!/org/slf4j/impl/StaticLoggerBinder.class]
attempt_201211101312_0007_m_000003_1: SLF4J: See
http://www.slf4j.org/codes.html#multiple_bindings for an
explanation.
12/11/10 14:12:54 INFO mapred.JobClient: Task Id :
attempt_201211101312_0007_m_000001_1, Status :
FAILED
Error:
com.google.common.collect.Lists.newCopyOnWriteArrayList()Ljava/util/concurrent/CopyOnWriteArrayList;
attempt_201211101312_0007_m_000001_1: SLF4J: Class path contains multiple
SLF4J
bindings.
attempt_201211101312_0007_m_000001_1: SLF4J: Found binding in
[jar:file:/usr/lib/zookeeper/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
attempt_201211101312_0007_m_000001_1: SLF4J: Found binding in
[jar:file:/media/sda/mapred/local/taskTracker/kamir/jobcache/job_201211101312_0007/jars/job.jar!/org/slf4j/impl/StaticLoggerBinder.class]
attempt_201211101312_0007_m_000001_1: SLF4J: See
http://www.slf4j.org/codes.html#multiple_bindings for an
explanation.
12/11/10 14:12:58 INFO mapred.JobClient: Task Id :
attempt_201211101312_0007_m_000002_1, Status :
FAILED
Error:
com.google.common.collect.Lists.newCopyOnWriteArrayList()Ljava/util/concurrent/CopyOnWriteArrayList;
attempt_201211101312_0007_m_000002_1: SLF4J: Class path contains multiple
SLF4J
bindings.
attempt_201211101312_0007_m_000002_1: SLF4J: Found binding in
[jar:file:/usr/lib/zookeeper/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
attempt_201211101312_0007_m_000002_1: SLF4J: Found binding in
[jar:file:/media/sda/mapred/local/taskTracker/kamir/jobcache/job_201211101312_0007/jars/job.jar!/org/slf4j/impl/StaticLoggerBinder.class]
attempt_201211101312_0007_m_000002_1: SLF4J: See
http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
12/11/10 14:12:59 INFO mapred.JobClient: Task Id :
attempt_201211101312_0007_m_000003_2, Status : FAILED
Error:
com.google.common.collect.Lists.newCopyOnWriteArrayList()Ljava/util/concurrent/CopyOnWriteArrayList;
attempt_201211101312_0007_m_000003_2: SLF4J: Class path contains multiple
SLF4J bindings.
attempt_201211101312_0007_m_000003_2: SLF4J: Found binding in
[jar:file:/usr/lib/zookeeper/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
attempt_201211101312_0007_m_000003_2: SLF4J: Found binding in
[jar:file:/media/sda/mapred/local/taskTracker/kamir/jobcache/job_201211101312_0007/jars/job.jar!/org/slf4j/impl/StaticLoggerBinder.class]
attempt_201211101312_0007_m_000003_2: SLF4J: See
http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
12/11/10 14:12:59 INFO mapred.JobClient: Task Id :
attempt_201211101312_0007_m_000001_2, Status : FAILED
Error:
com.google.common.collect.Lists.newCopyOnWriteArrayList()Ljava/util/concurrent/CopyOnWriteArrayList;
attempt_201211101312_0007_m_000001_2: SLF4J: Class path contains multiple
SLF4J bindings.
attempt_201211101312_0007_m_000001_2: SLF4J: Found binding in
[jar:file:/usr/lib/zookeeper/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
attempt_201211101312_0007_m_000001_2: SLF4J: Found binding in
[jar:file:/media/sda/mapred/local/taskTracker/kamir/jobcache/job_201211101312_0007/jars/job.jar!/org/slf4j/impl/StaticLoggerBinder.class]
attempt_201211101312_0007_m_000001_2: SLF4J: See
http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
12/11/10 14:13:04 INFO mapred.JobClient: Task Id :
attempt_201211101312_0007_m_000002_2, Status : FAILED
Error:
com.google.common.collect.Lists.newCopyOnWriteArrayList()Ljava/util/concurrent/CopyOnWriteArrayList;
attempt_201211101312_0007_m_000002_2: SLF4J: Class path contains multiple
SLF4J bindings.
attempt_201211101312_0007_m_000002_2: SLF4J: Found binding in
[jar:file:/usr/lib/zookeeper/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
attempt_201211101312_0007_m_000002_2: SLF4J: Found binding in
[jar:file:/media/sda/mapred/local/taskTracker/kamir/jobcache/job_201211101312_0007/jars/job.jar!/org/slf4j/impl/StaticLoggerBinder.class]
attempt_201211101312_0007_m_000002_2: SLF4J: See
http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
12/11/10 14:13:07 INFO mapred.JobClient: Job complete: job_201211101312_0007
12/11/10 14:13:07 INFO mapred.JobClient: Counters: 6
12/11/10 14:13:07 INFO mapred.JobClient: Job Counters
12/11/10 14:13:07 INFO mapred.JobClient: Failed map tasks=1
12/11/10 14:13:07 INFO mapred.JobClient: Launched map tasks=13
12/11/10 14:13:07 INFO mapred.JobClient: Total time spent by all maps
in occupied slots (ms)=51718
12/11/10 14:13:07 INFO mapred.JobClient: Total time spent by all
reduces in occupied slots (ms)=0
12/11/10 14:13:07 INFO mapred.JobClient: Total time spent by all maps
waiting after reserving slots (ms)=0
12/11/10 14:13:07 INFO mapred.JobClient: Total time spent by all
reduces waiting after reserving slots (ms)=0
[kamir@master2 trunk]$ hadoop fs -ls _bsp/
Found 1 items
drwxr-xr-x - kamir kamir 0 2012-11-10 14:12
_bsp/_defaultZkManagerDir
[kamir@master2 trunk]$ hadoop fs -ls _bsp//_defaultZkManagerDir
Found 1 items
drwxr-xr-x - kamir kamir 0 2012-11-10 14:12
_bsp/_defaultZkManagerDir/job_201211101312_0007
[kamir@master2 trunk]$
[kamir@master2 trunk]$ hadoop fs -ls
_bsp/_defaultZkManagerDir/job_201211101312_0007
Found 3 items
drwxr-xr-x - kamir kamir 0 2012-11-10 14:12
_bsp/_defaultZkManagerDir/job_201211101312_0007/_task
drwxr-xr-x - kamir kamir 0 2012-11-10 14:12
_bsp/_defaultZkManagerDir/job_201211101312_0007/_zkServer
-rw-r--r-- 3 kamir kamir 0 2012-11-10 14:12
_bsp/_defaultZkManagerDir/job_201211101312_0007/zkServerList_master2 0
[kamir@master2 trunk]$
On Wed, Nov 7, 2012 at 8:31 PM, Avery Ching <ac...@apache.org> wrote:
> Hi Franco,
>
> I haven't tried. Please let us know your experiences.
>
> Avery
>
>
> On 11/7/12 2:30 AM, Franco Maria Nardini wrote:
>
>> Dear all,
>>
>> I am trying to run Giraph on Hadoop Cloudera CDH4.1.1. I know that it is
>> not one of the Hadoop distributions reported to work with Giraph. However,
>> I was wondering if you guys try to do that? What is your experience on that?
>>
>> Thanks a lot in advance.
>>
>> Best,
>>
>> Franco Maria
>>
>
>
--
--
Mirko Kämpf
*Trainer* @ Cloudera
tel: +49 *176 20 63 51 99*
skype: *kamir1604*
mirko@cloudera.com
Re: Giraph and Hadoop Cloudera CDH4.1.1
Posted by Eugene Koontz <ek...@hiro-tan.org>.
Hi Mirko,
Thanks for looking at this and trying it on CDH! I filed a JIRA here:
https://issues.apache.org/jira/browse/GIRAPH-418
You are right that the docs need work too.
-Eugene