You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@giraph.apache.org by Silvio Di gregorio <si...@gmail.com> on 2013/10/04 13:06:04 UTC
Fwd: connected components example in giraph 1.0
Hi,
I hope I have sent to the right address.
i have a graph (directed and unweighted) stored in hdfs like a adjacency
list (140Milions of edges 6Milions of vertex)
node<tab>neighbors
23 2 1 3 4 3
1 999 99923 909 .......
......
hadoop version Hadoop 2.0.0-cdh4.3.0 - java 1.6
I have executed the giraph-1.0 connected components example, in this fashion
> > hadoop jar /usr/local/giraph/giraph-examples/target/giraph-
> examples-1.0.0-for-hadoop-0.20.203.0-jar-with-dependencies.jar
> org.apache.giraph.GiraphRunner org.apache.giraph.examples.ConnectedComponentsVertex
> -vif org.apache.giraph.io.formats.IntIntNullTextInputFormat -vip
> /user/hdfs/lista_adj_txt -op connectedgiraph --workers 4
and then fail with:
13/10/04 09:28:29 INFO utils.ConfigurationUtils: No edge input format
specified. Ensure your InputFormat does not require one.
13/10/04 09:28:29 INFO utils.ConfigurationUtils: No output format
specified. Ensure your OutputFormat does not require one.
13/10/04 09:28:30 INFO job.GiraphJob: run: Since checkpointing is disabled
(default), do not allow any task retries (setting mapred.map.max.attempts =
0, old value = 4)
13/10/04 09:28:31 WARN mapred.JobClient: Use GenericOptionsParser for
parsing the arguments. Applications should implement Tool for the same.
13/10/04 09:28:31 INFO mapred.JobClient: Cleaning up the staging area
hdfs://
srv-bigdata-dev-01.int.sose.it:8020/user/hdfs/.staging/job_201309181636_0535
Exception in thread "main" java.lang.IncompatibleClassChangeError: Found
interface org.apache.hadoop.mapreduce.JobContext, but class was expected
at
org.apache.giraph.bsp.BspOutputFormat.checkOutputSpecs(BspOutputFormat.java:43)
......
Thanks in advance
Re: connected components example in giraph 1.0
Posted by Silvio Di gregorio <si...@gmail.com>.
*- you've successfully built it against a Hadoop 2.0.x release? *
no i've successfully built giraph 1.0 with cdh4.3 distribution, which have
inside YARN then MR2, but also MR1 hive pig impala hue etc ....
I suggest you use MR1 not MR2 ( that is YARN, tell me it's not stable)
Right to point out i use CDH Cloudera distribution because i took courses
with Cloudera and it's more easy to install for a non-System-Administrator
like me, there are many other Distributions.
*then use -Dhadoop.version=2.0.5 to build? *
i have used :
mvn clean install -P hadoop_cdh4.1.2
i don't believe exist "-D" option for Maven, should be "-P", -D it is used
for hadoop command line parameters.
However the fulcrum is the "pom.xml" file of giraph, inside this you can
see all the build possibility.
You can report this the error, you may have to wrong input format out put
format and so on:
how is your graph rapresentation? adjacency list?
care to the isolated vertex, Marco told me that the vertex only in the
neighbors list can cause problem, ex:
<source>\t<node1\tnode2\t ....>
1 2 3
2 4 1
....
in the adjacency list, 3 and 4 are prensent only like neighbors [2,3] and
[4,1], i believe you must replicate
1 2 3
1 4 1
*3*
*4*
the guy (giraph) is very susceptible.
2013/10/8 Matthew Laird <la...@sfu.ca>
> Thanks, I appreciate the feedback! It's why open source projects are great.
>
> So just to be clear, I should grab the 1.0 release, and you've
> successfully built it against a Hadoop 2.0.x release? I add that patch
> from Claudio first, then use -Dhadoop.version=2.0.5 to build? It will
> still have the MR API in place?
>
> I'll get through that then see what happens with this whole split
> master-worker setup I have on my Hadoop installation which caused the
> examples to fail. Then on to trying to load my billion vertices and
> clustering the graph (I basically want all the sub-groups of connected
> components, these will be the related genes across all sequenced bacteria).
>
> Thanks again!
>
> On 13-10-07 12:56 PM, Silvio Di gregorio wrote:
>
>> with a working day in my shoulders and my children around i gave you
>> random version,
>> tomorrow morning in farm i'll be more precise.
>> though i have hadoop 2.0 i use MR1 not YARN.
>> I'm sorry again.
>>
>>
>> 2013/10/7 Matthew Laird <lairdm@sfu.ca <ma...@sfu.ca>>
>>
>> Hmm, it seems there have been some recent pushes that have broken
>> trunk as I discovered yesterday. :)
>>
>> But I guess that brings me back to my original question, to build
>> against 2.0.5 using the MR interface rather than YARN as Milinda
>> suggested, what profile would I use? Would I let it think it's
>> 0.20xxx or tell it 2.0.5, or something else?
>>
>> Thanks.
>>
>>
>> On 13-10-07 11:50 AM, Claudio Martella wrote:
>>
>> trunk is 1.1.0-SNAPSHOT. I believe the date refers to the last
>> update to
>> the site.
>>
>>
>> On Mon, Oct 7, 2013 at 8:48 PM, Matthew Laird <lairdm@sfu.ca
>> <ma...@sfu.ca>
>> <mailto:lairdm@sfu.ca <ma...@sfu.ca>>> wrote:
>>
>> Thanks, which is my next curiosity... if I head over to
>> http://giraph.apache.org/ I see at the top is says
>>
>> "Version: 1.1.0-SNAPSHOT | Last Published: 2013-08-10"
>>
>> However it's not clickable and the only download links I
>> see are for
>> 1.0.0. I'm assuming trunk in git isn't the same as the 1.1.0
>> snapshot since more pushes have occurred since August, and
>> I don't
>> see a tag in git for 1.1.0
>> (https://git-wip-us.apache.___**_org/repos/asf?p=giraph.git
>> <https://git-wip-us.apache.__**org/repos/asf?p=giraph.git
>> <https://git-wip-us.apache.**org/repos/asf?p=giraph.git<https://git-wip-us.apache.org/repos/asf?p=giraph.git>>>).
>> So where
>>
>> does one get 1.1.0-SNAPSHOT from?
>>
>> Thanks.
>>
>>
>> On 13-10-07 11:39 AM, Silvio Di gregorio wrote:
>>
>> Hi Matthew
>> I use giraph 1.1.0 and cdh4 hadoop 0.23
>>
>> the patch of Marco:
>>
>> http://www.mail-archive.com/__**____user@giraph.apache.org/___**
>> ___msg00945/check.diff<http://www.mail-archive.com/______user@giraph.apache.org/______msg00945/check.diff>
>> <http://www.mail-archive.com/_**___user@giraph.apache.org/____**
>> msg00945/check.diff<http://www.mail-archive.com/____user@giraph.apache.org/____msg00945/check.diff>
>> >
>> <http://www.mail-archive.com/_**___user@giraph.apache.org/____**
>> msg00945/check.diff<http://www.mail-archive.com/____user@giraph.apache.org/____msg00945/check.diff>
>> <http://www.mail-archive.com/_**_user@giraph.apache.org/__**
>> msg00945/check.diff<http://www.mail-archive.com/__user@giraph.apache.org/__msg00945/check.diff>
>> >>
>>
>>
>> <http://www.mail-archive.com/_**___user@giraph.apache.org/____**
>> msg00945/check.diff<http://www.mail-archive.com/____user@giraph.apache.org/____msg00945/check.diff>
>> <http://www.mail-archive.com/_**_user@giraph.apache.org/__**
>> msg00945/check.diff<http://www.mail-archive.com/__user@giraph.apache.org/__msg00945/check.diff>
>> >
>> <http://www.mail-archive.com/_**_user@giraph.apache.org/__**
>> msg00945/check.diff<http://www.mail-archive.com/__user@giraph.apache.org/__msg00945/check.diff>
>> <http://www.mail-archive.com/**user@giraph.apache.org/**
>> msg00945/check.diff<http://www.mail-archive.com/user@giraph.apache.org/msg00945/check.diff>
>> >>>
>>
>> and build
>>
>> mvn clean install -P hadoop_cdh4.1.2
>>
>>
>>
>> Il giorno 07/ott/2013 20:20, "Matthew Laird"
>> <lairdm@sfu.ca <ma...@sfu.ca>
>> <mailto:lairdm@sfu.ca <ma...@sfu.ca>>
>> <mailto:lairdm@sfu.ca <ma...@sfu.ca>
>> <mailto:lairdm@sfu.ca <ma...@sfu.ca>>>> ha scritto:
>>
>>
>>
>> Hello, so I'm watching this thread with interest
>> as well, since
>> that's the exact example I'll be trying to run as
>> well, and
>> I see
>> Silvio appears to be asking about running Giraph 1.0
>> against Hadoop
>> 2.0.0.
>>
>> I'm curious, is that a known working combination,
>> 1.0 vs
>> 2.0.x? And
>> if so, are there patches to apply first and what
>> maven
>> command line
>> should I build with? (ie. will a
>> -Dhadoop.version=2.0.5
>> give the
>> needed MR based API or will it use YARN?) I
>> figure it's
>> better to
>> ask than aimlessly try all the combinations. :)
>>
>> Thanks!
>>
>> On 13-10-07 11:01 AM, Claudio Martella wrote:
>>
>> try to increase the mappers' heap space through
>> mapred.child.java.opts.
>>
>>
>> On Mon, Oct 7, 2013 at 1:18 PM, Silvio Di
>> gregorio
>> <silvio.digregorio@gmail.com
>> <ma...@gmail.com>
>> >
>> <mailto:silvio.digregorio@__gm**ail.com <http://gmail.com>
>> <ma...@gmail.com>
>> >>
>> <mailto:silvio.digregorio@
>> <mailto:silvio.digregorio@>__g**m__ail.com <http://gm__ail.com> <
>> http://gmail.com>
>> <mailto:silvio.digregorio@__gm**ail.com <http://gmail.com>
>> <ma...@gmail.com>
>> >>>
>> <mailto:silvio.digregorio@ <mailto:silvio.digregorio@>
>> <mailto:silvio.digregorio@
>> <mailto:silvio.digregorio@>>__**g__m__ail.com<http://g__m__ail.com><
>> http://gm__ail.com>
>> <http://gmail.com>
>>
>>
>> <mailto:silvio.digregorio@
>> <mailto:silvio.digregorio@>__g**m__ail.com <http://gm__ail.com> <
>> http://gmail.com>
>> <mailto:silvio.digregorio@__gm**ail.com <http://gmail.com>
>> <ma...@gmail.com>>>>>>
>> wrote:
>>
>>
>> I think now my issue is another, thanks
>> to all.
>>
>>
>>
>>
>> java.lang.______**IllegalStateException:
>> run: Caught an
>>
>>
>> unrecoverable exception waitFor:
>> ExecutionException
>> occurred
>> while waiting for
>>
>>
>> org.apache.giraph.utils.______**ProgressableUtils$______**
>> FutureWaitable@73a01e8a
>> at
>>
>>
>> org.apache.giraph.graph.______**GraphMapper.run(GraphMapper.__**
>> ____java:102)
>>
>>
>>
>> ......
>> Caused by:
>> java.lang.______**IllegalStateException:
>>
>> waitFor:
>>
>> ExecutionException occurred while waiting for
>>
>>
>> org.apache.giraph.utils.______**ProgressableUtils$______**
>> FutureWaitable@73a01e8a
>> at
>>
>>
>> org.apache.giraph.utils.______**ProgressableUtils.waitFor(____**
>> __ProgressableUtils.java:151)
>>
>>
>>
>>
>> ... 7 more
>> Caused by:
>> java.util.concurrent.______**ExecutionException:
>> java.lang.OutOfMemoryError:*__**____Java heap
>> space*
>> at
>>
>>
>> org.apache.giraph.utils.______**ProgressableUtils.waitFor(____**
>> __ProgressableUtils.java:143)
>>
>>
>>
>>
>>
>> ... 13 more
>>
>>
>>
>> 2013/10/7 Claudio Martella
>> <claudio.martella@gmail.com <ma...@gmail.com>
>> >
>> <mailto:claudio.martella@__gma**il.com <http://gmail.com>
>> <mailto:claudio.martella@**gmail.com <cl...@gmail.com>
>> >>
>> <mailto:claudio.martella@
>> <mailto:claudio.martella@>__gm**a__il.com <http://gma__il.com> <
>> http://gmail.com>
>> <mailto:claudio.martella@__gma**il.com <http://gmail.com>
>> <mailto:claudio.martella@**gmail.com <cl...@gmail.com>
>> >>>
>> <mailto:claudio.martella@ <mailto:claudio.martella@>
>> <mailto:claudio.martella@
>> <mailto:claudio.martella@>>__g**m__a__il.com<http://gm__a__il.com><
>> http://gma__il.com>
>> <http://gmail.com>
>>
>>
>> <mailto:claudio.martella@
>> <mailto:claudio.martella@>__gm**a__il.com <http://gma__il.com> <
>> http://gmail.com>
>> <mailto:claudio.martella@__gma**il.com <http://gmail.com>
>> <mailto:claudio.martella@**gmail.com <cl...@gmail.com>
>> >>>>>
>>
>> OK, thanks.
>>
>> I really have to push that patch in.
>>
>>
>> On Mon, Oct 7, 2013 at 12:17 PM,
>> Silvio Di
>> gregorio
>> <silvio.digregorio@gmail.com <mailto:silvio.digregorio@**
>> gmail.com <si...@gmail.com>>
>> <mailto:silvio.digregorio@__gm**ail.com <http://gmail.com>
>> <ma...@gmail.com>
>> >>
>> <mailto:silvio.digregorio@
>> <mailto:silvio.digregorio@>__g**m__ail.com <http://gm__ail.com> <
>> http://gmail.com>
>> <mailto:silvio.digregorio@__gm**ail.com <http://gmail.com>
>> <ma...@gmail.com>
>> >>>
>> <mailto:silvio.digregorio@ <mailto:silvio.digregorio@>
>> <mailto:silvio.digregorio@
>> <mailto:silvio.digregorio@>>__**g__m__ail.com<http://g__m__ail.com><
>> http://gm__ail.com>
>> <http://gmail.com>
>>
>>
>> <mailto:silvio.digregorio@
>> <mailto:silvio.digregorio@>__g**m__ail.com <http://gm__ail.com> <
>> http://gmail.com>
>> <mailto:silvio.digregorio@__gm**ail.com <http://gmail.com>
>> <ma...@gmail.com>>>>>>
>> wrote:
>>
>> yes i do,
>> i have seen this in your post in:
>> http://www.mail-archive.com/__**____user@giraph.apache.org/___**
>> ___msg00957.html<http://www.mail-archive.com/______user@giraph.apache.org/______msg00957.html>
>> <http://www.mail-archive.com/_**___user@giraph.apache.org/____**
>> msg00957.html<http://www.mail-archive.com/____user@giraph.apache.org/____msg00957.html>
>> >
>> <http://www.mail-archive.com/_**___user@giraph.apache.org/____**
>> msg00957.html<http://www.mail-archive.com/____user@giraph.apache.org/____msg00957.html>
>> <http://www.mail-archive.com/_**_user@giraph.apache.org/__**
>> msg00957.html<http://www.mail-archive.com/__user@giraph.apache.org/__msg00957.html>
>> >>
>>
>>
>> <http://www.mail-archive.com/_**___user@giraph.apache.org/____**
>> msg00957.html<http://www.mail-archive.com/____user@giraph.apache.org/____msg00957.html>
>> <http://www.mail-archive.com/_**_user@giraph.apache.org/__**
>> msg00957.html<http://www.mail-archive.com/__user@giraph.apache.org/__msg00957.html>
>> >
>> <http://www.mail-archive.com/_**_user@giraph.apache.org/__**
>> msg00957.html<http://www.mail-archive.com/__user@giraph.apache.org/__msg00957.html>
>> <http://www.mail-archive.com/**user@giraph.apache.org/**
>> msg00957.html<http://www.mail-archive.com/user@giraph.apache.org/msg00957.html>
>> >>>
>>
>> excuse me if i had checked in
>> the mail-achive
>> first I would
>> have avoided the last post.
>>
>> Now zk issue are resolved.
>>
>>
>> 2013/10/7 Claudio Martella
>> <claudio.martella@gmail.com <ma...@gmail.com>
>> >
>> <mailto:claudio.martella@__gma**il.com <http://gmail.com>
>> <mailto:claudio.martella@**gmail.com <cl...@gmail.com>
>> >>
>> <mailto:claudio.martella@
>> <mailto:claudio.martella@>__gm**a__il.com <http://gma__il.com> <
>> http://gmail.com>
>> <mailto:claudio.martella@__gma**il.com <http://gmail.com>
>> <mailto:claudio.martella@**gmail.com <cl...@gmail.com>
>> >>>
>> <mailto:claudio.martella@ <mailto:claudio.martella@>
>> <mailto:claudio.martella@
>> <mailto:claudio.martella@>>__g**m__a__il.com<http://gm__a__il.com><
>> http://gma__il.com>
>> <http://gmail.com>
>>
>>
>> <mailto:claudio.martella@
>> <mailto:claudio.martella@>__gm**a__il.com <http://gma__il.com> <
>> http://gmail.com>
>> <mailto:claudio.martella@__gma**il.com <http://gmail.com>
>> <mailto:claudio.martella@**gmail.com <cl...@gmail.com>
>> >>>>>
>>
>> Can you try applying this one
>> first?
>>
>> <<< <
>> http://www.mail-archive.com/__**____user@giraph.apache.org/___**
>> ___msg00945/check.diff<http://www.mail-archive.com/______user@giraph.apache.org/______msg00945/check.diff>
>> <http://www.mail-archive.com/_**___user@giraph.apache.org/____**
>> msg00945/check.diff<http://www.mail-archive.com/____user@giraph.apache.org/____msg00945/check.diff>
>> >
>> <http://www.mail-archive.com/_**___user@giraph.apache.org/____**
>> msg00945/check.diff<http://www.mail-archive.com/____user@giraph.apache.org/____msg00945/check.diff>
>> <http://www.mail-archive.com/_**_user@giraph.apache.org/__**
>> msg00945/check.diff<http://www.mail-archive.com/__user@giraph.apache.org/__msg00945/check.diff>
>> >>
>>
>>
>> <http://www.mail-archive.com/_**___user@giraph.apache.org/____**
>> msg00945/check.diff<http://www.mail-archive.com/____user@giraph.apache.org/____msg00945/check.diff>
>> <http://www.mail-archive.com/_**_user@giraph.apache.org/__**
>> msg00945/check.diff<http://www.mail-archive.com/__user@giraph.apache.org/__msg00945/check.diff>
>> >
>> <http://www.mail-archive.com/_**_user@giraph.apache.org/__**
>> msg00945/check.diff<http://www.mail-archive.com/__user@giraph.apache.org/__msg00945/check.diff>
>> <http://www.mail-archive.com/**user@giraph.apache.org/**
>> msg00945/check.diff<http://www.mail-archive.com/user@giraph.apache.org/msg00945/check.diff>
>> >>><<<<
>>
>>
>> On Mon, Oct 7, 2013 at 8:40
>> AM, Silvio
>> Di gregorio
>> <silvio.digregorio@gmail.com <mailto:silvio.digregorio@**
>> gmail.com <si...@gmail.com>>
>> <mailto:silvio.digregorio@__gm**ail.com <http://gmail.com>
>> <ma...@gmail.com>
>> >>
>> <mailto:silvio.digregorio@
>> <mailto:silvio.digregorio@>__g**m__ail.com <http://gm__ail.com> <
>> http://gmail.com>
>> <mailto:silvio.digregorio@__gm**ail.com <http://gmail.com>
>> <ma...@gmail.com>
>> >>>
>> <mailto:silvio.digregorio@ <mailto:silvio.digregorio@>
>> <mailto:silvio.digregorio@
>> <mailto:silvio.digregorio@>>__**g__m__ail.com<http://g__m__ail.com><
>> http://gm__ail.com>
>> <http://gmail.com>
>>
>>
>> <mailto:silvio.digregorio@
>> <mailto:silvio.digregorio@>__g**m__ail.com <http://gm__ail.com> <
>> http://gmail.com>
>> <mailto:silvio.digregorio@__gm**ail.com <http://gmail.com>
>> <ma...@gmail.com>>>>>>
>> wrote:
>>
>>
>>
>>
>>
>>
>>
>> *As i said i have builded*
>>
>>
>>
>>
>>
>>
>>
>> /giraph-examples-1.0.0-for-___**___hadoop-2.0.0-cdh4.1.2-jar-_**
>> _____with-dependencies.jar/
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> *for cdh4, successfully.
>> The job
>> start to
>> monitoring the success rate:*
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> /13/10/07 08:28:45 INFO
>> mapred.JobClient:
>> map 0% reduce 0%/
>>
>>
>>
>>
>> *but then*
>>
>>
>>
>>
>> /Error running child
>>
>> java.lang.______**IllegalStateException: run:
>>
>>
>> Caught an unrecoverable exception
>> java.io.FileNotFoundException:
>> File
>>
>> _bsp/_defaultZkManagerDir/job_**______201309181636_0678/___**
>> zkServer
>>
>>
>> does not exist.
>> /
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> /.../
>>
>>
>>
>>
>>
>>
>>
>> /Caused by:
>> java.lang.RuntimeException:
>> java.io.FileNotFoundException: File
>>
>>
>> _bsp/_defaultZkManagerDir/job_**______201309181636_0678/___**zkServer
>> does
>>
>>
>> not exist./
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> 2013/10/5 Silvio Di
>> gregorio
>> <silvio.digregorio@gmail.com <mailto:silvio.digregorio@**
>> gmail.com <si...@gmail.com>>
>> <mailto:silvio.digregorio@__gm**ail.com <http://gmail.com>
>> <ma...@gmail.com>
>> >>
>> <mailto:silvio.digregorio@
>> <mailto:silvio.digregorio@>__g**m__ail.com <http://gm__ail.com> <
>> http://gmail.com>
>> <mailto:silvio.digregorio@__gm**ail.com <http://gmail.com>
>> <ma...@gmail.com>
>> >>>
>> <mailto:silvio.digregorio@ <mailto:silvio.digregorio@>
>> <mailto:silvio.digregorio@
>> <mailto:silvio.digregorio@>>__**g__m__ail.com<http://g__m__ail.com><
>> http://gm__ail.com>
>> <http://gmail.com>
>>
>>
>> <mailto:silvio.digregorio@
>> <mailto:silvio.digregorio@>__g**m__ail.com <http://gm__ail.com> <
>> http://gmail.com>
>> <mailto:silvio.digregorio@__gm**ail.com <http://gmail.com>
>> <ma...@gmail.com>
>> >>>>>
>>
>> I ha ve build w/
>> hadoop_cdh4.1.2 parameter.
>> Something is changed,
>> monday i
>> report the
>> result. Now the farm
>> is closed.
>>
>> Il giorno 05/ott/2013
>> 14:06,
>> "Claudio
>> Martella"
>> <claudio.martella@gmail.com <ma...@gmail.com>
>> >
>> <mailto:claudio.martella@__gma**il.com <http://gmail.com>
>> <mailto:claudio.martella@**gmail.com <cl...@gmail.com>
>> >>
>> <mailto:claudio.martella@
>> <mailto:claudio.martella@>__gm**a__il.com <http://gma__il.com> <
>> http://gmail.com>
>> <mailto:claudio.martella@__gma**il.com <http://gmail.com>
>> <mailto:claudio.martella@**gmail.com <cl...@gmail.com>
>> >>>
>> <mailto:claudio.martella@ <mailto:claudio.martella@>
>> <mailto:claudio.martella@
>> <mailto:claudio.martella@>>__g**m__a__il.com<http://gm__a__il.com><
>> http://gma__il.com>
>> <http://gmail.com>
>>
>>
>> <mailto:claudio.martella@
>> <mailto:claudio.martella@>__gm**a__il.com <http://gma__il.com> <
>> http://gmail.com>
>> <mailto:claudio.martella@__gma**il.com <http://gmail.com>
>> <mailto:claudio.martella@**gmail.com <cl...@gmail.com>>>>>>
>> ha scritto:
>>
>> Oh, right, -vof is
>> in
>> trunk. Anyway
>> it looks
>> like you built
>> giraph for
>> the wrong
>> profile.
>> You mentioned
>> you're
>> running on
>> 2.0, but
>> your giraph is
>> built for
>> 0.20.203 <tel:0.20.203> <tel:0.20.203 <tel:0.20.203>>
>> <tel:0.20.203 <tel:0.20.203> <tel:0.20.203 <tel:0.20.203>>>
>> <tel:0.20.203 <tel:0.20.203> <tel:0.20.203 <tel:0.20.203>>
>> <tel:0.20.203 <tel:0.20.203> <tel:0.20.203 <tel:0.20.203>>>>. try
>>
>> building with a profile
>>
>> for your hadoop
>> version.
>>
>>
>> On Fri, Oct 4,
>> 2013 at
>> 2:35 PM,
>> Silvio Di
>> gregorio
>> <silvio.digregorio@gmail.com <mailto:silvio.digregorio@**
>> gmail.com <si...@gmail.com>>
>> <mailto:silvio.digregorio@__gm**ail.com <http://gmail.com>
>> <ma...@gmail.com>
>> >>
>> <mailto:silvio.digregorio@
>> <mailto:silvio.digregorio@>__g**m__ail.com <http://gm__ail.com> <
>> http://gmail.com>
>> <mailto:silvio.digregorio@__gm**ail.com <http://gmail.com>
>> <ma...@gmail.com>
>> >>>
>> <mailto:silvio.digregorio@ <mailto:silvio.digregorio@>
>> <mailto:silvio.digregorio@
>> <mailto:silvio.digregorio@>>__**g__m__ail.com<http://g__m__ail.com><
>> http://gm__ail.com>
>> <http://gmail.com>
>>
>> <mailto:silvio.digregorio@
>> <mailto:silvio.digregorio@>__g**m__ail.com <http://gm__ail.com> <
>> http://gmail.com>
>> <mailto:silvio.digregorio@__gm**ail.com <http://gmail.com>
>> <ma...@gmail.com>>>>>>
>> wrote:
>>
>>
>>
>>
>> org.apache.commons.cli.______**UnrecognizedOptionException:
>>
>>
>> Unrecognized
>> option: -vof
>>
>> in 1.0 version
>> is
>>
>> -of,--outputFormat <arg>
>> Vertex output
>> format
>>
>> -op,--outputPath <arg>
>> Vertex output
>> path
>>
>>
>>
>> 2013/10/4
>> Claudio Martella
>> <claudio.martella@gmail.com <ma...@gmail.com>
>> >
>> <mailto:claudio.martella@__gma**il.com <http://gmail.com>
>> <mailto:claudio.martella@**gmail.com <cl...@gmail.com>
>> >>
>> <mailto:claudio.martella@
>> <mailto:claudio.martella@>__gm**a__il.com <http://gma__il.com> <
>> http://gmail.com>
>> <mailto:claudio.martella@__gma**il.com <http://gmail.com>
>> <mailto:claudio.martella@**gmail.com <cl...@gmail.com>
>> >>>
>> <mailto:claudio.martella@ <mailto:claudio.martella@>
>> <mailto:claudio.martella@
>> <mailto:claudio.martella@>>__g**m__a__il.com<http://gm__a__il.com><
>> http://gma__il.com>
>> <http://gmail.com>
>>
>>
>> <mailto:claudio.martella@
>> <mailto:claudio.martella@>__gm**a__il.com <http://gma__il.com> <
>> http://gmail.com>
>> <mailto:claudio.martella@__gma**il.com <http://gmail.com>
>> <mailto:claudio.martella@**gmail.com <cl...@gmail.com>
>> >>>>>
>>
>> did you
>> try the
>> argument
>> (-vof) i
>> suggested?
>>
>>
>> On Fri,
>> Oct 4,
>> 2013 at 2:13 PM,
>> Silvio Di
>> gregorio
>> <silvio.digregorio@gmail.com <mailto:silvio.digregorio@**
>> gmail.com <si...@gmail.com>>
>> <mailto:silvio.digregorio@__gm**ail.com <http://gmail.com>
>> <ma...@gmail.com>
>> >>
>> <mailto:silvio.digregorio@
>> <mailto:silvio.digregorio@>__g**m__ail.com <http://gm__ail.com> <
>> http://gmail.com>
>> <mailto:silvio.digregorio@__gm**ail.com <http://gmail.com>
>> <ma...@gmail.com>
>> >>>
>> <mailto:silvio.digregorio@ <mailto:silvio.digregorio@>
>> <mailto:silvio.digregorio@
>> <mailto:silvio.digregorio@>>__**g__m__ail.com<http://g__m__ail.com><
>> http://gm__ail.com>
>> <http://gmail.com>
>>
>>
>> <mailto:silvio.digregorio@
>> <mailto:silvio.digregorio@>__g**m__ail.com <http://gm__ail.com> <
>> http://gmail.com>
>> <mailto:silvio.digregorio@__gm**ail.com <http://gmail.com>
>> <ma...@gmail.com>>>>>>
>> wrote:
>>
>>
>> i've
>> specified
>> -of
>>
>>
>>
>> org.apache.giraph.io.formats._**_____**
>> IdWithValueTextOutputFormat
>>
>>
>>
>> but
>> the same
>> error was
>> produced
>>
>>
>> Exception in
>> thread "main"
>>
>> java.lang.______**IncompatibleClassChangeError:
>> Found
>> interface
>>
>> org.apache.hadoop.mapreduce.__**____JobContext,
>>
>>
>> but
>> class was
>> expected
>>
>> at
>>
>>
>>
>> org.apache.giraph.bsp.______**BspOutputFormat.______**
>> checkOutputSpecs(______**BspOutputFormat.java:43)
>>
>> at
>>
>>
>>
>> org.apache.hadoop.mapred._____**_JobClient$2.run(JobClient.___**
>> ___java:984)
>>
>> at
>>
>>
>>
>> org.apache.hadoop.mapred._____**_JobClient$2.run(JobClient.___**
>> ___java:945)
>>
>> at
>>
>>
>> java.security.______**AccessController.doPrivileged(**
>> ______Native
>> Method)
>>
>> at
>>
>>
>> javax.security.auth.Subject.__**____doAs(Subject.java:396)
>>
>> at
>>
>>
>>
>> org.apache.hadoop.security.___**___UserGroupInformation.doAs(_**
>> _____UserGroupInformation.**java:______1408)
>>
>> at
>>
>>
>>
>> org.apache.hadoop.mapred._____**_JobClient.submitJobInternal(_**
>> _____JobClient.java:945)
>>
>> at
>>
>>
>> org.apache.hadoop.mapreduce.__**____Job.submit(Job.java:566)
>>
>> at
>>
>>
>>
>> org.apache.hadoop.mapreduce.__**____Job.waitForCompletion(Job.**
>> ______java:596)
>>
>> at
>>
>>
>>
>> org.apache.giraph.job.______**GiraphJob.run(GiraphJob.java:_**
>> _____237)
>>
>> at
>>
>>
>>
>> org.apache.giraph.______**GiraphRunner.run(GiraphRunner.**
>> ______java:94)
>>
>> at
>>
>>
>>
>> org.apache.hadoop.util.______**ToolRunner.run(ToolRunner.____**
>> __java:70)
>>
>> at
>>
>>
>>
>> org.apache.hadoop.util.______**ToolRunner.run(ToolRunner.____**
>> __java:84)
>>
>> at
>>
>>
>>
>> org.apache.giraph.______**GiraphRunner.main(______**
>> GiraphRunner.java:124)
>>
>> at
>>
>>
>> sun.reflect.______**NativeMethodAccessorImpl._____**
>> _invoke0(Native
>> Method)
>>
>> at
>>
>>
>>
>> sun.reflect.______**NativeMethodAccessorImpl._____**
>> _invoke(______**NativeMethodAccessorImpl.java:**______39)
>>
>> at
>>
>>
>>
>> sun.reflect.______**DelegatingMethodAccessorImpl._**
>> _____invoke(______**DelegatingMethodAccessorImpl._**_____java:25)
>>
>> at
>>
>>
>> java.lang.reflect.Method._____**_invoke(Method.java:597)
>>
>> at
>>
>>
>> org.apache.hadoop.util.RunJar.**______main(RunJar.java:208)
>>
>>
>>
>>
>>
>>
>> 2013/10/4
>> Claudio Martella
>> <claudio.martella@gmail.com <ma...@gmail.com>
>> >
>> <mailto:claudio.martella@__gma**il.com <http://gmail.com>
>> <mailto:claudio.martella@**gmail.com <cl...@gmail.com>
>> >>
>> <mailto:claudio.martella@
>> <mailto:claudio.martella@>__gm**a__il.com <http://gma__il.com> <
>> http://gmail.com>
>> <mailto:claudio.martella@__gma**il.com <http://gmail.com>
>> <mailto:claudio.martella@**gmail.com <cl...@gmail.com>
>> >>>
>> <mailto:claudio.martella@ <mailto:claudio.martella@>
>> <mailto:claudio.martella@
>> <mailto:claudio.martella@>>__g**m__a__il.com<http://gm__a__il.com><
>> http://gma__il.com>
>> <http://gmail.com>
>>
>>
>> <mailto:claudio.martella@
>> <mailto:claudio.martella@>__gm**a__il.com <http://gma__il.com> <
>> http://gmail.com>
>> <mailto:claudio.martella@__gma**il.com <http://gmail.com>
>> <mailto:claudio.martella@**gmail.com <cl...@gmail.com>
>> >>>>>
>>
>> Hi,
>>
>>
>> you need
>> to specify the
>>
>> vertex
>> outputformat
>> class
>>
>> (-vof
>> option), e.g.
>>
>>
>>
>> org.apache.giraph.io.formats._**_______**
>> IdWithValueTextOutputFormat.
>>
>>
>>
>>
>>
>> On Fri,
>> Oct 4, 2013
>> at 1:06
>>
>> PM, Silvio
>> Di gregorio
>> <silvio.digregorio@gmail.com <mailto:silvio.digregorio@**
>> gmail.com <si...@gmail.com>>
>> <mailto:silvio.digregorio@__gm**ail.com <http://gmail.com>
>> <ma...@gmail.com>
>> >>
>> <mailto:silvio.digregorio@
>> <mailto:silvio.digregorio@>__g**m__ail.com <http://gm__ail.com> <
>> http://gmail.com>
>> <mailto:silvio.digregorio@__gm**ail.com <http://gmail.com>
>> <ma...@gmail.com>
>> >>>
>> <mailto:silvio.digregorio@ <mailto:silvio.digregorio@>
>> <mailto:silvio.digregorio@
>> <mailto:silvio.digregorio@>>__**g__m__ail.com<http://g__m__ail.com><
>> http://gm__ail.com>
>> <http://gmail.com>
>>
>>
>> <mailto:silvio.digregorio@
>> <mailto:silvio.digregorio@>__g**m__ail.com <http://gm__ail.com> <
>> http://gmail.com>
>> <mailto:silvio.digregorio@__gm**ail.com <http://gmail.com>
>> <ma...@gmail.com>
>> >>>>>
>>
>> wrote:
>>
>>
>>
>> Hi,
>>
>>
>> I hope
>> I have
>> sent to
>>
>> the
>> right address.
>>
>>
>> i have
>> a graph
>> (directed
>>
>> and
>> unweighted)
>> stored
>>
>> in
>> hdfs like a
>> adjacency
>>
>> list
>> (140Milions of
>>
>> edges
>> 6Milions
>> of vertex)
>>
>>
>> node<tab>neighbors
>>
>>
>> 23 2
>> 1 3
>> 4 3
>>
>> 1 999 99923 <tel:1%20999%2099923> <tel:1%20999%2099923>
>> <tel:1%20999%2099923>
>> <tel:1%20999%2099923>
>>
>> 909
>> .......
>>
>>
>> ......
>>
>>
>> hadoop
>> version
>> Hadoop
>>
>> 2.0.0-cdh4.3.0
>> - java 1.6
>>
>>
>>
>> I have
>> executed the
>>
>> giraph-1.0
>> connected
>>
>> components
>> example, in
>>
>> this
>> fashion
>>
>> > hadoop jar
>>
>>
>>
>> /usr/local/giraph/giraph-_____**_examples/target/giraph-______**
>> examples-1.0.0-for-hadoop-0.20.203.0
>> <tel:0.20.203.0>
>>
>> <tel:0.20.203.0 <tel:0.20.203.0>>
>> <tel:0.20.203.0 <tel:0.20.203.0> <tel:0.20.203.0 <tel:0.20.203.0
>> >>>
>> <tel:0.20.203.0 <tel:0.20.203.0> <tel:0.20.203.0
>> <tel:0.20.203.0>> <tel:0.20.203.0 <tel:0.20.203.0>
>> <tel:0.20.203.0 <tel:0.20.203.0>>>>-jar-with-__**
>> dependencies.____jar
>>
>> org.apache.giraph.GiraphRunner
>>
>>
>> org.apache.giraph.examples.___**___ConnectedComponentsVertex
>>
>> -vif
>>
>>
>> org.apache.giraph.io.formats._**_____IntIntNullTextInputFormat
>>
>>
>>
>> -vip
>>
>> /user/hdfs/lista_adj_txt
>>
>> -op
>> connectedgiraph
>>
>> --workers 4
>>
>>
>> and
>> then fail with:
>>
>>
>> 13/10/04
>> 09:28:29 INFO
>>
>> utils.ConfigurationUtils: No
>>
>> edge
>> input format
>>
>> specified.
>> Ensure your
>>
>> InputFormat
>> does not
>>
>> require one.
>>
>>
>> 13/10/04
>> 09:28:29 INFO
>>
>> utils.ConfigurationUtils: No
>>
>> output
>> format
>> specified.
>>
>> Ensure
>> your
>> OutputFormat
>>
>> does not
>> require one.
>>
>>
>> 13/10/04
>> 09:28:30 INFO
>>
>> job.GiraphJob: run:
>>
>> Since
>> checkpointing is
>>
>> disabled
>> (default), do
>>
>> not
>> allow any task
>>
>> retries (setting
>>
>> mapred.map.max.attempts
>>
>> = 0,
>> old value = 4)
>>
>>
>> 13/10/04
>> 09:28:31 WARN
>>
>> mapred.JobClient: Use
>>
>> GenericOptionsParser for
>>
>> parsing the
>> arguments.
>>
>> Applications should
>>
>> implement Tool
>> for the same.
>>
>>
>> 13/10/04
>> 09:28:31 INFO
>>
>> mapred.JobClient:
>>
>> Cleaning up the
>> staging
>>
>> area
>>
>> hdfs://srv-bigdata-dev-01.int
>> <http://srv-bigdata-dev-01.int**>
>> <http://srv-bigdata-dev-01.**int__<http://srv-bigdata-dev-01.int__>
>> >.____sose.it:8020/user/**hdfs/.______staging/job_**201309181636___0535<http://sose.it:8020/user/hdfs/.______staging/job_201309181636___0535>
>> <http://sose.it:8020/user/**hdfs/.____staging/job_**
>> 201309181636_0535<http://sose.it:8020/user/hdfs/.____staging/job_201309181636_0535>
>> >
>> <http://sose.it:8020/user/__**hdfs/.__staging/job___**
>> 201309181636_0535<http://sose.it:8020/user/__hdfs/.__staging/job___201309181636_0535>
>> <http://sose.it:8020/user/**hdfs/.__staging/job_**
>> 201309181636_0535<http://sose.it:8020/user/hdfs/.__staging/job_201309181636_0535>
>> >>
>>
>> <http://srv-bigdata-dev-01.__i**__nt.sose.it:8020/user/hdfs/._**
>> ___staging/job_201309181636_**0535<http://i__nt.sose.it:8020/user/hdfs/.____staging/job_201309181636_0535>
>> <http://int.sose.it:8020/user/**hdfs/.__staging/job_**
>> 201309181636_0535<http://int.sose.it:8020/user/hdfs/.__staging/job_201309181636_0535>
>> >
>> <http://srv-bigdata-dev-01.__i**nt.sose.it:8020/user/hdfs/.__**
>> staging/job_201309181636_0535<http://int.sose.it:8020/user/hdfs/.__staging/job_201309181636_0535>
>> <http://srv-bigdata-dev-01.**int.sose.it:8020/user/hdfs/.**
>> staging/job_201309181636_0535<http://srv-bigdata-dev-01.int.sose.it:8020/user/hdfs/.staging/job_201309181636_0535>
>> >**>__>
>> <http://srv-bigdata-dev-01.__i**____nt.sose.it:8020/user/hdfs/**
>> .______staging/job_**201309181636___0535<http://i____nt.sose.it:8020/user/hdfs/.______staging/job_201309181636___0535>
>> <http://i__nt.sose.it:8020/**user/hdfs/.____staging/job_**
>> 201309181636_0535<http://i__nt.sose.it:8020/user/hdfs/.____staging/job_201309181636_0535>
>> >
>> <http://int.sose.it:8020/user/**__hdfs/.__staging/job___**
>> 201309181636_0535<http://int.sose.it:8020/user/__hdfs/.__staging/job___201309181636_0535>
>> <http://int.sose.it:8020/user/**hdfs/.__staging/job_**
>> 201309181636_0535<http://int.sose.it:8020/user/hdfs/.__staging/job_201309181636_0535>
>> >>
>>
>> <http://srv-bigdata-dev-01.__i**__nt.sose.it:8020/user/hdfs/._**
>> ___staging/job_201309181636_**0535<http://i__nt.sose.it:8020/user/hdfs/.____staging/job_201309181636_0535>
>> <http://int.sose.it:8020/user/**hdfs/.__staging/job_**
>> 201309181636_0535<http://int.sose.it:8020/user/hdfs/.__staging/job_201309181636_0535>
>> >
>> <http://srv-bigdata-dev-01.__i**nt.sose.it:8020/user/hdfs/.__**
>> staging/job_201309181636_0535<http://int.sose.it:8020/user/hdfs/.__staging/job_201309181636_0535>
>> <http://srv-bigdata-dev-01.**int.sose.it:8020/user/hdfs/.**
>> staging/job_201309181636_0535<http://srv-bigdata-dev-01.int.sose.it:8020/user/hdfs/.staging/job_201309181636_0535>
>> >**>__>__>
>>
>>
>> Exception in thread
>> "main"
>>
>> java.lang.______**IncompatibleClassChangeError:
>>
>> Found
>> interface
>>
>> org.apache.hadoop.mapreduce.__**____JobContext,
>>
>>
>>
>> but
>> class was
>> expected
>>
>>
>> at
>>
>>
>>
>> org.apache.giraph.bsp.______**BspOutputFormat.______**
>> checkOutputSpecs(______**BspOutputFormat.java:43)
>>
>>
>>
>>
>> ......
>>
>>
>> Thanks
>> in advance
>>
>>
>>
>>
>>
>> --
>>
>> Claudio Martella
>> claudio.martella@gmail.com <ma...@gmail.com>
>> >
>> <mailto:claudio.martella@__gma**il.com <http://gmail.com>
>> <mailto:claudio.martella@**gmail.com <cl...@gmail.com>
>> >>
>> <mailto:claudio.martella@
>> <mailto:claudio.martella@>__gm**a__il.com <http://gma__il.com> <
>> http://gmail.com>
>> <mailto:claudio.martella@__gma**il.com <http://gmail.com>
>> <mailto:claudio.martella@**gmail.com <cl...@gmail.com>
>> >>>
>> <mailto:claudio.martella@ <mailto:claudio.martella@>
>> <mailto:claudio.martella@
>> <mailto:claudio.martella@>>__g**m__a__il.com<http://gm__a__il.com><
>> http://gma__il.com>
>> <http://gmail.com>
>>
>>
>> <mailto:claudio.martella@
>> <mailto:claudio.martella@>__gm**a__il.com <http://gma__il.com> <
>> http://gmail.com>
>> <mailto:claudio.martella@__gma**il.com <http://gmail.com>
>> <mailto:claudio.martella@**gmail.com <cl...@gmail.com>
>> >>>>
>>
>>
>>
>>
>>
>>
>> --
>>
>> Claudio Martella
>> claudio.martella@gmail.com <ma...@gmail.com>
>> >
>> <mailto:claudio.martella@__gma**il.com <http://gmail.com>
>> <mailto:claudio.martella@**gmail.com <cl...@gmail.com>
>> >>
>> <mailto:claudio.martella@
>> <mailto:claudio.martella@>__gm**a__il.com <http://gma__il.com> <
>> http://gmail.com>
>> <mailto:claudio.martella@__gma**il.com <http://gmail.com>
>> <mailto:claudio.martella@**gmail.com <cl...@gmail.com>
>> >>>
>> <mailto:claudio.martella@ <mailto:claudio.martella@>
>> <mailto:claudio.martella@
>> <mailto:claudio.martella@>>__g**m__a__il.com<http://gm__a__il.com><
>> http://gma__il.com>
>> <http://gmail.com>
>>
>>
>> <mailto:claudio.martella@
>> <mailto:claudio.martella@>__gm**a__il.com <http://gma__il.com> <
>> http://gmail.com>
>> <mailto:claudio.martella@__gma**il.com <http://gmail.com>
>> <mailto:claudio.martella@**gmail.com <cl...@gmail.com>
>> >>>>
>>
>>
>>
>>
>>
>> --
>> Claudio
>> Martella
>> claudio.martella@gmail.com <ma...@gmail.com>
>> >
>> <mailto:claudio.martella@__gma**il.com <http://gmail.com>
>> <mailto:claudio.martella@**gmail.com <cl...@gmail.com>
>> >>
>> <mailto:claudio.martella@
>> <mailto:claudio.martella@>__gm**a__il.com <http://gma__il.com> <
>> http://gmail.com>
>> <mailto:claudio.martella@__gma**il.com <http://gmail.com>
>> <mailto:claudio.martella@**gmail.com <cl...@gmail.com>
>> >>>
>> <mailto:claudio.martella@ <mailto:claudio.martella@>
>> <mailto:claudio.martella@
>> <mailto:claudio.martella@>>__g**m__a__il.com<http://gm__a__il.com><
>> http://gma__il.com>
>> <http://gmail.com>
>>
>>
>> <mailto:claudio.martella@
>> <mailto:claudio.martella@>__gm**a__il.com <http://gma__il.com> <
>> http://gmail.com>
>> <mailto:claudio.martella@__gma**il.com <http://gmail.com>
>> <mailto:claudio.martella@**gmail.com <cl...@gmail.com>
>> >>>>
>>
>>
>>
>>
>>
>> --
>> Claudio Martella
>> claudio.martella@gmail.com <ma...@gmail.com>
>> >
>> <mailto:claudio.martella@__gma**il.com <http://gmail.com>
>> <mailto:claudio.martella@**gmail.com <cl...@gmail.com>
>> >>
>> <mailto:claudio.martella@
>> <mailto:claudio.martella@>__gm**a__il.com <http://gma__il.com> <
>> http://gmail.com>
>> <mailto:claudio.martella@__gma**il.com <http://gmail.com>
>> <mailto:claudio.martella@**gmail.com <cl...@gmail.com>
>> >>>
>> <mailto:claudio.martella@ <mailto:claudio.martella@>
>> <mailto:claudio.martella@
>> <mailto:claudio.martella@>>__g**m__a__il.com<http://gm__a__il.com><
>> http://gma__il.com>
>> <http://gmail.com>
>>
>>
>> <mailto:claudio.martella@
>> <mailto:claudio.martella@>__gm**a__il.com <http://gma__il.com> <
>> http://gmail.com>
>> <mailto:claudio.martella@__gma**il.com <http://gmail.com>
>> <mailto:claudio.martella@**gmail.com <cl...@gmail.com>
>> >>>>
>>
>>
>>
>>
>>
>> --
>> Claudio Martella
>> claudio.martella@gmail.com <ma...@gmail.com>
>> >
>> <mailto:claudio.martella@__gma**il.com <http://gmail.com>
>> <mailto:claudio.martella@**gmail.com <cl...@gmail.com>
>> >>
>> <mailto:claudio.martella@
>> <mailto:claudio.martella@>__gm**a__il.com <http://gma__il.com> <
>> http://gmail.com>
>> <mailto:claudio.martella@__gma**il.com <http://gmail.com>
>> <mailto:claudio.martella@**gmail.com <cl...@gmail.com>
>> >>>
>> <mailto:claudio.martella@ <mailto:claudio.martella@>
>> <mailto:claudio.martella@
>> <mailto:claudio.martella@>>__g**m__a__il.com<http://gm__a__il.com><
>> http://gma__il.com>
>> <http://gmail.com>
>>
>>
>> <mailto:claudio.martella@
>> <mailto:claudio.martella@>__gm**a__il.com <http://gma__il.com> <
>> http://gmail.com>
>> <mailto:claudio.martella@__gma**il.com <http://gmail.com>
>> <mailto:claudio.martella@**gmail.com <cl...@gmail.com>
>> >>>>
>>
>>
>>
>>
>>
>> --
>> Claudio Martella
>> claudio.martella@gmail.com <ma...@gmail.com>
>> >
>> <mailto:claudio.martella@__gma**il.com <http://gmail.com>
>> <mailto:claudio.martella@**gmail.com <cl...@gmail.com>
>> >>
>> <mailto:claudio.martella@
>> <mailto:claudio.martella@>__gm**a__il.com <http://gma__il.com> <
>> http://gmail.com>
>> <mailto:claudio.martella@__gma**il.com <http://gmail.com>
>> <mailto:claudio.martella@**gmail.com <cl...@gmail.com>
>> >>>
>> <mailto:claudio.martella@ <mailto:claudio.martella@>
>> <mailto:claudio.martella@
>> <mailto:claudio.martella@>>__g**m__a__il.com<http://gm__a__il.com><
>> http://gma__il.com>
>> <http://gmail.com>
>>
>>
>> <mailto:claudio.martella@
>> <mailto:claudio.martella@>__gm**a__il.com <http://gma__il.com> <
>> http://gmail.com>
>> <mailto:claudio.martella@__gma**il.com <http://gmail.com>
>> <mailto:claudio.martella@**gmail.com <cl...@gmail.com>
>> >>>>
>>
>>
>> --
>> Matthew Laird
>> Lead Software Developer, Bioinformatics
>> Brinkman Laboratory
>> Simon Fraser University, Burnaby, BC, Canada
>
>
Re: connected components example in giraph 1.0
Posted by Matthew Laird <la...@sfu.ca>.
Thanks, I appreciate the feedback! It's why open source projects are great.
So just to be clear, I should grab the 1.0 release, and you've
successfully built it against a Hadoop 2.0.x release? I add that patch
from Claudio first, then use -Dhadoop.version=2.0.5 to build? It will
still have the MR API in place?
I'll get through that then see what happens with this whole split
master-worker setup I have on my Hadoop installation which caused the
examples to fail. Then on to trying to load my billion vertices and
clustering the graph (I basically want all the sub-groups of connected
components, these will be the related genes across all sequenced bacteria).
Thanks again!
On 13-10-07 12:56 PM, Silvio Di gregorio wrote:
> with a working day in my shoulders and my children around i gave you
> random version,
> tomorrow morning in farm i'll be more precise.
> though i have hadoop 2.0 i use MR1 not YARN.
> I'm sorry again.
>
>
> 2013/10/7 Matthew Laird <lairdm@sfu.ca <ma...@sfu.ca>>
>
> Hmm, it seems there have been some recent pushes that have broken
> trunk as I discovered yesterday. :)
>
> But I guess that brings me back to my original question, to build
> against 2.0.5 using the MR interface rather than YARN as Milinda
> suggested, what profile would I use? Would I let it think it's
> 0.20xxx or tell it 2.0.5, or something else?
>
> Thanks.
>
>
> On 13-10-07 11:50 AM, Claudio Martella wrote:
>
> trunk is 1.1.0-SNAPSHOT. I believe the date refers to the last
> update to
> the site.
>
>
> On Mon, Oct 7, 2013 at 8:48 PM, Matthew Laird <lairdm@sfu.ca
> <ma...@sfu.ca>
> <mailto:lairdm@sfu.ca <ma...@sfu.ca>>> wrote:
>
> Thanks, which is my next curiosity... if I head over to
> http://giraph.apache.org/ I see at the top is says
>
> "Version: 1.1.0-SNAPSHOT | Last Published: 2013-08-10"
>
> However it's not clickable and the only download links I
> see are for
> 1.0.0. I'm assuming trunk in git isn't the same as the 1.1.0
> snapshot since more pushes have occurred since August, and
> I don't
> see a tag in git for 1.1.0
> (https://git-wip-us.apache.____org/repos/asf?p=giraph.git
> <https://git-wip-us.apache.__org/repos/asf?p=giraph.git
> <https://git-wip-us.apache.org/repos/asf?p=giraph.git>>). So where
>
> does one get 1.1.0-SNAPSHOT from?
>
> Thanks.
>
>
> On 13-10-07 11:39 AM, Silvio Di gregorio wrote:
>
> Hi Matthew
> I use giraph 1.1.0 and cdh4 hadoop 0.23
>
> the patch of Marco:
>
> http://www.mail-archive.com/______user@giraph.apache.org/______msg00945/check.diff
> <http://www.mail-archive.com/____user@giraph.apache.org/____msg00945/check.diff>
> <http://www.mail-archive.com/____user@giraph.apache.org/____msg00945/check.diff
> <http://www.mail-archive.com/__user@giraph.apache.org/__msg00945/check.diff>>
>
>
> <http://www.mail-archive.com/____user@giraph.apache.org/____msg00945/check.diff
> <http://www.mail-archive.com/__user@giraph.apache.org/__msg00945/check.diff>
> <http://www.mail-archive.com/__user@giraph.apache.org/__msg00945/check.diff
> <http://www.mail-archive.com/user@giraph.apache.org/msg00945/check.diff>>>
>
> and build
>
> mvn clean install -P hadoop_cdh4.1.2
>
>
>
> Il giorno 07/ott/2013 20:20, "Matthew Laird"
> <lairdm@sfu.ca <ma...@sfu.ca>
> <mailto:lairdm@sfu.ca <ma...@sfu.ca>>
> <mailto:lairdm@sfu.ca <ma...@sfu.ca>
> <mailto:lairdm@sfu.ca <ma...@sfu.ca>>>> ha scritto:
>
>
>
> Hello, so I'm watching this thread with interest
> as well, since
> that's the exact example I'll be trying to run as
> well, and
> I see
> Silvio appears to be asking about running Giraph 1.0
> against Hadoop
> 2.0.0.
>
> I'm curious, is that a known working combination,
> 1.0 vs
> 2.0.x? And
> if so, are there patches to apply first and what maven
> command line
> should I build with? (ie. will a
> -Dhadoop.version=2.0.5
> give the
> needed MR based API or will it use YARN?) I
> figure it's
> better to
> ask than aimlessly try all the combinations. :)
>
> Thanks!
>
> On 13-10-07 11:01 AM, Claudio Martella wrote:
>
> try to increase the mappers' heap space through
> mapred.child.java.opts.
>
>
> On Mon, Oct 7, 2013 at 1:18 PM, Silvio Di gregorio
> <silvio.digregorio@gmail.com
> <ma...@gmail.com>
> <mailto:silvio.digregorio@__gmail.com
> <ma...@gmail.com>>
> <mailto:silvio.digregorio@
> <mailto:silvio.digregorio@>__gm__ail.com <http://gmail.com>
> <mailto:silvio.digregorio@__gmail.com
> <ma...@gmail.com>>>
> <mailto:silvio.digregorio@ <mailto:silvio.digregorio@>
> <mailto:silvio.digregorio@
> <mailto:silvio.digregorio@>>__g__m__ail.com <http://gm__ail.com>
> <http://gmail.com>
>
>
> <mailto:silvio.digregorio@
> <mailto:silvio.digregorio@>__gm__ail.com <http://gmail.com>
> <mailto:silvio.digregorio@__gmail.com
> <ma...@gmail.com>>>>> wrote:
>
>
> I think now my issue is another, thanks
> to all.
>
>
>
>
> java.lang.______IllegalStateException:
> run: Caught an
>
>
> unrecoverable exception waitFor:
> ExecutionException
> occurred
> while waiting for
>
>
> org.apache.giraph.utils.______ProgressableUtils$______FutureWaitable@73a01e8a
> at
>
>
> org.apache.giraph.graph.______GraphMapper.run(GraphMapper.______java:102)
>
>
>
> ......
> Caused by:
> java.lang.______IllegalStateException:
>
> waitFor:
>
> ExecutionException occurred while waiting for
>
>
> org.apache.giraph.utils.______ProgressableUtils$______FutureWaitable@73a01e8a
> at
>
>
> org.apache.giraph.utils.______ProgressableUtils.waitFor(______ProgressableUtils.java:151)
>
>
>
>
> ... 7 more
> Caused by:
> java.util.concurrent.______ExecutionException:
> java.lang.OutOfMemoryError:*______Java heap space*
> at
>
>
> org.apache.giraph.utils.______ProgressableUtils.waitFor(______ProgressableUtils.java:143)
>
>
>
>
>
> ... 13 more
>
>
>
> 2013/10/7 Claudio Martella
> <claudio.martella@gmail.com <ma...@gmail.com>
> <mailto:claudio.martella@__gmail.com
> <ma...@gmail.com>>
> <mailto:claudio.martella@
> <mailto:claudio.martella@>__gma__il.com <http://gmail.com>
> <mailto:claudio.martella@__gmail.com
> <ma...@gmail.com>>>
> <mailto:claudio.martella@ <mailto:claudio.martella@>
> <mailto:claudio.martella@
> <mailto:claudio.martella@>>__gm__a__il.com <http://gma__il.com>
> <http://gmail.com>
>
>
> <mailto:claudio.martella@
> <mailto:claudio.martella@>__gma__il.com <http://gmail.com>
> <mailto:claudio.martella@__gmail.com
> <ma...@gmail.com>>>>>
>
> OK, thanks.
>
> I really have to push that patch in.
>
>
> On Mon, Oct 7, 2013 at 12:17 PM,
> Silvio Di
> gregorio
> <silvio.digregorio@gmail.com <ma...@gmail.com>
> <mailto:silvio.digregorio@__gmail.com
> <ma...@gmail.com>>
> <mailto:silvio.digregorio@
> <mailto:silvio.digregorio@>__gm__ail.com <http://gmail.com>
> <mailto:silvio.digregorio@__gmail.com
> <ma...@gmail.com>>>
> <mailto:silvio.digregorio@ <mailto:silvio.digregorio@>
> <mailto:silvio.digregorio@
> <mailto:silvio.digregorio@>>__g__m__ail.com <http://gm__ail.com>
> <http://gmail.com>
>
>
> <mailto:silvio.digregorio@
> <mailto:silvio.digregorio@>__gm__ail.com <http://gmail.com>
> <mailto:silvio.digregorio@__gmail.com
> <ma...@gmail.com>>>>> wrote:
>
> yes i do,
> i have seen this in your post in:
> http://www.mail-archive.com/______user@giraph.apache.org/______msg00957.html
> <http://www.mail-archive.com/____user@giraph.apache.org/____msg00957.html>
> <http://www.mail-archive.com/____user@giraph.apache.org/____msg00957.html
> <http://www.mail-archive.com/__user@giraph.apache.org/__msg00957.html>>
>
>
> <http://www.mail-archive.com/____user@giraph.apache.org/____msg00957.html
> <http://www.mail-archive.com/__user@giraph.apache.org/__msg00957.html>
> <http://www.mail-archive.com/__user@giraph.apache.org/__msg00957.html
> <http://www.mail-archive.com/user@giraph.apache.org/msg00957.html>>>
>
> excuse me if i had checked in
> the mail-achive
> first I would
> have avoided the last post.
>
> Now zk issue are resolved.
>
>
> 2013/10/7 Claudio Martella
> <claudio.martella@gmail.com <ma...@gmail.com>
> <mailto:claudio.martella@__gmail.com
> <ma...@gmail.com>>
> <mailto:claudio.martella@
> <mailto:claudio.martella@>__gma__il.com <http://gmail.com>
> <mailto:claudio.martella@__gmail.com
> <ma...@gmail.com>>>
> <mailto:claudio.martella@ <mailto:claudio.martella@>
> <mailto:claudio.martella@
> <mailto:claudio.martella@>>__gm__a__il.com <http://gma__il.com>
> <http://gmail.com>
>
>
> <mailto:claudio.martella@
> <mailto:claudio.martella@>__gma__il.com <http://gmail.com>
> <mailto:claudio.martella@__gmail.com
> <ma...@gmail.com>>>>>
>
> Can you try applying this one
> first?
>
> <<< <
> http://www.mail-archive.com/______user@giraph.apache.org/______msg00945/check.diff
> <http://www.mail-archive.com/____user@giraph.apache.org/____msg00945/check.diff>
> <http://www.mail-archive.com/____user@giraph.apache.org/____msg00945/check.diff
> <http://www.mail-archive.com/__user@giraph.apache.org/__msg00945/check.diff>>
>
>
> <http://www.mail-archive.com/____user@giraph.apache.org/____msg00945/check.diff
> <http://www.mail-archive.com/__user@giraph.apache.org/__msg00945/check.diff>
> <http://www.mail-archive.com/__user@giraph.apache.org/__msg00945/check.diff
> <http://www.mail-archive.com/user@giraph.apache.org/msg00945/check.diff>>><<<<
>
>
> On Mon, Oct 7, 2013 at 8:40
> AM, Silvio
> Di gregorio
> <silvio.digregorio@gmail.com <ma...@gmail.com>
> <mailto:silvio.digregorio@__gmail.com
> <ma...@gmail.com>>
> <mailto:silvio.digregorio@
> <mailto:silvio.digregorio@>__gm__ail.com <http://gmail.com>
> <mailto:silvio.digregorio@__gmail.com
> <ma...@gmail.com>>>
> <mailto:silvio.digregorio@ <mailto:silvio.digregorio@>
> <mailto:silvio.digregorio@
> <mailto:silvio.digregorio@>>__g__m__ail.com <http://gm__ail.com>
> <http://gmail.com>
>
>
> <mailto:silvio.digregorio@
> <mailto:silvio.digregorio@>__gm__ail.com <http://gmail.com>
> <mailto:silvio.digregorio@__gmail.com
> <ma...@gmail.com>>>>> wrote:
>
>
>
>
>
>
>
> *As i said i have builded*
>
>
>
>
>
>
>
> /giraph-examples-1.0.0-for-______hadoop-2.0.0-cdh4.1.2-jar-______with-dependencies.jar/
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> *for cdh4, successfully.
> The job
> start to
> monitoring the success rate:*
>
>
>
>
>
>
>
>
>
>
>
>
>
> /13/10/07 08:28:45 INFO
> mapred.JobClient:
> map 0% reduce 0%/
>
>
>
>
> *but then*
>
>
>
>
> /Error running child
>
> java.lang.______IllegalStateException: run:
>
>
> Caught an unrecoverable exception
> java.io.FileNotFoundException:
> File
>
> _bsp/_defaultZkManagerDir/job_______201309181636_0678/___zkServer
>
>
> does not exist.
> /
>
>
>
>
>
>
>
>
>
>
>
>
>
> /.../
>
>
>
>
>
>
>
> /Caused by:
> java.lang.RuntimeException:
> java.io.FileNotFoundException: File
>
>
> _bsp/_defaultZkManagerDir/job_______201309181636_0678/___zkServer does
>
>
> not exist./
>
>
>
>
>
>
>
>
>
>
> 2013/10/5 Silvio Di gregorio
> <silvio.digregorio@gmail.com <ma...@gmail.com>
> <mailto:silvio.digregorio@__gmail.com
> <ma...@gmail.com>>
> <mailto:silvio.digregorio@
> <mailto:silvio.digregorio@>__gm__ail.com <http://gmail.com>
> <mailto:silvio.digregorio@__gmail.com
> <ma...@gmail.com>>>
> <mailto:silvio.digregorio@ <mailto:silvio.digregorio@>
> <mailto:silvio.digregorio@
> <mailto:silvio.digregorio@>>__g__m__ail.com <http://gm__ail.com>
> <http://gmail.com>
>
>
> <mailto:silvio.digregorio@
> <mailto:silvio.digregorio@>__gm__ail.com <http://gmail.com>
> <mailto:silvio.digregorio@__gmail.com
> <ma...@gmail.com>>>>>
>
> I ha ve build w/
> hadoop_cdh4.1.2 parameter.
> Something is changed,
> monday i
> report the
> result. Now the farm
> is closed.
>
> Il giorno 05/ott/2013
> 14:06,
> "Claudio
> Martella"
> <claudio.martella@gmail.com <ma...@gmail.com>
> <mailto:claudio.martella@__gmail.com
> <ma...@gmail.com>>
> <mailto:claudio.martella@
> <mailto:claudio.martella@>__gma__il.com <http://gmail.com>
> <mailto:claudio.martella@__gmail.com
> <ma...@gmail.com>>>
> <mailto:claudio.martella@ <mailto:claudio.martella@>
> <mailto:claudio.martella@
> <mailto:claudio.martella@>>__gm__a__il.com <http://gma__il.com>
> <http://gmail.com>
>
>
> <mailto:claudio.martella@
> <mailto:claudio.martella@>__gma__il.com <http://gmail.com>
> <mailto:claudio.martella@__gmail.com
> <ma...@gmail.com>>>>> ha scritto:
>
> Oh, right, -vof is in
> trunk. Anyway
> it looks
> like you built
> giraph for
> the wrong
> profile.
> You mentioned you're
> running on
> 2.0, but
> your giraph is
> built for
> 0.20.203 <tel:0.20.203> <tel:0.20.203 <tel:0.20.203>>
> <tel:0.20.203 <tel:0.20.203> <tel:0.20.203 <tel:0.20.203>>>
> <tel:0.20.203 <tel:0.20.203> <tel:0.20.203 <tel:0.20.203>>
> <tel:0.20.203 <tel:0.20.203> <tel:0.20.203 <tel:0.20.203>>>>. try
>
> building with a profile
>
> for your hadoop
> version.
>
>
> On Fri, Oct 4,
> 2013 at
> 2:35 PM,
> Silvio Di
> gregorio
> <silvio.digregorio@gmail.com <ma...@gmail.com>
> <mailto:silvio.digregorio@__gmail.com
> <ma...@gmail.com>>
> <mailto:silvio.digregorio@
> <mailto:silvio.digregorio@>__gm__ail.com <http://gmail.com>
> <mailto:silvio.digregorio@__gmail.com
> <ma...@gmail.com>>>
> <mailto:silvio.digregorio@ <mailto:silvio.digregorio@>
> <mailto:silvio.digregorio@
> <mailto:silvio.digregorio@>>__g__m__ail.com <http://gm__ail.com>
> <http://gmail.com>
>
> <mailto:silvio.digregorio@
> <mailto:silvio.digregorio@>__gm__ail.com <http://gmail.com>
> <mailto:silvio.digregorio@__gmail.com
> <ma...@gmail.com>>>>> wrote:
>
>
>
>
> org.apache.commons.cli.______UnrecognizedOptionException:
>
>
> Unrecognized
> option: -vof
>
> in 1.0 version is
>
> -of,--outputFormat <arg>
> Vertex output
> format
>
> -op,--outputPath <arg>
> Vertex output
> path
>
>
>
> 2013/10/4
> Claudio Martella
> <claudio.martella@gmail.com <ma...@gmail.com>
> <mailto:claudio.martella@__gmail.com
> <ma...@gmail.com>>
> <mailto:claudio.martella@
> <mailto:claudio.martella@>__gma__il.com <http://gmail.com>
> <mailto:claudio.martella@__gmail.com
> <ma...@gmail.com>>>
> <mailto:claudio.martella@ <mailto:claudio.martella@>
> <mailto:claudio.martella@
> <mailto:claudio.martella@>>__gm__a__il.com <http://gma__il.com>
> <http://gmail.com>
>
>
> <mailto:claudio.martella@
> <mailto:claudio.martella@>__gma__il.com <http://gmail.com>
> <mailto:claudio.martella@__gmail.com
> <ma...@gmail.com>>>>>
>
> did you
> try the
> argument
> (-vof) i
> suggested?
>
>
> On Fri,
> Oct 4,
> 2013 at 2:13 PM,
> Silvio Di
> gregorio
> <silvio.digregorio@gmail.com <ma...@gmail.com>
> <mailto:silvio.digregorio@__gmail.com
> <ma...@gmail.com>>
> <mailto:silvio.digregorio@
> <mailto:silvio.digregorio@>__gm__ail.com <http://gmail.com>
> <mailto:silvio.digregorio@__gmail.com
> <ma...@gmail.com>>>
> <mailto:silvio.digregorio@ <mailto:silvio.digregorio@>
> <mailto:silvio.digregorio@
> <mailto:silvio.digregorio@>>__g__m__ail.com <http://gm__ail.com>
> <http://gmail.com>
>
>
> <mailto:silvio.digregorio@
> <mailto:silvio.digregorio@>__gm__ail.com <http://gmail.com>
> <mailto:silvio.digregorio@__gmail.com
> <ma...@gmail.com>>>>> wrote:
>
>
> i've
> specified
> -of
>
>
>
> org.apache.giraph.io.formats.______IdWithValueTextOutputFormat
>
>
>
> but
> the same
> error was
> produced
>
>
> Exception in
> thread "main"
>
> java.lang.______IncompatibleClassChangeError:
> Found
> interface
>
> org.apache.hadoop.mapreduce.______JobContext,
>
>
> but
> class was
> expected
>
> at
>
>
>
> org.apache.giraph.bsp.______BspOutputFormat.______checkOutputSpecs(______BspOutputFormat.java:43)
>
> at
>
>
>
> org.apache.hadoop.mapred.______JobClient$2.run(JobClient.______java:984)
>
> at
>
>
>
> org.apache.hadoop.mapred.______JobClient$2.run(JobClient.______java:945)
>
> at
>
>
> java.security.______AccessController.doPrivileged(______Native
> Method)
>
> at
>
>
> javax.security.auth.Subject.______doAs(Subject.java:396)
>
> at
>
>
>
> org.apache.hadoop.security.______UserGroupInformation.doAs(______UserGroupInformation.java:______1408)
>
> at
>
>
>
> org.apache.hadoop.mapred.______JobClient.submitJobInternal(______JobClient.java:945)
>
> at
>
>
> org.apache.hadoop.mapreduce.______Job.submit(Job.java:566)
>
> at
>
>
>
> org.apache.hadoop.mapreduce.______Job.waitForCompletion(Job.______java:596)
>
> at
>
>
>
> org.apache.giraph.job.______GiraphJob.run(GiraphJob.java:______237)
>
> at
>
>
>
> org.apache.giraph.______GiraphRunner.run(GiraphRunner.______java:94)
>
> at
>
>
>
> org.apache.hadoop.util.______ToolRunner.run(ToolRunner.______java:70)
>
> at
>
>
>
> org.apache.hadoop.util.______ToolRunner.run(ToolRunner.______java:84)
>
> at
>
>
>
> org.apache.giraph.______GiraphRunner.main(______GiraphRunner.java:124)
>
> at
>
>
> sun.reflect.______NativeMethodAccessorImpl.______invoke0(Native
> Method)
>
> at
>
>
>
> sun.reflect.______NativeMethodAccessorImpl.______invoke(______NativeMethodAccessorImpl.java:______39)
>
> at
>
>
>
> sun.reflect.______DelegatingMethodAccessorImpl.______invoke(______DelegatingMethodAccessorImpl.______java:25)
>
> at
>
>
> java.lang.reflect.Method.______invoke(Method.java:597)
>
> at
>
>
> org.apache.hadoop.util.RunJar.______main(RunJar.java:208)
>
>
>
>
>
> 2013/10/4
> Claudio Martella
> <claudio.martella@gmail.com <ma...@gmail.com>
> <mailto:claudio.martella@__gmail.com
> <ma...@gmail.com>>
> <mailto:claudio.martella@
> <mailto:claudio.martella@>__gma__il.com <http://gmail.com>
> <mailto:claudio.martella@__gmail.com
> <ma...@gmail.com>>>
> <mailto:claudio.martella@ <mailto:claudio.martella@>
> <mailto:claudio.martella@
> <mailto:claudio.martella@>>__gm__a__il.com <http://gma__il.com>
> <http://gmail.com>
>
>
> <mailto:claudio.martella@
> <mailto:claudio.martella@>__gma__il.com <http://gmail.com>
> <mailto:claudio.martella@__gmail.com
> <ma...@gmail.com>>>>>
>
> Hi,
>
>
> you need
> to specify the
>
> vertex
> outputformat
> class
> (-vof
> option), e.g.
>
>
>
> org.apache.giraph.io.formats.________IdWithValueTextOutputFormat.
>
>
>
>
>
> On Fri,
> Oct 4, 2013
> at 1:06
>
> PM, Silvio
> Di gregorio
> <silvio.digregorio@gmail.com <ma...@gmail.com>
> <mailto:silvio.digregorio@__gmail.com
> <ma...@gmail.com>>
> <mailto:silvio.digregorio@
> <mailto:silvio.digregorio@>__gm__ail.com <http://gmail.com>
> <mailto:silvio.digregorio@__gmail.com
> <ma...@gmail.com>>>
> <mailto:silvio.digregorio@ <mailto:silvio.digregorio@>
> <mailto:silvio.digregorio@
> <mailto:silvio.digregorio@>>__g__m__ail.com <http://gm__ail.com>
> <http://gmail.com>
>
>
> <mailto:silvio.digregorio@
> <mailto:silvio.digregorio@>__gm__ail.com <http://gmail.com>
> <mailto:silvio.digregorio@__gmail.com
> <ma...@gmail.com>>>>>
>
> wrote:
>
>
>
> Hi,
>
>
> I hope
> I have
> sent to
>
> the
> right address.
>
>
> i have
> a graph
> (directed
>
> and
> unweighted)
> stored
>
> in
> hdfs like a
> adjacency
>
> list
> (140Milions of
>
> edges
> 6Milions
> of vertex)
>
>
> node<tab>neighbors
>
>
> 23 2
> 1 3
> 4 3
>
> 1 999 99923 <tel:1%20999%2099923> <tel:1%20999%2099923>
> <tel:1%20999%2099923>
> <tel:1%20999%2099923>
>
> 909
> .......
>
>
> ......
>
>
> hadoop
> version
> Hadoop
>
> 2.0.0-cdh4.3.0
> - java 1.6
>
>
>
> I have
> executed the
>
> giraph-1.0
> connected
>
> components
> example, in
>
> this
> fashion
>
> > hadoop jar
>
>
>
> /usr/local/giraph/giraph-______examples/target/giraph-______examples-1.0.0-for-hadoop-0.20.203.0
> <tel:0.20.203.0>
>
> <tel:0.20.203.0 <tel:0.20.203.0>>
> <tel:0.20.203.0 <tel:0.20.203.0> <tel:0.20.203.0 <tel:0.20.203.0>>>
> <tel:0.20.203.0 <tel:0.20.203.0> <tel:0.20.203.0
> <tel:0.20.203.0>> <tel:0.20.203.0 <tel:0.20.203.0>
> <tel:0.20.203.0 <tel:0.20.203.0>>>>-jar-with-__dependencies.____jar
>
> org.apache.giraph.GiraphRunner
>
>
> org.apache.giraph.examples.______ConnectedComponentsVertex
>
> -vif
>
>
> org.apache.giraph.io.formats.______IntIntNullTextInputFormat
>
>
>
> -vip
>
> /user/hdfs/lista_adj_txt
>
> -op
> connectedgiraph
>
> --workers 4
>
>
> and
> then fail with:
>
>
> 13/10/04
> 09:28:29 INFO
>
> utils.ConfigurationUtils: No
>
> edge
> input format
>
> specified.
> Ensure your
>
> InputFormat
> does not
>
> require one.
>
>
> 13/10/04
> 09:28:29 INFO
>
> utils.ConfigurationUtils: No
>
> output
> format
> specified.
>
> Ensure
> your
> OutputFormat
>
> does not
> require one.
>
>
> 13/10/04
> 09:28:30 INFO
>
> job.GiraphJob: run:
>
> Since
> checkpointing is
>
> disabled
> (default), do
>
> not
> allow any task
>
> retries (setting
>
> mapred.map.max.attempts
>
> = 0,
> old value = 4)
>
>
> 13/10/04
> 09:28:31 WARN
>
> mapred.JobClient: Use
>
> GenericOptionsParser for
>
> parsing the
> arguments.
>
> Applications should
>
> implement Tool
> for the same.
>
>
> 13/10/04
> 09:28:31 INFO
>
> mapred.JobClient:
>
> Cleaning up the
> staging
>
> area
>
> hdfs://srv-bigdata-dev-01.int
> <http://srv-bigdata-dev-01.int>
> <http://srv-bigdata-dev-01.int__>.____sose.it:8020/user/hdfs/.______staging/job_201309181636___0535
> <http://sose.it:8020/user/hdfs/.____staging/job_201309181636_0535>
> <http://sose.it:8020/user/__hdfs/.__staging/job___201309181636_0535
> <http://sose.it:8020/user/hdfs/.__staging/job_201309181636_0535>>
>
> <http://srv-bigdata-dev-01.__i__nt.sose.it:8020/user/hdfs/.____staging/job_201309181636_0535
> <http://int.sose.it:8020/user/hdfs/.__staging/job_201309181636_0535>
> <http://srv-bigdata-dev-01.__int.sose.it:8020/user/hdfs/.__staging/job_201309181636_0535
> <http://srv-bigdata-dev-01.int.sose.it:8020/user/hdfs/.staging/job_201309181636_0535>>__>
> <http://srv-bigdata-dev-01.__i____nt.sose.it:8020/user/hdfs/.______staging/job_201309181636___0535
> <http://i__nt.sose.it:8020/user/hdfs/.____staging/job_201309181636_0535>
> <http://int.sose.it:8020/user/__hdfs/.__staging/job___201309181636_0535
> <http://int.sose.it:8020/user/hdfs/.__staging/job_201309181636_0535>>
>
> <http://srv-bigdata-dev-01.__i__nt.sose.it:8020/user/hdfs/.____staging/job_201309181636_0535
> <http://int.sose.it:8020/user/hdfs/.__staging/job_201309181636_0535>
> <http://srv-bigdata-dev-01.__int.sose.it:8020/user/hdfs/.__staging/job_201309181636_0535
> <http://srv-bigdata-dev-01.int.sose.it:8020/user/hdfs/.staging/job_201309181636_0535>>__>__>
>
>
> Exception in thread
> "main"
>
> java.lang.______IncompatibleClassChangeError:
>
> Found
> interface
>
> org.apache.hadoop.mapreduce.______JobContext,
>
>
>
> but
> class was
> expected
>
>
> at
>
>
>
> org.apache.giraph.bsp.______BspOutputFormat.______checkOutputSpecs(______BspOutputFormat.java:43)
>
>
>
>
> ......
>
>
> Thanks
> in advance
>
>
>
>
>
> --
>
> Claudio Martella
> claudio.martella@gmail.com <ma...@gmail.com>
> <mailto:claudio.martella@__gmail.com
> <ma...@gmail.com>>
> <mailto:claudio.martella@
> <mailto:claudio.martella@>__gma__il.com <http://gmail.com>
> <mailto:claudio.martella@__gmail.com
> <ma...@gmail.com>>>
> <mailto:claudio.martella@ <mailto:claudio.martella@>
> <mailto:claudio.martella@
> <mailto:claudio.martella@>>__gm__a__il.com <http://gma__il.com>
> <http://gmail.com>
>
>
> <mailto:claudio.martella@
> <mailto:claudio.martella@>__gma__il.com <http://gmail.com>
> <mailto:claudio.martella@__gmail.com
> <ma...@gmail.com>>>>
>
>
>
>
>
>
> --
>
> Claudio Martella
> claudio.martella@gmail.com <ma...@gmail.com>
> <mailto:claudio.martella@__gmail.com
> <ma...@gmail.com>>
> <mailto:claudio.martella@
> <mailto:claudio.martella@>__gma__il.com <http://gmail.com>
> <mailto:claudio.martella@__gmail.com
> <ma...@gmail.com>>>
> <mailto:claudio.martella@ <mailto:claudio.martella@>
> <mailto:claudio.martella@
> <mailto:claudio.martella@>>__gm__a__il.com <http://gma__il.com>
> <http://gmail.com>
>
>
> <mailto:claudio.martella@
> <mailto:claudio.martella@>__gma__il.com <http://gmail.com>
> <mailto:claudio.martella@__gmail.com
> <ma...@gmail.com>>>>
>
>
>
>
>
> --
> Claudio Martella
> claudio.martella@gmail.com <ma...@gmail.com>
> <mailto:claudio.martella@__gmail.com
> <ma...@gmail.com>>
> <mailto:claudio.martella@
> <mailto:claudio.martella@>__gma__il.com <http://gmail.com>
> <mailto:claudio.martella@__gmail.com
> <ma...@gmail.com>>>
> <mailto:claudio.martella@ <mailto:claudio.martella@>
> <mailto:claudio.martella@
> <mailto:claudio.martella@>>__gm__a__il.com <http://gma__il.com>
> <http://gmail.com>
>
>
> <mailto:claudio.martella@
> <mailto:claudio.martella@>__gma__il.com <http://gmail.com>
> <mailto:claudio.martella@__gmail.com
> <ma...@gmail.com>>>>
>
>
>
>
>
> --
> Claudio Martella
> claudio.martella@gmail.com <ma...@gmail.com>
> <mailto:claudio.martella@__gmail.com
> <ma...@gmail.com>>
> <mailto:claudio.martella@
> <mailto:claudio.martella@>__gma__il.com <http://gmail.com>
> <mailto:claudio.martella@__gmail.com
> <ma...@gmail.com>>>
> <mailto:claudio.martella@ <mailto:claudio.martella@>
> <mailto:claudio.martella@
> <mailto:claudio.martella@>>__gm__a__il.com <http://gma__il.com>
> <http://gmail.com>
>
>
> <mailto:claudio.martella@
> <mailto:claudio.martella@>__gma__il.com <http://gmail.com>
> <mailto:claudio.martella@__gmail.com
> <ma...@gmail.com>>>>
>
>
>
>
>
> --
> Claudio Martella
> claudio.martella@gmail.com <ma...@gmail.com>
> <mailto:claudio.martella@__gmail.com
> <ma...@gmail.com>>
> <mailto:claudio.martella@
> <mailto:claudio.martella@>__gma__il.com <http://gmail.com>
> <mailto:claudio.martella@__gmail.com
> <ma...@gmail.com>>>
> <mailto:claudio.martella@ <mailto:claudio.martella@>
> <mailto:claudio.martella@
> <mailto:claudio.martella@>>__gm__a__il.com <http://gma__il.com>
> <http://gmail.com>
>
>
> <mailto:claudio.martella@
> <mailto:claudio.martella@>__gma__il.com <http://gmail.com>
> <mailto:claudio.martella@__gmail.com
> <ma...@gmail.com>>>>
>
>
>
>
>
> --
> Claudio Martella
> claudio.martella@gmail.com <ma...@gmail.com>
> <mailto:claudio.martella@__gmail.com
> <ma...@gmail.com>>
> <mailto:claudio.martella@
> <mailto:claudio.martella@>__gma__il.com <http://gmail.com>
> <mailto:claudio.martella@__gmail.com
> <ma...@gmail.com>>>
> <mailto:claudio.martella@ <mailto:claudio.martella@>
> <mailto:claudio.martella@
> <mailto:claudio.martella@>>__gm__a__il.com <http://gma__il.com>
> <http://gmail.com>
>
>
> <mailto:claudio.martella@
> <mailto:claudio.martella@>__gma__il.com <http://gmail.com>
> <mailto:claudio.martella@__gmail.com
> <ma...@gmail.com>>>>
>
>
> --
> Matthew Laird
> Lead Software Developer, Bioinformatics
> Brinkman Laboratory
> Simon Fraser University, Burnaby, BC, Canada
>
>
> --
> Matthew Laird
> Lead Software Developer, Bioinformatics
> Brinkman Laboratory
> Simon Fraser University, Burnaby, BC, Canada
>
>
>
>
> --
> Claudio Martella
> claudio.martella@gmail.com <ma...@gmail.com>
> <mailto:claudio.martella@__gmail.com
> <ma...@gmail.com>>
>
>
> --
> Matthew Laird
> Lead Software Developer, Bioinformatics
> Brinkman Laboratory
> Simon Fraser University, Burnaby, BC, Canada
>
>
--
Matthew Laird
Lead Software Developer, Bioinformatics
Brinkman Laboratory
Simon Fraser University, Burnaby, BC, Canada
Re: connected components example in giraph 1.0
Posted by Silvio Di gregorio <si...@gmail.com>.
Yes this case occur.
Il giorno 08/ott/2013 16:35, "Claudio Martella" <cl...@gmail.com>
ha scritto:
> One problem could be related to the fact that with a vertex inputformat if
> vertices only occur in the adjacency list of another vertex, they will only
> be created when a message is sent to them. I guess this could affect the
> execution of the algorithm.
> Could this be the case?
>
>
> On Tue, Oct 8, 2013 at 9:29 AM, Silvio Di gregorio <
> silvio.digregorio@gmail.com> wrote:
>
>> I have increased at 4GiB the heap space and the job has been terminated
>> successfully;
>> In the result file i've seen a strange thing, i have seen, like min
>> vertex id, "ZERO", that is:
>>
>> vertex_id<tab>[min vertex id namely label of connected components]
>> 12 0
>> 22 2
>> 23 2
>> but i don't have a vertex w/ named ZERO.
>> Has ZERO a special meaning? [for example not assigned etc ....]
>>
>> PS: i use giraph 1.0, CDH4.3 w/ MP1 0.20.203 not YARN i'm sorry.
>>
>>
>>
>> 2013/10/7 Silvio Di gregorio <si...@gmail.com>
>>
>>> with a working day in my shoulders and my children around i gave you
>>> random version,
>>> tomorrow morning in farm i'll be more precise.
>>> though i have hadoop 2.0 i use MR1 not YARN.
>>> I'm sorry again.
>>>
>>>
>>> 2013/10/7 Matthew Laird <la...@sfu.ca>
>>>
>>>> Hmm, it seems there have been some recent pushes that have broken trunk
>>>> as I discovered yesterday. :)
>>>>
>>>> But I guess that brings me back to my original question, to build
>>>> against 2.0.5 using the MR interface rather than YARN as Milinda suggested,
>>>> what profile would I use? Would I let it think it's 0.20xxx or tell it
>>>> 2.0.5, or something else?
>>>>
>>>> Thanks.
>>>>
>>>>
>>>> On 13-10-07 11:50 AM, Claudio Martella wrote:
>>>>
>>>>> trunk is 1.1.0-SNAPSHOT. I believe the date refers to the last update
>>>>> to
>>>>> the site.
>>>>>
>>>>>
>>>>> On Mon, Oct 7, 2013 at 8:48 PM, Matthew Laird <lairdm@sfu.ca
>>>>> <ma...@sfu.ca>> wrote:
>>>>>
>>>>> Thanks, which is my next curiosity... if I head over to
>>>>> http://giraph.apache.org/ I see at the top is says
>>>>>
>>>>> "Version: 1.1.0-SNAPSHOT | Last Published: 2013-08-10"
>>>>>
>>>>> However it's not clickable and the only download links I see are
>>>>> for
>>>>> 1.0.0. I'm assuming trunk in git isn't the same as the 1.1.0
>>>>> snapshot since more pushes have occurred since August, and I don't
>>>>> see a tag in git for 1.1.0
>>>>> (https://git-wip-us.apache.__**org/repos/asf?p=giraph.git
>>>>> <https://git-wip-us.apache.**org/repos/asf?p=giraph.git<https://git-wip-us.apache.org/repos/asf?p=giraph.git>>).
>>>>> So where
>>>>>
>>>>> does one get 1.1.0-SNAPSHOT from?
>>>>>
>>>>> Thanks.
>>>>>
>>>>>
>>>>> On 13-10-07 11:39 AM, Silvio Di gregorio wrote:
>>>>>
>>>>> Hi Matthew
>>>>> I use giraph 1.1.0 and cdh4 hadoop 0.23
>>>>>
>>>>> the patch of Marco:
>>>>>
>>>>> http://www.mail-archive.com/__**__user@giraph.apache.org/____*
>>>>> *msg00945/check.diff<http://www.mail-archive.com/____user@giraph.apache.org/____msg00945/check.diff>
>>>>> <http://www.mail-archive.com/_**_user@giraph.apache.org/__**
>>>>> msg00945/check.diff<http://www.mail-archive.com/__user@giraph.apache.org/__msg00945/check.diff>
>>>>> >
>>>>>
>>>>>
>>>>> <http://www.mail-archive.com/_**_user@giraph.apache.org/__**
>>>>> msg00945/check.diff<http://www.mail-archive.com/__user@giraph.apache.org/__msg00945/check.diff>
>>>>> <http://www.mail-archive.com/**user@giraph.apache.org/**
>>>>> msg00945/check.diff<http://www.mail-archive.com/user@giraph.apache.org/msg00945/check.diff>
>>>>> >>
>>>>>
>>>>> and build
>>>>>
>>>>> mvn clean install -P hadoop_cdh4.1.2
>>>>>
>>>>>
>>>>>
>>>>> Il giorno 07/ott/2013 20:20, "Matthew Laird" <lairdm@sfu.ca
>>>>> <ma...@sfu.ca>
>>>>> <mailto:lairdm@sfu.ca <ma...@sfu.ca>>> ha scritto:
>>>>>
>>>>>
>>>>>
>>>>> Hello, so I'm watching this thread with interest as well,
>>>>> since
>>>>> that's the exact example I'll be trying to run as well,
>>>>> and
>>>>> I see
>>>>> Silvio appears to be asking about running Giraph 1.0
>>>>> against Hadoop
>>>>> 2.0.0.
>>>>>
>>>>> I'm curious, is that a known working combination, 1.0 vs
>>>>> 2.0.x? And
>>>>> if so, are there patches to apply first and what maven
>>>>> command line
>>>>> should I build with? (ie. will a -Dhadoop.version=2.0.5
>>>>> give the
>>>>> needed MR based API or will it use YARN?) I figure it's
>>>>> better to
>>>>> ask than aimlessly try all the combinations. :)
>>>>>
>>>>> Thanks!
>>>>>
>>>>> On 13-10-07 11:01 AM, Claudio Martella wrote:
>>>>>
>>>>> try to increase the mappers' heap space through
>>>>> mapred.child.java.opts.
>>>>>
>>>>>
>>>>> On Mon, Oct 7, 2013 at 1:18 PM, Silvio Di gregorio
>>>>> <silvio.digregorio@gmail.com <mailto:silvio.digregorio@**
>>>>> gmail.com <si...@gmail.com>>
>>>>> <mailto:silvio.digregorio@__gm**ail.com <http://gmail.com>
>>>>> <ma...@gmail.com>
>>>>> >>
>>>>> <mailto:silvio.digregorio@
>>>>> <mailto:silvio.digregorio@>__g**m__ail.com<http://gm__ail.com><
>>>>> http://gmail.com>
>>>>>
>>>>>
>>>>> <mailto:silvio.digregorio@__gm**ail.com <http://gmail.com>
>>>>> <ma...@gmail.com>>>>>
>>>>> wrote:
>>>>>
>>>>>
>>>>> I think now my issue is another, thanks to all.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> java.lang.____**IllegalStateException: run:
>>>>> Caught an
>>>>>
>>>>>
>>>>> unrecoverable exception waitFor: ExecutionException
>>>>> occurred
>>>>> while waiting for
>>>>>
>>>>> org.apache.giraph.utils.____**ProgressableUtils$____**
>>>>> FutureWaitable@73a01e8a
>>>>> at
>>>>>
>>>>> org.apache.giraph.graph.____**GraphMapper.run(GraphMapper.__**
>>>>> __java:102)
>>>>>
>>>>>
>>>>>
>>>>> ......
>>>>> Caused by: java.lang.____**
>>>>> IllegalStateException:
>>>>>
>>>>> waitFor:
>>>>>
>>>>> ExecutionException occurred while waiting for
>>>>>
>>>>> org.apache.giraph.utils.____**ProgressableUtils$____**
>>>>> FutureWaitable@73a01e8a
>>>>> at
>>>>>
>>>>> org.apache.giraph.utils.____**ProgressableUtils.waitFor(____**
>>>>> ProgressableUtils.java:151)
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> ... 7 more
>>>>> Caused by:
>>>>> java.util.concurrent.____**ExecutionException:
>>>>> java.lang.OutOfMemoryError:*__**__Java heap space*
>>>>> at
>>>>>
>>>>> org.apache.giraph.utils.____**ProgressableUtils.waitFor(____**
>>>>> ProgressableUtils.java:143)
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> ... 13 more
>>>>>
>>>>>
>>>>>
>>>>> 2013/10/7 Claudio Martella
>>>>> <claudio.martella@gmail.com <mailto:claudio.martella@**
>>>>> gmail.com <cl...@gmail.com>>
>>>>> <mailto:claudio.martella@__gma**il.com <http://gmail.com>
>>>>> <ma...@gmail.com>
>>>>> >>
>>>>> <mailto:claudio.martella@
>>>>> <mailto:claudio.martella@>__gm**a__il.com <http://gma__il.com><
>>>>> http://gmail.com>
>>>>>
>>>>>
>>>>> <mailto:claudio.martella@__gma**il.com <http://gmail.com>
>>>>> <ma...@gmail.com>
>>>>> >>>>
>>>>>
>>>>> OK, thanks.
>>>>>
>>>>> I really have to push that patch in.
>>>>>
>>>>>
>>>>> On Mon, Oct 7, 2013 at 12:17 PM, Silvio Di
>>>>> gregorio
>>>>> <silvio.digregorio@gmail.com
>>>>> <ma...@gmail.com>
>>>>> >
>>>>> <mailto:silvio.digregorio@__gm**ail.com <http://gmail.com>
>>>>> <ma...@gmail.com>
>>>>> >>
>>>>> <mailto:silvio.digregorio@
>>>>> <mailto:silvio.digregorio@>__g**m__ail.com<http://gm__ail.com><
>>>>> http://gmail.com>
>>>>>
>>>>>
>>>>> <mailto:silvio.digregorio@__gm**ail.com <http://gmail.com>
>>>>> <ma...@gmail.com>>>>>
>>>>> wrote:
>>>>>
>>>>> yes i do,
>>>>> i have seen this in your post in:
>>>>> http://www.mail-archive.com/__**__user@giraph.apache.org/____*
>>>>> *msg00957.html<http://www.mail-archive.com/____user@giraph.apache.org/____msg00957.html>
>>>>> <http://www.mail-archive.com/_**_user@giraph.apache.org/__**
>>>>> msg00957.html<http://www.mail-archive.com/__user@giraph.apache.org/__msg00957.html>
>>>>> >
>>>>>
>>>>>
>>>>> <http://www.mail-archive.com/_**_user@giraph.apache.org/__**
>>>>> msg00957.html<http://www.mail-archive.com/__user@giraph.apache.org/__msg00957.html>
>>>>> <http://www.mail-archive.com/**user@giraph.apache.org/**
>>>>> msg00957.html<http://www.mail-archive.com/user@giraph.apache.org/msg00957.html>
>>>>> >>
>>>>>
>>>>> excuse me if i had checked in the
>>>>> mail-achive
>>>>> first I would
>>>>> have avoided the last post.
>>>>>
>>>>> Now zk issue are resolved.
>>>>>
>>>>>
>>>>> 2013/10/7 Claudio Martella
>>>>> <claudio.martella@gmail.com <mailto:claudio.martella@**
>>>>> gmail.com <cl...@gmail.com>>
>>>>> <mailto:claudio.martella@__gma**il.com <http://gmail.com>
>>>>> <ma...@gmail.com>
>>>>> >>
>>>>> <mailto:claudio.martella@
>>>>> <mailto:claudio.martella@>__gm**a__il.com <http://gma__il.com><
>>>>> http://gmail.com>
>>>>>
>>>>>
>>>>> <mailto:claudio.martella@__gma**il.com <http://gmail.com>
>>>>> <ma...@gmail.com>
>>>>> >>>>
>>>>>
>>>>> Can you try applying this one first?
>>>>>
>>>>> <<< <
>>>>> http://www.mail-archive.com/__**__user@giraph.apache.org/____*
>>>>> *msg00945/check.diff<http://www.mail-archive.com/____user@giraph.apache.org/____msg00945/check.diff>
>>>>> <http://www.mail-archive.com/_**_user@giraph.apache.org/__**
>>>>> msg00945/check.diff<http://www.mail-archive.com/__user@giraph.apache.org/__msg00945/check.diff>
>>>>> >
>>>>>
>>>>>
>>>>> <http://www.mail-archive.com/_**_user@giraph.apache.org/__**
>>>>> msg00945/check.diff<http://www.mail-archive.com/__user@giraph.apache.org/__msg00945/check.diff>
>>>>> <http://www.mail-archive.com/**user@giraph.apache.org/**
>>>>> msg00945/check.diff<http://www.mail-archive.com/user@giraph.apache.org/msg00945/check.diff>
>>>>> >><<<<
>>>>>
>>>>>
>>>>> On Mon, Oct 7, 2013 at 8:40 AM,
>>>>> Silvio
>>>>> Di gregorio
>>>>> <silvio.digregorio@gmail.com
>>>>> <ma...@gmail.com>
>>>>> >
>>>>> <mailto:silvio.digregorio@__gm**ail.com <http://gmail.com>
>>>>> <ma...@gmail.com>
>>>>> >>
>>>>> <mailto:silvio.digregorio@
>>>>> <mailto:silvio.digregorio@>__g**m__ail.com<http://gm__ail.com><
>>>>> http://gmail.com>
>>>>>
>>>>>
>>>>> <mailto:silvio.digregorio@__gm**ail.com <http://gmail.com>
>>>>> <ma...@gmail.com>>>>>
>>>>> wrote:
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> *As i said i have builded*
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> /giraph-examples-1.0.0-for-___**_hadoop-2.0.0-cdh4.1.2-jar-___
>>>>> **_with-dependencies.jar/
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> *for cdh4, successfully. The job
>>>>> start to
>>>>> monitoring the success rate:*
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> /13/10/07 08:28:45 INFO
>>>>> mapred.JobClient:
>>>>> map 0% reduce 0%/
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> *but then*
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> /Error running child
>>>>>
>>>>> java.lang.____**IllegalStateException: run:
>>>>>
>>>>>
>>>>> Caught an unrecoverable exception
>>>>> java.io.FileNotFoundException:
>>>>> File
>>>>> _bsp/_defaultZkManagerDir/job_**____201309181636_0678/_**
>>>>> zkServer
>>>>>
>>>>>
>>>>> does not exist.
>>>>> /
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> /.../
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> /Caused by:
>>>>> java.lang.RuntimeException:
>>>>> java.io.FileNotFoundException: File
>>>>>
>>>>> _bsp/_defaultZkManagerDir/job_**____201309181636_0678/_**zkServer
>>>>> does
>>>>>
>>>>>
>>>>> not exist./
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> 2013/10/5 Silvio Di gregorio
>>>>> <silvio.digregorio@gmail.com
>>>>> <ma...@gmail.com>
>>>>> >
>>>>> <mailto:silvio.digregorio@__gm**ail.com <http://gmail.com>
>>>>> <ma...@gmail.com>
>>>>> >>
>>>>> <mailto:silvio.digregorio@
>>>>> <mailto:silvio.digregorio@>__g**m__ail.com<http://gm__ail.com><
>>>>> http://gmail.com>
>>>>>
>>>>>
>>>>> <mailto:silvio.digregorio@__gm**ail.com <http://gmail.com>
>>>>> <ma...@gmail.com>
>>>>> >>>>
>>>>>
>>>>> I ha ve build w/
>>>>> hadoop_cdh4.1.2 parameter.
>>>>> Something is changed, monday
>>>>> i
>>>>> report the
>>>>> result. Now the farm is
>>>>> closed.
>>>>>
>>>>> Il giorno 05/ott/2013 14:06,
>>>>> "Claudio
>>>>> Martella"
>>>>> <claudio.martella@gmail.com <mailto:claudio.martella@**
>>>>> gmail.com <cl...@gmail.com>>
>>>>> <mailto:claudio.martella@__gma**il.com <http://gmail.com>
>>>>> <ma...@gmail.com>
>>>>> >>
>>>>> <mailto:claudio.martella@
>>>>> <mailto:claudio.martella@>__gm**a__il.com <http://gma__il.com><
>>>>> http://gmail.com>
>>>>>
>>>>>
>>>>> <mailto:claudio.martella@__gma**il.com <http://gmail.com>
>>>>> <ma...@gmail.com>>>>>
>>>>> ha scritto:
>>>>>
>>>>> Oh, right, -vof is in
>>>>> trunk. Anyway
>>>>> it looks
>>>>> like you built giraph for
>>>>> the wrong
>>>>> profile.
>>>>> You mentioned you're
>>>>> running on
>>>>> 2.0, but
>>>>> your giraph is built for
>>>>> 0.20.203 <tel:0.20.203>
>>>>> <tel:0.20.203 <tel:0.20.203>>
>>>>> <tel:0.20.203 <tel:0.20.203> <tel:0.20.203 <tel:0.20.203>>>.
>>>>> try
>>>>>
>>>>> building with a profile
>>>>>
>>>>> for your hadoop version.
>>>>>
>>>>>
>>>>> On Fri, Oct 4, 2013 at
>>>>> 2:35 PM,
>>>>> Silvio Di
>>>>> gregorio
>>>>> <silvio.digregorio@gmail.com
>>>>> <ma...@gmail.com>
>>>>> >
>>>>> <mailto:silvio.digregorio@__gm**ail.com <http://gmail.com>
>>>>> <ma...@gmail.com>
>>>>> >>
>>>>> <mailto:silvio.digregorio@
>>>>> <mailto:silvio.digregorio@>__g**m__ail.com<http://gm__ail.com><
>>>>> http://gmail.com>
>>>>>
>>>>> <mailto:silvio.digregorio@__gm**ail.com <http://gmail.com>
>>>>> <ma...@gmail.com>>>>>
>>>>> wrote:
>>>>>
>>>>>
>>>>>
>>>>> org.apache.commons.cli.____**
>>>>> UnrecognizedOptionException:
>>>>>
>>>>>
>>>>> Unrecognized option:
>>>>> -vof
>>>>>
>>>>> in 1.0 version is
>>>>> -of,--outputFormat
>>>>> <arg>
>>>>> Vertex output format
>>>>> -op,--outputPath
>>>>> <arg>
>>>>> Vertex output path
>>>>>
>>>>>
>>>>>
>>>>> 2013/10/4 Claudio
>>>>> Martella
>>>>> <claudio.martella@gmail.com <mailto:claudio.martella@**
>>>>> gmail.com <cl...@gmail.com>>
>>>>> <mailto:claudio.martella@__gma**il.com <http://gmail.com>
>>>>> <ma...@gmail.com>
>>>>> >>
>>>>> <mailto:claudio.martella@
>>>>> <mailto:claudio.martella@>__gm**a__il.com <http://gma__il.com><
>>>>> http://gmail.com>
>>>>>
>>>>>
>>>>> <mailto:claudio.martella@__gma**il.com <http://gmail.com>
>>>>> <ma...@gmail.com>
>>>>> >>>>
>>>>>
>>>>> did you try the
>>>>> argument
>>>>> (-vof) i
>>>>> suggested?
>>>>>
>>>>>
>>>>> On Fri, Oct 4,
>>>>> 2013 at 2:13 PM,
>>>>> Silvio Di
>>>>> gregorio
>>>>> <silvio.digregorio@gmail.com
>>>>> <ma...@gmail.com>
>>>>> >
>>>>> <mailto:silvio.digregorio@__gm**ail.com <http://gmail.com>
>>>>> <ma...@gmail.com>
>>>>> >>
>>>>> <mailto:silvio.digregorio@
>>>>> <mailto:silvio.digregorio@>__g**m__ail.com<http://gm__ail.com><
>>>>> http://gmail.com>
>>>>>
>>>>>
>>>>> <mailto:silvio.digregorio@__gm**ail.com <http://gmail.com>
>>>>> <ma...@gmail.com>>>>>
>>>>> wrote:
>>>>>
>>>>>
>>>>> i've
>>>>> specified
>>>>> -of
>>>>>
>>>>>
>>>>> org.apache.giraph.io.formats._**___IdWithValueTextOutputFormat
>>>>>
>>>>>
>>>>>
>>>>> but the same
>>>>> error was
>>>>> produced
>>>>>
>>>>> Exception in
>>>>> thread "main"
>>>>>
>>>>> java.lang.____**IncompatibleClassChangeError:
>>>>> Found
>>>>> interface
>>>>>
>>>>> org.apache.hadoop.mapreduce.__**__JobContext,
>>>>>
>>>>>
>>>>> but class was
>>>>> expected
>>>>> at
>>>>>
>>>>>
>>>>> org.apache.giraph.bsp.____**BspOutputFormat.____**
>>>>> checkOutputSpecs(____**BspOutputFormat.java:43)
>>>>> at
>>>>>
>>>>>
>>>>> org.apache.hadoop.mapred.____**JobClient$2.run(JobClient.____*
>>>>> *java:984)
>>>>> at
>>>>>
>>>>>
>>>>> org.apache.hadoop.mapred.____**JobClient$2.run(JobClient.____*
>>>>> *java:945)
>>>>> at
>>>>>
>>>>> java.security.____**AccessController.doPrivileged(**
>>>>> ____Native
>>>>> Method)
>>>>> at
>>>>>
>>>>> javax.security.auth.Subject.__**
>>>>> __doAs(Subject.java:396)
>>>>> at
>>>>>
>>>>>
>>>>> org.apache.hadoop.security.___**_UserGroupInformation.doAs(___
>>>>> **_UserGroupInformation.java:___**_1408)
>>>>> at
>>>>>
>>>>>
>>>>> org.apache.hadoop.mapred.____**JobClient.submitJobInternal(__*
>>>>> *__JobClient.java:945)
>>>>> at
>>>>>
>>>>> org.apache.hadoop.mapreduce.__**
>>>>> __Job.submit(Job.java:566)
>>>>> at
>>>>>
>>>>>
>>>>> org.apache.hadoop.mapreduce.__**__Job.waitForCompletion(Job.__
>>>>> **__java:596)
>>>>> at
>>>>>
>>>>>
>>>>> org.apache.giraph.job.____**GiraphJob.run(GiraphJob.java:_**
>>>>> ___237)
>>>>> at
>>>>>
>>>>>
>>>>> org.apache.giraph.____**GiraphRunner.run(GiraphRunner.**
>>>>> ____java:94)
>>>>> at
>>>>>
>>>>>
>>>>> org.apache.hadoop.util.____**ToolRunner.run(ToolRunner.____**
>>>>> java:70)
>>>>> at
>>>>>
>>>>>
>>>>> org.apache.hadoop.util.____**ToolRunner.run(ToolRunner.____**
>>>>> java:84)
>>>>> at
>>>>>
>>>>>
>>>>> org.apache.giraph.____**GiraphRunner.main(____**
>>>>> GiraphRunner.java:124)
>>>>> at
>>>>>
>>>>> sun.reflect.____**NativeMethodAccessorImpl.____**
>>>>> invoke0(Native
>>>>> Method)
>>>>> at
>>>>>
>>>>>
>>>>> sun.reflect.____**NativeMethodAccessorImpl.____**invoke(____**
>>>>> NativeMethodAccessorImpl.java:**____39)
>>>>> at
>>>>>
>>>>>
>>>>> sun.reflect.____**DelegatingMethodAccessorImpl._**
>>>>> ___invoke(____**DelegatingMethodAccessorImpl._**___java:25)
>>>>> at
>>>>>
>>>>> java.lang.reflect.Method.____**
>>>>> invoke(Method.java:597)
>>>>> at
>>>>>
>>>>> org.apache.hadoop.util.RunJar.**
>>>>> ____main(RunJar.java:208)
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> 2013/10/4
>>>>> Claudio Martella
>>>>> <claudio.martella@gmail.com <mailto:claudio.martella@**
>>>>> gmail.com <cl...@gmail.com>>
>>>>> <mailto:claudio.martella@__gma**il.com <http://gmail.com>
>>>>> <ma...@gmail.com>
>>>>> >>
>>>>> <mailto:claudio.martella@
>>>>> <mailto:claudio.martella@>__gm**a__il.com <http://gma__il.com><
>>>>> http://gmail.com>
>>>>>
>>>>>
>>>>> <mailto:claudio.martella@__gma**il.com <http://gmail.com>
>>>>> <ma...@gmail.com>
>>>>> >>>>
>>>>>
>>>>> Hi,
>>>>>
>>>>> you need
>>>>> to specify the
>>>>> vertex
>>>>> outputformat
>>>>> class
>>>>> (-vof
>>>>> option), e.g.
>>>>>
>>>>>
>>>>> org.apache.giraph.io.formats._**___**
>>>>> IdWithValueTextOutputFormat.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Fri,
>>>>> Oct 4, 2013
>>>>> at 1:06
>>>>> PM,
>>>>> Silvio
>>>>> Di gregorio
>>>>> <silvio.digregorio@gmail.com
>>>>> <ma...@gmail.com>
>>>>> >
>>>>> <mailto:silvio.digregorio@__gm**ail.com <http://gmail.com>
>>>>> <ma...@gmail.com>
>>>>> >>
>>>>> <mailto:silvio.digregorio@
>>>>> <mailto:silvio.digregorio@>__g**m__ail.com<http://gm__ail.com><
>>>>> http://gmail.com>
>>>>>
>>>>>
>>>>> <mailto:silvio.digregorio@__gm**ail.com <http://gmail.com>
>>>>> <ma...@gmail.com>
>>>>> >>>>
>>>>> wrote:
>>>>>
>>>>>
>>>>> Hi,
>>>>>
>>>>> I
>>>>> hope
>>>>> I have
>>>>> sent to
>>>>> the
>>>>> right address.
>>>>>
>>>>> i
>>>>> have
>>>>> a graph
>>>>> (directed
>>>>> and
>>>>> unweighted)
>>>>> stored
>>>>> in
>>>>> hdfs like a
>>>>> adjacency
>>>>> list
>>>>> (140Milions of
>>>>> edges
>>>>> 6Milions
>>>>> of vertex)
>>>>>
>>>>>
>>>>> node<tab>neighbors
>>>>>
>>>>> 23
>>>>> 2
>>>>> 1 3
>>>>> 4 3
>>>>>
>>>>> 1 999 99923 <tel:1%20999%2099923> <tel:1%20999%2099923>
>>>>> <tel:1%20999%2099923>
>>>>> 909
>>>>> .......
>>>>>
>>>>>
>>>>> ......
>>>>>
>>>>>
>>>>> hadoop
>>>>> version
>>>>> Hadoop
>>>>>
>>>>> 2.0.0-cdh4.3.0
>>>>> - java 1.6
>>>>>
>>>>>
>>>>> I
>>>>> have
>>>>> executed the
>>>>>
>>>>> giraph-1.0
>>>>> connected
>>>>>
>>>>> components
>>>>> example, in
>>>>> this
>>>>> fashion
>>>>>
>>>>> > hadoop jar
>>>>>
>>>>>
>>>>> /usr/local/giraph/giraph-____**examples/target/giraph-____**
>>>>> examples-1.0.0-for-hadoop-0.20.203.0
>>>>>
>>>>> <tel:0.20.203.0>
>>>>> <tel:0.20.203.0 <tel:0.20.203.0>>
>>>>> <tel:0.20.203.0 <tel:0.20.203.0> <tel:0.20.203.0
>>>>> <tel:0.20.203.0>>>-jar-with-**dependencies.____jar
>>>>>
>>>>> org.apache.giraph.GiraphRunner
>>>>>
>>>>> org.apache.giraph.examples.___**
>>>>> _ConnectedComponentsVertex
>>>>>
>>>>> -vif
>>>>>
>>>>> org.apache.giraph.io.formats._**
>>>>> ___IntIntNullTextInputFormat
>>>>>
>>>>>
>>>>>
>>>>> -vip
>>>>>
>>>>> /user/hdfs/lista_adj_txt
>>>>>
>>>>> -op
>>>>> connectedgiraph
>>>>>
>>>>> --workers 4
>>>>>
>>>>> and
>>>>> then fail with:
>>>>>
>>>>>
>>>>> 13/10/04
>>>>> 09:28:29 INFO
>>>>>
>>>>> utils.ConfigurationUtils: No
>>>>> edge
>>>>> input format
>>>>>
>>>>> specified.
>>>>> Ensure your
>>>>>
>>>>> InputFormat
>>>>> does not
>>>>>
>>>>> require one.
>>>>>
>>>>>
>>>>> 13/10/04
>>>>> 09:28:29 INFO
>>>>>
>>>>> utils.ConfigurationUtils: No
>>>>>
>>>>> output
>>>>> format
>>>>> specified.
>>>>>
>>>>> Ensure
>>>>> your
>>>>> OutputFormat
>>>>> does
>>>>> not
>>>>> require one.
>>>>>
>>>>>
>>>>> 13/10/04
>>>>> 09:28:30 INFO
>>>>>
>>>>> job.GiraphJob: run:
>>>>> Since
>>>>> checkpointing is
>>>>>
>>>>> disabled
>>>>> (default), do
>>>>> not
>>>>> allow any task
>>>>>
>>>>> retries (setting
>>>>>
>>>>> mapred.map.max.attempts
>>>>> = 0,
>>>>> old value = 4)
>>>>>
>>>>>
>>>>> 13/10/04
>>>>> 09:28:31 WARN
>>>>>
>>>>> mapred.JobClient: Use
>>>>>
>>>>> GenericOptionsParser for
>>>>>
>>>>> parsing the
>>>>> arguments.
>>>>>
>>>>> Applications should
>>>>>
>>>>> implement Tool
>>>>> for the same.
>>>>>
>>>>>
>>>>> 13/10/04
>>>>> 09:28:31 INFO
>>>>>
>>>>> mapred.JobClient:
>>>>>
>>>>> Cleaning up the
>>>>> staging
>>>>> area
>>>>>
>>>>> hdfs://srv-bigdata-dev-01.int
>>>>> <http://srv-bigdata-dev-01.int**>.____sose.it:8020/user/hdfs/.
>>>>> **____staging/job_201309181636_**0535<http://sose.it:8020/user/hdfs/.____staging/job_201309181636_0535>
>>>>> <http://sose.it:8020/user/**hdfs/.__staging/job_**
>>>>> 201309181636_0535<http://sose.it:8020/user/hdfs/.__staging/job_201309181636_0535>
>>>>> >
>>>>>
>>>>> <http://srv-bigdata-dev-01.__i**nt.sose.it:8020/user/hdfs/.__*
>>>>> *staging/job_201309181636_0535<http://int.sose.it:8020/user/hdfs/.__staging/job_201309181636_0535>
>>>>> <http://srv-bigdata-dev-01.**int.sose.it:8020/user/hdfs/.**
>>>>> staging/job_201309181636_0535<http://srv-bigdata-dev-01.int.sose.it:8020/user/hdfs/.staging/job_201309181636_0535>
>>>>> >**>
>>>>> <http://srv-bigdata-dev-01.__i**
>>>>> __nt.sose.it:8020/user/hdfs/._**___staging/job_201309181636_**0535<http://i__nt.sose.it:8020/user/hdfs/.____staging/job_201309181636_0535>
>>>>> <http://int.sose.it:8020/user/**hdfs/.__staging/job_**
>>>>> 201309181636_0535<http://int.sose.it:8020/user/hdfs/.__staging/job_201309181636_0535>
>>>>> >
>>>>>
>>>>> <http://srv-bigdata-dev-01.__i**nt.sose.it:8020/user/hdfs/.__*
>>>>> *staging/job_201309181636_0535<http://int.sose.it:8020/user/hdfs/.__staging/job_201309181636_0535>
>>>>> <http://srv-bigdata-dev-01.**int.sose.it:8020/user/hdfs/.**
>>>>> staging/job_201309181636_0535<http://srv-bigdata-dev-01.int.sose.it:8020/user/hdfs/.staging/job_201309181636_0535>
>>>>> >**>__>
>>>>>
>>>>>
>>>>> Exception in thread
>>>>> "main"
>>>>>
>>>>> java.lang.____**IncompatibleClassChangeError:
>>>>> Found
>>>>> interface
>>>>>
>>>>> org.apache.hadoop.mapreduce.__**__JobContext,
>>>>>
>>>>>
>>>>> but
>>>>> class was
>>>>> expected
>>>>>
>>>>>
>>>>> at
>>>>>
>>>>>
>>>>> org.apache.giraph.bsp.____**BspOutputFormat.____**
>>>>> checkOutputSpecs(____**BspOutputFormat.java:43)
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> ......
>>>>>
>>>>>
>>>>> Thanks
>>>>> in advance
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>>
>>>>> Claudio Martella
>>>>> claudio.martella@gmail.com <mailto:claudio.martella@**
>>>>> gmail.com <cl...@gmail.com>>
>>>>> <mailto:claudio.martella@__gma**il.com <http://gmail.com>
>>>>> <ma...@gmail.com>
>>>>> >>
>>>>> <mailto:claudio.martella@
>>>>> <mailto:claudio.martella@>__gm**a__il.com <http://gma__il.com><
>>>>> http://gmail.com>
>>>>>
>>>>>
>>>>> <mailto:claudio.martella@__gma**il.com <http://gmail.com>
>>>>> <ma...@gmail.com>
>>>>> >>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Claudio
>>>>> Martella
>>>>> claudio.martella@gmail.com <mailto:claudio.martella@**
>>>>> gmail.com <cl...@gmail.com>>
>>>>> <mailto:claudio.martella@__gma**il.com <http://gmail.com>
>>>>> <ma...@gmail.com>
>>>>> >>
>>>>> <mailto:claudio.martella@
>>>>> <mailto:claudio.martella@>__gm**a__il.com <http://gma__il.com><
>>>>> http://gmail.com>
>>>>>
>>>>>
>>>>> <mailto:claudio.martella@__gma**il.com <http://gmail.com>
>>>>> <ma...@gmail.com>
>>>>> >>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Claudio Martella
>>>>> claudio.martella@gmail.com <mailto:claudio.martella@**
>>>>> gmail.com <cl...@gmail.com>>
>>>>> <mailto:claudio.martella@__gma**il.com <http://gmail.com>
>>>>> <ma...@gmail.com>
>>>>> >>
>>>>> <mailto:claudio.martella@
>>>>> <mailto:claudio.martella@>__gm**a__il.com <http://gma__il.com><
>>>>> http://gmail.com>
>>>>>
>>>>>
>>>>> <mailto:claudio.martella@__gma**il.com <http://gmail.com>
>>>>> <ma...@gmail.com>
>>>>> >>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Claudio Martella
>>>>> claudio.martella@gmail.com <mailto:claudio.martella@**
>>>>> gmail.com <cl...@gmail.com>>
>>>>> <mailto:claudio.martella@__gma**il.com <http://gmail.com>
>>>>> <ma...@gmail.com>
>>>>> >>
>>>>> <mailto:claudio.martella@
>>>>> <mailto:claudio.martella@>__gm**a__il.com <http://gma__il.com><
>>>>> http://gmail.com>
>>>>>
>>>>>
>>>>> <mailto:claudio.martella@__gma**il.com <http://gmail.com>
>>>>> <ma...@gmail.com>
>>>>> >>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Claudio Martella
>>>>> claudio.martella@gmail.com <mailto:claudio.martella@**
>>>>> gmail.com <cl...@gmail.com>>
>>>>> <mailto:claudio.martella@__gma**il.com <http://gmail.com>
>>>>> <ma...@gmail.com>
>>>>> >>
>>>>> <mailto:claudio.martella@
>>>>> <mailto:claudio.martella@>__gm**a__il.com <http://gma__il.com><
>>>>> http://gmail.com>
>>>>>
>>>>>
>>>>> <mailto:claudio.martella@__gma**il.com <http://gmail.com>
>>>>> <ma...@gmail.com>
>>>>> >>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Claudio Martella
>>>>> claudio.martella@gmail.com <mailto:claudio.martella@**
>>>>> gmail.com <cl...@gmail.com>>
>>>>> <mailto:claudio.martella@__gma**il.com <http://gmail.com>
>>>>> <ma...@gmail.com>
>>>>> >>
>>>>> <mailto:claudio.martella@
>>>>> <mailto:claudio.martella@>__gm**a__il.com <http://gma__il.com><
>>>>> http://gmail.com>
>>>>>
>>>>>
>>>>> <mailto:claudio.martella@__gma**il.com <http://gmail.com>
>>>>> <ma...@gmail.com>
>>>>> >>>
>>>>>
>>>>>
>>>>> --
>>>>> Matthew Laird
>>>>> Lead Software Developer, Bioinformatics
>>>>> Brinkman Laboratory
>>>>> Simon Fraser University, Burnaby, BC, Canada
>>>>>
>>>>>
>>>>> --
>>>>> Matthew Laird
>>>>> Lead Software Developer, Bioinformatics
>>>>> Brinkman Laboratory
>>>>> Simon Fraser University, Burnaby, BC, Canada
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Claudio Martella
>>>>> claudio.martella@gmail.com <ma...@gmail.com>
>>>>> >
>>>>>
>>>>
>>>> --
>>>> Matthew Laird
>>>> Lead Software Developer, Bioinformatics
>>>> Brinkman Laboratory
>>>> Simon Fraser University, Burnaby, BC, Canada
>>>>
>>>
>>>
>>
>
>
> --
> Claudio Martella
> claudio.martella@gmail.com
>
Re: connected components example in giraph 1.0
Posted by Claudio Martella <cl...@gmail.com>.
One problem could be related to the fact that with a vertex inputformat if
vertices only occur in the adjacency list of another vertex, they will only
be created when a message is sent to them. I guess this could affect the
execution of the algorithm.
Could this be the case?
On Tue, Oct 8, 2013 at 9:29 AM, Silvio Di gregorio <
silvio.digregorio@gmail.com> wrote:
> I have increased at 4GiB the heap space and the job has been terminated
> successfully;
> In the result file i've seen a strange thing, i have seen, like min vertex
> id, "ZERO", that is:
>
> vertex_id<tab>[min vertex id namely label of connected components]
> 12 0
> 22 2
> 23 2
> but i don't have a vertex w/ named ZERO.
> Has ZERO a special meaning? [for example not assigned etc ....]
>
> PS: i use giraph 1.0, CDH4.3 w/ MP1 0.20.203 not YARN i'm sorry.
>
>
>
> 2013/10/7 Silvio Di gregorio <si...@gmail.com>
>
>> with a working day in my shoulders and my children around i gave you
>> random version,
>> tomorrow morning in farm i'll be more precise.
>> though i have hadoop 2.0 i use MR1 not YARN.
>> I'm sorry again.
>>
>>
>> 2013/10/7 Matthew Laird <la...@sfu.ca>
>>
>>> Hmm, it seems there have been some recent pushes that have broken trunk
>>> as I discovered yesterday. :)
>>>
>>> But I guess that brings me back to my original question, to build
>>> against 2.0.5 using the MR interface rather than YARN as Milinda suggested,
>>> what profile would I use? Would I let it think it's 0.20xxx or tell it
>>> 2.0.5, or something else?
>>>
>>> Thanks.
>>>
>>>
>>> On 13-10-07 11:50 AM, Claudio Martella wrote:
>>>
>>>> trunk is 1.1.0-SNAPSHOT. I believe the date refers to the last update to
>>>> the site.
>>>>
>>>>
>>>> On Mon, Oct 7, 2013 at 8:48 PM, Matthew Laird <lairdm@sfu.ca
>>>> <ma...@sfu.ca>> wrote:
>>>>
>>>> Thanks, which is my next curiosity... if I head over to
>>>> http://giraph.apache.org/ I see at the top is says
>>>>
>>>> "Version: 1.1.0-SNAPSHOT | Last Published: 2013-08-10"
>>>>
>>>> However it's not clickable and the only download links I see are for
>>>> 1.0.0. I'm assuming trunk in git isn't the same as the 1.1.0
>>>> snapshot since more pushes have occurred since August, and I don't
>>>> see a tag in git for 1.1.0
>>>> (https://git-wip-us.apache.__**org/repos/asf?p=giraph.git
>>>> <https://git-wip-us.apache.**org/repos/asf?p=giraph.git<https://git-wip-us.apache.org/repos/asf?p=giraph.git>>).
>>>> So where
>>>>
>>>> does one get 1.1.0-SNAPSHOT from?
>>>>
>>>> Thanks.
>>>>
>>>>
>>>> On 13-10-07 11:39 AM, Silvio Di gregorio wrote:
>>>>
>>>> Hi Matthew
>>>> I use giraph 1.1.0 and cdh4 hadoop 0.23
>>>>
>>>> the patch of Marco:
>>>>
>>>> http://www.mail-archive.com/__**__user@giraph.apache.org/____**
>>>> msg00945/check.diff<http://www.mail-archive.com/____user@giraph.apache.org/____msg00945/check.diff>
>>>> <http://www.mail-archive.com/_**_user@giraph.apache.org/__**
>>>> msg00945/check.diff<http://www.mail-archive.com/__user@giraph.apache.org/__msg00945/check.diff>
>>>> >
>>>>
>>>>
>>>> <http://www.mail-archive.com/_**_user@giraph.apache.org/__**
>>>> msg00945/check.diff<http://www.mail-archive.com/__user@giraph.apache.org/__msg00945/check.diff>
>>>> <http://www.mail-archive.com/**user@giraph.apache.org/**
>>>> msg00945/check.diff<http://www.mail-archive.com/user@giraph.apache.org/msg00945/check.diff>
>>>> >>
>>>>
>>>> and build
>>>>
>>>> mvn clean install -P hadoop_cdh4.1.2
>>>>
>>>>
>>>>
>>>> Il giorno 07/ott/2013 20:20, "Matthew Laird" <lairdm@sfu.ca
>>>> <ma...@sfu.ca>
>>>> <mailto:lairdm@sfu.ca <ma...@sfu.ca>>> ha scritto:
>>>>
>>>>
>>>>
>>>> Hello, so I'm watching this thread with interest as well,
>>>> since
>>>> that's the exact example I'll be trying to run as well, and
>>>> I see
>>>> Silvio appears to be asking about running Giraph 1.0
>>>> against Hadoop
>>>> 2.0.0.
>>>>
>>>> I'm curious, is that a known working combination, 1.0 vs
>>>> 2.0.x? And
>>>> if so, are there patches to apply first and what maven
>>>> command line
>>>> should I build with? (ie. will a -Dhadoop.version=2.0.5
>>>> give the
>>>> needed MR based API or will it use YARN?) I figure it's
>>>> better to
>>>> ask than aimlessly try all the combinations. :)
>>>>
>>>> Thanks!
>>>>
>>>> On 13-10-07 11:01 AM, Claudio Martella wrote:
>>>>
>>>> try to increase the mappers' heap space through
>>>> mapred.child.java.opts.
>>>>
>>>>
>>>> On Mon, Oct 7, 2013 at 1:18 PM, Silvio Di gregorio
>>>> <silvio.digregorio@gmail.com <mailto:silvio.digregorio@**
>>>> gmail.com <si...@gmail.com>>
>>>> <mailto:silvio.digregorio@__gm**ail.com <http://gmail.com>
>>>> <ma...@gmail.com>
>>>> >>
>>>> <mailto:silvio.digregorio@
>>>> <mailto:silvio.digregorio@>__g**m__ail.com <http://gm__ail.com><
>>>> http://gmail.com>
>>>>
>>>>
>>>> <mailto:silvio.digregorio@__gm**ail.com <http://gmail.com>
>>>> <ma...@gmail.com>>>>>
>>>> wrote:
>>>>
>>>>
>>>> I think now my issue is another, thanks to all.
>>>>
>>>>
>>>>
>>>>
>>>> java.lang.____**IllegalStateException: run:
>>>> Caught an
>>>>
>>>>
>>>> unrecoverable exception waitFor: ExecutionException
>>>> occurred
>>>> while waiting for
>>>>
>>>> org.apache.giraph.utils.____**ProgressableUtils$____**
>>>> FutureWaitable@73a01e8a
>>>> at
>>>>
>>>> org.apache.giraph.graph.____**GraphMapper.run(GraphMapper.__**
>>>> __java:102)
>>>>
>>>>
>>>>
>>>> ......
>>>> Caused by: java.lang.____**IllegalStateException:
>>>>
>>>> waitFor:
>>>>
>>>> ExecutionException occurred while waiting for
>>>>
>>>> org.apache.giraph.utils.____**ProgressableUtils$____**
>>>> FutureWaitable@73a01e8a
>>>> at
>>>>
>>>> org.apache.giraph.utils.____**ProgressableUtils.waitFor(____**
>>>> ProgressableUtils.java:151)
>>>>
>>>>
>>>>
>>>>
>>>> ... 7 more
>>>> Caused by:
>>>> java.util.concurrent.____**ExecutionException:
>>>> java.lang.OutOfMemoryError:*__**__Java heap space*
>>>> at
>>>>
>>>> org.apache.giraph.utils.____**ProgressableUtils.waitFor(____**
>>>> ProgressableUtils.java:143)
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> ... 13 more
>>>>
>>>>
>>>>
>>>> 2013/10/7 Claudio Martella
>>>> <claudio.martella@gmail.com <mailto:claudio.martella@**
>>>> gmail.com <cl...@gmail.com>>
>>>> <mailto:claudio.martella@__gma**il.com <http://gmail.com>
>>>> <ma...@gmail.com>
>>>> >>
>>>> <mailto:claudio.martella@
>>>> <mailto:claudio.martella@>__gm**a__il.com <http://gma__il.com><
>>>> http://gmail.com>
>>>>
>>>>
>>>> <mailto:claudio.martella@__gma**il.com <http://gmail.com>
>>>> <ma...@gmail.com>
>>>> >>>>
>>>>
>>>> OK, thanks.
>>>>
>>>> I really have to push that patch in.
>>>>
>>>>
>>>> On Mon, Oct 7, 2013 at 12:17 PM, Silvio Di
>>>> gregorio
>>>> <silvio.digregorio@gmail.com
>>>> <ma...@gmail.com>
>>>> >
>>>> <mailto:silvio.digregorio@__gm**ail.com <http://gmail.com>
>>>> <ma...@gmail.com>
>>>> >>
>>>> <mailto:silvio.digregorio@
>>>> <mailto:silvio.digregorio@>__g**m__ail.com <http://gm__ail.com><
>>>> http://gmail.com>
>>>>
>>>>
>>>> <mailto:silvio.digregorio@__gm**ail.com <http://gmail.com>
>>>> <ma...@gmail.com>>>>>
>>>> wrote:
>>>>
>>>> yes i do,
>>>> i have seen this in your post in:
>>>> http://www.mail-archive.com/__**__user@giraph.apache.org/____**
>>>> msg00957.html<http://www.mail-archive.com/____user@giraph.apache.org/____msg00957.html>
>>>> <http://www.mail-archive.com/_**_user@giraph.apache.org/__**
>>>> msg00957.html<http://www.mail-archive.com/__user@giraph.apache.org/__msg00957.html>
>>>> >
>>>>
>>>>
>>>> <http://www.mail-archive.com/_**_user@giraph.apache.org/__**
>>>> msg00957.html<http://www.mail-archive.com/__user@giraph.apache.org/__msg00957.html>
>>>> <http://www.mail-archive.com/**user@giraph.apache.org/**
>>>> msg00957.html<http://www.mail-archive.com/user@giraph.apache.org/msg00957.html>
>>>> >>
>>>>
>>>> excuse me if i had checked in the
>>>> mail-achive
>>>> first I would
>>>> have avoided the last post.
>>>>
>>>> Now zk issue are resolved.
>>>>
>>>>
>>>> 2013/10/7 Claudio Martella
>>>> <claudio.martella@gmail.com <mailto:claudio.martella@**
>>>> gmail.com <cl...@gmail.com>>
>>>> <mailto:claudio.martella@__gma**il.com <http://gmail.com>
>>>> <ma...@gmail.com>
>>>> >>
>>>> <mailto:claudio.martella@
>>>> <mailto:claudio.martella@>__gm**a__il.com <http://gma__il.com><
>>>> http://gmail.com>
>>>>
>>>>
>>>> <mailto:claudio.martella@__gma**il.com <http://gmail.com>
>>>> <ma...@gmail.com>
>>>> >>>>
>>>>
>>>> Can you try applying this one first?
>>>>
>>>> <<< <
>>>> http://www.mail-archive.com/__**__user@giraph.apache.org/____**
>>>> msg00945/check.diff<http://www.mail-archive.com/____user@giraph.apache.org/____msg00945/check.diff>
>>>> <http://www.mail-archive.com/_**_user@giraph.apache.org/__**
>>>> msg00945/check.diff<http://www.mail-archive.com/__user@giraph.apache.org/__msg00945/check.diff>
>>>> >
>>>>
>>>>
>>>> <http://www.mail-archive.com/_**_user@giraph.apache.org/__**
>>>> msg00945/check.diff<http://www.mail-archive.com/__user@giraph.apache.org/__msg00945/check.diff>
>>>> <http://www.mail-archive.com/**user@giraph.apache.org/**
>>>> msg00945/check.diff<http://www.mail-archive.com/user@giraph.apache.org/msg00945/check.diff>
>>>> >><<<<
>>>>
>>>>
>>>> On Mon, Oct 7, 2013 at 8:40 AM, Silvio
>>>> Di gregorio
>>>> <silvio.digregorio@gmail.com
>>>> <ma...@gmail.com>
>>>> >
>>>> <mailto:silvio.digregorio@__gm**ail.com <http://gmail.com>
>>>> <ma...@gmail.com>
>>>> >>
>>>> <mailto:silvio.digregorio@
>>>> <mailto:silvio.digregorio@>__g**m__ail.com <http://gm__ail.com><
>>>> http://gmail.com>
>>>>
>>>>
>>>> <mailto:silvio.digregorio@__gm**ail.com <http://gmail.com>
>>>> <ma...@gmail.com>>>>>
>>>> wrote:
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> *As i said i have builded*
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> /giraph-examples-1.0.0-for-___**_hadoop-2.0.0-cdh4.1.2-jar-___*
>>>> *_with-dependencies.jar/
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> *for cdh4, successfully. The job
>>>> start to
>>>> monitoring the success rate:*
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> /13/10/07 08:28:45 INFO
>>>> mapred.JobClient:
>>>> map 0% reduce 0%/
>>>>
>>>>
>>>>
>>>>
>>>> *but then*
>>>>
>>>>
>>>>
>>>>
>>>> /Error running child
>>>>
>>>> java.lang.____**IllegalStateException: run:
>>>>
>>>>
>>>> Caught an unrecoverable exception
>>>> java.io.FileNotFoundException:
>>>> File
>>>> _bsp/_defaultZkManagerDir/job_**____201309181636_0678/_**
>>>> zkServer
>>>>
>>>>
>>>> does not exist.
>>>> /
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> /.../
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> /Caused by:
>>>> java.lang.RuntimeException:
>>>> java.io.FileNotFoundException: File
>>>>
>>>> _bsp/_defaultZkManagerDir/job_**____201309181636_0678/_**zkServer
>>>> does
>>>>
>>>>
>>>> not exist./
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> 2013/10/5 Silvio Di gregorio
>>>> <silvio.digregorio@gmail.com
>>>> <ma...@gmail.com>
>>>> >
>>>> <mailto:silvio.digregorio@__gm**ail.com <http://gmail.com>
>>>> <ma...@gmail.com>
>>>> >>
>>>> <mailto:silvio.digregorio@
>>>> <mailto:silvio.digregorio@>__g**m__ail.com <http://gm__ail.com><
>>>> http://gmail.com>
>>>>
>>>>
>>>> <mailto:silvio.digregorio@__gm**ail.com <http://gmail.com>
>>>> <ma...@gmail.com>
>>>> >>>>
>>>>
>>>> I ha ve build w/
>>>> hadoop_cdh4.1.2 parameter.
>>>> Something is changed, monday i
>>>> report the
>>>> result. Now the farm is
>>>> closed.
>>>>
>>>> Il giorno 05/ott/2013 14:06,
>>>> "Claudio
>>>> Martella"
>>>> <claudio.martella@gmail.com <mailto:claudio.martella@**
>>>> gmail.com <cl...@gmail.com>>
>>>> <mailto:claudio.martella@__gma**il.com <http://gmail.com>
>>>> <ma...@gmail.com>
>>>> >>
>>>> <mailto:claudio.martella@
>>>> <mailto:claudio.martella@>__gm**a__il.com <http://gma__il.com><
>>>> http://gmail.com>
>>>>
>>>>
>>>> <mailto:claudio.martella@__gma**il.com <http://gmail.com>
>>>> <ma...@gmail.com>>>>>
>>>> ha scritto:
>>>>
>>>> Oh, right, -vof is in
>>>> trunk. Anyway
>>>> it looks
>>>> like you built giraph for
>>>> the wrong
>>>> profile.
>>>> You mentioned you're
>>>> running on
>>>> 2.0, but
>>>> your giraph is built for
>>>> 0.20.203 <tel:0.20.203>
>>>> <tel:0.20.203 <tel:0.20.203>>
>>>> <tel:0.20.203 <tel:0.20.203> <tel:0.20.203 <tel:0.20.203>>>.
>>>> try
>>>>
>>>> building with a profile
>>>>
>>>> for your hadoop version.
>>>>
>>>>
>>>> On Fri, Oct 4, 2013 at
>>>> 2:35 PM,
>>>> Silvio Di
>>>> gregorio
>>>> <silvio.digregorio@gmail.com
>>>> <ma...@gmail.com>
>>>> >
>>>> <mailto:silvio.digregorio@__gm**ail.com <http://gmail.com>
>>>> <ma...@gmail.com>
>>>> >>
>>>> <mailto:silvio.digregorio@
>>>> <mailto:silvio.digregorio@>__g**m__ail.com <http://gm__ail.com><
>>>> http://gmail.com>
>>>>
>>>> <mailto:silvio.digregorio@__gm**ail.com <http://gmail.com>
>>>> <ma...@gmail.com>>>>>
>>>> wrote:
>>>>
>>>>
>>>>
>>>> org.apache.commons.cli.____**
>>>> UnrecognizedOptionException:
>>>>
>>>>
>>>> Unrecognized option:
>>>> -vof
>>>>
>>>> in 1.0 version is
>>>> -of,--outputFormat
>>>> <arg>
>>>> Vertex output format
>>>> -op,--outputPath
>>>> <arg>
>>>> Vertex output path
>>>>
>>>>
>>>>
>>>> 2013/10/4 Claudio
>>>> Martella
>>>> <claudio.martella@gmail.com <mailto:claudio.martella@**
>>>> gmail.com <cl...@gmail.com>>
>>>> <mailto:claudio.martella@__gma**il.com <http://gmail.com>
>>>> <ma...@gmail.com>
>>>> >>
>>>> <mailto:claudio.martella@
>>>> <mailto:claudio.martella@>__gm**a__il.com <http://gma__il.com><
>>>> http://gmail.com>
>>>>
>>>>
>>>> <mailto:claudio.martella@__gma**il.com <http://gmail.com>
>>>> <ma...@gmail.com>
>>>> >>>>
>>>>
>>>> did you try the
>>>> argument
>>>> (-vof) i
>>>> suggested?
>>>>
>>>>
>>>> On Fri, Oct 4,
>>>> 2013 at 2:13 PM,
>>>> Silvio Di gregorio
>>>> <silvio.digregorio@gmail.com
>>>> <ma...@gmail.com>
>>>> >
>>>> <mailto:silvio.digregorio@__gm**ail.com <http://gmail.com>
>>>> <ma...@gmail.com>
>>>> >>
>>>> <mailto:silvio.digregorio@
>>>> <mailto:silvio.digregorio@>__g**m__ail.com <http://gm__ail.com><
>>>> http://gmail.com>
>>>>
>>>>
>>>> <mailto:silvio.digregorio@__gm**ail.com <http://gmail.com>
>>>> <ma...@gmail.com>>>>>
>>>> wrote:
>>>>
>>>>
>>>> i've specified
>>>> -of
>>>>
>>>>
>>>> org.apache.giraph.io.formats._**___IdWithValueTextOutputFormat
>>>>
>>>>
>>>>
>>>> but the same
>>>> error was
>>>> produced
>>>>
>>>> Exception in
>>>> thread "main"
>>>>
>>>> java.lang.____**IncompatibleClassChangeError:
>>>> Found
>>>> interface
>>>>
>>>> org.apache.hadoop.mapreduce.__**__JobContext,
>>>>
>>>>
>>>> but class was
>>>> expected
>>>> at
>>>>
>>>>
>>>> org.apache.giraph.bsp.____**BspOutputFormat.____**
>>>> checkOutputSpecs(____**BspOutputFormat.java:43)
>>>> at
>>>>
>>>>
>>>> org.apache.hadoop.mapred.____**JobClient$2.run(JobClient.____**
>>>> java:984)
>>>> at
>>>>
>>>>
>>>> org.apache.hadoop.mapred.____**JobClient$2.run(JobClient.____**
>>>> java:945)
>>>> at
>>>>
>>>> java.security.____**AccessController.doPrivileged(**
>>>> ____Native
>>>> Method)
>>>> at
>>>>
>>>> javax.security.auth.Subject.__**
>>>> __doAs(Subject.java:396)
>>>> at
>>>>
>>>>
>>>> org.apache.hadoop.security.___**_UserGroupInformation.doAs(___*
>>>> *_UserGroupInformation.java:___**_1408)
>>>> at
>>>>
>>>>
>>>> org.apache.hadoop.mapred.____**JobClient.submitJobInternal(__**
>>>> __JobClient.java:945)
>>>> at
>>>>
>>>> org.apache.hadoop.mapreduce.__**
>>>> __Job.submit(Job.java:566)
>>>> at
>>>>
>>>>
>>>> org.apache.hadoop.mapreduce.__**__Job.waitForCompletion(Job.__*
>>>> *__java:596)
>>>> at
>>>>
>>>>
>>>> org.apache.giraph.job.____**GiraphJob.run(GiraphJob.java:_**
>>>> ___237)
>>>> at
>>>>
>>>>
>>>> org.apache.giraph.____**GiraphRunner.run(GiraphRunner.**
>>>> ____java:94)
>>>> at
>>>>
>>>>
>>>> org.apache.hadoop.util.____**ToolRunner.run(ToolRunner.____**
>>>> java:70)
>>>> at
>>>>
>>>>
>>>> org.apache.hadoop.util.____**ToolRunner.run(ToolRunner.____**
>>>> java:84)
>>>> at
>>>>
>>>>
>>>> org.apache.giraph.____**GiraphRunner.main(____**
>>>> GiraphRunner.java:124)
>>>> at
>>>>
>>>> sun.reflect.____**NativeMethodAccessorImpl.____**
>>>> invoke0(Native
>>>> Method)
>>>> at
>>>>
>>>>
>>>> sun.reflect.____**NativeMethodAccessorImpl.____**invoke(____**
>>>> NativeMethodAccessorImpl.java:**____39)
>>>> at
>>>>
>>>>
>>>> sun.reflect.____**DelegatingMethodAccessorImpl._**
>>>> ___invoke(____**DelegatingMethodAccessorImpl._**___java:25)
>>>> at
>>>>
>>>> java.lang.reflect.Method.____**invoke(Method.java:597)
>>>> at
>>>>
>>>> org.apache.hadoop.util.RunJar.**
>>>> ____main(RunJar.java:208)
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> 2013/10/4
>>>> Claudio Martella
>>>> <claudio.martella@gmail.com <mailto:claudio.martella@**
>>>> gmail.com <cl...@gmail.com>>
>>>> <mailto:claudio.martella@__gma**il.com <http://gmail.com>
>>>> <ma...@gmail.com>
>>>> >>
>>>> <mailto:claudio.martella@
>>>> <mailto:claudio.martella@>__gm**a__il.com <http://gma__il.com><
>>>> http://gmail.com>
>>>>
>>>>
>>>> <mailto:claudio.martella@__gma**il.com <http://gmail.com>
>>>> <ma...@gmail.com>
>>>> >>>>
>>>>
>>>> Hi,
>>>>
>>>> you need
>>>> to specify the
>>>> vertex
>>>> outputformat
>>>> class
>>>> (-vof
>>>> option), e.g.
>>>>
>>>>
>>>> org.apache.giraph.io.formats._**___**
>>>> IdWithValueTextOutputFormat.
>>>>
>>>>
>>>>
>>>>
>>>> On Fri,
>>>> Oct 4, 2013
>>>> at 1:06
>>>> PM, Silvio
>>>> Di gregorio
>>>> <silvio.digregorio@gmail.com
>>>> <ma...@gmail.com>
>>>> >
>>>> <mailto:silvio.digregorio@__gm**ail.com <http://gmail.com>
>>>> <ma...@gmail.com>
>>>> >>
>>>> <mailto:silvio.digregorio@
>>>> <mailto:silvio.digregorio@>__g**m__ail.com <http://gm__ail.com><
>>>> http://gmail.com>
>>>>
>>>>
>>>> <mailto:silvio.digregorio@__gm**ail.com <http://gmail.com>
>>>> <ma...@gmail.com>
>>>> >>>>
>>>> wrote:
>>>>
>>>>
>>>> Hi,
>>>>
>>>> I hope
>>>> I have
>>>> sent to
>>>> the
>>>> right address.
>>>>
>>>> i have
>>>> a graph
>>>> (directed
>>>> and
>>>> unweighted)
>>>> stored
>>>> in
>>>> hdfs like a
>>>> adjacency
>>>> list
>>>> (140Milions of
>>>> edges
>>>> 6Milions
>>>> of vertex)
>>>>
>>>>
>>>> node<tab>neighbors
>>>>
>>>> 23 2
>>>> 1 3
>>>> 4 3
>>>>
>>>> 1 999 99923 <tel:1%20999%2099923> <tel:1%20999%2099923>
>>>> <tel:1%20999%2099923>
>>>> 909
>>>> .......
>>>>
>>>> ......
>>>>
>>>> hadoop
>>>> version
>>>> Hadoop
>>>>
>>>> 2.0.0-cdh4.3.0
>>>> - java 1.6
>>>>
>>>>
>>>> I have
>>>> executed the
>>>>
>>>> giraph-1.0
>>>> connected
>>>>
>>>> components
>>>> example, in
>>>> this
>>>> fashion
>>>>
>>>> > hadoop jar
>>>>
>>>>
>>>> /usr/local/giraph/giraph-____**examples/target/giraph-____**
>>>> examples-1.0.0-for-hadoop-0.20.203.0
>>>>
>>>> <tel:0.20.203.0>
>>>> <tel:0.20.203.0 <tel:0.20.203.0>>
>>>> <tel:0.20.203.0 <tel:0.20.203.0> <tel:0.20.203.0
>>>> <tel:0.20.203.0>>>-jar-with-**dependencies.____jar
>>>>
>>>> org.apache.giraph.GiraphRunner
>>>>
>>>> org.apache.giraph.examples.___**
>>>> _ConnectedComponentsVertex
>>>>
>>>> -vif
>>>>
>>>> org.apache.giraph.io.formats._**
>>>> ___IntIntNullTextInputFormat
>>>>
>>>>
>>>>
>>>> -vip
>>>>
>>>> /user/hdfs/lista_adj_txt
>>>>
>>>> -op
>>>> connectedgiraph
>>>>
>>>> --workers 4
>>>>
>>>> and
>>>> then fail with:
>>>>
>>>>
>>>> 13/10/04
>>>> 09:28:29 INFO
>>>>
>>>> utils.ConfigurationUtils: No
>>>> edge
>>>> input format
>>>>
>>>> specified.
>>>> Ensure your
>>>>
>>>> InputFormat
>>>> does not
>>>>
>>>> require one.
>>>>
>>>>
>>>> 13/10/04
>>>> 09:28:29 INFO
>>>>
>>>> utils.ConfigurationUtils: No
>>>> output
>>>> format
>>>> specified.
>>>> Ensure
>>>> your
>>>> OutputFormat
>>>> does
>>>> not
>>>> require one.
>>>>
>>>>
>>>> 13/10/04
>>>> 09:28:30 INFO
>>>>
>>>> job.GiraphJob: run:
>>>> Since
>>>> checkpointing is
>>>>
>>>> disabled
>>>> (default), do
>>>> not
>>>> allow any task
>>>>
>>>> retries (setting
>>>>
>>>> mapred.map.max.attempts
>>>> = 0,
>>>> old value = 4)
>>>>
>>>>
>>>> 13/10/04
>>>> 09:28:31 WARN
>>>>
>>>> mapred.JobClient: Use
>>>>
>>>> GenericOptionsParser for
>>>>
>>>> parsing the
>>>> arguments.
>>>>
>>>> Applications should
>>>>
>>>> implement Tool
>>>> for the same.
>>>>
>>>>
>>>> 13/10/04
>>>> 09:28:31 INFO
>>>>
>>>> mapred.JobClient:
>>>>
>>>> Cleaning up the
>>>> staging
>>>> area
>>>>
>>>> hdfs://srv-bigdata-dev-01.int
>>>> <http://srv-bigdata-dev-01.int**>.____sose.it:8020/user/hdfs/.*
>>>> *____staging/job_201309181636_**0535<http://sose.it:8020/user/hdfs/.____staging/job_201309181636_0535>
>>>> <http://sose.it:8020/user/**hdfs/.__staging/job_**
>>>> 201309181636_0535<http://sose.it:8020/user/hdfs/.__staging/job_201309181636_0535>
>>>> >
>>>>
>>>> <http://srv-bigdata-dev-01.__i**nt.sose.it:8020/user/hdfs/.__**
>>>> staging/job_201309181636_0535<http://int.sose.it:8020/user/hdfs/.__staging/job_201309181636_0535>
>>>> <http://srv-bigdata-dev-01.**int.sose.it:8020/user/hdfs/.**
>>>> staging/job_201309181636_0535<http://srv-bigdata-dev-01.int.sose.it:8020/user/hdfs/.staging/job_201309181636_0535>
>>>> >**>
>>>> <http://srv-bigdata-dev-01.__i**__nt.sose.it:8020/user/hdfs/._
>>>> **___staging/job_201309181636_**0535<http://i__nt.sose.it:8020/user/hdfs/.____staging/job_201309181636_0535>
>>>> <http://int.sose.it:8020/user/**hdfs/.__staging/job_**
>>>> 201309181636_0535<http://int.sose.it:8020/user/hdfs/.__staging/job_201309181636_0535>
>>>> >
>>>>
>>>> <http://srv-bigdata-dev-01.__i**nt.sose.it:8020/user/hdfs/.__**
>>>> staging/job_201309181636_0535<http://int.sose.it:8020/user/hdfs/.__staging/job_201309181636_0535>
>>>> <http://srv-bigdata-dev-01.**int.sose.it:8020/user/hdfs/.**
>>>> staging/job_201309181636_0535<http://srv-bigdata-dev-01.int.sose.it:8020/user/hdfs/.staging/job_201309181636_0535>
>>>> >**>__>
>>>>
>>>>
>>>> Exception in thread
>>>> "main"
>>>>
>>>> java.lang.____**IncompatibleClassChangeError:
>>>> Found
>>>> interface
>>>>
>>>> org.apache.hadoop.mapreduce.__**__JobContext,
>>>>
>>>>
>>>> but
>>>> class was
>>>> expected
>>>>
>>>>
>>>> at
>>>>
>>>>
>>>> org.apache.giraph.bsp.____**BspOutputFormat.____**
>>>> checkOutputSpecs(____**BspOutputFormat.java:43)
>>>>
>>>>
>>>>
>>>> ......
>>>>
>>>> Thanks
>>>> in advance
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>>
>>>> Claudio Martella
>>>> claudio.martella@gmail.com <ma...@gmail.com>
>>>> >
>>>> <mailto:claudio.martella@__gma**il.com <http://gmail.com>
>>>> <ma...@gmail.com>
>>>> >>
>>>> <mailto:claudio.martella@
>>>> <mailto:claudio.martella@>__gm**a__il.com <http://gma__il.com><
>>>> http://gmail.com>
>>>>
>>>>
>>>> <mailto:claudio.martella@__gma**il.com <http://gmail.com>
>>>> <ma...@gmail.com>
>>>> >>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Claudio
>>>> Martella
>>>> claudio.martella@gmail.com <ma...@gmail.com>
>>>> >
>>>> <mailto:claudio.martella@__gma**il.com <http://gmail.com>
>>>> <ma...@gmail.com>
>>>> >>
>>>> <mailto:claudio.martella@
>>>> <mailto:claudio.martella@>__gm**a__il.com <http://gma__il.com><
>>>> http://gmail.com>
>>>>
>>>>
>>>> <mailto:claudio.martella@__gma**il.com <http://gmail.com>
>>>> <ma...@gmail.com>
>>>> >>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Claudio Martella
>>>> claudio.martella@gmail.com <ma...@gmail.com>
>>>> >
>>>> <mailto:claudio.martella@__gma**il.com <http://gmail.com>
>>>> <ma...@gmail.com>
>>>> >>
>>>> <mailto:claudio.martella@
>>>> <mailto:claudio.martella@>__gm**a__il.com <http://gma__il.com><
>>>> http://gmail.com>
>>>>
>>>>
>>>> <mailto:claudio.martella@__gma**il.com <http://gmail.com>
>>>> <ma...@gmail.com>
>>>> >>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Claudio Martella
>>>> claudio.martella@gmail.com <ma...@gmail.com>
>>>> >
>>>> <mailto:claudio.martella@__gma**il.com <http://gmail.com>
>>>> <ma...@gmail.com>
>>>> >>
>>>> <mailto:claudio.martella@
>>>> <mailto:claudio.martella@>__gm**a__il.com <http://gma__il.com><
>>>> http://gmail.com>
>>>>
>>>>
>>>> <mailto:claudio.martella@__gma**il.com <http://gmail.com>
>>>> <ma...@gmail.com>
>>>> >>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Claudio Martella
>>>> claudio.martella@gmail.com <ma...@gmail.com>
>>>> >
>>>> <mailto:claudio.martella@__gma**il.com <http://gmail.com>
>>>> <ma...@gmail.com>
>>>> >>
>>>> <mailto:claudio.martella@
>>>> <mailto:claudio.martella@>__gm**a__il.com <http://gma__il.com><
>>>> http://gmail.com>
>>>>
>>>>
>>>> <mailto:claudio.martella@__gma**il.com <http://gmail.com>
>>>> <ma...@gmail.com>
>>>> >>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Claudio Martella
>>>> claudio.martella@gmail.com <ma...@gmail.com>
>>>> >
>>>> <mailto:claudio.martella@__gma**il.com <http://gmail.com>
>>>> <ma...@gmail.com>
>>>> >>
>>>> <mailto:claudio.martella@
>>>> <mailto:claudio.martella@>__gm**a__il.com <http://gma__il.com><
>>>> http://gmail.com>
>>>>
>>>>
>>>> <mailto:claudio.martella@__gma**il.com <http://gmail.com>
>>>> <ma...@gmail.com>
>>>> >>>
>>>>
>>>>
>>>> --
>>>> Matthew Laird
>>>> Lead Software Developer, Bioinformatics
>>>> Brinkman Laboratory
>>>> Simon Fraser University, Burnaby, BC, Canada
>>>>
>>>>
>>>> --
>>>> Matthew Laird
>>>> Lead Software Developer, Bioinformatics
>>>> Brinkman Laboratory
>>>> Simon Fraser University, Burnaby, BC, Canada
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Claudio Martella
>>>> claudio.martella@gmail.com <ma...@gmail.com>
>>>> >
>>>>
>>>
>>> --
>>> Matthew Laird
>>> Lead Software Developer, Bioinformatics
>>> Brinkman Laboratory
>>> Simon Fraser University, Burnaby, BC, Canada
>>>
>>
>>
>
--
Claudio Martella
claudio.martella@gmail.com
Re: connected components example in giraph 1.0
Posted by Silvio Di gregorio <si...@gmail.com>.
I have increased at 4GiB the heap space and the job has been terminated
successfully;
In the result file i've seen a strange thing, i have seen, like min vertex
id, "ZERO", that is:
vertex_id<tab>[min vertex id namely label of connected components]
12 0
22 2
23 2
but i don't have a vertex w/ named ZERO.
Has ZERO a special meaning? [for example not assigned etc ....]
PS: i use giraph 1.0, CDH4.3 w/ MP1 0.20.203 not YARN i'm sorry.
2013/10/7 Silvio Di gregorio <si...@gmail.com>
> with a working day in my shoulders and my children around i gave you
> random version,
> tomorrow morning in farm i'll be more precise.
> though i have hadoop 2.0 i use MR1 not YARN.
> I'm sorry again.
>
>
> 2013/10/7 Matthew Laird <la...@sfu.ca>
>
>> Hmm, it seems there have been some recent pushes that have broken trunk
>> as I discovered yesterday. :)
>>
>> But I guess that brings me back to my original question, to build against
>> 2.0.5 using the MR interface rather than YARN as Milinda suggested, what
>> profile would I use? Would I let it think it's 0.20xxx or tell it 2.0.5,
>> or something else?
>>
>> Thanks.
>>
>>
>> On 13-10-07 11:50 AM, Claudio Martella wrote:
>>
>>> trunk is 1.1.0-SNAPSHOT. I believe the date refers to the last update to
>>> the site.
>>>
>>>
>>> On Mon, Oct 7, 2013 at 8:48 PM, Matthew Laird <lairdm@sfu.ca
>>> <ma...@sfu.ca>> wrote:
>>>
>>> Thanks, which is my next curiosity... if I head over to
>>> http://giraph.apache.org/ I see at the top is says
>>>
>>> "Version: 1.1.0-SNAPSHOT | Last Published: 2013-08-10"
>>>
>>> However it's not clickable and the only download links I see are for
>>> 1.0.0. I'm assuming trunk in git isn't the same as the 1.1.0
>>> snapshot since more pushes have occurred since August, and I don't
>>> see a tag in git for 1.1.0
>>> (https://git-wip-us.apache.__**org/repos/asf?p=giraph.git
>>> <https://git-wip-us.apache.**org/repos/asf?p=giraph.git<https://git-wip-us.apache.org/repos/asf?p=giraph.git>>).
>>> So where
>>>
>>> does one get 1.1.0-SNAPSHOT from?
>>>
>>> Thanks.
>>>
>>>
>>> On 13-10-07 11:39 AM, Silvio Di gregorio wrote:
>>>
>>> Hi Matthew
>>> I use giraph 1.1.0 and cdh4 hadoop 0.23
>>>
>>> the patch of Marco:
>>>
>>> http://www.mail-archive.com/__**__user@giraph.apache.org/____**
>>> msg00945/check.diff<http://www.mail-archive.com/____user@giraph.apache.org/____msg00945/check.diff>
>>> <http://www.mail-archive.com/_**_user@giraph.apache.org/__**
>>> msg00945/check.diff<http://www.mail-archive.com/__user@giraph.apache.org/__msg00945/check.diff>
>>> >
>>>
>>>
>>> <http://www.mail-archive.com/_**_user@giraph.apache.org/__**
>>> msg00945/check.diff<http://www.mail-archive.com/__user@giraph.apache.org/__msg00945/check.diff>
>>> <http://www.mail-archive.com/**user@giraph.apache.org/**
>>> msg00945/check.diff<http://www.mail-archive.com/user@giraph.apache.org/msg00945/check.diff>
>>> >>
>>>
>>> and build
>>>
>>> mvn clean install -P hadoop_cdh4.1.2
>>>
>>>
>>>
>>> Il giorno 07/ott/2013 20:20, "Matthew Laird" <lairdm@sfu.ca
>>> <ma...@sfu.ca>
>>> <mailto:lairdm@sfu.ca <ma...@sfu.ca>>> ha scritto:
>>>
>>>
>>>
>>> Hello, so I'm watching this thread with interest as well,
>>> since
>>> that's the exact example I'll be trying to run as well, and
>>> I see
>>> Silvio appears to be asking about running Giraph 1.0
>>> against Hadoop
>>> 2.0.0.
>>>
>>> I'm curious, is that a known working combination, 1.0 vs
>>> 2.0.x? And
>>> if so, are there patches to apply first and what maven
>>> command line
>>> should I build with? (ie. will a -Dhadoop.version=2.0.5
>>> give the
>>> needed MR based API or will it use YARN?) I figure it's
>>> better to
>>> ask than aimlessly try all the combinations. :)
>>>
>>> Thanks!
>>>
>>> On 13-10-07 11:01 AM, Claudio Martella wrote:
>>>
>>> try to increase the mappers' heap space through
>>> mapred.child.java.opts.
>>>
>>>
>>> On Mon, Oct 7, 2013 at 1:18 PM, Silvio Di gregorio
>>> <silvio.digregorio@gmail.com <mailto:silvio.digregorio@**
>>> gmail.com <si...@gmail.com>>
>>> <mailto:silvio.digregorio@__gm**ail.com <http://gmail.com>
>>> <ma...@gmail.com>
>>> >>
>>> <mailto:silvio.digregorio@
>>> <mailto:silvio.digregorio@>__g**m__ail.com <http://gm__ail.com><
>>> http://gmail.com>
>>>
>>>
>>> <mailto:silvio.digregorio@__gm**ail.com <http://gmail.com>
>>> <ma...@gmail.com>>>>>
>>> wrote:
>>>
>>>
>>> I think now my issue is another, thanks to all.
>>>
>>>
>>>
>>>
>>> java.lang.____**IllegalStateException: run:
>>> Caught an
>>>
>>>
>>> unrecoverable exception waitFor: ExecutionException
>>> occurred
>>> while waiting for
>>>
>>> org.apache.giraph.utils.____**ProgressableUtils$____**
>>> FutureWaitable@73a01e8a
>>> at
>>>
>>> org.apache.giraph.graph.____**GraphMapper.run(GraphMapper.__**
>>> __java:102)
>>>
>>>
>>>
>>> ......
>>> Caused by: java.lang.____**IllegalStateException:
>>>
>>> waitFor:
>>>
>>> ExecutionException occurred while waiting for
>>>
>>> org.apache.giraph.utils.____**ProgressableUtils$____**
>>> FutureWaitable@73a01e8a
>>> at
>>>
>>> org.apache.giraph.utils.____**ProgressableUtils.waitFor(____**
>>> ProgressableUtils.java:151)
>>>
>>>
>>>
>>>
>>> ... 7 more
>>> Caused by:
>>> java.util.concurrent.____**ExecutionException:
>>> java.lang.OutOfMemoryError:*__**__Java heap space*
>>> at
>>>
>>> org.apache.giraph.utils.____**ProgressableUtils.waitFor(____**
>>> ProgressableUtils.java:143)
>>>
>>>
>>>
>>>
>>>
>>> ... 13 more
>>>
>>>
>>>
>>> 2013/10/7 Claudio Martella
>>> <claudio.martella@gmail.com <ma...@gmail.com>
>>> >
>>> <mailto:claudio.martella@__gma**il.com <http://gmail.com>
>>> <ma...@gmail.com>
>>> >>
>>> <mailto:claudio.martella@
>>> <mailto:claudio.martella@>__gm**a__il.com <http://gma__il.com> <
>>> http://gmail.com>
>>>
>>>
>>> <mailto:claudio.martella@__gma**il.com <http://gmail.com>
>>> <ma...@gmail.com>
>>> >>>>
>>>
>>> OK, thanks.
>>>
>>> I really have to push that patch in.
>>>
>>>
>>> On Mon, Oct 7, 2013 at 12:17 PM, Silvio Di
>>> gregorio
>>> <silvio.digregorio@gmail.com
>>> <ma...@gmail.com>
>>> >
>>> <mailto:silvio.digregorio@__gm**ail.com <http://gmail.com>
>>> <ma...@gmail.com>
>>> >>
>>> <mailto:silvio.digregorio@
>>> <mailto:silvio.digregorio@>__g**m__ail.com <http://gm__ail.com><
>>> http://gmail.com>
>>>
>>>
>>> <mailto:silvio.digregorio@__gm**ail.com <http://gmail.com>
>>> <ma...@gmail.com>>>>>
>>> wrote:
>>>
>>> yes i do,
>>> i have seen this in your post in:
>>> http://www.mail-archive.com/__**__user@giraph.apache.org/____**
>>> msg00957.html<http://www.mail-archive.com/____user@giraph.apache.org/____msg00957.html>
>>> <http://www.mail-archive.com/_**_user@giraph.apache.org/__**
>>> msg00957.html<http://www.mail-archive.com/__user@giraph.apache.org/__msg00957.html>
>>> >
>>>
>>>
>>> <http://www.mail-archive.com/_**_user@giraph.apache.org/__**
>>> msg00957.html<http://www.mail-archive.com/__user@giraph.apache.org/__msg00957.html>
>>> <http://www.mail-archive.com/**user@giraph.apache.org/**
>>> msg00957.html<http://www.mail-archive.com/user@giraph.apache.org/msg00957.html>
>>> >>
>>>
>>> excuse me if i had checked in the
>>> mail-achive
>>> first I would
>>> have avoided the last post.
>>>
>>> Now zk issue are resolved.
>>>
>>>
>>> 2013/10/7 Claudio Martella
>>> <claudio.martella@gmail.com <ma...@gmail.com>
>>> >
>>> <mailto:claudio.martella@__gma**il.com <http://gmail.com>
>>> <ma...@gmail.com>
>>> >>
>>> <mailto:claudio.martella@
>>> <mailto:claudio.martella@>__gm**a__il.com <http://gma__il.com> <
>>> http://gmail.com>
>>>
>>>
>>> <mailto:claudio.martella@__gma**il.com <http://gmail.com>
>>> <ma...@gmail.com>
>>> >>>>
>>>
>>> Can you try applying this one first?
>>>
>>> <<< <
>>> http://www.mail-archive.com/__**__user@giraph.apache.org/____**
>>> msg00945/check.diff<http://www.mail-archive.com/____user@giraph.apache.org/____msg00945/check.diff>
>>> <http://www.mail-archive.com/_**_user@giraph.apache.org/__**
>>> msg00945/check.diff<http://www.mail-archive.com/__user@giraph.apache.org/__msg00945/check.diff>
>>> >
>>>
>>>
>>> <http://www.mail-archive.com/_**_user@giraph.apache.org/__**
>>> msg00945/check.diff<http://www.mail-archive.com/__user@giraph.apache.org/__msg00945/check.diff>
>>> <http://www.mail-archive.com/**user@giraph.apache.org/**
>>> msg00945/check.diff<http://www.mail-archive.com/user@giraph.apache.org/msg00945/check.diff>
>>> >><<<<
>>>
>>>
>>> On Mon, Oct 7, 2013 at 8:40 AM, Silvio
>>> Di gregorio
>>> <silvio.digregorio@gmail.com
>>> <ma...@gmail.com>
>>> >
>>> <mailto:silvio.digregorio@__gm**ail.com <http://gmail.com>
>>> <ma...@gmail.com>
>>> >>
>>> <mailto:silvio.digregorio@
>>> <mailto:silvio.digregorio@>__g**m__ail.com <http://gm__ail.com><
>>> http://gmail.com>
>>>
>>>
>>> <mailto:silvio.digregorio@__gm**ail.com <http://gmail.com>
>>> <ma...@gmail.com>>>>>
>>> wrote:
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> *As i said i have builded*
>>>
>>>
>>>
>>>
>>>
>>>
>>> /giraph-examples-1.0.0-for-___**_hadoop-2.0.0-cdh4.1.2-jar-___**
>>> _with-dependencies.jar/
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> *for cdh4, successfully. The job
>>> start to
>>> monitoring the success rate:*
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> /13/10/07 08:28:45 INFO
>>> mapred.JobClient:
>>> map 0% reduce 0%/
>>>
>>>
>>>
>>>
>>> *but then*
>>>
>>>
>>>
>>>
>>> /Error running child
>>>
>>> java.lang.____**IllegalStateException: run:
>>>
>>>
>>> Caught an unrecoverable exception
>>> java.io.FileNotFoundException:
>>> File
>>> _bsp/_defaultZkManagerDir/job_**____201309181636_0678/_**
>>> zkServer
>>>
>>>
>>> does not exist.
>>> /
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> /.../
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> /Caused by:
>>> java.lang.RuntimeException:
>>> java.io.FileNotFoundException: File
>>>
>>> _bsp/_defaultZkManagerDir/job_**____201309181636_0678/_**zkServer
>>> does
>>>
>>>
>>> not exist./
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> 2013/10/5 Silvio Di gregorio
>>> <silvio.digregorio@gmail.com
>>> <ma...@gmail.com>
>>> >
>>> <mailto:silvio.digregorio@__gm**ail.com <http://gmail.com>
>>> <ma...@gmail.com>
>>> >>
>>> <mailto:silvio.digregorio@
>>> <mailto:silvio.digregorio@>__g**m__ail.com <http://gm__ail.com><
>>> http://gmail.com>
>>>
>>>
>>> <mailto:silvio.digregorio@__gm**ail.com <http://gmail.com>
>>> <ma...@gmail.com>
>>> >>>>
>>>
>>> I ha ve build w/
>>> hadoop_cdh4.1.2 parameter.
>>> Something is changed, monday i
>>> report the
>>> result. Now the farm is closed.
>>>
>>> Il giorno 05/ott/2013 14:06,
>>> "Claudio
>>> Martella"
>>> <claudio.martella@gmail.com <ma...@gmail.com>
>>> >
>>> <mailto:claudio.martella@__gma**il.com <http://gmail.com>
>>> <ma...@gmail.com>
>>> >>
>>> <mailto:claudio.martella@
>>> <mailto:claudio.martella@>__gm**a__il.com <http://gma__il.com> <
>>> http://gmail.com>
>>>
>>>
>>> <mailto:claudio.martella@__gma**il.com <http://gmail.com>
>>> <ma...@gmail.com>>>>>
>>> ha scritto:
>>>
>>> Oh, right, -vof is in
>>> trunk. Anyway
>>> it looks
>>> like you built giraph for
>>> the wrong
>>> profile.
>>> You mentioned you're
>>> running on
>>> 2.0, but
>>> your giraph is built for
>>> 0.20.203 <tel:0.20.203>
>>> <tel:0.20.203 <tel:0.20.203>>
>>> <tel:0.20.203 <tel:0.20.203> <tel:0.20.203 <tel:0.20.203>>>. try
>>>
>>> building with a profile
>>>
>>> for your hadoop version.
>>>
>>>
>>> On Fri, Oct 4, 2013 at
>>> 2:35 PM,
>>> Silvio Di
>>> gregorio
>>> <silvio.digregorio@gmail.com
>>> <ma...@gmail.com>
>>> >
>>> <mailto:silvio.digregorio@__gm**ail.com <http://gmail.com>
>>> <ma...@gmail.com>
>>> >>
>>> <mailto:silvio.digregorio@
>>> <mailto:silvio.digregorio@>__g**m__ail.com <http://gm__ail.com><
>>> http://gmail.com>
>>>
>>> <mailto:silvio.digregorio@__gm**ail.com <http://gmail.com>
>>> <ma...@gmail.com>>>>>
>>> wrote:
>>>
>>>
>>>
>>> org.apache.commons.cli.____**
>>> UnrecognizedOptionException:
>>>
>>>
>>> Unrecognized option:
>>> -vof
>>>
>>> in 1.0 version is
>>> -of,--outputFormat
>>> <arg>
>>> Vertex output format
>>> -op,--outputPath
>>> <arg>
>>> Vertex output path
>>>
>>>
>>>
>>> 2013/10/4 Claudio
>>> Martella
>>> <claudio.martella@gmail.com <ma...@gmail.com>
>>> >
>>> <mailto:claudio.martella@__gma**il.com <http://gmail.com>
>>> <ma...@gmail.com>
>>> >>
>>> <mailto:claudio.martella@
>>> <mailto:claudio.martella@>__gm**a__il.com <http://gma__il.com> <
>>> http://gmail.com>
>>>
>>>
>>> <mailto:claudio.martella@__gma**il.com <http://gmail.com>
>>> <ma...@gmail.com>
>>> >>>>
>>>
>>> did you try the
>>> argument
>>> (-vof) i
>>> suggested?
>>>
>>>
>>> On Fri, Oct 4,
>>> 2013 at 2:13 PM,
>>> Silvio Di gregorio
>>> <silvio.digregorio@gmail.com
>>> <ma...@gmail.com>
>>> >
>>> <mailto:silvio.digregorio@__gm**ail.com <http://gmail.com>
>>> <ma...@gmail.com>
>>> >>
>>> <mailto:silvio.digregorio@
>>> <mailto:silvio.digregorio@>__g**m__ail.com <http://gm__ail.com><
>>> http://gmail.com>
>>>
>>>
>>> <mailto:silvio.digregorio@__gm**ail.com <http://gmail.com>
>>> <ma...@gmail.com>>>>>
>>> wrote:
>>>
>>>
>>> i've specified
>>> -of
>>>
>>>
>>> org.apache.giraph.io.formats._**___IdWithValueTextOutputFormat
>>>
>>>
>>>
>>> but the same
>>> error was
>>> produced
>>>
>>> Exception in
>>> thread "main"
>>>
>>> java.lang.____**IncompatibleClassChangeError:
>>> Found interface
>>>
>>> org.apache.hadoop.mapreduce.__**__JobContext,
>>>
>>>
>>> but class was
>>> expected
>>> at
>>>
>>>
>>> org.apache.giraph.bsp.____**BspOutputFormat.____**
>>> checkOutputSpecs(____**BspOutputFormat.java:43)
>>> at
>>>
>>>
>>> org.apache.hadoop.mapred.____**JobClient$2.run(JobClient.____**
>>> java:984)
>>> at
>>>
>>>
>>> org.apache.hadoop.mapred.____**JobClient$2.run(JobClient.____**
>>> java:945)
>>> at
>>>
>>> java.security.____**AccessController.doPrivileged(**
>>> ____Native
>>> Method)
>>> at
>>>
>>> javax.security.auth.Subject.__**
>>> __doAs(Subject.java:396)
>>> at
>>>
>>>
>>> org.apache.hadoop.security.___**_UserGroupInformation.doAs(___**
>>> _UserGroupInformation.java:___**_1408)
>>> at
>>>
>>>
>>> org.apache.hadoop.mapred.____**JobClient.submitJobInternal(__**
>>> __JobClient.java:945)
>>> at
>>>
>>> org.apache.hadoop.mapreduce.__**
>>> __Job.submit(Job.java:566)
>>> at
>>>
>>>
>>> org.apache.hadoop.mapreduce.__**__Job.waitForCompletion(Job.__**
>>> __java:596)
>>> at
>>>
>>>
>>> org.apache.giraph.job.____**GiraphJob.run(GiraphJob.java:_**
>>> ___237)
>>> at
>>>
>>>
>>> org.apache.giraph.____**GiraphRunner.run(GiraphRunner.**
>>> ____java:94)
>>> at
>>>
>>>
>>> org.apache.hadoop.util.____**ToolRunner.run(ToolRunner.____**
>>> java:70)
>>> at
>>>
>>>
>>> org.apache.hadoop.util.____**ToolRunner.run(ToolRunner.____**
>>> java:84)
>>> at
>>>
>>>
>>> org.apache.giraph.____**GiraphRunner.main(____**
>>> GiraphRunner.java:124)
>>> at
>>>
>>> sun.reflect.____**NativeMethodAccessorImpl.____**
>>> invoke0(Native
>>> Method)
>>> at
>>>
>>>
>>> sun.reflect.____**NativeMethodAccessorImpl.____**invoke(____**
>>> NativeMethodAccessorImpl.java:**____39)
>>> at
>>>
>>>
>>> sun.reflect.____**DelegatingMethodAccessorImpl._**___invoke(____
>>> **DelegatingMethodAccessorImpl._**___java:25)
>>> at
>>>
>>> java.lang.reflect.Method.____**invoke(Method.java:597)
>>> at
>>>
>>> org.apache.hadoop.util.RunJar.**
>>> ____main(RunJar.java:208)
>>>
>>>
>>>
>>>
>>>
>>> 2013/10/4
>>> Claudio Martella
>>> <claudio.martella@gmail.com <ma...@gmail.com>
>>> >
>>> <mailto:claudio.martella@__gma**il.com <http://gmail.com>
>>> <ma...@gmail.com>
>>> >>
>>> <mailto:claudio.martella@
>>> <mailto:claudio.martella@>__gm**a__il.com <http://gma__il.com> <
>>> http://gmail.com>
>>>
>>>
>>> <mailto:claudio.martella@__gma**il.com <http://gmail.com>
>>> <ma...@gmail.com>
>>> >>>>
>>>
>>> Hi,
>>>
>>> you need
>>> to specify the
>>> vertex
>>> outputformat
>>> class
>>> (-vof
>>> option), e.g.
>>>
>>>
>>> org.apache.giraph.io.formats._**___**
>>> IdWithValueTextOutputFormat.
>>>
>>>
>>>
>>>
>>> On Fri,
>>> Oct 4, 2013
>>> at 1:06
>>> PM, Silvio
>>> Di gregorio
>>> <silvio.digregorio@gmail.com
>>> <ma...@gmail.com>
>>> >
>>> <mailto:silvio.digregorio@__gm**ail.com <http://gmail.com>
>>> <ma...@gmail.com>
>>> >>
>>> <mailto:silvio.digregorio@
>>> <mailto:silvio.digregorio@>__g**m__ail.com <http://gm__ail.com><
>>> http://gmail.com>
>>>
>>>
>>> <mailto:silvio.digregorio@__gm**ail.com <http://gmail.com>
>>> <ma...@gmail.com>
>>> >>>>
>>> wrote:
>>>
>>>
>>> Hi,
>>>
>>> I hope
>>> I have
>>> sent to
>>> the
>>> right address.
>>>
>>> i have
>>> a graph
>>> (directed
>>> and
>>> unweighted)
>>> stored
>>> in
>>> hdfs like a
>>> adjacency
>>> list
>>> (140Milions of
>>> edges
>>> 6Milions
>>> of vertex)
>>>
>>>
>>> node<tab>neighbors
>>>
>>> 23 2
>>> 1 3
>>> 4 3
>>>
>>> 1 999 99923 <tel:1%20999%2099923> <tel:1%20999%2099923>
>>> <tel:1%20999%2099923>
>>> 909
>>> .......
>>>
>>> ......
>>>
>>> hadoop
>>> version
>>> Hadoop
>>>
>>> 2.0.0-cdh4.3.0
>>> - java 1.6
>>>
>>>
>>> I have
>>> executed the
>>>
>>> giraph-1.0
>>> connected
>>>
>>> components
>>> example, in
>>> this
>>> fashion
>>>
>>> > hadoop jar
>>>
>>>
>>> /usr/local/giraph/giraph-____**examples/target/giraph-____**
>>> examples-1.0.0-for-hadoop-0.20.203.0
>>>
>>> <tel:0.20.203.0>
>>> <tel:0.20.203.0 <tel:0.20.203.0>>
>>> <tel:0.20.203.0 <tel:0.20.203.0> <tel:0.20.203.0
>>> <tel:0.20.203.0>>>-jar-with-**dependencies.____jar
>>>
>>> org.apache.giraph.GiraphRunner
>>>
>>> org.apache.giraph.examples.___**
>>> _ConnectedComponentsVertex
>>>
>>> -vif
>>>
>>> org.apache.giraph.io.formats._**
>>> ___IntIntNullTextInputFormat
>>>
>>>
>>>
>>> -vip
>>>
>>> /user/hdfs/lista_adj_txt
>>> -op
>>> connectedgiraph
>>>
>>> --workers 4
>>>
>>> and
>>> then fail with:
>>>
>>>
>>> 13/10/04
>>> 09:28:29 INFO
>>>
>>> utils.ConfigurationUtils: No
>>> edge
>>> input format
>>>
>>> specified.
>>> Ensure your
>>>
>>> InputFormat
>>> does not
>>>
>>> require one.
>>>
>>>
>>> 13/10/04
>>> 09:28:29 INFO
>>>
>>> utils.ConfigurationUtils: No
>>> output
>>> format
>>> specified.
>>> Ensure
>>> your
>>> OutputFormat
>>> does
>>> not
>>> require one.
>>>
>>>
>>> 13/10/04
>>> 09:28:30 INFO
>>>
>>> job.GiraphJob: run:
>>> Since
>>> checkpointing is
>>>
>>> disabled
>>> (default), do
>>> not
>>> allow any task
>>>
>>> retries (setting
>>>
>>> mapred.map.max.attempts
>>> = 0,
>>> old value = 4)
>>>
>>>
>>> 13/10/04
>>> 09:28:31 WARN
>>>
>>> mapred.JobClient: Use
>>>
>>> GenericOptionsParser for
>>>
>>> parsing the
>>> arguments.
>>>
>>> Applications should
>>>
>>> implement Tool
>>> for the same.
>>>
>>>
>>> 13/10/04
>>> 09:28:31 INFO
>>>
>>> mapred.JobClient:
>>>
>>> Cleaning up the
>>> staging
>>> area
>>>
>>> hdfs://srv-bigdata-dev-01.int
>>> <http://srv-bigdata-dev-01.int**>.____sose.it:8020/user/hdfs/.**
>>> ____staging/job_201309181636_**0535<http://sose.it:8020/user/hdfs/.____staging/job_201309181636_0535>
>>> <http://sose.it:8020/user/**hdfs/.__staging/job_**
>>> 201309181636_0535<http://sose.it:8020/user/hdfs/.__staging/job_201309181636_0535>
>>> >
>>>
>>> <http://srv-bigdata-dev-01.__i**nt.sose.it:8020/user/hdfs/.__**
>>> staging/job_201309181636_0535<http://int.sose.it:8020/user/hdfs/.__staging/job_201309181636_0535>
>>> <http://srv-bigdata-dev-01.**int.sose.it:8020/user/hdfs/.**
>>> staging/job_201309181636_0535<http://srv-bigdata-dev-01.int.sose.it:8020/user/hdfs/.staging/job_201309181636_0535>
>>> >**>
>>> <http://srv-bigdata-dev-01.__i**__nt.sose.it:8020/user/hdfs/._*
>>> *___staging/job_201309181636_**0535<http://i__nt.sose.it:8020/user/hdfs/.____staging/job_201309181636_0535>
>>> <http://int.sose.it:8020/user/**hdfs/.__staging/job_**
>>> 201309181636_0535<http://int.sose.it:8020/user/hdfs/.__staging/job_201309181636_0535>
>>> >
>>>
>>> <http://srv-bigdata-dev-01.__i**nt.sose.it:8020/user/hdfs/.__**
>>> staging/job_201309181636_0535<http://int.sose.it:8020/user/hdfs/.__staging/job_201309181636_0535>
>>> <http://srv-bigdata-dev-01.**int.sose.it:8020/user/hdfs/.**
>>> staging/job_201309181636_0535<http://srv-bigdata-dev-01.int.sose.it:8020/user/hdfs/.staging/job_201309181636_0535>
>>> >**>__>
>>>
>>>
>>> Exception in thread
>>> "main"
>>>
>>> java.lang.____**IncompatibleClassChangeError:
>>> Found
>>> interface
>>>
>>> org.apache.hadoop.mapreduce.__**__JobContext,
>>>
>>>
>>> but
>>> class was
>>> expected
>>>
>>>
>>> at
>>>
>>>
>>> org.apache.giraph.bsp.____**BspOutputFormat.____**
>>> checkOutputSpecs(____**BspOutputFormat.java:43)
>>>
>>>
>>>
>>> ......
>>>
>>> Thanks
>>> in advance
>>>
>>>
>>>
>>>
>>>
>>> --
>>>
>>> Claudio Martella
>>> claudio.martella@gmail.com <ma...@gmail.com>
>>> >
>>> <mailto:claudio.martella@__gma**il.com <http://gmail.com>
>>> <ma...@gmail.com>
>>> >>
>>> <mailto:claudio.martella@
>>> <mailto:claudio.martella@>__gm**a__il.com <http://gma__il.com> <
>>> http://gmail.com>
>>>
>>>
>>> <mailto:claudio.martella@__gma**il.com <http://gmail.com>
>>> <ma...@gmail.com>
>>> >>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> --
>>> Claudio
>>> Martella
>>> claudio.martella@gmail.com <ma...@gmail.com>
>>> >
>>> <mailto:claudio.martella@__gma**il.com <http://gmail.com>
>>> <ma...@gmail.com>
>>> >>
>>> <mailto:claudio.martella@
>>> <mailto:claudio.martella@>__gm**a__il.com <http://gma__il.com> <
>>> http://gmail.com>
>>>
>>>
>>> <mailto:claudio.martella@__gma**il.com <http://gmail.com>
>>> <ma...@gmail.com>
>>> >>>
>>>
>>>
>>>
>>>
>>>
>>> --
>>> Claudio Martella
>>> claudio.martella@gmail.com <ma...@gmail.com>
>>> >
>>> <mailto:claudio.martella@__gma**il.com <http://gmail.com>
>>> <ma...@gmail.com>
>>> >>
>>> <mailto:claudio.martella@
>>> <mailto:claudio.martella@>__gm**a__il.com <http://gma__il.com> <
>>> http://gmail.com>
>>>
>>>
>>> <mailto:claudio.martella@__gma**il.com <http://gmail.com>
>>> <ma...@gmail.com>
>>> >>>
>>>
>>>
>>>
>>>
>>>
>>> --
>>> Claudio Martella
>>> claudio.martella@gmail.com <ma...@gmail.com>
>>> >
>>> <mailto:claudio.martella@__gma**il.com <http://gmail.com>
>>> <ma...@gmail.com>
>>> >>
>>> <mailto:claudio.martella@
>>> <mailto:claudio.martella@>__gm**a__il.com <http://gma__il.com> <
>>> http://gmail.com>
>>>
>>>
>>> <mailto:claudio.martella@__gma**il.com <http://gmail.com>
>>> <ma...@gmail.com>
>>> >>>
>>>
>>>
>>>
>>>
>>>
>>> --
>>> Claudio Martella
>>> claudio.martella@gmail.com <ma...@gmail.com>
>>> >
>>> <mailto:claudio.martella@__gma**il.com <http://gmail.com>
>>> <ma...@gmail.com>
>>> >>
>>> <mailto:claudio.martella@
>>> <mailto:claudio.martella@>__gm**a__il.com <http://gma__il.com> <
>>> http://gmail.com>
>>>
>>>
>>> <mailto:claudio.martella@__gma**il.com <http://gmail.com>
>>> <ma...@gmail.com>
>>> >>>
>>>
>>>
>>>
>>>
>>>
>>> --
>>> Claudio Martella
>>> claudio.martella@gmail.com <ma...@gmail.com>
>>> >
>>> <mailto:claudio.martella@__gma**il.com <http://gmail.com>
>>> <ma...@gmail.com>
>>> >>
>>> <mailto:claudio.martella@
>>> <mailto:claudio.martella@>__gm**a__il.com <http://gma__il.com> <
>>> http://gmail.com>
>>>
>>>
>>> <mailto:claudio.martella@__gma**il.com <http://gmail.com>
>>> <ma...@gmail.com>
>>> >>>
>>>
>>>
>>> --
>>> Matthew Laird
>>> Lead Software Developer, Bioinformatics
>>> Brinkman Laboratory
>>> Simon Fraser University, Burnaby, BC, Canada
>>>
>>>
>>> --
>>> Matthew Laird
>>> Lead Software Developer, Bioinformatics
>>> Brinkman Laboratory
>>> Simon Fraser University, Burnaby, BC, Canada
>>>
>>>
>>>
>>>
>>> --
>>> Claudio Martella
>>> claudio.martella@gmail.com <ma...@gmail.com>
>>> >
>>>
>>
>> --
>> Matthew Laird
>> Lead Software Developer, Bioinformatics
>> Brinkman Laboratory
>> Simon Fraser University, Burnaby, BC, Canada
>>
>
>
Re: connected components example in giraph 1.0
Posted by Silvio Di gregorio <si...@gmail.com>.
with a working day in my shoulders and my children around i gave you
random version,
tomorrow morning in farm i'll be more precise.
though i have hadoop 2.0 i use MR1 not YARN.
I'm sorry again.
2013/10/7 Matthew Laird <la...@sfu.ca>
> Hmm, it seems there have been some recent pushes that have broken trunk as
> I discovered yesterday. :)
>
> But I guess that brings me back to my original question, to build against
> 2.0.5 using the MR interface rather than YARN as Milinda suggested, what
> profile would I use? Would I let it think it's 0.20xxx or tell it 2.0.5,
> or something else?
>
> Thanks.
>
>
> On 13-10-07 11:50 AM, Claudio Martella wrote:
>
>> trunk is 1.1.0-SNAPSHOT. I believe the date refers to the last update to
>> the site.
>>
>>
>> On Mon, Oct 7, 2013 at 8:48 PM, Matthew Laird <lairdm@sfu.ca
>> <ma...@sfu.ca>> wrote:
>>
>> Thanks, which is my next curiosity... if I head over to
>> http://giraph.apache.org/ I see at the top is says
>>
>> "Version: 1.1.0-SNAPSHOT | Last Published: 2013-08-10"
>>
>> However it's not clickable and the only download links I see are for
>> 1.0.0. I'm assuming trunk in git isn't the same as the 1.1.0
>> snapshot since more pushes have occurred since August, and I don't
>> see a tag in git for 1.1.0
>> (https://git-wip-us.apache.__**org/repos/asf?p=giraph.git
>> <https://git-wip-us.apache.**org/repos/asf?p=giraph.git<https://git-wip-us.apache.org/repos/asf?p=giraph.git>>).
>> So where
>>
>> does one get 1.1.0-SNAPSHOT from?
>>
>> Thanks.
>>
>>
>> On 13-10-07 11:39 AM, Silvio Di gregorio wrote:
>>
>> Hi Matthew
>> I use giraph 1.1.0 and cdh4 hadoop 0.23
>>
>> the patch of Marco:
>>
>> http://www.mail-archive.com/__**__user@giraph.apache.org/____**
>> msg00945/check.diff<http://www.mail-archive.com/____user@giraph.apache.org/____msg00945/check.diff>
>> <http://www.mail-archive.com/_**_user@giraph.apache.org/__**
>> msg00945/check.diff<http://www.mail-archive.com/__user@giraph.apache.org/__msg00945/check.diff>
>> >
>>
>>
>> <http://www.mail-archive.com/_**_user@giraph.apache.org/__**
>> msg00945/check.diff<http://www.mail-archive.com/__user@giraph.apache.org/__msg00945/check.diff>
>> <http://www.mail-archive.com/**user@giraph.apache.org/**
>> msg00945/check.diff<http://www.mail-archive.com/user@giraph.apache.org/msg00945/check.diff>
>> >>
>>
>> and build
>>
>> mvn clean install -P hadoop_cdh4.1.2
>>
>>
>>
>> Il giorno 07/ott/2013 20:20, "Matthew Laird" <lairdm@sfu.ca
>> <ma...@sfu.ca>
>> <mailto:lairdm@sfu.ca <ma...@sfu.ca>>> ha scritto:
>>
>>
>>
>> Hello, so I'm watching this thread with interest as well,
>> since
>> that's the exact example I'll be trying to run as well, and
>> I see
>> Silvio appears to be asking about running Giraph 1.0
>> against Hadoop
>> 2.0.0.
>>
>> I'm curious, is that a known working combination, 1.0 vs
>> 2.0.x? And
>> if so, are there patches to apply first and what maven
>> command line
>> should I build with? (ie. will a -Dhadoop.version=2.0.5
>> give the
>> needed MR based API or will it use YARN?) I figure it's
>> better to
>> ask than aimlessly try all the combinations. :)
>>
>> Thanks!
>>
>> On 13-10-07 11:01 AM, Claudio Martella wrote:
>>
>> try to increase the mappers' heap space through
>> mapred.child.java.opts.
>>
>>
>> On Mon, Oct 7, 2013 at 1:18 PM, Silvio Di gregorio
>> <silvio.digregorio@gmail.com <mailto:silvio.digregorio@**
>> gmail.com <si...@gmail.com>>
>> <mailto:silvio.digregorio@__gm**ail.com <http://gmail.com>
>> <ma...@gmail.com>
>> >>
>> <mailto:silvio.digregorio@
>> <mailto:silvio.digregorio@>__g**m__ail.com <http://gm__ail.com> <
>> http://gmail.com>
>>
>>
>> <mailto:silvio.digregorio@__gm**ail.com <http://gmail.com>
>> <ma...@gmail.com>>>>>
>> wrote:
>>
>>
>> I think now my issue is another, thanks to all.
>>
>>
>>
>>
>> java.lang.____**IllegalStateException: run: Caught
>> an
>>
>>
>> unrecoverable exception waitFor: ExecutionException
>> occurred
>> while waiting for
>>
>> org.apache.giraph.utils.____**ProgressableUtils$____**
>> FutureWaitable@73a01e8a
>> at
>>
>> org.apache.giraph.graph.____**GraphMapper.run(GraphMapper.__**
>> __java:102)
>>
>>
>>
>> ......
>> Caused by: java.lang.____**IllegalStateException:
>>
>> waitFor:
>>
>> ExecutionException occurred while waiting for
>>
>> org.apache.giraph.utils.____**ProgressableUtils$____**
>> FutureWaitable@73a01e8a
>> at
>>
>> org.apache.giraph.utils.____**ProgressableUtils.waitFor(____**
>> ProgressableUtils.java:151)
>>
>>
>>
>>
>> ... 7 more
>> Caused by:
>> java.util.concurrent.____**ExecutionException:
>> java.lang.OutOfMemoryError:*__**__Java heap space*
>> at
>>
>> org.apache.giraph.utils.____**ProgressableUtils.waitFor(____**
>> ProgressableUtils.java:143)
>>
>>
>>
>>
>>
>> ... 13 more
>>
>>
>>
>> 2013/10/7 Claudio Martella
>> <claudio.martella@gmail.com <ma...@gmail.com>
>> >
>> <mailto:claudio.martella@__gma**il.com <http://gmail.com>
>> <mailto:claudio.martella@**gmail.com <cl...@gmail.com>
>> >>
>> <mailto:claudio.martella@
>> <mailto:claudio.martella@>__gm**a__il.com <http://gma__il.com> <
>> http://gmail.com>
>>
>>
>> <mailto:claudio.martella@__gma**il.com <http://gmail.com>
>> <mailto:claudio.martella@**gmail.com <cl...@gmail.com>
>> >>>>
>>
>> OK, thanks.
>>
>> I really have to push that patch in.
>>
>>
>> On Mon, Oct 7, 2013 at 12:17 PM, Silvio Di
>> gregorio
>> <silvio.digregorio@gmail.com
>> <ma...@gmail.com>
>> >
>> <mailto:silvio.digregorio@__gm**ail.com <http://gmail.com>
>> <ma...@gmail.com>
>> >>
>> <mailto:silvio.digregorio@
>> <mailto:silvio.digregorio@>__g**m__ail.com <http://gm__ail.com> <
>> http://gmail.com>
>>
>>
>> <mailto:silvio.digregorio@__gm**ail.com <http://gmail.com>
>> <ma...@gmail.com>>>>>
>> wrote:
>>
>> yes i do,
>> i have seen this in your post in:
>> http://www.mail-archive.com/__**__user@giraph.apache.org/____**
>> msg00957.html<http://www.mail-archive.com/____user@giraph.apache.org/____msg00957.html>
>> <http://www.mail-archive.com/_**_user@giraph.apache.org/__**
>> msg00957.html<http://www.mail-archive.com/__user@giraph.apache.org/__msg00957.html>
>> >
>>
>>
>> <http://www.mail-archive.com/_**_user@giraph.apache.org/__**
>> msg00957.html<http://www.mail-archive.com/__user@giraph.apache.org/__msg00957.html>
>> <http://www.mail-archive.com/**user@giraph.apache.org/**
>> msg00957.html<http://www.mail-archive.com/user@giraph.apache.org/msg00957.html>
>> >>
>>
>> excuse me if i had checked in the
>> mail-achive
>> first I would
>> have avoided the last post.
>>
>> Now zk issue are resolved.
>>
>>
>> 2013/10/7 Claudio Martella
>> <claudio.martella@gmail.com <ma...@gmail.com>
>> >
>> <mailto:claudio.martella@__gma**il.com <http://gmail.com>
>> <mailto:claudio.martella@**gmail.com <cl...@gmail.com>
>> >>
>> <mailto:claudio.martella@
>> <mailto:claudio.martella@>__gm**a__il.com <http://gma__il.com> <
>> http://gmail.com>
>>
>>
>> <mailto:claudio.martella@__gma**il.com <http://gmail.com>
>> <mailto:claudio.martella@**gmail.com <cl...@gmail.com>
>> >>>>
>>
>> Can you try applying this one first?
>>
>> <<< <
>> http://www.mail-archive.com/__**__user@giraph.apache.org/____**
>> msg00945/check.diff<http://www.mail-archive.com/____user@giraph.apache.org/____msg00945/check.diff>
>> <http://www.mail-archive.com/_**_user@giraph.apache.org/__**
>> msg00945/check.diff<http://www.mail-archive.com/__user@giraph.apache.org/__msg00945/check.diff>
>> >
>>
>>
>> <http://www.mail-archive.com/_**_user@giraph.apache.org/__**
>> msg00945/check.diff<http://www.mail-archive.com/__user@giraph.apache.org/__msg00945/check.diff>
>> <http://www.mail-archive.com/**user@giraph.apache.org/**
>> msg00945/check.diff<http://www.mail-archive.com/user@giraph.apache.org/msg00945/check.diff>
>> >><<<<
>>
>>
>> On Mon, Oct 7, 2013 at 8:40 AM, Silvio
>> Di gregorio
>> <silvio.digregorio@gmail.com
>> <ma...@gmail.com>
>> >
>> <mailto:silvio.digregorio@__gm**ail.com <http://gmail.com>
>> <ma...@gmail.com>
>> >>
>> <mailto:silvio.digregorio@
>> <mailto:silvio.digregorio@>__g**m__ail.com <http://gm__ail.com> <
>> http://gmail.com>
>>
>>
>> <mailto:silvio.digregorio@__gm**ail.com <http://gmail.com>
>> <ma...@gmail.com>>>>>
>> wrote:
>>
>>
>>
>>
>>
>>
>>
>> *As i said i have builded*
>>
>>
>>
>>
>>
>>
>> /giraph-examples-1.0.0-for-___**_hadoop-2.0.0-cdh4.1.2-jar-___**
>> _with-dependencies.jar/
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> *for cdh4, successfully. The job
>> start to
>> monitoring the success rate:*
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> /13/10/07 08:28:45 INFO
>> mapred.JobClient:
>> map 0% reduce 0%/
>>
>>
>>
>>
>> *but then*
>>
>>
>>
>>
>> /Error running child
>>
>> java.lang.____**IllegalStateException: run:
>>
>>
>> Caught an unrecoverable exception
>> java.io.FileNotFoundException:
>> File
>> _bsp/_defaultZkManagerDir/job_**____201309181636_0678/_**zkServer
>>
>>
>> does not exist.
>> /
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> /.../
>>
>>
>>
>>
>>
>>
>>
>> /Caused by:
>> java.lang.RuntimeException:
>> java.io.FileNotFoundException: File
>>
>> _bsp/_defaultZkManagerDir/job_**____201309181636_0678/_**zkServer
>> does
>>
>>
>> not exist./
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> 2013/10/5 Silvio Di gregorio
>> <silvio.digregorio@gmail.com
>> <ma...@gmail.com>
>> >
>> <mailto:silvio.digregorio@__gm**ail.com <http://gmail.com>
>> <ma...@gmail.com>
>> >>
>> <mailto:silvio.digregorio@
>> <mailto:silvio.digregorio@>__g**m__ail.com <http://gm__ail.com> <
>> http://gmail.com>
>>
>>
>> <mailto:silvio.digregorio@__gm**ail.com <http://gmail.com>
>> <ma...@gmail.com>
>> >>>>
>>
>> I ha ve build w/
>> hadoop_cdh4.1.2 parameter.
>> Something is changed, monday i
>> report the
>> result. Now the farm is closed.
>>
>> Il giorno 05/ott/2013 14:06,
>> "Claudio
>> Martella"
>> <claudio.martella@gmail.com <ma...@gmail.com>
>> >
>> <mailto:claudio.martella@__gma**il.com <http://gmail.com>
>> <mailto:claudio.martella@**gmail.com <cl...@gmail.com>
>> >>
>> <mailto:claudio.martella@
>> <mailto:claudio.martella@>__gm**a__il.com <http://gma__il.com> <
>> http://gmail.com>
>>
>>
>> <mailto:claudio.martella@__gma**il.com <http://gmail.com>
>> <mailto:claudio.martella@**gmail.com <cl...@gmail.com>>>>>
>> ha scritto:
>>
>> Oh, right, -vof is in
>> trunk. Anyway
>> it looks
>> like you built giraph for
>> the wrong
>> profile.
>> You mentioned you're
>> running on
>> 2.0, but
>> your giraph is built for
>> 0.20.203 <tel:0.20.203>
>> <tel:0.20.203 <tel:0.20.203>>
>> <tel:0.20.203 <tel:0.20.203> <tel:0.20.203 <tel:0.20.203>>>. try
>>
>> building with a profile
>>
>> for your hadoop version.
>>
>>
>> On Fri, Oct 4, 2013 at
>> 2:35 PM,
>> Silvio Di
>> gregorio
>> <silvio.digregorio@gmail.com
>> <ma...@gmail.com>
>> >
>> <mailto:silvio.digregorio@__gm**ail.com <http://gmail.com>
>> <ma...@gmail.com>
>> >>
>> <mailto:silvio.digregorio@
>> <mailto:silvio.digregorio@>__g**m__ail.com <http://gm__ail.com> <
>> http://gmail.com>
>>
>> <mailto:silvio.digregorio@__gm**ail.com <http://gmail.com>
>> <ma...@gmail.com>>>>>
>> wrote:
>>
>>
>>
>> org.apache.commons.cli.____**
>> UnrecognizedOptionException:
>>
>>
>> Unrecognized option:
>> -vof
>>
>> in 1.0 version is
>> -of,--outputFormat
>> <arg>
>> Vertex output format
>> -op,--outputPath <arg>
>> Vertex output path
>>
>>
>>
>> 2013/10/4 Claudio
>> Martella
>> <claudio.martella@gmail.com <ma...@gmail.com>
>> >
>> <mailto:claudio.martella@__gma**il.com <http://gmail.com>
>> <mailto:claudio.martella@**gmail.com <cl...@gmail.com>
>> >>
>> <mailto:claudio.martella@
>> <mailto:claudio.martella@>__gm**a__il.com <http://gma__il.com> <
>> http://gmail.com>
>>
>>
>> <mailto:claudio.martella@__gma**il.com <http://gmail.com>
>> <mailto:claudio.martella@**gmail.com <cl...@gmail.com>
>> >>>>
>>
>> did you try the
>> argument
>> (-vof) i
>> suggested?
>>
>>
>> On Fri, Oct 4,
>> 2013 at 2:13 PM,
>> Silvio Di gregorio
>> <silvio.digregorio@gmail.com
>> <ma...@gmail.com>
>> >
>> <mailto:silvio.digregorio@__gm**ail.com <http://gmail.com>
>> <ma...@gmail.com>
>> >>
>> <mailto:silvio.digregorio@
>> <mailto:silvio.digregorio@>__g**m__ail.com <http://gm__ail.com> <
>> http://gmail.com>
>>
>>
>> <mailto:silvio.digregorio@__gm**ail.com <http://gmail.com>
>> <ma...@gmail.com>>>>>
>> wrote:
>>
>>
>> i've specified
>> -of
>>
>>
>> org.apache.giraph.io.formats._**___IdWithValueTextOutputFormat
>>
>>
>>
>> but the same
>> error was
>> produced
>>
>> Exception in
>> thread "main"
>>
>> java.lang.____**IncompatibleClassChangeError:
>> Found interface
>>
>> org.apache.hadoop.mapreduce.__**__JobContext,
>>
>>
>> but class was
>> expected
>> at
>>
>>
>> org.apache.giraph.bsp.____**BspOutputFormat.____**
>> checkOutputSpecs(____**BspOutputFormat.java:43)
>> at
>>
>>
>> org.apache.hadoop.mapred.____**JobClient$2.run(JobClient.____**
>> java:984)
>> at
>>
>>
>> org.apache.hadoop.mapred.____**JobClient$2.run(JobClient.____**
>> java:945)
>> at
>>
>> java.security.____**AccessController.doPrivileged(**
>> ____Native
>> Method)
>> at
>>
>> javax.security.auth.Subject.__**__doAs(Subject.java:396)
>> at
>>
>>
>> org.apache.hadoop.security.___**_UserGroupInformation.doAs(___**
>> _UserGroupInformation.java:___**_1408)
>> at
>>
>>
>> org.apache.hadoop.mapred.____**JobClient.submitJobInternal(__**
>> __JobClient.java:945)
>> at
>>
>> org.apache.hadoop.mapreduce.__**
>> __Job.submit(Job.java:566)
>> at
>>
>>
>> org.apache.hadoop.mapreduce.__**__Job.waitForCompletion(Job.__**
>> __java:596)
>> at
>>
>>
>> org.apache.giraph.job.____**GiraphJob.run(GiraphJob.java:_**
>> ___237)
>> at
>>
>>
>> org.apache.giraph.____**GiraphRunner.run(GiraphRunner.**
>> ____java:94)
>> at
>>
>>
>> org.apache.hadoop.util.____**ToolRunner.run(ToolRunner.____**
>> java:70)
>> at
>>
>>
>> org.apache.hadoop.util.____**ToolRunner.run(ToolRunner.____**
>> java:84)
>> at
>>
>>
>> org.apache.giraph.____**GiraphRunner.main(____**
>> GiraphRunner.java:124)
>> at
>>
>> sun.reflect.____**NativeMethodAccessorImpl.____**
>> invoke0(Native
>> Method)
>> at
>>
>>
>> sun.reflect.____**NativeMethodAccessorImpl.____**invoke(____**
>> NativeMethodAccessorImpl.java:**____39)
>> at
>>
>>
>> sun.reflect.____**DelegatingMethodAccessorImpl._**___invoke(____*
>> *DelegatingMethodAccessorImpl._**___java:25)
>> at
>>
>> java.lang.reflect.Method.____**invoke(Method.java:597)
>> at
>>
>> org.apache.hadoop.util.RunJar.**
>> ____main(RunJar.java:208)
>>
>>
>>
>>
>>
>> 2013/10/4
>> Claudio Martella
>> <claudio.martella@gmail.com <ma...@gmail.com>
>> >
>> <mailto:claudio.martella@__gma**il.com <http://gmail.com>
>> <mailto:claudio.martella@**gmail.com <cl...@gmail.com>
>> >>
>> <mailto:claudio.martella@
>> <mailto:claudio.martella@>__gm**a__il.com <http://gma__il.com> <
>> http://gmail.com>
>>
>>
>> <mailto:claudio.martella@__gma**il.com <http://gmail.com>
>> <mailto:claudio.martella@**gmail.com <cl...@gmail.com>
>> >>>>
>>
>> Hi,
>>
>> you need
>> to specify the
>> vertex
>> outputformat
>> class
>> (-vof
>> option), e.g.
>>
>>
>> org.apache.giraph.io.formats._**___**IdWithValueTextOutputFormat.
>>
>>
>>
>>
>> On Fri,
>> Oct 4, 2013
>> at 1:06
>> PM, Silvio
>> Di gregorio
>> <silvio.digregorio@gmail.com
>> <ma...@gmail.com>
>> >
>> <mailto:silvio.digregorio@__gm**ail.com <http://gmail.com>
>> <ma...@gmail.com>
>> >>
>> <mailto:silvio.digregorio@
>> <mailto:silvio.digregorio@>__g**m__ail.com <http://gm__ail.com> <
>> http://gmail.com>
>>
>>
>> <mailto:silvio.digregorio@__gm**ail.com <http://gmail.com>
>> <ma...@gmail.com>
>> >>>>
>> wrote:
>>
>>
>> Hi,
>>
>> I hope
>> I have
>> sent to
>> the
>> right address.
>>
>> i have
>> a graph
>> (directed
>> and
>> unweighted)
>> stored
>> in
>> hdfs like a
>> adjacency
>> list
>> (140Milions of
>> edges
>> 6Milions
>> of vertex)
>>
>>
>> node<tab>neighbors
>>
>> 23 2
>> 1 3
>> 4 3
>>
>> 1 999 99923 <tel:1%20999%2099923> <tel:1%20999%2099923>
>> <tel:1%20999%2099923>
>> 909
>> .......
>>
>> ......
>>
>> hadoop
>> version
>> Hadoop
>>
>> 2.0.0-cdh4.3.0
>> - java 1.6
>>
>>
>> I have
>> executed the
>>
>> giraph-1.0
>> connected
>>
>> components
>> example, in
>> this
>> fashion
>>
>> > hadoop jar
>>
>>
>> /usr/local/giraph/giraph-____**examples/target/giraph-____**
>> examples-1.0.0-for-hadoop-0.20.203.0
>>
>> <tel:0.20.203.0>
>> <tel:0.20.203.0 <tel:0.20.203.0>>
>> <tel:0.20.203.0 <tel:0.20.203.0> <tel:0.20.203.0
>> <tel:0.20.203.0>>>-jar-with-**dependencies.____jar
>>
>> org.apache.giraph.GiraphRunner
>>
>> org.apache.giraph.examples.___**
>> _ConnectedComponentsVertex
>> -vif
>>
>> org.apache.giraph.io.formats._**
>> ___IntIntNullTextInputFormat
>>
>>
>> -vip
>>
>> /user/hdfs/lista_adj_txt
>> -op
>> connectedgiraph
>>
>> --workers 4
>>
>> and
>> then fail with:
>>
>> 13/10/04
>> 09:28:29 INFO
>>
>> utils.ConfigurationUtils: No
>> edge
>> input format
>>
>> specified.
>> Ensure your
>>
>> InputFormat
>> does not
>>
>> require one.
>>
>> 13/10/04
>> 09:28:29 INFO
>>
>> utils.ConfigurationUtils: No
>> output
>> format
>> specified.
>> Ensure
>> your
>> OutputFormat
>> does not
>> require one.
>>
>> 13/10/04
>> 09:28:30 INFO
>>
>> job.GiraphJob: run:
>> Since
>> checkpointing is
>> disabled
>> (default), do
>> not
>> allow any task
>>
>> retries (setting
>>
>> mapred.map.max.attempts
>> = 0,
>> old value = 4)
>>
>> 13/10/04
>> 09:28:31 WARN
>>
>> mapred.JobClient: Use
>>
>> GenericOptionsParser for
>>
>> parsing the
>> arguments.
>>
>> Applications should
>>
>> implement Tool
>> for the same.
>>
>> 13/10/04
>> 09:28:31 INFO
>>
>> mapred.JobClient:
>>
>> Cleaning up the
>> staging
>> area
>>
>> hdfs://srv-bigdata-dev-01.int
>> <http://srv-bigdata-dev-01.int**>.____sose.it:8020/user/hdfs/.**
>> ____staging/job_201309181636_**0535<http://sose.it:8020/user/hdfs/.____staging/job_201309181636_0535>
>> <http://sose.it:8020/user/**hdfs/.__staging/job_**
>> 201309181636_0535<http://sose.it:8020/user/hdfs/.__staging/job_201309181636_0535>
>> >
>>
>> <http://srv-bigdata-dev-01.__i**nt.sose.it:8020/user/hdfs/.__**
>> staging/job_201309181636_0535<http://int.sose.it:8020/user/hdfs/.__staging/job_201309181636_0535>
>> <http://srv-bigdata-dev-01.**int.sose.it:8020/user/hdfs/.**
>> staging/job_201309181636_0535<http://srv-bigdata-dev-01.int.sose.it:8020/user/hdfs/.staging/job_201309181636_0535>
>> >**>
>> <http://srv-bigdata-dev-01.__i**__nt.sose.it:8020/user/hdfs/._**
>> ___staging/job_201309181636_**0535<http://i__nt.sose.it:8020/user/hdfs/.____staging/job_201309181636_0535>
>> <http://int.sose.it:8020/user/**hdfs/.__staging/job_**
>> 201309181636_0535<http://int.sose.it:8020/user/hdfs/.__staging/job_201309181636_0535>
>> >
>>
>> <http://srv-bigdata-dev-01.__i**nt.sose.it:8020/user/hdfs/.__**
>> staging/job_201309181636_0535<http://int.sose.it:8020/user/hdfs/.__staging/job_201309181636_0535>
>> <http://srv-bigdata-dev-01.**int.sose.it:8020/user/hdfs/.**
>> staging/job_201309181636_0535<http://srv-bigdata-dev-01.int.sose.it:8020/user/hdfs/.staging/job_201309181636_0535>
>> >**>__>
>>
>>
>> Exception in thread
>> "main"
>>
>> java.lang.____**IncompatibleClassChangeError:
>> Found
>> interface
>>
>> org.apache.hadoop.mapreduce.__**__JobContext,
>>
>>
>> but
>> class was
>> expected
>>
>>
>> at
>>
>>
>> org.apache.giraph.bsp.____**BspOutputFormat.____**
>> checkOutputSpecs(____**BspOutputFormat.java:43)
>>
>>
>>
>> ......
>>
>> Thanks
>> in advance
>>
>>
>>
>>
>>
>> --
>>
>> Claudio Martella
>> claudio.martella@gmail.com <ma...@gmail.com>
>> >
>> <mailto:claudio.martella@__gma**il.com <http://gmail.com>
>> <mailto:claudio.martella@**gmail.com <cl...@gmail.com>
>> >>
>> <mailto:claudio.martella@
>> <mailto:claudio.martella@>__gm**a__il.com <http://gma__il.com> <
>> http://gmail.com>
>>
>>
>> <mailto:claudio.martella@__gma**il.com <http://gmail.com>
>> <mailto:claudio.martella@**gmail.com <cl...@gmail.com>
>> >>>
>>
>>
>>
>>
>>
>>
>> --
>> Claudio Martella
>> claudio.martella@gmail.com <ma...@gmail.com>
>> >
>> <mailto:claudio.martella@__gma**il.com <http://gmail.com>
>> <mailto:claudio.martella@**gmail.com <cl...@gmail.com>
>> >>
>> <mailto:claudio.martella@
>> <mailto:claudio.martella@>__gm**a__il.com <http://gma__il.com> <
>> http://gmail.com>
>>
>>
>> <mailto:claudio.martella@__gma**il.com <http://gmail.com>
>> <mailto:claudio.martella@**gmail.com <cl...@gmail.com>
>> >>>
>>
>>
>>
>>
>>
>> --
>> Claudio Martella
>> claudio.martella@gmail.com <ma...@gmail.com>
>> >
>> <mailto:claudio.martella@__gma**il.com <http://gmail.com>
>> <mailto:claudio.martella@**gmail.com <cl...@gmail.com>
>> >>
>> <mailto:claudio.martella@
>> <mailto:claudio.martella@>__gm**a__il.com <http://gma__il.com> <
>> http://gmail.com>
>>
>>
>> <mailto:claudio.martella@__gma**il.com <http://gmail.com>
>> <mailto:claudio.martella@**gmail.com <cl...@gmail.com>
>> >>>
>>
>>
>>
>>
>>
>> --
>> Claudio Martella
>> claudio.martella@gmail.com <ma...@gmail.com>
>> >
>> <mailto:claudio.martella@__gma**il.com <http://gmail.com>
>> <mailto:claudio.martella@**gmail.com <cl...@gmail.com>
>> >>
>> <mailto:claudio.martella@
>> <mailto:claudio.martella@>__gm**a__il.com <http://gma__il.com> <
>> http://gmail.com>
>>
>>
>> <mailto:claudio.martella@__gma**il.com <http://gmail.com>
>> <mailto:claudio.martella@**gmail.com <cl...@gmail.com>
>> >>>
>>
>>
>>
>>
>>
>> --
>> Claudio Martella
>> claudio.martella@gmail.com <ma...@gmail.com>
>> >
>> <mailto:claudio.martella@__gma**il.com <http://gmail.com>
>> <mailto:claudio.martella@**gmail.com <cl...@gmail.com>
>> >>
>> <mailto:claudio.martella@
>> <mailto:claudio.martella@>__gm**a__il.com <http://gma__il.com> <
>> http://gmail.com>
>>
>>
>> <mailto:claudio.martella@__gma**il.com <http://gmail.com>
>> <mailto:claudio.martella@**gmail.com <cl...@gmail.com>
>> >>>
>>
>>
>>
>>
>>
>> --
>> Claudio Martella
>> claudio.martella@gmail.com <ma...@gmail.com>
>> >
>> <mailto:claudio.martella@__gma**il.com <http://gmail.com>
>> <mailto:claudio.martella@**gmail.com <cl...@gmail.com>
>> >>
>> <mailto:claudio.martella@
>> <mailto:claudio.martella@>__gm**a__il.com <http://gma__il.com> <
>> http://gmail.com>
>>
>>
>> <mailto:claudio.martella@__gma**il.com <http://gmail.com>
>> <mailto:claudio.martella@**gmail.com <cl...@gmail.com>
>> >>>
>>
>>
>> --
>> Matthew Laird
>> Lead Software Developer, Bioinformatics
>> Brinkman Laboratory
>> Simon Fraser University, Burnaby, BC, Canada
>>
>>
>> --
>> Matthew Laird
>> Lead Software Developer, Bioinformatics
>> Brinkman Laboratory
>> Simon Fraser University, Burnaby, BC, Canada
>>
>>
>>
>>
>> --
>> Claudio Martella
>> claudio.martella@gmail.com <ma...@gmail.com>
>> >
>>
>
> --
> Matthew Laird
> Lead Software Developer, Bioinformatics
> Brinkman Laboratory
> Simon Fraser University, Burnaby, BC, Canada
>
Re: connected components example in giraph 1.0
Posted by Silvio Di gregorio <si...@gmail.com>.
Excuse me giraph 1.0
Il giorno 07/ott/2013 21:10, "Matthew Laird" <la...@sfu.ca> ha scritto:
> Hmm, it seems there have been some recent pushes that have broken trunk as
> I discovered yesterday. :)
>
> But I guess that brings me back to my original question, to build against
> 2.0.5 using the MR interface rather than YARN as Milinda suggested, what
> profile would I use? Would I let it think it's 0.20xxx or tell it 2.0.5,
> or something else?
>
> Thanks.
>
> On 13-10-07 11:50 AM, Claudio Martella wrote:
>
>> trunk is 1.1.0-SNAPSHOT. I believe the date refers to the last update to
>> the site.
>>
>>
>> On Mon, Oct 7, 2013 at 8:48 PM, Matthew Laird <lairdm@sfu.ca
>> <ma...@sfu.ca>> wrote:
>>
>> Thanks, which is my next curiosity... if I head over to
>> http://giraph.apache.org/ I see at the top is says
>>
>> "Version: 1.1.0-SNAPSHOT | Last Published: 2013-08-10"
>>
>> However it's not clickable and the only download links I see are for
>> 1.0.0. I'm assuming trunk in git isn't the same as the 1.1.0
>> snapshot since more pushes have occurred since August, and I don't
>> see a tag in git for 1.1.0
>> (https://git-wip-us.apache.__**org/repos/asf?p=giraph.git
>> <https://git-wip-us.apache.**org/repos/asf?p=giraph.git<https://git-wip-us.apache.org/repos/asf?p=giraph.git>>).
>> So where
>> does one get 1.1.0-SNAPSHOT from?
>>
>> Thanks.
>>
>>
>> On 13-10-07 11:39 AM, Silvio Di gregorio wrote:
>>
>> Hi Matthew
>> I use giraph 1.1.0 and cdh4 hadoop 0.23
>>
>> the patch of Marco:
>>
>> http://www.mail-archive.com/__**__user@giraph.apache.org/____**
>> msg00945/check.diff<http://www.mail-archive.com/____user@giraph.apache.org/____msg00945/check.diff>
>> <http://www.mail-archive.com/_**_user@giraph.apache.org/__**
>> msg00945/check.diff<http://www.mail-archive.com/__user@giraph.apache.org/__msg00945/check.diff>
>> >
>>
>> <http://www.mail-archive.com/_**_user@giraph.apache.org/__**
>> msg00945/check.diff<http://www.mail-archive.com/__user@giraph.apache.org/__msg00945/check.diff>
>> <http://www.mail-archive.com/**user@giraph.apache.org/**
>> msg00945/check.diff<http://www.mail-archive.com/user@giraph.apache.org/msg00945/check.diff>
>> >>
>>
>> and build
>>
>> mvn clean install -P hadoop_cdh4.1.2
>>
>>
>>
>> Il giorno 07/ott/2013 20:20, "Matthew Laird" <lairdm@sfu.ca
>> <ma...@sfu.ca>
>> <mailto:lairdm@sfu.ca <ma...@sfu.ca>>> ha scritto:
>>
>>
>> Hello, so I'm watching this thread with interest as well,
>> since
>> that's the exact example I'll be trying to run as well, and
>> I see
>> Silvio appears to be asking about running Giraph 1.0
>> against Hadoop
>> 2.0.0.
>>
>> I'm curious, is that a known working combination, 1.0 vs
>> 2.0.x? And
>> if so, are there patches to apply first and what maven
>> command line
>> should I build with? (ie. will a -Dhadoop.version=2.0.5
>> give the
>> needed MR based API or will it use YARN?) I figure it's
>> better to
>> ask than aimlessly try all the combinations. :)
>>
>> Thanks!
>>
>> On 13-10-07 11:01 AM, Claudio Martella wrote:
>>
>> try to increase the mappers' heap space through
>> mapred.child.java.opts.
>>
>>
>> On Mon, Oct 7, 2013 at 1:18 PM, Silvio Di gregorio
>> <silvio.digregorio@gmail.com <mailto:silvio.digregorio@**
>> gmail.com <si...@gmail.com>>
>> <mailto:silvio.digregorio@__gm**ail.com <http://gmail.com>
>> <ma...@gmail.com>
>> >>
>> <mailto:silvio.digregorio@
>> <mailto:silvio.digregorio@>__g**m__ail.com <http://gm__ail.com> <
>> http://gmail.com>
>>
>> <mailto:silvio.digregorio@__gm**ail.com <http://gmail.com>
>> <ma...@gmail.com>>>>>
>> wrote:
>>
>>
>> I think now my issue is another, thanks to all.
>>
>>
>>
>>
>> java.lang.____**IllegalStateException: run: Caught
>> an
>>
>> unrecoverable exception waitFor: ExecutionException
>> occurred
>> while waiting for
>>
>> org.apache.giraph.utils.____**ProgressableUtils$____**
>> FutureWaitable@73a01e8a
>> at
>>
>> org.apache.giraph.graph.____**GraphMapper.run(GraphMapper.__**
>> __java:102)
>>
>>
>>
>> ......
>> Caused by: java.lang.____**IllegalStateException:
>> waitFor:
>>
>> ExecutionException occurred while waiting for
>>
>> org.apache.giraph.utils.____**ProgressableUtils$____**
>> FutureWaitable@73a01e8a
>> at
>>
>> org.apache.giraph.utils.____**ProgressableUtils.waitFor(____**
>> ProgressableUtils.java:151)
>>
>>
>>
>> ... 7 more
>> Caused by:
>> java.util.concurrent.____**ExecutionException:
>> java.lang.OutOfMemoryError:*__**__Java heap space*
>> at
>>
>> org.apache.giraph.utils.____**ProgressableUtils.waitFor(____**
>> ProgressableUtils.java:143)
>>
>>
>>
>>
>> ... 13 more
>>
>>
>>
>> 2013/10/7 Claudio Martella
>> <claudio.martella@gmail.com <ma...@gmail.com>
>> >
>> <mailto:claudio.martella@__gma**il.com <http://gmail.com>
>> <mailto:claudio.martella@**gmail.com <cl...@gmail.com>
>> >>
>> <mailto:claudio.martella@
>> <mailto:claudio.martella@>__gm**a__il.com <http://gma__il.com> <
>> http://gmail.com>
>>
>> <mailto:claudio.martella@__gma**il.com <http://gmail.com>
>> <mailto:claudio.martella@**gmail.com <cl...@gmail.com>
>> >>>>
>>
>> OK, thanks.
>>
>> I really have to push that patch in.
>>
>>
>> On Mon, Oct 7, 2013 at 12:17 PM, Silvio Di
>> gregorio
>> <silvio.digregorio@gmail.com
>> <ma...@gmail.com>
>> >
>> <mailto:silvio.digregorio@__gm**ail.com <http://gmail.com>
>> <ma...@gmail.com>
>> >>
>> <mailto:silvio.digregorio@
>> <mailto:silvio.digregorio@>__g**m__ail.com <http://gm__ail.com> <
>> http://gmail.com>
>>
>> <mailto:silvio.digregorio@__gm**ail.com <http://gmail.com>
>> <ma...@gmail.com>>>>>
>> wrote:
>>
>> yes i do,
>> i have seen this in your post in:
>> http://www.mail-archive.com/__**__user@giraph.apache.org/____**
>> msg00957.html<http://www.mail-archive.com/____user@giraph.apache.org/____msg00957.html>
>> <http://www.mail-archive.com/_**_user@giraph.apache.org/__**
>> msg00957.html<http://www.mail-archive.com/__user@giraph.apache.org/__msg00957.html>
>> >
>>
>> <http://www.mail-archive.com/_**_user@giraph.apache.org/__**
>> msg00957.html<http://www.mail-archive.com/__user@giraph.apache.org/__msg00957.html>
>> <http://www.mail-archive.com/**user@giraph.apache.org/**
>> msg00957.html<http://www.mail-archive.com/user@giraph.apache.org/msg00957.html>
>> >>
>>
>> excuse me if i had checked in the
>> mail-achive
>> first I would
>> have avoided the last post.
>>
>> Now zk issue are resolved.
>>
>>
>> 2013/10/7 Claudio Martella
>> <claudio.martella@gmail.com <ma...@gmail.com>
>> >
>> <mailto:claudio.martella@__gma**il.com <http://gmail.com>
>> <mailto:claudio.martella@**gmail.com <cl...@gmail.com>
>> >>
>> <mailto:claudio.martella@
>> <mailto:claudio.martella@>__gm**a__il.com <http://gma__il.com> <
>> http://gmail.com>
>>
>> <mailto:claudio.martella@__gma**il.com <http://gmail.com>
>> <mailto:claudio.martella@**gmail.com <cl...@gmail.com>
>> >>>>
>>
>> Can you try applying this one first?
>>
>> <<< <
>> http://www.mail-archive.com/__**__user@giraph.apache.org/____**
>> msg00945/check.diff<http://www.mail-archive.com/____user@giraph.apache.org/____msg00945/check.diff>
>> <http://www.mail-archive.com/_**_user@giraph.apache.org/__**
>> msg00945/check.diff<http://www.mail-archive.com/__user@giraph.apache.org/__msg00945/check.diff>
>> >
>>
>> <http://www.mail-archive.com/_**_user@giraph.apache.org/__**
>> msg00945/check.diff<http://www.mail-archive.com/__user@giraph.apache.org/__msg00945/check.diff>
>> <http://www.mail-archive.com/**user@giraph.apache.org/**
>> msg00945/check.diff<http://www.mail-archive.com/user@giraph.apache.org/msg00945/check.diff>
>> >><<<<
>>
>>
>> On Mon, Oct 7, 2013 at 8:40 AM, Silvio
>> Di gregorio
>> <silvio.digregorio@gmail.com
>> <ma...@gmail.com>
>> >
>> <mailto:silvio.digregorio@__gm**ail.com <http://gmail.com>
>> <ma...@gmail.com>
>> >>
>> <mailto:silvio.digregorio@
>> <mailto:silvio.digregorio@>__g**m__ail.com <http://gm__ail.com> <
>> http://gmail.com>
>>
>> <mailto:silvio.digregorio@__gm**ail.com <http://gmail.com>
>> <ma...@gmail.com>>>>>
>> wrote:
>>
>>
>>
>>
>>
>>
>>
>> *As i said i have builded*
>>
>>
>>
>>
>>
>>
>> /giraph-examples-1.0.0-for-___**_hadoop-2.0.0-cdh4.1.2-jar-___**
>> _with-dependencies.jar/
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> *for cdh4, successfully. The job
>> start to
>> monitoring the success rate:*
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> /13/10/07 08:28:45 INFO
>> mapred.JobClient:
>> map 0% reduce 0%/
>>
>>
>>
>>
>> *but then*
>>
>>
>>
>>
>> /Error running child
>>
>> java.lang.____**IllegalStateException: run:
>>
>> Caught an unrecoverable exception
>> java.io.FileNotFoundException:
>> File
>> _bsp/_defaultZkManagerDir/job_**____201309181636_0678/_**zkServer
>>
>> does not exist.
>> /
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> /.../
>>
>>
>>
>>
>>
>>
>>
>> /Caused by:
>> java.lang.RuntimeException:
>> java.io.FileNotFoundException: File
>>
>> _bsp/_defaultZkManagerDir/job_**____201309181636_0678/_**zkServer
>> does
>>
>> not exist./
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> 2013/10/5 Silvio Di gregorio
>> <silvio.digregorio@gmail.com
>> <ma...@gmail.com>
>> >
>> <mailto:silvio.digregorio@__gm**ail.com <http://gmail.com>
>> <ma...@gmail.com>
>> >>
>> <mailto:silvio.digregorio@
>> <mailto:silvio.digregorio@>__g**m__ail.com <http://gm__ail.com> <
>> http://gmail.com>
>>
>> <mailto:silvio.digregorio@__gm**ail.com <http://gmail.com>
>> <ma...@gmail.com>
>> >>>>
>>
>> I ha ve build w/
>> hadoop_cdh4.1.2 parameter.
>> Something is changed, monday i
>> report the
>> result. Now the farm is closed.
>>
>> Il giorno 05/ott/2013 14:06,
>> "Claudio
>> Martella"
>> <claudio.martella@gmail.com <ma...@gmail.com>
>> >
>> <mailto:claudio.martella@__gma**il.com <http://gmail.com>
>> <mailto:claudio.martella@**gmail.com <cl...@gmail.com>
>> >>
>> <mailto:claudio.martella@
>> <mailto:claudio.martella@>__gm**a__il.com <http://gma__il.com> <
>> http://gmail.com>
>>
>> <mailto:claudio.martella@__gma**il.com <http://gmail.com>
>> <mailto:claudio.martella@**gmail.com <cl...@gmail.com>>>>>
>> ha scritto:
>>
>> Oh, right, -vof is in
>> trunk. Anyway
>> it looks
>> like you built giraph for
>> the wrong
>> profile.
>> You mentioned you're
>> running on
>> 2.0, but
>> your giraph is built for
>> 0.20.203 <tel:0.20.203>
>> <tel:0.20.203 <tel:0.20.203>>
>> <tel:0.20.203 <tel:0.20.203> <tel:0.20.203 <tel:0.20.203>>>. try
>> building with a profile
>>
>> for your hadoop version.
>>
>>
>> On Fri, Oct 4, 2013 at
>> 2:35 PM,
>> Silvio Di
>> gregorio
>> <silvio.digregorio@gmail.com
>> <ma...@gmail.com>
>> >
>> <mailto:silvio.digregorio@__gm**ail.com <http://gmail.com>
>> <ma...@gmail.com>
>> >>
>> <mailto:silvio.digregorio@
>> <mailto:silvio.digregorio@>__g**m__ail.com <http://gm__ail.com> <
>> http://gmail.com>
>> <mailto:silvio.digregorio@__gm**ail.com <http://gmail.com>
>> <ma...@gmail.com>>>>>
>> wrote:
>>
>>
>>
>> org.apache.commons.cli.____**
>> UnrecognizedOptionException:
>>
>> Unrecognized option:
>> -vof
>>
>> in 1.0 version is
>> -of,--outputFormat
>> <arg>
>> Vertex output format
>> -op,--outputPath <arg>
>> Vertex output path
>>
>>
>>
>> 2013/10/4 Claudio
>> Martella
>> <claudio.martella@gmail.com <ma...@gmail.com>
>> >
>> <mailto:claudio.martella@__gma**il.com <http://gmail.com>
>> <mailto:claudio.martella@**gmail.com <cl...@gmail.com>
>> >>
>> <mailto:claudio.martella@
>> <mailto:claudio.martella@>__gm**a__il.com <http://gma__il.com> <
>> http://gmail.com>
>>
>> <mailto:claudio.martella@__gma**il.com <http://gmail.com>
>> <mailto:claudio.martella@**gmail.com <cl...@gmail.com>
>> >>>>
>>
>> did you try the
>> argument
>> (-vof) i
>> suggested?
>>
>>
>> On Fri, Oct 4,
>> 2013 at 2:13 PM,
>> Silvio Di gregorio
>> <silvio.digregorio@gmail.com
>> <ma...@gmail.com>
>> >
>> <mailto:silvio.digregorio@__gm**ail.com <http://gmail.com>
>> <ma...@gmail.com>
>> >>
>> <mailto:silvio.digregorio@
>> <mailto:silvio.digregorio@>__g**m__ail.com <http://gm__ail.com> <
>> http://gmail.com>
>>
>> <mailto:silvio.digregorio@__gm**ail.com <http://gmail.com>
>> <ma...@gmail.com>>>>>
>> wrote:
>>
>>
>> i've specified
>> -of
>>
>>
>> org.apache.giraph.io.formats._**___IdWithValueTextOutputFormat
>>
>>
>> but the same
>> error was
>> produced
>>
>> Exception in
>> thread "main"
>>
>> java.lang.____**IncompatibleClassChangeError:
>> Found interface
>>
>> org.apache.hadoop.mapreduce.__**__JobContext,
>>
>> but class was
>> expected
>> at
>>
>>
>> org.apache.giraph.bsp.____**BspOutputFormat.____**
>> checkOutputSpecs(____**BspOutputFormat.java:43)
>> at
>>
>>
>> org.apache.hadoop.mapred.____**JobClient$2.run(JobClient.____**
>> java:984)
>> at
>>
>>
>> org.apache.hadoop.mapred.____**JobClient$2.run(JobClient.____**
>> java:945)
>> at
>>
>> java.security.____**AccessController.doPrivileged(**
>> ____Native
>> Method)
>> at
>>
>> javax.security.auth.Subject.__**__doAs(Subject.java:396)
>> at
>>
>>
>> org.apache.hadoop.security.___**_UserGroupInformation.doAs(___**
>> _UserGroupInformation.java:___**_1408)
>> at
>>
>>
>> org.apache.hadoop.mapred.____**JobClient.submitJobInternal(__**
>> __JobClient.java:945)
>> at
>>
>> org.apache.hadoop.mapreduce.__**
>> __Job.submit(Job.java:566)
>> at
>>
>>
>> org.apache.hadoop.mapreduce.__**__Job.waitForCompletion(Job.__**
>> __java:596)
>> at
>>
>>
>> org.apache.giraph.job.____**GiraphJob.run(GiraphJob.java:_**
>> ___237)
>> at
>>
>>
>> org.apache.giraph.____**GiraphRunner.run(GiraphRunner.**
>> ____java:94)
>> at
>>
>>
>> org.apache.hadoop.util.____**ToolRunner.run(ToolRunner.____**
>> java:70)
>> at
>>
>>
>> org.apache.hadoop.util.____**ToolRunner.run(ToolRunner.____**
>> java:84)
>> at
>>
>>
>> org.apache.giraph.____**GiraphRunner.main(____**
>> GiraphRunner.java:124)
>> at
>>
>> sun.reflect.____**NativeMethodAccessorImpl.____**
>> invoke0(Native
>> Method)
>> at
>>
>>
>> sun.reflect.____**NativeMethodAccessorImpl.____**invoke(____**
>> NativeMethodAccessorImpl.java:**____39)
>> at
>>
>>
>> sun.reflect.____**DelegatingMethodAccessorImpl._**___invoke(____*
>> *DelegatingMethodAccessorImpl._**___java:25)
>> at
>>
>> java.lang.reflect.Method.____**invoke(Method.java:597)
>> at
>>
>> org.apache.hadoop.util.RunJar.**
>> ____main(RunJar.java:208)
>>
>>
>>
>>
>> 2013/10/4
>> Claudio Martella
>> <claudio.martella@gmail.com <ma...@gmail.com>
>> >
>> <mailto:claudio.martella@__gma**il.com <http://gmail.com>
>> <mailto:claudio.martella@**gmail.com <cl...@gmail.com>
>> >>
>> <mailto:claudio.martella@
>> <mailto:claudio.martella@>__gm**a__il.com <http://gma__il.com> <
>> http://gmail.com>
>>
>> <mailto:claudio.martella@__gma**il.com <http://gmail.com>
>> <mailto:claudio.martella@**gmail.com <cl...@gmail.com>
>> >>>>
>>
>> Hi,
>>
>> you need
>> to specify the
>> vertex
>> outputformat
>> class
>> (-vof
>> option), e.g.
>>
>>
>> org.apache.giraph.io.formats._**___**IdWithValueTextOutputFormat.
>>
>>
>>
>> On Fri,
>> Oct 4, 2013
>> at 1:06
>> PM, Silvio
>> Di gregorio
>> <silvio.digregorio@gmail.com
>> <ma...@gmail.com>
>> >
>> <mailto:silvio.digregorio@__gm**ail.com <http://gmail.com>
>> <ma...@gmail.com>
>> >>
>> <mailto:silvio.digregorio@
>> <mailto:silvio.digregorio@>__g**m__ail.com <http://gm__ail.com> <
>> http://gmail.com>
>>
>> <mailto:silvio.digregorio@__gm**ail.com <http://gmail.com>
>> <ma...@gmail.com>
>> >>>>
>> wrote:
>>
>>
>> Hi,
>>
>> I hope
>> I have
>> sent to
>> the
>> right address.
>>
>> i have
>> a graph
>> (directed
>> and
>> unweighted)
>> stored
>> in
>> hdfs like a
>> adjacency
>> list
>> (140Milions of
>> edges
>> 6Milions
>> of vertex)
>>
>>
>> node<tab>neighbors
>>
>> 23 2
>> 1 3
>> 4 3
>>
>> 1 999 99923 <tel:1%20999%2099923> <tel:1%20999%2099923>
>> <tel:1%20999%2099923>
>> 909
>> .......
>>
>> ......
>>
>> hadoop
>> version
>> Hadoop
>>
>> 2.0.0-cdh4.3.0
>> - java 1.6
>>
>>
>> I have
>> executed the
>>
>> giraph-1.0
>> connected
>>
>> components
>> example, in
>> this
>> fashion
>>
>> > hadoop jar
>>
>>
>> /usr/local/giraph/giraph-____**examples/target/giraph-____**
>> examples-1.0.0-for-hadoop-0.20.203.0
>> <tel:0.20.203.0>
>> <tel:0.20.203.0 <tel:0.20.203.0>>
>> <tel:0.20.203.0 <tel:0.20.203.0> <tel:0.20.203.0
>> <tel:0.20.203.0>>>-jar-with-**dependencies.____jar
>>
>> org.apache.giraph.GiraphRunner
>>
>> org.apache.giraph.examples.___**
>> _ConnectedComponentsVertex
>> -vif
>>
>> org.apache.giraph.io.formats._**
>> ___IntIntNullTextInputFormat
>>
>> -vip
>>
>> /user/hdfs/lista_adj_txt
>> -op
>> connectedgiraph
>>
>> --workers 4
>>
>> and
>> then fail with:
>>
>> 13/10/04
>> 09:28:29 INFO
>>
>> utils.ConfigurationUtils: No
>> edge
>> input format
>>
>> specified.
>> Ensure your
>>
>> InputFormat
>> does not
>>
>> require one.
>>
>> 13/10/04
>> 09:28:29 INFO
>>
>> utils.ConfigurationUtils: No
>> output
>> format
>> specified.
>> Ensure
>> your
>> OutputFormat
>> does not
>> require one.
>>
>> 13/10/04
>> 09:28:30 INFO
>>
>> job.GiraphJob: run:
>> Since
>> checkpointing is
>> disabled
>> (default), do
>> not
>> allow any task
>>
>> retries (setting
>>
>> mapred.map.max.attempts
>> = 0,
>> old value = 4)
>>
>> 13/10/04
>> 09:28:31 WARN
>>
>> mapred.JobClient: Use
>>
>> GenericOptionsParser for
>>
>> parsing the
>> arguments.
>>
>> Applications should
>>
>> implement Tool
>> for the same.
>>
>> 13/10/04
>> 09:28:31 INFO
>>
>> mapred.JobClient:
>>
>> Cleaning up the
>> staging
>> area
>>
>> hdfs://srv-bigdata-dev-01.int
>> <http://srv-bigdata-dev-01.int**>.____sose.it:8020/user/hdfs/.**
>> ____staging/job_201309181636_**0535<http://sose.it:8020/user/hdfs/.____staging/job_201309181636_0535>
>> <http://sose.it:8020/user/**hdfs/.__staging/job_**
>> 201309181636_0535<http://sose.it:8020/user/hdfs/.__staging/job_201309181636_0535>
>> >
>> <http://srv-bigdata-dev-01.__i**nt.sose.it:8020/user/hdfs/.__**
>> staging/job_201309181636_0535<http://int.sose.it:8020/user/hdfs/.__staging/job_201309181636_0535>
>> <http://srv-bigdata-dev-01.**int.sose.it:8020/user/hdfs/.**
>> staging/job_201309181636_0535<http://srv-bigdata-dev-01.int.sose.it:8020/user/hdfs/.staging/job_201309181636_0535>
>> >**>
>> <http://srv-bigdata-dev-01.__i**__nt.sose.it:8020/user/hdfs/._**
>> ___staging/job_201309181636_**0535<http://i__nt.sose.it:8020/user/hdfs/.____staging/job_201309181636_0535>
>> <http://int.sose.it:8020/user/**hdfs/.__staging/job_**
>> 201309181636_0535<http://int.sose.it:8020/user/hdfs/.__staging/job_201309181636_0535>
>> >
>>
>> <http://srv-bigdata-dev-01.__i**nt.sose.it:8020/user/hdfs/.__**
>> staging/job_201309181636_0535<http://int.sose.it:8020/user/hdfs/.__staging/job_201309181636_0535>
>> <http://srv-bigdata-dev-01.**int.sose.it:8020/user/hdfs/.**
>> staging/job_201309181636_0535<http://srv-bigdata-dev-01.int.sose.it:8020/user/hdfs/.staging/job_201309181636_0535>
>> >**>__>
>>
>>
>> Exception in thread
>> "main"
>>
>> java.lang.____**IncompatibleClassChangeError:
>> Found
>> interface
>>
>> org.apache.hadoop.mapreduce.__**__JobContext,
>>
>> but
>> class was
>> expected
>>
>>
>> at
>>
>>
>> org.apache.giraph.bsp.____**BspOutputFormat.____**
>> checkOutputSpecs(____**BspOutputFormat.java:43)
>>
>>
>> ......
>>
>> Thanks
>> in advance
>>
>>
>>
>>
>>
>> --
>>
>> Claudio Martella
>> claudio.martella@gmail.com <ma...@gmail.com>
>> >
>> <mailto:claudio.martella@__gma**il.com <http://gmail.com>
>> <mailto:claudio.martella@**gmail.com <cl...@gmail.com>
>> >>
>> <mailto:claudio.martella@
>> <mailto:claudio.martella@>__gm**a__il.com <http://gma__il.com> <
>> http://gmail.com>
>>
>> <mailto:claudio.martella@__gma**il.com <http://gmail.com>
>> <mailto:claudio.martella@**gmail.com <cl...@gmail.com>
>> >>>
>>
>>
>>
>>
>>
>>
>> --
>> Claudio Martella
>> claudio.martella@gmail.com <ma...@gmail.com>
>> >
>> <mailto:claudio.martella@__gma**il.com <http://gmail.com>
>> <mailto:claudio.martella@**gmail.com <cl...@gmail.com>
>> >>
>> <mailto:claudio.martella@
>> <mailto:claudio.martella@>__gm**a__il.com <http://gma__il.com> <
>> http://gmail.com>
>>
>> <mailto:claudio.martella@__gma**il.com <http://gmail.com>
>> <mailto:claudio.martella@**gmail.com <cl...@gmail.com>
>> >>>
>>
>>
>>
>>
>>
>> --
>> Claudio Martella
>> claudio.martella@gmail.com <ma...@gmail.com>
>> >
>> <mailto:claudio.martella@__gma**il.com <http://gmail.com>
>> <mailto:claudio.martella@**gmail.com <cl...@gmail.com>
>> >>
>> <mailto:claudio.martella@
>> <mailto:claudio.martella@>__gm**a__il.com <http://gma__il.com> <
>> http://gmail.com>
>>
>> <mailto:claudio.martella@__gma**il.com <http://gmail.com>
>> <mailto:claudio.martella@**gmail.com <cl...@gmail.com>
>> >>>
>>
>>
>>
>>
>>
>> --
>> Claudio Martella
>> claudio.martella@gmail.com <ma...@gmail.com>
>> >
>> <mailto:claudio.martella@__gma**il.com <http://gmail.com>
>> <mailto:claudio.martella@**gmail.com <cl...@gmail.com>
>> >>
>> <mailto:claudio.martella@
>> <mailto:claudio.martella@>__gm**a__il.com <http://gma__il.com> <
>> http://gmail.com>
>>
>> <mailto:claudio.martella@__gma**il.com <http://gmail.com>
>> <mailto:claudio.martella@**gmail.com <cl...@gmail.com>
>> >>>
>>
>>
>>
>>
>>
>> --
>> Claudio Martella
>> claudio.martella@gmail.com <ma...@gmail.com>
>> >
>> <mailto:claudio.martella@__gma**il.com <http://gmail.com>
>> <mailto:claudio.martella@**gmail.com <cl...@gmail.com>
>> >>
>> <mailto:claudio.martella@
>> <mailto:claudio.martella@>__gm**a__il.com <http://gma__il.com> <
>> http://gmail.com>
>>
>> <mailto:claudio.martella@__gma**il.com <http://gmail.com>
>> <mailto:claudio.martella@**gmail.com <cl...@gmail.com>
>> >>>
>>
>>
>>
>>
>>
>> --
>> Claudio Martella
>> claudio.martella@gmail.com <ma...@gmail.com>
>> >
>> <mailto:claudio.martella@__gma**il.com <http://gmail.com>
>> <mailto:claudio.martella@**gmail.com <cl...@gmail.com>
>> >>
>> <mailto:claudio.martella@
>> <mailto:claudio.martella@>__gm**a__il.com <http://gma__il.com> <
>> http://gmail.com>
>>
>> <mailto:claudio.martella@__gma**il.com <http://gmail.com>
>> <mailto:claudio.martella@**gmail.com <cl...@gmail.com>
>> >>>
>>
>>
>> --
>> Matthew Laird
>> Lead Software Developer, Bioinformatics
>> Brinkman Laboratory
>> Simon Fraser University, Burnaby, BC, Canada
>>
>>
>> --
>> Matthew Laird
>> Lead Software Developer, Bioinformatics
>> Brinkman Laboratory
>> Simon Fraser University, Burnaby, BC, Canada
>>
>>
>>
>>
>> --
>> Claudio Martella
>> claudio.martella@gmail.com <ma...@gmail.com>
>> >
>>
>
> --
> Matthew Laird
> Lead Software Developer, Bioinformatics
> Brinkman Laboratory
> Simon Fraser University, Burnaby, BC, Canada
>
Re: connected components example in giraph 1.0
Posted by Matthew Laird <la...@sfu.ca>.
Hmm, it seems there have been some recent pushes that have broken trunk
as I discovered yesterday. :)
But I guess that brings me back to my original question, to build
against 2.0.5 using the MR interface rather than YARN as Milinda
suggested, what profile would I use? Would I let it think it's 0.20xxx
or tell it 2.0.5, or something else?
Thanks.
On 13-10-07 11:50 AM, Claudio Martella wrote:
> trunk is 1.1.0-SNAPSHOT. I believe the date refers to the last update to
> the site.
>
>
> On Mon, Oct 7, 2013 at 8:48 PM, Matthew Laird <lairdm@sfu.ca
> <ma...@sfu.ca>> wrote:
>
> Thanks, which is my next curiosity... if I head over to
> http://giraph.apache.org/ I see at the top is says
>
> "Version: 1.1.0-SNAPSHOT | Last Published: 2013-08-10"
>
> However it's not clickable and the only download links I see are for
> 1.0.0. I'm assuming trunk in git isn't the same as the 1.1.0
> snapshot since more pushes have occurred since August, and I don't
> see a tag in git for 1.1.0
> (https://git-wip-us.apache.__org/repos/asf?p=giraph.git
> <https://git-wip-us.apache.org/repos/asf?p=giraph.git>). So where
> does one get 1.1.0-SNAPSHOT from?
>
> Thanks.
>
>
> On 13-10-07 11:39 AM, Silvio Di gregorio wrote:
>
> Hi Matthew
> I use giraph 1.1.0 and cdh4 hadoop 0.23
>
> the patch of Marco:
>
> http://www.mail-archive.com/____user@giraph.apache.org/____msg00945/check.diff
> <http://www.mail-archive.com/__user@giraph.apache.org/__msg00945/check.diff>
>
> <http://www.mail-archive.com/__user@giraph.apache.org/__msg00945/check.diff
> <http://www.mail-archive.com/user@giraph.apache.org/msg00945/check.diff>>
>
> and build
>
> mvn clean install -P hadoop_cdh4.1.2
>
>
>
> Il giorno 07/ott/2013 20:20, "Matthew Laird" <lairdm@sfu.ca
> <ma...@sfu.ca>
> <mailto:lairdm@sfu.ca <ma...@sfu.ca>>> ha scritto:
>
>
> Hello, so I'm watching this thread with interest as well, since
> that's the exact example I'll be trying to run as well, and
> I see
> Silvio appears to be asking about running Giraph 1.0
> against Hadoop
> 2.0.0.
>
> I'm curious, is that a known working combination, 1.0 vs
> 2.0.x? And
> if so, are there patches to apply first and what maven
> command line
> should I build with? (ie. will a -Dhadoop.version=2.0.5
> give the
> needed MR based API or will it use YARN?) I figure it's
> better to
> ask than aimlessly try all the combinations. :)
>
> Thanks!
>
> On 13-10-07 11:01 AM, Claudio Martella wrote:
>
> try to increase the mappers' heap space through
> mapred.child.java.opts.
>
>
> On Mon, Oct 7, 2013 at 1:18 PM, Silvio Di gregorio
> <silvio.digregorio@gmail.com <ma...@gmail.com>
> <mailto:silvio.digregorio@__gmail.com
> <ma...@gmail.com>>
> <mailto:silvio.digregorio@
> <mailto:silvio.digregorio@>__gm__ail.com <http://gmail.com>
>
> <mailto:silvio.digregorio@__gmail.com
> <ma...@gmail.com>>>> wrote:
>
>
> I think now my issue is another, thanks to all.
>
>
>
>
> java.lang.____IllegalStateException: run: Caught an
>
> unrecoverable exception waitFor: ExecutionException
> occurred
> while waiting for
>
> org.apache.giraph.utils.____ProgressableUtils$____FutureWaitable@73a01e8a
> at
>
> org.apache.giraph.graph.____GraphMapper.run(GraphMapper.____java:102)
>
>
>
> ......
> Caused by: java.lang.____IllegalStateException:
> waitFor:
>
> ExecutionException occurred while waiting for
>
> org.apache.giraph.utils.____ProgressableUtils$____FutureWaitable@73a01e8a
> at
>
> org.apache.giraph.utils.____ProgressableUtils.waitFor(____ProgressableUtils.java:151)
>
>
>
> ... 7 more
> Caused by:
> java.util.concurrent.____ExecutionException:
> java.lang.OutOfMemoryError:*____Java heap space*
> at
>
> org.apache.giraph.utils.____ProgressableUtils.waitFor(____ProgressableUtils.java:143)
>
>
>
>
> ... 13 more
>
>
>
> 2013/10/7 Claudio Martella
> <claudio.martella@gmail.com <ma...@gmail.com>
> <mailto:claudio.martella@__gmail.com
> <ma...@gmail.com>>
> <mailto:claudio.martella@
> <mailto:claudio.martella@>__gma__il.com <http://gmail.com>
>
> <mailto:claudio.martella@__gmail.com
> <ma...@gmail.com>>>>
>
> OK, thanks.
>
> I really have to push that patch in.
>
>
> On Mon, Oct 7, 2013 at 12:17 PM, Silvio Di
> gregorio
> <silvio.digregorio@gmail.com
> <ma...@gmail.com>
> <mailto:silvio.digregorio@__gmail.com
> <ma...@gmail.com>>
> <mailto:silvio.digregorio@
> <mailto:silvio.digregorio@>__gm__ail.com <http://gmail.com>
>
> <mailto:silvio.digregorio@__gmail.com
> <ma...@gmail.com>>>> wrote:
>
> yes i do,
> i have seen this in your post in:
> http://www.mail-archive.com/____user@giraph.apache.org/____msg00957.html
> <http://www.mail-archive.com/__user@giraph.apache.org/__msg00957.html>
>
> <http://www.mail-archive.com/__user@giraph.apache.org/__msg00957.html
> <http://www.mail-archive.com/user@giraph.apache.org/msg00957.html>>
>
> excuse me if i had checked in the mail-achive
> first I would
> have avoided the last post.
>
> Now zk issue are resolved.
>
>
> 2013/10/7 Claudio Martella
> <claudio.martella@gmail.com <ma...@gmail.com>
> <mailto:claudio.martella@__gmail.com
> <ma...@gmail.com>>
> <mailto:claudio.martella@
> <mailto:claudio.martella@>__gma__il.com <http://gmail.com>
>
> <mailto:claudio.martella@__gmail.com
> <ma...@gmail.com>>>>
>
> Can you try applying this one first?
>
> <<< <
> http://www.mail-archive.com/____user@giraph.apache.org/____msg00945/check.diff
> <http://www.mail-archive.com/__user@giraph.apache.org/__msg00945/check.diff>
>
> <http://www.mail-archive.com/__user@giraph.apache.org/__msg00945/check.diff
> <http://www.mail-archive.com/user@giraph.apache.org/msg00945/check.diff>><<<<
>
>
> On Mon, Oct 7, 2013 at 8:40 AM, Silvio
> Di gregorio
> <silvio.digregorio@gmail.com
> <ma...@gmail.com>
> <mailto:silvio.digregorio@__gmail.com
> <ma...@gmail.com>>
> <mailto:silvio.digregorio@
> <mailto:silvio.digregorio@>__gm__ail.com <http://gmail.com>
>
> <mailto:silvio.digregorio@__gmail.com
> <ma...@gmail.com>>>> wrote:
>
>
>
>
>
>
>
> *As i said i have builded*
>
>
>
>
>
>
> /giraph-examples-1.0.0-for-____hadoop-2.0.0-cdh4.1.2-jar-____with-dependencies.jar/
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> *for cdh4, successfully. The job
> start to
> monitoring the success rate:*
>
>
>
>
>
>
>
>
>
>
>
>
>
> /13/10/07 08:28:45 INFO
> mapred.JobClient:
> map 0% reduce 0%/
>
>
>
>
> *but then*
>
>
>
>
> /Error running child
>
> java.lang.____IllegalStateException: run:
>
> Caught an unrecoverable exception
> java.io.FileNotFoundException:
> File
> _bsp/_defaultZkManagerDir/job_____201309181636_0678/_zkServer
>
> does not exist.
> /
>
>
>
>
>
>
>
>
>
>
>
>
>
> /.../
>
>
>
>
>
>
>
> /Caused by:
> java.lang.RuntimeException:
> java.io.FileNotFoundException: File
>
> _bsp/_defaultZkManagerDir/job_____201309181636_0678/_zkServer does
>
> not exist./
>
>
>
>
>
>
>
>
>
>
> 2013/10/5 Silvio Di gregorio
> <silvio.digregorio@gmail.com
> <ma...@gmail.com>
> <mailto:silvio.digregorio@__gmail.com
> <ma...@gmail.com>>
> <mailto:silvio.digregorio@
> <mailto:silvio.digregorio@>__gm__ail.com <http://gmail.com>
>
> <mailto:silvio.digregorio@__gmail.com
> <ma...@gmail.com>>>>
>
> I ha ve build w/
> hadoop_cdh4.1.2 parameter.
> Something is changed, monday i
> report the
> result. Now the farm is closed.
>
> Il giorno 05/ott/2013 14:06,
> "Claudio
> Martella"
> <claudio.martella@gmail.com <ma...@gmail.com>
> <mailto:claudio.martella@__gmail.com
> <ma...@gmail.com>>
> <mailto:claudio.martella@
> <mailto:claudio.martella@>__gma__il.com <http://gmail.com>
>
> <mailto:claudio.martella@__gmail.com
> <ma...@gmail.com>>>> ha scritto:
>
> Oh, right, -vof is in
> trunk. Anyway
> it looks
> like you built giraph for
> the wrong
> profile.
> You mentioned you're
> running on
> 2.0, but
> your giraph is built for
> 0.20.203 <tel:0.20.203>
> <tel:0.20.203 <tel:0.20.203>>
> <tel:0.20.203 <tel:0.20.203> <tel:0.20.203 <tel:0.20.203>>>. try
> building with a profile
>
> for your hadoop version.
>
>
> On Fri, Oct 4, 2013 at
> 2:35 PM,
> Silvio Di
> gregorio
> <silvio.digregorio@gmail.com
> <ma...@gmail.com>
> <mailto:silvio.digregorio@__gmail.com
> <ma...@gmail.com>>
> <mailto:silvio.digregorio@
> <mailto:silvio.digregorio@>__gm__ail.com <http://gmail.com>
> <mailto:silvio.digregorio@__gmail.com
> <ma...@gmail.com>>>> wrote:
>
>
>
> org.apache.commons.cli.____UnrecognizedOptionException:
>
> Unrecognized option: -vof
>
> in 1.0 version is
> -of,--outputFormat <arg>
> Vertex output format
> -op,--outputPath <arg>
> Vertex output path
>
>
>
> 2013/10/4 Claudio Martella
> <claudio.martella@gmail.com <ma...@gmail.com>
> <mailto:claudio.martella@__gmail.com
> <ma...@gmail.com>>
> <mailto:claudio.martella@
> <mailto:claudio.martella@>__gma__il.com <http://gmail.com>
>
> <mailto:claudio.martella@__gmail.com
> <ma...@gmail.com>>>>
>
> did you try the
> argument
> (-vof) i
> suggested?
>
>
> On Fri, Oct 4,
> 2013 at 2:13 PM,
> Silvio Di gregorio
> <silvio.digregorio@gmail.com
> <ma...@gmail.com>
> <mailto:silvio.digregorio@__gmail.com
> <ma...@gmail.com>>
> <mailto:silvio.digregorio@
> <mailto:silvio.digregorio@>__gm__ail.com <http://gmail.com>
>
> <mailto:silvio.digregorio@__gmail.com
> <ma...@gmail.com>>>> wrote:
>
>
> i've specified
> -of
>
>
> org.apache.giraph.io.formats.____IdWithValueTextOutputFormat
>
>
> but the same
> error was
> produced
>
> Exception in
> thread "main"
>
> java.lang.____IncompatibleClassChangeError:
> Found interface
>
> org.apache.hadoop.mapreduce.____JobContext,
>
> but class was
> expected
> at
>
>
> org.apache.giraph.bsp.____BspOutputFormat.____checkOutputSpecs(____BspOutputFormat.java:43)
> at
>
>
> org.apache.hadoop.mapred.____JobClient$2.run(JobClient.____java:984)
> at
>
>
> org.apache.hadoop.mapred.____JobClient$2.run(JobClient.____java:945)
> at
>
> java.security.____AccessController.doPrivileged(____Native
> Method)
> at
>
> javax.security.auth.Subject.____doAs(Subject.java:396)
> at
>
>
> org.apache.hadoop.security.____UserGroupInformation.doAs(____UserGroupInformation.java:____1408)
> at
>
>
> org.apache.hadoop.mapred.____JobClient.submitJobInternal(____JobClient.java:945)
> at
>
> org.apache.hadoop.mapreduce.____Job.submit(Job.java:566)
> at
>
>
> org.apache.hadoop.mapreduce.____Job.waitForCompletion(Job.____java:596)
> at
>
>
> org.apache.giraph.job.____GiraphJob.run(GiraphJob.java:____237)
> at
>
>
> org.apache.giraph.____GiraphRunner.run(GiraphRunner.____java:94)
> at
>
>
> org.apache.hadoop.util.____ToolRunner.run(ToolRunner.____java:70)
> at
>
>
> org.apache.hadoop.util.____ToolRunner.run(ToolRunner.____java:84)
> at
>
>
> org.apache.giraph.____GiraphRunner.main(____GiraphRunner.java:124)
> at
>
> sun.reflect.____NativeMethodAccessorImpl.____invoke0(Native
> Method)
> at
>
>
> sun.reflect.____NativeMethodAccessorImpl.____invoke(____NativeMethodAccessorImpl.java:____39)
> at
>
>
> sun.reflect.____DelegatingMethodAccessorImpl.____invoke(____DelegatingMethodAccessorImpl.____java:25)
> at
>
> java.lang.reflect.Method.____invoke(Method.java:597)
> at
>
> org.apache.hadoop.util.RunJar.____main(RunJar.java:208)
>
>
>
>
> 2013/10/4
> Claudio Martella
> <claudio.martella@gmail.com <ma...@gmail.com>
> <mailto:claudio.martella@__gmail.com
> <ma...@gmail.com>>
> <mailto:claudio.martella@
> <mailto:claudio.martella@>__gma__il.com <http://gmail.com>
>
> <mailto:claudio.martella@__gmail.com
> <ma...@gmail.com>>>>
>
> Hi,
>
> you need
> to specify the
> vertex
> outputformat
> class
> (-vof
> option), e.g.
>
>
> org.apache.giraph.io.formats.____IdWithValueTextOutputFormat.
>
>
>
> On Fri,
> Oct 4, 2013
> at 1:06
> PM, Silvio
> Di gregorio
> <silvio.digregorio@gmail.com
> <ma...@gmail.com>
> <mailto:silvio.digregorio@__gmail.com
> <ma...@gmail.com>>
> <mailto:silvio.digregorio@
> <mailto:silvio.digregorio@>__gm__ail.com <http://gmail.com>
>
> <mailto:silvio.digregorio@__gmail.com
> <ma...@gmail.com>>>>
> wrote:
>
>
> Hi,
>
> I hope
> I have
> sent to
> the
> right address.
>
> i have
> a graph
> (directed
> and
> unweighted)
> stored
> in
> hdfs like a
> adjacency
> list
> (140Milions of
> edges
> 6Milions
> of vertex)
>
>
> node<tab>neighbors
>
> 23 2
> 1 3
> 4 3
>
> 1 999 99923 <tel:1%20999%2099923> <tel:1%20999%2099923>
> <tel:1%20999%2099923>
> 909
> .......
>
> ......
>
> hadoop
> version
> Hadoop
>
> 2.0.0-cdh4.3.0
> - java 1.6
>
>
> I have
> executed the
> giraph-1.0
> connected
> components
> example, in
> this
> fashion
>
> > hadoop jar
>
>
> /usr/local/giraph/giraph-____examples/target/giraph-____examples-1.0.0-for-hadoop-0.20.203.0
> <tel:0.20.203.0>
> <tel:0.20.203.0 <tel:0.20.203.0>>
> <tel:0.20.203.0 <tel:0.20.203.0> <tel:0.20.203.0
> <tel:0.20.203.0>>>-jar-with-dependencies.____jar
>
> org.apache.giraph.GiraphRunner
>
> org.apache.giraph.examples.____ConnectedComponentsVertex
> -vif
>
> org.apache.giraph.io.formats.____IntIntNullTextInputFormat
>
> -vip
>
> /user/hdfs/lista_adj_txt
> -op
> connectedgiraph
>
> --workers 4
>
> and
> then fail with:
>
> 13/10/04
> 09:28:29 INFO
>
> utils.ConfigurationUtils: No
> edge
> input format
> specified.
> Ensure your
>
> InputFormat
> does not
>
> require one.
>
> 13/10/04
> 09:28:29 INFO
>
> utils.ConfigurationUtils: No
> output
> format
> specified.
> Ensure
> your
> OutputFormat
> does not
> require one.
>
> 13/10/04
> 09:28:30 INFO
>
> job.GiraphJob: run:
> Since
> checkpointing is
> disabled
> (default), do
> not
> allow any task
>
> retries (setting
>
> mapred.map.max.attempts
> = 0,
> old value = 4)
>
> 13/10/04
> 09:28:31 WARN
>
> mapred.JobClient: Use
>
> GenericOptionsParser for
>
> parsing the
> arguments.
>
> Applications should
>
> implement Tool
> for the same.
>
> 13/10/04
> 09:28:31 INFO
>
> mapred.JobClient:
>
> Cleaning up the
> staging
> area
>
> hdfs://srv-bigdata-dev-01.int
> <http://srv-bigdata-dev-01.int>.____sose.it:8020/user/hdfs/.____staging/job_201309181636_0535
> <http://sose.it:8020/user/hdfs/.__staging/job_201309181636_0535>
> <http://srv-bigdata-dev-01.__int.sose.it:8020/user/hdfs/.__staging/job_201309181636_0535
> <http://srv-bigdata-dev-01.int.sose.it:8020/user/hdfs/.staging/job_201309181636_0535>>
> <http://srv-bigdata-dev-01.__i__nt.sose.it:8020/user/hdfs/.____staging/job_201309181636_0535
> <http://int.sose.it:8020/user/hdfs/.__staging/job_201309181636_0535>
>
> <http://srv-bigdata-dev-01.__int.sose.it:8020/user/hdfs/.__staging/job_201309181636_0535
> <http://srv-bigdata-dev-01.int.sose.it:8020/user/hdfs/.staging/job_201309181636_0535>>__>
>
>
> Exception in thread
> "main"
>
> java.lang.____IncompatibleClassChangeError:
> Found
> interface
>
> org.apache.hadoop.mapreduce.____JobContext,
>
> but
> class was
> expected
>
>
> at
>
>
> org.apache.giraph.bsp.____BspOutputFormat.____checkOutputSpecs(____BspOutputFormat.java:43)
>
>
> ......
>
> Thanks
> in advance
>
>
>
>
>
> --
>
> Claudio Martella
> claudio.martella@gmail.com <ma...@gmail.com>
> <mailto:claudio.martella@__gmail.com
> <ma...@gmail.com>>
> <mailto:claudio.martella@
> <mailto:claudio.martella@>__gma__il.com <http://gmail.com>
>
> <mailto:claudio.martella@__gmail.com
> <ma...@gmail.com>>>
>
>
>
>
>
>
> --
> Claudio Martella
> claudio.martella@gmail.com <ma...@gmail.com>
> <mailto:claudio.martella@__gmail.com
> <ma...@gmail.com>>
> <mailto:claudio.martella@
> <mailto:claudio.martella@>__gma__il.com <http://gmail.com>
>
> <mailto:claudio.martella@__gmail.com
> <ma...@gmail.com>>>
>
>
>
>
>
> --
> Claudio Martella
> claudio.martella@gmail.com <ma...@gmail.com>
> <mailto:claudio.martella@__gmail.com
> <ma...@gmail.com>>
> <mailto:claudio.martella@
> <mailto:claudio.martella@>__gma__il.com <http://gmail.com>
>
> <mailto:claudio.martella@__gmail.com
> <ma...@gmail.com>>>
>
>
>
>
>
> --
> Claudio Martella
> claudio.martella@gmail.com <ma...@gmail.com>
> <mailto:claudio.martella@__gmail.com
> <ma...@gmail.com>>
> <mailto:claudio.martella@
> <mailto:claudio.martella@>__gma__il.com <http://gmail.com>
>
> <mailto:claudio.martella@__gmail.com
> <ma...@gmail.com>>>
>
>
>
>
>
> --
> Claudio Martella
> claudio.martella@gmail.com <ma...@gmail.com>
> <mailto:claudio.martella@__gmail.com
> <ma...@gmail.com>>
> <mailto:claudio.martella@
> <mailto:claudio.martella@>__gma__il.com <http://gmail.com>
>
> <mailto:claudio.martella@__gmail.com
> <ma...@gmail.com>>>
>
>
>
>
>
> --
> Claudio Martella
> claudio.martella@gmail.com <ma...@gmail.com>
> <mailto:claudio.martella@__gmail.com
> <ma...@gmail.com>>
> <mailto:claudio.martella@
> <mailto:claudio.martella@>__gma__il.com <http://gmail.com>
>
> <mailto:claudio.martella@__gmail.com
> <ma...@gmail.com>>>
>
>
> --
> Matthew Laird
> Lead Software Developer, Bioinformatics
> Brinkman Laboratory
> Simon Fraser University, Burnaby, BC, Canada
>
>
> --
> Matthew Laird
> Lead Software Developer, Bioinformatics
> Brinkman Laboratory
> Simon Fraser University, Burnaby, BC, Canada
>
>
>
>
> --
> Claudio Martella
> claudio.martella@gmail.com <ma...@gmail.com>
--
Matthew Laird
Lead Software Developer, Bioinformatics
Brinkman Laboratory
Simon Fraser University, Burnaby, BC, Canada
Re: connected components example in giraph 1.0
Posted by Claudio Martella <cl...@gmail.com>.
trunk is 1.1.0-SNAPSHOT. I believe the date refers to the last update to
the site.
On Mon, Oct 7, 2013 at 8:48 PM, Matthew Laird <la...@sfu.ca> wrote:
> Thanks, which is my next curiosity... if I head over to
> http://giraph.apache.org/ I see at the top is says
>
> "Version: 1.1.0-SNAPSHOT | Last Published: 2013-08-10"
>
> However it's not clickable and the only download links I see are for
> 1.0.0. I'm assuming trunk in git isn't the same as the 1.1.0 snapshot
> since more pushes have occurred since August, and I don't see a tag in git
> for 1.1.0 (https://git-wip-us.apache.**org/repos/asf?p=giraph.git<https://git-wip-us.apache.org/repos/asf?p=giraph.git>).
> So where does one get 1.1.0-SNAPSHOT from?
>
> Thanks.
>
>
> On 13-10-07 11:39 AM, Silvio Di gregorio wrote:
>
>> Hi Matthew
>> I use giraph 1.1.0 and cdh4 hadoop 0.23
>>
>> the patch of Marco:
>>
>> http://www.mail-archive.com/__**user@giraph.apache.org/__**
>> msg00945/check.diff<http://www.mail-archive.com/__user@giraph.apache.org/__msg00945/check.diff>
>>
>> <http://www.mail-archive.com/**user@giraph.apache.org/**
>> msg00945/check.diff<http://www.mail-archive.com/user@giraph.apache.org/msg00945/check.diff>
>> >
>>
>> and build
>>
>> mvn clean install -P hadoop_cdh4.1.2
>>
>>
>>
>> Il giorno 07/ott/2013 20:20, "Matthew Laird" <lairdm@sfu.ca
>> <ma...@sfu.ca>> ha scritto:
>>
>>
>> Hello, so I'm watching this thread with interest as well, since
>> that's the exact example I'll be trying to run as well, and I see
>> Silvio appears to be asking about running Giraph 1.0 against Hadoop
>> 2.0.0.
>>
>> I'm curious, is that a known working combination, 1.0 vs 2.0.x? And
>> if so, are there patches to apply first and what maven command line
>> should I build with? (ie. will a -Dhadoop.version=2.0.5 give the
>> needed MR based API or will it use YARN?) I figure it's better to
>> ask than aimlessly try all the combinations. :)
>>
>> Thanks!
>>
>> On 13-10-07 11:01 AM, Claudio Martella wrote:
>>
>> try to increase the mappers' heap space through
>> mapred.child.java.opts.
>>
>>
>> On Mon, Oct 7, 2013 at 1:18 PM, Silvio Di gregorio
>> <silvio.digregorio@gmail.com
>> <ma...@gmail.com>
>> >
>> <mailto:silvio.digregorio@__gm**ail.com <http://gmail.com>
>>
>> <ma...@gmail.com>>>>
>> wrote:
>>
>>
>> I think now my issue is another, thanks to all.
>>
>>
>>
>>
>> java.lang.__**IllegalStateException: run: Caught an
>>
>> unrecoverable exception waitFor: ExecutionException occurred
>> while waiting for
>> org.apache.giraph.utils.__**ProgressableUtils$__**
>> FutureWaitable@73a01e8a
>> at
>> org.apache.giraph.graph.__**GraphMapper.run(GraphMapper.__**
>> java:102)
>>
>>
>>
>> ......
>> Caused by: java.lang.__**IllegalStateException: waitFor:
>>
>> ExecutionException occurred while waiting for
>> org.apache.giraph.utils.__**ProgressableUtils$__**
>> FutureWaitable@73a01e8a
>> at
>> org.apache.giraph.utils.__**ProgressableUtils.waitFor(__**
>> ProgressableUtils.java:151)
>>
>>
>>
>> ... 7 more
>> Caused by: java.util.concurrent.__**ExecutionException:
>> java.lang.OutOfMemoryError:*__**Java heap space*
>> at
>> org.apache.giraph.utils.__**ProgressableUtils.waitFor(__**
>> ProgressableUtils.java:143)
>>
>>
>>
>>
>> ... 13 more
>>
>>
>>
>> 2013/10/7 Claudio Martella <claudio.martella@gmail.com
>> <mailto:claudio.martella@**gmail.com <cl...@gmail.com>
>> >
>> <mailto:claudio.martella@__gma**il.com <http://gmail.com>
>>
>> <mailto:claudio.martella@**gmail.com <cl...@gmail.com>
>> >>>
>>
>> OK, thanks.
>>
>> I really have to push that patch in.
>>
>>
>> On Mon, Oct 7, 2013 at 12:17 PM, Silvio Di gregorio
>> <silvio.digregorio@gmail.com <mailto:silvio.digregorio@**
>> gmail.com <si...@gmail.com>>
>> <mailto:silvio.digregorio@__gm**ail.com <http://gmail.com>
>>
>> <ma...@gmail.com>>>>
>> wrote:
>>
>> yes i do,
>> i have seen this in your post in:
>> http://www.mail-archive.com/__**user@giraph.apache.org/__**
>> msg00957.html<http://www.mail-archive.com/__user@giraph.apache.org/__msg00957.html>
>>
>> <http://www.mail-archive.com/**user@giraph.apache.org/**
>> msg00957.html<http://www.mail-archive.com/user@giraph.apache.org/msg00957.html>
>> >
>>
>> excuse me if i had checked in the mail-achive
>> first I would
>> have avoided the last post.
>>
>> Now zk issue are resolved.
>>
>>
>> 2013/10/7 Claudio Martella
>> <claudio.martella@gmail.com <ma...@gmail.com>
>> >
>> <mailto:claudio.martella@__gma**il.com <http://gmail.com>
>>
>> <mailto:claudio.martella@**gmail.com <cl...@gmail.com>
>> >>>
>>
>> Can you try applying this one first?
>>
>> <<< <
>> http://www.mail-archive.com/__**user@giraph.apache.org/__**
>> msg00945/check.diff<http://www.mail-archive.com/__user@giraph.apache.org/__msg00945/check.diff>
>>
>> <http://www.mail-archive.com/**user@giraph.apache.org/**
>> msg00945/check.diff<http://www.mail-archive.com/user@giraph.apache.org/msg00945/check.diff>
>> ><<<<
>>
>>
>> On Mon, Oct 7, 2013 at 8:40 AM, Silvio Di
>> gregorio
>> <silvio.digregorio@gmail.com <mailto:silvio.digregorio@**
>> gmail.com <si...@gmail.com>>
>> <mailto:silvio.digregorio@__gm**ail.com <http://gmail.com>
>>
>> <ma...@gmail.com>>>>
>> wrote:
>>
>>
>>
>>
>>
>>
>>
>> *As i said i have builded*
>>
>>
>>
>>
>>
>> /giraph-examples-1.0.0-for-__**hadoop-2.0.0-cdh4.1.2-jar-__**
>> with-dependencies.jar/
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> *for cdh4, successfully. The job start to
>> monitoring the success rate:*
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> /13/10/07 08:28:45 INFO mapred.JobClient:
>> map 0% reduce 0%/
>>
>>
>>
>>
>> *but then*
>>
>>
>>
>>
>> /Error running child
>> java.lang.__**IllegalStateException: run:
>>
>> Caught an unrecoverable exception java.io.FileNotFoundException:
>> File _bsp/_defaultZkManagerDir/job_**
>> __201309181636_0678/_zkServer
>>
>> does not exist.
>> /
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> /.../
>>
>>
>>
>>
>>
>>
>>
>> /Caused by: java.lang.RuntimeException:
>> java.io.FileNotFoundException: File
>> _bsp/_defaultZkManagerDir/job_**__201309181636_0678/_zkServer
>> does
>>
>> not exist./
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> 2013/10/5 Silvio Di gregorio
>> <silvio.digregorio@gmail.com <mailto:silvio.digregorio@**
>> gmail.com <si...@gmail.com>>
>> <mailto:silvio.digregorio@__gm**ail.com <http://gmail.com>
>>
>> <ma...@gmail.com>
>> >>>
>>
>> I ha ve build w/ hadoop_cdh4.1.2
>> parameter.
>> Something is changed, monday i report the
>> result. Now the farm is closed.
>>
>> Il giorno 05/ott/2013 14:06, "Claudio
>> Martella"
>> <claudio.martella@gmail.com <ma...@gmail.com>
>> >
>> <mailto:claudio.martella@__gma**il.com <http://gmail.com>
>>
>> <mailto:claudio.martella@**gmail.com <cl...@gmail.com>>>>
>> ha scritto:
>>
>> Oh, right, -vof is in trunk. Anyway
>> it looks
>> like you built giraph for the wrong
>> profile.
>> You mentioned you're running on
>> 2.0, but
>> your giraph is built for 0.20.203
>> <tel:0.20.203>
>> <tel:0.20.203 <tel:0.20.203>>. try building with a profile
>>
>> for your hadoop version.
>>
>>
>> On Fri, Oct 4, 2013 at 2:35 PM,
>> Silvio Di
>> gregorio
>> <silvio.digregorio@gmail.com <mailto:silvio.digregorio@**
>> gmail.com <si...@gmail.com>>
>> <mailto:silvio.digregorio@__gm**ail.com <http://gmail.com>
>> <ma...@gmail.com>>>>
>> wrote:
>>
>>
>>
>> org.apache.commons.cli.__**UnrecognizedOptionException:
>>
>> Unrecognized option: -vof
>>
>> in 1.0 version is
>> -of,--outputFormat <arg>
>> Vertex output format
>> -op,--outputPath <arg>
>> Vertex output path
>>
>>
>>
>> 2013/10/4 Claudio Martella
>> <claudio.martella@gmail.com <ma...@gmail.com>
>> >
>> <mailto:claudio.martella@__gma**il.com <http://gmail.com>
>>
>> <mailto:claudio.martella@**gmail.com <cl...@gmail.com>
>> >>>
>>
>> did you try the argument
>> (-vof) i
>> suggested?
>>
>>
>> On Fri, Oct 4, 2013 at 2:13
>> PM,
>> Silvio Di gregorio
>> <silvio.digregorio@gmail.com <mailto:silvio.digregorio@**
>> gmail.com <si...@gmail.com>>
>> <mailto:silvio.digregorio@__gm**ail.com <http://gmail.com>
>>
>> <ma...@gmail.com>>>>
>> wrote:
>>
>>
>> i've specified -of
>>
>> org.apache.giraph.io.formats._**_IdWithValueTextOutputFormat
>>
>>
>> but the same error was
>> produced
>>
>> Exception in thread
>> "main"
>>
>> java.lang.__**IncompatibleClassChangeError:
>> Found interface
>>
>> org.apache.hadoop.mapreduce.__**JobContext,
>>
>> but class was expected
>> at
>>
>> org.apache.giraph.bsp.__**BspOutputFormat.__**checkOutputSpecs(__
>> **BspOutputFormat.java:43)
>> at
>>
>> org.apache.hadoop.mapred.__**JobClient$2.run(JobClient.__**
>> java:984)
>> at
>>
>> org.apache.hadoop.mapred.__**JobClient$2.run(JobClient.__**
>> java:945)
>> at
>>
>> java.security.__**AccessController.doPrivileged(**__Native
>> Method)
>> at
>>
>> javax.security.auth.Subject.__**doAs(Subject.java:396)
>> at
>>
>> org.apache.hadoop.security.__**UserGroupInformation.doAs(__**
>> UserGroupInformation.java:__**1408)
>> at
>>
>> org.apache.hadoop.mapred.__**JobClient.submitJobInternal(__**
>> JobClient.java:945)
>> at
>>
>> org.apache.hadoop.mapreduce.__**Job.submit(Job.java:566)
>> at
>>
>> org.apache.hadoop.mapreduce.__**Job.waitForCompletion(Job.__**
>> java:596)
>> at
>>
>> org.apache.giraph.job.__**GiraphJob.run(GiraphJob.java:_**_237)
>> at
>>
>> org.apache.giraph.__**GiraphRunner.run(GiraphRunner.**__java:94)
>> at
>>
>> org.apache.hadoop.util.__**ToolRunner.run(ToolRunner.__**java:70)
>> at
>>
>> org.apache.hadoop.util.__**ToolRunner.run(ToolRunner.__**java:84)
>> at
>>
>> org.apache.giraph.__**GiraphRunner.main(__**
>> GiraphRunner.java:124)
>> at
>>
>> sun.reflect.__**NativeMethodAccessorImpl.__**invoke0(Native
>> Method)
>> at
>>
>> sun.reflect.__**NativeMethodAccessorImpl.__**invoke(__**
>> NativeMethodAccessorImpl.java:**__39)
>> at
>>
>> sun.reflect.__**DelegatingMethodAccessorImpl._**_invoke(__**
>> DelegatingMethodAccessorImpl._**_java:25)
>> at
>>
>> java.lang.reflect.Method.__**invoke(Method.java:597)
>> at
>>
>> org.apache.hadoop.util.RunJar.**__main(RunJar.java:208)
>>
>>
>>
>>
>> 2013/10/4 Claudio
>> Martella
>> <claudio.martella@gmail.com <ma...@gmail.com>
>> >
>> <mailto:claudio.martella@__gma**il.com <http://gmail.com>
>>
>> <mailto:claudio.martella@**gmail.com <cl...@gmail.com>
>> >>>
>>
>> Hi,
>>
>> you need to specify
>> the
>> vertex outputformat
>> class
>> (-vof option), e.g.
>>
>> org.apache.giraph.io.formats._**_IdWithValueTextOutputFormat.
>>
>>
>>
>> On Fri, Oct 4, 2013
>> at 1:06
>> PM, Silvio Di
>> gregorio
>> <silvio.digregorio@gmail.com <mailto:silvio.digregorio@**
>> gmail.com <si...@gmail.com>>
>> <mailto:silvio.digregorio@__gm**ail.com <http://gmail.com>
>>
>> <ma...@gmail.com>
>> >>>
>> wrote:
>>
>>
>> Hi,
>>
>> I hope I have
>> sent to
>> the right
>> address.
>>
>> i have a graph
>> (directed
>> and unweighted)
>> stored
>> in hdfs like a
>> adjacency
>> list (140Milions
>> of
>> edges 6Milions
>> of vertex)
>>
>>
>> node<tab>neighbors
>>
>> 23 2 1 3
>> 4 3
>>
>> 1 999 99923 <tel:1%20999%2099923>
>> <tel:1%20999%2099923>
>> 909 .......
>>
>> ......
>>
>> hadoop version
>> Hadoop
>> 2.0.0-cdh4.3.0
>> - java 1.6
>>
>>
>> I have executed
>> the
>> giraph-1.0
>> connected
>> components
>> example, in
>> this fashion
>>
>> > hadoop jar
>>
>> /usr/local/giraph/giraph-__**examples/target/giraph-__**
>> examples-1.0.0-for-hadoop-0.20.203.0
>> <tel:0.20.203.0>
>> <tel:0.20.203.0 <tel:0.20.203.0>>-jar-with-dependencies.**__jar
>>
>> org.apache.giraph.GiraphRunner
>>
>> org.apache.giraph.examples.__**ConnectedComponentsVertex
>> -vif
>>
>> org.apache.giraph.io.formats._**_IntIntNullTextInputFormat
>>
>> -vip
>>
>> /user/hdfs/lista_adj_txt
>> -op
>> connectedgiraph
>> --workers 4
>>
>> and then fail
>> with:
>>
>> 13/10/04
>> 09:28:29 INFO
>>
>> utils.ConfigurationUtils: No
>> edge input format
>> specified.
>> Ensure your
>> InputFormat
>> does not
>> require one.
>>
>> 13/10/04
>> 09:28:29 INFO
>>
>> utils.ConfigurationUtils: No
>> output format
>> specified.
>> Ensure your
>> OutputFormat
>> does not
>> require one.
>>
>> 13/10/04
>> 09:28:30 INFO
>> job.GiraphJob:
>> run:
>> Since
>> checkpointing is
>> disabled
>> (default), do
>> not allow any
>> task
>> retries (setting
>>
>> mapred.map.max.attempts
>> = 0, old value =
>> 4)
>>
>> 13/10/04
>> 09:28:31 WARN
>>
>> mapred.JobClient: Use
>>
>> GenericOptionsParser for
>> parsing the
>> arguments.
>> Applications
>> should
>> implement Tool
>> for the same.
>>
>> 13/10/04
>> 09:28:31 INFO
>> mapred.JobClient:
>> Cleaning up the
>> staging
>> area
>>
>> hdfs://srv-bigdata-dev-01.int.**__sose.it:8020/user/hdfs/.__**
>> staging/job_201309181636_0535<http://sose.it:8020/user/hdfs/.__staging/job_201309181636_0535>
>> <http://srv-bigdata-dev-01.**int.sose.it:8020/user/hdfs/.**
>> staging/job_201309181636_0535<http://srv-bigdata-dev-01.int.sose.it:8020/user/hdfs/.staging/job_201309181636_0535>
>> >
>> <http://srv-bigdata-dev-01.__i**nt.sose.it:8020/user/hdfs/.__**
>> staging/job_201309181636_0535<http://int.sose.it:8020/user/hdfs/.__staging/job_201309181636_0535>
>>
>> <http://srv-bigdata-dev-01.**int.sose.it:8020/user/hdfs/.**
>> staging/job_201309181636_0535<http://srv-bigdata-dev-01.int.sose.it:8020/user/hdfs/.staging/job_201309181636_0535>
>> >**>
>>
>> Exception in
>> thread
>> "main"
>>
>> java.lang.__**IncompatibleClassChangeError:
>> Found interface
>>
>> org.apache.hadoop.mapreduce.__**JobContext,
>>
>> but class was
>> expected
>>
>> at
>>
>> org.apache.giraph.bsp.__**BspOutputFormat.__**checkOutputSpecs(__
>> **BspOutputFormat.java:43)
>>
>>
>> ......
>>
>> Thanks in advance
>>
>>
>>
>>
>>
>> --
>> Claudio Martella
>> claudio.martella@gmail.com <ma...@gmail.com>
>> >
>> <mailto:claudio.martella@__gma**il.com <http://gmail.com>
>>
>> <mailto:claudio.martella@**gmail.com <cl...@gmail.com>
>> >>
>>
>>
>>
>>
>>
>>
>> --
>> Claudio Martella
>> claudio.martella@gmail.com <ma...@gmail.com>
>> >
>> <mailto:claudio.martella@__gma**il.com <http://gmail.com>
>>
>> <mailto:claudio.martella@**gmail.com <cl...@gmail.com>
>> >>
>>
>>
>>
>>
>>
>> --
>> Claudio Martella
>> claudio.martella@gmail.com <ma...@gmail.com>
>> >
>> <mailto:claudio.martella@__gma**il.com <http://gmail.com>
>>
>> <mailto:claudio.martella@**gmail.com <cl...@gmail.com>
>> >>
>>
>>
>>
>>
>>
>> --
>> Claudio Martella
>> claudio.martella@gmail.com <ma...@gmail.com>
>> >
>> <mailto:claudio.martella@__gma**il.com <http://gmail.com>
>>
>> <mailto:claudio.martella@**gmail.com <cl...@gmail.com>
>> >>
>>
>>
>>
>>
>>
>> --
>> Claudio Martella
>> claudio.martella@gmail.com <ma...@gmail.com>
>> >
>> <mailto:claudio.martella@__gma**il.com <http://gmail.com>
>>
>> <mailto:claudio.martella@**gmail.com <cl...@gmail.com>
>> >>
>>
>>
>>
>>
>>
>> --
>> Claudio Martella
>> claudio.martella@gmail.com <ma...@gmail.com>
>> >
>> <mailto:claudio.martella@__gma**il.com <http://gmail.com>
>>
>> <mailto:claudio.martella@**gmail.com <cl...@gmail.com>
>> >>
>>
>>
>> --
>> Matthew Laird
>> Lead Software Developer, Bioinformatics
>> Brinkman Laboratory
>> Simon Fraser University, Burnaby, BC, Canada
>>
>>
> --
> Matthew Laird
> Lead Software Developer, Bioinformatics
> Brinkman Laboratory
> Simon Fraser University, Burnaby, BC, Canada
>
--
Claudio Martella
claudio.martella@gmail.com
Re: connected components example in giraph 1.0
Posted by Matthew Laird <la...@sfu.ca>.
Thanks, which is my next curiosity... if I head over to
http://giraph.apache.org/ I see at the top is says
"Version: 1.1.0-SNAPSHOT | Last Published: 2013-08-10"
However it's not clickable and the only download links I see are for
1.0.0. I'm assuming trunk in git isn't the same as the 1.1.0 snapshot
since more pushes have occurred since August, and I don't see a tag in
git for 1.1.0 (https://git-wip-us.apache.org/repos/asf?p=giraph.git).
So where does one get 1.1.0-SNAPSHOT from?
Thanks.
On 13-10-07 11:39 AM, Silvio Di gregorio wrote:
> Hi Matthew
> I use giraph 1.1.0 and cdh4 hadoop 0.23
>
> the patch of Marco:
>
> http://www.mail-archive.com/__user@giraph.apache.org/__msg00945/check.diff
> <http://www.mail-archive.com/user@giraph.apache.org/msg00945/check.diff>
>
> and build
>
> mvn clean install -P hadoop_cdh4.1.2
>
>
>
> Il giorno 07/ott/2013 20:20, "Matthew Laird" <lairdm@sfu.ca
> <ma...@sfu.ca>> ha scritto:
>
> Hello, so I'm watching this thread with interest as well, since
> that's the exact example I'll be trying to run as well, and I see
> Silvio appears to be asking about running Giraph 1.0 against Hadoop
> 2.0.0.
>
> I'm curious, is that a known working combination, 1.0 vs 2.0.x? And
> if so, are there patches to apply first and what maven command line
> should I build with? (ie. will a -Dhadoop.version=2.0.5 give the
> needed MR based API or will it use YARN?) I figure it's better to
> ask than aimlessly try all the combinations. :)
>
> Thanks!
>
> On 13-10-07 11:01 AM, Claudio Martella wrote:
>
> try to increase the mappers' heap space through
> mapred.child.java.opts.
>
>
> On Mon, Oct 7, 2013 at 1:18 PM, Silvio Di gregorio
> <silvio.digregorio@gmail.com
> <ma...@gmail.com>
> <mailto:silvio.digregorio@__gmail.com
> <ma...@gmail.com>>> wrote:
>
>
> I think now my issue is another, thanks to all.
>
>
>
>
> java.lang.__IllegalStateException: run: Caught an
> unrecoverable exception waitFor: ExecutionException occurred
> while waiting for
> org.apache.giraph.utils.__ProgressableUtils$__FutureWaitable@73a01e8a
> at
> org.apache.giraph.graph.__GraphMapper.run(GraphMapper.__java:102)
>
>
>
> ......
> Caused by: java.lang.__IllegalStateException: waitFor:
> ExecutionException occurred while waiting for
> org.apache.giraph.utils.__ProgressableUtils$__FutureWaitable@73a01e8a
> at
> org.apache.giraph.utils.__ProgressableUtils.waitFor(__ProgressableUtils.java:151)
>
>
>
> ... 7 more
> Caused by: java.util.concurrent.__ExecutionException:
> java.lang.OutOfMemoryError:*__Java heap space*
> at
> org.apache.giraph.utils.__ProgressableUtils.waitFor(__ProgressableUtils.java:143)
>
>
>
> ... 13 more
>
>
>
> 2013/10/7 Claudio Martella <claudio.martella@gmail.com
> <ma...@gmail.com>
> <mailto:claudio.martella@__gmail.com
> <ma...@gmail.com>>>
>
> OK, thanks.
>
> I really have to push that patch in.
>
>
> On Mon, Oct 7, 2013 at 12:17 PM, Silvio Di gregorio
> <silvio.digregorio@gmail.com <ma...@gmail.com>
> <mailto:silvio.digregorio@__gmail.com
> <ma...@gmail.com>>> wrote:
>
> yes i do,
> i have seen this in your post in:
> http://www.mail-archive.com/__user@giraph.apache.org/__msg00957.html
> <http://www.mail-archive.com/user@giraph.apache.org/msg00957.html>
>
> excuse me if i had checked in the mail-achive
> first I would
> have avoided the last post.
>
> Now zk issue are resolved.
>
>
> 2013/10/7 Claudio Martella
> <claudio.martella@gmail.com <ma...@gmail.com>
> <mailto:claudio.martella@__gmail.com
> <ma...@gmail.com>>>
>
> Can you try applying this one first?
>
> <<< <
> http://www.mail-archive.com/__user@giraph.apache.org/__msg00945/check.diff
> <http://www.mail-archive.com/user@giraph.apache.org/msg00945/check.diff><<<<
>
>
> On Mon, Oct 7, 2013 at 8:40 AM, Silvio Di gregorio
> <silvio.digregorio@gmail.com <ma...@gmail.com>
> <mailto:silvio.digregorio@__gmail.com
> <ma...@gmail.com>>> wrote:
>
>
>
>
>
>
>
> *As i said i have builded*
>
>
>
>
>
> /giraph-examples-1.0.0-for-__hadoop-2.0.0-cdh4.1.2-jar-__with-dependencies.jar/
>
>
>
>
>
>
>
>
>
>
>
>
>
> *for cdh4, successfully. The job start to
> monitoring the success rate:*
>
>
>
>
>
>
>
>
>
>
>
>
>
> /13/10/07 08:28:45 INFO mapred.JobClient:
> map 0% reduce 0%/
>
>
>
>
> *but then*
>
>
>
>
> /Error running child
> java.lang.__IllegalStateException: run:
> Caught an unrecoverable exception java.io.FileNotFoundException:
> File _bsp/_defaultZkManagerDir/job___201309181636_0678/_zkServer
> does not exist.
> /
>
>
>
>
>
>
>
>
>
>
>
>
>
> /.../
>
>
>
>
>
>
>
> /Caused by: java.lang.RuntimeException:
> java.io.FileNotFoundException: File
> _bsp/_defaultZkManagerDir/job___201309181636_0678/_zkServer does
> not exist./
>
>
>
>
>
>
>
>
>
>
> 2013/10/5 Silvio Di gregorio
> <silvio.digregorio@gmail.com <ma...@gmail.com>
> <mailto:silvio.digregorio@__gmail.com
> <ma...@gmail.com>>>
>
> I ha ve build w/ hadoop_cdh4.1.2 parameter.
> Something is changed, monday i report the
> result. Now the farm is closed.
>
> Il giorno 05/ott/2013 14:06, "Claudio
> Martella"
> <claudio.martella@gmail.com <ma...@gmail.com>
> <mailto:claudio.martella@__gmail.com
> <ma...@gmail.com>>> ha scritto:
>
> Oh, right, -vof is in trunk. Anyway
> it looks
> like you built giraph for the wrong
> profile.
> You mentioned you're running on
> 2.0, but
> your giraph is built for 0.20.203
> <tel:0.20.203>
> <tel:0.20.203 <tel:0.20.203>>. try building with a profile
> for your hadoop version.
>
>
> On Fri, Oct 4, 2013 at 2:35 PM,
> Silvio Di
> gregorio
> <silvio.digregorio@gmail.com <ma...@gmail.com>
> <mailto:silvio.digregorio@__gmail.com
> <ma...@gmail.com>>> wrote:
>
>
>
> org.apache.commons.cli.__UnrecognizedOptionException:
> Unrecognized option: -vof
>
> in 1.0 version is
> -of,--outputFormat <arg>
> Vertex output format
> -op,--outputPath <arg>
> Vertex output path
>
>
>
> 2013/10/4 Claudio Martella
> <claudio.martella@gmail.com <ma...@gmail.com>
> <mailto:claudio.martella@__gmail.com
> <ma...@gmail.com>>>
>
> did you try the argument
> (-vof) i
> suggested?
>
>
> On Fri, Oct 4, 2013 at 2:13 PM,
> Silvio Di gregorio
> <silvio.digregorio@gmail.com <ma...@gmail.com>
> <mailto:silvio.digregorio@__gmail.com
> <ma...@gmail.com>>> wrote:
>
>
> i've specified -of
>
> org.apache.giraph.io.formats.__IdWithValueTextOutputFormat
>
> but the same error was
> produced
>
> Exception in thread "main"
>
> java.lang.__IncompatibleClassChangeError:
> Found interface
>
> org.apache.hadoop.mapreduce.__JobContext,
> but class was expected
> at
>
> org.apache.giraph.bsp.__BspOutputFormat.__checkOutputSpecs(__BspOutputFormat.java:43)
> at
>
> org.apache.hadoop.mapred.__JobClient$2.run(JobClient.__java:984)
> at
>
> org.apache.hadoop.mapred.__JobClient$2.run(JobClient.__java:945)
> at
>
> java.security.__AccessController.doPrivileged(__Native
> Method)
> at
>
> javax.security.auth.Subject.__doAs(Subject.java:396)
> at
>
> org.apache.hadoop.security.__UserGroupInformation.doAs(__UserGroupInformation.java:__1408)
> at
>
> org.apache.hadoop.mapred.__JobClient.submitJobInternal(__JobClient.java:945)
> at
>
> org.apache.hadoop.mapreduce.__Job.submit(Job.java:566)
> at
>
> org.apache.hadoop.mapreduce.__Job.waitForCompletion(Job.__java:596)
> at
>
> org.apache.giraph.job.__GiraphJob.run(GiraphJob.java:__237)
> at
>
> org.apache.giraph.__GiraphRunner.run(GiraphRunner.__java:94)
> at
>
> org.apache.hadoop.util.__ToolRunner.run(ToolRunner.__java:70)
> at
>
> org.apache.hadoop.util.__ToolRunner.run(ToolRunner.__java:84)
> at
>
> org.apache.giraph.__GiraphRunner.main(__GiraphRunner.java:124)
> at
>
> sun.reflect.__NativeMethodAccessorImpl.__invoke0(Native
> Method)
> at
>
> sun.reflect.__NativeMethodAccessorImpl.__invoke(__NativeMethodAccessorImpl.java:__39)
> at
>
> sun.reflect.__DelegatingMethodAccessorImpl.__invoke(__DelegatingMethodAccessorImpl.__java:25)
> at
>
> java.lang.reflect.Method.__invoke(Method.java:597)
> at
>
> org.apache.hadoop.util.RunJar.__main(RunJar.java:208)
>
>
>
> 2013/10/4 Claudio Martella
> <claudio.martella@gmail.com <ma...@gmail.com>
> <mailto:claudio.martella@__gmail.com
> <ma...@gmail.com>>>
>
> Hi,
>
> you need to specify the
> vertex outputformat
> class
> (-vof option), e.g.
>
> org.apache.giraph.io.formats.__IdWithValueTextOutputFormat.
>
>
> On Fri, Oct 4, 2013
> at 1:06
> PM, Silvio Di gregorio
> <silvio.digregorio@gmail.com <ma...@gmail.com>
> <mailto:silvio.digregorio@__gmail.com
> <ma...@gmail.com>>>
> wrote:
>
>
> Hi,
>
> I hope I have
> sent to
> the right address.
>
> i have a graph
> (directed
> and unweighted)
> stored
> in hdfs like a
> adjacency
> list (140Milions of
> edges 6Milions
> of vertex)
>
> node<tab>neighbors
>
> 23 2 1 3
> 4 3
>
> 1 999 99923 <tel:1%20999%2099923>
> <tel:1%20999%2099923>
> 909 .......
>
> ......
>
> hadoop version
> Hadoop
> 2.0.0-cdh4.3.0
> - java 1.6
>
>
> I have executed the
> giraph-1.0
> connected
> components
> example, in
> this fashion
>
> > hadoop jar
>
> /usr/local/giraph/giraph-__examples/target/giraph-__examples-1.0.0-for-hadoop-0.20.203.0
> <tel:0.20.203.0>
> <tel:0.20.203.0 <tel:0.20.203.0>>-jar-with-dependencies.__jar
>
> org.apache.giraph.GiraphRunner
>
> org.apache.giraph.examples.__ConnectedComponentsVertex
> -vif
>
> org.apache.giraph.io.formats.__IntIntNullTextInputFormat
> -vip
>
> /user/hdfs/lista_adj_txt
> -op
> connectedgiraph
> --workers 4
>
> and then fail with:
>
> 13/10/04
> 09:28:29 INFO
>
> utils.ConfigurationUtils: No
> edge input format
> specified.
> Ensure your
> InputFormat
> does not
> require one.
>
> 13/10/04
> 09:28:29 INFO
>
> utils.ConfigurationUtils: No
> output format
> specified.
> Ensure your
> OutputFormat
> does not
> require one.
>
> 13/10/04
> 09:28:30 INFO
> job.GiraphJob: run:
> Since
> checkpointing is
> disabled
> (default), do
> not allow any task
> retries (setting
>
> mapred.map.max.attempts
> = 0, old value = 4)
>
> 13/10/04
> 09:28:31 WARN
>
> mapred.JobClient: Use
>
> GenericOptionsParser for
> parsing the
> arguments.
> Applications should
> implement Tool
> for the same.
>
> 13/10/04
> 09:28:31 INFO
> mapred.JobClient:
> Cleaning up the
> staging
> area
>
> hdfs://srv-bigdata-dev-01.int.__sose.it:8020/user/hdfs/.__staging/job_201309181636_0535
> <http://srv-bigdata-dev-01.int.sose.it:8020/user/hdfs/.staging/job_201309181636_0535>
> <http://srv-bigdata-dev-01.__int.sose.it:8020/user/hdfs/.__staging/job_201309181636_0535
> <http://srv-bigdata-dev-01.int.sose.it:8020/user/hdfs/.staging/job_201309181636_0535>>
>
> Exception in thread
> "main"
>
> java.lang.__IncompatibleClassChangeError:
> Found interface
>
> org.apache.hadoop.mapreduce.__JobContext,
> but class was
> expected
>
> at
>
> org.apache.giraph.bsp.__BspOutputFormat.__checkOutputSpecs(__BspOutputFormat.java:43)
>
> ......
>
> Thanks in advance
>
>
>
>
>
> --
> Claudio Martella
> claudio.martella@gmail.com <ma...@gmail.com>
> <mailto:claudio.martella@__gmail.com
> <ma...@gmail.com>>
>
>
>
>
>
>
> --
> Claudio Martella
> claudio.martella@gmail.com <ma...@gmail.com>
> <mailto:claudio.martella@__gmail.com
> <ma...@gmail.com>>
>
>
>
>
>
> --
> Claudio Martella
> claudio.martella@gmail.com <ma...@gmail.com>
> <mailto:claudio.martella@__gmail.com
> <ma...@gmail.com>>
>
>
>
>
>
> --
> Claudio Martella
> claudio.martella@gmail.com <ma...@gmail.com>
> <mailto:claudio.martella@__gmail.com
> <ma...@gmail.com>>
>
>
>
>
>
> --
> Claudio Martella
> claudio.martella@gmail.com <ma...@gmail.com>
> <mailto:claudio.martella@__gmail.com
> <ma...@gmail.com>>
>
>
>
>
>
> --
> Claudio Martella
> claudio.martella@gmail.com <ma...@gmail.com>
> <mailto:claudio.martella@__gmail.com
> <ma...@gmail.com>>
>
>
> --
> Matthew Laird
> Lead Software Developer, Bioinformatics
> Brinkman Laboratory
> Simon Fraser University, Burnaby, BC, Canada
>
--
Matthew Laird
Lead Software Developer, Bioinformatics
Brinkman Laboratory
Simon Fraser University, Burnaby, BC, Canada
Re: connected components example in giraph 1.0
Posted by Silvio Di gregorio <si...@gmail.com>.
Hi Matthew
I use giraph 1.1.0 and cdh4 hadoop 0.23
the patch of Marco:
http://www.mail-archive.com/**user@giraph.apache.org/**msg00945/check.diff<http://www.mail-archive.com/user@giraph.apache.org/msg00945/check.diff>
and build
mvn clean install -P hadoop_cdh4.1.2
Il giorno 07/ott/2013 20:20, "Matthew Laird" <la...@sfu.ca> ha scritto:
> Hello, so I'm watching this thread with interest as well, since that's the
> exact example I'll be trying to run as well, and I see Silvio appears to be
> asking about running Giraph 1.0 against Hadoop 2.0.0.
>
> I'm curious, is that a known working combination, 1.0 vs 2.0.x? And if
> so, are there patches to apply first and what maven command line should I
> build with? (ie. will a -Dhadoop.version=2.0.5 give the needed MR based API
> or will it use YARN?) I figure it's better to ask than aimlessly try all
> the combinations. :)
>
> Thanks!
>
> On 13-10-07 11:01 AM, Claudio Martella wrote:
>
>> try to increase the mappers' heap space through mapred.child.java.opts.
>>
>>
>> On Mon, Oct 7, 2013 at 1:18 PM, Silvio Di gregorio
>> <silvio.digregorio@gmail.com <ma...@gmail.com>>>
>> wrote:
>>
>>
>> I think now my issue is another, thanks to all.
>>
>>
>>
>>
>> java.lang.**IllegalStateException: run: Caught an unrecoverable
>> exception waitFor: ExecutionException occurred while waiting for
>> org.apache.giraph.utils.**ProgressableUtils$**FutureWaitable@73a01e8a
>> at org.apache.giraph.graph.**GraphMapper.run(GraphMapper.**
>> java:102)
>>
>>
>>
>> ......
>> Caused by: java.lang.**IllegalStateException: waitFor:
>> ExecutionException occurred while waiting for org.apache.giraph.utils.**
>> ProgressableUtils$**FutureWaitable@73a01e8a
>> at org.apache.giraph.utils.**ProgressableUtils.waitFor(**
>> ProgressableUtils.java:151)
>>
>>
>>
>> ... 7 more
>> Caused by: java.util.concurrent.**ExecutionException:
>> java.lang.OutOfMemoryError:***Java heap space*
>> at org.apache.giraph.utils.**ProgressableUtils.waitFor(**
>> ProgressableUtils.java:143)
>>
>>
>>
>> ... 13 more
>>
>>
>>
>> 2013/10/7 Claudio Martella <claudio.martella@gmail.com
>> <mailto:claudio.martella@**gmail.com <cl...@gmail.com>>>
>>
>> OK, thanks.
>>
>> I really have to push that patch in.
>>
>>
>> On Mon, Oct 7, 2013 at 12:17 PM, Silvio Di gregorio
>> <silvio.digregorio@gmail.com
>> <ma...@gmail.com>>>
>> wrote:
>>
>> yes i do,
>> i have seen this in your post in:
>> http://www.mail-archive.com/**user@giraph.apache.org/**
>> msg00957.html<http://www.mail-archive.com/user@giraph.apache.org/msg00957.html>
>>
>> excuse me if i had checked in the mail-achive first I would
>> have avoided the last post.
>>
>> Now zk issue are resolved.
>>
>>
>> 2013/10/7 Claudio Martella <claudio.martella@gmail.com
>> <ma...@gmail.com>
>> >>
>>
>> Can you try applying this one first?
>>
>> <<< < http://www.mail-archive.com/**user@giraph.apache.org/
>> **msg00945/check.diff<http://www.mail-archive.com/user@giraph.apache.org/msg00945/check.diff>
>> <<<<
>>
>>
>> On Mon, Oct 7, 2013 at 8:40 AM, Silvio Di gregorio
>> <silvio.digregorio@gmail.com
>> <ma...@gmail.com>>>
>> wrote:
>>
>>
>>
>>
>>
>>
>>
>> *As i said i have builded*
>>
>>
>>
>>
>> /giraph-examples-1.0.0-for-**
>> hadoop-2.0.0-cdh4.1.2-jar-**with-dependencies.jar/
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> *for cdh4, successfully. The job start to monitoring
>> the success rate:*
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> /13/10/07 08:28:45 INFO mapred.JobClient: map 0%
>> reduce 0%/
>>
>>
>>
>>
>> *but then*
>>
>>
>>
>>
>> /Error running child
>> java.lang.**IllegalStateException: run: Caught an
>> unrecoverable exception java.io.FileNotFoundException: File
>> _bsp/_defaultZkManagerDir/job_**201309181636_0678/_zkServer does not
>> exist.
>> /
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> /.../
>>
>>
>>
>>
>>
>>
>>
>> /Caused by: java.lang.RuntimeException:
>> java.io.FileNotFoundException: File _bsp/_defaultZkManagerDir/job_**201309181636_0678/_zkServer
>> does not exist./
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> 2013/10/5 Silvio Di gregorio
>> <silvio.digregorio@gmail.com
>> <ma...@gmail.com>
>> >>
>>
>> I ha ve build w/ hadoop_cdh4.1.2 parameter.
>> Something is changed, monday i report the
>> result. Now the farm is closed.
>>
>> Il giorno 05/ott/2013 14:06, "Claudio Martella"
>> <claudio.martella@gmail.com
>> <ma...@gmail.com>>>
>> ha scritto:
>>
>> Oh, right, -vof is in trunk. Anyway it looks
>> like you built giraph for the wrong profile.
>> You mentioned you're running on 2.0, but
>> your giraph is built for 0.20.203
>> <tel:0.20.203>. try building with a profile
>> for your hadoop version.
>>
>>
>> On Fri, Oct 4, 2013 at 2:35 PM, Silvio Di
>> gregorio <silvio.digregorio@gmail.com
>> <ma...@gmail.com>>>
>> wrote:
>>
>>
>> org.apache.commons.cli.**
>> UnrecognizedOptionException:
>> Unrecognized option: -vof
>>
>> in 1.0 version is
>> -of,--outputFormat <arg>
>> Vertex output format
>> -op,--outputPath <arg>
>> Vertex output path
>>
>>
>>
>> 2013/10/4 Claudio Martella
>> <claudio.martella@gmail.com
>> <ma...@gmail.com>
>> >>
>>
>> did you try the argument (-vof) i
>> suggested?
>>
>>
>> On Fri, Oct 4, 2013 at 2:13 PM,
>> Silvio Di gregorio
>> <silvio.digregorio@gmail.com
>> <ma...@gmail.com>>>
>> wrote:
>>
>>
>> i've specified -of
>> org.apache.giraph.io.formats.**
>> IdWithValueTextOutputFormat
>>
>> but the same error was produced
>>
>> Exception in thread "main"
>> java.lang.**
>> IncompatibleClassChangeError:
>> Found interface
>> org.apache.hadoop.mapreduce.**
>> JobContext,
>> but class was expected
>> at
>> org.apache.giraph.bsp.**
>> BspOutputFormat.**checkOutputSpecs(**BspOutputFormat.java:43)
>> at
>> org.apache.hadoop.mapred.**
>> JobClient$2.run(JobClient.**java:984)
>> at
>> org.apache.hadoop.mapred.**
>> JobClient$2.run(JobClient.**java:945)
>> at
>> java.security.**
>> AccessController.doPrivileged(**Native
>> Method)
>> at
>> javax.security.auth.Subject.**
>> doAs(Subject.java:396)
>> at
>> org.apache.hadoop.security.**
>> UserGroupInformation.doAs(**UserGroupInformation.java:**1408)
>> at
>> org.apache.hadoop.mapred.**
>> JobClient.submitJobInternal(**JobClient.java:945)
>> at
>> org.apache.hadoop.mapreduce.**
>> Job.submit(Job.java:566)
>> at
>> org.apache.hadoop.mapreduce.**
>> Job.waitForCompletion(Job.**java:596)
>> at
>> org.apache.giraph.job.**
>> GiraphJob.run(GiraphJob.java:**237)
>> at
>> org.apache.giraph.**
>> GiraphRunner.run(GiraphRunner.**java:94)
>> at
>> org.apache.hadoop.util.**
>> ToolRunner.run(ToolRunner.**java:70)
>> at
>> org.apache.hadoop.util.**
>> ToolRunner.run(ToolRunner.**java:84)
>> at
>> org.apache.giraph.**
>> GiraphRunner.main(**GiraphRunner.java:124)
>> at
>> sun.reflect.**
>> NativeMethodAccessorImpl.**invoke0(Native
>> Method)
>> at
>> sun.reflect.**
>> NativeMethodAccessorImpl.**invoke(**NativeMethodAccessorImpl.java:**39)
>> at
>> sun.reflect.**
>> DelegatingMethodAccessorImpl.**invoke(**DelegatingMethodAccessorImpl.**
>> java:25)
>> at
>> java.lang.reflect.Method.**
>> invoke(Method.java:597)
>> at
>> org.apache.hadoop.util.RunJar.**
>> main(RunJar.java:208)
>>
>>
>>
>> 2013/10/4 Claudio Martella
>> <claudio.martella@gmail.com
>> <mailto:claudio.martella@**
>> gmail.com <cl...@gmail.com>>>
>>
>> Hi,
>>
>> you need to specify the
>> vertex outputformat class
>> (-vof option), e.g.
>> org.apache.giraph.io.formats.
>> **IdWithValueTextOutputFormat.
>>
>>
>> On Fri, Oct 4, 2013 at 1:06
>> PM, Silvio Di gregorio
>> <silvio.digregorio@gmail.com
>> <mailto:silvio.digregorio@**
>> gmail.com <si...@gmail.com>>>
>> wrote:
>>
>>
>> Hi,
>>
>> I hope I have sent to
>> the right address.
>>
>> i have a graph (directed
>> and unweighted) stored
>> in hdfs like a adjacency
>> list (140Milions of
>> edges 6Milions of vertex)
>>
>> node<tab>neighbors
>>
>> 23 2 1 3 4 3
>>
>> 1 999 99923
>> <tel:1%20999%2099923>
>> 909 .......
>>
>> ......
>>
>> hadoop version Hadoop
>> 2.0.0-cdh4.3.0 - java 1.6
>>
>>
>> I have executed the
>> giraph-1.0 connected
>> components example, in
>> this fashion
>>
>> > hadoop jar
>>
>> /usr/local/giraph/giraph-**examples/target/giraph-**
>> examples-1.0.0-for-hadoop-0.20.203.0
>> <tel:0.20.203.0
>> >-jar-with-dependencies.**jar
>>
>> org.apache.giraph.GiraphRunner
>>
>> org.apache.giraph.examples.**ConnectedComponentsVertex
>> -vif
>>
>> org.apache.giraph.io.formats.**IntIntNullTextInputFormat
>> -vip
>>
>> /user/hdfs/lista_adj_txt
>> -op connectedgiraph
>> --workers 4
>>
>> and then fail with:
>>
>> 13/10/04 09:28:29 INFO
>> utils.ConfigurationUtils:
>> No
>> edge input format
>> specified. Ensure your
>> InputFormat does not
>> require one.
>>
>> 13/10/04 09:28:29 INFO
>> utils.ConfigurationUtils:
>> No
>> output format specified.
>> Ensure your OutputFormat
>> does not require one.
>>
>> 13/10/04 09:28:30 INFO
>> job.GiraphJob: run:
>> Since checkpointing is
>> disabled (default), do
>> not allow any task
>> retries (setting
>> mapred.map.max.attempts
>> = 0, old value = 4)
>>
>> 13/10/04 09:28:31 WARN
>> mapred.JobClient: Use
>> GenericOptionsParser for
>> parsing the arguments.
>> Applications should
>> implement Tool for the
>> same.
>>
>> 13/10/04 09:28:31 INFO
>> mapred.JobClient:
>> Cleaning up the staging
>> area
>> hdfs://
>> srv-bigdata-dev-01.int.**sose.it:8020/user/hdfs/.**
>> staging/job_201309181636_0535<http://srv-bigdata-dev-01.int.sose.it:8020/user/hdfs/.staging/job_201309181636_0535>
>> <
>> http://srv-bigdata-dev-01.**int.sose.it:8020/user/hdfs/.**
>> staging/job_201309181636_0535<http://srv-bigdata-dev-01.int.sose.it:8020/user/hdfs/.staging/job_201309181636_0535>
>> >
>>
>> Exception in thread
>> "main"
>> java.lang.**
>> IncompatibleClassChangeError:
>> Found interface
>>
>> org.apache.hadoop.mapreduce.**JobContext,
>> but class was expected
>>
>> at
>> org.apache.giraph.bsp.**
>> BspOutputFormat.**checkOutputSpecs(**BspOutputFormat.java:43)
>>
>> ......
>>
>> Thanks in advance
>>
>>
>>
>>
>>
>> --
>> Claudio Martella
>> claudio.martella@gmail.com
>> <mailto:claudio.martella@**
>> gmail.com <cl...@gmail.com>>
>>
>>
>>
>>
>>
>>
>> --
>> Claudio Martella
>> claudio.martella@gmail.com
>> <ma...@gmail.com>
>> >
>>
>>
>>
>>
>>
>> --
>> Claudio Martella
>> claudio.martella@gmail.com
>> <ma...@gmail.com>
>> >
>>
>>
>>
>>
>>
>> --
>> Claudio Martella
>> claudio.martella@gmail.com
>> <ma...@gmail.com>
>> >
>>
>>
>>
>>
>>
>> --
>> Claudio Martella
>> claudio.martella@gmail.com <ma...@gmail.com>
>> >
>>
>>
>>
>>
>>
>> --
>> Claudio Martella
>> claudio.martella@gmail.com <ma...@gmail.com>
>> >
>>
>
> --
> Matthew Laird
> Lead Software Developer, Bioinformatics
> Brinkman Laboratory
> Simon Fraser University, Burnaby, BC, Canada
>
Re: connected components example in giraph 1.0
Posted by Matthew Laird <la...@sfu.ca>.
Hello, so I'm watching this thread with interest as well, since that's
the exact example I'll be trying to run as well, and I see Silvio
appears to be asking about running Giraph 1.0 against Hadoop 2.0.0.
I'm curious, is that a known working combination, 1.0 vs 2.0.x? And if
so, are there patches to apply first and what maven command line should
I build with? (ie. will a -Dhadoop.version=2.0.5 give the needed MR
based API or will it use YARN?) I figure it's better to ask than
aimlessly try all the combinations. :)
Thanks!
On 13-10-07 11:01 AM, Claudio Martella wrote:
> try to increase the mappers' heap space through mapred.child.java.opts.
>
>
> On Mon, Oct 7, 2013 at 1:18 PM, Silvio Di gregorio
> <silvio.digregorio@gmail.com <ma...@gmail.com>> wrote:
>
>
> I think now my issue is another, thanks to all.
>
>
>
>
> java.lang.IllegalStateException: run: Caught an unrecoverable exception waitFor: ExecutionException occurred while waiting for org.apache.giraph.utils.ProgressableUtils$FutureWaitable@73a01e8a
> at org.apache.giraph.graph.GraphMapper.run(GraphMapper.java:102)
>
>
>
> ......
> Caused by: java.lang.IllegalStateException: waitFor: ExecutionException occurred while waiting for org.apache.giraph.utils.ProgressableUtils$FutureWaitable@73a01e8a
> at org.apache.giraph.utils.ProgressableUtils.waitFor(ProgressableUtils.java:151)
>
>
>
> ... 7 more
> Caused by: java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError:*Java heap space*
> at org.apache.giraph.utils.ProgressableUtils.waitFor(ProgressableUtils.java:143)
>
>
>
> ... 13 more
>
>
>
> 2013/10/7 Claudio Martella <claudio.martella@gmail.com
> <ma...@gmail.com>>
>
> OK, thanks.
>
> I really have to push that patch in.
>
>
> On Mon, Oct 7, 2013 at 12:17 PM, Silvio Di gregorio
> <silvio.digregorio@gmail.com
> <ma...@gmail.com>> wrote:
>
> yes i do,
> i have seen this in your post in:
> http://www.mail-archive.com/user@giraph.apache.org/msg00957.html
>
> excuse me if i had checked in the mail-achive first I would
> have avoided the last post.
>
> Now zk issue are resolved.
>
>
> 2013/10/7 Claudio Martella <claudio.martella@gmail.com
> <ma...@gmail.com>>
>
> Can you try applying this one first?
>
> http://www.mail-archive.com/user@giraph.apache.org/msg00945/check.diff
>
>
> On Mon, Oct 7, 2013 at 8:40 AM, Silvio Di gregorio
> <silvio.digregorio@gmail.com
> <ma...@gmail.com>> wrote:
>
>
>
>
>
>
>
> *As i said i have builded*
>
>
>
>
> /giraph-examples-1.0.0-for-hadoop-2.0.0-cdh4.1.2-jar-with-dependencies.jar/
>
>
>
>
>
>
>
>
>
>
>
>
>
> *for cdh4, successfully. The job start to monitoring the success rate:*
>
>
>
>
>
>
>
>
>
>
>
>
>
> /13/10/07 08:28:45 INFO mapred.JobClient: map 0% reduce 0%/
>
>
>
>
> *but then*
>
>
>
>
> /Error running child
> java.lang.IllegalStateException: run: Caught an unrecoverable exception java.io.FileNotFoundException: File _bsp/_defaultZkManagerDir/job_201309181636_0678/_zkServer does not exist.
> /
>
>
>
>
>
>
>
>
>
>
>
>
>
> /.../
>
>
>
>
>
>
>
> /Caused by: java.lang.RuntimeException: java.io.FileNotFoundException: File _bsp/_defaultZkManagerDir/job_201309181636_0678/_zkServer does not exist./
>
>
>
>
>
>
>
>
>
>
> 2013/10/5 Silvio Di gregorio
> <silvio.digregorio@gmail.com
> <ma...@gmail.com>>
>
> I ha ve build w/ hadoop_cdh4.1.2 parameter.
> Something is changed, monday i report the
> result. Now the farm is closed.
>
> Il giorno 05/ott/2013 14:06, "Claudio Martella"
> <claudio.martella@gmail.com
> <ma...@gmail.com>> ha scritto:
>
> Oh, right, -vof is in trunk. Anyway it looks
> like you built giraph for the wrong profile.
> You mentioned you're running on 2.0, but
> your giraph is built for 0.20.203
> <tel:0.20.203>. try building with a profile
> for your hadoop version.
>
>
> On Fri, Oct 4, 2013 at 2:35 PM, Silvio Di
> gregorio <silvio.digregorio@gmail.com
> <ma...@gmail.com>> wrote:
>
>
> org.apache.commons.cli.UnrecognizedOptionException:
> Unrecognized option: -vof
>
> in 1.0 version is
> -of,--outputFormat <arg>
> Vertex output format
> -op,--outputPath <arg>
> Vertex output path
>
>
>
> 2013/10/4 Claudio Martella
> <claudio.martella@gmail.com
> <ma...@gmail.com>>
>
> did you try the argument (-vof) i
> suggested?
>
>
> On Fri, Oct 4, 2013 at 2:13 PM,
> Silvio Di gregorio
> <silvio.digregorio@gmail.com
> <ma...@gmail.com>> wrote:
>
>
> i've specified -of
> org.apache.giraph.io.formats.IdWithValueTextOutputFormat
>
> but the same error was produced
>
> Exception in thread "main"
> java.lang.IncompatibleClassChangeError:
> Found interface
> org.apache.hadoop.mapreduce.JobContext,
> but class was expected
> at
> org.apache.giraph.bsp.BspOutputFormat.checkOutputSpecs(BspOutputFormat.java:43)
> at
> org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:984)
> at
> org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:945)
> at
> java.security.AccessController.doPrivileged(Native
> Method)
> at
> javax.security.auth.Subject.doAs(Subject.java:396)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
> at
> org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:945)
> at
> org.apache.hadoop.mapreduce.Job.submit(Job.java:566)
> at
> org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:596)
> at
> org.apache.giraph.job.GiraphJob.run(GiraphJob.java:237)
> at
> org.apache.giraph.GiraphRunner.run(GiraphRunner.java:94)
> at
> org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at
> org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
> at
> org.apache.giraph.GiraphRunner.main(GiraphRunner.java:124)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at
> java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.hadoop.util.RunJar.main(RunJar.java:208)
>
>
>
> 2013/10/4 Claudio Martella
> <claudio.martella@gmail.com
> <ma...@gmail.com>>
>
> Hi,
>
> you need to specify the
> vertex outputformat class
> (-vof option), e.g.
> org.apache.giraph.io.formats.IdWithValueTextOutputFormat.
>
>
> On Fri, Oct 4, 2013 at 1:06
> PM, Silvio Di gregorio
> <silvio.digregorio@gmail.com
> <ma...@gmail.com>>
> wrote:
>
>
> Hi,
>
> I hope I have sent to
> the right address.
>
> i have a graph (directed
> and unweighted) stored
> in hdfs like a adjacency
> list (140Milions of
> edges 6Milions of vertex)
>
> node<tab>neighbors
>
> 23 2 1 3 4 3
>
> 1 999 99923
> <tel:1%20999%2099923>
> 909 .......
>
> ......
>
> hadoop version Hadoop
> 2.0.0-cdh4.3.0 - java 1.6
>
>
> I have executed the
> giraph-1.0 connected
> components example, in
> this fashion
>
> > hadoop jar
> /usr/local/giraph/giraph-examples/target/giraph-examples-1.0.0-for-hadoop-0.20.203.0
> <tel:0.20.203.0>-jar-with-dependencies.jar
> org.apache.giraph.GiraphRunner
> org.apache.giraph.examples.ConnectedComponentsVertex
> -vif
> org.apache.giraph.io.formats.IntIntNullTextInputFormat
> -vip
> /user/hdfs/lista_adj_txt
> -op connectedgiraph
> --workers 4
>
> and then fail with:
>
> 13/10/04 09:28:29 INFO
> utils.ConfigurationUtils: No
> edge input format
> specified. Ensure your
> InputFormat does not
> require one.
>
> 13/10/04 09:28:29 INFO
> utils.ConfigurationUtils: No
> output format specified.
> Ensure your OutputFormat
> does not require one.
>
> 13/10/04 09:28:30 INFO
> job.GiraphJob: run:
> Since checkpointing is
> disabled (default), do
> not allow any task
> retries (setting
> mapred.map.max.attempts
> = 0, old value = 4)
>
> 13/10/04 09:28:31 WARN
> mapred.JobClient: Use
> GenericOptionsParser for
> parsing the arguments.
> Applications should
> implement Tool for the same.
>
> 13/10/04 09:28:31 INFO
> mapred.JobClient:
> Cleaning up the staging
> area
> hdfs://srv-bigdata-dev-01.int.sose.it:8020/user/hdfs/.staging/job_201309181636_0535
> <http://srv-bigdata-dev-01.int.sose.it:8020/user/hdfs/.staging/job_201309181636_0535>
>
> Exception in thread
> "main"
> java.lang.IncompatibleClassChangeError:
> Found interface
> org.apache.hadoop.mapreduce.JobContext,
> but class was expected
>
> at
> org.apache.giraph.bsp.BspOutputFormat.checkOutputSpecs(BspOutputFormat.java:43)
>
> ......
>
> Thanks in advance
>
>
>
>
>
> --
> Claudio Martella
> claudio.martella@gmail.com
> <ma...@gmail.com>
>
>
>
>
>
>
> --
> Claudio Martella
> claudio.martella@gmail.com
> <ma...@gmail.com>
>
>
>
>
>
> --
> Claudio Martella
> claudio.martella@gmail.com
> <ma...@gmail.com>
>
>
>
>
>
> --
> Claudio Martella
> claudio.martella@gmail.com
> <ma...@gmail.com>
>
>
>
>
>
> --
> Claudio Martella
> claudio.martella@gmail.com <ma...@gmail.com>
>
>
>
>
>
> --
> Claudio Martella
> claudio.martella@gmail.com <ma...@gmail.com>
--
Matthew Laird
Lead Software Developer, Bioinformatics
Brinkman Laboratory
Simon Fraser University, Burnaby, BC, Canada
Re: connected components example in giraph 1.0
Posted by Claudio Martella <cl...@gmail.com>.
try to increase the mappers' heap space through mapred.child.java.opts.
On Mon, Oct 7, 2013 at 1:18 PM, Silvio Di gregorio <
silvio.digregorio@gmail.com> wrote:
> I think now my issue is another, thanks to all.
>
>
>
> java.lang.IllegalStateException: run: Caught an unrecoverable exception waitFor: ExecutionException occurred while waiting for org.apache.giraph.utils.ProgressableUtils$FutureWaitable@73a01e8a
> at org.apache.giraph.graph.GraphMapper.run(GraphMapper.java:102)
>
>
> ......
> Caused by: java.lang.IllegalStateException: waitFor: ExecutionException occurred while waiting for org.apache.giraph.utils.ProgressableUtils$FutureWaitable@73a01e8a
> at org.apache.giraph.utils.ProgressableUtils.waitFor(ProgressableUtils.java:151)
>
>
> ... 7 more
> Caused by: java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: *Java heap space*
> at org.apache.giraph.utils.ProgressableUtils.waitFor(ProgressableUtils.java:143)
>
>
> ... 13 more
>
>
>
> 2013/10/7 Claudio Martella <cl...@gmail.com>
>
>> OK, thanks.
>>
>> I really have to push that patch in.
>>
>>
>> On Mon, Oct 7, 2013 at 12:17 PM, Silvio Di gregorio <
>> silvio.digregorio@gmail.com> wrote:
>>
>>> yes i do,
>>> i have seen this in your post in:
>>> http://www.mail-archive.com/user@giraph.apache.org/msg00957.html
>>>
>>> excuse me if i had checked in the mail-achive first I would have
>>> avoided the last post.
>>>
>>> Now zk issue are resolved.
>>>
>>>
>>> 2013/10/7 Claudio Martella <cl...@gmail.com>
>>>
>>>> Can you try applying this one first?
>>>>
>>>> http://www.mail-archive.com/user@giraph.apache.org/msg00945/check.diff
>>>>
>>>>
>>>> On Mon, Oct 7, 2013 at 8:40 AM, Silvio Di gregorio <
>>>> silvio.digregorio@gmail.com> wrote:
>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> *As i said i have builded*
>>>>>
>>>>>
>>>>>
>>>>> *giraph-examples-1.0.0-for-hadoop-2.0.0-cdh4.1.2-jar-with-dependencies.jar*
>>>>>
>>>>>
>>>>>
>>>>> *for cdh4, successfully. The job start to monitoring the success rate:*
>>>>>
>>>>>
>>>>>
>>>>> *13/10/07 08:28:45 INFO mapred.JobClient: map 0% reduce 0%*
>>>>>
>>>>>
>>>>>
>>>>> *but then*
>>>>>
>>>>>
>>>>>
>>>>> *Error running child
>>>>> java.lang.IllegalStateException: run: Caught an unrecoverable exception java.io.FileNotFoundException: File _bsp/_defaultZkManagerDir/job_201309181636_0678/_zkServer does not exist.
>>>>> *
>>>>>
>>>>>
>>>>>
>>>>> *...*
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> *Caused by: java.lang.RuntimeException: java.io.FileNotFoundException: File _bsp/_defaultZkManagerDir/job_201309181636_0678/_zkServer does not exist.*
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> 2013/10/5 Silvio Di gregorio <si...@gmail.com>
>>>>>
>>>>>> I ha ve build w/ hadoop_cdh4.1.2 parameter.
>>>>>> Something is changed, monday i report the result. Now the farm is
>>>>>> closed.
>>>>>> Il giorno 05/ott/2013 14:06, "Claudio Martella" <
>>>>>> claudio.martella@gmail.com> ha scritto:
>>>>>>
>>>>>> Oh, right, -vof is in trunk. Anyway it looks like you built giraph
>>>>>>> for the wrong profile. You mentioned you're running on 2.0, but your giraph
>>>>>>> is built for 0.20.203. try building with a profile for your hadoop
>>>>>>> version.
>>>>>>>
>>>>>>>
>>>>>>> On Fri, Oct 4, 2013 at 2:35 PM, Silvio Di gregorio <
>>>>>>> silvio.digregorio@gmail.com> wrote:
>>>>>>>
>>>>>>>>
>>>>>>>> org.apache.commons.cli.UnrecognizedOptionException: Unrecognized
>>>>>>>> option: -vof
>>>>>>>>
>>>>>>>> in 1.0 version is
>>>>>>>> -of,--outputFormat <arg> Vertex output format
>>>>>>>> -op,--outputPath <arg> Vertex output path
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> 2013/10/4 Claudio Martella <cl...@gmail.com>
>>>>>>>>
>>>>>>>>> did you try the argument (-vof) i suggested?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Fri, Oct 4, 2013 at 2:13 PM, Silvio Di gregorio <
>>>>>>>>> silvio.digregorio@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> i've specified -of
>>>>>>>>>> org.apache.giraph.io.formats.IdWithValueTextOutputFormat
>>>>>>>>>>
>>>>>>>>>> but the same error was produced
>>>>>>>>>>
>>>>>>>>>> Exception in thread "main"
>>>>>>>>>> java.lang.IncompatibleClassChangeError: Found interface
>>>>>>>>>> org.apache.hadoop.mapreduce.JobContext, but class was expected
>>>>>>>>>> at
>>>>>>>>>> org.apache.giraph.bsp.BspOutputFormat.checkOutputSpecs(BspOutputFormat.java:43)
>>>>>>>>>> at
>>>>>>>>>> org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:984)
>>>>>>>>>> at
>>>>>>>>>> org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:945)
>>>>>>>>>> at java.security.AccessController.doPrivileged(Native
>>>>>>>>>> Method)
>>>>>>>>>> at javax.security.auth.Subject.doAs(Subject.java:396)
>>>>>>>>>> at
>>>>>>>>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
>>>>>>>>>> at
>>>>>>>>>> org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:945)
>>>>>>>>>> at org.apache.hadoop.mapreduce.Job.submit(Job.java:566)
>>>>>>>>>> at
>>>>>>>>>> org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:596)
>>>>>>>>>> at org.apache.giraph.job.GiraphJob.run(GiraphJob.java:237)
>>>>>>>>>> at
>>>>>>>>>> org.apache.giraph.GiraphRunner.run(GiraphRunner.java:94)
>>>>>>>>>> at
>>>>>>>>>> org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
>>>>>>>>>> at
>>>>>>>>>> org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
>>>>>>>>>> at
>>>>>>>>>> org.apache.giraph.GiraphRunner.main(GiraphRunner.java:124)
>>>>>>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
>>>>>>>>>> Method)
>>>>>>>>>> at
>>>>>>>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>>>>>>>>> at
>>>>>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>>>>>>>> at java.lang.reflect.Method.invoke(Method.java:597)
>>>>>>>>>> at org.apache.hadoop.util.RunJar.main(RunJar.java:208)
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> 2013/10/4 Claudio Martella <cl...@gmail.com>
>>>>>>>>>>
>>>>>>>>>>> Hi,
>>>>>>>>>>>
>>>>>>>>>>> you need to specify the vertex outputformat class (-vof option),
>>>>>>>>>>> e.g. org.apache.giraph.io.formats.IdWithValueTextOutputFormat.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Fri, Oct 4, 2013 at 1:06 PM, Silvio Di gregorio <
>>>>>>>>>>> silvio.digregorio@gmail.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Hi,
>>>>>>>>>>>>
>>>>>>>>>>>> I hope I have sent to the right address.
>>>>>>>>>>>>
>>>>>>>>>>>> i have a graph (directed and unweighted) stored in hdfs like a
>>>>>>>>>>>> adjacency list (140Milions of edges 6Milions of vertex)
>>>>>>>>>>>>
>>>>>>>>>>>> node<tab>neighbors
>>>>>>>>>>>>
>>>>>>>>>>>> 23 2 1 3 4 3
>>>>>>>>>>>>
>>>>>>>>>>>> 1 999 99923 909 .......
>>>>>>>>>>>>
>>>>>>>>>>>> ......
>>>>>>>>>>>>
>>>>>>>>>>>> hadoop version Hadoop 2.0.0-cdh4.3.0 - java 1.6
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> I have executed the giraph-1.0 connected components example, in
>>>>>>>>>>>> this fashion
>>>>>>>>>>>>
>>>>>>>>>>>>> > hadoop jar /usr/local/giraph/giraph-examples/target/giraph-
>>>>>>>>>>>>> examples-1.0.0-for-hadoop-0.20.203.0-jar-with-dependencies.jar
>>>>>>>>>>>>> org.apache.giraph.GiraphRunner org.apache.giraph.examples.ConnectedComponentsVertex
>>>>>>>>>>>>> -vif org.apache.giraph.io.formats.IntIntNullTextInputFormat
>>>>>>>>>>>>> -vip /user/hdfs/lista_adj_txt -op connectedgiraph --workers 4
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> and then fail with:
>>>>>>>>>>>>
>>>>>>>>>>>> 13/10/04 09:28:29 INFO utils.ConfigurationUtils: No edge input
>>>>>>>>>>>> format specified. Ensure your InputFormat does not require one.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> 13/10/04 09:28:29 INFO utils.ConfigurationUtils: No output
>>>>>>>>>>>> format specified. Ensure your OutputFormat does not require one.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> 13/10/04 09:28:30 INFO job.GiraphJob: run: Since checkpointing
>>>>>>>>>>>> is disabled (default), do not allow any task retries (setting
>>>>>>>>>>>> mapred.map.max.attempts = 0, old value = 4)
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> 13/10/04 09:28:31 WARN mapred.JobClient: Use
>>>>>>>>>>>> GenericOptionsParser for parsing the arguments. Applications should
>>>>>>>>>>>> implement Tool for the same.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> 13/10/04 09:28:31 INFO mapred.JobClient: Cleaning up the
>>>>>>>>>>>> staging area hdfs://
>>>>>>>>>>>> srv-bigdata-dev-01.int.sose.it:8020/user/hdfs/.staging/job_201309181636_0535
>>>>>>>>>>>>
>>>>>>>>>>>> Exception in thread "main"
>>>>>>>>>>>> java.lang.IncompatibleClassChangeError: Found interface
>>>>>>>>>>>> org.apache.hadoop.mapreduce.JobContext, but class was expected
>>>>>>>>>>>>
>>>>>>>>>>>> at
>>>>>>>>>>>> org.apache.giraph.bsp.BspOutputFormat.checkOutputSpecs(BspOutputFormat.java:43)
>>>>>>>>>>>>
>>>>>>>>>>>> ......
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks in advance
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Claudio Martella
>>>>>>>>>>> claudio.martella@gmail.com
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Claudio Martella
>>>>>>>>> claudio.martella@gmail.com
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Claudio Martella
>>>>>>> claudio.martella@gmail.com
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Claudio Martella
>>>> claudio.martella@gmail.com
>>>>
>>>
>>>
>>
>>
>> --
>> Claudio Martella
>> claudio.martella@gmail.com
>>
>
>
--
Claudio Martella
claudio.martella@gmail.com
Re: connected components example in giraph 1.0
Posted by Silvio Di gregorio <si...@gmail.com>.
I think now my issue is another, thanks to all.
java.lang.IllegalStateException: run: Caught an unrecoverable
exception waitFor: ExecutionException occurred while waiting for
org.apache.giraph.utils.ProgressableUtils$FutureWaitable@73a01e8a
at org.apache.giraph.graph.GraphMapper.run(GraphMapper.java:102)
......
Caused by: java.lang.IllegalStateException: waitFor:
ExecutionException occurred while waiting for
org.apache.giraph.utils.ProgressableUtils$FutureWaitable@73a01e8a
at org.apache.giraph.utils.ProgressableUtils.waitFor(ProgressableUtils.java:151)
... 7 more
Caused by: java.util.concurrent.ExecutionException:
java.lang.OutOfMemoryError: *Java heap space*
at org.apache.giraph.utils.ProgressableUtils.waitFor(ProgressableUtils.java:143)
... 13 more
2013/10/7 Claudio Martella <cl...@gmail.com>
> OK, thanks.
>
> I really have to push that patch in.
>
>
> On Mon, Oct 7, 2013 at 12:17 PM, Silvio Di gregorio <
> silvio.digregorio@gmail.com> wrote:
>
>> yes i do,
>> i have seen this in your post in:
>> http://www.mail-archive.com/user@giraph.apache.org/msg00957.html
>>
>> excuse me if i had checked in the mail-achive first I would have avoided
>> the last post.
>>
>> Now zk issue are resolved.
>>
>>
>> 2013/10/7 Claudio Martella <cl...@gmail.com>
>>
>>> Can you try applying this one first?
>>>
>>> http://www.mail-archive.com/user@giraph.apache.org/msg00945/check.diff
>>>
>>>
>>> On Mon, Oct 7, 2013 at 8:40 AM, Silvio Di gregorio <
>>> silvio.digregorio@gmail.com> wrote:
>>>
>>>>
>>>>
>>>> *As i said i have builded*
>>>>
>>>>
>>>> *giraph-examples-1.0.0-for-hadoop-2.0.0-cdh4.1.2-jar-with-dependencies.jar*
>>>>
>>>>
>>>> *for cdh4, successfully. The job start to monitoring the success rate:*
>>>>
>>>>
>>>> *13/10/07 08:28:45 INFO mapred.JobClient: map 0% reduce 0%*
>>>>
>>>>
>>>> *but then*
>>>>
>>>>
>>>> *Error running child
>>>> java.lang.IllegalStateException: run: Caught an unrecoverable exception java.io.FileNotFoundException: File _bsp/_defaultZkManagerDir/job_201309181636_0678/_zkServer does not exist.
>>>> *
>>>>
>>>>
>>>> *...*
>>>>
>>>>
>>>>
>>>> *Caused by: java.lang.RuntimeException: java.io.FileNotFoundException: File _bsp/_defaultZkManagerDir/job_201309181636_0678/_zkServer does not exist.*
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> 2013/10/5 Silvio Di gregorio <si...@gmail.com>
>>>>
>>>>> I ha ve build w/ hadoop_cdh4.1.2 parameter.
>>>>> Something is changed, monday i report the result. Now the farm is
>>>>> closed.
>>>>> Il giorno 05/ott/2013 14:06, "Claudio Martella" <
>>>>> claudio.martella@gmail.com> ha scritto:
>>>>>
>>>>> Oh, right, -vof is in trunk. Anyway it looks like you built giraph
>>>>>> for the wrong profile. You mentioned you're running on 2.0, but your giraph
>>>>>> is built for 0.20.203. try building with a profile for your hadoop
>>>>>> version.
>>>>>>
>>>>>>
>>>>>> On Fri, Oct 4, 2013 at 2:35 PM, Silvio Di gregorio <
>>>>>> silvio.digregorio@gmail.com> wrote:
>>>>>>
>>>>>>>
>>>>>>> org.apache.commons.cli.UnrecognizedOptionException: Unrecognized
>>>>>>> option: -vof
>>>>>>>
>>>>>>> in 1.0 version is
>>>>>>> -of,--outputFormat <arg> Vertex output format
>>>>>>> -op,--outputPath <arg> Vertex output path
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> 2013/10/4 Claudio Martella <cl...@gmail.com>
>>>>>>>
>>>>>>>> did you try the argument (-vof) i suggested?
>>>>>>>>
>>>>>>>>
>>>>>>>> On Fri, Oct 4, 2013 at 2:13 PM, Silvio Di gregorio <
>>>>>>>> silvio.digregorio@gmail.com> wrote:
>>>>>>>>
>>>>>>>>>
>>>>>>>>> i've specified -of
>>>>>>>>> org.apache.giraph.io.formats.IdWithValueTextOutputFormat
>>>>>>>>>
>>>>>>>>> but the same error was produced
>>>>>>>>>
>>>>>>>>> Exception in thread "main" java.lang.IncompatibleClassChangeError:
>>>>>>>>> Found interface org.apache.hadoop.mapreduce.JobContext, but class was
>>>>>>>>> expected
>>>>>>>>> at
>>>>>>>>> org.apache.giraph.bsp.BspOutputFormat.checkOutputSpecs(BspOutputFormat.java:43)
>>>>>>>>> at
>>>>>>>>> org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:984)
>>>>>>>>> at
>>>>>>>>> org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:945)
>>>>>>>>> at java.security.AccessController.doPrivileged(Native
>>>>>>>>> Method)
>>>>>>>>> at javax.security.auth.Subject.doAs(Subject.java:396)
>>>>>>>>> at
>>>>>>>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
>>>>>>>>> at
>>>>>>>>> org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:945)
>>>>>>>>> at org.apache.hadoop.mapreduce.Job.submit(Job.java:566)
>>>>>>>>> at
>>>>>>>>> org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:596)
>>>>>>>>> at org.apache.giraph.job.GiraphJob.run(GiraphJob.java:237)
>>>>>>>>> at org.apache.giraph.GiraphRunner.run(GiraphRunner.java:94)
>>>>>>>>> at
>>>>>>>>> org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
>>>>>>>>> at
>>>>>>>>> org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
>>>>>>>>> at
>>>>>>>>> org.apache.giraph.GiraphRunner.main(GiraphRunner.java:124)
>>>>>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
>>>>>>>>> Method)
>>>>>>>>> at
>>>>>>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>>>>>>>> at
>>>>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>>>>>>> at java.lang.reflect.Method.invoke(Method.java:597)
>>>>>>>>> at org.apache.hadoop.util.RunJar.main(RunJar.java:208)
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> 2013/10/4 Claudio Martella <cl...@gmail.com>
>>>>>>>>>
>>>>>>>>>> Hi,
>>>>>>>>>>
>>>>>>>>>> you need to specify the vertex outputformat class (-vof option),
>>>>>>>>>> e.g. org.apache.giraph.io.formats.IdWithValueTextOutputFormat.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Fri, Oct 4, 2013 at 1:06 PM, Silvio Di gregorio <
>>>>>>>>>> silvio.digregorio@gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Hi,
>>>>>>>>>>>
>>>>>>>>>>> I hope I have sent to the right address.
>>>>>>>>>>>
>>>>>>>>>>> i have a graph (directed and unweighted) stored in hdfs like a
>>>>>>>>>>> adjacency list (140Milions of edges 6Milions of vertex)
>>>>>>>>>>>
>>>>>>>>>>> node<tab>neighbors
>>>>>>>>>>>
>>>>>>>>>>> 23 2 1 3 4 3
>>>>>>>>>>>
>>>>>>>>>>> 1 999 99923 909 .......
>>>>>>>>>>>
>>>>>>>>>>> ......
>>>>>>>>>>>
>>>>>>>>>>> hadoop version Hadoop 2.0.0-cdh4.3.0 - java 1.6
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> I have executed the giraph-1.0 connected components example, in
>>>>>>>>>>> this fashion
>>>>>>>>>>>
>>>>>>>>>>>> > hadoop jar /usr/local/giraph/giraph-examples/target/giraph-
>>>>>>>>>>>> examples-1.0.0-for-hadoop-0.20.203.0-jar-with-dependencies.jar
>>>>>>>>>>>> org.apache.giraph.GiraphRunner org.apache.giraph.examples.ConnectedComponentsVertex
>>>>>>>>>>>> -vif org.apache.giraph.io.formats.IntIntNullTextInputFormat
>>>>>>>>>>>> -vip /user/hdfs/lista_adj_txt -op connectedgiraph --workers 4
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> and then fail with:
>>>>>>>>>>>
>>>>>>>>>>> 13/10/04 09:28:29 INFO utils.ConfigurationUtils: No edge input
>>>>>>>>>>> format specified. Ensure your InputFormat does not require one.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> 13/10/04 09:28:29 INFO utils.ConfigurationUtils: No output
>>>>>>>>>>> format specified. Ensure your OutputFormat does not require one.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> 13/10/04 09:28:30 INFO job.GiraphJob: run: Since checkpointing
>>>>>>>>>>> is disabled (default), do not allow any task retries (setting
>>>>>>>>>>> mapred.map.max.attempts = 0, old value = 4)
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> 13/10/04 09:28:31 WARN mapred.JobClient: Use
>>>>>>>>>>> GenericOptionsParser for parsing the arguments. Applications should
>>>>>>>>>>> implement Tool for the same.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> 13/10/04 09:28:31 INFO mapred.JobClient: Cleaning up the staging
>>>>>>>>>>> area hdfs://
>>>>>>>>>>> srv-bigdata-dev-01.int.sose.it:8020/user/hdfs/.staging/job_201309181636_0535
>>>>>>>>>>>
>>>>>>>>>>> Exception in thread "main"
>>>>>>>>>>> java.lang.IncompatibleClassChangeError: Found interface
>>>>>>>>>>> org.apache.hadoop.mapreduce.JobContext, but class was expected
>>>>>>>>>>>
>>>>>>>>>>> at
>>>>>>>>>>> org.apache.giraph.bsp.BspOutputFormat.checkOutputSpecs(BspOutputFormat.java:43)
>>>>>>>>>>>
>>>>>>>>>>> ......
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Thanks in advance
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Claudio Martella
>>>>>>>>>> claudio.martella@gmail.com
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Claudio Martella
>>>>>>>> claudio.martella@gmail.com
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Claudio Martella
>>>>>> claudio.martella@gmail.com
>>>>>>
>>>>>
>>>>
>>>
>>>
>>> --
>>> Claudio Martella
>>> claudio.martella@gmail.com
>>>
>>
>>
>
>
> --
> Claudio Martella
> claudio.martella@gmail.com
>
Re: connected components example in giraph 1.0
Posted by Claudio Martella <cl...@gmail.com>.
OK, thanks.
I really have to push that patch in.
On Mon, Oct 7, 2013 at 12:17 PM, Silvio Di gregorio <
silvio.digregorio@gmail.com> wrote:
> yes i do,
> i have seen this in your post in:
> http://www.mail-archive.com/user@giraph.apache.org/msg00957.html
>
> excuse me if i had checked in the mail-achive first I would have avoided
> the last post.
>
> Now zk issue are resolved.
>
>
> 2013/10/7 Claudio Martella <cl...@gmail.com>
>
>> Can you try applying this one first?
>>
>> http://www.mail-archive.com/user@giraph.apache.org/msg00945/check.diff
>>
>>
>> On Mon, Oct 7, 2013 at 8:40 AM, Silvio Di gregorio <
>> silvio.digregorio@gmail.com> wrote:
>>
>>>
>>>
>>> *As i said i have builded*
>>>
>>>
>>> *giraph-examples-1.0.0-for-hadoop-2.0.0-cdh4.1.2-jar-with-dependencies.jar*
>>>
>>>
>>> *for cdh4, successfully. The job start to monitoring the success rate:*
>>>
>>>
>>> *13/10/07 08:28:45 INFO mapred.JobClient: map 0% reduce 0%*
>>>
>>>
>>> *but then*
>>>
>>>
>>> *Error running child
>>> java.lang.IllegalStateException: run: Caught an unrecoverable exception java.io.FileNotFoundException: File _bsp/_defaultZkManagerDir/job_201309181636_0678/_zkServer does not exist.
>>> *
>>>
>>>
>>> *...*
>>>
>>>
>>>
>>> *Caused by: java.lang.RuntimeException: java.io.FileNotFoundException: File _bsp/_defaultZkManagerDir/job_201309181636_0678/_zkServer does not exist.*
>>>
>>>
>>>
>>>
>>>
>>>
>>> 2013/10/5 Silvio Di gregorio <si...@gmail.com>
>>>
>>>> I ha ve build w/ hadoop_cdh4.1.2 parameter.
>>>> Something is changed, monday i report the result. Now the farm is
>>>> closed.
>>>> Il giorno 05/ott/2013 14:06, "Claudio Martella" <
>>>> claudio.martella@gmail.com> ha scritto:
>>>>
>>>> Oh, right, -vof is in trunk. Anyway it looks like you built giraph
>>>>> for the wrong profile. You mentioned you're running on 2.0, but your giraph
>>>>> is built for 0.20.203. try building with a profile for your hadoop
>>>>> version.
>>>>>
>>>>>
>>>>> On Fri, Oct 4, 2013 at 2:35 PM, Silvio Di gregorio <
>>>>> silvio.digregorio@gmail.com> wrote:
>>>>>
>>>>>>
>>>>>> org.apache.commons.cli.UnrecognizedOptionException: Unrecognized
>>>>>> option: -vof
>>>>>>
>>>>>> in 1.0 version is
>>>>>> -of,--outputFormat <arg> Vertex output format
>>>>>> -op,--outputPath <arg> Vertex output path
>>>>>>
>>>>>>
>>>>>>
>>>>>> 2013/10/4 Claudio Martella <cl...@gmail.com>
>>>>>>
>>>>>>> did you try the argument (-vof) i suggested?
>>>>>>>
>>>>>>>
>>>>>>> On Fri, Oct 4, 2013 at 2:13 PM, Silvio Di gregorio <
>>>>>>> silvio.digregorio@gmail.com> wrote:
>>>>>>>
>>>>>>>>
>>>>>>>> i've specified -of
>>>>>>>> org.apache.giraph.io.formats.IdWithValueTextOutputFormat
>>>>>>>>
>>>>>>>> but the same error was produced
>>>>>>>>
>>>>>>>> Exception in thread "main" java.lang.IncompatibleClassChangeError:
>>>>>>>> Found interface org.apache.hadoop.mapreduce.JobContext, but class was
>>>>>>>> expected
>>>>>>>> at
>>>>>>>> org.apache.giraph.bsp.BspOutputFormat.checkOutputSpecs(BspOutputFormat.java:43)
>>>>>>>> at
>>>>>>>> org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:984)
>>>>>>>> at
>>>>>>>> org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:945)
>>>>>>>> at java.security.AccessController.doPrivileged(Native
>>>>>>>> Method)
>>>>>>>> at javax.security.auth.Subject.doAs(Subject.java:396)
>>>>>>>> at
>>>>>>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
>>>>>>>> at
>>>>>>>> org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:945)
>>>>>>>> at org.apache.hadoop.mapreduce.Job.submit(Job.java:566)
>>>>>>>> at
>>>>>>>> org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:596)
>>>>>>>> at org.apache.giraph.job.GiraphJob.run(GiraphJob.java:237)
>>>>>>>> at org.apache.giraph.GiraphRunner.run(GiraphRunner.java:94)
>>>>>>>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
>>>>>>>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
>>>>>>>> at
>>>>>>>> org.apache.giraph.GiraphRunner.main(GiraphRunner.java:124)
>>>>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
>>>>>>>> Method)
>>>>>>>> at
>>>>>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>>>>>>> at
>>>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>>>>>> at java.lang.reflect.Method.invoke(Method.java:597)
>>>>>>>> at org.apache.hadoop.util.RunJar.main(RunJar.java:208)
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> 2013/10/4 Claudio Martella <cl...@gmail.com>
>>>>>>>>
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> you need to specify the vertex outputformat class (-vof option),
>>>>>>>>> e.g. org.apache.giraph.io.formats.IdWithValueTextOutputFormat.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Fri, Oct 4, 2013 at 1:06 PM, Silvio Di gregorio <
>>>>>>>>> silvio.digregorio@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Hi,
>>>>>>>>>>
>>>>>>>>>> I hope I have sent to the right address.
>>>>>>>>>>
>>>>>>>>>> i have a graph (directed and unweighted) stored in hdfs like a
>>>>>>>>>> adjacency list (140Milions of edges 6Milions of vertex)
>>>>>>>>>>
>>>>>>>>>> node<tab>neighbors
>>>>>>>>>>
>>>>>>>>>> 23 2 1 3 4 3
>>>>>>>>>>
>>>>>>>>>> 1 999 99923 909 .......
>>>>>>>>>>
>>>>>>>>>> ......
>>>>>>>>>>
>>>>>>>>>> hadoop version Hadoop 2.0.0-cdh4.3.0 - java 1.6
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> I have executed the giraph-1.0 connected components example, in
>>>>>>>>>> this fashion
>>>>>>>>>>
>>>>>>>>>>> > hadoop jar /usr/local/giraph/giraph-examples/target/giraph-
>>>>>>>>>>> examples-1.0.0-for-hadoop-0.20.203.0-jar-with-dependencies.jar
>>>>>>>>>>> org.apache.giraph.GiraphRunner org.apache.giraph.examples.ConnectedComponentsVertex
>>>>>>>>>>> -vif org.apache.giraph.io.formats.IntIntNullTextInputFormat
>>>>>>>>>>> -vip /user/hdfs/lista_adj_txt -op connectedgiraph --workers 4
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> and then fail with:
>>>>>>>>>>
>>>>>>>>>> 13/10/04 09:28:29 INFO utils.ConfigurationUtils: No edge input
>>>>>>>>>> format specified. Ensure your InputFormat does not require one.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> 13/10/04 09:28:29 INFO utils.ConfigurationUtils: No output format
>>>>>>>>>> specified. Ensure your OutputFormat does not require one.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> 13/10/04 09:28:30 INFO job.GiraphJob: run: Since checkpointing is
>>>>>>>>>> disabled (default), do not allow any task retries (setting
>>>>>>>>>> mapred.map.max.attempts = 0, old value = 4)
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> 13/10/04 09:28:31 WARN mapred.JobClient: Use GenericOptionsParser
>>>>>>>>>> for parsing the arguments. Applications should implement Tool for the same.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> 13/10/04 09:28:31 INFO mapred.JobClient: Cleaning up the staging
>>>>>>>>>> area hdfs://
>>>>>>>>>> srv-bigdata-dev-01.int.sose.it:8020/user/hdfs/.staging/job_201309181636_0535
>>>>>>>>>>
>>>>>>>>>> Exception in thread "main"
>>>>>>>>>> java.lang.IncompatibleClassChangeError: Found interface
>>>>>>>>>> org.apache.hadoop.mapreduce.JobContext, but class was expected
>>>>>>>>>>
>>>>>>>>>> at
>>>>>>>>>> org.apache.giraph.bsp.BspOutputFormat.checkOutputSpecs(BspOutputFormat.java:43)
>>>>>>>>>>
>>>>>>>>>> ......
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Thanks in advance
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Claudio Martella
>>>>>>>>> claudio.martella@gmail.com
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Claudio Martella
>>>>>>> claudio.martella@gmail.com
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Claudio Martella
>>>>> claudio.martella@gmail.com
>>>>>
>>>>
>>>
>>
>>
>> --
>> Claudio Martella
>> claudio.martella@gmail.com
>>
>
>
--
Claudio Martella
claudio.martella@gmail.com
Re: connected components example in giraph 1.0
Posted by Silvio Di gregorio <si...@gmail.com>.
yes i do,
i have seen this in your post in:
http://www.mail-archive.com/user@giraph.apache.org/msg00957.html
excuse me if i had checked in the mail-achive first I would have avoided
the last post.
Now zk issue are resolved.
2013/10/7 Claudio Martella <cl...@gmail.com>
> Can you try applying this one first?
>
> http://www.mail-archive.com/user@giraph.apache.org/msg00945/check.diff
>
>
> On Mon, Oct 7, 2013 at 8:40 AM, Silvio Di gregorio <
> silvio.digregorio@gmail.com> wrote:
>
>> *As i said i have builded*
>>
>> *giraph-examples-1.0.0-for-hadoop-2.0.0-cdh4.1.2-jar-with-dependencies.jar*
>>
>> *for cdh4, successfully. The job start to monitoring the success rate:*
>>
>> *13/10/07 08:28:45 INFO mapred.JobClient: map 0% reduce 0%*
>>
>> *but then*
>>
>> *Error running child
>> java.lang.IllegalStateException: run: Caught an unrecoverable exception java.io.FileNotFoundException: File _bsp/_defaultZkManagerDir/job_201309181636_0678/_zkServer does not exist.
>> *
>>
>> *...*
>>
>> *Caused by: java.lang.RuntimeException: java.io.FileNotFoundException: File _bsp/_defaultZkManagerDir/job_201309181636_0678/_zkServer does not exist.*
>>
>>
>>
>>
>> 2013/10/5 Silvio Di gregorio <si...@gmail.com>
>>
>>> I ha ve build w/ hadoop_cdh4.1.2 parameter.
>>> Something is changed, monday i report the result. Now the farm is closed.
>>> Il giorno 05/ott/2013 14:06, "Claudio Martella" <
>>> claudio.martella@gmail.com> ha scritto:
>>>
>>> Oh, right, -vof is in trunk. Anyway it looks like you built giraph for
>>>> the wrong profile. You mentioned you're running on 2.0, but your giraph is
>>>> built for 0.20.203. try building with a profile for your hadoop
>>>> version.
>>>>
>>>>
>>>> On Fri, Oct 4, 2013 at 2:35 PM, Silvio Di gregorio <
>>>> silvio.digregorio@gmail.com> wrote:
>>>>
>>>>>
>>>>> org.apache.commons.cli.UnrecognizedOptionException: Unrecognized
>>>>> option: -vof
>>>>>
>>>>> in 1.0 version is
>>>>> -of,--outputFormat <arg> Vertex output format
>>>>> -op,--outputPath <arg> Vertex output path
>>>>>
>>>>>
>>>>>
>>>>> 2013/10/4 Claudio Martella <cl...@gmail.com>
>>>>>
>>>>>> did you try the argument (-vof) i suggested?
>>>>>>
>>>>>>
>>>>>> On Fri, Oct 4, 2013 at 2:13 PM, Silvio Di gregorio <
>>>>>> silvio.digregorio@gmail.com> wrote:
>>>>>>
>>>>>>>
>>>>>>> i've specified -of
>>>>>>> org.apache.giraph.io.formats.IdWithValueTextOutputFormat
>>>>>>>
>>>>>>> but the same error was produced
>>>>>>>
>>>>>>> Exception in thread "main" java.lang.IncompatibleClassChangeError:
>>>>>>> Found interface org.apache.hadoop.mapreduce.JobContext, but class was
>>>>>>> expected
>>>>>>> at
>>>>>>> org.apache.giraph.bsp.BspOutputFormat.checkOutputSpecs(BspOutputFormat.java:43)
>>>>>>> at
>>>>>>> org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:984)
>>>>>>> at
>>>>>>> org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:945)
>>>>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>>>>> at javax.security.auth.Subject.doAs(Subject.java:396)
>>>>>>> at
>>>>>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
>>>>>>> at
>>>>>>> org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:945)
>>>>>>> at org.apache.hadoop.mapreduce.Job.submit(Job.java:566)
>>>>>>> at
>>>>>>> org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:596)
>>>>>>> at org.apache.giraph.job.GiraphJob.run(GiraphJob.java:237)
>>>>>>> at org.apache.giraph.GiraphRunner.run(GiraphRunner.java:94)
>>>>>>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
>>>>>>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
>>>>>>> at org.apache.giraph.GiraphRunner.main(GiraphRunner.java:124)
>>>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
>>>>>>> Method)
>>>>>>> at
>>>>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>>>>>> at
>>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>>>>> at java.lang.reflect.Method.invoke(Method.java:597)
>>>>>>> at org.apache.hadoop.util.RunJar.main(RunJar.java:208)
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> 2013/10/4 Claudio Martella <cl...@gmail.com>
>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> you need to specify the vertex outputformat class (-vof option),
>>>>>>>> e.g. org.apache.giraph.io.formats.IdWithValueTextOutputFormat.
>>>>>>>>
>>>>>>>>
>>>>>>>> On Fri, Oct 4, 2013 at 1:06 PM, Silvio Di gregorio <
>>>>>>>> silvio.digregorio@gmail.com> wrote:
>>>>>>>>
>>>>>>>>>
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> I hope I have sent to the right address.
>>>>>>>>>
>>>>>>>>> i have a graph (directed and unweighted) stored in hdfs like a
>>>>>>>>> adjacency list (140Milions of edges 6Milions of vertex)
>>>>>>>>>
>>>>>>>>> node<tab>neighbors
>>>>>>>>>
>>>>>>>>> 23 2 1 3 4 3
>>>>>>>>>
>>>>>>>>> 1 999 99923 909 .......
>>>>>>>>>
>>>>>>>>> ......
>>>>>>>>>
>>>>>>>>> hadoop version Hadoop 2.0.0-cdh4.3.0 - java 1.6
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> I have executed the giraph-1.0 connected components example, in
>>>>>>>>> this fashion
>>>>>>>>>
>>>>>>>>>> > hadoop jar /usr/local/giraph/giraph-examples/target/giraph-
>>>>>>>>>> examples-1.0.0-for-hadoop-0.20.203.0-jar-with-dependencies.jar
>>>>>>>>>> org.apache.giraph.GiraphRunner org.apache.giraph.examples.ConnectedComponentsVertex
>>>>>>>>>> -vif org.apache.giraph.io.formats.IntIntNullTextInputFormat -vip
>>>>>>>>>> /user/hdfs/lista_adj_txt -op connectedgiraph --workers 4
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> and then fail with:
>>>>>>>>>
>>>>>>>>> 13/10/04 09:28:29 INFO utils.ConfigurationUtils: No edge input
>>>>>>>>> format specified. Ensure your InputFormat does not require one.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> 13/10/04 09:28:29 INFO utils.ConfigurationUtils: No output format
>>>>>>>>> specified. Ensure your OutputFormat does not require one.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> 13/10/04 09:28:30 INFO job.GiraphJob: run: Since checkpointing is
>>>>>>>>> disabled (default), do not allow any task retries (setting
>>>>>>>>> mapred.map.max.attempts = 0, old value = 4)
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> 13/10/04 09:28:31 WARN mapred.JobClient: Use GenericOptionsParser
>>>>>>>>> for parsing the arguments. Applications should implement Tool for the same.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> 13/10/04 09:28:31 INFO mapred.JobClient: Cleaning up the staging
>>>>>>>>> area hdfs://
>>>>>>>>> srv-bigdata-dev-01.int.sose.it:8020/user/hdfs/.staging/job_201309181636_0535
>>>>>>>>>
>>>>>>>>> Exception in thread "main" java.lang.IncompatibleClassChangeError:
>>>>>>>>> Found interface org.apache.hadoop.mapreduce.JobContext, but class was
>>>>>>>>> expected
>>>>>>>>>
>>>>>>>>> at
>>>>>>>>> org.apache.giraph.bsp.BspOutputFormat.checkOutputSpecs(BspOutputFormat.java:43)
>>>>>>>>>
>>>>>>>>> ......
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Thanks in advance
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Claudio Martella
>>>>>>>> claudio.martella@gmail.com
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Claudio Martella
>>>>>> claudio.martella@gmail.com
>>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Claudio Martella
>>>> claudio.martella@gmail.com
>>>>
>>>
>>
>
>
> --
> Claudio Martella
> claudio.martella@gmail.com
>
Re: connected components example in giraph 1.0
Posted by Claudio Martella <cl...@gmail.com>.
Can you try applying this one first?
http://www.mail-archive.com/user@giraph.apache.org/msg00945/check.diff
On Mon, Oct 7, 2013 at 8:40 AM, Silvio Di gregorio <
silvio.digregorio@gmail.com> wrote:
> *As i said i have builded*
>
> *giraph-examples-1.0.0-for-hadoop-2.0.0-cdh4.1.2-jar-with-dependencies.jar*
>
> *for cdh4, successfully. The job start to monitoring the success rate:*
>
> *13/10/07 08:28:45 INFO mapred.JobClient: map 0% reduce 0%*
>
> *but then*
>
> *Error running child
> java.lang.IllegalStateException: run: Caught an unrecoverable exception java.io.FileNotFoundException: File _bsp/_defaultZkManagerDir/job_201309181636_0678/_zkServer does not exist.
> *
>
> *...*
>
> *Caused by: java.lang.RuntimeException: java.io.FileNotFoundException: File _bsp/_defaultZkManagerDir/job_201309181636_0678/_zkServer does not exist.*
>
>
>
>
> 2013/10/5 Silvio Di gregorio <si...@gmail.com>
>
>> I ha ve build w/ hadoop_cdh4.1.2 parameter.
>> Something is changed, monday i report the result. Now the farm is closed.
>> Il giorno 05/ott/2013 14:06, "Claudio Martella" <
>> claudio.martella@gmail.com> ha scritto:
>>
>> Oh, right, -vof is in trunk. Anyway it looks like you built giraph for
>>> the wrong profile. You mentioned you're running on 2.0, but your giraph is
>>> built for 0.20.203. try building with a profile for your hadoop version.
>>>
>>>
>>> On Fri, Oct 4, 2013 at 2:35 PM, Silvio Di gregorio <
>>> silvio.digregorio@gmail.com> wrote:
>>>
>>>>
>>>> org.apache.commons.cli.UnrecognizedOptionException: Unrecognized
>>>> option: -vof
>>>>
>>>> in 1.0 version is
>>>> -of,--outputFormat <arg> Vertex output format
>>>> -op,--outputPath <arg> Vertex output path
>>>>
>>>>
>>>>
>>>> 2013/10/4 Claudio Martella <cl...@gmail.com>
>>>>
>>>>> did you try the argument (-vof) i suggested?
>>>>>
>>>>>
>>>>> On Fri, Oct 4, 2013 at 2:13 PM, Silvio Di gregorio <
>>>>> silvio.digregorio@gmail.com> wrote:
>>>>>
>>>>>>
>>>>>> i've specified -of
>>>>>> org.apache.giraph.io.formats.IdWithValueTextOutputFormat
>>>>>>
>>>>>> but the same error was produced
>>>>>>
>>>>>> Exception in thread "main" java.lang.IncompatibleClassChangeError:
>>>>>> Found interface org.apache.hadoop.mapreduce.JobContext, but class was
>>>>>> expected
>>>>>> at
>>>>>> org.apache.giraph.bsp.BspOutputFormat.checkOutputSpecs(BspOutputFormat.java:43)
>>>>>> at
>>>>>> org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:984)
>>>>>> at
>>>>>> org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:945)
>>>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>>>> at javax.security.auth.Subject.doAs(Subject.java:396)
>>>>>> at
>>>>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
>>>>>> at
>>>>>> org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:945)
>>>>>> at org.apache.hadoop.mapreduce.Job.submit(Job.java:566)
>>>>>> at
>>>>>> org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:596)
>>>>>> at org.apache.giraph.job.GiraphJob.run(GiraphJob.java:237)
>>>>>> at org.apache.giraph.GiraphRunner.run(GiraphRunner.java:94)
>>>>>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
>>>>>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
>>>>>> at org.apache.giraph.GiraphRunner.main(GiraphRunner.java:124)
>>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>>>> at
>>>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>>>>> at
>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>>>> at java.lang.reflect.Method.invoke(Method.java:597)
>>>>>> at org.apache.hadoop.util.RunJar.main(RunJar.java:208)
>>>>>>
>>>>>>
>>>>>>
>>>>>> 2013/10/4 Claudio Martella <cl...@gmail.com>
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> you need to specify the vertex outputformat class (-vof option),
>>>>>>> e.g. org.apache.giraph.io.formats.IdWithValueTextOutputFormat.
>>>>>>>
>>>>>>>
>>>>>>> On Fri, Oct 4, 2013 at 1:06 PM, Silvio Di gregorio <
>>>>>>> silvio.digregorio@gmail.com> wrote:
>>>>>>>
>>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> I hope I have sent to the right address.
>>>>>>>>
>>>>>>>> i have a graph (directed and unweighted) stored in hdfs like a
>>>>>>>> adjacency list (140Milions of edges 6Milions of vertex)
>>>>>>>>
>>>>>>>> node<tab>neighbors
>>>>>>>>
>>>>>>>> 23 2 1 3 4 3
>>>>>>>>
>>>>>>>> 1 999 99923 909 .......
>>>>>>>>
>>>>>>>> ......
>>>>>>>>
>>>>>>>> hadoop version Hadoop 2.0.0-cdh4.3.0 - java 1.6
>>>>>>>>
>>>>>>>>
>>>>>>>> I have executed the giraph-1.0 connected components example, in
>>>>>>>> this fashion
>>>>>>>>
>>>>>>>>> > hadoop jar /usr/local/giraph/giraph-examples/target/giraph-
>>>>>>>>> examples-1.0.0-for-hadoop-0.20.203.0-jar-with-dependencies.jar
>>>>>>>>> org.apache.giraph.GiraphRunner org.apache.giraph.examples.ConnectedComponentsVertex
>>>>>>>>> -vif org.apache.giraph.io.formats.IntIntNullTextInputFormat -vip
>>>>>>>>> /user/hdfs/lista_adj_txt -op connectedgiraph --workers 4
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> and then fail with:
>>>>>>>>
>>>>>>>> 13/10/04 09:28:29 INFO utils.ConfigurationUtils: No edge input
>>>>>>>> format specified. Ensure your InputFormat does not require one.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> 13/10/04 09:28:29 INFO utils.ConfigurationUtils: No output format
>>>>>>>> specified. Ensure your OutputFormat does not require one.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> 13/10/04 09:28:30 INFO job.GiraphJob: run: Since checkpointing is
>>>>>>>> disabled (default), do not allow any task retries (setting
>>>>>>>> mapred.map.max.attempts = 0, old value = 4)
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> 13/10/04 09:28:31 WARN mapred.JobClient: Use GenericOptionsParser
>>>>>>>> for parsing the arguments. Applications should implement Tool for the same.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> 13/10/04 09:28:31 INFO mapred.JobClient: Cleaning up the staging
>>>>>>>> area hdfs://
>>>>>>>> srv-bigdata-dev-01.int.sose.it:8020/user/hdfs/.staging/job_201309181636_0535
>>>>>>>>
>>>>>>>> Exception in thread "main" java.lang.IncompatibleClassChangeError:
>>>>>>>> Found interface org.apache.hadoop.mapreduce.JobContext, but class was
>>>>>>>> expected
>>>>>>>>
>>>>>>>> at
>>>>>>>> org.apache.giraph.bsp.BspOutputFormat.checkOutputSpecs(BspOutputFormat.java:43)
>>>>>>>>
>>>>>>>> ......
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Thanks in advance
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Claudio Martella
>>>>>>> claudio.martella@gmail.com
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Claudio Martella
>>>>> claudio.martella@gmail.com
>>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> Claudio Martella
>>> claudio.martella@gmail.com
>>>
>>
>
--
Claudio Martella
claudio.martella@gmail.com
Re: connected components example in giraph 1.0
Posted by Silvio Di gregorio <si...@gmail.com>.
*As i said i have builded*
*giraph-examples-1.0.0-for-hadoop-2.0.0-cdh4.1.2-jar-with-dependencies.jar*
*for cdh4, successfully. The job start to monitoring the success rate:*
*13/10/07 08:28:45 INFO mapred.JobClient: map 0% reduce 0%*
*but then*
*Error running child
java.lang.IllegalStateException: run: Caught an unrecoverable
exception java.io.FileNotFoundException: File
_bsp/_defaultZkManagerDir/job_201309181636_0678/_zkServer does not
exist.
*
*...*
*Caused by: java.lang.RuntimeException: java.io.FileNotFoundException:
File _bsp/_defaultZkManagerDir/job_201309181636_0678/_zkServer does
not exist.*
2013/10/5 Silvio Di gregorio <si...@gmail.com>
> I ha ve build w/ hadoop_cdh4.1.2 parameter.
> Something is changed, monday i report the result. Now the farm is closed.
> Il giorno 05/ott/2013 14:06, "Claudio Martella" <
> claudio.martella@gmail.com> ha scritto:
>
> Oh, right, -vof is in trunk. Anyway it looks like you built giraph for the
>> wrong profile. You mentioned you're running on 2.0, but your giraph is
>> built for 0.20.203. try building with a profile for your hadoop version.
>>
>>
>> On Fri, Oct 4, 2013 at 2:35 PM, Silvio Di gregorio <
>> silvio.digregorio@gmail.com> wrote:
>>
>>>
>>> org.apache.commons.cli.UnrecognizedOptionException: Unrecognized option:
>>> -vof
>>>
>>> in 1.0 version is
>>> -of,--outputFormat <arg> Vertex output format
>>> -op,--outputPath <arg> Vertex output path
>>>
>>>
>>>
>>> 2013/10/4 Claudio Martella <cl...@gmail.com>
>>>
>>>> did you try the argument (-vof) i suggested?
>>>>
>>>>
>>>> On Fri, Oct 4, 2013 at 2:13 PM, Silvio Di gregorio <
>>>> silvio.digregorio@gmail.com> wrote:
>>>>
>>>>>
>>>>> i've specified -of
>>>>> org.apache.giraph.io.formats.IdWithValueTextOutputFormat
>>>>>
>>>>> but the same error was produced
>>>>>
>>>>> Exception in thread "main" java.lang.IncompatibleClassChangeError:
>>>>> Found interface org.apache.hadoop.mapreduce.JobContext, but class was
>>>>> expected
>>>>> at
>>>>> org.apache.giraph.bsp.BspOutputFormat.checkOutputSpecs(BspOutputFormat.java:43)
>>>>> at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:984)
>>>>> at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:945)
>>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>>> at javax.security.auth.Subject.doAs(Subject.java:396)
>>>>> at
>>>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
>>>>> at
>>>>> org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:945)
>>>>> at org.apache.hadoop.mapreduce.Job.submit(Job.java:566)
>>>>> at
>>>>> org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:596)
>>>>> at org.apache.giraph.job.GiraphJob.run(GiraphJob.java:237)
>>>>> at org.apache.giraph.GiraphRunner.run(GiraphRunner.java:94)
>>>>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
>>>>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
>>>>> at org.apache.giraph.GiraphRunner.main(GiraphRunner.java:124)
>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>>> at
>>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>>>> at
>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>>> at java.lang.reflect.Method.invoke(Method.java:597)
>>>>> at org.apache.hadoop.util.RunJar.main(RunJar.java:208)
>>>>>
>>>>>
>>>>>
>>>>> 2013/10/4 Claudio Martella <cl...@gmail.com>
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> you need to specify the vertex outputformat class (-vof option), e.g.
>>>>>> org.apache.giraph.io.formats.IdWithValueTextOutputFormat.
>>>>>>
>>>>>>
>>>>>> On Fri, Oct 4, 2013 at 1:06 PM, Silvio Di gregorio <
>>>>>> silvio.digregorio@gmail.com> wrote:
>>>>>>
>>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> I hope I have sent to the right address.
>>>>>>>
>>>>>>> i have a graph (directed and unweighted) stored in hdfs like a
>>>>>>> adjacency list (140Milions of edges 6Milions of vertex)
>>>>>>>
>>>>>>> node<tab>neighbors
>>>>>>>
>>>>>>> 23 2 1 3 4 3
>>>>>>>
>>>>>>> 1 999 99923 909 .......
>>>>>>>
>>>>>>> ......
>>>>>>>
>>>>>>> hadoop version Hadoop 2.0.0-cdh4.3.0 - java 1.6
>>>>>>>
>>>>>>>
>>>>>>> I have executed the giraph-1.0 connected components example, in this
>>>>>>> fashion
>>>>>>>
>>>>>>>> > hadoop jar /usr/local/giraph/giraph-examples/target/giraph-
>>>>>>>> examples-1.0.0-for-hadoop-0.20.203.0-jar-with-dependencies.jar
>>>>>>>> org.apache.giraph.GiraphRunner org.apache.giraph.examples.ConnectedComponentsVertex
>>>>>>>> -vif org.apache.giraph.io.formats.IntIntNullTextInputFormat -vip
>>>>>>>> /user/hdfs/lista_adj_txt -op connectedgiraph --workers 4
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> and then fail with:
>>>>>>>
>>>>>>> 13/10/04 09:28:29 INFO utils.ConfigurationUtils: No edge input
>>>>>>> format specified. Ensure your InputFormat does not require one.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> 13/10/04 09:28:29 INFO utils.ConfigurationUtils: No output format
>>>>>>> specified. Ensure your OutputFormat does not require one.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> 13/10/04 09:28:30 INFO job.GiraphJob: run: Since checkpointing is
>>>>>>> disabled (default), do not allow any task retries (setting
>>>>>>> mapred.map.max.attempts = 0, old value = 4)
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> 13/10/04 09:28:31 WARN mapred.JobClient: Use GenericOptionsParser
>>>>>>> for parsing the arguments. Applications should implement Tool for the same.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> 13/10/04 09:28:31 INFO mapred.JobClient: Cleaning up the staging
>>>>>>> area hdfs://
>>>>>>> srv-bigdata-dev-01.int.sose.it:8020/user/hdfs/.staging/job_201309181636_0535
>>>>>>>
>>>>>>> Exception in thread "main" java.lang.IncompatibleClassChangeError:
>>>>>>> Found interface org.apache.hadoop.mapreduce.JobContext, but class was
>>>>>>> expected
>>>>>>>
>>>>>>> at
>>>>>>> org.apache.giraph.bsp.BspOutputFormat.checkOutputSpecs(BspOutputFormat.java:43)
>>>>>>>
>>>>>>> ......
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Thanks in advance
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Claudio Martella
>>>>>> claudio.martella@gmail.com
>>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Claudio Martella
>>>> claudio.martella@gmail.com
>>>>
>>>
>>>
>>
>>
>> --
>> Claudio Martella
>> claudio.martella@gmail.com
>>
>
Re: connected components example in giraph 1.0
Posted by Silvio Di gregorio <si...@gmail.com>.
I ha ve build w/ hadoop_cdh4.1.2 parameter.
Something is changed, monday i report the result. Now the farm is closed.
Il giorno 05/ott/2013 14:06, "Claudio Martella" <cl...@gmail.com>
ha scritto:
> Oh, right, -vof is in trunk. Anyway it looks like you built giraph for the
> wrong profile. You mentioned you're running on 2.0, but your giraph is
> built for 0.20.203. try building with a profile for your hadoop version.
>
>
> On Fri, Oct 4, 2013 at 2:35 PM, Silvio Di gregorio <
> silvio.digregorio@gmail.com> wrote:
>
>>
>> org.apache.commons.cli.UnrecognizedOptionException: Unrecognized option:
>> -vof
>>
>> in 1.0 version is
>> -of,--outputFormat <arg> Vertex output format
>> -op,--outputPath <arg> Vertex output path
>>
>>
>>
>> 2013/10/4 Claudio Martella <cl...@gmail.com>
>>
>>> did you try the argument (-vof) i suggested?
>>>
>>>
>>> On Fri, Oct 4, 2013 at 2:13 PM, Silvio Di gregorio <
>>> silvio.digregorio@gmail.com> wrote:
>>>
>>>>
>>>> i've specified -of
>>>> org.apache.giraph.io.formats.IdWithValueTextOutputFormat
>>>>
>>>> but the same error was produced
>>>>
>>>> Exception in thread "main" java.lang.IncompatibleClassChangeError:
>>>> Found interface org.apache.hadoop.mapreduce.JobContext, but class was
>>>> expected
>>>> at
>>>> org.apache.giraph.bsp.BspOutputFormat.checkOutputSpecs(BspOutputFormat.java:43)
>>>> at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:984)
>>>> at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:945)
>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>> at javax.security.auth.Subject.doAs(Subject.java:396)
>>>> at
>>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
>>>> at
>>>> org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:945)
>>>> at org.apache.hadoop.mapreduce.Job.submit(Job.java:566)
>>>> at
>>>> org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:596)
>>>> at org.apache.giraph.job.GiraphJob.run(GiraphJob.java:237)
>>>> at org.apache.giraph.GiraphRunner.run(GiraphRunner.java:94)
>>>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
>>>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
>>>> at org.apache.giraph.GiraphRunner.main(GiraphRunner.java:124)
>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>> at
>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>>> at
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>> at java.lang.reflect.Method.invoke(Method.java:597)
>>>> at org.apache.hadoop.util.RunJar.main(RunJar.java:208)
>>>>
>>>>
>>>>
>>>> 2013/10/4 Claudio Martella <cl...@gmail.com>
>>>>
>>>>> Hi,
>>>>>
>>>>> you need to specify the vertex outputformat class (-vof option), e.g.
>>>>> org.apache.giraph.io.formats.IdWithValueTextOutputFormat.
>>>>>
>>>>>
>>>>> On Fri, Oct 4, 2013 at 1:06 PM, Silvio Di gregorio <
>>>>> silvio.digregorio@gmail.com> wrote:
>>>>>
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> I hope I have sent to the right address.
>>>>>>
>>>>>> i have a graph (directed and unweighted) stored in hdfs like a
>>>>>> adjacency list (140Milions of edges 6Milions of vertex)
>>>>>>
>>>>>> node<tab>neighbors
>>>>>>
>>>>>> 23 2 1 3 4 3
>>>>>>
>>>>>> 1 999 99923 909 .......
>>>>>>
>>>>>> ......
>>>>>>
>>>>>> hadoop version Hadoop 2.0.0-cdh4.3.0 - java 1.6
>>>>>>
>>>>>>
>>>>>> I have executed the giraph-1.0 connected components example, in this
>>>>>> fashion
>>>>>>
>>>>>>> > hadoop jar /usr/local/giraph/giraph-examples/target/giraph-
>>>>>>> examples-1.0.0-for-hadoop-0.20.203.0-jar-with-dependencies.jar
>>>>>>> org.apache.giraph.GiraphRunner org.apache.giraph.examples.ConnectedComponentsVertex
>>>>>>> -vif org.apache.giraph.io.formats.IntIntNullTextInputFormat -vip
>>>>>>> /user/hdfs/lista_adj_txt -op connectedgiraph --workers 4
>>>>>>
>>>>>>
>>>>>>
>>>>>> and then fail with:
>>>>>>
>>>>>> 13/10/04 09:28:29 INFO utils.ConfigurationUtils: No edge input format
>>>>>> specified. Ensure your InputFormat does not require one.
>>>>>>
>>>>>>
>>>>>>
>>>>>> 13/10/04 09:28:29 INFO utils.ConfigurationUtils: No output format
>>>>>> specified. Ensure your OutputFormat does not require one.
>>>>>>
>>>>>>
>>>>>>
>>>>>> 13/10/04 09:28:30 INFO job.GiraphJob: run: Since checkpointing is
>>>>>> disabled (default), do not allow any task retries (setting
>>>>>> mapred.map.max.attempts = 0, old value = 4)
>>>>>>
>>>>>>
>>>>>>
>>>>>> 13/10/04 09:28:31 WARN mapred.JobClient: Use GenericOptionsParser for
>>>>>> parsing the arguments. Applications should implement Tool for the same.
>>>>>>
>>>>>>
>>>>>>
>>>>>> 13/10/04 09:28:31 INFO mapred.JobClient: Cleaning up the staging area
>>>>>> hdfs://
>>>>>> srv-bigdata-dev-01.int.sose.it:8020/user/hdfs/.staging/job_201309181636_0535
>>>>>>
>>>>>> Exception in thread "main" java.lang.IncompatibleClassChangeError:
>>>>>> Found interface org.apache.hadoop.mapreduce.JobContext, but class was
>>>>>> expected
>>>>>>
>>>>>> at
>>>>>> org.apache.giraph.bsp.BspOutputFormat.checkOutputSpecs(BspOutputFormat.java:43)
>>>>>>
>>>>>> ......
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> Thanks in advance
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Claudio Martella
>>>>> claudio.martella@gmail.com
>>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> Claudio Martella
>>> claudio.martella@gmail.com
>>>
>>
>>
>
>
> --
> Claudio Martella
> claudio.martella@gmail.com
>
Re: connected components example in giraph 1.0
Posted by Claudio Martella <cl...@gmail.com>.
Oh, right, -vof is in trunk. Anyway it looks like you built giraph for the
wrong profile. You mentioned you're running on 2.0, but your giraph is
built for 0.20.203. try building with a profile for your hadoop version.
On Fri, Oct 4, 2013 at 2:35 PM, Silvio Di gregorio <
silvio.digregorio@gmail.com> wrote:
>
> org.apache.commons.cli.UnrecognizedOptionException: Unrecognized option:
> -vof
>
> in 1.0 version is
> -of,--outputFormat <arg> Vertex output format
> -op,--outputPath <arg> Vertex output path
>
>
>
> 2013/10/4 Claudio Martella <cl...@gmail.com>
>
>> did you try the argument (-vof) i suggested?
>>
>>
>> On Fri, Oct 4, 2013 at 2:13 PM, Silvio Di gregorio <
>> silvio.digregorio@gmail.com> wrote:
>>
>>>
>>> i've specified -of
>>> org.apache.giraph.io.formats.IdWithValueTextOutputFormat
>>>
>>> but the same error was produced
>>>
>>> Exception in thread "main" java.lang.IncompatibleClassChangeError: Found
>>> interface org.apache.hadoop.mapreduce.JobContext, but class was expected
>>> at
>>> org.apache.giraph.bsp.BspOutputFormat.checkOutputSpecs(BspOutputFormat.java:43)
>>> at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:984)
>>> at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:945)
>>> at java.security.AccessController.doPrivileged(Native Method)
>>> at javax.security.auth.Subject.doAs(Subject.java:396)
>>> at
>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
>>> at
>>> org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:945)
>>> at org.apache.hadoop.mapreduce.Job.submit(Job.java:566)
>>> at
>>> org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:596)
>>> at org.apache.giraph.job.GiraphJob.run(GiraphJob.java:237)
>>> at org.apache.giraph.GiraphRunner.run(GiraphRunner.java:94)
>>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
>>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
>>> at org.apache.giraph.GiraphRunner.main(GiraphRunner.java:124)
>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>> at
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>> at
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>> at java.lang.reflect.Method.invoke(Method.java:597)
>>> at org.apache.hadoop.util.RunJar.main(RunJar.java:208)
>>>
>>>
>>>
>>> 2013/10/4 Claudio Martella <cl...@gmail.com>
>>>
>>>> Hi,
>>>>
>>>> you need to specify the vertex outputformat class (-vof option), e.g.
>>>> org.apache.giraph.io.formats.IdWithValueTextOutputFormat.
>>>>
>>>>
>>>> On Fri, Oct 4, 2013 at 1:06 PM, Silvio Di gregorio <
>>>> silvio.digregorio@gmail.com> wrote:
>>>>
>>>>>
>>>>> Hi,
>>>>>
>>>>> I hope I have sent to the right address.
>>>>>
>>>>> i have a graph (directed and unweighted) stored in hdfs like a
>>>>> adjacency list (140Milions of edges 6Milions of vertex)
>>>>>
>>>>> node<tab>neighbors
>>>>>
>>>>> 23 2 1 3 4 3
>>>>>
>>>>> 1 999 99923 909 .......
>>>>>
>>>>> ......
>>>>>
>>>>> hadoop version Hadoop 2.0.0-cdh4.3.0 - java 1.6
>>>>>
>>>>>
>>>>> I have executed the giraph-1.0 connected components example, in this
>>>>> fashion
>>>>>
>>>>>> > hadoop jar /usr/local/giraph/giraph-examples/target/giraph-
>>>>>> examples-1.0.0-for-hadoop-0.20.203.0-jar-with-dependencies.jar
>>>>>> org.apache.giraph.GiraphRunner org.apache.giraph.examples.ConnectedComponentsVertex
>>>>>> -vif org.apache.giraph.io.formats.IntIntNullTextInputFormat -vip
>>>>>> /user/hdfs/lista_adj_txt -op connectedgiraph --workers 4
>>>>>
>>>>>
>>>>>
>>>>> and then fail with:
>>>>>
>>>>> 13/10/04 09:28:29 INFO utils.ConfigurationUtils: No edge input format
>>>>> specified. Ensure your InputFormat does not require one.
>>>>>
>>>>>
>>>>>
>>>>> 13/10/04 09:28:29 INFO utils.ConfigurationUtils: No output format
>>>>> specified. Ensure your OutputFormat does not require one.
>>>>>
>>>>>
>>>>>
>>>>> 13/10/04 09:28:30 INFO job.GiraphJob: run: Since checkpointing is
>>>>> disabled (default), do not allow any task retries (setting
>>>>> mapred.map.max.attempts = 0, old value = 4)
>>>>>
>>>>>
>>>>>
>>>>> 13/10/04 09:28:31 WARN mapred.JobClient: Use GenericOptionsParser for
>>>>> parsing the arguments. Applications should implement Tool for the same.
>>>>>
>>>>>
>>>>>
>>>>> 13/10/04 09:28:31 INFO mapred.JobClient: Cleaning up the staging area
>>>>> hdfs://
>>>>> srv-bigdata-dev-01.int.sose.it:8020/user/hdfs/.staging/job_201309181636_0535
>>>>>
>>>>> Exception in thread "main" java.lang.IncompatibleClassChangeError:
>>>>> Found interface org.apache.hadoop.mapreduce.JobContext, but class was
>>>>> expected
>>>>>
>>>>> at
>>>>> org.apache.giraph.bsp.BspOutputFormat.checkOutputSpecs(BspOutputFormat.java:43)
>>>>>
>>>>> ......
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> Thanks in advance
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Claudio Martella
>>>> claudio.martella@gmail.com
>>>>
>>>
>>>
>>
>>
>> --
>> Claudio Martella
>> claudio.martella@gmail.com
>>
>
>
--
Claudio Martella
claudio.martella@gmail.com
Re: connected components example in giraph 1.0
Posted by Silvio Di gregorio <si...@gmail.com>.
org.apache.commons.cli.UnrecognizedOptionException: Unrecognized option:
-vof
in 1.0 version is
-of,--outputFormat <arg> Vertex output format
-op,--outputPath <arg> Vertex output path
2013/10/4 Claudio Martella <cl...@gmail.com>
> did you try the argument (-vof) i suggested?
>
>
> On Fri, Oct 4, 2013 at 2:13 PM, Silvio Di gregorio <
> silvio.digregorio@gmail.com> wrote:
>
>>
>> i've specified -of
>> org.apache.giraph.io.formats.IdWithValueTextOutputFormat
>>
>> but the same error was produced
>>
>> Exception in thread "main" java.lang.IncompatibleClassChangeError: Found
>> interface org.apache.hadoop.mapreduce.JobContext, but class was expected
>> at
>> org.apache.giraph.bsp.BspOutputFormat.checkOutputSpecs(BspOutputFormat.java:43)
>> at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:984)
>> at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:945)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at javax.security.auth.Subject.doAs(Subject.java:396)
>> at
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
>> at
>> org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:945)
>> at org.apache.hadoop.mapreduce.Job.submit(Job.java:566)
>> at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:596)
>> at org.apache.giraph.job.GiraphJob.run(GiraphJob.java:237)
>> at org.apache.giraph.GiraphRunner.run(GiraphRunner.java:94)
>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
>> at org.apache.giraph.GiraphRunner.main(GiraphRunner.java:124)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> at java.lang.reflect.Method.invoke(Method.java:597)
>> at org.apache.hadoop.util.RunJar.main(RunJar.java:208)
>>
>>
>>
>> 2013/10/4 Claudio Martella <cl...@gmail.com>
>>
>>> Hi,
>>>
>>> you need to specify the vertex outputformat class (-vof option), e.g.
>>> org.apache.giraph.io.formats.IdWithValueTextOutputFormat.
>>>
>>>
>>> On Fri, Oct 4, 2013 at 1:06 PM, Silvio Di gregorio <
>>> silvio.digregorio@gmail.com> wrote:
>>>
>>>>
>>>> Hi,
>>>>
>>>> I hope I have sent to the right address.
>>>>
>>>> i have a graph (directed and unweighted) stored in hdfs like a
>>>> adjacency list (140Milions of edges 6Milions of vertex)
>>>>
>>>> node<tab>neighbors
>>>>
>>>> 23 2 1 3 4 3
>>>>
>>>> 1 999 99923 909 .......
>>>>
>>>> ......
>>>>
>>>> hadoop version Hadoop 2.0.0-cdh4.3.0 - java 1.6
>>>>
>>>>
>>>> I have executed the giraph-1.0 connected components example, in this
>>>> fashion
>>>>
>>>>> > hadoop jar /usr/local/giraph/giraph-examples/target/giraph-
>>>>> examples-1.0.0-for-hadoop-0.20.203.0-jar-with-dependencies.jar
>>>>> org.apache.giraph.GiraphRunner org.apache.giraph.examples.ConnectedComponentsVertex
>>>>> -vif org.apache.giraph.io.formats.IntIntNullTextInputFormat -vip
>>>>> /user/hdfs/lista_adj_txt -op connectedgiraph --workers 4
>>>>
>>>>
>>>>
>>>> and then fail with:
>>>>
>>>> 13/10/04 09:28:29 INFO utils.ConfigurationUtils: No edge input format
>>>> specified. Ensure your InputFormat does not require one.
>>>>
>>>>
>>>>
>>>> 13/10/04 09:28:29 INFO utils.ConfigurationUtils: No output format
>>>> specified. Ensure your OutputFormat does not require one.
>>>>
>>>>
>>>>
>>>> 13/10/04 09:28:30 INFO job.GiraphJob: run: Since checkpointing is
>>>> disabled (default), do not allow any task retries (setting
>>>> mapred.map.max.attempts = 0, old value = 4)
>>>>
>>>>
>>>>
>>>> 13/10/04 09:28:31 WARN mapred.JobClient: Use GenericOptionsParser for
>>>> parsing the arguments. Applications should implement Tool for the same.
>>>>
>>>>
>>>>
>>>> 13/10/04 09:28:31 INFO mapred.JobClient: Cleaning up the staging area
>>>> hdfs://
>>>> srv-bigdata-dev-01.int.sose.it:8020/user/hdfs/.staging/job_201309181636_0535
>>>>
>>>> Exception in thread "main" java.lang.IncompatibleClassChangeError:
>>>> Found interface org.apache.hadoop.mapreduce.JobContext, but class was
>>>> expected
>>>>
>>>> at
>>>> org.apache.giraph.bsp.BspOutputFormat.checkOutputSpecs(BspOutputFormat.java:43)
>>>>
>>>> ......
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> Thanks in advance
>>>>
>>>>
>>>
>>>
>>> --
>>> Claudio Martella
>>> claudio.martella@gmail.com
>>>
>>
>>
>
>
> --
> Claudio Martella
> claudio.martella@gmail.com
>
Re: connected components example in giraph 1.0
Posted by Claudio Martella <cl...@gmail.com>.
did you try the argument (-vof) i suggested?
On Fri, Oct 4, 2013 at 2:13 PM, Silvio Di gregorio <
silvio.digregorio@gmail.com> wrote:
>
> i've specified -of
> org.apache.giraph.io.formats.IdWithValueTextOutputFormat
>
> but the same error was produced
>
> Exception in thread "main" java.lang.IncompatibleClassChangeError: Found
> interface org.apache.hadoop.mapreduce.JobContext, but class was expected
> at
> org.apache.giraph.bsp.BspOutputFormat.checkOutputSpecs(BspOutputFormat.java:43)
> at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:984)
> at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:945)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
> at
> org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:945)
> at org.apache.hadoop.mapreduce.Job.submit(Job.java:566)
> at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:596)
> at org.apache.giraph.job.GiraphJob.run(GiraphJob.java:237)
> at org.apache.giraph.GiraphRunner.run(GiraphRunner.java:94)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
> at org.apache.giraph.GiraphRunner.main(GiraphRunner.java:124)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:208)
>
>
>
> 2013/10/4 Claudio Martella <cl...@gmail.com>
>
>> Hi,
>>
>> you need to specify the vertex outputformat class (-vof option), e.g.
>> org.apache.giraph.io.formats.IdWithValueTextOutputFormat.
>>
>>
>> On Fri, Oct 4, 2013 at 1:06 PM, Silvio Di gregorio <
>> silvio.digregorio@gmail.com> wrote:
>>
>>>
>>> Hi,
>>>
>>> I hope I have sent to the right address.
>>>
>>> i have a graph (directed and unweighted) stored in hdfs like a adjacency
>>> list (140Milions of edges 6Milions of vertex)
>>>
>>> node<tab>neighbors
>>>
>>> 23 2 1 3 4 3
>>>
>>> 1 999 99923 909 .......
>>>
>>> ......
>>>
>>> hadoop version Hadoop 2.0.0-cdh4.3.0 - java 1.6
>>>
>>>
>>> I have executed the giraph-1.0 connected components example, in this
>>> fashion
>>>
>>>> > hadoop jar /usr/local/giraph/giraph-examples/target/giraph-
>>>> examples-1.0.0-for-hadoop-0.20.203.0-jar-with-dependencies.jar
>>>> org.apache.giraph.GiraphRunner org.apache.giraph.examples.ConnectedComponentsVertex
>>>> -vif org.apache.giraph.io.formats.IntIntNullTextInputFormat -vip
>>>> /user/hdfs/lista_adj_txt -op connectedgiraph --workers 4
>>>
>>>
>>>
>>> and then fail with:
>>>
>>> 13/10/04 09:28:29 INFO utils.ConfigurationUtils: No edge input format
>>> specified. Ensure your InputFormat does not require one.
>>>
>>>
>>>
>>> 13/10/04 09:28:29 INFO utils.ConfigurationUtils: No output format
>>> specified. Ensure your OutputFormat does not require one.
>>>
>>>
>>>
>>> 13/10/04 09:28:30 INFO job.GiraphJob: run: Since checkpointing is
>>> disabled (default), do not allow any task retries (setting
>>> mapred.map.max.attempts = 0, old value = 4)
>>>
>>>
>>>
>>> 13/10/04 09:28:31 WARN mapred.JobClient: Use GenericOptionsParser for
>>> parsing the arguments. Applications should implement Tool for the same.
>>>
>>>
>>>
>>> 13/10/04 09:28:31 INFO mapred.JobClient: Cleaning up the staging area
>>> hdfs://
>>> srv-bigdata-dev-01.int.sose.it:8020/user/hdfs/.staging/job_201309181636_0535
>>>
>>> Exception in thread "main" java.lang.IncompatibleClassChangeError: Found
>>> interface org.apache.hadoop.mapreduce.JobContext, but class was expected
>>>
>>> at
>>> org.apache.giraph.bsp.BspOutputFormat.checkOutputSpecs(BspOutputFormat.java:43)
>>>
>>> ......
>>>
>>>
>>>
>>>
>>>
>>> Thanks in advance
>>>
>>>
>>
>>
>> --
>> Claudio Martella
>> claudio.martella@gmail.com
>>
>
>
--
Claudio Martella
claudio.martella@gmail.com
Re: connected components example in giraph 1.0
Posted by Silvio Di gregorio <si...@gmail.com>.
i've specified -of
org.apache.giraph.io.formats.IdWithValueTextOutputFormat
but the same error was produced
Exception in thread "main" java.lang.IncompatibleClassChangeError: Found
interface org.apache.hadoop.mapreduce.JobContext, but class was expected
at
org.apache.giraph.bsp.BspOutputFormat.checkOutputSpecs(BspOutputFormat.java:43)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:984)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:945)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
at
org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:945)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:566)
at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:596)
at org.apache.giraph.job.GiraphJob.run(GiraphJob.java:237)
at org.apache.giraph.GiraphRunner.run(GiraphRunner.java:94)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
at org.apache.giraph.GiraphRunner.main(GiraphRunner.java:124)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:208)
2013/10/4 Claudio Martella <cl...@gmail.com>
> Hi,
>
> you need to specify the vertex outputformat class (-vof option), e.g.
> org.apache.giraph.io.formats.IdWithValueTextOutputFormat.
>
>
> On Fri, Oct 4, 2013 at 1:06 PM, Silvio Di gregorio <
> silvio.digregorio@gmail.com> wrote:
>
>>
>> Hi,
>>
>> I hope I have sent to the right address.
>>
>> i have a graph (directed and unweighted) stored in hdfs like a adjacency
>> list (140Milions of edges 6Milions of vertex)
>>
>> node<tab>neighbors
>>
>> 23 2 1 3 4 3
>>
>> 1 999 99923 909 .......
>>
>> ......
>>
>> hadoop version Hadoop 2.0.0-cdh4.3.0 - java 1.6
>>
>>
>> I have executed the giraph-1.0 connected components example, in this
>> fashion
>>
>>> > hadoop jar /usr/local/giraph/giraph-examples/target/giraph-
>>> examples-1.0.0-for-hadoop-0.20.203.0-jar-with-dependencies.jar
>>> org.apache.giraph.GiraphRunner org.apache.giraph.examples.ConnectedComponentsVertex
>>> -vif org.apache.giraph.io.formats.IntIntNullTextInputFormat -vip
>>> /user/hdfs/lista_adj_txt -op connectedgiraph --workers 4
>>
>>
>>
>> and then fail with:
>>
>> 13/10/04 09:28:29 INFO utils.ConfigurationUtils: No edge input format
>> specified. Ensure your InputFormat does not require one.
>>
>>
>>
>> 13/10/04 09:28:29 INFO utils.ConfigurationUtils: No output format
>> specified. Ensure your OutputFormat does not require one.
>>
>>
>>
>> 13/10/04 09:28:30 INFO job.GiraphJob: run: Since checkpointing is
>> disabled (default), do not allow any task retries (setting
>> mapred.map.max.attempts = 0, old value = 4)
>>
>>
>>
>> 13/10/04 09:28:31 WARN mapred.JobClient: Use GenericOptionsParser for
>> parsing the arguments. Applications should implement Tool for the same.
>>
>>
>>
>> 13/10/04 09:28:31 INFO mapred.JobClient: Cleaning up the staging area
>> hdfs://
>> srv-bigdata-dev-01.int.sose.it:8020/user/hdfs/.staging/job_201309181636_0535
>>
>> Exception in thread "main" java.lang.IncompatibleClassChangeError: Found
>> interface org.apache.hadoop.mapreduce.JobContext, but class was expected
>>
>> at
>> org.apache.giraph.bsp.BspOutputFormat.checkOutputSpecs(BspOutputFormat.java:43)
>>
>> ......
>>
>>
>>
>>
>>
>> Thanks in advance
>>
>>
>
>
> --
> Claudio Martella
> claudio.martella@gmail.com
>
Re: connected components example in giraph 1.0
Posted by Claudio Martella <cl...@gmail.com>.
Hi,
you need to specify the vertex outputformat class (-vof option), e.g.
org.apache.giraph.io.formats.IdWithValueTextOutputFormat.
On Fri, Oct 4, 2013 at 1:06 PM, Silvio Di gregorio <
silvio.digregorio@gmail.com> wrote:
>
> Hi,
>
> I hope I have sent to the right address.
>
> i have a graph (directed and unweighted) stored in hdfs like a adjacency
> list (140Milions of edges 6Milions of vertex)
>
> node<tab>neighbors
>
> 23 2 1 3 4 3
>
> 1 999 99923 909 .......
>
> ......
>
> hadoop version Hadoop 2.0.0-cdh4.3.0 - java 1.6
>
>
> I have executed the giraph-1.0 connected components example, in this
> fashion
>
>> > hadoop jar /usr/local/giraph/giraph-examples/target/giraph-
>> examples-1.0.0-for-hadoop-0.20.203.0-jar-with-dependencies.jar
>> org.apache.giraph.GiraphRunner org.apache.giraph.examples.ConnectedComponentsVertex
>> -vif org.apache.giraph.io.formats.IntIntNullTextInputFormat -vip
>> /user/hdfs/lista_adj_txt -op connectedgiraph --workers 4
>
>
>
> and then fail with:
>
> 13/10/04 09:28:29 INFO utils.ConfigurationUtils: No edge input format
> specified. Ensure your InputFormat does not require one.
>
>
>
> 13/10/04 09:28:29 INFO utils.ConfigurationUtils: No output format
> specified. Ensure your OutputFormat does not require one.
>
>
>
> 13/10/04 09:28:30 INFO job.GiraphJob: run: Since checkpointing is disabled
> (default), do not allow any task retries (setting mapred.map.max.attempts =
> 0, old value = 4)
>
>
>
> 13/10/04 09:28:31 WARN mapred.JobClient: Use GenericOptionsParser for
> parsing the arguments. Applications should implement Tool for the same.
>
>
>
> 13/10/04 09:28:31 INFO mapred.JobClient: Cleaning up the staging area
> hdfs://
> srv-bigdata-dev-01.int.sose.it:8020/user/hdfs/.staging/job_201309181636_0535
>
> Exception in thread "main" java.lang.IncompatibleClassChangeError: Found
> interface org.apache.hadoop.mapreduce.JobContext, but class was expected
>
> at
> org.apache.giraph.bsp.BspOutputFormat.checkOutputSpecs(BspOutputFormat.java:43)
>
> ......
>
>
>
>
>
> Thanks in advance
>
>
--
Claudio Martella
claudio.martella@gmail.com