You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@oozie.apache.org by Christian Nardi <ch...@gmail.com> on 2012/03/01 22:40:30 UTC
Giraph
Hi all,
Does anybody know if there is any way to lunch a giraph job from oozie?
Thanks
Christian
Re: Giraph
Posted by Mayank Bansal <ma...@gmail.com>.
please paste the error here which you are getting while running java
action. It may be u have to set the classpath for zk in each task tracker.
Thanks,
Mayank
On Thu, Mar 1, 2012 at 2:26 PM, Christian Nardi
<ch...@gmail.com>wrote:
> I tried with a Map Reduce job, but it tries to connect with a
> Zookeeper server 10 times and then fails. I guess this is because
> GiraphJob does something that standard hadoop jobs does not...
>
>
> On Thu, Mar 1, 2012 at 7:23 PM, Mayank Bansal <ma...@apache.org> wrote:
> > can't we use java action?
> >
> > Thanks,
> > Mayank
> >
> > On Thu, Mar 1, 2012 at 2:19 PM, Params <pa...@gmail.com> wrote:
> >
> >> Hi Christian,
> >>
> >> This is a way to do this (not sure if its the best way though). You can
> >> write a pig script that makes a native MAPREDUCE call with the required
> >> libraries and arguments for your giraph job. And, then use a <pig>
> action
> >> to run it via Oozie.
> >>
> >> You can look at:
> >> http://pig.apache.org/docs/r0.8.1/piglatin_ref2.html#MAPREDUCE
> >>
> >> Basically, its a way to convert a giraph job that you would submit via
> >> command line (such as below) into pig .
> >> $ hadoop jar giraph-0.70-jar-with-dependencies.jar
> >> org.apache.giraph.benchmark.PageRankBenchmark -e 1 -s 3 -v -V 50000000
> -w
> >> 30
> >>
> >>
> >>
> >> Thanks,
> >> --
> >> Params
> >>
> >> On Thu, Mar 1, 2012 at 1:40 PM, Christian Nardi
> >> <ch...@gmail.com>wrote:
> >>
> >> > Hi all,
> >> >
> >> > Does anybody know if there is any way to lunch a giraph job from
> oozie?
> >> >
> >> > Thanks
> >> > Christian
> >> >
> >>
> >>
> >>
> >> --
> >> params
> >>
>
--
Thanks and Regards,
Mayank
Cell: 408-718-9370
Re: Giraph
Posted by Christian Nardi <ch...@gmail.com>.
As I supposed, it was a dependencies problem.
It worked OK as a java action.
Thank you all for your help and suggestions
Christian
On Thu, Mar 1, 2012 at 11:05 PM, Mohammad Islam <mi...@yahoo.com> wrote:
> yes. it looks strange.
> For testing purpose, can you refer or instantiate the missing class in your Java code before calling Giraph.
> I suspect the class is available in the Java code but it was not passed to Giraph properly when you called .
>
> Based on this, we can look for the solution.
>
> Btw, if possible exchange some relevant section of you java code. Like how you execute Giraph. Because it might be important. I believe none of us (Params!) actually has used it.
>
> Regards,
> Mohammad
>
>
>
>
> ----- Original Message -----
> From: Christian Nardi <ch...@gmail.com>
> To: oozie-users@incubator.apache.org; Mohammad Islam <mi...@yahoo.com>
> Cc:
> Sent: Thursday, March 1, 2012 4:46 PM
> Subject: Re: Giraph
>
> Yes, It is. In fact it is the same jar that contains the main class
> that i´m trying to execute.
> There are also other jars in the lib directory because other parts of
> the workflow require them. Maybe there´s a problem with another jar
> (maybe some colision with versions or something). Is there any way of
> limiting the classpath libraries included in the execution?
>
>
>
> On Thu, Mar 1, 2012 at 9:32 PM, Mohammad Islam <mi...@yahoo.com> wrote:
>> Hi Christian,
>> Stack trace shows the some giraph libraries are already available like
>> org.apache.giraph.
>>
>> Looks like missing class "com.company.MyVertexOutputFormat" is not a Giraph class.
>> It is a user class. is it right?
>> In that case, was that library also included into WF/lib directory and could be seen in Launcher Mapper log.
>>
>> Regards,
>> Mohammad
>>
>>
>> ________________________________
>> From: Christian Nardi <ch...@gmail.com>
>> To: oozie-users@incubator.apache.org
>> Sent: Thursday, March 1, 2012 2:39 PM
>> Subject: Re: Giraph
>>
>> I know, it is there and it appears in the classpath logged on the map
>> job in hadoop, but it throws:
>>
>> java.lang.RuntimeException: java.lang.RuntimeException:
>> java.lang.ClassNotFoundException: com.company.MyVertexOutputFormat
>> at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:898)
>> at org.apache.giraph.graph.BspUtils.getVertexOutputFormatClass(BspUtils.java:161)
>> at org.apache.giraph.bsp.BspOutputFormat.getOutputCommitter(BspOutputFormat.java:56)
>> at org.apache.hadoop.mapred.Task.initialize(Task.java:515)
>> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:353)
>> at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at javax.security.auth.Subject.doAs(Subject.java:396)
>> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
>> at org.apache.hadoop.mapred.Child.main(Child.java:249)
>> Caused by: java.lang.RuntimeException:
>> java.lang.ClassNotFoundException: com.company.MyVertexOutputFormat
>> at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:866)
>> at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:890)
>> ... 9 more
>>
>> And that sounds really strange...That should be the right way...I must
>> be making a silly mistake...I will keep trying and will let you
>> know...
>>
>> Thanks
>>
>>
>> On Thu, Mar 1, 2012 at 7:32 PM, Mayank Bansal <ma...@apache.org> wrote:
>>> for the class not found exception you have to put that jar to lib directory
>>> of workflow
>>>
>>> On Thu, Mar 1, 2012 at 2:30 PM, Christian Nardi
>>> <ch...@gmail.com>wrote:
>>>
>>>> And with standard java actions, it throws a ClassNotFoundException of
>>>> one of my classes
>>>>
>>>> On Thu, Mar 1, 2012 at 7:26 PM, Christian Nardi
>>>> <ch...@gmail.com> wrote:
>>>> > I tried with a Map Reduce job, but it tries to connect with a
>>>> > Zookeeper server 10 times and then fails. I guess this is because
>>>> > GiraphJob does something that standard hadoop jobs does not...
>>>> >
>>>> >
>>>> > On Thu, Mar 1, 2012 at 7:23 PM, Mayank Bansal <ma...@apache.org> wrote:
>>>> >> can't we use java action?
>>>> >>
>>>> >> Thanks,
>>>> >> Mayank
>>>> >>
>>>> >> On Thu, Mar 1, 2012 at 2:19 PM, Params <pa...@gmail.com> wrote:
>>>> >>
>>>> >>> Hi Christian,
>>>> >>>
>>>> >>> This is a way to do this (not sure if its the best way though). You can
>>>> >>> write a pig script that makes a native MAPREDUCE call with the required
>>>> >>> libraries and arguments for your giraph job. And, then use a <pig>
>>>> action
>>>> >>> to run it via Oozie.
>>>> >>>
>>>> >>> You can look at:
>>>> >>> http://pig.apache.org/docs/r0.8.1/piglatin_ref2.html#MAPREDUCE
>>>> >>>
>>>> >>> Basically, its a way to convert a giraph job that you would submit via
>>>> >>> command line (such as below) into pig .
>>>> >>> $ hadoop jar giraph-0.70-jar-with-dependencies.jar
>>>> >>> org.apache.giraph.benchmark.PageRankBenchmark -e 1 -s 3 -v -V 50000000
>>>> -w
>>>> >>> 30
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>> Thanks,
>>>> >>> --
>>>> >>> Params
>>>> >>>
>>>> >>> On Thu, Mar 1, 2012 at 1:40 PM, Christian Nardi
>>>> >>> <ch...@gmail.com>wrote:
>>>> >>>
>>>> >>> > Hi all,
>>>> >>> >
>>>> >>> > Does anybody know if there is any way to lunch a giraph job from
>>>> oozie?
>>>> >>> >
>>>> >>> > Thanks
>>>> >>> > Christian
>>>> >>> >
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>> --
>>>> >>> params
>>>> >>>
>>>>
>
Re: Giraph
Posted by Mohammad Islam <mi...@yahoo.com>.
yes. it looks strange.
For testing purpose, can you refer or instantiate the missing class in your Java code before calling Giraph.
I suspect the class is available in the Java code but it was not passed to Giraph properly when you called .
Based on this, we can look for the solution.
Btw, if possible exchange some relevant section of you java code. Like how you execute Giraph. Because it might be important. I believe none of us (Params!) actually has used it.
Regards,
Mohammad
----- Original Message -----
From: Christian Nardi <ch...@gmail.com>
To: oozie-users@incubator.apache.org; Mohammad Islam <mi...@yahoo.com>
Cc:
Sent: Thursday, March 1, 2012 4:46 PM
Subject: Re: Giraph
Yes, It is. In fact it is the same jar that contains the main class
that i´m trying to execute.
There are also other jars in the lib directory because other parts of
the workflow require them. Maybe there´s a problem with another jar
(maybe some colision with versions or something). Is there any way of
limiting the classpath libraries included in the execution?
On Thu, Mar 1, 2012 at 9:32 PM, Mohammad Islam <mi...@yahoo.com> wrote:
> Hi Christian,
> Stack trace shows the some giraph libraries are already available like
> org.apache.giraph.
>
> Looks like missing class "com.company.MyVertexOutputFormat" is not a Giraph class.
> It is a user class. is it right?
> In that case, was that library also included into WF/lib directory and could be seen in Launcher Mapper log.
>
> Regards,
> Mohammad
>
>
> ________________________________
> From: Christian Nardi <ch...@gmail.com>
> To: oozie-users@incubator.apache.org
> Sent: Thursday, March 1, 2012 2:39 PM
> Subject: Re: Giraph
>
> I know, it is there and it appears in the classpath logged on the map
> job in hadoop, but it throws:
>
> java.lang.RuntimeException: java.lang.RuntimeException:
> java.lang.ClassNotFoundException: com.company.MyVertexOutputFormat
> at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:898)
> at org.apache.giraph.graph.BspUtils.getVertexOutputFormatClass(BspUtils.java:161)
> at org.apache.giraph.bsp.BspOutputFormat.getOutputCommitter(BspOutputFormat.java:56)
> at org.apache.hadoop.mapred.Task.initialize(Task.java:515)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:353)
> at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
> at org.apache.hadoop.mapred.Child.main(Child.java:249)
> Caused by: java.lang.RuntimeException:
> java.lang.ClassNotFoundException: com.company.MyVertexOutputFormat
> at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:866)
> at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:890)
> ... 9 more
>
> And that sounds really strange...That should be the right way...I must
> be making a silly mistake...I will keep trying and will let you
> know...
>
> Thanks
>
>
> On Thu, Mar 1, 2012 at 7:32 PM, Mayank Bansal <ma...@apache.org> wrote:
>> for the class not found exception you have to put that jar to lib directory
>> of workflow
>>
>> On Thu, Mar 1, 2012 at 2:30 PM, Christian Nardi
>> <ch...@gmail.com>wrote:
>>
>>> And with standard java actions, it throws a ClassNotFoundException of
>>> one of my classes
>>>
>>> On Thu, Mar 1, 2012 at 7:26 PM, Christian Nardi
>>> <ch...@gmail.com> wrote:
>>> > I tried with a Map Reduce job, but it tries to connect with a
>>> > Zookeeper server 10 times and then fails. I guess this is because
>>> > GiraphJob does something that standard hadoop jobs does not...
>>> >
>>> >
>>> > On Thu, Mar 1, 2012 at 7:23 PM, Mayank Bansal <ma...@apache.org> wrote:
>>> >> can't we use java action?
>>> >>
>>> >> Thanks,
>>> >> Mayank
>>> >>
>>> >> On Thu, Mar 1, 2012 at 2:19 PM, Params <pa...@gmail.com> wrote:
>>> >>
>>> >>> Hi Christian,
>>> >>>
>>> >>> This is a way to do this (not sure if its the best way though). You can
>>> >>> write a pig script that makes a native MAPREDUCE call with the required
>>> >>> libraries and arguments for your giraph job. And, then use a <pig>
>>> action
>>> >>> to run it via Oozie.
>>> >>>
>>> >>> You can look at:
>>> >>> http://pig.apache.org/docs/r0.8.1/piglatin_ref2.html#MAPREDUCE
>>> >>>
>>> >>> Basically, its a way to convert a giraph job that you would submit via
>>> >>> command line (such as below) into pig .
>>> >>> $ hadoop jar giraph-0.70-jar-with-dependencies.jar
>>> >>> org.apache.giraph.benchmark.PageRankBenchmark -e 1 -s 3 -v -V 50000000
>>> -w
>>> >>> 30
>>> >>>
>>> >>>
>>> >>>
>>> >>> Thanks,
>>> >>> --
>>> >>> Params
>>> >>>
>>> >>> On Thu, Mar 1, 2012 at 1:40 PM, Christian Nardi
>>> >>> <ch...@gmail.com>wrote:
>>> >>>
>>> >>> > Hi all,
>>> >>> >
>>> >>> > Does anybody know if there is any way to lunch a giraph job from
>>> oozie?
>>> >>> >
>>> >>> > Thanks
>>> >>> > Christian
>>> >>> >
>>> >>>
>>> >>>
>>> >>>
>>> >>> --
>>> >>> params
>>> >>>
>>>
Re: Giraph
Posted by Christian Nardi <ch...@gmail.com>.
Yes, It is. In fact it is the same jar that contains the main class
that i´m trying to execute.
There are also other jars in the lib directory because other parts of
the workflow require them. Maybe there´s a problem with another jar
(maybe some colision with versions or something). Is there any way of
limiting the classpath libraries included in the execution?
On Thu, Mar 1, 2012 at 9:32 PM, Mohammad Islam <mi...@yahoo.com> wrote:
> Hi Christian,
> Stack trace shows the some giraph libraries are already available like
> org.apache.giraph.
>
> Looks like missing class "com.company.MyVertexOutputFormat" is not a Giraph class.
> It is a user class. is it right?
> In that case, was that library also included into WF/lib directory and could be seen in Launcher Mapper log.
>
> Regards,
> Mohammad
>
>
> ________________________________
> From: Christian Nardi <ch...@gmail.com>
> To: oozie-users@incubator.apache.org
> Sent: Thursday, March 1, 2012 2:39 PM
> Subject: Re: Giraph
>
> I know, it is there and it appears in the classpath logged on the map
> job in hadoop, but it throws:
>
> java.lang.RuntimeException: java.lang.RuntimeException:
> java.lang.ClassNotFoundException: com.company.MyVertexOutputFormat
> at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:898)
> at org.apache.giraph.graph.BspUtils.getVertexOutputFormatClass(BspUtils.java:161)
> at org.apache.giraph.bsp.BspOutputFormat.getOutputCommitter(BspOutputFormat.java:56)
> at org.apache.hadoop.mapred.Task.initialize(Task.java:515)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:353)
> at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
> at org.apache.hadoop.mapred.Child.main(Child.java:249)
> Caused by: java.lang.RuntimeException:
> java.lang.ClassNotFoundException: com.company.MyVertexOutputFormat
> at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:866)
> at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:890)
> ... 9 more
>
> And that sounds really strange...That should be the right way...I must
> be making a silly mistake...I will keep trying and will let you
> know...
>
> Thanks
>
>
> On Thu, Mar 1, 2012 at 7:32 PM, Mayank Bansal <ma...@apache.org> wrote:
>> for the class not found exception you have to put that jar to lib directory
>> of workflow
>>
>> On Thu, Mar 1, 2012 at 2:30 PM, Christian Nardi
>> <ch...@gmail.com>wrote:
>>
>>> And with standard java actions, it throws a ClassNotFoundException of
>>> one of my classes
>>>
>>> On Thu, Mar 1, 2012 at 7:26 PM, Christian Nardi
>>> <ch...@gmail.com> wrote:
>>> > I tried with a Map Reduce job, but it tries to connect with a
>>> > Zookeeper server 10 times and then fails. I guess this is because
>>> > GiraphJob does something that standard hadoop jobs does not...
>>> >
>>> >
>>> > On Thu, Mar 1, 2012 at 7:23 PM, Mayank Bansal <ma...@apache.org> wrote:
>>> >> can't we use java action?
>>> >>
>>> >> Thanks,
>>> >> Mayank
>>> >>
>>> >> On Thu, Mar 1, 2012 at 2:19 PM, Params <pa...@gmail.com> wrote:
>>> >>
>>> >>> Hi Christian,
>>> >>>
>>> >>> This is a way to do this (not sure if its the best way though). You can
>>> >>> write a pig script that makes a native MAPREDUCE call with the required
>>> >>> libraries and arguments for your giraph job. And, then use a <pig>
>>> action
>>> >>> to run it via Oozie.
>>> >>>
>>> >>> You can look at:
>>> >>> http://pig.apache.org/docs/r0.8.1/piglatin_ref2.html#MAPREDUCE
>>> >>>
>>> >>> Basically, its a way to convert a giraph job that you would submit via
>>> >>> command line (such as below) into pig .
>>> >>> $ hadoop jar giraph-0.70-jar-with-dependencies.jar
>>> >>> org.apache.giraph.benchmark.PageRankBenchmark -e 1 -s 3 -v -V 50000000
>>> -w
>>> >>> 30
>>> >>>
>>> >>>
>>> >>>
>>> >>> Thanks,
>>> >>> --
>>> >>> Params
>>> >>>
>>> >>> On Thu, Mar 1, 2012 at 1:40 PM, Christian Nardi
>>> >>> <ch...@gmail.com>wrote:
>>> >>>
>>> >>> > Hi all,
>>> >>> >
>>> >>> > Does anybody know if there is any way to lunch a giraph job from
>>> oozie?
>>> >>> >
>>> >>> > Thanks
>>> >>> > Christian
>>> >>> >
>>> >>>
>>> >>>
>>> >>>
>>> >>> --
>>> >>> params
>>> >>>
>>>
Re: Giraph
Posted by Mohammad Islam <mi...@yahoo.com>.
Hi Christian,
Stack trace shows the some giraph libraries are already available like
org.apache.giraph.
Looks like missing class "com.company.MyVertexOutputFormat" is not a Giraph class.
It is a user class. is it right?
In that case, was that library also included into WF/lib directory and could be seen in Launcher Mapper log.
Regards,
Mohammad
________________________________
From: Christian Nardi <ch...@gmail.com>
To: oozie-users@incubator.apache.org
Sent: Thursday, March 1, 2012 2:39 PM
Subject: Re: Giraph
I know, it is there and it appears in the classpath logged on the map
job in hadoop, but it throws:
java.lang.RuntimeException: java.lang.RuntimeException:
java.lang.ClassNotFoundException: com.company.MyVertexOutputFormat
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:898)
at org.apache.giraph.graph.BspUtils.getVertexOutputFormatClass(BspUtils.java:161)
at org.apache.giraph.bsp.BspOutputFormat.getOutputCommitter(BspOutputFormat.java:56)
at org.apache.hadoop.mapred.Task.initialize(Task.java:515)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:353)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
at org.apache.hadoop.mapred.Child.main(Child.java:249)
Caused by: java.lang.RuntimeException:
java.lang.ClassNotFoundException: com.company.MyVertexOutputFormat
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:866)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:890)
... 9 more
And that sounds really strange...That should be the right way...I must
be making a silly mistake...I will keep trying and will let you
know...
Thanks
On Thu, Mar 1, 2012 at 7:32 PM, Mayank Bansal <ma...@apache.org> wrote:
> for the class not found exception you have to put that jar to lib directory
> of workflow
>
> On Thu, Mar 1, 2012 at 2:30 PM, Christian Nardi
> <ch...@gmail.com>wrote:
>
>> And with standard java actions, it throws a ClassNotFoundException of
>> one of my classes
>>
>> On Thu, Mar 1, 2012 at 7:26 PM, Christian Nardi
>> <ch...@gmail.com> wrote:
>> > I tried with a Map Reduce job, but it tries to connect with a
>> > Zookeeper server 10 times and then fails. I guess this is because
>> > GiraphJob does something that standard hadoop jobs does not...
>> >
>> >
>> > On Thu, Mar 1, 2012 at 7:23 PM, Mayank Bansal <ma...@apache.org> wrote:
>> >> can't we use java action?
>> >>
>> >> Thanks,
>> >> Mayank
>> >>
>> >> On Thu, Mar 1, 2012 at 2:19 PM, Params <pa...@gmail.com> wrote:
>> >>
>> >>> Hi Christian,
>> >>>
>> >>> This is a way to do this (not sure if its the best way though). You can
>> >>> write a pig script that makes a native MAPREDUCE call with the required
>> >>> libraries and arguments for your giraph job. And, then use a <pig>
>> action
>> >>> to run it via Oozie.
>> >>>
>> >>> You can look at:
>> >>> http://pig.apache.org/docs/r0.8.1/piglatin_ref2.html#MAPREDUCE
>> >>>
>> >>> Basically, its a way to convert a giraph job that you would submit via
>> >>> command line (such as below) into pig .
>> >>> $ hadoop jar giraph-0.70-jar-with-dependencies.jar
>> >>> org.apache.giraph.benchmark.PageRankBenchmark -e 1 -s 3 -v -V 50000000
>> -w
>> >>> 30
>> >>>
>> >>>
>> >>>
>> >>> Thanks,
>> >>> --
>> >>> Params
>> >>>
>> >>> On Thu, Mar 1, 2012 at 1:40 PM, Christian Nardi
>> >>> <ch...@gmail.com>wrote:
>> >>>
>> >>> > Hi all,
>> >>> >
>> >>> > Does anybody know if there is any way to lunch a giraph job from
>> oozie?
>> >>> >
>> >>> > Thanks
>> >>> > Christian
>> >>> >
>> >>>
>> >>>
>> >>>
>> >>> --
>> >>> params
>> >>>
>>
Re: Giraph
Posted by Christian Nardi <ch...@gmail.com>.
I know, it is there and it appears in the classpath logged on the map
job in hadoop, but it throws:
java.lang.RuntimeException: java.lang.RuntimeException:
java.lang.ClassNotFoundException: com.company.MyVertexOutputFormat
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:898)
at org.apache.giraph.graph.BspUtils.getVertexOutputFormatClass(BspUtils.java:161)
at org.apache.giraph.bsp.BspOutputFormat.getOutputCommitter(BspOutputFormat.java:56)
at org.apache.hadoop.mapred.Task.initialize(Task.java:515)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:353)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
at org.apache.hadoop.mapred.Child.main(Child.java:249)
Caused by: java.lang.RuntimeException:
java.lang.ClassNotFoundException: com.company.MyVertexOutputFormat
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:866)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:890)
... 9 more
And that sounds really strange...That should be the right way...I must
be making a silly mistake...I will keep trying and will let you
know...
Thanks
On Thu, Mar 1, 2012 at 7:32 PM, Mayank Bansal <ma...@apache.org> wrote:
> for the class not found exception you have to put that jar to lib directory
> of workflow
>
> On Thu, Mar 1, 2012 at 2:30 PM, Christian Nardi
> <ch...@gmail.com>wrote:
>
>> And with standard java actions, it throws a ClassNotFoundException of
>> one of my classes
>>
>> On Thu, Mar 1, 2012 at 7:26 PM, Christian Nardi
>> <ch...@gmail.com> wrote:
>> > I tried with a Map Reduce job, but it tries to connect with a
>> > Zookeeper server 10 times and then fails. I guess this is because
>> > GiraphJob does something that standard hadoop jobs does not...
>> >
>> >
>> > On Thu, Mar 1, 2012 at 7:23 PM, Mayank Bansal <ma...@apache.org> wrote:
>> >> can't we use java action?
>> >>
>> >> Thanks,
>> >> Mayank
>> >>
>> >> On Thu, Mar 1, 2012 at 2:19 PM, Params <pa...@gmail.com> wrote:
>> >>
>> >>> Hi Christian,
>> >>>
>> >>> This is a way to do this (not sure if its the best way though). You can
>> >>> write a pig script that makes a native MAPREDUCE call with the required
>> >>> libraries and arguments for your giraph job. And, then use a <pig>
>> action
>> >>> to run it via Oozie.
>> >>>
>> >>> You can look at:
>> >>> http://pig.apache.org/docs/r0.8.1/piglatin_ref2.html#MAPREDUCE
>> >>>
>> >>> Basically, its a way to convert a giraph job that you would submit via
>> >>> command line (such as below) into pig .
>> >>> $ hadoop jar giraph-0.70-jar-with-dependencies.jar
>> >>> org.apache.giraph.benchmark.PageRankBenchmark -e 1 -s 3 -v -V 50000000
>> -w
>> >>> 30
>> >>>
>> >>>
>> >>>
>> >>> Thanks,
>> >>> --
>> >>> Params
>> >>>
>> >>> On Thu, Mar 1, 2012 at 1:40 PM, Christian Nardi
>> >>> <ch...@gmail.com>wrote:
>> >>>
>> >>> > Hi all,
>> >>> >
>> >>> > Does anybody know if there is any way to lunch a giraph job from
>> oozie?
>> >>> >
>> >>> > Thanks
>> >>> > Christian
>> >>> >
>> >>>
>> >>>
>> >>>
>> >>> --
>> >>> params
>> >>>
>>
Re: Giraph
Posted by Mayank Bansal <ma...@apache.org>.
for the class not found exception you have to put that jar to lib directory
of workflow
On Thu, Mar 1, 2012 at 2:30 PM, Christian Nardi
<ch...@gmail.com>wrote:
> And with standard java actions, it throws a ClassNotFoundException of
> one of my classes
>
> On Thu, Mar 1, 2012 at 7:26 PM, Christian Nardi
> <ch...@gmail.com> wrote:
> > I tried with a Map Reduce job, but it tries to connect with a
> > Zookeeper server 10 times and then fails. I guess this is because
> > GiraphJob does something that standard hadoop jobs does not...
> >
> >
> > On Thu, Mar 1, 2012 at 7:23 PM, Mayank Bansal <ma...@apache.org> wrote:
> >> can't we use java action?
> >>
> >> Thanks,
> >> Mayank
> >>
> >> On Thu, Mar 1, 2012 at 2:19 PM, Params <pa...@gmail.com> wrote:
> >>
> >>> Hi Christian,
> >>>
> >>> This is a way to do this (not sure if its the best way though). You can
> >>> write a pig script that makes a native MAPREDUCE call with the required
> >>> libraries and arguments for your giraph job. And, then use a <pig>
> action
> >>> to run it via Oozie.
> >>>
> >>> You can look at:
> >>> http://pig.apache.org/docs/r0.8.1/piglatin_ref2.html#MAPREDUCE
> >>>
> >>> Basically, its a way to convert a giraph job that you would submit via
> >>> command line (such as below) into pig .
> >>> $ hadoop jar giraph-0.70-jar-with-dependencies.jar
> >>> org.apache.giraph.benchmark.PageRankBenchmark -e 1 -s 3 -v -V 50000000
> -w
> >>> 30
> >>>
> >>>
> >>>
> >>> Thanks,
> >>> --
> >>> Params
> >>>
> >>> On Thu, Mar 1, 2012 at 1:40 PM, Christian Nardi
> >>> <ch...@gmail.com>wrote:
> >>>
> >>> > Hi all,
> >>> >
> >>> > Does anybody know if there is any way to lunch a giraph job from
> oozie?
> >>> >
> >>> > Thanks
> >>> > Christian
> >>> >
> >>>
> >>>
> >>>
> >>> --
> >>> params
> >>>
>
Re: Giraph
Posted by Christian Nardi <ch...@gmail.com>.
And with standard java actions, it throws a ClassNotFoundException of
one of my classes
On Thu, Mar 1, 2012 at 7:26 PM, Christian Nardi
<ch...@gmail.com> wrote:
> I tried with a Map Reduce job, but it tries to connect with a
> Zookeeper server 10 times and then fails. I guess this is because
> GiraphJob does something that standard hadoop jobs does not...
>
>
> On Thu, Mar 1, 2012 at 7:23 PM, Mayank Bansal <ma...@apache.org> wrote:
>> can't we use java action?
>>
>> Thanks,
>> Mayank
>>
>> On Thu, Mar 1, 2012 at 2:19 PM, Params <pa...@gmail.com> wrote:
>>
>>> Hi Christian,
>>>
>>> This is a way to do this (not sure if its the best way though). You can
>>> write a pig script that makes a native MAPREDUCE call with the required
>>> libraries and arguments for your giraph job. And, then use a <pig> action
>>> to run it via Oozie.
>>>
>>> You can look at:
>>> http://pig.apache.org/docs/r0.8.1/piglatin_ref2.html#MAPREDUCE
>>>
>>> Basically, its a way to convert a giraph job that you would submit via
>>> command line (such as below) into pig .
>>> $ hadoop jar giraph-0.70-jar-with-dependencies.jar
>>> org.apache.giraph.benchmark.PageRankBenchmark -e 1 -s 3 -v -V 50000000 -w
>>> 30
>>>
>>>
>>>
>>> Thanks,
>>> --
>>> Params
>>>
>>> On Thu, Mar 1, 2012 at 1:40 PM, Christian Nardi
>>> <ch...@gmail.com>wrote:
>>>
>>> > Hi all,
>>> >
>>> > Does anybody know if there is any way to lunch a giraph job from oozie?
>>> >
>>> > Thanks
>>> > Christian
>>> >
>>>
>>>
>>>
>>> --
>>> params
>>>
Re: Giraph
Posted by Christian Nardi <ch...@gmail.com>.
I tried with a Map Reduce job, but it tries to connect with a
Zookeeper server 10 times and then fails. I guess this is because
GiraphJob does something that standard hadoop jobs does not...
On Thu, Mar 1, 2012 at 7:23 PM, Mayank Bansal <ma...@apache.org> wrote:
> can't we use java action?
>
> Thanks,
> Mayank
>
> On Thu, Mar 1, 2012 at 2:19 PM, Params <pa...@gmail.com> wrote:
>
>> Hi Christian,
>>
>> This is a way to do this (not sure if its the best way though). You can
>> write a pig script that makes a native MAPREDUCE call with the required
>> libraries and arguments for your giraph job. And, then use a <pig> action
>> to run it via Oozie.
>>
>> You can look at:
>> http://pig.apache.org/docs/r0.8.1/piglatin_ref2.html#MAPREDUCE
>>
>> Basically, its a way to convert a giraph job that you would submit via
>> command line (such as below) into pig .
>> $ hadoop jar giraph-0.70-jar-with-dependencies.jar
>> org.apache.giraph.benchmark.PageRankBenchmark -e 1 -s 3 -v -V 50000000 -w
>> 30
>>
>>
>>
>> Thanks,
>> --
>> Params
>>
>> On Thu, Mar 1, 2012 at 1:40 PM, Christian Nardi
>> <ch...@gmail.com>wrote:
>>
>> > Hi all,
>> >
>> > Does anybody know if there is any way to lunch a giraph job from oozie?
>> >
>> > Thanks
>> > Christian
>> >
>>
>>
>>
>> --
>> params
>>
Re: Giraph
Posted by Mayank Bansal <ma...@apache.org>.
can't we use java action?
Thanks,
Mayank
On Thu, Mar 1, 2012 at 2:19 PM, Params <pa...@gmail.com> wrote:
> Hi Christian,
>
> This is a way to do this (not sure if its the best way though). You can
> write a pig script that makes a native MAPREDUCE call with the required
> libraries and arguments for your giraph job. And, then use a <pig> action
> to run it via Oozie.
>
> You can look at:
> http://pig.apache.org/docs/r0.8.1/piglatin_ref2.html#MAPREDUCE
>
> Basically, its a way to convert a giraph job that you would submit via
> command line (such as below) into pig .
> $ hadoop jar giraph-0.70-jar-with-dependencies.jar
> org.apache.giraph.benchmark.PageRankBenchmark -e 1 -s 3 -v -V 50000000 -w
> 30
>
>
>
> Thanks,
> --
> Params
>
> On Thu, Mar 1, 2012 at 1:40 PM, Christian Nardi
> <ch...@gmail.com>wrote:
>
> > Hi all,
> >
> > Does anybody know if there is any way to lunch a giraph job from oozie?
> >
> > Thanks
> > Christian
> >
>
>
>
> --
> params
>
Re: Giraph
Posted by Params <pa...@gmail.com>.
Hi Christian,
This is a way to do this (not sure if its the best way though). You can
write a pig script that makes a native MAPREDUCE call with the required
libraries and arguments for your giraph job. And, then use a <pig> action
to run it via Oozie.
You can look at:
http://pig.apache.org/docs/r0.8.1/piglatin_ref2.html#MAPREDUCE
Basically, its a way to convert a giraph job that you would submit via
command line (such as below) into pig .
$ hadoop jar giraph-0.70-jar-with-dependencies.jar
org.apache.giraph.benchmark.PageRankBenchmark -e 1 -s 3 -v -V 50000000 -w 30
Thanks,
--
Params
On Thu, Mar 1, 2012 at 1:40 PM, Christian Nardi
<ch...@gmail.com>wrote:
> Hi all,
>
> Does anybody know if there is any way to lunch a giraph job from oozie?
>
> Thanks
> Christian
>
--
params