You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@giraph.apache.org by David Garcia <dg...@potomacfusion.com> on 2012/02/09 02:17:12 UTC
Giraph Architecture bug in
This is a very subtle bug. GiraphJob inherits from
org.apache.mapreduce.Job. However, the methods submit() and
waitForCompletion() are not overridden. I assumed that they were
implemented, so when I called either one of these methods, the framework
started up identity mappers/reducers. A simple fix is to throw
unsupported operation exceptions or to implement these methods. Perhaps
this has been done already?
-David
On 2/7/12 7:46 PM, "David Garcia" <dg...@potomacfusion.com> wrote:
>I am running into a weird error that I haven't seen yet (I suppose I've
>been lucky). I see the following in the logging:
>
>org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop
>library for your platform... using builtin-java classes where applicable
>
>
>In the job definition, the property "mapreduce.map.class" is not even
>defined. For Giraph, this is usually set to
>"mapreduce.map.class=org.apache.giraph.graph.GraphMapper"
>
>I'm building my project with hadoop 0.20.204.
>
>When I build the GiraphProject myself (and run my own tests with the
>projects dependencies), I have no problems. The main difference is that
>I'm using a Giraph dependency in my work project. All input is welcome.
>Thx!!
>
>-David
>
Re: Giraph Architecture bug in
Posted by Avery Ching <ac...@apache.org>.
AFAIK we don't have any SOP for opening issues. Maybe I'll take a crack
at this one tonight if I find some time, unless you were planning to
work on it David.
Avery
On 2/8/12 5:46 PM, David Garcia wrote:
> I opened up
>
> * GIRAPH-144<https://issues.apache.org/jira/browse/GIRAPH-144>
>
>
> I apologize if I didn't do it up according to project SOP's. I haven't
> had time to read it thoroughly.
>
> -David
>
>
> On 2/8/12 7:29 PM, "David Garcia"<dg...@potomacfusion.com> wrote:
>
>> Yeah, I'll write something up.
>>
>>
>> On 2/8/12 7:26 PM, "Avery Ching"<ac...@apache.org> wrote:
>>
>>> Since we call waitForCompletion() (which calls submit() internally) in
>>> GiraphJob#run(), we cannot override those methods. A better fix would
>>> probably be to use composition rather than inheritance (i.e.
>>>
>>> public class GiraphJob {
>>> Job internalJob;
>>> }
>>>
>>> and expose the methods we would like as necessary. There are other
>>> methods we don't want the user to call, (i.e. setMapperClass(), etc.).
>>> David, can you please open an issue for this?
>>>
>>> Avery
>>>
>>> On 2/8/12 5:17 PM, David Garcia wrote:
>>>> This is a very subtle bug. GiraphJob inherits from
>>>> org.apache.mapreduce.Job. However, the methods submit() and
>>>> waitForCompletion() are not overridden. I assumed that they were
>>>> implemented, so when I called either one of these methods, the
>>>> framework
>>>> started up identity mappers/reducers. A simple fix is to throw
>>>> unsupported operation exceptions or to implement these methods.
>>>> Perhaps
>>>> this has been done already?
>>>>
>>>> -David
>>>>
>>>> On 2/7/12 7:46 PM, "David Garcia"<dg...@potomacfusion.com> wrote:
>>>>
>>>>> I am running into a weird error that I haven't seen yet (I suppose
>>>>> I've
>>>>> been lucky). I see the following in the logging:
>>>>>
>>>>> org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop
>>>>> library for your platform... using builtin-java classes where
>>>>> applicable
>>>>>
>>>>>
>>>>> In the job definition, the property "mapreduce.map.class" is not even
>>>>> defined. For Giraph, this is usually set to
>>>>> "mapreduce.map.class=org.apache.giraph.graph.GraphMapper"
>>>>>
>>>>> I'm building my project with hadoop 0.20.204.
>>>>>
>>>>> When I build the GiraphProject myself (and run my own tests with the
>>>>> projects dependencies), I have no problems. The main difference is
>>>>> that
>>>>> I'm using a Giraph dependency in my work project. All input is
>>>>> welcome.
>>>>> Thx!!
>>>>>
>>>>> -David
>>>>>
Re: Giraph Architecture bug in
Posted by David Garcia <dg...@potomacfusion.com>.
I opened up
* GIRAPH-144 <https://issues.apache.org/jira/browse/GIRAPH-144>
I apologize if I didn't do it up according to project SOP's. I haven't
had time to read it thoroughly.
-David
On 2/8/12 7:29 PM, "David Garcia" <dg...@potomacfusion.com> wrote:
>Yeah, I'll write something up.
>
>
>On 2/8/12 7:26 PM, "Avery Ching" <ac...@apache.org> wrote:
>
>>Since we call waitForCompletion() (which calls submit() internally) in
>>GiraphJob#run(), we cannot override those methods. A better fix would
>>probably be to use composition rather than inheritance (i.e.
>>
>>public class GiraphJob {
>> Job internalJob;
>>}
>>
>>and expose the methods we would like as necessary. There are other
>>methods we don't want the user to call, (i.e. setMapperClass(), etc.).
>>David, can you please open an issue for this?
>>
>>Avery
>>
>>On 2/8/12 5:17 PM, David Garcia wrote:
>>> This is a very subtle bug. GiraphJob inherits from
>>> org.apache.mapreduce.Job. However, the methods submit() and
>>> waitForCompletion() are not overridden. I assumed that they were
>>> implemented, so when I called either one of these methods, the
>>>framework
>>> started up identity mappers/reducers. A simple fix is to throw
>>> unsupported operation exceptions or to implement these methods.
>>>Perhaps
>>> this has been done already?
>>>
>>> -David
>>>
>>> On 2/7/12 7:46 PM, "David Garcia"<dg...@potomacfusion.com> wrote:
>>>
>>>> I am running into a weird error that I haven't seen yet (I suppose
>>>>I've
>>>> been lucky). I see the following in the logging:
>>>>
>>>> org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop
>>>> library for your platform... using builtin-java classes where
>>>>applicable
>>>>
>>>>
>>>> In the job definition, the property "mapreduce.map.class" is not even
>>>> defined. For Giraph, this is usually set to
>>>> "mapreduce.map.class=org.apache.giraph.graph.GraphMapper"
>>>>
>>>> I'm building my project with hadoop 0.20.204.
>>>>
>>>> When I build the GiraphProject myself (and run my own tests with the
>>>> projects dependencies), I have no problems. The main difference is
>>>>that
>>>> I'm using a Giraph dependency in my work project. All input is
>>>>welcome.
>>>> Thx!!
>>>>
>>>> -David
>>>>
>>
>
Re: Giraph Architecture bug in
Posted by David Garcia <dg...@potomacfusion.com>.
Yeah, I'll write something up.
On 2/8/12 7:26 PM, "Avery Ching" <ac...@apache.org> wrote:
>Since we call waitForCompletion() (which calls submit() internally) in
>GiraphJob#run(), we cannot override those methods. A better fix would
>probably be to use composition rather than inheritance (i.e.
>
>public class GiraphJob {
> Job internalJob;
>}
>
>and expose the methods we would like as necessary. There are other
>methods we don't want the user to call, (i.e. setMapperClass(), etc.).
>David, can you please open an issue for this?
>
>Avery
>
>On 2/8/12 5:17 PM, David Garcia wrote:
>> This is a very subtle bug. GiraphJob inherits from
>> org.apache.mapreduce.Job. However, the methods submit() and
>> waitForCompletion() are not overridden. I assumed that they were
>> implemented, so when I called either one of these methods, the framework
>> started up identity mappers/reducers. A simple fix is to throw
>> unsupported operation exceptions or to implement these methods. Perhaps
>> this has been done already?
>>
>> -David
>>
>> On 2/7/12 7:46 PM, "David Garcia"<dg...@potomacfusion.com> wrote:
>>
>>> I am running into a weird error that I haven't seen yet (I suppose I've
>>> been lucky). I see the following in the logging:
>>>
>>> org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop
>>> library for your platform... using builtin-java classes where
>>>applicable
>>>
>>>
>>> In the job definition, the property "mapreduce.map.class" is not even
>>> defined. For Giraph, this is usually set to
>>> "mapreduce.map.class=org.apache.giraph.graph.GraphMapper"
>>>
>>> I'm building my project with hadoop 0.20.204.
>>>
>>> When I build the GiraphProject myself (and run my own tests with the
>>> projects dependencies), I have no problems. The main difference is
>>>that
>>> I'm using a Giraph dependency in my work project. All input is
>>>welcome.
>>> Thx!!
>>>
>>> -David
>>>
>
Re: Giraph Architecture bug in
Posted by Avery Ching <ac...@apache.org>.
Since we call waitForCompletion() (which calls submit() internally) in
GiraphJob#run(), we cannot override those methods. A better fix would
probably be to use composition rather than inheritance (i.e.
public class GiraphJob {
Job internalJob;
}
and expose the methods we would like as necessary. There are other
methods we don't want the user to call, (i.e. setMapperClass(), etc.).
David, can you please open an issue for this?
Avery
On 2/8/12 5:17 PM, David Garcia wrote:
> This is a very subtle bug. GiraphJob inherits from
> org.apache.mapreduce.Job. However, the methods submit() and
> waitForCompletion() are not overridden. I assumed that they were
> implemented, so when I called either one of these methods, the framework
> started up identity mappers/reducers. A simple fix is to throw
> unsupported operation exceptions or to implement these methods. Perhaps
> this has been done already?
>
> -David
>
> On 2/7/12 7:46 PM, "David Garcia"<dg...@potomacfusion.com> wrote:
>
>> I am running into a weird error that I haven't seen yet (I suppose I've
>> been lucky). I see the following in the logging:
>>
>> org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop
>> library for your platform... using builtin-java classes where applicable
>>
>>
>> In the job definition, the property "mapreduce.map.class" is not even
>> defined. For Giraph, this is usually set to
>> "mapreduce.map.class=org.apache.giraph.graph.GraphMapper"
>>
>> I'm building my project with hadoop 0.20.204.
>>
>> When I build the GiraphProject myself (and run my own tests with the
>> projects dependencies), I have no problems. The main difference is that
>> I'm using a Giraph dependency in my work project. All input is welcome.
>> Thx!!
>>
>> -David
>>