Giraph out-of-core feature is not helping!

[Khaled Ammar <kh...@gmail.com>]

Hi,

I am using Giraph-1.1.0 to do large graph processing. I was trying to do a
hashMin (WCC) algorithm on a large graph but it failed with out of memory
error. I thought the out-of-core option may help, but it did not.

Is there any advice about how to enable out-of-core processing?

I followed this URL: http://giraph.apache.org/ooc.html

-- 
Thanks,
-Khaled

Re: Giraph out-of-core feature is not helping!

[Hassan Eslami <hs...@gmail.com>]

OOC features (both for graph and messages) are under active development at
the moment. The previous version (1.0.0 and 1.1.0) did have several issues
(mostly performance issues and also falling short interacting with
stop-the-world GC pauses while OOC tries to spill data to disk). A new
mechanism for OOC is already pushed into the code base. However, the new
mechanism does not work properly for OOC messages. An entirely new approach
to OOC is under development and soon will be available in the code base.

Best,
Hassan

On Thu, Dec 10, 2015 at 8:55 PM, Khaled Ammar <kh...@gmail.com>
wrote:

> Thank you Roman,
>
> This is interesting. I think I used OOC on the older Giraph version 1.0.0.
> Does the new version introduce new bugs?
>
> I checked the archive, but could not see relevant information. Does any
> one have experience with OOC on 1.1.0 or 1.2 ?!!
>
> Thanks,
> -Khaled
>
>
> On Fri, Dec 4, 2015 at 8:56 PM, Roman Shaposhnik <ro...@shaposhnik.org>
> wrote:
>
>> I remember seeing a discussion that discouraged users from
>> utilizing OOC. I am vague on the details but you can try
>> searching the archives.
>>
>> Thanks,
>> Roman.
>>
>> On Fri, Dec 4, 2015 at 4:34 PM, Khaled Ammar <kh...@gmail.com>
>> wrote:
>> > Hi,
>> >
>> > I am using Giraph-1.1.0 to do large graph processing. I was trying to
>> do a
>> > hashMin (WCC) algorithm on a large graph but it failed with out of
>> memory
>> > error. I thought the out-of-core option may help, but it did not.
>> >
>> > Is there any advice about how to enable out-of-core processing?
>> >
>> > I followed this URL: http://giraph.apache.org/ooc.html
>> >
>> > --
>> > Thanks,
>> > -Khaled
>>
>
>
>
> --
> Thanks,
> -Khaled
>

Re: Giraph out-of-core feature is not helping!

[Khaled Ammar <kh...@gmail.com>]

Thank you Roman,

This is interesting. I think I used OOC on the older Giraph version 1.0.0.
Does the new version introduce new bugs?

I checked the archive, but could not see relevant information. Does any one
have experience with OOC on 1.1.0 or 1.2 ?!!

Thanks,
-Khaled


On Fri, Dec 4, 2015 at 8:56 PM, Roman Shaposhnik <ro...@shaposhnik.org>
wrote:

> I remember seeing a discussion that discouraged users from
> utilizing OOC. I am vague on the details but you can try
> searching the archives.
>
> Thanks,
> Roman.
>
> On Fri, Dec 4, 2015 at 4:34 PM, Khaled Ammar <kh...@gmail.com>
> wrote:
> > Hi,
> >
> > I am using Giraph-1.1.0 to do large graph processing. I was trying to do
> a
> > hashMin (WCC) algorithm on a large graph but it failed with out of memory
> > error. I thought the out-of-core option may help, but it did not.
> >
> > Is there any advice about how to enable out-of-core processing?
> >
> > I followed this URL: http://giraph.apache.org/ooc.html
> >
> > --
> > Thanks,
> > -Khaled
>



-- 
Thanks,
-Khaled

Re: Giraph out-of-core feature is not helping!

[Roman Shaposhnik <ro...@shaposhnik.org>]

I remember seeing a discussion that discouraged users from
utilizing OOC. I am vague on the details but you can try
searching the archives.

Thanks,
Roman.

On Fri, Dec 4, 2015 at 4:34 PM, Khaled Ammar <kh...@gmail.com> wrote:
> Hi,
>
> I am using Giraph-1.1.0 to do large graph processing. I was trying to do a
> hashMin (WCC) algorithm on a large graph but it failed with out of memory
> error. I thought the out-of-core option may help, but it did not.
>
> Is there any advice about how to enable out-of-core processing?
>
> I followed this URL: http://giraph.apache.org/ooc.html
>
> --
> Thanks,
> -Khaled