You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by otherwise777 <wo...@onzichtbaar.net> on 2016/11/28 20:43:51 UTC
Executing graph algorithms on Gelly that are larger then memmory
I read somewhere that Flink and Gelly should be able to handle graph
algorithms that require more space then the available memory, i'm currently
getting java OutOfMemoryError heap space and if it would use disk space that
wouldn't happen.
Currently my algorithms use dense graphs with 10m edges, the algorithms that
i use are on my github [1], i did get the same heap space errors when using
the Gelly algorithms, i also tried playing with setting the parallelism but
it mostly gives the same error
Can anyone help me with some methods on how to solve this so that i can
scale up to bigger graphs?
[1] https://github.com/otherwise777/Temporal_Graph_library/
--
View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Executing-graph-algorithms-on-Gelly-that-are-larger-then-memmory-tp10358.html
Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.
Re: Executing graph algorithms on Gelly that are larger then memmory
Posted by Vasiliki Kalavri <va...@gmail.com>.
Hi,
can you give us some more details about the algorithm you are testing and
your configuration?
Flink DataSet operators like join, coGroup, reduce, etc. spill to disk if
there is not enough memory. If you are using a delta iteration operator
though, the state that is kept across iterations (solution set) must fit in
memory.
-Vasia.
On 28 November 2016 at 21:44, otherwise777 <wo...@onzichtbaar.net> wrote:
> Small addition, i'm currently running the programs via my IDE intelij
>
>
>
> --
> View this message in context: http://apache-flink-user-
> mailing-list-archive.2336050.n4.nabble.com/Executing-graph-
> algorithms-on-Gelly-that-are-larger-then-memmory-tp10358p10359.html
> Sent from the Apache Flink User Mailing List archive. mailing list archive
> at Nabble.com.
>
Re: Executing graph algorithms on Gelly that are larger then
memmory
Posted by otherwise777 <wo...@onzichtbaar.net>.
Small addition, i'm currently running the programs via my IDE intelij
--
View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Executing-graph-algorithms-on-Gelly-that-are-larger-then-memmory-tp10358p10359.html
Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.