You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Russell Jurney <ru...@gmail.com> on 2025/01/11 08:22:36 UTC
GraphFrames' ConnectedComponentSuite test 'two components and two dangling vertices' fails with OutOfMemoryError: Java heap space
Friends of GraphFrames (github.com/graphframes/graphframes), I have a
question for you...
I can't get the unit test 'two components and two dangling vertices' in the
org.graphframes.lib.ConnectedComponentsSuite
<https://github.com/graphframes/graphframes/blob/649094caf58cfda0eea3e8cd66785aa38104d771/src/test/scala/org/graphframes/lib/ConnectedComponentsSuite.scala#L138-L148>
to pass. It fails with an 'OutOfMemoryError: Java heap space' error. I am a
little stuck on completing a docs release with a motif finding tutorial
<https://github.com/graphframes/graphframes/pull/473> due to this issue.
The problem is outlined in this gist:
https://gist.github.com/rjurney/6abeffbd59c67df5e5243c8f6619b6bf
Can someone else please try this and see if it passes on the master branch?
> build/sbt clean compile package test
I've tried giving it lots of RAM just to see if it would help, as much as
32g driver and 16g for executors and... it has no effect. The test graph is 8
nodes and 6 edges
<https://gist.github.com/rjurney/6abeffbd59c67df5e5243c8f6619b6bf#file-connectedcomponentsuite-scala-L22-L26>,
so it shouldn't have a memory problem... yet when it runs, all 24 cores of
my CPU get used, it spikes as indicated in the image in the gist.
I am running the following setup:
* Ubuntu 20.04 (22.04 in the Docker image)
* OpenJDK 11 (I also tried 8, same problem)
* Scala 2.12.20 (I also tried 2.13, same problem)
* Python 3.11 (I also tried 3.9, same problem)
Or I am running the Dockerfile in the gist
<https://gist.github.com/rjurney/6abeffbd59c67df5e5243c8f6619b6bf#file-dockerfile>
.
Any help much appreciated! Thanks
-----------------------------------------------------------------
Oh, some new community stuff for GraphFrames. Hackathon announced next week
:)
- GraphFrames Mailing List <https://groups.google.com/g/graphframes/>:
ask questions about GraphFrames on our Google Group
- #graphframes Discord Channel on GraphGeeks
<https://discord.com/channels/1162999022819225631/1326257052368113674>
Thanks!
Russell Jurney @rjurney <http://twitter.com/rjurney>
russell.jurney@gmail.com LI <http://linkedin.com/in/russelljurney> FB
<http://facebook.com/jurney> datasyndrome.com
Re: GraphFrames' ConnectedComponentSuite test 'two components and two dangling vertices' fails with OutOfMemoryError: Java heap space
Posted by Russell Jurney <ru...@gmail.com>.
Okay, I just sent you some instructions privately and will update the list
tomorrow.
On Thu, Jan 23, 2025 at 11:34 PM Ángel <an...@gmail.com>
wrote:
> Super helpful. Thanks, Russell.
>
> I'm planning to dedicate the entire weekend to this issue, and I hope to
> create some Spark and/or Graphframes PRs as well.
>
> El vie, 24 ene 2025 a las 5:31, Russell Jurney (<ru...@gmail.com>)
> escribió:
>
>> I have the stats.meta.stackexchange.com graph I’m hoping to wrangle into
>> the tests, which is around 100K edges. I should have the tutorial and
>> associated tools read in a week or two. I could send you the node and edge
>> Parquets if it will be helpful?
>>
>> On Tue, Jan 21, 2025 at 6:15 PM Ángel <an...@gmail.com>
>> wrote:
>>
>>> Hi Team,
>>>
>>> I believe I’ve finally managed to fix the OOM issue, though I’m still
>>> analyzing some details and conducting tests to confirm my findings. The
>>> issue seems to be related to how Spark updates and propagates the plan
>>> across the internal bus (AdaptiveSparkPlanExec.onUpdatePlan) but, above
>>> all, to how Spark generates the physical plan when DataFrames are cached
>>> iteratively—such as in GraphFrames. I’ve fixed the OOM issue by adding some
>>> minor improvements to the Spark code, but I’d also like to explore whether
>>> a solution might be achievable by modifying only the GraphFrames code, in
>>> case my improvements to Spark are not accepted.
>>>
>>> Over the next few weeks, I’m planning to write three articles
>>> summarizing everything I’ve discovered. I’ll share the links once they’re
>>> published, and I’d greatly appreciate any feedback you can provide.
>>>
>>> @Bjørn Jørgensen, I was genuinely surprised by the links you shared last
>>> week. Shouldn’t Connected Components be thoroughly tested in the
>>> GraphFrames unit tests? Regardless, while this other issue appears to be
>>> also fixed when AQE is disabled, I believe there isn't a connection. It
>>> seems like the way GraphFrames assigned ids only functioned correctly when
>>> data partitioning was static—that is, in older Spark versions where AQE
>>> didn’t exist or was disabled by default. Do you happen to have a simple
>>> dataset we could use for further testing?
>>>
>>> Regards,
>>> Ángel
>>>
>>> El jue, 16 ene 2025 a las 9:00, Bjørn Jørgensen (<
>>> bjornjorgensen@gmail.com>) escribió:
>>>
>>>> Connectes components produce non correct results
>>>> <https://github.com/graphframes/graphframes/issues/428> and Connected
>>>> Components gives wrong results
>>>> <https://github.com/graphframes/graphframes/issues/453>
>>>>
>>>> tor. 16. jan. 2025 kl. 08:21 skrev Ángel <
>>>> angel.alvarez.pascua@gmail.com>:
>>>>
>>>>> Don't thank me; I'm really learning a lot about Spark and Graphframes
>>>>> internals and having fun digging into this issue. I'm planning to write an
>>>>> article about it next weekend.
>>>>>
>>>>> I'm getting closer and closer...
>>>>>
>>>>> Btw., I've also noticed a couple of issues with Spark when working
>>>>> with large plans. I find it weird that nobody has realized this before. I
>>>>> want to double-check them - and also test them in Spark 4 - , but if I'm
>>>>> correct, I'll open a ticket on the Spark project and submit a PR this week.
>>>>> As just mentioned, I could be wrong, but with the improvements, the test
>>>>> went from taking 3–4 minutes to only 1.5 minutes.
>>>>>
>>>>>
>>>>> El jue, 16 ene 2025 a las 1:15, Russell Jurney (<
>>>>> russell.jurney@gmail.com>) escribió:
>>>>>
>>>>>> Thank you SO much for looking into this! GraphFrames lives!
>>>>>>
>>>>>> As an update, here's the most popular logo from a poll I took:
>>>>>>
>>>>>> [image: image.png]
>>>>>>
>>>>>> Although this one in second place fits more with the Apache logo
>>>>>> directory <https://www.apache.org/logos/>:
>>>>>>
>>>>>> [image: image.png]
>>>>>>
>>>>>> Which one makes you want to work on this issue more? :D
>>>>>>
>>>>>> Russ
>>>>>>
>>>>>> On Wed, Jan 15, 2025 at 5:04 AM Ángel <an...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> My only change was increasing 4 to 10 in the
>>>>>>> spark.sql.shuffle.partitions setting (still doesn't work for a value lower
>>>>>>> than 10).
>>>>>>>
>>>>>>> [image: image.png]
>>>>>>>
>>>>>>> I've performed some tests, changed some code ... and I'm starting
>>>>>>> to grasp what's behind this OOM, but I'm still working on it .. quite
>>>>>>> interesting indeed this issue! (thanks)
>>>>>>>
>>>>>>> According to MAT <https://eclipse.dev/mat/> ... a humongous
>>>>>>> execution plan is being created ...
>>>>>>> [image: image.png]
>>>>>>>
>>>>>>> El mar, 14 ene 2025 a las 21:52, Russell Jurney (<
>>>>>>> russell.jurney@gmail.com>) escribió:
>>>>>>>
>>>>>>>> Can you please share the code? It doesn't seem an ideal solution,
>>>>>>>> but if AQE is confused, disabling it makes sense. I can't figure out why a
>>>>>>>> low partition count for an 8 node, 6 edge network would require a lot of
>>>>>>>> partitions... users may have different numbers... do you suggest we enforce
>>>>>>>> some minimum partition count?
>>>>>>>>
>>>>>>>> On Tue, Jan 14, 2025 at 7:33 AM Ángel <
>>>>>>>> angel.alvarez.pascua@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> Are you sure that temporarily disabling a global setting like AQE
>>>>>>>>> is the best approach to fix this issue?
>>>>>>>>> I increased the number of shuffle partitions in the Spark session
>>>>>>>>> configure in the GraphFrameTestSparkContext.scala from 4 to 10, and the
>>>>>>>>> "checkpoint interval" test ran perfectly without throwing an OOM error.
>>>>>>>>> Why? No idea, but it worked.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> El lun, 13 ene 2025 a las 16:45, Russell Jurney (<
>>>>>>>>> russell.jurney@gmail.com>) escribió:
>>>>>>>>>
>>>>>>>>>> Merged, thanks guys!
>>>>>>>>>>
>>>>>>>>>> Russ
>>>>>>>>>>
>>>>>>>>>> On Sun, Jan 12, 2025 at 2:23 PM Bjørn Jørgensen <
>>>>>>>>>> bjornjorgensen@gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> I tested stuff in this PR
>>>>>>>>>>> https://github.com/graphframes/graphframes/pull/477 and then I
>>>>>>>>>>> made this PR https://github.com/graphframes/graphframes/pull/478
>>>>>>>>>>>
>>>>>>>>>>> søn. 12. jan. 2025 kl. 23:10 skrev Ángel <
>>>>>>>>>>> angel.alvarez.pascua@gmail.com>:
>>>>>>>>>>>
>>>>>>>>>>>> Hi Russell,
>>>>>>>>>>>>
>>>>>>>>>>>> I've just got the OOM error during Test 13. I'm running it from
>>>>>>>>>>>> IntelliJ on Windows with Java 11.
>>>>>>>>>>>>
>>>>>>>>>>>> [image: image.png]
>>>>>>>>>>>> I'll look into it over the course of the next week.
>>>>>>>>>>>>
>>>>>>>>>>>> Regards,
>>>>>>>>>>>> Ángel
>>>>>>>>>>>>
>>>>>>>>>>>> El sáb, 11 ene 2025 a las 9:23, Russell Jurney (<
>>>>>>>>>>>> russell.jurney@gmail.com>) escribió:
>>>>>>>>>>>>
>>>>>>>>>>>>> Friends of GraphFrames (github.com/graphframes/graphframes),
>>>>>>>>>>>>> I have a question for you...
>>>>>>>>>>>>>
>>>>>>>>>>>>> I can't get the unit test 'two components and two dangling
>>>>>>>>>>>>> vertices' in the org.graphframes.lib.ConnectedComponentsSuite
>>>>>>>>>>>>> <https://github.com/graphframes/graphframes/blob/649094caf58cfda0eea3e8cd66785aa38104d771/src/test/scala/org/graphframes/lib/ConnectedComponentsSuite.scala#L138-L148>
>>>>>>>>>>>>> to pass. It fails with an 'OutOfMemoryError: Java heap space' error. I am a
>>>>>>>>>>>>> little stuck on completing a docs release with a motif
>>>>>>>>>>>>> finding tutorial
>>>>>>>>>>>>> <https://github.com/graphframes/graphframes/pull/473> due to
>>>>>>>>>>>>> this issue.
>>>>>>>>>>>>>
>>>>>>>>>>>>> The problem is outlined in this gist:
>>>>>>>>>>>>> https://gist.github.com/rjurney/6abeffbd59c67df5e5243c8f6619b6bf
>>>>>>>>>>>>>
>>>>>>>>>>>>> Can someone else please try this and see if it passes on the
>>>>>>>>>>>>> master branch?
>>>>>>>>>>>>>
>>>>>>>>>>>>> > build/sbt clean compile package test
>>>>>>>>>>>>>
>>>>>>>>>>>>> I've tried giving it lots of RAM just to see if it would help,
>>>>>>>>>>>>> as much as 32g driver and 16g for executors and... it has no effect. The
>>>>>>>>>>>>> test graph is 8 nodes and 6 edges
>>>>>>>>>>>>> <https://gist.github.com/rjurney/6abeffbd59c67df5e5243c8f6619b6bf#file-connectedcomponentsuite-scala-L22-L26>,
>>>>>>>>>>>>> so it shouldn't have a memory problem... yet when it runs, all 24 cores of
>>>>>>>>>>>>> my CPU get used, it spikes as indicated in the image in the gist.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I am running the following setup:
>>>>>>>>>>>>>
>>>>>>>>>>>>> * Ubuntu 20.04 (22.04 in the Docker image)
>>>>>>>>>>>>> * OpenJDK 11 (I also tried 8, same problem)
>>>>>>>>>>>>> * Scala 2.12.20 (I also tried 2.13, same problem)
>>>>>>>>>>>>> * Python 3.11 (I also tried 3.9, same problem)
>>>>>>>>>>>>>
>>>>>>>>>>>>> Or I am running the Dockerfile in the gist
>>>>>>>>>>>>> <https://gist.github.com/rjurney/6abeffbd59c67df5e5243c8f6619b6bf#file-dockerfile>
>>>>>>>>>>>>> .
>>>>>>>>>>>>>
>>>>>>>>>>>>> Any help much appreciated! Thanks
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> -----------------------------------------------------------------
>>>>>>>>>>>>> Oh, some new community stuff for GraphFrames. Hackathon
>>>>>>>>>>>>> announced next week :)
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> - GraphFrames Mailing List
>>>>>>>>>>>>> <https://groups.google.com/g/graphframes/>: ask questions
>>>>>>>>>>>>> about GraphFrames on our Google Group
>>>>>>>>>>>>> - #graphframes Discord Channel on GraphGeeks
>>>>>>>>>>>>> <https://discord.com/channels/1162999022819225631/1326257052368113674>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks!
>>>>>>>>>>>>> Russell Jurney @rjurney <http://twitter.com/rjurney>
>>>>>>>>>>>>> russell.jurney@gmail.com LI
>>>>>>>>>>>>> <http://linkedin.com/in/russelljurney> FB
>>>>>>>>>>>>> <http://facebook.com/jurney> datasyndrome.com
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Bjørn Jørgensen
>>>>>>>>>>> Vestre Aspehaug 4, 6010 Ålesund
>>>>>>>>>>> <https://www.google.com/maps/search/Vestre+Aspehaug+4,+6010+%C3%85lesund++Norge?entry=gmail&source=g>
>>>>>>>>>>> Norge
>>>>>>>>>>> <https://www.google.com/maps/search/Vestre+Aspehaug+4,+6010+%C3%85lesund++Norge?entry=gmail&source=g>
>>>>>>>>>>>
>>>>>>>>>>> +47 480 94 297
>>>>>>>>>>>
>>>>>>>>>>
>>>>
>>>> --
>>>> Bjørn Jørgensen
>>>> Vestre Aspehaug 4, 6010 Ålesund
>>>> <https://www.google.com/maps/search/Vestre+Aspehaug+4,+6010+%C3%85lesund++Norge?entry=gmail&source=g>
>>>> Norge
>>>> <https://www.google.com/maps/search/Vestre+Aspehaug+4,+6010+%C3%85lesund++Norge?entry=gmail&source=g>
>>>>
>>>> +47 480 94 297
>>>>
>>>
Re: GraphFrames' ConnectedComponentSuite test 'two components and two dangling vertices' fails with OutOfMemoryError: Java heap space
Posted by Ángel <an...@gmail.com>.
Super helpful. Thanks, Russell.
I'm planning to dedicate the entire weekend to this issue, and I hope to
create some Spark and/or Graphframes PRs as well.
El vie, 24 ene 2025 a las 5:31, Russell Jurney (<ru...@gmail.com>)
escribió:
> I have the stats.meta.stackexchange.com graph I’m hoping to wrangle into
> the tests, which is around 100K edges. I should have the tutorial and
> associated tools read in a week or two. I could send you the node and edge
> Parquets if it will be helpful?
>
> On Tue, Jan 21, 2025 at 6:15 PM Ángel <an...@gmail.com>
> wrote:
>
>> Hi Team,
>>
>> I believe I’ve finally managed to fix the OOM issue, though I’m still
>> analyzing some details and conducting tests to confirm my findings. The
>> issue seems to be related to how Spark updates and propagates the plan
>> across the internal bus (AdaptiveSparkPlanExec.onUpdatePlan) but, above
>> all, to how Spark generates the physical plan when DataFrames are cached
>> iteratively—such as in GraphFrames. I’ve fixed the OOM issue by adding some
>> minor improvements to the Spark code, but I’d also like to explore whether
>> a solution might be achievable by modifying only the GraphFrames code, in
>> case my improvements to Spark are not accepted.
>>
>> Over the next few weeks, I’m planning to write three articles summarizing
>> everything I’ve discovered. I’ll share the links once they’re published,
>> and I’d greatly appreciate any feedback you can provide.
>>
>> @Bjørn Jørgensen, I was genuinely surprised by the links you shared last
>> week. Shouldn’t Connected Components be thoroughly tested in the
>> GraphFrames unit tests? Regardless, while this other issue appears to be
>> also fixed when AQE is disabled, I believe there isn't a connection. It
>> seems like the way GraphFrames assigned ids only functioned correctly when
>> data partitioning was static—that is, in older Spark versions where AQE
>> didn’t exist or was disabled by default. Do you happen to have a simple
>> dataset we could use for further testing?
>>
>> Regards,
>> Ángel
>>
>> El jue, 16 ene 2025 a las 9:00, Bjørn Jørgensen (<
>> bjornjorgensen@gmail.com>) escribió:
>>
>>> Connectes components produce non correct results
>>> <https://github.com/graphframes/graphframes/issues/428> and Connected
>>> Components gives wrong results
>>> <https://github.com/graphframes/graphframes/issues/453>
>>>
>>> tor. 16. jan. 2025 kl. 08:21 skrev Ángel <angel.alvarez.pascua@gmail.com
>>> >:
>>>
>>>> Don't thank me; I'm really learning a lot about Spark and Graphframes
>>>> internals and having fun digging into this issue. I'm planning to write an
>>>> article about it next weekend.
>>>>
>>>> I'm getting closer and closer...
>>>>
>>>> Btw., I've also noticed a couple of issues with Spark when working with
>>>> large plans. I find it weird that nobody has realized this before. I want
>>>> to double-check them - and also test them in Spark 4 - , but if I'm
>>>> correct, I'll open a ticket on the Spark project and submit a PR this week.
>>>> As just mentioned, I could be wrong, but with the improvements, the test
>>>> went from taking 3–4 minutes to only 1.5 minutes.
>>>>
>>>>
>>>> El jue, 16 ene 2025 a las 1:15, Russell Jurney (<
>>>> russell.jurney@gmail.com>) escribió:
>>>>
>>>>> Thank you SO much for looking into this! GraphFrames lives!
>>>>>
>>>>> As an update, here's the most popular logo from a poll I took:
>>>>>
>>>>> [image: image.png]
>>>>>
>>>>> Although this one in second place fits more with the Apache logo
>>>>> directory <https://www.apache.org/logos/>:
>>>>>
>>>>> [image: image.png]
>>>>>
>>>>> Which one makes you want to work on this issue more? :D
>>>>>
>>>>> Russ
>>>>>
>>>>> On Wed, Jan 15, 2025 at 5:04 AM Ángel <an...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> My only change was increasing 4 to 10 in the
>>>>>> spark.sql.shuffle.partitions setting (still doesn't work for a value lower
>>>>>> than 10).
>>>>>>
>>>>>> [image: image.png]
>>>>>>
>>>>>> I've performed some tests, changed some code ... and I'm starting to
>>>>>> grasp what's behind this OOM, but I'm still working on it .. quite
>>>>>> interesting indeed this issue! (thanks)
>>>>>>
>>>>>> According to MAT <https://eclipse.dev/mat/> ... a humongous
>>>>>> execution plan is being created ...
>>>>>> [image: image.png]
>>>>>>
>>>>>> El mar, 14 ene 2025 a las 21:52, Russell Jurney (<
>>>>>> russell.jurney@gmail.com>) escribió:
>>>>>>
>>>>>>> Can you please share the code? It doesn't seem an ideal solution,
>>>>>>> but if AQE is confused, disabling it makes sense. I can't figure out why a
>>>>>>> low partition count for an 8 node, 6 edge network would require a lot of
>>>>>>> partitions... users may have different numbers... do you suggest we enforce
>>>>>>> some minimum partition count?
>>>>>>>
>>>>>>> On Tue, Jan 14, 2025 at 7:33 AM Ángel <
>>>>>>> angel.alvarez.pascua@gmail.com> wrote:
>>>>>>>
>>>>>>>> Are you sure that temporarily disabling a global setting like AQE
>>>>>>>> is the best approach to fix this issue?
>>>>>>>> I increased the number of shuffle partitions in the Spark session
>>>>>>>> configure in the GraphFrameTestSparkContext.scala from 4 to 10, and the
>>>>>>>> "checkpoint interval" test ran perfectly without throwing an OOM error.
>>>>>>>> Why? No idea, but it worked.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> El lun, 13 ene 2025 a las 16:45, Russell Jurney (<
>>>>>>>> russell.jurney@gmail.com>) escribió:
>>>>>>>>
>>>>>>>>> Merged, thanks guys!
>>>>>>>>>
>>>>>>>>> Russ
>>>>>>>>>
>>>>>>>>> On Sun, Jan 12, 2025 at 2:23 PM Bjørn Jørgensen <
>>>>>>>>> bjornjorgensen@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> I tested stuff in this PR
>>>>>>>>>> https://github.com/graphframes/graphframes/pull/477 and then I
>>>>>>>>>> made this PR https://github.com/graphframes/graphframes/pull/478
>>>>>>>>>>
>>>>>>>>>> søn. 12. jan. 2025 kl. 23:10 skrev Ángel <
>>>>>>>>>> angel.alvarez.pascua@gmail.com>:
>>>>>>>>>>
>>>>>>>>>>> Hi Russell,
>>>>>>>>>>>
>>>>>>>>>>> I've just got the OOM error during Test 13. I'm running it from
>>>>>>>>>>> IntelliJ on Windows with Java 11.
>>>>>>>>>>>
>>>>>>>>>>> [image: image.png]
>>>>>>>>>>> I'll look into it over the course of the next week.
>>>>>>>>>>>
>>>>>>>>>>> Regards,
>>>>>>>>>>> Ángel
>>>>>>>>>>>
>>>>>>>>>>> El sáb, 11 ene 2025 a las 9:23, Russell Jurney (<
>>>>>>>>>>> russell.jurney@gmail.com>) escribió:
>>>>>>>>>>>
>>>>>>>>>>>> Friends of GraphFrames (github.com/graphframes/graphframes), I
>>>>>>>>>>>> have a question for you...
>>>>>>>>>>>>
>>>>>>>>>>>> I can't get the unit test 'two components and two dangling
>>>>>>>>>>>> vertices' in the org.graphframes.lib.ConnectedComponentsSuite
>>>>>>>>>>>> <https://github.com/graphframes/graphframes/blob/649094caf58cfda0eea3e8cd66785aa38104d771/src/test/scala/org/graphframes/lib/ConnectedComponentsSuite.scala#L138-L148>
>>>>>>>>>>>> to pass. It fails with an 'OutOfMemoryError: Java heap space' error. I am a
>>>>>>>>>>>> little stuck on completing a docs release with a motif finding
>>>>>>>>>>>> tutorial <https://github.com/graphframes/graphframes/pull/473>
>>>>>>>>>>>> due to this issue.
>>>>>>>>>>>>
>>>>>>>>>>>> The problem is outlined in this gist:
>>>>>>>>>>>> https://gist.github.com/rjurney/6abeffbd59c67df5e5243c8f6619b6bf
>>>>>>>>>>>>
>>>>>>>>>>>> Can someone else please try this and see if it passes on the
>>>>>>>>>>>> master branch?
>>>>>>>>>>>>
>>>>>>>>>>>> > build/sbt clean compile package test
>>>>>>>>>>>>
>>>>>>>>>>>> I've tried giving it lots of RAM just to see if it would help,
>>>>>>>>>>>> as much as 32g driver and 16g for executors and... it has no effect. The
>>>>>>>>>>>> test graph is 8 nodes and 6 edges
>>>>>>>>>>>> <https://gist.github.com/rjurney/6abeffbd59c67df5e5243c8f6619b6bf#file-connectedcomponentsuite-scala-L22-L26>,
>>>>>>>>>>>> so it shouldn't have a memory problem... yet when it runs, all 24 cores of
>>>>>>>>>>>> my CPU get used, it spikes as indicated in the image in the gist.
>>>>>>>>>>>>
>>>>>>>>>>>> I am running the following setup:
>>>>>>>>>>>>
>>>>>>>>>>>> * Ubuntu 20.04 (22.04 in the Docker image)
>>>>>>>>>>>> * OpenJDK 11 (I also tried 8, same problem)
>>>>>>>>>>>> * Scala 2.12.20 (I also tried 2.13, same problem)
>>>>>>>>>>>> * Python 3.11 (I also tried 3.9, same problem)
>>>>>>>>>>>>
>>>>>>>>>>>> Or I am running the Dockerfile in the gist
>>>>>>>>>>>> <https://gist.github.com/rjurney/6abeffbd59c67df5e5243c8f6619b6bf#file-dockerfile>
>>>>>>>>>>>> .
>>>>>>>>>>>>
>>>>>>>>>>>> Any help much appreciated! Thanks
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> -----------------------------------------------------------------
>>>>>>>>>>>> Oh, some new community stuff for GraphFrames. Hackathon
>>>>>>>>>>>> announced next week :)
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> - GraphFrames Mailing List
>>>>>>>>>>>> <https://groups.google.com/g/graphframes/>: ask questions
>>>>>>>>>>>> about GraphFrames on our Google Group
>>>>>>>>>>>> - #graphframes Discord Channel on GraphGeeks
>>>>>>>>>>>> <https://discord.com/channels/1162999022819225631/1326257052368113674>
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks!
>>>>>>>>>>>> Russell Jurney @rjurney <http://twitter.com/rjurney>
>>>>>>>>>>>> russell.jurney@gmail.com LI
>>>>>>>>>>>> <http://linkedin.com/in/russelljurney> FB
>>>>>>>>>>>> <http://facebook.com/jurney> datasyndrome.com
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Bjørn Jørgensen
>>>>>>>>>> Vestre Aspehaug 4, 6010 Ålesund
>>>>>>>>>> <https://www.google.com/maps/search/Vestre+Aspehaug+4,+6010+%C3%85lesund++Norge?entry=gmail&source=g>
>>>>>>>>>> Norge
>>>>>>>>>> <https://www.google.com/maps/search/Vestre+Aspehaug+4,+6010+%C3%85lesund++Norge?entry=gmail&source=g>
>>>>>>>>>>
>>>>>>>>>> +47 480 94 297
>>>>>>>>>>
>>>>>>>>>
>>>
>>> --
>>> Bjørn Jørgensen
>>> Vestre Aspehaug 4, 6010 Ålesund
>>> <https://www.google.com/maps/search/Vestre+Aspehaug+4,+6010+%C3%85lesund++Norge?entry=gmail&source=g>
>>> Norge
>>> <https://www.google.com/maps/search/Vestre+Aspehaug+4,+6010+%C3%85lesund++Norge?entry=gmail&source=g>
>>>
>>> +47 480 94 297
>>>
>>
Re: GraphFrames' ConnectedComponentSuite test 'two components and two dangling vertices' fails with OutOfMemoryError: Java heap space
Posted by Russell Jurney <ru...@gmail.com>.
I have the stats.meta.stackexchange.com graph I’m hoping to wrangle into
the tests, which is around 100K edges. I should have the tutorial and
associated tools read in a week or two. I could send you the node and edge
Parquets if it will be helpful?
On Tue, Jan 21, 2025 at 6:15 PM Ángel <an...@gmail.com>
wrote:
> Hi Team,
>
> I believe I’ve finally managed to fix the OOM issue, though I’m still
> analyzing some details and conducting tests to confirm my findings. The
> issue seems to be related to how Spark updates and propagates the plan
> across the internal bus (AdaptiveSparkPlanExec.onUpdatePlan) but, above
> all, to how Spark generates the physical plan when DataFrames are cached
> iteratively—such as in GraphFrames. I’ve fixed the OOM issue by adding some
> minor improvements to the Spark code, but I’d also like to explore whether
> a solution might be achievable by modifying only the GraphFrames code, in
> case my improvements to Spark are not accepted.
>
> Over the next few weeks, I’m planning to write three articles summarizing
> everything I’ve discovered. I’ll share the links once they’re published,
> and I’d greatly appreciate any feedback you can provide.
>
> @Bjørn Jørgensen, I was genuinely surprised by the links you shared last
> week. Shouldn’t Connected Components be thoroughly tested in the
> GraphFrames unit tests? Regardless, while this other issue appears to be
> also fixed when AQE is disabled, I believe there isn't a connection. It
> seems like the way GraphFrames assigned ids only functioned correctly when
> data partitioning was static—that is, in older Spark versions where AQE
> didn’t exist or was disabled by default. Do you happen to have a simple
> dataset we could use for further testing?
>
> Regards,
> Ángel
>
> El jue, 16 ene 2025 a las 9:00, Bjørn Jørgensen (<bj...@gmail.com>)
> escribió:
>
>> Connectes components produce non correct results
>> <https://github.com/graphframes/graphframes/issues/428> and Connected
>> Components gives wrong results
>> <https://github.com/graphframes/graphframes/issues/453>
>>
>> tor. 16. jan. 2025 kl. 08:21 skrev Ángel <angel.alvarez.pascua@gmail.com
>> >:
>>
>>> Don't thank me; I'm really learning a lot about Spark and Graphframes
>>> internals and having fun digging into this issue. I'm planning to write an
>>> article about it next weekend.
>>>
>>> I'm getting closer and closer...
>>>
>>> Btw., I've also noticed a couple of issues with Spark when working with
>>> large plans. I find it weird that nobody has realized this before. I want
>>> to double-check them - and also test them in Spark 4 - , but if I'm
>>> correct, I'll open a ticket on the Spark project and submit a PR this week.
>>> As just mentioned, I could be wrong, but with the improvements, the test
>>> went from taking 3–4 minutes to only 1.5 minutes.
>>>
>>>
>>> El jue, 16 ene 2025 a las 1:15, Russell Jurney (<
>>> russell.jurney@gmail.com>) escribió:
>>>
>>>> Thank you SO much for looking into this! GraphFrames lives!
>>>>
>>>> As an update, here's the most popular logo from a poll I took:
>>>>
>>>> [image: image.png]
>>>>
>>>> Although this one in second place fits more with the Apache logo
>>>> directory <https://www.apache.org/logos/>:
>>>>
>>>> [image: image.png]
>>>>
>>>> Which one makes you want to work on this issue more? :D
>>>>
>>>> Russ
>>>>
>>>> On Wed, Jan 15, 2025 at 5:04 AM Ángel <an...@gmail.com>
>>>> wrote:
>>>>
>>>>> My only change was increasing 4 to 10 in the
>>>>> spark.sql.shuffle.partitions setting (still doesn't work for a value lower
>>>>> than 10).
>>>>>
>>>>> [image: image.png]
>>>>>
>>>>> I've performed some tests, changed some code ... and I'm starting to
>>>>> grasp what's behind this OOM, but I'm still working on it .. quite
>>>>> interesting indeed this issue! (thanks)
>>>>>
>>>>> According to MAT <https://eclipse.dev/mat/> ... a humongous execution
>>>>> plan is being created ...
>>>>> [image: image.png]
>>>>>
>>>>> El mar, 14 ene 2025 a las 21:52, Russell Jurney (<
>>>>> russell.jurney@gmail.com>) escribió:
>>>>>
>>>>>> Can you please share the code? It doesn't seem an ideal solution, but
>>>>>> if AQE is confused, disabling it makes sense. I can't figure out why a low
>>>>>> partition count for an 8 node, 6 edge network would require a lot of
>>>>>> partitions... users may have different numbers... do you suggest we enforce
>>>>>> some minimum partition count?
>>>>>>
>>>>>> On Tue, Jan 14, 2025 at 7:33 AM Ángel <an...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Are you sure that temporarily disabling a global setting like AQE is
>>>>>>> the best approach to fix this issue?
>>>>>>> I increased the number of shuffle partitions in the Spark session
>>>>>>> configure in the GraphFrameTestSparkContext.scala from 4 to 10, and the
>>>>>>> "checkpoint interval" test ran perfectly without throwing an OOM error.
>>>>>>> Why? No idea, but it worked.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> El lun, 13 ene 2025 a las 16:45, Russell Jurney (<
>>>>>>> russell.jurney@gmail.com>) escribió:
>>>>>>>
>>>>>>>> Merged, thanks guys!
>>>>>>>>
>>>>>>>> Russ
>>>>>>>>
>>>>>>>> On Sun, Jan 12, 2025 at 2:23 PM Bjørn Jørgensen <
>>>>>>>> bjornjorgensen@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> I tested stuff in this PR
>>>>>>>>> https://github.com/graphframes/graphframes/pull/477 and then I
>>>>>>>>> made this PR https://github.com/graphframes/graphframes/pull/478
>>>>>>>>>
>>>>>>>>> søn. 12. jan. 2025 kl. 23:10 skrev Ángel <
>>>>>>>>> angel.alvarez.pascua@gmail.com>:
>>>>>>>>>
>>>>>>>>>> Hi Russell,
>>>>>>>>>>
>>>>>>>>>> I've just got the OOM error during Test 13. I'm running it from
>>>>>>>>>> IntelliJ on Windows with Java 11.
>>>>>>>>>>
>>>>>>>>>> [image: image.png]
>>>>>>>>>> I'll look into it over the course of the next week.
>>>>>>>>>>
>>>>>>>>>> Regards,
>>>>>>>>>> Ángel
>>>>>>>>>>
>>>>>>>>>> El sáb, 11 ene 2025 a las 9:23, Russell Jurney (<
>>>>>>>>>> russell.jurney@gmail.com>) escribió:
>>>>>>>>>>
>>>>>>>>>>> Friends of GraphFrames (github.com/graphframes/graphframes), I
>>>>>>>>>>> have a question for you...
>>>>>>>>>>>
>>>>>>>>>>> I can't get the unit test 'two components and two dangling
>>>>>>>>>>> vertices' in the org.graphframes.lib.ConnectedComponentsSuite
>>>>>>>>>>> <https://github.com/graphframes/graphframes/blob/649094caf58cfda0eea3e8cd66785aa38104d771/src/test/scala/org/graphframes/lib/ConnectedComponentsSuite.scala#L138-L148>
>>>>>>>>>>> to pass. It fails with an 'OutOfMemoryError: Java heap space' error. I am a
>>>>>>>>>>> little stuck on completing a docs release with a motif finding
>>>>>>>>>>> tutorial <https://github.com/graphframes/graphframes/pull/473>
>>>>>>>>>>> due to this issue.
>>>>>>>>>>>
>>>>>>>>>>> The problem is outlined in this gist:
>>>>>>>>>>> https://gist.github.com/rjurney/6abeffbd59c67df5e5243c8f6619b6bf
>>>>>>>>>>>
>>>>>>>>>>> Can someone else please try this and see if it passes on the
>>>>>>>>>>> master branch?
>>>>>>>>>>>
>>>>>>>>>>> > build/sbt clean compile package test
>>>>>>>>>>>
>>>>>>>>>>> I've tried giving it lots of RAM just to see if it would help,
>>>>>>>>>>> as much as 32g driver and 16g for executors and... it has no effect. The
>>>>>>>>>>> test graph is 8 nodes and 6 edges
>>>>>>>>>>> <https://gist.github.com/rjurney/6abeffbd59c67df5e5243c8f6619b6bf#file-connectedcomponentsuite-scala-L22-L26>,
>>>>>>>>>>> so it shouldn't have a memory problem... yet when it runs, all 24 cores of
>>>>>>>>>>> my CPU get used, it spikes as indicated in the image in the gist.
>>>>>>>>>>>
>>>>>>>>>>> I am running the following setup:
>>>>>>>>>>>
>>>>>>>>>>> * Ubuntu 20.04 (22.04 in the Docker image)
>>>>>>>>>>> * OpenJDK 11 (I also tried 8, same problem)
>>>>>>>>>>> * Scala 2.12.20 (I also tried 2.13, same problem)
>>>>>>>>>>> * Python 3.11 (I also tried 3.9, same problem)
>>>>>>>>>>>
>>>>>>>>>>> Or I am running the Dockerfile in the gist
>>>>>>>>>>> <https://gist.github.com/rjurney/6abeffbd59c67df5e5243c8f6619b6bf#file-dockerfile>
>>>>>>>>>>> .
>>>>>>>>>>>
>>>>>>>>>>> Any help much appreciated! Thanks
>>>>>>>>>>>
>>>>>>>>>>> -----------------------------------------------------------------
>>>>>>>>>>> Oh, some new community stuff for GraphFrames. Hackathon
>>>>>>>>>>> announced next week :)
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> - GraphFrames Mailing List
>>>>>>>>>>> <https://groups.google.com/g/graphframes/>: ask questions
>>>>>>>>>>> about GraphFrames on our Google Group
>>>>>>>>>>> - #graphframes Discord Channel on GraphGeeks
>>>>>>>>>>> <https://discord.com/channels/1162999022819225631/1326257052368113674>
>>>>>>>>>>>
>>>>>>>>>>> Thanks!
>>>>>>>>>>> Russell Jurney @rjurney <http://twitter.com/rjurney>
>>>>>>>>>>> russell.jurney@gmail.com LI
>>>>>>>>>>> <http://linkedin.com/in/russelljurney> FB
>>>>>>>>>>> <http://facebook.com/jurney> datasyndrome.com
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Bjørn Jørgensen
>>>>>>>>> Vestre Aspehaug 4, 6010 Ålesund
>>>>>>>>> <https://www.google.com/maps/search/Vestre+Aspehaug+4,+6010+%C3%85lesund++Norge?entry=gmail&source=g>
>>>>>>>>> Norge
>>>>>>>>> <https://www.google.com/maps/search/Vestre+Aspehaug+4,+6010+%C3%85lesund++Norge?entry=gmail&source=g>
>>>>>>>>>
>>>>>>>>> +47 480 94 297
>>>>>>>>>
>>>>>>>>
>>
>> --
>> Bjørn Jørgensen
>> Vestre Aspehaug 4, 6010 Ålesund
>> <https://www.google.com/maps/search/Vestre+Aspehaug+4,+6010+%C3%85lesund++Norge?entry=gmail&source=g>
>> Norge
>> <https://www.google.com/maps/search/Vestre+Aspehaug+4,+6010+%C3%85lesund++Norge?entry=gmail&source=g>
>>
>> +47 480 94 297
>>
>
Re: GraphFrames' ConnectedComponentSuite test 'two components and two dangling vertices' fails with OutOfMemoryError: Java heap space
Posted by Ángel <an...@gmail.com>.
The second part of the mini-series ... link
<https://medium.com/towards-data-engineering/apache-spark-wtf-i-like-it-when-a-plan-comes-together-part-ii-dc59def302b3>
El sáb, 1 feb 2025 a las 18:20, Russell Jurney (<rj...@graphlet.ai>)
escribió:
> Very cool, thanks for sharing!
>
> Thanks,
>
> Russell Jurney | rjurney@graphlet.ai | graphlet.ai | Graphlet AI Blog
> <https://blog.graphlet.ai/> | LinkedIn
> <https://linkedin.com/in/russelljurney> | BlueSky
> <https://bsky.app/profile/rjurney.bsky.social>
>
>
> On Sat, Feb 1, 2025 at 7:54 AM Ángel <an...@gmail.com>
> wrote:
>
>> My first article about this issue:
>> https://medium.com/@angel.alvarez.pascua/apache-spark-wtf-i-like-it-when-a-plan-comes-together-part-i-48c52a667288
>>
>> El mié, 22 ene 2025, 3:13, Ángel <an...@gmail.com>
>> escribió:
>>
>>> Hi Team,
>>>
>>> I believe I’ve finally managed to fix the OOM issue, though I’m still
>>> analyzing some details and conducting tests to confirm my findings. The
>>> issue seems to be related to how Spark updates and propagates the plan
>>> across the internal bus (AdaptiveSparkPlanExec.onUpdatePlan) but, above
>>> all, to how Spark generates the physical plan when DataFrames are cached
>>> iteratively—such as in GraphFrames. I’ve fixed the OOM issue by adding some
>>> minor improvements to the Spark code, but I’d also like to explore whether
>>> a solution might be achievable by modifying only the GraphFrames code, in
>>> case my improvements to Spark are not accepted.
>>>
>>> Over the next few weeks, I’m planning to write three articles
>>> summarizing everything I’ve discovered. I’ll share the links once they’re
>>> published, and I’d greatly appreciate any feedback you can provide.
>>>
>>> @Bjørn Jørgensen, I was genuinely surprised by the links you shared last
>>> week. Shouldn’t Connected Components be thoroughly tested in the
>>> GraphFrames unit tests? Regardless, while this other issue appears to be
>>> also fixed when AQE is disabled, I believe there isn't a connection. It
>>> seems like the way GraphFrames assigned ids only functioned correctly when
>>> data partitioning was static—that is, in older Spark versions where AQE
>>> didn’t exist or was disabled by default. Do you happen to have a simple
>>> dataset we could use for further testing?
>>>
>>> Regards,
>>> Ángel
>>>
>>> El jue, 16 ene 2025 a las 9:00, Bjørn Jørgensen (<
>>> bjornjorgensen@gmail.com>) escribió:
>>>
>>>> Connectes components produce non correct results
>>>> <https://github.com/graphframes/graphframes/issues/428> and Connected
>>>> Components gives wrong results
>>>> <https://github.com/graphframes/graphframes/issues/453>
>>>>
>>>> tor. 16. jan. 2025 kl. 08:21 skrev Ángel <
>>>> angel.alvarez.pascua@gmail.com>:
>>>>
>>>>> Don't thank me; I'm really learning a lot about Spark and Graphframes
>>>>> internals and having fun digging into this issue. I'm planning to write an
>>>>> article about it next weekend.
>>>>>
>>>>> I'm getting closer and closer...
>>>>>
>>>>> Btw., I've also noticed a couple of issues with Spark when working
>>>>> with large plans. I find it weird that nobody has realized this before. I
>>>>> want to double-check them - and also test them in Spark 4 - , but if I'm
>>>>> correct, I'll open a ticket on the Spark project and submit a PR this week.
>>>>> As just mentioned, I could be wrong, but with the improvements, the test
>>>>> went from taking 3–4 minutes to only 1.5 minutes.
>>>>>
>>>>>
>>>>> El jue, 16 ene 2025 a las 1:15, Russell Jurney (<
>>>>> russell.jurney@gmail.com>) escribió:
>>>>>
>>>>>> Thank you SO much for looking into this! GraphFrames lives!
>>>>>>
>>>>>> As an update, here's the most popular logo from a poll I took:
>>>>>>
>>>>>> [image: image.png]
>>>>>>
>>>>>> Although this one in second place fits more with the Apache logo
>>>>>> directory <https://www.apache.org/logos/>:
>>>>>>
>>>>>> [image: image.png]
>>>>>>
>>>>>> Which one makes you want to work on this issue more? :D
>>>>>>
>>>>>> Russ
>>>>>>
>>>>>> On Wed, Jan 15, 2025 at 5:04 AM Ángel <an...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> My only change was increasing 4 to 10 in the
>>>>>>> spark.sql.shuffle.partitions setting (still doesn't work for a value lower
>>>>>>> than 10).
>>>>>>>
>>>>>>> [image: image.png]
>>>>>>>
>>>>>>> I've performed some tests, changed some code ... and I'm starting
>>>>>>> to grasp what's behind this OOM, but I'm still working on it .. quite
>>>>>>> interesting indeed this issue! (thanks)
>>>>>>>
>>>>>>> According to MAT <https://eclipse.dev/mat/> ... a humongous
>>>>>>> execution plan is being created ...
>>>>>>> [image: image.png]
>>>>>>>
>>>>>>> El mar, 14 ene 2025 a las 21:52, Russell Jurney (<
>>>>>>> russell.jurney@gmail.com>) escribió:
>>>>>>>
>>>>>>>> Can you please share the code? It doesn't seem an ideal solution,
>>>>>>>> but if AQE is confused, disabling it makes sense. I can't figure out why a
>>>>>>>> low partition count for an 8 node, 6 edge network would require a lot of
>>>>>>>> partitions... users may have different numbers... do you suggest we enforce
>>>>>>>> some minimum partition count?
>>>>>>>>
>>>>>>>> On Tue, Jan 14, 2025 at 7:33 AM Ángel <
>>>>>>>> angel.alvarez.pascua@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> Are you sure that temporarily disabling a global setting like AQE
>>>>>>>>> is the best approach to fix this issue?
>>>>>>>>> I increased the number of shuffle partitions in the Spark session
>>>>>>>>> configure in the GraphFrameTestSparkContext.scala from 4 to 10, and the
>>>>>>>>> "checkpoint interval" test ran perfectly without throwing an OOM error.
>>>>>>>>> Why? No idea, but it worked.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> El lun, 13 ene 2025 a las 16:45, Russell Jurney (<
>>>>>>>>> russell.jurney@gmail.com>) escribió:
>>>>>>>>>
>>>>>>>>>> Merged, thanks guys!
>>>>>>>>>>
>>>>>>>>>> Russ
>>>>>>>>>>
>>>>>>>>>> On Sun, Jan 12, 2025 at 2:23 PM Bjørn Jørgensen <
>>>>>>>>>> bjornjorgensen@gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> I tested stuff in this PR
>>>>>>>>>>> https://github.com/graphframes/graphframes/pull/477 and then I
>>>>>>>>>>> made this PR https://github.com/graphframes/graphframes/pull/478
>>>>>>>>>>>
>>>>>>>>>>> søn. 12. jan. 2025 kl. 23:10 skrev Ángel <
>>>>>>>>>>> angel.alvarez.pascua@gmail.com>:
>>>>>>>>>>>
>>>>>>>>>>>> Hi Russell,
>>>>>>>>>>>>
>>>>>>>>>>>> I've just got the OOM error during Test 13. I'm running it from
>>>>>>>>>>>> IntelliJ on Windows with Java 11.
>>>>>>>>>>>>
>>>>>>>>>>>> [image: image.png]
>>>>>>>>>>>> I'll look into it over the course of the next week.
>>>>>>>>>>>>
>>>>>>>>>>>> Regards,
>>>>>>>>>>>> Ángel
>>>>>>>>>>>>
>>>>>>>>>>>> El sáb, 11 ene 2025 a las 9:23, Russell Jurney (<
>>>>>>>>>>>> russell.jurney@gmail.com>) escribió:
>>>>>>>>>>>>
>>>>>>>>>>>>> Friends of GraphFrames (github.com/graphframes/graphframes),
>>>>>>>>>>>>> I have a question for you...
>>>>>>>>>>>>>
>>>>>>>>>>>>> I can't get the unit test 'two components and two dangling
>>>>>>>>>>>>> vertices' in the org.graphframes.lib.ConnectedComponentsSuite
>>>>>>>>>>>>> <https://github.com/graphframes/graphframes/blob/649094caf58cfda0eea3e8cd66785aa38104d771/src/test/scala/org/graphframes/lib/ConnectedComponentsSuite.scala#L138-L148>
>>>>>>>>>>>>> to pass. It fails with an 'OutOfMemoryError: Java heap space' error. I am a
>>>>>>>>>>>>> little stuck on completing a docs release with a motif
>>>>>>>>>>>>> finding tutorial
>>>>>>>>>>>>> <https://github.com/graphframes/graphframes/pull/473> due to
>>>>>>>>>>>>> this issue.
>>>>>>>>>>>>>
>>>>>>>>>>>>> The problem is outlined in this gist:
>>>>>>>>>>>>> https://gist.github.com/rjurney/6abeffbd59c67df5e5243c8f6619b6bf
>>>>>>>>>>>>>
>>>>>>>>>>>>> Can someone else please try this and see if it passes on the
>>>>>>>>>>>>> master branch?
>>>>>>>>>>>>>
>>>>>>>>>>>>> > build/sbt clean compile package test
>>>>>>>>>>>>>
>>>>>>>>>>>>> I've tried giving it lots of RAM just to see if it would help,
>>>>>>>>>>>>> as much as 32g driver and 16g for executors and... it has no effect. The
>>>>>>>>>>>>> test graph is 8 nodes and 6 edges
>>>>>>>>>>>>> <https://gist.github.com/rjurney/6abeffbd59c67df5e5243c8f6619b6bf#file-connectedcomponentsuite-scala-L22-L26>,
>>>>>>>>>>>>> so it shouldn't have a memory problem... yet when it runs, all 24 cores of
>>>>>>>>>>>>> my CPU get used, it spikes as indicated in the image in the gist.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I am running the following setup:
>>>>>>>>>>>>>
>>>>>>>>>>>>> * Ubuntu 20.04 (22.04 in the Docker image)
>>>>>>>>>>>>> * OpenJDK 11 (I also tried 8, same problem)
>>>>>>>>>>>>> * Scala 2.12.20 (I also tried 2.13, same problem)
>>>>>>>>>>>>> * Python 3.11 (I also tried 3.9, same problem)
>>>>>>>>>>>>>
>>>>>>>>>>>>> Or I am running the Dockerfile in the gist
>>>>>>>>>>>>> <https://gist.github.com/rjurney/6abeffbd59c67df5e5243c8f6619b6bf#file-dockerfile>
>>>>>>>>>>>>> .
>>>>>>>>>>>>>
>>>>>>>>>>>>> Any help much appreciated! Thanks
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> -----------------------------------------------------------------
>>>>>>>>>>>>> Oh, some new community stuff for GraphFrames. Hackathon
>>>>>>>>>>>>> announced next week :)
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> - GraphFrames Mailing List
>>>>>>>>>>>>> <https://groups.google.com/g/graphframes/>: ask questions
>>>>>>>>>>>>> about GraphFrames on our Google Group
>>>>>>>>>>>>> - #graphframes Discord Channel on GraphGeeks
>>>>>>>>>>>>> <https://discord.com/channels/1162999022819225631/1326257052368113674>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks!
>>>>>>>>>>>>> Russell Jurney @rjurney <http://twitter.com/rjurney>
>>>>>>>>>>>>> russell.jurney@gmail.com LI
>>>>>>>>>>>>> <http://linkedin.com/in/russelljurney> FB
>>>>>>>>>>>>> <http://facebook.com/jurney> datasyndrome.com
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Bjørn Jørgensen
>>>>>>>>>>> Vestre Aspehaug 4, 6010 Ålesund
>>>>>>>>>>> <https://www.google.com/maps/search/Vestre+Aspehaug+4,+6010+%C3%85lesund++Norge?entry=gmail&source=g>
>>>>>>>>>>> Norge
>>>>>>>>>>> <https://www.google.com/maps/search/Vestre+Aspehaug+4,+6010+%C3%85lesund++Norge?entry=gmail&source=g>
>>>>>>>>>>>
>>>>>>>>>>> +47 480 94 297
>>>>>>>>>>>
>>>>>>>>>>
>>>>
>>>> --
>>>> Bjørn Jørgensen
>>>> Vestre Aspehaug 4, 6010 Ålesund
>>>> <https://www.google.com/maps/search/Vestre+Aspehaug+4,+6010+%C3%85lesund++Norge?entry=gmail&source=g>
>>>> Norge
>>>> <https://www.google.com/maps/search/Vestre+Aspehaug+4,+6010+%C3%85lesund++Norge?entry=gmail&source=g>
>>>>
>>>> +47 480 94 297
>>>>
>>> --
>> You received this message because you are subscribed to the Google Groups
>> "GraphFrames" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to graphframes+unsubscribe@googlegroups.com.
>> To view this discussion visit
>> https://groups.google.com/d/msgid/graphframes/CAGUyL8ifNPF-0fs6aubJi4REd4JS4-SPEgfHM7E7ugd7a_J5XQ%40mail.gmail.com
>> <https://groups.google.com/d/msgid/graphframes/CAGUyL8ifNPF-0fs6aubJi4REd4JS4-SPEgfHM7E7ugd7a_J5XQ%40mail.gmail.com?utm_medium=email&utm_source=footer>
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
Re: GraphFrames' ConnectedComponentSuite test 'two components and two dangling vertices' fails with OutOfMemoryError: Java heap space
Posted by Russell Jurney <rj...@graphlet.ai>.
Very cool, thanks for sharing!
Thanks,
Russell Jurney | rjurney@graphlet.ai | graphlet.ai | Graphlet AI Blog
<https://blog.graphlet.ai/> | LinkedIn
<https://linkedin.com/in/russelljurney> | BlueSky
<https://bsky.app/profile/rjurney.bsky.social>
On Sat, Feb 1, 2025 at 7:54 AM Ángel <an...@gmail.com> wrote:
> My first article about this issue:
> https://medium.com/@angel.alvarez.pascua/apache-spark-wtf-i-like-it-when-a-plan-comes-together-part-i-48c52a667288
>
> El mié, 22 ene 2025, 3:13, Ángel <an...@gmail.com>
> escribió:
>
>> Hi Team,
>>
>> I believe I’ve finally managed to fix the OOM issue, though I’m still
>> analyzing some details and conducting tests to confirm my findings. The
>> issue seems to be related to how Spark updates and propagates the plan
>> across the internal bus (AdaptiveSparkPlanExec.onUpdatePlan) but, above
>> all, to how Spark generates the physical plan when DataFrames are cached
>> iteratively—such as in GraphFrames. I’ve fixed the OOM issue by adding some
>> minor improvements to the Spark code, but I’d also like to explore whether
>> a solution might be achievable by modifying only the GraphFrames code, in
>> case my improvements to Spark are not accepted.
>>
>> Over the next few weeks, I’m planning to write three articles summarizing
>> everything I’ve discovered. I’ll share the links once they’re published,
>> and I’d greatly appreciate any feedback you can provide.
>>
>> @Bjørn Jørgensen, I was genuinely surprised by the links you shared last
>> week. Shouldn’t Connected Components be thoroughly tested in the
>> GraphFrames unit tests? Regardless, while this other issue appears to be
>> also fixed when AQE is disabled, I believe there isn't a connection. It
>> seems like the way GraphFrames assigned ids only functioned correctly when
>> data partitioning was static—that is, in older Spark versions where AQE
>> didn’t exist or was disabled by default. Do you happen to have a simple
>> dataset we could use for further testing?
>>
>> Regards,
>> Ángel
>>
>> El jue, 16 ene 2025 a las 9:00, Bjørn Jørgensen (<
>> bjornjorgensen@gmail.com>) escribió:
>>
>>> Connectes components produce non correct results
>>> <https://github.com/graphframes/graphframes/issues/428> and Connected
>>> Components gives wrong results
>>> <https://github.com/graphframes/graphframes/issues/453>
>>>
>>> tor. 16. jan. 2025 kl. 08:21 skrev Ángel <angel.alvarez.pascua@gmail.com
>>> >:
>>>
>>>> Don't thank me; I'm really learning a lot about Spark and Graphframes
>>>> internals and having fun digging into this issue. I'm planning to write an
>>>> article about it next weekend.
>>>>
>>>> I'm getting closer and closer...
>>>>
>>>> Btw., I've also noticed a couple of issues with Spark when working with
>>>> large plans. I find it weird that nobody has realized this before. I want
>>>> to double-check them - and also test them in Spark 4 - , but if I'm
>>>> correct, I'll open a ticket on the Spark project and submit a PR this week.
>>>> As just mentioned, I could be wrong, but with the improvements, the test
>>>> went from taking 3–4 minutes to only 1.5 minutes.
>>>>
>>>>
>>>> El jue, 16 ene 2025 a las 1:15, Russell Jurney (<
>>>> russell.jurney@gmail.com>) escribió:
>>>>
>>>>> Thank you SO much for looking into this! GraphFrames lives!
>>>>>
>>>>> As an update, here's the most popular logo from a poll I took:
>>>>>
>>>>> [image: image.png]
>>>>>
>>>>> Although this one in second place fits more with the Apache logo
>>>>> directory <https://www.apache.org/logos/>:
>>>>>
>>>>> [image: image.png]
>>>>>
>>>>> Which one makes you want to work on this issue more? :D
>>>>>
>>>>> Russ
>>>>>
>>>>> On Wed, Jan 15, 2025 at 5:04 AM Ángel <an...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> My only change was increasing 4 to 10 in the
>>>>>> spark.sql.shuffle.partitions setting (still doesn't work for a value lower
>>>>>> than 10).
>>>>>>
>>>>>> [image: image.png]
>>>>>>
>>>>>> I've performed some tests, changed some code ... and I'm starting to
>>>>>> grasp what's behind this OOM, but I'm still working on it .. quite
>>>>>> interesting indeed this issue! (thanks)
>>>>>>
>>>>>> According to MAT <https://eclipse.dev/mat/> ... a humongous
>>>>>> execution plan is being created ...
>>>>>> [image: image.png]
>>>>>>
>>>>>> El mar, 14 ene 2025 a las 21:52, Russell Jurney (<
>>>>>> russell.jurney@gmail.com>) escribió:
>>>>>>
>>>>>>> Can you please share the code? It doesn't seem an ideal solution,
>>>>>>> but if AQE is confused, disabling it makes sense. I can't figure out why a
>>>>>>> low partition count for an 8 node, 6 edge network would require a lot of
>>>>>>> partitions... users may have different numbers... do you suggest we enforce
>>>>>>> some minimum partition count?
>>>>>>>
>>>>>>> On Tue, Jan 14, 2025 at 7:33 AM Ángel <
>>>>>>> angel.alvarez.pascua@gmail.com> wrote:
>>>>>>>
>>>>>>>> Are you sure that temporarily disabling a global setting like AQE
>>>>>>>> is the best approach to fix this issue?
>>>>>>>> I increased the number of shuffle partitions in the Spark session
>>>>>>>> configure in the GraphFrameTestSparkContext.scala from 4 to 10, and the
>>>>>>>> "checkpoint interval" test ran perfectly without throwing an OOM error.
>>>>>>>> Why? No idea, but it worked.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> El lun, 13 ene 2025 a las 16:45, Russell Jurney (<
>>>>>>>> russell.jurney@gmail.com>) escribió:
>>>>>>>>
>>>>>>>>> Merged, thanks guys!
>>>>>>>>>
>>>>>>>>> Russ
>>>>>>>>>
>>>>>>>>> On Sun, Jan 12, 2025 at 2:23 PM Bjørn Jørgensen <
>>>>>>>>> bjornjorgensen@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> I tested stuff in this PR
>>>>>>>>>> https://github.com/graphframes/graphframes/pull/477 and then I
>>>>>>>>>> made this PR https://github.com/graphframes/graphframes/pull/478
>>>>>>>>>>
>>>>>>>>>> søn. 12. jan. 2025 kl. 23:10 skrev Ángel <
>>>>>>>>>> angel.alvarez.pascua@gmail.com>:
>>>>>>>>>>
>>>>>>>>>>> Hi Russell,
>>>>>>>>>>>
>>>>>>>>>>> I've just got the OOM error during Test 13. I'm running it from
>>>>>>>>>>> IntelliJ on Windows with Java 11.
>>>>>>>>>>>
>>>>>>>>>>> [image: image.png]
>>>>>>>>>>> I'll look into it over the course of the next week.
>>>>>>>>>>>
>>>>>>>>>>> Regards,
>>>>>>>>>>> Ángel
>>>>>>>>>>>
>>>>>>>>>>> El sáb, 11 ene 2025 a las 9:23, Russell Jurney (<
>>>>>>>>>>> russell.jurney@gmail.com>) escribió:
>>>>>>>>>>>
>>>>>>>>>>>> Friends of GraphFrames (github.com/graphframes/graphframes), I
>>>>>>>>>>>> have a question for you...
>>>>>>>>>>>>
>>>>>>>>>>>> I can't get the unit test 'two components and two dangling
>>>>>>>>>>>> vertices' in the org.graphframes.lib.ConnectedComponentsSuite
>>>>>>>>>>>> <https://github.com/graphframes/graphframes/blob/649094caf58cfda0eea3e8cd66785aa38104d771/src/test/scala/org/graphframes/lib/ConnectedComponentsSuite.scala#L138-L148>
>>>>>>>>>>>> to pass. It fails with an 'OutOfMemoryError: Java heap space' error. I am a
>>>>>>>>>>>> little stuck on completing a docs release with a motif finding
>>>>>>>>>>>> tutorial <https://github.com/graphframes/graphframes/pull/473>
>>>>>>>>>>>> due to this issue.
>>>>>>>>>>>>
>>>>>>>>>>>> The problem is outlined in this gist:
>>>>>>>>>>>> https://gist.github.com/rjurney/6abeffbd59c67df5e5243c8f6619b6bf
>>>>>>>>>>>>
>>>>>>>>>>>> Can someone else please try this and see if it passes on the
>>>>>>>>>>>> master branch?
>>>>>>>>>>>>
>>>>>>>>>>>> > build/sbt clean compile package test
>>>>>>>>>>>>
>>>>>>>>>>>> I've tried giving it lots of RAM just to see if it would help,
>>>>>>>>>>>> as much as 32g driver and 16g for executors and... it has no effect. The
>>>>>>>>>>>> test graph is 8 nodes and 6 edges
>>>>>>>>>>>> <https://gist.github.com/rjurney/6abeffbd59c67df5e5243c8f6619b6bf#file-connectedcomponentsuite-scala-L22-L26>,
>>>>>>>>>>>> so it shouldn't have a memory problem... yet when it runs, all 24 cores of
>>>>>>>>>>>> my CPU get used, it spikes as indicated in the image in the gist.
>>>>>>>>>>>>
>>>>>>>>>>>> I am running the following setup:
>>>>>>>>>>>>
>>>>>>>>>>>> * Ubuntu 20.04 (22.04 in the Docker image)
>>>>>>>>>>>> * OpenJDK 11 (I also tried 8, same problem)
>>>>>>>>>>>> * Scala 2.12.20 (I also tried 2.13, same problem)
>>>>>>>>>>>> * Python 3.11 (I also tried 3.9, same problem)
>>>>>>>>>>>>
>>>>>>>>>>>> Or I am running the Dockerfile in the gist
>>>>>>>>>>>> <https://gist.github.com/rjurney/6abeffbd59c67df5e5243c8f6619b6bf#file-dockerfile>
>>>>>>>>>>>> .
>>>>>>>>>>>>
>>>>>>>>>>>> Any help much appreciated! Thanks
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> -----------------------------------------------------------------
>>>>>>>>>>>> Oh, some new community stuff for GraphFrames. Hackathon
>>>>>>>>>>>> announced next week :)
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> - GraphFrames Mailing List
>>>>>>>>>>>> <https://groups.google.com/g/graphframes/>: ask questions
>>>>>>>>>>>> about GraphFrames on our Google Group
>>>>>>>>>>>> - #graphframes Discord Channel on GraphGeeks
>>>>>>>>>>>> <https://discord.com/channels/1162999022819225631/1326257052368113674>
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks!
>>>>>>>>>>>> Russell Jurney @rjurney <http://twitter.com/rjurney>
>>>>>>>>>>>> russell.jurney@gmail.com LI
>>>>>>>>>>>> <http://linkedin.com/in/russelljurney> FB
>>>>>>>>>>>> <http://facebook.com/jurney> datasyndrome.com
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Bjørn Jørgensen
>>>>>>>>>> Vestre Aspehaug 4, 6010 Ålesund
>>>>>>>>>> <https://www.google.com/maps/search/Vestre+Aspehaug+4,+6010+%C3%85lesund++Norge?entry=gmail&source=g>
>>>>>>>>>> Norge
>>>>>>>>>> <https://www.google.com/maps/search/Vestre+Aspehaug+4,+6010+%C3%85lesund++Norge?entry=gmail&source=g>
>>>>>>>>>>
>>>>>>>>>> +47 480 94 297
>>>>>>>>>>
>>>>>>>>>
>>>
>>> --
>>> Bjørn Jørgensen
>>> Vestre Aspehaug 4, 6010 Ålesund
>>> <https://www.google.com/maps/search/Vestre+Aspehaug+4,+6010+%C3%85lesund++Norge?entry=gmail&source=g>
>>> Norge
>>> <https://www.google.com/maps/search/Vestre+Aspehaug+4,+6010+%C3%85lesund++Norge?entry=gmail&source=g>
>>>
>>> +47 480 94 297
>>>
>> --
> You received this message because you are subscribed to the Google Groups
> "GraphFrames" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to graphframes+unsubscribe@googlegroups.com.
> To view this discussion visit
> https://groups.google.com/d/msgid/graphframes/CAGUyL8ifNPF-0fs6aubJi4REd4JS4-SPEgfHM7E7ugd7a_J5XQ%40mail.gmail.com
> <https://groups.google.com/d/msgid/graphframes/CAGUyL8ifNPF-0fs6aubJi4REd4JS4-SPEgfHM7E7ugd7a_J5XQ%40mail.gmail.com?utm_medium=email&utm_source=footer>
> .
> For more options, visit https://groups.google.com/d/optout.
>
Re: GraphFrames' ConnectedComponentSuite test 'two components and two dangling vertices' fails with OutOfMemoryError: Java heap space
Posted by Ángel <an...@gmail.com>.
My first article about this issue:
https://medium.com/@angel.alvarez.pascua/apache-spark-wtf-i-like-it-when-a-plan-comes-together-part-i-48c52a667288
El mié, 22 ene 2025, 3:13, Ángel <an...@gmail.com> escribió:
> Hi Team,
>
> I believe I’ve finally managed to fix the OOM issue, though I’m still
> analyzing some details and conducting tests to confirm my findings. The
> issue seems to be related to how Spark updates and propagates the plan
> across the internal bus (AdaptiveSparkPlanExec.onUpdatePlan) but, above
> all, to how Spark generates the physical plan when DataFrames are cached
> iteratively—such as in GraphFrames. I’ve fixed the OOM issue by adding some
> minor improvements to the Spark code, but I’d also like to explore whether
> a solution might be achievable by modifying only the GraphFrames code, in
> case my improvements to Spark are not accepted.
>
> Over the next few weeks, I’m planning to write three articles summarizing
> everything I’ve discovered. I’ll share the links once they’re published,
> and I’d greatly appreciate any feedback you can provide.
>
> @Bjørn Jørgensen, I was genuinely surprised by the links you shared last
> week. Shouldn’t Connected Components be thoroughly tested in the
> GraphFrames unit tests? Regardless, while this other issue appears to be
> also fixed when AQE is disabled, I believe there isn't a connection. It
> seems like the way GraphFrames assigned ids only functioned correctly when
> data partitioning was static—that is, in older Spark versions where AQE
> didn’t exist or was disabled by default. Do you happen to have a simple
> dataset we could use for further testing?
>
> Regards,
> Ángel
>
> El jue, 16 ene 2025 a las 9:00, Bjørn Jørgensen (<bj...@gmail.com>)
> escribió:
>
>> Connectes components produce non correct results
>> <https://github.com/graphframes/graphframes/issues/428> and Connected
>> Components gives wrong results
>> <https://github.com/graphframes/graphframes/issues/453>
>>
>> tor. 16. jan. 2025 kl. 08:21 skrev Ángel <angel.alvarez.pascua@gmail.com
>> >:
>>
>>> Don't thank me; I'm really learning a lot about Spark and Graphframes
>>> internals and having fun digging into this issue. I'm planning to write an
>>> article about it next weekend.
>>>
>>> I'm getting closer and closer...
>>>
>>> Btw., I've also noticed a couple of issues with Spark when working with
>>> large plans. I find it weird that nobody has realized this before. I want
>>> to double-check them - and also test them in Spark 4 - , but if I'm
>>> correct, I'll open a ticket on the Spark project and submit a PR this week.
>>> As just mentioned, I could be wrong, but with the improvements, the test
>>> went from taking 3–4 minutes to only 1.5 minutes.
>>>
>>>
>>> El jue, 16 ene 2025 a las 1:15, Russell Jurney (<
>>> russell.jurney@gmail.com>) escribió:
>>>
>>>> Thank you SO much for looking into this! GraphFrames lives!
>>>>
>>>> As an update, here's the most popular logo from a poll I took:
>>>>
>>>> [image: image.png]
>>>>
>>>> Although this one in second place fits more with the Apache logo
>>>> directory <https://www.apache.org/logos/>:
>>>>
>>>> [image: image.png]
>>>>
>>>> Which one makes you want to work on this issue more? :D
>>>>
>>>> Russ
>>>>
>>>> On Wed, Jan 15, 2025 at 5:04 AM Ángel <an...@gmail.com>
>>>> wrote:
>>>>
>>>>> My only change was increasing 4 to 10 in the
>>>>> spark.sql.shuffle.partitions setting (still doesn't work for a value lower
>>>>> than 10).
>>>>>
>>>>> [image: image.png]
>>>>>
>>>>> I've performed some tests, changed some code ... and I'm starting to
>>>>> grasp what's behind this OOM, but I'm still working on it .. quite
>>>>> interesting indeed this issue! (thanks)
>>>>>
>>>>> According to MAT <https://eclipse.dev/mat/> ... a humongous execution
>>>>> plan is being created ...
>>>>> [image: image.png]
>>>>>
>>>>> El mar, 14 ene 2025 a las 21:52, Russell Jurney (<
>>>>> russell.jurney@gmail.com>) escribió:
>>>>>
>>>>>> Can you please share the code? It doesn't seem an ideal solution, but
>>>>>> if AQE is confused, disabling it makes sense. I can't figure out why a low
>>>>>> partition count for an 8 node, 6 edge network would require a lot of
>>>>>> partitions... users may have different numbers... do you suggest we enforce
>>>>>> some minimum partition count?
>>>>>>
>>>>>> On Tue, Jan 14, 2025 at 7:33 AM Ángel <an...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Are you sure that temporarily disabling a global setting like AQE is
>>>>>>> the best approach to fix this issue?
>>>>>>> I increased the number of shuffle partitions in the Spark session
>>>>>>> configure in the GraphFrameTestSparkContext.scala from 4 to 10, and the
>>>>>>> "checkpoint interval" test ran perfectly without throwing an OOM error.
>>>>>>> Why? No idea, but it worked.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> El lun, 13 ene 2025 a las 16:45, Russell Jurney (<
>>>>>>> russell.jurney@gmail.com>) escribió:
>>>>>>>
>>>>>>>> Merged, thanks guys!
>>>>>>>>
>>>>>>>> Russ
>>>>>>>>
>>>>>>>> On Sun, Jan 12, 2025 at 2:23 PM Bjørn Jørgensen <
>>>>>>>> bjornjorgensen@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> I tested stuff in this PR
>>>>>>>>> https://github.com/graphframes/graphframes/pull/477 and then I
>>>>>>>>> made this PR https://github.com/graphframes/graphframes/pull/478
>>>>>>>>>
>>>>>>>>> søn. 12. jan. 2025 kl. 23:10 skrev Ángel <
>>>>>>>>> angel.alvarez.pascua@gmail.com>:
>>>>>>>>>
>>>>>>>>>> Hi Russell,
>>>>>>>>>>
>>>>>>>>>> I've just got the OOM error during Test 13. I'm running it from
>>>>>>>>>> IntelliJ on Windows with Java 11.
>>>>>>>>>>
>>>>>>>>>> [image: image.png]
>>>>>>>>>> I'll look into it over the course of the next week.
>>>>>>>>>>
>>>>>>>>>> Regards,
>>>>>>>>>> Ángel
>>>>>>>>>>
>>>>>>>>>> El sáb, 11 ene 2025 a las 9:23, Russell Jurney (<
>>>>>>>>>> russell.jurney@gmail.com>) escribió:
>>>>>>>>>>
>>>>>>>>>>> Friends of GraphFrames (github.com/graphframes/graphframes), I
>>>>>>>>>>> have a question for you...
>>>>>>>>>>>
>>>>>>>>>>> I can't get the unit test 'two components and two dangling
>>>>>>>>>>> vertices' in the org.graphframes.lib.ConnectedComponentsSuite
>>>>>>>>>>> <https://github.com/graphframes/graphframes/blob/649094caf58cfda0eea3e8cd66785aa38104d771/src/test/scala/org/graphframes/lib/ConnectedComponentsSuite.scala#L138-L148>
>>>>>>>>>>> to pass. It fails with an 'OutOfMemoryError: Java heap space' error. I am a
>>>>>>>>>>> little stuck on completing a docs release with a motif finding
>>>>>>>>>>> tutorial <https://github.com/graphframes/graphframes/pull/473>
>>>>>>>>>>> due to this issue.
>>>>>>>>>>>
>>>>>>>>>>> The problem is outlined in this gist:
>>>>>>>>>>> https://gist.github.com/rjurney/6abeffbd59c67df5e5243c8f6619b6bf
>>>>>>>>>>>
>>>>>>>>>>> Can someone else please try this and see if it passes on the
>>>>>>>>>>> master branch?
>>>>>>>>>>>
>>>>>>>>>>> > build/sbt clean compile package test
>>>>>>>>>>>
>>>>>>>>>>> I've tried giving it lots of RAM just to see if it would help,
>>>>>>>>>>> as much as 32g driver and 16g for executors and... it has no effect. The
>>>>>>>>>>> test graph is 8 nodes and 6 edges
>>>>>>>>>>> <https://gist.github.com/rjurney/6abeffbd59c67df5e5243c8f6619b6bf#file-connectedcomponentsuite-scala-L22-L26>,
>>>>>>>>>>> so it shouldn't have a memory problem... yet when it runs, all 24 cores of
>>>>>>>>>>> my CPU get used, it spikes as indicated in the image in the gist.
>>>>>>>>>>>
>>>>>>>>>>> I am running the following setup:
>>>>>>>>>>>
>>>>>>>>>>> * Ubuntu 20.04 (22.04 in the Docker image)
>>>>>>>>>>> * OpenJDK 11 (I also tried 8, same problem)
>>>>>>>>>>> * Scala 2.12.20 (I also tried 2.13, same problem)
>>>>>>>>>>> * Python 3.11 (I also tried 3.9, same problem)
>>>>>>>>>>>
>>>>>>>>>>> Or I am running the Dockerfile in the gist
>>>>>>>>>>> <https://gist.github.com/rjurney/6abeffbd59c67df5e5243c8f6619b6bf#file-dockerfile>
>>>>>>>>>>> .
>>>>>>>>>>>
>>>>>>>>>>> Any help much appreciated! Thanks
>>>>>>>>>>>
>>>>>>>>>>> -----------------------------------------------------------------
>>>>>>>>>>> Oh, some new community stuff for GraphFrames. Hackathon
>>>>>>>>>>> announced next week :)
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> - GraphFrames Mailing List
>>>>>>>>>>> <https://groups.google.com/g/graphframes/>: ask questions
>>>>>>>>>>> about GraphFrames on our Google Group
>>>>>>>>>>> - #graphframes Discord Channel on GraphGeeks
>>>>>>>>>>> <https://discord.com/channels/1162999022819225631/1326257052368113674>
>>>>>>>>>>>
>>>>>>>>>>> Thanks!
>>>>>>>>>>> Russell Jurney @rjurney <http://twitter.com/rjurney>
>>>>>>>>>>> russell.jurney@gmail.com LI
>>>>>>>>>>> <http://linkedin.com/in/russelljurney> FB
>>>>>>>>>>> <http://facebook.com/jurney> datasyndrome.com
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Bjørn Jørgensen
>>>>>>>>> Vestre Aspehaug 4, 6010 Ålesund
>>>>>>>>> <https://www.google.com/maps/search/Vestre+Aspehaug+4,+6010+%C3%85lesund++Norge?entry=gmail&source=g>
>>>>>>>>> Norge
>>>>>>>>> <https://www.google.com/maps/search/Vestre+Aspehaug+4,+6010+%C3%85lesund++Norge?entry=gmail&source=g>
>>>>>>>>>
>>>>>>>>> +47 480 94 297
>>>>>>>>>
>>>>>>>>
>>
>> --
>> Bjørn Jørgensen
>> Vestre Aspehaug 4, 6010 Ålesund
>> Norge
>>
>> +47 480 94 297
>>
>
Re: GraphFrames' ConnectedComponentSuite test 'two components and two dangling vertices' fails with OutOfMemoryError: Java heap space
Posted by Ángel <an...@gmail.com>.
Hi Team,
I believe I’ve finally managed to fix the OOM issue, though I’m still
analyzing some details and conducting tests to confirm my findings. The
issue seems to be related to how Spark updates and propagates the plan
across the internal bus (AdaptiveSparkPlanExec.onUpdatePlan) but, above
all, to how Spark generates the physical plan when DataFrames are cached
iteratively—such as in GraphFrames. I’ve fixed the OOM issue by adding some
minor improvements to the Spark code, but I’d also like to explore whether
a solution might be achievable by modifying only the GraphFrames code, in
case my improvements to Spark are not accepted.
Over the next few weeks, I’m planning to write three articles summarizing
everything I’ve discovered. I’ll share the links once they’re published,
and I’d greatly appreciate any feedback you can provide.
@Bjørn Jørgensen, I was genuinely surprised by the links you shared last
week. Shouldn’t Connected Components be thoroughly tested in the
GraphFrames unit tests? Regardless, while this other issue appears to be
also fixed when AQE is disabled, I believe there isn't a connection. It
seems like the way GraphFrames assigned ids only functioned correctly when
data partitioning was static—that is, in older Spark versions where AQE
didn’t exist or was disabled by default. Do you happen to have a simple
dataset we could use for further testing?
Regards,
Ángel
El jue, 16 ene 2025 a las 9:00, Bjørn Jørgensen (<bj...@gmail.com>)
escribió:
> Connectes components produce non correct results
> <https://github.com/graphframes/graphframes/issues/428> and Connected
> Components gives wrong results
> <https://github.com/graphframes/graphframes/issues/453>
>
> tor. 16. jan. 2025 kl. 08:21 skrev Ángel <an...@gmail.com>:
>
>> Don't thank me; I'm really learning a lot about Spark and Graphframes
>> internals and having fun digging into this issue. I'm planning to write an
>> article about it next weekend.
>>
>> I'm getting closer and closer...
>>
>> Btw., I've also noticed a couple of issues with Spark when working with
>> large plans. I find it weird that nobody has realized this before. I want
>> to double-check them - and also test them in Spark 4 - , but if I'm
>> correct, I'll open a ticket on the Spark project and submit a PR this week.
>> As just mentioned, I could be wrong, but with the improvements, the test
>> went from taking 3–4 minutes to only 1.5 minutes.
>>
>>
>> El jue, 16 ene 2025 a las 1:15, Russell Jurney (<ru...@gmail.com>)
>> escribió:
>>
>>> Thank you SO much for looking into this! GraphFrames lives!
>>>
>>> As an update, here's the most popular logo from a poll I took:
>>>
>>> [image: image.png]
>>>
>>> Although this one in second place fits more with the Apache logo
>>> directory <https://www.apache.org/logos/>:
>>>
>>> [image: image.png]
>>>
>>> Which one makes you want to work on this issue more? :D
>>>
>>> Russ
>>>
>>> On Wed, Jan 15, 2025 at 5:04 AM Ángel <an...@gmail.com>
>>> wrote:
>>>
>>>> My only change was increasing 4 to 10 in the
>>>> spark.sql.shuffle.partitions setting (still doesn't work for a value lower
>>>> than 10).
>>>>
>>>> [image: image.png]
>>>>
>>>> I've performed some tests, changed some code ... and I'm starting to
>>>> grasp what's behind this OOM, but I'm still working on it .. quite
>>>> interesting indeed this issue! (thanks)
>>>>
>>>> According to MAT <https://eclipse.dev/mat/> ... a humongous execution
>>>> plan is being created ...
>>>> [image: image.png]
>>>>
>>>> El mar, 14 ene 2025 a las 21:52, Russell Jurney (<
>>>> russell.jurney@gmail.com>) escribió:
>>>>
>>>>> Can you please share the code? It doesn't seem an ideal solution, but
>>>>> if AQE is confused, disabling it makes sense. I can't figure out why a low
>>>>> partition count for an 8 node, 6 edge network would require a lot of
>>>>> partitions... users may have different numbers... do you suggest we enforce
>>>>> some minimum partition count?
>>>>>
>>>>> On Tue, Jan 14, 2025 at 7:33 AM Ángel <an...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Are you sure that temporarily disabling a global setting like AQE is
>>>>>> the best approach to fix this issue?
>>>>>> I increased the number of shuffle partitions in the Spark session
>>>>>> configure in the GraphFrameTestSparkContext.scala from 4 to 10, and the
>>>>>> "checkpoint interval" test ran perfectly without throwing an OOM error.
>>>>>> Why? No idea, but it worked.
>>>>>>
>>>>>>
>>>>>>
>>>>>> El lun, 13 ene 2025 a las 16:45, Russell Jurney (<
>>>>>> russell.jurney@gmail.com>) escribió:
>>>>>>
>>>>>>> Merged, thanks guys!
>>>>>>>
>>>>>>> Russ
>>>>>>>
>>>>>>> On Sun, Jan 12, 2025 at 2:23 PM Bjørn Jørgensen <
>>>>>>> bjornjorgensen@gmail.com> wrote:
>>>>>>>
>>>>>>>> I tested stuff in this PR
>>>>>>>> https://github.com/graphframes/graphframes/pull/477 and then I
>>>>>>>> made this PR https://github.com/graphframes/graphframes/pull/478
>>>>>>>>
>>>>>>>> søn. 12. jan. 2025 kl. 23:10 skrev Ángel <
>>>>>>>> angel.alvarez.pascua@gmail.com>:
>>>>>>>>
>>>>>>>>> Hi Russell,
>>>>>>>>>
>>>>>>>>> I've just got the OOM error during Test 13. I'm running it from
>>>>>>>>> IntelliJ on Windows with Java 11.
>>>>>>>>>
>>>>>>>>> [image: image.png]
>>>>>>>>> I'll look into it over the course of the next week.
>>>>>>>>>
>>>>>>>>> Regards,
>>>>>>>>> Ángel
>>>>>>>>>
>>>>>>>>> El sáb, 11 ene 2025 a las 9:23, Russell Jurney (<
>>>>>>>>> russell.jurney@gmail.com>) escribió:
>>>>>>>>>
>>>>>>>>>> Friends of GraphFrames (github.com/graphframes/graphframes), I
>>>>>>>>>> have a question for you...
>>>>>>>>>>
>>>>>>>>>> I can't get the unit test 'two components and two dangling
>>>>>>>>>> vertices' in the org.graphframes.lib.ConnectedComponentsSuite
>>>>>>>>>> <https://github.com/graphframes/graphframes/blob/649094caf58cfda0eea3e8cd66785aa38104d771/src/test/scala/org/graphframes/lib/ConnectedComponentsSuite.scala#L138-L148>
>>>>>>>>>> to pass. It fails with an 'OutOfMemoryError: Java heap space' error. I am a
>>>>>>>>>> little stuck on completing a docs release with a motif finding
>>>>>>>>>> tutorial <https://github.com/graphframes/graphframes/pull/473>
>>>>>>>>>> due to this issue.
>>>>>>>>>>
>>>>>>>>>> The problem is outlined in this gist:
>>>>>>>>>> https://gist.github.com/rjurney/6abeffbd59c67df5e5243c8f6619b6bf
>>>>>>>>>>
>>>>>>>>>> Can someone else please try this and see if it passes on the
>>>>>>>>>> master branch?
>>>>>>>>>>
>>>>>>>>>> > build/sbt clean compile package test
>>>>>>>>>>
>>>>>>>>>> I've tried giving it lots of RAM just to see if it would help, as
>>>>>>>>>> much as 32g driver and 16g for executors and... it has no effect. The test
>>>>>>>>>> graph is 8 nodes and 6 edges
>>>>>>>>>> <https://gist.github.com/rjurney/6abeffbd59c67df5e5243c8f6619b6bf#file-connectedcomponentsuite-scala-L22-L26>,
>>>>>>>>>> so it shouldn't have a memory problem... yet when it runs, all 24 cores of
>>>>>>>>>> my CPU get used, it spikes as indicated in the image in the gist.
>>>>>>>>>>
>>>>>>>>>> I am running the following setup:
>>>>>>>>>>
>>>>>>>>>> * Ubuntu 20.04 (22.04 in the Docker image)
>>>>>>>>>> * OpenJDK 11 (I also tried 8, same problem)
>>>>>>>>>> * Scala 2.12.20 (I also tried 2.13, same problem)
>>>>>>>>>> * Python 3.11 (I also tried 3.9, same problem)
>>>>>>>>>>
>>>>>>>>>> Or I am running the Dockerfile in the gist
>>>>>>>>>> <https://gist.github.com/rjurney/6abeffbd59c67df5e5243c8f6619b6bf#file-dockerfile>
>>>>>>>>>> .
>>>>>>>>>>
>>>>>>>>>> Any help much appreciated! Thanks
>>>>>>>>>>
>>>>>>>>>> -----------------------------------------------------------------
>>>>>>>>>> Oh, some new community stuff for GraphFrames. Hackathon announced
>>>>>>>>>> next week :)
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> - GraphFrames Mailing List
>>>>>>>>>> <https://groups.google.com/g/graphframes/>: ask questions
>>>>>>>>>> about GraphFrames on our Google Group
>>>>>>>>>> - #graphframes Discord Channel on GraphGeeks
>>>>>>>>>> <https://discord.com/channels/1162999022819225631/1326257052368113674>
>>>>>>>>>>
>>>>>>>>>> Thanks!
>>>>>>>>>> Russell Jurney @rjurney <http://twitter.com/rjurney>
>>>>>>>>>> russell.jurney@gmail.com LI
>>>>>>>>>> <http://linkedin.com/in/russelljurney> FB
>>>>>>>>>> <http://facebook.com/jurney> datasyndrome.com
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Bjørn Jørgensen
>>>>>>>> Vestre Aspehaug 4, 6010 Ålesund
>>>>>>>> <https://www.google.com/maps/search/Vestre+Aspehaug+4,+6010+%C3%85lesund++Norge?entry=gmail&source=g>
>>>>>>>> Norge
>>>>>>>> <https://www.google.com/maps/search/Vestre+Aspehaug+4,+6010+%C3%85lesund++Norge?entry=gmail&source=g>
>>>>>>>>
>>>>>>>> +47 480 94 297
>>>>>>>>
>>>>>>>
>
> --
> Bjørn Jørgensen
> Vestre Aspehaug 4, 6010 Ålesund
> Norge
>
> +47 480 94 297
>
Re: GraphFrames' ConnectedComponentSuite test 'two components and two dangling vertices' fails with OutOfMemoryError: Java heap space
Posted by Bjørn Jørgensen <bj...@gmail.com>.
Connectes components produce non correct results
<https://github.com/graphframes/graphframes/issues/428> and Connected
Components gives wrong results
<https://github.com/graphframes/graphframes/issues/453>
tor. 16. jan. 2025 kl. 08:21 skrev Ángel <an...@gmail.com>:
> Don't thank me; I'm really learning a lot about Spark and Graphframes
> internals and having fun digging into this issue. I'm planning to write an
> article about it next weekend.
>
> I'm getting closer and closer...
>
> Btw., I've also noticed a couple of issues with Spark when working with
> large plans. I find it weird that nobody has realized this before. I want
> to double-check them - and also test them in Spark 4 - , but if I'm
> correct, I'll open a ticket on the Spark project and submit a PR this week.
> As just mentioned, I could be wrong, but with the improvements, the test
> went from taking 3–4 minutes to only 1.5 minutes.
>
>
> El jue, 16 ene 2025 a las 1:15, Russell Jurney (<ru...@gmail.com>)
> escribió:
>
>> Thank you SO much for looking into this! GraphFrames lives!
>>
>> As an update, here's the most popular logo from a poll I took:
>>
>> [image: image.png]
>>
>> Although this one in second place fits more with the Apache logo
>> directory <https://www.apache.org/logos/>:
>>
>> [image: image.png]
>>
>> Which one makes you want to work on this issue more? :D
>>
>> Russ
>>
>> On Wed, Jan 15, 2025 at 5:04 AM Ángel <an...@gmail.com>
>> wrote:
>>
>>> My only change was increasing 4 to 10 in the
>>> spark.sql.shuffle.partitions setting (still doesn't work for a value lower
>>> than 10).
>>>
>>> [image: image.png]
>>>
>>> I've performed some tests, changed some code ... and I'm starting to
>>> grasp what's behind this OOM, but I'm still working on it .. quite
>>> interesting indeed this issue! (thanks)
>>>
>>> According to MAT <https://eclipse.dev/mat/> ... a humongous execution
>>> plan is being created ...
>>> [image: image.png]
>>>
>>> El mar, 14 ene 2025 a las 21:52, Russell Jurney (<
>>> russell.jurney@gmail.com>) escribió:
>>>
>>>> Can you please share the code? It doesn't seem an ideal solution, but
>>>> if AQE is confused, disabling it makes sense. I can't figure out why a low
>>>> partition count for an 8 node, 6 edge network would require a lot of
>>>> partitions... users may have different numbers... do you suggest we enforce
>>>> some minimum partition count?
>>>>
>>>> On Tue, Jan 14, 2025 at 7:33 AM Ángel <an...@gmail.com>
>>>> wrote:
>>>>
>>>>> Are you sure that temporarily disabling a global setting like AQE is
>>>>> the best approach to fix this issue?
>>>>> I increased the number of shuffle partitions in the Spark session
>>>>> configure in the GraphFrameTestSparkContext.scala from 4 to 10, and the
>>>>> "checkpoint interval" test ran perfectly without throwing an OOM error.
>>>>> Why? No idea, but it worked.
>>>>>
>>>>>
>>>>>
>>>>> El lun, 13 ene 2025 a las 16:45, Russell Jurney (<
>>>>> russell.jurney@gmail.com>) escribió:
>>>>>
>>>>>> Merged, thanks guys!
>>>>>>
>>>>>> Russ
>>>>>>
>>>>>> On Sun, Jan 12, 2025 at 2:23 PM Bjørn Jørgensen <
>>>>>> bjornjorgensen@gmail.com> wrote:
>>>>>>
>>>>>>> I tested stuff in this PR
>>>>>>> https://github.com/graphframes/graphframes/pull/477 and then I made
>>>>>>> this PR https://github.com/graphframes/graphframes/pull/478
>>>>>>>
>>>>>>> søn. 12. jan. 2025 kl. 23:10 skrev Ángel <
>>>>>>> angel.alvarez.pascua@gmail.com>:
>>>>>>>
>>>>>>>> Hi Russell,
>>>>>>>>
>>>>>>>> I've just got the OOM error during Test 13. I'm running it from
>>>>>>>> IntelliJ on Windows with Java 11.
>>>>>>>>
>>>>>>>> [image: image.png]
>>>>>>>> I'll look into it over the course of the next week.
>>>>>>>>
>>>>>>>> Regards,
>>>>>>>> Ángel
>>>>>>>>
>>>>>>>> El sáb, 11 ene 2025 a las 9:23, Russell Jurney (<
>>>>>>>> russell.jurney@gmail.com>) escribió:
>>>>>>>>
>>>>>>>>> Friends of GraphFrames (github.com/graphframes/graphframes), I
>>>>>>>>> have a question for you...
>>>>>>>>>
>>>>>>>>> I can't get the unit test 'two components and two dangling
>>>>>>>>> vertices' in the org.graphframes.lib.ConnectedComponentsSuite
>>>>>>>>> <https://github.com/graphframes/graphframes/blob/649094caf58cfda0eea3e8cd66785aa38104d771/src/test/scala/org/graphframes/lib/ConnectedComponentsSuite.scala#L138-L148>
>>>>>>>>> to pass. It fails with an 'OutOfMemoryError: Java heap space' error. I am a
>>>>>>>>> little stuck on completing a docs release with a motif finding
>>>>>>>>> tutorial <https://github.com/graphframes/graphframes/pull/473>
>>>>>>>>> due to this issue.
>>>>>>>>>
>>>>>>>>> The problem is outlined in this gist:
>>>>>>>>> https://gist.github.com/rjurney/6abeffbd59c67df5e5243c8f6619b6bf
>>>>>>>>>
>>>>>>>>> Can someone else please try this and see if it passes on the
>>>>>>>>> master branch?
>>>>>>>>>
>>>>>>>>> > build/sbt clean compile package test
>>>>>>>>>
>>>>>>>>> I've tried giving it lots of RAM just to see if it would help, as
>>>>>>>>> much as 32g driver and 16g for executors and... it has no effect. The test
>>>>>>>>> graph is 8 nodes and 6 edges
>>>>>>>>> <https://gist.github.com/rjurney/6abeffbd59c67df5e5243c8f6619b6bf#file-connectedcomponentsuite-scala-L22-L26>,
>>>>>>>>> so it shouldn't have a memory problem... yet when it runs, all 24 cores of
>>>>>>>>> my CPU get used, it spikes as indicated in the image in the gist.
>>>>>>>>>
>>>>>>>>> I am running the following setup:
>>>>>>>>>
>>>>>>>>> * Ubuntu 20.04 (22.04 in the Docker image)
>>>>>>>>> * OpenJDK 11 (I also tried 8, same problem)
>>>>>>>>> * Scala 2.12.20 (I also tried 2.13, same problem)
>>>>>>>>> * Python 3.11 (I also tried 3.9, same problem)
>>>>>>>>>
>>>>>>>>> Or I am running the Dockerfile in the gist
>>>>>>>>> <https://gist.github.com/rjurney/6abeffbd59c67df5e5243c8f6619b6bf#file-dockerfile>
>>>>>>>>> .
>>>>>>>>>
>>>>>>>>> Any help much appreciated! Thanks
>>>>>>>>>
>>>>>>>>> -----------------------------------------------------------------
>>>>>>>>> Oh, some new community stuff for GraphFrames. Hackathon announced
>>>>>>>>> next week :)
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> - GraphFrames Mailing List
>>>>>>>>> <https://groups.google.com/g/graphframes/>: ask questions
>>>>>>>>> about GraphFrames on our Google Group
>>>>>>>>> - #graphframes Discord Channel on GraphGeeks
>>>>>>>>> <https://discord.com/channels/1162999022819225631/1326257052368113674>
>>>>>>>>>
>>>>>>>>> Thanks!
>>>>>>>>> Russell Jurney @rjurney <http://twitter.com/rjurney>
>>>>>>>>> russell.jurney@gmail.com LI <http://linkedin.com/in/russelljurney>
>>>>>>>>> FB <http://facebook.com/jurney> datasyndrome.com
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Bjørn Jørgensen
>>>>>>> Vestre Aspehaug 4, 6010 Ålesund
>>>>>>> <https://www.google.com/maps/search/Vestre+Aspehaug+4,+6010+%C3%85lesund++Norge?entry=gmail&source=g>
>>>>>>> Norge
>>>>>>> <https://www.google.com/maps/search/Vestre+Aspehaug+4,+6010+%C3%85lesund++Norge?entry=gmail&source=g>
>>>>>>>
>>>>>>> +47 480 94 297
>>>>>>>
>>>>>>
--
Bjørn Jørgensen
Vestre Aspehaug 4, 6010 Ålesund
Norge
+47 480 94 297
Re: GraphFrames' ConnectedComponentSuite test 'two components and two dangling vertices' fails with OutOfMemoryError: Java heap space
Posted by Ángel <an...@gmail.com>.
Don't thank me; I'm really learning a lot about Spark and Graphframes
internals and having fun digging into this issue. I'm planning to write an
article about it next weekend.
I'm getting closer and closer...
Btw., I've also noticed a couple of issues with Spark when working with
large plans. I find it weird that nobody has realized this before. I want
to double-check them - and also test them in Spark 4 - , but if I'm
correct, I'll open a ticket on the Spark project and submit a PR this week.
As just mentioned, I could be wrong, but with the improvements, the test
went from taking 3–4 minutes to only 1.5 minutes.
El jue, 16 ene 2025 a las 1:15, Russell Jurney (<ru...@gmail.com>)
escribió:
> Thank you SO much for looking into this! GraphFrames lives!
>
> As an update, here's the most popular logo from a poll I took:
>
> [image: image.png]
>
> Although this one in second place fits more with the Apache logo directory
> <https://www.apache.org/logos/>:
>
> [image: image.png]
>
> Which one makes you want to work on this issue more? :D
>
> Russ
>
> On Wed, Jan 15, 2025 at 5:04 AM Ángel <an...@gmail.com>
> wrote:
>
>> My only change was increasing 4 to 10 in the spark.sql.shuffle.partitions
>> setting (still doesn't work for a value lower than 10).
>>
>> [image: image.png]
>>
>> I've performed some tests, changed some code ... and I'm starting to
>> grasp what's behind this OOM, but I'm still working on it .. quite
>> interesting indeed this issue! (thanks)
>>
>> According to MAT <https://eclipse.dev/mat/> ... a humongous execution
>> plan is being created ...
>> [image: image.png]
>>
>> El mar, 14 ene 2025 a las 21:52, Russell Jurney (<
>> russell.jurney@gmail.com>) escribió:
>>
>>> Can you please share the code? It doesn't seem an ideal solution, but if
>>> AQE is confused, disabling it makes sense. I can't figure out why a low
>>> partition count for an 8 node, 6 edge network would require a lot of
>>> partitions... users may have different numbers... do you suggest we enforce
>>> some minimum partition count?
>>>
>>> On Tue, Jan 14, 2025 at 7:33 AM Ángel <an...@gmail.com>
>>> wrote:
>>>
>>>> Are you sure that temporarily disabling a global setting like AQE is
>>>> the best approach to fix this issue?
>>>> I increased the number of shuffle partitions in the Spark session
>>>> configure in the GraphFrameTestSparkContext.scala from 4 to 10, and the
>>>> "checkpoint interval" test ran perfectly without throwing an OOM error.
>>>> Why? No idea, but it worked.
>>>>
>>>>
>>>>
>>>> El lun, 13 ene 2025 a las 16:45, Russell Jurney (<
>>>> russell.jurney@gmail.com>) escribió:
>>>>
>>>>> Merged, thanks guys!
>>>>>
>>>>> Russ
>>>>>
>>>>> On Sun, Jan 12, 2025 at 2:23 PM Bjørn Jørgensen <
>>>>> bjornjorgensen@gmail.com> wrote:
>>>>>
>>>>>> I tested stuff in this PR
>>>>>> https://github.com/graphframes/graphframes/pull/477 and then I made
>>>>>> this PR https://github.com/graphframes/graphframes/pull/478
>>>>>>
>>>>>> søn. 12. jan. 2025 kl. 23:10 skrev Ángel <
>>>>>> angel.alvarez.pascua@gmail.com>:
>>>>>>
>>>>>>> Hi Russell,
>>>>>>>
>>>>>>> I've just got the OOM error during Test 13. I'm running it from
>>>>>>> IntelliJ on Windows with Java 11.
>>>>>>>
>>>>>>> [image: image.png]
>>>>>>> I'll look into it over the course of the next week.
>>>>>>>
>>>>>>> Regards,
>>>>>>> Ángel
>>>>>>>
>>>>>>> El sáb, 11 ene 2025 a las 9:23, Russell Jurney (<
>>>>>>> russell.jurney@gmail.com>) escribió:
>>>>>>>
>>>>>>>> Friends of GraphFrames (github.com/graphframes/graphframes), I
>>>>>>>> have a question for you...
>>>>>>>>
>>>>>>>> I can't get the unit test 'two components and two dangling
>>>>>>>> vertices' in the org.graphframes.lib.ConnectedComponentsSuite
>>>>>>>> <https://github.com/graphframes/graphframes/blob/649094caf58cfda0eea3e8cd66785aa38104d771/src/test/scala/org/graphframes/lib/ConnectedComponentsSuite.scala#L138-L148>
>>>>>>>> to pass. It fails with an 'OutOfMemoryError: Java heap space' error. I am a
>>>>>>>> little stuck on completing a docs release with a motif finding
>>>>>>>> tutorial <https://github.com/graphframes/graphframes/pull/473> due
>>>>>>>> to this issue.
>>>>>>>>
>>>>>>>> The problem is outlined in this gist:
>>>>>>>> https://gist.github.com/rjurney/6abeffbd59c67df5e5243c8f6619b6bf
>>>>>>>>
>>>>>>>> Can someone else please try this and see if it passes on the master
>>>>>>>> branch?
>>>>>>>>
>>>>>>>> > build/sbt clean compile package test
>>>>>>>>
>>>>>>>> I've tried giving it lots of RAM just to see if it would help, as
>>>>>>>> much as 32g driver and 16g for executors and... it has no effect. The test
>>>>>>>> graph is 8 nodes and 6 edges
>>>>>>>> <https://gist.github.com/rjurney/6abeffbd59c67df5e5243c8f6619b6bf#file-connectedcomponentsuite-scala-L22-L26>,
>>>>>>>> so it shouldn't have a memory problem... yet when it runs, all 24 cores of
>>>>>>>> my CPU get used, it spikes as indicated in the image in the gist.
>>>>>>>>
>>>>>>>> I am running the following setup:
>>>>>>>>
>>>>>>>> * Ubuntu 20.04 (22.04 in the Docker image)
>>>>>>>> * OpenJDK 11 (I also tried 8, same problem)
>>>>>>>> * Scala 2.12.20 (I also tried 2.13, same problem)
>>>>>>>> * Python 3.11 (I also tried 3.9, same problem)
>>>>>>>>
>>>>>>>> Or I am running the Dockerfile in the gist
>>>>>>>> <https://gist.github.com/rjurney/6abeffbd59c67df5e5243c8f6619b6bf#file-dockerfile>
>>>>>>>> .
>>>>>>>>
>>>>>>>> Any help much appreciated! Thanks
>>>>>>>>
>>>>>>>> -----------------------------------------------------------------
>>>>>>>> Oh, some new community stuff for GraphFrames. Hackathon announced
>>>>>>>> next week :)
>>>>>>>>
>>>>>>>>
>>>>>>>> - GraphFrames Mailing List
>>>>>>>> <https://groups.google.com/g/graphframes/>: ask questions about
>>>>>>>> GraphFrames on our Google Group
>>>>>>>> - #graphframes Discord Channel on GraphGeeks
>>>>>>>> <https://discord.com/channels/1162999022819225631/1326257052368113674>
>>>>>>>>
>>>>>>>> Thanks!
>>>>>>>> Russell Jurney @rjurney <http://twitter.com/rjurney>
>>>>>>>> russell.jurney@gmail.com LI <http://linkedin.com/in/russelljurney>
>>>>>>>> FB <http://facebook.com/jurney> datasyndrome.com
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>> --
>>>>>> Bjørn Jørgensen
>>>>>> Vestre Aspehaug 4, 6010 Ålesund
>>>>>> <https://www.google.com/maps/search/Vestre+Aspehaug+4,+6010+%C3%85lesund++Norge?entry=gmail&source=g>
>>>>>> Norge
>>>>>> <https://www.google.com/maps/search/Vestre+Aspehaug+4,+6010+%C3%85lesund++Norge?entry=gmail&source=g>
>>>>>>
>>>>>> +47 480 94 297
>>>>>>
>>>>>
Re: GraphFrames' ConnectedComponentSuite test 'two components and two dangling vertices' fails with OutOfMemoryError: Java heap space
Posted by Russell Jurney <ru...@gmail.com>.
Thank you SO much for looking into this! GraphFrames lives!
As an update, here's the most popular logo from a poll I took:
[image: image.png]
Although this one in second place fits more with the Apache logo directory
<https://www.apache.org/logos/>:
[image: image.png]
Which one makes you want to work on this issue more? :D
Russ
On Wed, Jan 15, 2025 at 5:04 AM Ángel <an...@gmail.com>
wrote:
> My only change was increasing 4 to 10 in the spark.sql.shuffle.partitions
> setting (still doesn't work for a value lower than 10).
>
> [image: image.png]
>
> I've performed some tests, changed some code ... and I'm starting to
> grasp what's behind this OOM, but I'm still working on it .. quite
> interesting indeed this issue! (thanks)
>
> According to MAT <https://eclipse.dev/mat/> ... a humongous execution
> plan is being created ...
> [image: image.png]
>
> El mar, 14 ene 2025 a las 21:52, Russell Jurney (<ru...@gmail.com>)
> escribió:
>
>> Can you please share the code? It doesn't seem an ideal solution, but if
>> AQE is confused, disabling it makes sense. I can't figure out why a low
>> partition count for an 8 node, 6 edge network would require a lot of
>> partitions... users may have different numbers... do you suggest we enforce
>> some minimum partition count?
>>
>> On Tue, Jan 14, 2025 at 7:33 AM Ángel <an...@gmail.com>
>> wrote:
>>
>>> Are you sure that temporarily disabling a global setting like AQE is the
>>> best approach to fix this issue?
>>> I increased the number of shuffle partitions in the Spark session
>>> configure in the GraphFrameTestSparkContext.scala from 4 to 10, and the
>>> "checkpoint interval" test ran perfectly without throwing an OOM error.
>>> Why? No idea, but it worked.
>>>
>>>
>>>
>>> El lun, 13 ene 2025 a las 16:45, Russell Jurney (<
>>> russell.jurney@gmail.com>) escribió:
>>>
>>>> Merged, thanks guys!
>>>>
>>>> Russ
>>>>
>>>> On Sun, Jan 12, 2025 at 2:23 PM Bjørn Jørgensen <
>>>> bjornjorgensen@gmail.com> wrote:
>>>>
>>>>> I tested stuff in this PR
>>>>> https://github.com/graphframes/graphframes/pull/477 and then I made
>>>>> this PR https://github.com/graphframes/graphframes/pull/478
>>>>>
>>>>> søn. 12. jan. 2025 kl. 23:10 skrev Ángel <
>>>>> angel.alvarez.pascua@gmail.com>:
>>>>>
>>>>>> Hi Russell,
>>>>>>
>>>>>> I've just got the OOM error during Test 13. I'm running it from
>>>>>> IntelliJ on Windows with Java 11.
>>>>>>
>>>>>> [image: image.png]
>>>>>> I'll look into it over the course of the next week.
>>>>>>
>>>>>> Regards,
>>>>>> Ángel
>>>>>>
>>>>>> El sáb, 11 ene 2025 a las 9:23, Russell Jurney (<
>>>>>> russell.jurney@gmail.com>) escribió:
>>>>>>
>>>>>>> Friends of GraphFrames (github.com/graphframes/graphframes), I have
>>>>>>> a question for you...
>>>>>>>
>>>>>>> I can't get the unit test 'two components and two dangling
>>>>>>> vertices' in the org.graphframes.lib.ConnectedComponentsSuite
>>>>>>> <https://github.com/graphframes/graphframes/blob/649094caf58cfda0eea3e8cd66785aa38104d771/src/test/scala/org/graphframes/lib/ConnectedComponentsSuite.scala#L138-L148>
>>>>>>> to pass. It fails with an 'OutOfMemoryError: Java heap space' error. I am a
>>>>>>> little stuck on completing a docs release with a motif finding
>>>>>>> tutorial <https://github.com/graphframes/graphframes/pull/473> due
>>>>>>> to this issue.
>>>>>>>
>>>>>>> The problem is outlined in this gist:
>>>>>>> https://gist.github.com/rjurney/6abeffbd59c67df5e5243c8f6619b6bf
>>>>>>>
>>>>>>> Can someone else please try this and see if it passes on the master
>>>>>>> branch?
>>>>>>>
>>>>>>> > build/sbt clean compile package test
>>>>>>>
>>>>>>> I've tried giving it lots of RAM just to see if it would help, as
>>>>>>> much as 32g driver and 16g for executors and... it has no effect. The test
>>>>>>> graph is 8 nodes and 6 edges
>>>>>>> <https://gist.github.com/rjurney/6abeffbd59c67df5e5243c8f6619b6bf#file-connectedcomponentsuite-scala-L22-L26>,
>>>>>>> so it shouldn't have a memory problem... yet when it runs, all 24 cores of
>>>>>>> my CPU get used, it spikes as indicated in the image in the gist.
>>>>>>>
>>>>>>> I am running the following setup:
>>>>>>>
>>>>>>> * Ubuntu 20.04 (22.04 in the Docker image)
>>>>>>> * OpenJDK 11 (I also tried 8, same problem)
>>>>>>> * Scala 2.12.20 (I also tried 2.13, same problem)
>>>>>>> * Python 3.11 (I also tried 3.9, same problem)
>>>>>>>
>>>>>>> Or I am running the Dockerfile in the gist
>>>>>>> <https://gist.github.com/rjurney/6abeffbd59c67df5e5243c8f6619b6bf#file-dockerfile>
>>>>>>> .
>>>>>>>
>>>>>>> Any help much appreciated! Thanks
>>>>>>>
>>>>>>> -----------------------------------------------------------------
>>>>>>> Oh, some new community stuff for GraphFrames. Hackathon announced
>>>>>>> next week :)
>>>>>>>
>>>>>>>
>>>>>>> - GraphFrames Mailing List
>>>>>>> <https://groups.google.com/g/graphframes/>: ask questions about
>>>>>>> GraphFrames on our Google Group
>>>>>>> - #graphframes Discord Channel on GraphGeeks
>>>>>>> <https://discord.com/channels/1162999022819225631/1326257052368113674>
>>>>>>>
>>>>>>> Thanks!
>>>>>>> Russell Jurney @rjurney <http://twitter.com/rjurney>
>>>>>>> russell.jurney@gmail.com LI <http://linkedin.com/in/russelljurney>
>>>>>>> FB <http://facebook.com/jurney> datasyndrome.com
>>>>>>>
>>>>>>
>>>>>
>>>>> --
>>>>> Bjørn Jørgensen
>>>>> Vestre Aspehaug 4, 6010 Ålesund
>>>>> <https://www.google.com/maps/search/Vestre+Aspehaug+4,+6010+%C3%85lesund++Norge?entry=gmail&source=g>
>>>>> Norge
>>>>> <https://www.google.com/maps/search/Vestre+Aspehaug+4,+6010+%C3%85lesund++Norge?entry=gmail&source=g>
>>>>>
>>>>> +47 480 94 297
>>>>>
>>>>
Re: GraphFrames' ConnectedComponentSuite test 'two components and two dangling vertices' fails with OutOfMemoryError: Java heap space
Posted by Ángel <an...@gmail.com>.
My only change was increasing 4 to 10 in the spark.sql.shuffle.partitions
setting (still doesn't work for a value lower than 10).
[image: image.png]
I've performed some tests, changed some code ... and I'm starting to grasp
what's behind this OOM, but I'm still working on it .. quite interesting
indeed this issue! (thanks)
According to MAT <https://eclipse.dev/mat/> ... a humongous execution plan
is being created ...
[image: image.png]
El mar, 14 ene 2025 a las 21:52, Russell Jurney (<ru...@gmail.com>)
escribió:
> Can you please share the code? It doesn't seem an ideal solution, but if
> AQE is confused, disabling it makes sense. I can't figure out why a low
> partition count for an 8 node, 6 edge network would require a lot of
> partitions... users may have different numbers... do you suggest we enforce
> some minimum partition count?
>
> On Tue, Jan 14, 2025 at 7:33 AM Ángel <an...@gmail.com>
> wrote:
>
>> Are you sure that temporarily disabling a global setting like AQE is the
>> best approach to fix this issue?
>> I increased the number of shuffle partitions in the Spark session
>> configure in the GraphFrameTestSparkContext.scala from 4 to 10, and the
>> "checkpoint interval" test ran perfectly without throwing an OOM error.
>> Why? No idea, but it worked.
>>
>>
>>
>> El lun, 13 ene 2025 a las 16:45, Russell Jurney (<
>> russell.jurney@gmail.com>) escribió:
>>
>>> Merged, thanks guys!
>>>
>>> Russ
>>>
>>> On Sun, Jan 12, 2025 at 2:23 PM Bjørn Jørgensen <
>>> bjornjorgensen@gmail.com> wrote:
>>>
>>>> I tested stuff in this PR
>>>> https://github.com/graphframes/graphframes/pull/477 and then I made
>>>> this PR https://github.com/graphframes/graphframes/pull/478
>>>>
>>>> søn. 12. jan. 2025 kl. 23:10 skrev Ángel <
>>>> angel.alvarez.pascua@gmail.com>:
>>>>
>>>>> Hi Russell,
>>>>>
>>>>> I've just got the OOM error during Test 13. I'm running it from
>>>>> IntelliJ on Windows with Java 11.
>>>>>
>>>>> [image: image.png]
>>>>> I'll look into it over the course of the next week.
>>>>>
>>>>> Regards,
>>>>> Ángel
>>>>>
>>>>> El sáb, 11 ene 2025 a las 9:23, Russell Jurney (<
>>>>> russell.jurney@gmail.com>) escribió:
>>>>>
>>>>>> Friends of GraphFrames (github.com/graphframes/graphframes), I have
>>>>>> a question for you...
>>>>>>
>>>>>> I can't get the unit test 'two components and two dangling vertices'
>>>>>> in the org.graphframes.lib.ConnectedComponentsSuite
>>>>>> <https://github.com/graphframes/graphframes/blob/649094caf58cfda0eea3e8cd66785aa38104d771/src/test/scala/org/graphframes/lib/ConnectedComponentsSuite.scala#L138-L148>
>>>>>> to pass. It fails with an 'OutOfMemoryError: Java heap space' error. I am a
>>>>>> little stuck on completing a docs release with a motif finding
>>>>>> tutorial <https://github.com/graphframes/graphframes/pull/473> due
>>>>>> to this issue.
>>>>>>
>>>>>> The problem is outlined in this gist:
>>>>>> https://gist.github.com/rjurney/6abeffbd59c67df5e5243c8f6619b6bf
>>>>>>
>>>>>> Can someone else please try this and see if it passes on the master
>>>>>> branch?
>>>>>>
>>>>>> > build/sbt clean compile package test
>>>>>>
>>>>>> I've tried giving it lots of RAM just to see if it would help, as
>>>>>> much as 32g driver and 16g for executors and... it has no effect. The test
>>>>>> graph is 8 nodes and 6 edges
>>>>>> <https://gist.github.com/rjurney/6abeffbd59c67df5e5243c8f6619b6bf#file-connectedcomponentsuite-scala-L22-L26>,
>>>>>> so it shouldn't have a memory problem... yet when it runs, all 24 cores of
>>>>>> my CPU get used, it spikes as indicated in the image in the gist.
>>>>>>
>>>>>> I am running the following setup:
>>>>>>
>>>>>> * Ubuntu 20.04 (22.04 in the Docker image)
>>>>>> * OpenJDK 11 (I also tried 8, same problem)
>>>>>> * Scala 2.12.20 (I also tried 2.13, same problem)
>>>>>> * Python 3.11 (I also tried 3.9, same problem)
>>>>>>
>>>>>> Or I am running the Dockerfile in the gist
>>>>>> <https://gist.github.com/rjurney/6abeffbd59c67df5e5243c8f6619b6bf#file-dockerfile>
>>>>>> .
>>>>>>
>>>>>> Any help much appreciated! Thanks
>>>>>>
>>>>>> -----------------------------------------------------------------
>>>>>> Oh, some new community stuff for GraphFrames. Hackathon announced
>>>>>> next week :)
>>>>>>
>>>>>>
>>>>>> - GraphFrames Mailing List
>>>>>> <https://groups.google.com/g/graphframes/>: ask questions about
>>>>>> GraphFrames on our Google Group
>>>>>> - #graphframes Discord Channel on GraphGeeks
>>>>>> <https://discord.com/channels/1162999022819225631/1326257052368113674>
>>>>>>
>>>>>> Thanks!
>>>>>> Russell Jurney @rjurney <http://twitter.com/rjurney>
>>>>>> russell.jurney@gmail.com LI <http://linkedin.com/in/russelljurney> FB
>>>>>> <http://facebook.com/jurney> datasyndrome.com
>>>>>>
>>>>>
>>>>
>>>> --
>>>> Bjørn Jørgensen
>>>> Vestre Aspehaug 4, 6010 Ålesund
>>>> <https://www.google.com/maps/search/Vestre+Aspehaug+4,+6010+%C3%85lesund++Norge?entry=gmail&source=g>
>>>> Norge
>>>> <https://www.google.com/maps/search/Vestre+Aspehaug+4,+6010+%C3%85lesund++Norge?entry=gmail&source=g>
>>>>
>>>> +47 480 94 297
>>>>
>>>
Re: GraphFrames' ConnectedComponentSuite test 'two components and two dangling vertices' fails with OutOfMemoryError: Java heap space
Posted by Russell Jurney <ru...@gmail.com>.
Can you please share the code? It doesn't seem an ideal solution, but if
AQE is confused, disabling it makes sense. I can't figure out why a low
partition count for an 8 node, 6 edge network would require a lot of
partitions... users may have different numbers... do you suggest we enforce
some minimum partition count?
On Tue, Jan 14, 2025 at 7:33 AM Ángel <an...@gmail.com>
wrote:
> Are you sure that temporarily disabling a global setting like AQE is the
> best approach to fix this issue?
> I increased the number of shuffle partitions in the Spark session
> configure in the GraphFrameTestSparkContext.scala from 4 to 10, and the
> "checkpoint interval" test ran perfectly without throwing an OOM error.
> Why? No idea, but it worked.
>
>
>
> El lun, 13 ene 2025 a las 16:45, Russell Jurney (<ru...@gmail.com>)
> escribió:
>
>> Merged, thanks guys!
>>
>> Russ
>>
>> On Sun, Jan 12, 2025 at 2:23 PM Bjørn Jørgensen <bj...@gmail.com>
>> wrote:
>>
>>> I tested stuff in this PR
>>> https://github.com/graphframes/graphframes/pull/477 and then I made
>>> this PR https://github.com/graphframes/graphframes/pull/478
>>>
>>> søn. 12. jan. 2025 kl. 23:10 skrev Ángel <angel.alvarez.pascua@gmail.com
>>> >:
>>>
>>>> Hi Russell,
>>>>
>>>> I've just got the OOM error during Test 13. I'm running it from
>>>> IntelliJ on Windows with Java 11.
>>>>
>>>> [image: image.png]
>>>> I'll look into it over the course of the next week.
>>>>
>>>> Regards,
>>>> Ángel
>>>>
>>>> El sáb, 11 ene 2025 a las 9:23, Russell Jurney (<
>>>> russell.jurney@gmail.com>) escribió:
>>>>
>>>>> Friends of GraphFrames (github.com/graphframes/graphframes), I have a
>>>>> question for you...
>>>>>
>>>>> I can't get the unit test 'two components and two dangling vertices'
>>>>> in the org.graphframes.lib.ConnectedComponentsSuite
>>>>> <https://github.com/graphframes/graphframes/blob/649094caf58cfda0eea3e8cd66785aa38104d771/src/test/scala/org/graphframes/lib/ConnectedComponentsSuite.scala#L138-L148>
>>>>> to pass. It fails with an 'OutOfMemoryError: Java heap space' error. I am a
>>>>> little stuck on completing a docs release with a motif finding
>>>>> tutorial <https://github.com/graphframes/graphframes/pull/473> due to
>>>>> this issue.
>>>>>
>>>>> The problem is outlined in this gist:
>>>>> https://gist.github.com/rjurney/6abeffbd59c67df5e5243c8f6619b6bf
>>>>>
>>>>> Can someone else please try this and see if it passes on the master
>>>>> branch?
>>>>>
>>>>> > build/sbt clean compile package test
>>>>>
>>>>> I've tried giving it lots of RAM just to see if it would help, as much
>>>>> as 32g driver and 16g for executors and... it has no effect. The test graph
>>>>> is 8 nodes and 6 edges
>>>>> <https://gist.github.com/rjurney/6abeffbd59c67df5e5243c8f6619b6bf#file-connectedcomponentsuite-scala-L22-L26>,
>>>>> so it shouldn't have a memory problem... yet when it runs, all 24 cores of
>>>>> my CPU get used, it spikes as indicated in the image in the gist.
>>>>>
>>>>> I am running the following setup:
>>>>>
>>>>> * Ubuntu 20.04 (22.04 in the Docker image)
>>>>> * OpenJDK 11 (I also tried 8, same problem)
>>>>> * Scala 2.12.20 (I also tried 2.13, same problem)
>>>>> * Python 3.11 (I also tried 3.9, same problem)
>>>>>
>>>>> Or I am running the Dockerfile in the gist
>>>>> <https://gist.github.com/rjurney/6abeffbd59c67df5e5243c8f6619b6bf#file-dockerfile>
>>>>> .
>>>>>
>>>>> Any help much appreciated! Thanks
>>>>>
>>>>> -----------------------------------------------------------------
>>>>> Oh, some new community stuff for GraphFrames. Hackathon announced next
>>>>> week :)
>>>>>
>>>>>
>>>>> - GraphFrames Mailing List
>>>>> <https://groups.google.com/g/graphframes/>: ask questions about
>>>>> GraphFrames on our Google Group
>>>>> - #graphframes Discord Channel on GraphGeeks
>>>>> <https://discord.com/channels/1162999022819225631/1326257052368113674>
>>>>>
>>>>> Thanks!
>>>>> Russell Jurney @rjurney <http://twitter.com/rjurney>
>>>>> russell.jurney@gmail.com LI <http://linkedin.com/in/russelljurney> FB
>>>>> <http://facebook.com/jurney> datasyndrome.com
>>>>>
>>>>
>>>
>>> --
>>> Bjørn Jørgensen
>>> Vestre Aspehaug 4, 6010 Ålesund
>>> <https://www.google.com/maps/search/Vestre+Aspehaug+4,+6010+%C3%85lesund++Norge?entry=gmail&source=g>
>>> Norge
>>> <https://www.google.com/maps/search/Vestre+Aspehaug+4,+6010+%C3%85lesund++Norge?entry=gmail&source=g>
>>>
>>> +47 480 94 297
>>>
>>
Re: GraphFrames' ConnectedComponentSuite test 'two components and two dangling vertices' fails with OutOfMemoryError: Java heap space
Posted by Ángel <an...@gmail.com>.
Are you sure that temporarily disabling a global setting like AQE is the
best approach to fix this issue?
I increased the number of shuffle partitions in the Spark session configure
in the GraphFrameTestSparkContext.scala from 4 to 10, and the "checkpoint
interval" test ran perfectly without throwing an OOM error. Why? No idea,
but it worked.
El lun, 13 ene 2025 a las 16:45, Russell Jurney (<ru...@gmail.com>)
escribió:
> Merged, thanks guys!
>
> Russ
>
> On Sun, Jan 12, 2025 at 2:23 PM Bjørn Jørgensen <bj...@gmail.com>
> wrote:
>
>> I tested stuff in this PR
>> https://github.com/graphframes/graphframes/pull/477 and then I made this
>> PR https://github.com/graphframes/graphframes/pull/478
>>
>> søn. 12. jan. 2025 kl. 23:10 skrev Ángel <angel.alvarez.pascua@gmail.com
>> >:
>>
>>> Hi Russell,
>>>
>>> I've just got the OOM error during Test 13. I'm running it from IntelliJ
>>> on Windows with Java 11.
>>>
>>> [image: image.png]
>>> I'll look into it over the course of the next week.
>>>
>>> Regards,
>>> Ángel
>>>
>>> El sáb, 11 ene 2025 a las 9:23, Russell Jurney (<
>>> russell.jurney@gmail.com>) escribió:
>>>
>>>> Friends of GraphFrames (github.com/graphframes/graphframes), I have a
>>>> question for you...
>>>>
>>>> I can't get the unit test 'two components and two dangling vertices'
>>>> in the org.graphframes.lib.ConnectedComponentsSuite
>>>> <https://github.com/graphframes/graphframes/blob/649094caf58cfda0eea3e8cd66785aa38104d771/src/test/scala/org/graphframes/lib/ConnectedComponentsSuite.scala#L138-L148>
>>>> to pass. It fails with an 'OutOfMemoryError: Java heap space' error. I am a
>>>> little stuck on completing a docs release with a motif finding tutorial
>>>> <https://github.com/graphframes/graphframes/pull/473> due to this
>>>> issue.
>>>>
>>>> The problem is outlined in this gist:
>>>> https://gist.github.com/rjurney/6abeffbd59c67df5e5243c8f6619b6bf
>>>>
>>>> Can someone else please try this and see if it passes on the master
>>>> branch?
>>>>
>>>> > build/sbt clean compile package test
>>>>
>>>> I've tried giving it lots of RAM just to see if it would help, as much
>>>> as 32g driver and 16g for executors and... it has no effect. The test graph
>>>> is 8 nodes and 6 edges
>>>> <https://gist.github.com/rjurney/6abeffbd59c67df5e5243c8f6619b6bf#file-connectedcomponentsuite-scala-L22-L26>,
>>>> so it shouldn't have a memory problem... yet when it runs, all 24 cores of
>>>> my CPU get used, it spikes as indicated in the image in the gist.
>>>>
>>>> I am running the following setup:
>>>>
>>>> * Ubuntu 20.04 (22.04 in the Docker image)
>>>> * OpenJDK 11 (I also tried 8, same problem)
>>>> * Scala 2.12.20 (I also tried 2.13, same problem)
>>>> * Python 3.11 (I also tried 3.9, same problem)
>>>>
>>>> Or I am running the Dockerfile in the gist
>>>> <https://gist.github.com/rjurney/6abeffbd59c67df5e5243c8f6619b6bf#file-dockerfile>
>>>> .
>>>>
>>>> Any help much appreciated! Thanks
>>>>
>>>> -----------------------------------------------------------------
>>>> Oh, some new community stuff for GraphFrames. Hackathon announced next
>>>> week :)
>>>>
>>>>
>>>> - GraphFrames Mailing List
>>>> <https://groups.google.com/g/graphframes/>: ask questions about
>>>> GraphFrames on our Google Group
>>>> - #graphframes Discord Channel on GraphGeeks
>>>> <https://discord.com/channels/1162999022819225631/1326257052368113674>
>>>>
>>>> Thanks!
>>>> Russell Jurney @rjurney <http://twitter.com/rjurney>
>>>> russell.jurney@gmail.com LI <http://linkedin.com/in/russelljurney> FB
>>>> <http://facebook.com/jurney> datasyndrome.com
>>>>
>>>
>>
>> --
>> Bjørn Jørgensen
>> Vestre Aspehaug 4, 6010 Ålesund
>> <https://www.google.com/maps/search/Vestre+Aspehaug+4,+6010+%C3%85lesund++Norge?entry=gmail&source=g>
>> Norge
>> <https://www.google.com/maps/search/Vestre+Aspehaug+4,+6010+%C3%85lesund++Norge?entry=gmail&source=g>
>>
>> +47 480 94 297
>>
>
Re: Storing a JDBC-based table in a catalog for direct use in Spark SQL
Posted by Aaron Grubb <aa...@kaden.ai>.
In case anyone comes across this, I managed to accomplish this using the Hive Standalone Metastore, with help from an article [1]. Required
configurations:
spark.sql.warehouse.dir=s3a://<BUCKET>/
spark.sql.catalogImplementation=hive
hive.metastore.uris=thrift://<URL>:9083
This enables me to run the example successfully. I would still be interested in knowing if there's another way to accomplish this,
particularly without an external service.
[1]
https://christiaan-viljoen.medium.com/how-to-deploy-a-minimal-apache-hive-standalone-metastore-to-enable-delta-lake-support-for-trino-on-6b03964882a2
On Mon, 2025-01-13 at 18:49 +0000, Aaron Grubb wrote:
> Hi all,
>
> I'm trying to figure out how to persist a table definition in a catalog that can be used from different sessions. Something along the lines
> of
>
> -------------------
> CREATE TABLE spark_catalog.default.test_table (
> name string
> )
> USING jdbc
> OPTIONS (
> driver 'com.mysql.cj.jdbc.Driver',
> url 'jdbc:mysql://example.com:3306/db',
> user 'user',
> password 'pass',
> query 'SELECT name FROM test_table WHERE type_id = 2'
> )
> --------------------
>
> and then from another session, directly calling
>
> --------------------
> SparkSession.builder.getOrCreate().sql('SELECT * FROM spark_catalog.default.test_table').show()
> --------------------
>
> However, I'm unable to figure out how to accomplish this. I initially tried to define the table in a Nessie catalog but it won't accept
> "USING JDBC". I have also tried setting spark.sql.warehouse.dir to a location in S3 and setting enableHiveSupport() on the session, however
> creating this table under these circumstances only creates an empty directory and the table doesn't show up in the next session. Would I
> need
> to set up a Hive Metastore to accomplish this or what other options do I have?
>
> Thanks,
> Aaron
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: user-unsubscribe@spark.apache.org
>
---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org
Re: Incorrect Results and SIGSEGV on Read with Iceberg + PySpark + Nessie
Posted by Aaron Grubb <aa...@kaden.ai>.
Someone just replied to the bug, it was already known about and will be fixed in the upcoming Iceberg 1.7.2 release.
On Thu, 2025-02-06 at 09:35 +0000, Aaron Grubb wrote:
> Hi all,
>
> I filed a bug with the Iceberg team [1] but I'm not sure that it's 100% specific to Iceberg (I assume it is as data in the related parquet
> file is correct and session.read.parquet always returns correct results) so I figured I would flag it here in case anyone has some insight.
> Currently planning on trying different versions of Iceberg to see if I can get correct results but this is a major blocker to my company's
> infrastructure planning so any help would be appreciated. Maybe it's wiser to try different Java versions or is it possible this is related
> to using aarch64 somehow? All JARs are precompiled versions.
>
> Thanks,
> Aaron
>
> [1] https://github.com/apache/iceberg/issues/12178
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: user-unsubscribe@spark.apache.org
>
---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org
Incorrect Results and SIGSEGV on Read with Iceberg + PySpark + Nessie
Posted by Aaron Grubb <aa...@kaden.ai>.
Hi all,
I filed a bug with the Iceberg team [1] but I'm not sure that it's 100% specific to Iceberg (I assume it is as data in the related parquet
file is correct and session.read.parquet always returns correct results) so I figured I would flag it here in case anyone has some insight.
Currently planning on trying different versions of Iceberg to see if I can get correct results but this is a major blocker to my company's
infrastructure planning so any help would be appreciated. Maybe it's wiser to try different Java versions or is it possible this is related
to using aarch64 somehow? All JARs are precompiled versions.
Thanks,
Aaron
[1] https://github.com/apache/iceberg/issues/12178
---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org
Storing a JDBC-based table in a catalog for direct use in Spark SQL
Posted by Aaron Grubb <aa...@kaden.ai>.
Hi all,
I'm trying to figure out how to persist a table definition in a catalog that can be used from different sessions. Something along the lines
of
-------------------
CREATE TABLE spark_catalog.default.test_table (
name string
)
USING jdbc
OPTIONS (
driver 'com.mysql.cj.jdbc.Driver',
url 'jdbc:mysql://example.com:3306/db',
user 'user',
password 'pass',
query 'SELECT name FROM test_table WHERE type_id = 2'
)
--------------------
and then from another session, directly calling
--------------------
SparkSession.builder.getOrCreate().sql('SELECT * FROM spark_catalog.default.test_table').show()
--------------------
However, I'm unable to figure out how to accomplish this. I initially tried to define the table in a Nessie catalog but it won't accept
"USING JDBC". I have also tried setting spark.sql.warehouse.dir to a location in S3 and setting enableHiveSupport() on the session, however
creating this table under these circumstances only creates an empty directory and the table doesn't show up in the next session. Would I need
to set up a Hive Metastore to accomplish this or what other options do I have?
Thanks,
Aaron
---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org
Re: GraphFrames' ConnectedComponentSuite test 'two components and two dangling vertices' fails with OutOfMemoryError: Java heap space
Posted by Russell Jurney <ru...@gmail.com>.
Merged, thanks guys!
Russ
On Sun, Jan 12, 2025 at 2:23 PM Bjørn Jørgensen <bj...@gmail.com>
wrote:
> I tested stuff in this PR
> https://github.com/graphframes/graphframes/pull/477 and then I made this
> PR https://github.com/graphframes/graphframes/pull/478
>
> søn. 12. jan. 2025 kl. 23:10 skrev Ángel <an...@gmail.com>:
>
>> Hi Russell,
>>
>> I've just got the OOM error during Test 13. I'm running it from IntelliJ
>> on Windows with Java 11.
>>
>> [image: image.png]
>> I'll look into it over the course of the next week.
>>
>> Regards,
>> Ángel
>>
>> El sáb, 11 ene 2025 a las 9:23, Russell Jurney (<ru...@gmail.com>)
>> escribió:
>>
>>> Friends of GraphFrames (github.com/graphframes/graphframes), I have a
>>> question for you...
>>>
>>> I can't get the unit test 'two components and two dangling vertices' in
>>> the org.graphframes.lib.ConnectedComponentsSuite
>>> <https://github.com/graphframes/graphframes/blob/649094caf58cfda0eea3e8cd66785aa38104d771/src/test/scala/org/graphframes/lib/ConnectedComponentsSuite.scala#L138-L148>
>>> to pass. It fails with an 'OutOfMemoryError: Java heap space' error. I am a
>>> little stuck on completing a docs release with a motif finding tutorial
>>> <https://github.com/graphframes/graphframes/pull/473> due to this issue.
>>>
>>> The problem is outlined in this gist:
>>> https://gist.github.com/rjurney/6abeffbd59c67df5e5243c8f6619b6bf
>>>
>>> Can someone else please try this and see if it passes on the master
>>> branch?
>>>
>>> > build/sbt clean compile package test
>>>
>>> I've tried giving it lots of RAM just to see if it would help, as much
>>> as 32g driver and 16g for executors and... it has no effect. The test graph
>>> is 8 nodes and 6 edges
>>> <https://gist.github.com/rjurney/6abeffbd59c67df5e5243c8f6619b6bf#file-connectedcomponentsuite-scala-L22-L26>,
>>> so it shouldn't have a memory problem... yet when it runs, all 24 cores of
>>> my CPU get used, it spikes as indicated in the image in the gist.
>>>
>>> I am running the following setup:
>>>
>>> * Ubuntu 20.04 (22.04 in the Docker image)
>>> * OpenJDK 11 (I also tried 8, same problem)
>>> * Scala 2.12.20 (I also tried 2.13, same problem)
>>> * Python 3.11 (I also tried 3.9, same problem)
>>>
>>> Or I am running the Dockerfile in the gist
>>> <https://gist.github.com/rjurney/6abeffbd59c67df5e5243c8f6619b6bf#file-dockerfile>
>>> .
>>>
>>> Any help much appreciated! Thanks
>>>
>>> -----------------------------------------------------------------
>>> Oh, some new community stuff for GraphFrames. Hackathon announced next
>>> week :)
>>>
>>>
>>> - GraphFrames Mailing List <https://groups.google.com/g/graphframes/>:
>>> ask questions about GraphFrames on our Google Group
>>> - #graphframes Discord Channel on GraphGeeks
>>> <https://discord.com/channels/1162999022819225631/1326257052368113674>
>>>
>>> Thanks!
>>> Russell Jurney @rjurney <http://twitter.com/rjurney>
>>> russell.jurney@gmail.com LI <http://linkedin.com/in/russelljurney> FB
>>> <http://facebook.com/jurney> datasyndrome.com
>>>
>>
>
> --
> Bjørn Jørgensen
> Vestre Aspehaug 4, 6010 Ålesund
> <https://www.google.com/maps/search/Vestre+Aspehaug+4,+6010+%C3%85lesund++Norge?entry=gmail&source=g>
> Norge
> <https://www.google.com/maps/search/Vestre+Aspehaug+4,+6010+%C3%85lesund++Norge?entry=gmail&source=g>
>
> +47 480 94 297
>
Re: GraphFrames' ConnectedComponentSuite test 'two components and two dangling vertices' fails with OutOfMemoryError: Java heap space
Posted by Bjørn Jørgensen <bj...@gmail.com>.
I tested stuff in this PR
https://github.com/graphframes/graphframes/pull/477 and then I made this PR
https://github.com/graphframes/graphframes/pull/478
søn. 12. jan. 2025 kl. 23:10 skrev Ángel <an...@gmail.com>:
> Hi Russell,
>
> I've just got the OOM error during Test 13. I'm running it from IntelliJ
> on Windows with Java 11.
>
> [image: image.png]
> I'll look into it over the course of the next week.
>
> Regards,
> Ángel
>
> El sáb, 11 ene 2025 a las 9:23, Russell Jurney (<ru...@gmail.com>)
> escribió:
>
>> Friends of GraphFrames (github.com/graphframes/graphframes), I have a
>> question for you...
>>
>> I can't get the unit test 'two components and two dangling vertices' in
>> the org.graphframes.lib.ConnectedComponentsSuite
>> <https://github.com/graphframes/graphframes/blob/649094caf58cfda0eea3e8cd66785aa38104d771/src/test/scala/org/graphframes/lib/ConnectedComponentsSuite.scala#L138-L148>
>> to pass. It fails with an 'OutOfMemoryError: Java heap space' error. I am a
>> little stuck on completing a docs release with a motif finding tutorial
>> <https://github.com/graphframes/graphframes/pull/473> due to this issue.
>>
>> The problem is outlined in this gist:
>> https://gist.github.com/rjurney/6abeffbd59c67df5e5243c8f6619b6bf
>>
>> Can someone else please try this and see if it passes on the master
>> branch?
>>
>> > build/sbt clean compile package test
>>
>> I've tried giving it lots of RAM just to see if it would help, as much as
>> 32g driver and 16g for executors and... it has no effect. The test graph is 8
>> nodes and 6 edges
>> <https://gist.github.com/rjurney/6abeffbd59c67df5e5243c8f6619b6bf#file-connectedcomponentsuite-scala-L22-L26>,
>> so it shouldn't have a memory problem... yet when it runs, all 24 cores of
>> my CPU get used, it spikes as indicated in the image in the gist.
>>
>> I am running the following setup:
>>
>> * Ubuntu 20.04 (22.04 in the Docker image)
>> * OpenJDK 11 (I also tried 8, same problem)
>> * Scala 2.12.20 (I also tried 2.13, same problem)
>> * Python 3.11 (I also tried 3.9, same problem)
>>
>> Or I am running the Dockerfile in the gist
>> <https://gist.github.com/rjurney/6abeffbd59c67df5e5243c8f6619b6bf#file-dockerfile>
>> .
>>
>> Any help much appreciated! Thanks
>>
>> -----------------------------------------------------------------
>> Oh, some new community stuff for GraphFrames. Hackathon announced next
>> week :)
>>
>>
>> - GraphFrames Mailing List <https://groups.google.com/g/graphframes/>:
>> ask questions about GraphFrames on our Google Group
>> - #graphframes Discord Channel on GraphGeeks
>> <https://discord.com/channels/1162999022819225631/1326257052368113674>
>>
>> Thanks!
>> Russell Jurney @rjurney <http://twitter.com/rjurney>
>> russell.jurney@gmail.com LI <http://linkedin.com/in/russelljurney> FB
>> <http://facebook.com/jurney> datasyndrome.com
>>
>
--
Bjørn Jørgensen
Vestre Aspehaug 4, 6010 Ålesund
Norge
+47 480 94 297
Re: GraphFrames' ConnectedComponentSuite test 'two components and two dangling vertices' fails with OutOfMemoryError: Java heap space
Posted by Ángel <an...@gmail.com>.
Hi Russell,
I've just got the OOM error during Test 13. I'm running it from IntelliJ on
Windows with Java 11.
[image: image.png]
I'll look into it over the course of the next week.
Regards,
Ángel
El sáb, 11 ene 2025 a las 9:23, Russell Jurney (<ru...@gmail.com>)
escribió:
> Friends of GraphFrames (github.com/graphframes/graphframes), I have a
> question for you...
>
> I can't get the unit test 'two components and two dangling vertices' in
> the org.graphframes.lib.ConnectedComponentsSuite
> <https://github.com/graphframes/graphframes/blob/649094caf58cfda0eea3e8cd66785aa38104d771/src/test/scala/org/graphframes/lib/ConnectedComponentsSuite.scala#L138-L148>
> to pass. It fails with an 'OutOfMemoryError: Java heap space' error. I am a
> little stuck on completing a docs release with a motif finding tutorial
> <https://github.com/graphframes/graphframes/pull/473> due to this issue.
>
> The problem is outlined in this gist:
> https://gist.github.com/rjurney/6abeffbd59c67df5e5243c8f6619b6bf
>
> Can someone else please try this and see if it passes on the master branch?
>
> > build/sbt clean compile package test
>
> I've tried giving it lots of RAM just to see if it would help, as much as
> 32g driver and 16g for executors and... it has no effect. The test graph is 8
> nodes and 6 edges
> <https://gist.github.com/rjurney/6abeffbd59c67df5e5243c8f6619b6bf#file-connectedcomponentsuite-scala-L22-L26>,
> so it shouldn't have a memory problem... yet when it runs, all 24 cores of
> my CPU get used, it spikes as indicated in the image in the gist.
>
> I am running the following setup:
>
> * Ubuntu 20.04 (22.04 in the Docker image)
> * OpenJDK 11 (I also tried 8, same problem)
> * Scala 2.12.20 (I also tried 2.13, same problem)
> * Python 3.11 (I also tried 3.9, same problem)
>
> Or I am running the Dockerfile in the gist
> <https://gist.github.com/rjurney/6abeffbd59c67df5e5243c8f6619b6bf#file-dockerfile>
> .
>
> Any help much appreciated! Thanks
>
> -----------------------------------------------------------------
> Oh, some new community stuff for GraphFrames. Hackathon announced next
> week :)
>
>
> - GraphFrames Mailing List <https://groups.google.com/g/graphframes/>:
> ask questions about GraphFrames on our Google Group
> - #graphframes Discord Channel on GraphGeeks
> <https://discord.com/channels/1162999022819225631/1326257052368113674>
>
> Thanks!
> Russell Jurney @rjurney <http://twitter.com/rjurney>
> russell.jurney@gmail.com LI <http://linkedin.com/in/russelljurney> FB
> <http://facebook.com/jurney> datasyndrome.com
>