You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@sedona.apache.org by "nullbutt (via GitHub)" <gi...@apache.org> on 2026/01/09 15:00:00 UTC
[I] DBSCAN performance regression in 1.8.1 [sedona]
nullbutt opened a new issue, #2589:
URL: https://github.com/apache/sedona/issues/2589
Hi, flagging that I've noticed a significant performance drop in DBSCAN following upgrade to 1.8.1. My jobs are taking ~4x the time they took before when using 1.8.0 and tasks seem to have doubled.
Perhaps related to the graphframes upgrade and changes to connected components?
https://github.com/graphframes/graphframes/issues/758
### Perf Comparison
| Metric | Before (GF 0.9.2) | After (GF 0.10.0) |
|--------|-------------------|-------------------|
| Duration | 4m 1s | 15m 53s |
| Spark tasks | 25,310 | 53,188 |
| Shuffle write | 12.3MB | 20.2MB |
### What I've Tried
None of these resolved the issue:
1. **Disabling AQE**
```python
spark.conf.set("spark.sql.adaptive.enabled", "false")
```
Result: Duration improved slightly to ~14 min, but tasks increased to 75K
2. **Setting broadcastThreshold to -1 with AQE enabled** (as recommended in GraphFrames issue)
```python
spark.conf.set("spark.sql.adaptive.enabled", "true")
spark.conf.set("graphframes.connected.components.broadcastThreshold", "-1")
```
Result: No significant improvement
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] DBSCAN performance regression in 1.8.1 [sedona]
Posted by "james-willis (via GitHub)" <gi...@apache.org>.
james-willis commented on issue #2589:
URL: https://github.com/apache/sedona/issues/2589#issuecomment-3737630752
Probably related to https://github.com/graphframes/graphframes/issues/759
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] DBSCAN performance regression in 1.8.1 [sedona]
Posted by "jiayuasu (via GitHub)" <gi...@apache.org>.
jiayuasu commented on issue #2589:
URL: https://github.com/apache/sedona/issues/2589#issuecomment-3765009722
@james-willis do you want to create a PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] DBSCAN performance regression in 1.8.1 [sedona]
Posted by "jiayuasu (via GitHub)" <gi...@apache.org>.
jiayuasu commented on issue #2589:
URL: https://github.com/apache/sedona/issues/2589#issuecomment-3737276724
@james-willis can you please take a look?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] DBSCAN performance regression in 1.8.1 [sedona]
Posted by "jiayuasu (via GitHub)" <gi...@apache.org>.
jiayuasu closed issue #2589: DBSCAN performance regression in 1.8.1
URL: https://github.com/apache/sedona/issues/2589
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] DBSCAN performance regression in 1.8.1 [sedona]
Posted by "james-willis (via GitHub)" <gi...@apache.org>.
james-willis commented on issue #2589:
URL: https://github.com/apache/sedona/issues/2589#issuecomment-3750207753
We've reverted what we believe to be the regressing code changes in graphframes. We have asked the users who reported the original gf bug to test with the snapshot release.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] DBSCAN performance regression in 1.8.1 [sedona]
Posted by "james-willis (via GitHub)" <gi...@apache.org>.
james-willis commented on issue #2589:
URL: https://github.com/apache/sedona/issues/2589#issuecomment-3764092337
@jiayuasu a user confirmed 0.10.1 has foxed the regression. Can we change the version we use in sedona and create a 1.8.2 release?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] DBSCAN performance regression in 1.8.1 [sedona]
Posted by "james-willis (via GitHub)" <gi...@apache.org>.
james-willis commented on issue #2589:
URL: https://github.com/apache/sedona/issues/2589#issuecomment-3765614031
yes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] DBSCAN performance regression in 1.8.1 [sedona]
Posted by "james-willis (via GitHub)" <gi...@apache.org>.
james-willis commented on issue #2589:
URL: https://github.com/apache/sedona/issues/2589#issuecomment-3761511438
I released a 0.10.1 version of graphframes to try to revert this perf regression.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org