You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@sedona.apache.org by "Kontinuation (via GitHub)" <gi...@apache.org> on 2025/09/05 13:41:52 UTC
[PR] Run queries in python benchmarks using only one thread [sedona-db]
Kontinuation opened a new pull request, #24:
URL: https://github.com/apache/sedona-db/pull/24
SedonaDB may run queries using multiple threads so the results are not directly comparable with other engines. See comment https://github.com/apache/sedona-db/pull/10#issuecomment-3257793272 for details. This patch configures SedonaDB and DuckDB to run queries using one single thread.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Run queries in python benchmarks using only one thread [sedona-db]
Posted by "petern48 (via GitHub)" <gi...@apache.org>.
petern48 commented on code in PR #24:
URL: https://github.com/apache/sedona-db/pull/24#discussion_r2411136954
##########
python/sedonadb/python/sedonadb/testing.py:
##########
@@ -302,6 +302,7 @@ def __init__(self):
import sedonadb
self.con = sedonadb.connect()
+ self.con.sql("SET datafusion.execution.target_partitions TO 1")
Review Comment:
> @Kontinuation so we probably should just put the single thread conf to here https://github.com/apache/sedona-db/blob/main/benchmarks/test_bench_base.py?
How about we add a helper function like `force_one_thread()` (I didn't think too hard about the name) to SedonaDB and DuckDB engines so we can isolate this behavior to the benchmarks. At the moment, this would slow down the SedonaDB engine during the tests as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Run queries in python benchmarks using only one thread [sedona-db]
Posted by "Kontinuation (via GitHub)" <gi...@apache.org>.
Kontinuation commented on PR #24:
URL: https://github.com/apache/sedona-db/pull/24#issuecomment-3258576653
I was using my local build of duckdb-spatial using the latest trunk, the duckdb-python is also built using my locally built latest 1.4.1-dev version.
I have tried re-installing duckdb using `pip install duckdb --upgrade --pre` and ran another benchmark (test_st_buffer), the behavior is basically the same:
```
----------------------------------------------------------------------------------- benchmark 'table=collections_complex': 2 tests ----------------------------------------------------------------------------------
Name (time in ms) Min Max Mean StdDev Median IQR Outliers OPS Rounds Iterations
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_st_buffer[collections_complex-SedonaDB] 115.0520 (1.0) 157.6825 (1.0) 128.0455 (1.0) 13.9384 (1.73) 122.8113 (1.0) 10.1471 (1.0) 2;1 7.8097 (1.0) 9 1
test_st_buffer[collections_complex-DuckDB] 611.7650 (5.32) 633.0960 (4.02) 623.6101 (4.87) 8.0625 (1.0) 623.5183 (5.08) 10.8149 (1.07) 2;0 1.6036 (0.21) 5 1
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------- benchmark 'table=collections_simple': 2 tests -----------------------------------------------------------------------------------
Name (time in ms) Min Max Mean StdDev Median IQR Outliers OPS Rounds Iterations
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
test_st_buffer[collections_simple-SedonaDB] 111.6307 (1.0) 128.6885 (1.0) 120.1398 (1.0) 5.7616 (1.0) 121.7260 (1.0) 8.5031 (1.0) 3;0 8.3236 (1.0) 9 1
test_st_buffer[collections_simple-DuckDB] 619.3594 (5.55) 698.7082 (5.43) 655.4287 (5.46) 34.3422 (5.96) 657.1137 (5.40) 60.9305 (7.17) 2;0 1.5257 (0.18) 5 1
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Run queries in python benchmarks using only one thread [sedona-db]
Posted by "petern48 (via GitHub)" <gi...@apache.org>.
petern48 commented on code in PR #24:
URL: https://github.com/apache/sedona-db/pull/24#discussion_r2414077330
##########
benchmarks/README.md:
##########
@@ -79,3 +79,36 @@ test_st_buffer[collections_simple-PostGIS] 855.3329 (9.96) 854.7194 (9.
```
For more details and command line options, refer to the official [pytest-benchmark documentation](https://pytest-benchmark.readthedocs.io/en/latest/usage.html)
+
+### Adding New Benchmarks
+
+There are two categories of benchmarks, each mapped to a pytest marker:
+
+- `@pytest.mark.bench_udf`:
+ Micro / UDF benchmarks that measure the per-function cost (e.g. ST_Area, ST_Contains). These should run engines in a comparable, single-thread style configuration (where possible) to make function-level performance differences clearer.
+- `@pytest.mark.bench_query`:
+ Macro / complex query benchmarks (e.g. KNN joins) that represent perceived end-user performance. Engines run with their default / natural configuration (multi-threading, internal parallelism, etc.).
Review Comment:
I like how you phrased it as "default / natural configuration" + very concise 👍
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Run queries in python benchmarks using only one thread [sedona-db]
Posted by "jiayuasu (via GitHub)" <gi...@apache.org>.
jiayuasu commented on PR #24:
URL: https://github.com/apache/sedona-db/pull/24#issuecomment-3373428757
should we make this default?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Run queries in python benchmarks using only one thread [sedona-db]
Posted by "jiayuasu (via GitHub)" <gi...@apache.org>.
jiayuasu commented on PR #24:
URL: https://github.com/apache/sedona-db/pull/24#issuecomment-3375166067
I think the main purpose of this benchmark is to serve as a unit test — to check whether our scalar function implementation introduces any unreasonable overhead. If we don’t make it default to a single thread, as DuckDB and PostGIS do for small datasets, then the results become largely meaningless.
It doesn’t make sense to showcase SedonaDB’s performance using these numbers. For meaningful performance evaluation, we should instead use SpatialBench, which is designed exactly for that purpose.
So let's merge it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Run queries in python benchmarks using only one thread [sedona-db]
Posted by "jiayuasu (via GitHub)" <gi...@apache.org>.
jiayuasu commented on PR #24:
URL: https://github.com/apache/sedona-db/pull/24#issuecomment-3375181583
@Kontinuation so we probably should just put the single thread conf to here https://github.com/apache/sedona-db/blob/main/benchmarks/test_bench_base.py?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Run queries in python benchmarks using only one thread [sedona-db]
Posted by "Kontinuation (via GitHub)" <gi...@apache.org>.
Kontinuation commented on PR #24:
URL: https://github.com/apache/sedona-db/pull/24#issuecomment-3383905276
> ```python
> class DuckDBSingleThread(DuckDB):
> ```
I have also thought about this approach. My initial thought was that we lean towards using more pytest features to write benchmarks more concisely, and refactor the engine connection to pytest fixtures. I'll switch to your suggested approach since pytest-benchmark is not favored.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Run queries in python benchmarks using only one thread [sedona-db]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on PR #24:
URL: https://github.com/apache/sedona-db/pull/24#issuecomment-3276193755
I can see how there are two types of benchmarks that are equally valuable:
- Integration-style benchmarks that use the defaults and read from Parquet (e.g., check the perceived speed of a relatively realistic query)
- Unit-style benchmarks that are just a way to check if our particular implementation of a scalar function/iteration overhead is reasonable. Running these from memory on one thread (or the same number of threads) is possibly more comparable but forcing a single thread is maybe unrealistic because in practice some of our per-batch and per-item overhead is amortized over multiple threads and is possibly not something we should spend time optimizing yet.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Run queries in python benchmarks using only one thread [sedona-db]
Posted by "petern48 (via GitHub)" <gi...@apache.org>.
petern48 commented on PR #24:
URL: https://github.com/apache/sedona-db/pull/24#issuecomment-3374800284
> I can see how there are two types of benchmarks that are equally valuable:
>
> * Integration-style benchmarks that use the defaults and read from Parquet (e.g., check the perceived speed of a relatively realistic query)
> * Unit-style benchmarks that are just a way to check if our particular implementation of a scalar function/iteration overhead is reasonable. Running these from memory on one thread (or the same number of threads) is possibly more comparable but forcing a single thread is maybe unrealistic because in practice some of our per-batch and per-item overhead is amortized over multiple threads and is possibly not something we should spend time optimizing yet.
I agree with this. Which one of these do we care more about (to be the default)? 1) realistic benchmarks for "bragging" to the public OR 2) benchmarks meant for development and understanding how good our function implementations are.
For 2), sure maybe using one thread would be better for comparing for development purposes. In that case, I'll mention again that pre-loading the parquet data (instead of using a view) would further isolate the comparison to the function implementations.
For 1), I don't think forcing them both to one thread would be more "fair." How well the engines leverage all the threads they have access to is a core part of how "good" they are. Ok here comes my main point:
> I did some experiments and found that it is not related to whether we are using a test/benchmark framework or not. It is related to the size of dataset.
I interpret this to mean that DuckDB decides to use one thread because it thinks this will be optimal compared to using more (correct me if I'm wrong), since multithreading comes with an overhead. If one engine *chooses* to only use one thread when it's given multiple, that shouldn't force everyone else to only use one. It might be right, it might be wrong. If you want perf numbers to showcase publicly, we could increase the dataset size until DuckDB uses as many threads, to kill any potential argument for an "unfair" benchmark.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Run queries in python benchmarks using only one thread [sedona-db]
Posted by "Kontinuation (via GitHub)" <gi...@apache.org>.
Kontinuation commented on PR #24:
URL: https://github.com/apache/sedona-db/pull/24#issuecomment-3258587619
> * We don't benchmark predicates with realistic input (they are ST_Contains with two identical inputs). The array/scalar case is probably best to focus on (more likely to affect the perceived speed of our engine's fist release).
I also found this problem. This will drastically affect the benchmarking results for predicates such as ST_Intersects since some implementations may return early for such cases. I'll fix the benchmarking data by generating uniformly distributed pairs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Run queries in python benchmarks using only one thread [sedona-db]
Posted by "petern48 (via GitHub)" <gi...@apache.org>.
petern48 commented on PR #24:
URL: https://github.com/apache/sedona-db/pull/24#issuecomment-3259069837
> - We don't benchmark predicates with realistic input (they are ST_Contains with two identical inputs). The array/scalar case is probably best to focus on (more likely to affect the perceived speed of our engine's fist release).
Yeah, I knowingly made `geom1` and `geom2` identical [here](https://github.com/apache/sedona-db/blob/8bddfa3ca5916fd42bec1968441cf516ba7fb08b/benchmarks/test_bench_base.py#L73-L79). I was primarily focused on the non-predicate functions, so I just put something together quick for binary functions. I meant to circle back to it later, but ran into other things.
> - They benchmark on a "table" and not a Parquet scan. We have the edge on a Parquet scan, PostGIS and DuckDB have an edge with their native table format. The Parquet scan probably is more realistic.
I knowingly did this too actually. When I started this was just supposed to be a scuffed development tool rather than something we'd use for presenting results publicly. I created these benchmarks with the intention of comparing purely the function implementations (ignoring parquet reading optimizations), since I've been focused on function implementations.
I would still argue using "table" is probably more useful as a developer when it comes to optimizing functions, since it would be a raw 1-to-1 comparison. We could make it configurable (later potentially). For now, It's totally fine if we want to change it to using parquet scans.
Sorry for all of these issues 😬
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Run queries in python benchmarks using only one thread [sedona-db]
Posted by "Kontinuation (via GitHub)" <gi...@apache.org>.
Kontinuation commented on PR #24:
URL: https://github.com/apache/sedona-db/pull/24#issuecomment-3381378443
I have added markers for marking the purposes of benchmarks (`@pytest.mark.bench_udf` and `@pytest.mark.bench_query`). Benchmarks marked by `bench_udf` will run using only one thread; benchmarks marked by `bench_query` will run using the default configuration.
The marker was applied by an auto fixture, which looks quite hacky. A more pytest-native way is to define engines as fixtures and use [markers to pass data to fixtures](https://docs.pytest.org/en/stable/how-to/fixtures.html#using-markers-to-pass-data-to-fixtures) directly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Run queries in python benchmarks using only one thread [sedona-db]
Posted by "Kontinuation (via GitHub)" <gi...@apache.org>.
Kontinuation commented on PR #24:
URL: https://github.com/apache/sedona-db/pull/24#issuecomment-3273209632
> I don't see any issues running these without the benchmark suite:
>
> ```python
> from sedonadb.testing import SedonaDB, DuckDB
>
> sedonadb = SedonaDB()
> duckdb = DuckDB()
>
> random = sedonadb.con.sql("""
> SELECT geometry FROM sd_random_geometry('{
> "geom_type": "Point",
> "target_rows": 10000000,
> "vertices_per_linestring_range": [2, 2]
> }')""")
>
>
> duckdb.create_table_arrow("random", random)
> sedonadb.create_table_arrow("random", random)
>
> %time duckdb.execute_and_collect("SELECT ST_Buffer(geometry, 0.1) FROM random")
> #> CPU times: user 1min 20s, sys: 2.04 s, total: 1min 22s
> #> Wall time: 7.53 s
>
> %time sedonadb.execute_and_collect("SELECT ST_Buffer(geometry, 0.1) FROM random")
> #> CPU times: user 1min 31s, sys: 3.05 s, total: 1min 35s
> #> Wall time: 9.75 s
> ```
>
> Maybe `duckdb` always runs single threaded under `pytest`? Probably it makes sense to roll our own benchmark suite.
I did some experiments and found that it is not related to whether we are using a test/benchmark framework or not. It is related to the size of dataset.
I tried this workload with various `target_rows` configurations when generating the test dataset. Here are the results:
* **target_rows = 100000**: DuckDB uses a single thread, SedonaDB uses 10 threads
* **target_rows = 500000**: DuckDB uses 4 threads, SedonaDB uses 10 threads
* **target_rows = 1000000 or larger**: DuckDB uses 5 threads, SedonaDB uses 10 threads
Generally, we observe less parallelism when the base table contains less data. This is probably related to how DuckDB [estimates thread count](https://github.com/duckdb/duckdb/blob/c11813f1b4e89aa7096b8d18ca9eb608e27168a8/src/execution/physical_operator.cpp#L56-L73). This behavior is also documented [here](https://duckdb.org/docs/stable/guides/performance/how_to_tune_workloads.html#the-effect-of-row-groups-on-parallelism).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Run queries in python benchmarks using only one thread [sedona-db]
Posted by "Kontinuation (via GitHub)" <gi...@apache.org>.
Kontinuation commented on PR #24:
URL: https://github.com/apache/sedona-db/pull/24#issuecomment-3258606787
You can run the `test_st_buffer` benchmark yourself and see if this single-thread behavior happens only on my site.
The benchmark results for ST_Contains represented by Peter seems to be single-threaded performance. I'm unsure if I'm doing anything wrong to achieve this multi-core concurrency behavior when running SedonaDB.
<img width="1816" height="336" alt="image" src="https://github.com/user-attachments/assets/e6c44105-4031-407f-bc23-6833b4ba87de" />
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Run queries in python benchmarks using only one thread [sedona-db]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on PR #24:
URL: https://github.com/apache/sedona-db/pull/24#issuecomment-3277306627
> How does multithreading amortize per-batch and per-item overhead? Could you explain that a bit more?
It might not! I am mostly just skeptical of creating too much of an artificial situation for the sake of fairness (although we can certainly do it for ourselves if it's giving us useful information/finding performance issues in specific benchmarks). Parts of Sedona and Sedona DB also adapt various configuration based on statistics...it's all part of the magic of the engine(s) that we want to measure.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Run queries in python benchmarks using only one thread [sedona-db]
Posted by "petern48 (via GitHub)" <gi...@apache.org>.
petern48 commented on PR #24:
URL: https://github.com/apache/sedona-db/pull/24#issuecomment-3375648214
That sounds good to me. Also double-checking, we're good with (continuing to) pre-load the parquet data instead of using a view? I prefer it this way, so we can ignore any advantages/disadvantages from parquet reading.
The code currently does this here:
https://github.com/apache/sedona-db/blob/e45d6f9a145be216e4fa62408a51aa9b22a0d14e/benchmarks/test_bench_base.py#L128C27-L130
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Run queries in python benchmarks using only one thread [sedona-db]
Posted by "Kontinuation (via GitHub)" <gi...@apache.org>.
Kontinuation merged PR #24:
URL: https://github.com/apache/sedona-db/pull/24
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Run queries in python benchmarks using only one thread [sedona-db]
Posted by "Kontinuation (via GitHub)" <gi...@apache.org>.
Kontinuation commented on PR #24:
URL: https://github.com/apache/sedona-db/pull/24#issuecomment-3375175278
I agree that we should benchmark individual ST functions using one thread (scenario 2), as it is a must for making the benchmark results comparable across multiple engines. We focus on the quality of function implementation rather than the overall performance of the system in this scenario, which helps us catch performance bugs. By the time I authored this PR, all the benchmarks we've written were for benchmarking individual ST functions, that is why I decided to force using single thread.
I also agree that we need to use the default settings of the systems to benchmark for perceived performance. This setting should be used for benchmarking more complex scenarios such as joins, spatial bench, etc. The conclusion is that we'd better use different settings for different types of benchmarks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Run queries in python benchmarks using only one thread [sedona-db]
Posted by "petern48 (via GitHub)" <gi...@apache.org>.
petern48 commented on PR #24:
URL: https://github.com/apache/sedona-db/pull/24#issuecomment-3377530343
We're on the same page. I primarily meant that there's no IO during the query, don't worry about me saying "instead of a view."
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Run queries in python benchmarks using only one thread [sedona-db]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on PR #24:
URL: https://github.com/apache/sedona-db/pull/24#issuecomment-3259089548
I don't see any issues running these without the benchmark suite:
```python
from sedonadb.testing import SedonaDB, DuckDB
sedonadb = SedonaDB()
duckdb = DuckDB()
random = sedonadb.con.sql("""
SELECT geometry FROM sd_random_geometry('{
"geom_type": "Point",
"target_rows": 10000000,
"vertices_per_linestring_range": [2, 2]
}')""")
duckdb.create_table_arrow("random", random)
sedonadb.create_table_arrow("random", random)
%time duckdb.execute_and_collect("SELECT ST_Buffer(geometry, 0.1) FROM random")
#> CPU times: user 1min 20s, sys: 2.04 s, total: 1min 22s
#> Wall time: 7.53 s
%time sedonadb.execute_and_collect("SELECT ST_Buffer(geometry, 0.1) FROM random")
#> CPU times: user 1min 31s, sys: 3.05 s, total: 1min 35s
#> Wall time: 9.75 s
```
Maybe `duckdb` always runs single threaded under `pytest`? Probably it makes sense to roll our own benchmark suite.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Run queries in python benchmarks using only one thread [sedona-db]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on PR #24:
URL: https://github.com/apache/sedona-db/pull/24#issuecomment-3258486907
I haven't run into DuckDB running stuff on a single thread before although I have dealt with wildly different timings with the CLI vs Python. It's worth checking something other than ST_Contains, too, and also ensuring we're benchmarking against forthcoming DuckDB (`pip install duckdb --upgrade --pre`) since these will all change in a week.
Probably the benchmark we want to focus on is the "user-facing default", since that is how people will perceive the speed of our engine? We might want to consider tuning the PostGIS instance to be smarter since the defaults are very bad for geometry (the last time I did this was https://dewey.dunnington.ca/post/2024/wrangling-and-joining-130m-points-with-duckdb--the-open-source-spatial-stack/#postgis ).
Some other issues with the current benchmarks:
- They benchmark on a "table" and not a Parquet scan. We have the edge on a Parquet scan, PostGIS and DuckDB have an edge with their native table format. The Parquet scan probably is more realistic.
- We don't benchmark predicates with realistic input (they are ST_Contains with two identical inputs). The array/scalar case is probably best to focus on (more likely to affect the perceived speed of our engine's fist release).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Run queries in python benchmarks using only one thread [sedona-db]
Posted by "jiayuasu (via GitHub)" <gi...@apache.org>.
jiayuasu commented on PR #24:
URL: https://github.com/apache/sedona-db/pull/24#issuecomment-3259538220
+1 to add Parquet scan. We can have both or default to compare against Parquet scan. Internal storage format always have huge impact on performance. We are targeting at the lakehouse architecture which use open format like parquet for storage
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Run queries in python benchmarks using only one thread [sedona-db]
Posted by "jiayuasu (via GitHub)" <gi...@apache.org>.
jiayuasu commented on PR #24:
URL: https://github.com/apache/sedona-db/pull/24#issuecomment-3273340239
Interesting. I wonder if this also happens if we run spatial join against DuckDB.
This estimation gives us non-deterministic performance. But looks like there is a way to force DuckDB to use a fixed number of threads as well?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Run queries in python benchmarks using only one thread [sedona-db]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on PR #24:
URL: https://github.com/apache/sedona-db/pull/24#issuecomment-3258723379
When I run
```python
import duckdb
duckdb.load_extension("spatial")
duckdb.sql(
"SELECT count(*) FROM 'submodules/geoarrow-data/microsoft-buildings/files/microsoft-buildings_point_geo.parquet' WHERE ST_Intersects(geometry, geometry)"
).to_arrow_table()
```
...I see Activity Manager reporting 900% CPU usage (query runs in 30s). I'll check the benchmark too in a bit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Run queries in python benchmarks using only one thread [sedona-db]
Posted by "Kontinuation (via GitHub)" <gi...@apache.org>.
Kontinuation commented on PR #24:
URL: https://github.com/apache/sedona-db/pull/24#issuecomment-3277149127
> I wonder if this also happens if we run spatial join against DuckDB.
I have investigated the performance characteristics using a profiler, both are using all available processors when running the spatial join benchmarks. But this could be a problem if we run benchmarks in a different environment, or using a different dataset. We should always understand the reasons behind those numbers produced by benchmarks
> forcing a single thread is maybe unrealistic because in practice some of our per-batch and per-item overhead is amortized over multiple threads
How does multithreading amortize per-batch and per-item overhead? Could you explain that a bit more?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Run queries in python benchmarks using only one thread [sedona-db]
Posted by "petern48 (via GitHub)" <gi...@apache.org>.
petern48 commented on code in PR #24:
URL: https://github.com/apache/sedona-db/pull/24#discussion_r2411136954
##########
python/sedonadb/python/sedonadb/testing.py:
##########
@@ -302,6 +302,7 @@ def __init__(self):
import sedonadb
self.con = sedonadb.connect()
+ self.con.sql("SET datafusion.execution.target_partitions TO 1")
Review Comment:
> @Kontinuation so we probably should just put the single thread conf to here https://github.com/apache/sedona-db/blob/main/benchmarks/test_bench_base.py?
How about we add a helper function like `force_one_thread()` (I didn't think too hard about the name) to SedonaDB and DuckDB engines so we can isolate this behavior to the benchmarks. At the moment, this would slow down the SedonaDB engine during the tests as well.
Or we add a parameter to the constructor
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Run queries in python benchmarks using only one thread [sedona-db]
Posted by "Kontinuation (via GitHub)" <gi...@apache.org>.
Kontinuation commented on PR #24:
URL: https://github.com/apache/sedona-db/pull/24#issuecomment-3375720563
Using views should be fine. The current approach has already `execute_and_collect`ed the data as in-memory arrow tables before creating the views, so the views should refer to in-memory batches as well. There's no IO involved when scanning these views.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Run queries in python benchmarks using only one thread [sedona-db]
Posted by "petern48 (via GitHub)" <gi...@apache.org>.
petern48 commented on code in PR #24:
URL: https://github.com/apache/sedona-db/pull/24#discussion_r2414077330
##########
benchmarks/README.md:
##########
@@ -79,3 +79,36 @@ test_st_buffer[collections_simple-PostGIS] 855.3329 (9.96) 854.7194 (9.
```
For more details and command line options, refer to the official [pytest-benchmark documentation](https://pytest-benchmark.readthedocs.io/en/latest/usage.html)
+
+### Adding New Benchmarks
+
+There are two categories of benchmarks, each mapped to a pytest marker:
+
+- `@pytest.mark.bench_udf`:
+ Micro / UDF benchmarks that measure the per-function cost (e.g. ST_Area, ST_Contains). These should run engines in a comparable, single-thread style configuration (where possible) to make function-level performance differences clearer.
+- `@pytest.mark.bench_query`:
+ Macro / complex query benchmarks (e.g. KNN joins) that represent perceived end-user performance. Engines run with their default / natural configuration (multi-threading, internal parallelism, etc.).
Review Comment:
I like how you phrased it + very concise 👍
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@sedona.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org