You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "alamb (via GitHub)" <gi...@apache.org> on 2024/01/27 09:39:41 UTC
[PR] [WIP] Add timeseries / observability benchmarks [arrow-datafusion]
alamb opened a new pull request, #9017:
URL: https://github.com/apache/arrow-datafusion/pull/9017
## Which issue does this PR close?
Closes https://github.com/apache/arrow-datafusion/issues/8791
## Rationale for this change
There are usecases for several DataFusion users (like IOx) that store observability data, that is often characterized by low cardinality string data encoded as dictionaries. While the current parquet_filter pushdown benchmarks (TODO LINK) cover this example, we don't have an end to end test that does.
This has caused problems when have made changes such as https://github.com/apache/arrow-datafusion/issues/7647 that should improve the performance of these queries but we had no reproducible way to measure the impact, and couldn't evaluate if the change was beneficial enough to warrant additional code complexity
There in systems such as IOx the data is very often sorted and the sort order is quite important for performance. However, DataFusion's existing benchmark coverage does not have any pre-sorted data
## What changes are included in this PR?
1. Add a datafusion specific data set to to model common patterns in timeseries data -- http access logs / metrics and tracing data specifically. This uses the same generator as used in several other parts of DataFusion
2. Add a XXX benchmark to dfbench, runnable by `bench.sh` along with several queries
## Are these changes tested?
All tests
## Are there any user-facing changes?
No
## TODO
- [ ] add ticket / extend to model logging data as well
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [WIP] Add timeseries / observability benchmarks [datafusion]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #9017: [WIP] Add timeseries / observability benchmarks
URL: https://github.com/apache/datafusion/pull/9017
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org
Re: [PR] [WIP] Add timeseries / observability benchmarks [arrow-datafusion]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #9017:
URL: https://github.com/apache/arrow-datafusion/pull/9017#issuecomment-2052916447
Thank you for your contribution. Unfortunately, this pull request is stale because it has been open 60 days with no activity. Please remove the stale label or comment or this will be closed in 7 days.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org