[GitHub] [arrow-datafusion] Ted-Jiang opened a new issue, #3806: Change tdigest centroids number for better accuracy

[GitBox <gi...@apache.org>]

Ted-Jiang opened a new issue, #3806:
URL: https://github.com/apache/arrow-datafusion/issues/3806

   **Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
   We have a lot of use in approx_percentile in our product. which base on `tdigest`,  we facing large amount data we face the issue which loss a lot accuracy.
   In the paper,  centroids number means  how many sample data in your  `tdigest`, so i think we should increase centroids number default size.
   
   like in clickhouse  set to 2048 
   
   https://github.com/ClickHouse/ClickHouse/blob/927af1255adb37ace1b95cc3ec4316553b4cb4b4/src/AggregateFunctions/QuantileTDigest.h#L71-L87
   
   in spark and hive set to 10000 . https://github.com/apache/hive/blob/7b3ecf617a6d46f48a3b6f77e0339fd4ad95a420/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFPercentileApprox.java#L58-L59
   
   **Describe the solution you'd like**
   Change DEFAULT_MAX_SIZE to 10000
   
   **Describe alternatives you've considered**
   A clear and concise description of any alternative solutions or features you've considered.
   
   **Additional context**
   Add any other context or screenshots about the feature request here.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org