You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "kat-grayson (via GitHub)" <gi...@apache.org> on 2023/05/09 10:38:37 UTC
[GitHub] [arrow] kat-grayson opened a new issue, #35508: adding data to tdigest in pyarrow
kat-grayson opened a new issue, #35508:
URL: https://github.com/apache/arrow/issues/35508
### Describe the enhancement requested
Requests for the python implementation of the t-digest algorithm:
1. Currently it is only possible to compute one t-digest from a data array and return a given percentile. There doesn't seem to be an option to continuously 'add' data to the digest, as described in the clustering algorithm 2: https://arxiv.org/pdf/1902.04023.pdf. However I think this is happening under the hood, as reducing the buffer size, I assume, converts it the clustering algorithm variant. Would it be possible to include a a function such as pyarrow.compute.tdigest.add(), where you can add data points incrementally (as in the case of data streaming)?
2. Is it possible to alter the scale function used in the implementation? I'm not sure which scale function you have implemented but it would be nice to have some control over this!
Thanks
### Component(s)
Python
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kinow commented on issue #35508: [C++][Python] Adding data to tdigest in pyarrow
Posted by "kinow (via GitHub)" <gi...@apache.org>.
kinow commented on issue #35508:
URL: https://github.com/apache/arrow/issues/35508#issuecomment-1542123031
So from what I could understand, in the `pyarrow` code;
```py
from pyarrow._compute import function_registry
tdigest' in reg.list_functions() # True
```
And here, where [`_make_global_functions`](https://github.com/apache/arrow/blob/ec29c6ffc3cb1af4db4903d9877b2f0b548a3ad9/python/pyarrow/compute.py#L302C5-L331) is automatically called, the functions are exposed in the Python module.
That's how we have `pyarrow.compute.tdigest` (somewhere else `tdigest` must be added to the registry of functions).
What's actually added is [a wrapper to the actual arrow tdigest function](https://github.com/apache/arrow/blob/ec29c6ffc3cb1af4db4903d9877b2f0b548a3ad9/python/pyarrow/compute.py#L121-L127).
```bash
>>> pac.tdigest.__arrow_compute_function__
{'name': 'tdigest', 'arity': 1, 'options_class': 'TDigestOptions', 'options_required': False}
```
I couldn't find a way to access any instance of TDigest. The namespaces of the C++ classes are within `arrow.internal`. So I guess the Arrow C++ implementation was intentionally hidden from the public, intended to be used internally, and exposed via aggregation functions or via a call in Python that returns a panda array-like object. Does that sound correct @westonpace ?
If that's the case, I guess our best option would be to try to create a bind for the Arrow C++ code ourselves, or use another implementation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] westonpace commented on issue #35508: [C++][Python] Adding data to tdigest in pyarrow
Posted by "westonpace (via GitHub)" <gi...@apache.org>.
westonpace commented on issue #35508:
URL: https://github.com/apache/arrow/issues/35508#issuecomment-1544148978
> My guess, and probably @westonpace would agree with me, is that there is little hope in exposing this to PyArrow.
There's little hope that I'll get to it myself, true. Someone else could always be motivated to work on this however. If anyone wants pointers or a rough sketch I'd be happy to provide more details on possible implementation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++][Python] Adding data to tdigest in pyarrow [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #35508:
URL: https://github.com/apache/arrow/issues/35508#issuecomment-5813080518
This issue has been marked as stale because it has had no activity in the past 365 days. Please remove the stale label or comment below, or this issue will be closed in 14 days. If this improvement is still desired but has no current owner, please add the 'Status: needs champion' label.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] marsupialtail commented on issue #35508: [C++][Python] Adding data to tdigest in pyarrow
Posted by "marsupialtail (via GitHub)" <gi...@apache.org>.
marsupialtail commented on issue #35508:
URL: https://github.com/apache/arrow/issues/35508#issuecomment-1546284622
@kinow I suppose I should add that if you want large scale distributed approximate quantile with t-digest, which is what I suspect you are doing with this API, my library https://github.com/marsupialtail/quokka does this with a Python API. After all that's why I wrote my C++ code
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] westonpace commented on issue #35508: [C++][Python] Adding data to tdigest in pyarrow
Posted by "westonpace (via GitHub)" <gi...@apache.org>.
westonpace commented on issue #35508:
URL: https://github.com/apache/arrow/issues/35508#issuecomment-1540932823
> Is it possible to alter the scale function used in the implementation? I'm not sure which scale function you have implemented but it would be nice to have some control over this!
I don't know enough about tdigest to answer this. Here are the current options:
```
Help on function tdigest in module pyarrow.compute:
tdigest(array, /, q=0.5, *, delta=100, buffer_size=500, skip_nulls=True, min_count=0, options=None, memory_pool=None)
Approximate quantiles of a numeric array with T-Digest algorithm.
By default, 0.5 quantile (median) is returned.
Nulls and NaNs are ignored.
An array of nulls is returned if there is no valid data point.
Parameters
----------
array : Array-like
Argument to compute function.
q : double or sequence of double, default 0.5
Quantiles to approximate. All values must be in [0, 1].
delta : int, default 100
Compression parameter for the T-digest algorithm.
buffer_size : int, default 500
Buffer size for the T-digest algorithm.
skip_nulls : bool, default True
Whether to skip (ignore) nulls in the input.
If False, any null in the input forces the output to null.
min_count : int, default 0
Minimum number of non-null values in the input. If the number
of non-null values is below `min_count`, the output is null.
options : pyarrow.compute.TDigestOptions, optional
Alternative way of passing options.
memory_pool : pyarrow.MemoryPool, optional
If not passed, will allocate memory from the default memory pool.
```
If that isn't enough then you could probably open a separate issue to request a configurable scale.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] marsupialtail commented on issue #35508: adding data to tdigest in pyarrow
Posted by "marsupialtail (via GitHub)" <gi...@apache.org>.
marsupialtail commented on issue #35508:
URL: https://github.com/apache/arrow/issues/35508#issuecomment-1540750019
Hey, if you are not picky about using random python projects, try this:
pip3 install ldbpy
Then:
```
#!/usr/bin/env python3
import pyarrow as pa
import numpy as np
from pyarrow.cffi import ffi
c_schema = ffi.new("struct ArrowSchema*")
schema_ptr = int(ffi.cast("uintptr_t", c_schema))
c_array = ffi.new("struct ArrowArray*")
array_ptr = int(ffi.cast("uintptr_t", c_array))
import polars
lineitem = polars.read_parquet("demo-tpch/lineitem.parquet")
arr = lineitem.to_arrow()["l_tax"].combine_chunks()
arr._export_to_c(array_ptr, schema_ptr)
import ldbpy, time
a = ldbpy.NTDigest(20,100,10000)
start = time.time()
a.batch_add_arrow([array_ptr] * 20, [schema_ptr] * 20)
print(a.quantile(0, 0.5))
print(a.quantile(1, 0.1))
print(time.time() - start)
import pyarrow.compute as pac
start = time.time()
print(pac.tdigest(arr, 0.5))
print(time.time() - start)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] marsupialtail commented on issue #35508: [C++][Python] Adding data to tdigest in pyarrow
Posted by "marsupialtail (via GitHub)" <gi...@apache.org>.
marsupialtail commented on issue #35508:
URL: https://github.com/apache/arrow/issues/35508#issuecomment-1556083301
I'm sorry @kat-grayson I was away for the past two days.
This is understandable -- since the array_ptr is not pointing to a real arrow array. I recommend referencing the implementation here: https://github.com/marsupialtail/quokka/blob/master/pyquokka/datastream.py#L957 or maybe just here: https://github.com/marsupialtail/ldb/blob/master/src/test1.py
The input values are pretty much a direct mapover to the tdigest code that currently lives in arrow c++. I would refer you to that code: just search tdigest in arrow github.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++][Python] Adding data to tdigest in pyarrow [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #35508: [C++][Python] Adding data to tdigest in pyarrow
URL: https://github.com/apache/arrow/issues/35508
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] marsupialtail commented on issue #35508: [C++][Python] Adding data to tdigest in pyarrow
Posted by "marsupialtail (via GitHub)" <gi...@apache.org>.
marsupialtail commented on issue #35508:
URL: https://github.com/apache/arrow/issues/35508#issuecomment-1542671374
@kinow "try to create a bind for the Arrow C++ code ourselves": that is what I did. If you do decide to do that I can give you my current implementation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] cyb70289 commented on issue #35508: [C++][Python] Adding data to tdigest in pyarrow
Posted by "cyb70289 (via GitHub)" <gi...@apache.org>.
cyb70289 commented on issue #35508:
URL: https://github.com/apache/arrow/issues/35508#issuecomment-1562290508
> Is it possible to alter the scale function used in the implementation?
It's currently hard coded in [c++ source](https://github.com/apache/arrow/blob/main/cpp/src/arrow/util/tdigest.cc#L63). It's possible to add new scale functions and export an option to choose one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++][Python] Adding data to tdigest in pyarrow [arrow]
Posted by "alexeykudinkin (via GitHub)" <gi...@apache.org>.
alexeykudinkin commented on issue #35508:
URL: https://github.com/apache/arrow/issues/35508#issuecomment-2654907547
This seems like a no-brainer architecture that would allow multi-stage aggregations to work.
Any plans to add support for it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on issue #35508: [C++][Python] Adding data to tdigest in pyarrow
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on issue #35508:
URL: https://github.com/apache/arrow/issues/35508#issuecomment-1561535770
cc @cyb70289
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++][Python] Adding data to tdigest in pyarrow [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on issue #35508:
URL: https://github.com/apache/arrow/issues/35508#issuecomment-2655895831
@alexeykudinkin Please read the discussion above, and feel free to propose APIs and their implementation if you're interested in this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++][Python] Adding data to tdigest in pyarrow [arrow]
Posted by "gitmodimo (via GitHub)" <gi...@apache.org>.
gitmodimo commented on issue #35508:
URL: https://github.com/apache/arrow/issues/35508#issuecomment-3281051722
I have implemented [C++ part](https://github.com/gitmodimo/arrow/tree/tdigest_map). To fully restore TDigest state, in addition to `mean` and `weight` vectors, values of `count`, `min` and `max` are also needed. Final struct holding mergable tdigest state is:
```
struct_({
field("mean", fixed_size_list(float64(), output_size), false),
field("weight", fixed_size_list(float64(), output_size), false),
field("count", uint64(), false),
field("min", float64(), true),
field("max", float64(), true)
})
```
Where output_size=delta.
@pitrou Can you take a look at it? It is my first kernel written in arrow. IIUC all that is needed for complete feature is python bindings and probably docs update. I am also open to renaming of the functions if you have better ones.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] marsupialtail commented on issue #35508: [C++][Python] Adding data to tdigest in pyarrow
Posted by "marsupialtail (via GitHub)" <gi...@apache.org>.
marsupialtail commented on issue #35508:
URL: https://github.com/apache/arrow/issues/35508#issuecomment-1542824553
Yes: https://github.com/marsupialtail/ldb.
My guess, and probably @westonpace would agree with me, is that there is little hope in exposing this to PyArrow.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++][Python] Adding data to tdigest in pyarrow [arrow]
Posted by "thobai (via GitHub)" <gi...@apache.org>.
thobai commented on issue #35508:
URL: https://github.com/apache/arrow/issues/35508#issuecomment-2506136373
Thanks @kinow and @kat-grayson! I'll have a look at crick. Would be interested in your paper draft. I'd be happy to receive a copy ({firstname}.{lastname}@appian.com)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++][Python] Adding data to tdigest in pyarrow [arrow]
Posted by "thobai (via GitHub)" <gi...@apache.org>.
thobai commented on issue #35508:
URL: https://github.com/apache/arrow/issues/35508#issuecomment-2500034788
@kat-grayson and @kinow: With which solution did you end up? I'm working on a distributed Polars project, which has similar requirements, but they're slightly different. We need to be able produce the tdigests on different workers (map) and then merge them on our head (reduce). We've implemented a Polars plugin called [polars-tdigest](https://github.com/appian/polars-tdigest) for this that wraps around the [tdigest Rust crate](https://github.com/MnO2/t-digest). Yet, the tdigest rust implementation isn't actively maintained and we're having issues with its accuracy, so we are looking for better maintained alternatives. Would be very interested which solution you picked
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] marsupialtail commented on issue #35508: [C++][Python] Adding data to tdigest in pyarrow
Posted by "marsupialtail (via GitHub)" <gi...@apache.org>.
marsupialtail commented on issue #35508:
URL: https://github.com/apache/arrow/issues/35508#issuecomment-1543980710
I just fixed the memory leak btw. I use the C data interface, so I only support certain data types. Of course t-digest itself only supports numerics
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++][Python] Adding data to tdigest in pyarrow [arrow]
Posted by "gitmodimo (via GitHub)" <gi...@apache.org>.
gitmodimo commented on issue #35508:
URL: https://github.com/apache/arrow/issues/35508#issuecomment-3270116720
Hello Everyone. I am willing to contribute in C++ part.
> 1. Currently it is only possible to compute one t-digest from a data array and return a given percentile. There doesn't seem to be an option to continuously 'add' data to the digest,...
I believe what you need can be achieved by introducing 3 additional functions.:
1. tdigest_map - functionally the same as tdigest but instead of quantiles outputs `centroids_vector` as fixed_size_list of length `delta` (with nullable values) each element would be struct{double mean; double weight}
2. tdigest_merge - function that takes vector of `centroids_vector`s and output merged `centroids_vector`
3. tdigest_guantile - function that takes `centroids_vector` and calculates quantiles as tdigest does
> 2\. Is it possible to alter the scale function used in the implementation? I'm not sure which scale function you have implemented but it would be nice to have some control over this!
This should be straight forward to allow selecting between ScalerK0/ScalerK1 implementation. Enabling custom scaler functions is way more complicated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++][Python] Adding data to tdigest in pyarrow [arrow]
Posted by "kinow (via GitHub)" <gi...@apache.org>.
kinow commented on issue #35508:
URL: https://github.com/apache/arrow/issues/35508#issuecomment-2503852529
Hi @thobai in the end we used `dask/crick`, and I haven't worked with tdigest in a while. I pinged a colleague that's responsible for that now, and he may comment here after they read the thread if they have something useful :+1: Cheers
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kinow commented on issue #35508: [C++][Python] Adding data to tdigest in pyarrow
Posted by "kinow (via GitHub)" <gi...@apache.org>.
kinow commented on issue #35508:
URL: https://github.com/apache/arrow/issues/35508#issuecomment-1543644241
> Yes: https://github.com/marsupialtail/ldb.
>
> My guess, and probably @westonpace would agree with me, is that there is little hope in exposing this to PyArrow.
Hi @marsupialtail . Let's wait for the reply rfom @westonpace or another maintainer, but in the meantime I will start looking at your code. Thank you!!!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kinow commented on issue #35508: [C++][Python] Adding data to tdigest in pyarrow
Posted by "kinow (via GitHub)" <gi...@apache.org>.
kinow commented on issue #35508:
URL: https://github.com/apache/arrow/issues/35508#issuecomment-1542717776
> @kinow "try to create a bind for the Arrow C++ code ourselves": that is what I did. If you do decide to do that I can give you my current implementation.
That's be helpful. Is the code public?
Or maybe we could discuss working on a pull request to Arrow too. Since there are users for this use case, maybe the dev team will be willing to expose the tdigest class in the Python bindings.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] westonpace commented on issue #35508: [C++][Python] Adding data to tdigest in pyarrow
Posted by "westonpace (via GitHub)" <gi...@apache.org>.
westonpace commented on issue #35508:
URL: https://github.com/apache/arrow/issues/35508#issuecomment-1540931214
Yes, internally, most aggregate functions are implemented in an incremental map/reduce style. A lot of the pieces are in place to expose this but not everything.
However, C++ changes would be required (I think).
You could choose to do this completely outside of Acero using the function registry directly. You would end up creating something that looks quite a bit like the aggregate node so I'd recommend starting by looking at that and getting familiar.
On the other hand, I think I'd prefer an approach reusing Acero since it already does this. Acero can take streaming (and potentially infinite inputs). This is not very well exposed to pyarrow today (I think you can use an iterator of batches as the input to a scanner). It might be nice to be able to create a push-based pyarrow/acero data source though.
The other problem is that you would need an aggregate node that emitted its results every time a batch arrived. This shouldn't be too hard and could probably be done with a flag to the aggregate node. The only wrinkle I think is that the aggregate node, today, has one state per thread. For this to work you would want to synchronize all the threads on a single state (or run without threads).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kat-grayson commented on issue #35508: [C++][Python] Adding data to tdigest in pyarrow
Posted by "kat-grayson (via GitHub)" <gi...@apache.org>.
kat-grayson commented on issue #35508:
URL: https://github.com/apache/arrow/issues/35508#issuecomment-1552795987
Hi @marsupialtail, I have just tried to use your implementation above using the package `ldbpy` and unfortunately I'm running into some errors. Very simply the lines I tried to run are:
`import pyarrow as pa
import numpy as np
from pyarrow.cffi import ffi
import polars
import ldbpy, time
import pyarrow.compute as pac
c_schema = ffi.new("struct ArrowSchema*")
schema_ptr = int(ffi.cast("uintptr_t", c_schema))
c_array = ffi.new("struct ArrowArray*")
array_ptr = int(ffi.cast("uintptr_t", c_array))
a = ldbpy.NTDigest(20,100,10000)
start = time.time()
a.batch_add_arrow([array_ptr] * 20, [schema_ptr] * 20)`
Running the last line with the function `batch_add_arrow` caused the error: `Assertion failed: arrowSchema->release != nullptr
Error message: arrowSchema was released.
terminate called recursively
Aborted`
Do you know what's happening here? Also some more general questions, the line `a = ldbpy.NTDigest(20,100,10000)`, what do the input values represent?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++][Python] Adding data to tdigest in pyarrow [arrow]
Posted by "kat-grayson (via GitHub)" <gi...@apache.org>.
kat-grayson commented on issue #35508:
URL: https://github.com/apache/arrow/issues/35508#issuecomment-2505772889
Hi @thobai, yes as @kinow said we are using the dask/crick implementation https://github.com/dask/crick after some preliminary accuracy tests and it performed quite well. If you want I can e-mail you a draft of a paper where we perform some analysis on the package's accuracy for some climate variables
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kinow commented on issue #35508: [C++][Python] Adding data to tdigest in pyarrow
Posted by "kinow (via GitHub)" <gi...@apache.org>.
kinow commented on issue #35508:
URL: https://github.com/apache/arrow/issues/35508#issuecomment-1541965615
HI @marsupialtail
>Hey, if you are not picky about using random python projects, try this:
>
>pip3 install ldbpy
I had a look at the project on PyPI, I think you are the maintainer? I couldn't locate the source code, and it doesn't have a sdist file, only wheels. Would you have the link to the project source, please?
Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kinow commented on issue #35508: [C++][Python] Adding data to tdigest in pyarrow
Posted by "kinow (via GitHub)" <gi...@apache.org>.
kinow commented on issue #35508:
URL: https://github.com/apache/arrow/issues/35508#issuecomment-1542061091
Hi @westonpace
I work with @kat-grayson on the same project , but on another component that does not use t-digest directly (so I am a bit lost, still learning the ropes around streaming, t-digest, etc.).
>You could choose to do this completely outside of Acero using the function registry directly. You would end up creating something that looks quite a bit like the aggregate node so I'd recommend starting by looking at that and getting familiar.
A little more context here; our project has already a streaming component, and Apache Arrow is not used anywhere in the project (I think, but could be wrong… large project).
But still a good suggestion to look at the [aggregate node in Acero](https://arrow.apache.org/docs/cpp/streaming_execution.html#aggregate). From a brief look, I **think** the aggregate node [accepts functions](https://github.com/apache/arrow/blob/ec29c6ffc3cb1af4db4903d9877b2f0b548a3ad9/cpp/src/arrow/acero/aggregate_node.cc) to apply to the streaming data. One being an [aggregate function with tdigest](https://github.com/apache/arrow/blob/ec29c6ffc3cb1af4db4903d9877b2f0b548a3ad9/cpp/src/arrow/compute/kernels/aggregate_tdigest.cc#L81).
@kat-grayson I think that aggregate calls `NanAdd` in [the tdigest](https://github.com/apache/arrow/blob/ec29c6ffc3cb1af4db4903d9877b2f0b548a3ad9/cpp/src/arrow/util/tdigest.h#L73). So the C++ code seems to support adding to an existing t-digest.
But I think the Python version was made simpler, without exposing some functions from the C++ t-digest object.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kinow commented on issue #35508: [C++][Python] Adding data to tdigest in pyarrow
Posted by "kinow (via GitHub)" <gi...@apache.org>.
kinow commented on issue #35508:
URL: https://github.com/apache/arrow/issues/35508#issuecomment-1544185758
@westonpace,
> If anyone wants pointers or a rough sketch I'd be happy to provide more details on possible implementation.
Thanks that'd be helpful.
But if we worked on an implementation and created a pull request to apache/arrow exposing the C++ tdigest in the PyArrow Python binding, do you think that would be considered to be included as part of a future release of Arrow and PyArrow?
I'm asking in case others are against doing it for some reason (as it appears to have been a design decision not to expose everything but instead create the wrapper to compute functions with a similar return type and options objects).
If a pull request for that is welcome, @kat-grayson , myself and others involved in the same project could work to prepare this pull request (maybe @marsupialtail would join too with their experience).
Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++][Python] Adding data to tdigest in pyarrow [arrow]
Posted by "gitmodimo (via GitHub)" <gi...@apache.org>.
gitmodimo commented on issue #35508:
URL: https://github.com/apache/arrow/issues/35508#issuecomment-3291256232
I have realized that delta does not necessarily has to be constant throughout map->reduce stages. I added delta as options to tdigest_reduce and tdigest_quantile kernels and changes type to list<double not null>. Now tdigest centroid structs is as follows:
```
struct_({
field("mean", list(field("item", float64(), false)), false),
field("weight", list(field("item", float64(), false)), false),
field("min", float64(), true),
field("max", float64(), true),
field("count", uint64(), false),
})
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org