You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "totoroyyb (via GitHub)" <gi...@apache.org> on 2025/01/03 11:15:55 UTC
[I] Optionally disable data validation [arrow-rs]
totoroyyb opened a new issue, #6933:
URL: https://github.com/apache/arrow-rs/issues/6933
**Which part is this question about**
Regarding the library API usage.
**Describe your question**
I am using high-level API (`FileReader` and `FileDecoder`) to read IPC files via mmap. I have noticed that `validate_data()` in the Array building process ([here](https://github.com/apache/arrow-rs/blob/f7263e253655b2ee613be97f9d00e063444d3df5/arrow-data/src/data.rs#L1918-L1945)) adds significant overhead.
I am targeting an ultra-low-latency scenario. With `validate_data` I got 290ms for reading a 2.2GB IPC file (via mmap), and 3.8ms without `validate_data`, which I tested locally by commenting that out. 3.8ms latency is pretty much identical to c++ arrow implementation I tested, and I suspect c++ codebase didn't do this sanity check (not entirely sure).
The functions for the "unchecked" building are here in the codebase, but they are not accessible from high-level API, where I can easily disable them without creating my own array and everything on top of it.
**I wonder if there is any better way to achieve that?**
**Additional context**
Low latency is critical in my case. Thus, I am trying to avoid any additional overhead (C++ codebase as the baseline, maybe?)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Optionally disable data validation [arrow-rs]
Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on issue #6933:
URL: https://github.com/apache/arrow-rs/issues/6933#issuecomment-2569227683
I think this is a duplicate of https://github.com/apache/arrow-rs/issues/3287
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Optionally disable data validation for arrow-ipc [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb closed issue #6933: Optionally disable data validation for arrow-ipc
URL: https://github.com/apache/arrow-rs/issues/6933
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Optionally disable data validation for arrow-ipc [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #6933:
URL: https://github.com/apache/arrow-rs/issues/6933#issuecomment-2584116584
I moved the relevant content to https://github.com/apache/arrow-rs/issues/3287#issuecomment-2584115139 and so let's close this ticket and continue the conversation there
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org