You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "TrevorADHD (via GitHub)" <gi...@apache.org> on 2025/09/18 09:14:15 UTC
[PR] fix: reset the offset of 'file_for_view' [arrow-rs]
TrevorADHD opened a new pull request, #8381:
URL: https://github.com/apache/arrow-rs/pull/8381
# Which issue does this PR close?
- Closes #8380
# Rationale for this change
It will fix the example to help user get the result of utf8view performance.
# What changes are included in this PR?
Reset the file handle's offset before using it in next time.
# Are these changes tested?
```
arrow-avro/examples/read_with_utf8view.rs
```
The example can run successfully
```shell
(base) ➜ arrow-avro git:(fix/example_of_utf8view) cargo run --package arrow-avro --example read_with_utf8view -- test/data/nested_record_reuse.avro
Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.30s
Running `/Users/trevor.wang/Workspace/rust/fix-arrow-rs/arrow-rs/target/debug/examples/read_with_utf8view test/data/nested_record_reuse.avro`
Read 2 rows from test/data/nested_record_reuse.avro
Reading with StringArray: 2.095417ms
Reading with StringViewArray: 179.333µs
StringViewArray was 11.68x faster
```
# Are there any user-facing changes?
Every user will get the right result and run the example to get the perfomrance data.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: reset the offset of 'file_for_view' [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #8381:
URL: https://github.com/apache/arrow-rs/pull/8381#issuecomment-3313432080
Thank you @TrevorADHD @jecsand838 and @mbrobbel
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: reset the offset of 'file_for_view' [arrow-rs]
Posted by "TrevorADHD (via GitHub)" <gi...@apache.org>.
TrevorADHD commented on code in PR #8381:
URL: https://github.com/apache/arrow-rs/pull/8381#discussion_r2359539208
##########
arrow-avro/examples/read_with_utf8view.rs:
##########
@@ -48,6 +48,7 @@ fn main() -> Result<(), Box<dyn std::error::Error>> {
let batches: Vec<RecordBatch> = avro_reader.collect::<Result<_, _>>()?;
let regular_duration = start.elapsed();
+ file_for_view.seek(SeekFrom::Start(0))?;
Review Comment:
In my personal opinion, no need to modify the reader's logic. User who clone the file handle should know, realize and conscious control the file handle. If reader doesn't have to keep a mut file handle.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: reset the offset of 'file_for_view' [arrow-rs]
Posted by "TrevorADHD (via GitHub)" <gi...@apache.org>.
TrevorADHD commented on code in PR #8381:
URL: https://github.com/apache/arrow-rs/pull/8381#discussion_r2359539208
##########
arrow-avro/examples/read_with_utf8view.rs:
##########
@@ -48,6 +48,7 @@ fn main() -> Result<(), Box<dyn std::error::Error>> {
let batches: Vec<RecordBatch> = avro_reader.collect::<Result<_, _>>()?;
let regular_duration = start.elapsed();
+ file_for_view.seek(SeekFrom::Start(0))?;
Review Comment:
In my personal opinion, no need to modify the reader's logic. User who clone the file handle should know, realize and conscious control the file handle.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: reset the offset of 'file_for_view' [arrow-rs]
Posted by "TrevorADHD (via GitHub)" <gi...@apache.org>.
TrevorADHD commented on code in PR #8381:
URL: https://github.com/apache/arrow-rs/pull/8381#discussion_r2359618525
##########
arrow-avro/examples/read_with_utf8view.rs:
##########
@@ -48,6 +48,7 @@ fn main() -> Result<(), Box<dyn std::error::Error>> {
let batches: Vec<RecordBatch> = avro_reader.collect::<Result<_, _>>()?;
let regular_duration = start.elapsed();
+ file_for_view.seek(SeekFrom::Start(0))?;
Review Comment:
For example, i use confluent avro and apache avro tgt. So I can skip confluent magic byte + schema id , and pass it to reader. But if reader reset the offset, it will become not convenient.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: reset the offset of 'file_for_view' [arrow-rs]
Posted by "TrevorADHD (via GitHub)" <gi...@apache.org>.
TrevorADHD commented on code in PR #8381:
URL: https://github.com/apache/arrow-rs/pull/8381#discussion_r2359539208
##########
arrow-avro/examples/read_with_utf8view.rs:
##########
@@ -48,6 +48,7 @@ fn main() -> Result<(), Box<dyn std::error::Error>> {
let batches: Vec<RecordBatch> = avro_reader.collect::<Result<_, _>>()?;
let regular_duration = start.elapsed();
+ file_for_view.seek(SeekFrom::Start(0))?;
Review Comment:
In my personal opinion, no need to modify the reader's logic. User who clone the file handle should know, realize and conscious controlcontrol the file handle. If reader doesn't have to keep a mut file handle.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: reset the offset of 'file_for_view' [arrow-rs]
Posted by "mbrobbel (via GitHub)" <gi...@apache.org>.
mbrobbel commented on code in PR #8381:
URL: https://github.com/apache/arrow-rs/pull/8381#discussion_r2358787569
##########
arrow-avro/examples/read_with_utf8view.rs:
##########
@@ -48,6 +48,7 @@ fn main() -> Result<(), Box<dyn std::error::Error>> {
let batches: Vec<RecordBatch> = avro_reader.collect::<Result<_, _>>()?;
let regular_duration = start.elapsed();
+ file_for_view.seek(SeekFrom::Start(0))?;
Review Comment:
Maybe the reader should seek to the start?
https://github.com/apache/arrow-rs/blob/aed2f3b6a72375acb06cf958c9e3ff3c6ecb760f/arrow-avro/src/reader/mod.rs#L226-L243
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: reset the offset of 'file_for_view' [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb merged PR #8381:
URL: https://github.com/apache/arrow-rs/pull/8381
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org