You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "ZENOTME (via GitHub)" <gi...@apache.org> on 2023/10/26 10:20:25 UTC
[I] Return error instead of assert when meeting incompatitble type [arrow-rs]
ZENOTME opened a new issue, #4995:
URL: https://github.com/apache/arrow-rs/issues/4995
https://github.com/apache/arrow-rs/blob/b07dabea870404c4d2d42e16e33d12b152f364d5/parquet/src/arrow/arrow_writer/levels.rs#L135
When I use the writer and pass incompatitble type, it will panic instead of return a error. Maybe it should return a error here or other place? 🧐
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Return error instead of assert when meeting incompatitble type [arrow-rs]
Posted by "carols10cents (via GitHub)" <gi...@apache.org>.
carols10cents commented on issue #4995:
URL: https://github.com/apache/arrow-rs/issues/4995#issuecomment-1914839326
I've just hit this panic as well, and it's definitely because I'm messing up the schemas somewhere, but it's hard to tell what the problem is from the panic message.
At the very least, I expected some Arrow method in the call stack of the backtrace to have documentation with a "Panics" section that details the invariant that I'm not upholding, but I don't see any (and I'm not sure exactly where it should go). I'd be happy to send in a documentation PR if given some guidance for where it should go!
<details>
<summary>Backtrace</summary>
```
hread \'main\' panicked at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/parquet-50.0.0/src/arrow/arrow_writer/levels.rs:135:9:
assertion `left == right` failed
left: Int64
right: Float64
stack backtrace:
0: rust_begin_unwind
at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:645:5
1: core::panicking::panic_fmt
at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/panicking.rs:72:14
2: core::panicking::assert_failed_inner
3: core::panicking::assert_failed
at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/panicking.rs:279:5
4: parquet::arrow::arrow_writer::levels::LevelInfoBuilder::try_new
at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/parquet-50.0.0/src/arrow/arrow_writer/levels.rs:135:9
5: parquet::arrow::arrow_writer::levels::calculate_array_levels
at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/parquet-50.0.0/src/arrow/arrow_writer/levels.rs:54:23
6: parquet::arrow::arrow_writer::compute_leaves
at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/parquet-50.0.0/src/arrow/arrow_writer/mod.rs:355:18
7: parquet::arrow::arrow_writer::ArrowRowGroupWriter::write
at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/parquet-50.0.0/src/arrow/arrow_writer/mod.rs:534:25
8: parquet::arrow::arrow_writer::ArrowWriter<W>::write
at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/parquet-50.0.0/src/arrow/arrow_writer/mod.rs:194:9
9: bulk_ingester::fsm::create_intermediate_files::IntermediateFileManager::add_batch
at ./src/fsm/create_intermediate_files.rs:253:12
10: bulk_ingester::fsm::create_intermediate_files::partition_file::{{closure}}::{{closure}}
at ./src/fsm/create_intermediate_files.rs:185:17
11: core::option::Option<T>::map
at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/option.rs:1072:29
12: bulk_ingester::fsm::create_intermediate_files::partition_file::{{closure}}
at ./src/fsm/create_intermediate_files.rs:182:9
13: bulk_ingester::fsm::create_intermediate_files::<impl bulk_ingester::fsm::Fsm<bulk_ingester::fsm::create_intermediate_files::CreateIntermediateFiles>>::create_intermediate_files::{{closure}}
at ./src/fsm/create_intermediate_files.rs:116:18
14: bulk_ingester::fsm::FsmDriver::try_transition::{{closure}}::{{closure}}::{{closure}}
at ./src/fsm.rs:116:26
15: state_machine_utils::AlwaysSome<T>::mutate_async::{{closure}}
at /home/rust/project/state_machine_utils/src/lib.rs:55:73
16: bulk_ingester::fsm::FsmDriver::try_transition::{{closure}}
at ./src/fsm.rs:125:14
17: bulk_ingester::bulk_ingest::{{closure}}
at ./src/main.rs:273:37
18: bulk_ingester::main::{{closure}}
at ./src/main.rs:160:45
19: <tracing::instrument::Instrumented<T> as core::future::future::Future>::poll
at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tracing-0.1.40/src/instrument.rs:321:9
20: tokio::runtime::park::CachedParkThread::block_on::{{closure}}
at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/park.rs:282:63
21: tokio::runtime::coop::with_budget
at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/coop.rs:107:5
22: tokio::runtime::coop::budget
at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/coop.rs:73:5
23: tokio::runtime::park::CachedParkThread::block_on
at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/park.rs:282:31
24: tokio::runtime::context::blocking::BlockingRegionGuard::block_on
at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/context/blocking.rs:66:9
25: tokio::runtime::scheduler::multi_thread::MultiThread::block_on::{{closure}}
at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/scheduler/multi_thread/mod.rs:87:13
26: tokio::runtime::context::runtime::enter_runtime
at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/context/runtime.rs:65:16
27: tokio::runtime::scheduler::multi_thread::MultiThread::block_on
at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/scheduler/multi_thread/mod.rs:86:9
28: tokio::runtime::runtime::Runtime::block_on
at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.35.1/src/runtime/runtime.rs:350:45
29: bulk_ingester::main
at ./src/main.rs:160:5
30: core::ops::function::FnOnce::call_once
at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/ops/function.rs:250:5
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
```
Places I looked for documentation:
- `parquet::arrow::arrow_writer::levels::LevelInfoBuilder::try_new` - private
- `parquet::arrow::arrow_writer::levels::calculate_array_levels` - private
- `parquet::arrow::arrow_writer::compute_leaves` - [no "Panics" section](https://docs.rs/parquet/latest/parquet/arrow/arrow_writer/fn.compute_leaves.html)
- `parquet::arrow::arrow_writer::ArrowRowGroupWriter::write` - private
- `parquet::arrow::arrow_writer::ArrowWriter` - [no "Panics" section](https://docs.rs/parquet/latest/parquet/arrow/arrow_writer/struct.ArrowWriter.html), this is where I'd guess the invariants should be documented, as this is what I'm calling
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Return error instead of assert when meeting incompatitble type [arrow-rs]
Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold closed issue #4995: Return error instead of assert when meeting incompatitble type
URL: https://github.com/apache/arrow-rs/issues/4995
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Return error instead of assert when meeting incompatitble type [arrow-rs]
Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on issue #4995:
URL: https://github.com/apache/arrow-rs/issues/4995#issuecomment-1914858466
Oh I see what has happened, at some point the schema check got removed from ArrowWriter, possibly #4871, you can see it in this PR - https://github.com/apache/arrow-rs/pull/4027/files#diff-dbb6b7806a48ce2b6fb40ce4a429599808a87671274ee6b2dc7c1f87429320b3R135
As such you can now hit this panic. We should change it to be an error with a more descriptive message
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Return error instead of assert when meeting incompatitble type [arrow-rs]
Posted by "tustvold (via GitHub)" <gi...@apache.org>.
tustvold commented on issue #4995:
URL: https://github.com/apache/arrow-rs/issues/4995#issuecomment-1780912062
In general we use assertions for invariants that are not expected to be encountered in a well-formed program. I'm not sure that a well-formed program should be able to hit this as it would indicate some form of schema inconsistency, but then I'm also not opposed to making this an error.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org