You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2023/01/05 12:11:55 UTC
[GitHub] [arrow-rs] alamb opened a new issue, #3460: Arrow IPC integration test "generated_map_non_canonical" failure
alamb opened a new issue, #3460:
URL: https://github.com/apache/arrow-rs/issues/3460
**Describe the bug**
There is some problem reading the Arrow Map files from the IPC format
in the reader tests: https://github.com/apache/arrow-rs/blob/81abc1a942cd13a92231f4a828077ad60fdabe36/arrow-integration-testing/tests/ipc_reader.rs#L120-L122
and the writer tests: https://github.com/apache/arrow-rs/blob/81abc1a942cd13a92231f4a828077ad60fdabe36/arrow-integration-testing/tests/ipc_writer.rs#L73-L74
**To Reproduce**
Uncomment one the lines referenced above and the test will fail like the following (looks like the names of the fields are different -- "entries" vs "some_entries")
```
Verifying /Users/alamb/Software/arrow-rs2/arrow/../testing/data/arrow-ipc-stream/integration/1.0.0-littleendian/generated_map_non_canonical.stream
verifying content
thread 'read_1_0_0_littleendian' panicked at 'assertion failed: `(left == right)`
left: `Field { name: "map_other_names", data_type: Map(Field { name: "some_entries", data_type: Struct([Field { name: "some_key", data_type: Utf8, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: "some_value", data_type: Int32, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }]), nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {} }, false), nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }`,
right: `Field { name: "map_other_names", data_type: Map(Field { name: "entries", data_type: Struct([Field { name: "key", data_type: Utf8, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: "value", data_type: Int32, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }]), nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {} }, false), nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }`: Arrow fields not the same', arrow-integration-test/src/lib.rs:253:17
stack backtrace:
0: rust_begin_unwind
at /rustc/69f9c33d71c871fc16ac445211281c6e7a340943/library/std/src/panicking.rs:575:5
1: core::panicking::panic_fmt
at /rustc/69f9c33d71c871fc16ac445211281c6e7a340943/library/core/src/panicking.rs:65:14
2: core::panicking::assert_failed_inner
at /rustc/69f9c33d71c871fc16ac445211281c6e7a340943/library/core/src/panicking.rs:240:23
3: core::panicking::assert_failed
at /rustc/69f9c33d71c871fc16ac445211281c6e7a340943/library/core/src/panicking.rs:203:5
4: arrow_integration_test::ArrowJsonField::equals_field
at /Users/alamb/Software/arrow-rs2/arrow-integration-test/src/lib.rs:253:17
5: arrow_integration_test::ArrowJsonSchema::equals_schema
at /Users/alamb/Software/arrow-rs2/arrow-integration-test/src/lib.rs:217:17
6: arrow_integration_test::ArrowJson::equals_reader
at /Users/alamb/Software/arrow-rs2/arrow-integration-test/src/lib.rs:166:13
7: ipc_reader::verify_arrow_stream
at ./tests/ipc_reader.rs:218:17
8: ipc_reader::read_1_0_0_littleendian::{{closure}}
at ./tests/ipc_reader.rs:140:9
9: <core::slice::iter::Iter<T> as core::iter::traits::iterator::Iterator>::for_each
at /rustc/69f9c33d71c871fc16ac445211281c6e7a340943/library/core/src/slice/iter/macros.rs:211:21
10: ipc_reader::read_1_0_0_littleendian
at ./tests/ipc_reader.rs:138:5
11: ipc_reader::read_1_0_0_littleendian::{{closure}}
at ./tests/ipc_reader.rs:109:1
12: core::ops::function::FnOnce::call_once
at /rustc/69f9c33d71c871fc16ac445211281c6e7a340943/library/core/src/ops/function.rs:251:5
13: core::ops::function::FnOnce::call_once
at /rustc/69f9c33d71c871fc16ac445211281c6e7a340943/library/core/src/ops/function.rs:251:5
```
**Expected behavior**
<!--
A clear and concise description of what you expected to happen.
-->
**Additional context**
Found on https://github.com/apache/arrow-rs/pull/3427
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] alamb commented on issue #3460: Arrow IPC integration test "generated_map_non_canonical" failure
Posted by GitBox <gi...@apache.org>.
alamb commented on issue #3460:
URL: https://github.com/apache/arrow-rs/issues/3460#issuecomment-1374881831
> Other approach is to enable Archery tests for Rust for littleendian folder and remove the tests from the newly written integration tests.
I think the idea of using the existing archery tests from the overall integration suite would be a great improvement 👍
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] alamb commented on issue #3460: Arrow IPC integration test "generated_map_non_canonical" failure
Posted by GitBox <gi...@apache.org>.
alamb commented on issue #3460:
URL: https://github.com/apache/arrow-rs/issues/3460#issuecomment-1374881892
Thank you for the thorough research @askoa
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] askoa commented on issue #3460: Arrow IPC integration test "generated_map_non_canonical" failure
Posted by GitBox <gi...@apache.org>.
askoa commented on issue #3460:
URL: https://github.com/apache/arrow-rs/issues/3460#issuecomment-1372700758
I am not sure what's the ask is here. I suspect there might be some issue with test case.
The [specification for file format](https://arrow.apache.org/docs/format/Columnar.html#ipc-file-format) is that it is IPC Data encapsulated with some additional preceding and succeeding data. However when I look at arrow file `generated_map_non_canonical.arrow_file`, I could see that field names are `some_entries`, `some_keys` and `some_values`. Whereas, in the stream file `generated_map_non_canonical.stream`, the field names are `entries`, `keys` and `values`. This is a break from specification linked above.
Also, the different field names in json file and stream file are the reason the test case is breaking. I see that the stream file, with has field names `entries`, `keys` and `values`, is compared with json file with field names `some_entries`, `some_keys` and `some_values`. The mismatch is because of the contents of the file used and not because of the code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] askoa commented on issue #3460: Arrow IPC integration test "generated_map_non_canonical" failure
Posted by GitBox <gi...@apache.org>.
askoa commented on issue #3460:
URL: https://github.com/apache/arrow-rs/issues/3460#issuecomment-1373697069
The value `some_entries` from json file is compared with value `entries` from the stream file. Should the JSON file be processed through ipc reader before being compared? Then we can fix ipc reader for any differences.
https://github.com/apache/arrow-rs/blob/acefeef1cb5698a6afe1d3061644f6276d39117c/arrow-integration-testing/tests/ipc_reader.rs#L216-L217
I was trying to find if any of the existing arrow projects utilize the ipc test data set. I checked golang and java. They seem to validate only the arrow files and not the stream files. Do you know any projects that validate the stream files?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] alamb commented on issue #3460: Arrow IPC integration test "generated_map_non_canonical" failure
Posted by GitBox <gi...@apache.org>.
alamb commented on issue #3460:
URL: https://github.com/apache/arrow-rs/issues/3460#issuecomment-1373890208
I do not -- the C++ would be the one I would expect
On Fri, Jan 6, 2023 at 9:13 AM askoa ***@***.***> wrote:
> The value some_entries from json file is compared with value entries from
> the stream file. Should the JSON file be processed through ipc reader
> before being compared? Then we can fix ipc reader for any differences.
>
>
> https://github.com/apache/arrow-rs/blob/acefeef1cb5698a6afe1d3061644f6276d39117c/arrow-integration-testing/tests/ipc_reader.rs#L216-L217
>
> I was trying to find if any of the existing arrow projects utilize the ipc
> test data set. I checked golang and java. They seem to validate only the
> arrow files and not the stream files. Do you know any projects that
> validate the stream files?
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/apache/arrow-rs/issues/3460#issuecomment-1373697069>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AADXZMOBWAG67FOORDOF5EDWRASCHANCNFSM6AAAAAATR4F6HM>
> .
> You are receiving this because you authored the thread.Message ID:
> ***@***.***>
>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] askoa commented on issue #3460: Arrow IPC integration test "generated_map_non_canonical" failure
Posted by GitBox <gi...@apache.org>.
askoa commented on issue #3460:
URL: https://github.com/apache/arrow-rs/issues/3460#issuecomment-1374169123
The `*.stream` files are used in Archery integration tests. Though `1.0.0-littleendian` folder is skipped for Rust, the files that are being tested gives insight on how non-canonical field names are handled. In Rust, a function named `canonicalize_schema` strips away any non-canonical names from the schemas before validation.
- One solution is to use the function `canonicalize_schema` before comparing schemas. I checked Java and Go validations, they have custom JSON parser which pretty much does same thing.
- Other approach is to enable Archery tests for Rust for littleendian folder and remove the tests from the newly written integration tests.
##### 1.0.0-littleendian folder is skipped:
https://github.com/apache/arrow/blob/a580f2711750ef507cc57ce48cb431dd700a6166/dev/archery/archery/integration/runner.py#L135-L139
##### canonicalize_schema:
https://github.com/apache/arrow-rs/blob/d325ad2d0f2b39972d1b299479d06c5fd9a41d42/arrow-integration-testing/src/bin/arrow-json-integration-test.rs#L115
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] alamb commented on issue #3460: Arrow IPC integration test "generated_map_non_canonical" failure
Posted by GitBox <gi...@apache.org>.
alamb commented on issue #3460:
URL: https://github.com/apache/arrow-rs/issues/3460#issuecomment-1373006367
Thanks @askoa -- I think the ask here is to figure out what should be done with the test case.
Given the dataset has "non_canonical" in the name I suspect there is some corner case it is meant to sort out (perhaps the names should be normalized as part of the arrow ipc reader?)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org