You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@arrow.apache.org by Ying Zhou <yz...@gmail.com> on 2020/12/30 09:29:03 UTC
[C++] Weird Rust Linter error in CICD & Float/Double equality
Hi,
When finalizing my Arrow2ORC C++ pull request I found a weird Rust-related and IPC-related error in the Linter that didn’t happen just 2 days ago despite my code having nothing to do with either Rust or IPC. Here is the check: https://github.com/apache/arrow/pull/8648/checks?check_run_id=1625423680 <https://github.com/apache/arrow/pull/8648/checks?check_run_id=1625423680> Here is the part of the output I think is relevant:
source_dir /arrow/cpp/src --quiet
1413
<https://github.com/apache/arrow/pull/8648/checks?check_run_id=1625423680#step:7:1413>[1/1] cd /tmp/arrow-lint-_9i678pu/cpp-build && /usr/local/bin/python /arrow/cpp/build-support/lint_cpp_cli.py /arrow/cpp/src
1414
<https://github.com/apache/arrow/pull/8648/checks?check_run_id=1625423680#step:7:1414>INFO:archery:Running Python formatter (autopep8)
1415
<https://github.com/apache/arrow/pull/8648/checks?check_run_id=1625423680#step:7:1415>INFO:archery:Running Python linter (flake8)
1416
<https://github.com/apache/arrow/pull/8648/checks?check_run_id=1625423680#step:7:1416>INFO:archery:Running cmake-format linters
1417
<https://github.com/apache/arrow/pull/8648/checks?check_run_id=1625423680#step:7:1417>WARNING:archery:run-cmake-format modifies files, regardless of --fix
1418
<https://github.com/apache/arrow/pull/8648/checks?check_run_id=1625423680#step:7:1418>INFO:archery:Running apache-rat linter
1419
<https://github.com/apache/arrow/pull/8648/checks?check_run_id=1625423680#step:7:1419>INFO:archery:Running R linter
1420
<https://github.com/apache/arrow/pull/8648/checks?check_run_id=1625423680#step:7:1420>INFO:archery:Running Rust linter
1421
<https://github.com/apache/arrow/pull/8648/checks?check_run_id=1625423680#step:7:1421>Diff in /arrow/rust/arrow/src/ipc/reader.rs at line 160:
1422
<https://github.com/apache/arrow/pull/8648/checks?check_run_id=1625423680#step:7:1422> let null_count = struct_node.null_count() as usize;
1423
<https://github.com/apache/arrow/pull/8648/checks?check_run_id=1625423680#step:7:1423> let struct_array = if null_count > 0 {
1424
<https://github.com/apache/arrow/pull/8648/checks?check_run_id=1625423680#step:7:1424> // create struct array from fields, arrays and null data
1425
<https://github.com/apache/arrow/pull/8648/checks?check_run_id=1625423680#step:7:1425>- StructArray::from((
1426
<https://github.com/apache/arrow/pull/8648/checks?check_run_id=1625423680#step:7:1426>- struct_arrays,
1427
<https://github.com/apache/arrow/pull/8648/checks?check_run_id=1625423680#step:7:1427>- null_buffer,
1428
<https://github.com/apache/arrow/pull/8648/checks?check_run_id=1625423680#step:7:1428>- ))
1429
<https://github.com/apache/arrow/pull/8648/checks?check_run_id=1625423680#step:7:1429>+ StructArray::from((struct_arrays, null_buffer))
1430
<https://github.com/apache/arrow/pull/8648/checks?check_run_id=1625423680#step:7:1430> } else {
1431
<https://github.com/apache/arrow/pull/8648/checks?check_run_id=1625423680#step:7:1431> StructArray::from(struct_arrays)
1432
<https://github.com/apache/arrow/pull/8648/checks?check_run_id=1625423680#step:7:1432> };
1433
<https://github.com/apache/arrow/pull/8648/checks?check_run_id=1625423680#step:7:1433>INFO:archery:Running Docker linter
1434
<https://github.com/apache/arrow/pull/8648/checks?check_run_id=1625423680#step:7:1434>Error: `docker-compose --file /home/runner/work/arrow/arrow/docker-compose.yml run --rm ubuntu-lint` exited with a non-zero exit code 1, see the process log above.
There is no C++ or reasonbly C++-related error. Does anyone know why the error happen?
I also would like to ask about table equality when some columns are float/double. In this case do we have some built-in epsilon so that 0.6 == 0.6 work? Right now I have separate tests for these types that looks like the following and they are pretty clumsy:
EXPECT_TRUE(outputTable->schema()->Equals(*(table->schema())));
EXPECT_TRUE(outputTable->column(0)
->chunk(0)
->Slice(0, numRows / 2)
->ApproxEquals(table->column(0)->chunk(1)));
EXPECT_TRUE(outputTable->column(0)
->chunk(0)
->Slice(numRows / 2, numRows / 2)
->ApproxEquals(table->column(0)->chunk(3)));
EXPECT_TRUE(outputTable->column(1)
->chunk(0)
->Slice(0, numRows / 2)
->ApproxEquals(table->column(1)->chunk(1)));
EXPECT_TRUE(outputTable->column(1)
->chunk(0)
->Slice(numRows / 2, numRows / 2)
->ApproxEquals(table->column(1)->chunk(3)));
Thanks,
Ying
Re: [C++] Weird Rust Linter error in CICD & Float/Double equality
Posted by Andrew Lamb <al...@influxdata.com>.
https://github.com/apache/arrow/pull/9041 has a fix -- I'll merge it as
soon as it passes CI
On Wed, Dec 30, 2020 at 7:01 AM Andrew Lamb <al...@influxdata.com> wrote:
> I think this may be related to something merged in
> https://github.com/apache/arrow/pull/9041#issuecomment-752266447
>
> We'll get a fix in ASAP -- sorry about that
>
> On Wed, Dec 30, 2020 at 4:29 AM Ying Zhou <yz...@gmail.com> wrote:
>
>> Hi,
>>
>> When finalizing my Arrow2ORC C++ pull request I found a weird
>> Rust-related and IPC-related error in the Linter that didn’t happen just 2
>> days ago despite my code having nothing to do with either Rust or IPC. Here
>> is the check:
>> https://github.com/apache/arrow/pull/8648/checks?check_run_id=1625423680
>> <https://github.com/apache/arrow/pull/8648/checks?check_run_id=1625423680>
>> Here is the part of the output I think is relevant:
>>
>> source_dir /arrow/cpp/src --quiet
>> 1413
>> <
>> https://github.com/apache/arrow/pull/8648/checks?check_run_id=1625423680#step:7:1413>[1/1]
>> cd /tmp/arrow-lint-_9i678pu/cpp-build && /usr/local/bin/python
>> /arrow/cpp/build-support/lint_cpp_cli.py /arrow/cpp/src
>> 1414
>> <
>> https://github.com/apache/arrow/pull/8648/checks?check_run_id=1625423680#step:7:1414>INFO:archery:Running
>> Python formatter (autopep8)
>> 1415
>> <
>> https://github.com/apache/arrow/pull/8648/checks?check_run_id=1625423680#step:7:1415>INFO:archery:Running
>> Python linter (flake8)
>> 1416
>> <
>> https://github.com/apache/arrow/pull/8648/checks?check_run_id=1625423680#step:7:1416>INFO:archery:Running
>> cmake-format linters
>> 1417
>> <
>> https://github.com/apache/arrow/pull/8648/checks?check_run_id=1625423680#step:7:1417>WARNING:archery:run-cmake-format
>> modifies files, regardless of --fix
>> 1418
>> <
>> https://github.com/apache/arrow/pull/8648/checks?check_run_id=1625423680#step:7:1418>INFO:archery:Running
>> apache-rat linter
>> 1419
>> <
>> https://github.com/apache/arrow/pull/8648/checks?check_run_id=1625423680#step:7:1419>INFO:archery:Running
>> R linter
>> 1420
>> <
>> https://github.com/apache/arrow/pull/8648/checks?check_run_id=1625423680#step:7:1420>INFO:archery:Running
>> Rust linter
>> 1421
>> <
>> https://github.com/apache/arrow/pull/8648/checks?check_run_id=1625423680#step:7:1421>Diff
>> in /arrow/rust/arrow/src/ipc/reader.rs at line 160:
>> 1422
>> <
>> https://github.com/apache/arrow/pull/8648/checks?check_run_id=1625423680#step:7:1422>
>> let null_count = struct_node.null_count() as usize;
>> 1423
>> <
>> https://github.com/apache/arrow/pull/8648/checks?check_run_id=1625423680#step:7:1423>
>> let struct_array = if null_count > 0 {
>> 1424
>> <
>> https://github.com/apache/arrow/pull/8648/checks?check_run_id=1625423680#step:7:1424>
>> // create struct array from fields, arrays and null data
>> 1425
>> <
>> https://github.com/apache/arrow/pull/8648/checks?check_run_id=1625423680#step:7:1425>-
>> StructArray::from((
>> 1426
>> <
>> https://github.com/apache/arrow/pull/8648/checks?check_run_id=1625423680#step:7:1426>-
>> struct_arrays,
>> 1427
>> <
>> https://github.com/apache/arrow/pull/8648/checks?check_run_id=1625423680#step:7:1427>-
>> null_buffer,
>> 1428
>> <
>> https://github.com/apache/arrow/pull/8648/checks?check_run_id=1625423680#step:7:1428>-
>> ))
>> 1429
>> <
>> https://github.com/apache/arrow/pull/8648/checks?check_run_id=1625423680#step:7:1429>+
>> StructArray::from((struct_arrays, null_buffer))
>> 1430
>> <
>> https://github.com/apache/arrow/pull/8648/checks?check_run_id=1625423680#step:7:1430>
>> } else {
>> 1431
>> <
>> https://github.com/apache/arrow/pull/8648/checks?check_run_id=1625423680#step:7:1431>
>> StructArray::from(struct_arrays)
>> 1432
>> <
>> https://github.com/apache/arrow/pull/8648/checks?check_run_id=1625423680#step:7:1432>
>> };
>> 1433
>> <
>> https://github.com/apache/arrow/pull/8648/checks?check_run_id=1625423680#step:7:1433>INFO:archery:Running
>> Docker linter
>> 1434
>> <
>> https://github.com/apache/arrow/pull/8648/checks?check_run_id=1625423680#step:7:1434>Error:
>> `docker-compose --file /home/runner/work/arrow/arrow/docker-compose.yml run
>> --rm ubuntu-lint` exited with a non-zero exit code 1, see the process log
>> above.
>>
>> There is no C++ or reasonbly C++-related error. Does anyone know why the
>> error happen?
>>
>> I also would like to ask about table equality when some columns are
>> float/double. In this case do we have some built-in epsilon so that 0.6 ==
>> 0.6 work? Right now I have separate tests for these types that looks like
>> the following and they are pretty clumsy:
>>
>> EXPECT_TRUE(outputTable->schema()->Equals(*(table->schema())));
>> EXPECT_TRUE(outputTable->column(0)
>> ->chunk(0)
>> ->Slice(0, numRows / 2)
>> ->ApproxEquals(table->column(0)->chunk(1)));
>> EXPECT_TRUE(outputTable->column(0)
>> ->chunk(0)
>> ->Slice(numRows / 2, numRows / 2)
>> ->ApproxEquals(table->column(0)->chunk(3)));
>> EXPECT_TRUE(outputTable->column(1)
>> ->chunk(0)
>> ->Slice(0, numRows / 2)
>> ->ApproxEquals(table->column(1)->chunk(1)));
>> EXPECT_TRUE(outputTable->column(1)
>> ->chunk(0)
>> ->Slice(numRows / 2, numRows / 2)
>> ->ApproxEquals(table->column(1)->chunk(3)));
>>
>> Thanks,
>> Ying
>
>
Re: [C++] Weird Rust Linter error in CICD & Float/Double equality
Posted by Andrew Lamb <al...@influxdata.com>.
I think this may be related to something merged in
https://github.com/apache/arrow/pull/9041#issuecomment-752266447
We'll get a fix in ASAP -- sorry about that
On Wed, Dec 30, 2020 at 4:29 AM Ying Zhou <yz...@gmail.com> wrote:
> Hi,
>
> When finalizing my Arrow2ORC C++ pull request I found a weird Rust-related
> and IPC-related error in the Linter that didn’t happen just 2 days ago
> despite my code having nothing to do with either Rust or IPC. Here is the
> check:
> https://github.com/apache/arrow/pull/8648/checks?check_run_id=1625423680 <
> https://github.com/apache/arrow/pull/8648/checks?check_run_id=1625423680>
> Here is the part of the output I think is relevant:
>
> source_dir /arrow/cpp/src --quiet
> 1413
> <
> https://github.com/apache/arrow/pull/8648/checks?check_run_id=1625423680#step:7:1413>[1/1]
> cd /tmp/arrow-lint-_9i678pu/cpp-build && /usr/local/bin/python
> /arrow/cpp/build-support/lint_cpp_cli.py /arrow/cpp/src
> 1414
> <
> https://github.com/apache/arrow/pull/8648/checks?check_run_id=1625423680#step:7:1414>INFO:archery:Running
> Python formatter (autopep8)
> 1415
> <
> https://github.com/apache/arrow/pull/8648/checks?check_run_id=1625423680#step:7:1415>INFO:archery:Running
> Python linter (flake8)
> 1416
> <
> https://github.com/apache/arrow/pull/8648/checks?check_run_id=1625423680#step:7:1416>INFO:archery:Running
> cmake-format linters
> 1417
> <
> https://github.com/apache/arrow/pull/8648/checks?check_run_id=1625423680#step:7:1417>WARNING:archery:run-cmake-format
> modifies files, regardless of --fix
> 1418
> <
> https://github.com/apache/arrow/pull/8648/checks?check_run_id=1625423680#step:7:1418>INFO:archery:Running
> apache-rat linter
> 1419
> <
> https://github.com/apache/arrow/pull/8648/checks?check_run_id=1625423680#step:7:1419>INFO:archery:Running
> R linter
> 1420
> <
> https://github.com/apache/arrow/pull/8648/checks?check_run_id=1625423680#step:7:1420>INFO:archery:Running
> Rust linter
> 1421
> <
> https://github.com/apache/arrow/pull/8648/checks?check_run_id=1625423680#step:7:1421>Diff
> in /arrow/rust/arrow/src/ipc/reader.rs at line 160:
> 1422
> <
> https://github.com/apache/arrow/pull/8648/checks?check_run_id=1625423680#step:7:1422>
> let null_count = struct_node.null_count() as usize;
> 1423
> <
> https://github.com/apache/arrow/pull/8648/checks?check_run_id=1625423680#step:7:1423>
> let struct_array = if null_count > 0 {
> 1424
> <
> https://github.com/apache/arrow/pull/8648/checks?check_run_id=1625423680#step:7:1424>
> // create struct array from fields, arrays and null data
> 1425
> <
> https://github.com/apache/arrow/pull/8648/checks?check_run_id=1625423680#step:7:1425>-
> StructArray::from((
> 1426
> <
> https://github.com/apache/arrow/pull/8648/checks?check_run_id=1625423680#step:7:1426>-
> struct_arrays,
> 1427
> <
> https://github.com/apache/arrow/pull/8648/checks?check_run_id=1625423680#step:7:1427>-
> null_buffer,
> 1428
> <
> https://github.com/apache/arrow/pull/8648/checks?check_run_id=1625423680#step:7:1428>-
> ))
> 1429
> <
> https://github.com/apache/arrow/pull/8648/checks?check_run_id=1625423680#step:7:1429>+
> StructArray::from((struct_arrays, null_buffer))
> 1430
> <
> https://github.com/apache/arrow/pull/8648/checks?check_run_id=1625423680#step:7:1430>
> } else {
> 1431
> <
> https://github.com/apache/arrow/pull/8648/checks?check_run_id=1625423680#step:7:1431>
> StructArray::from(struct_arrays)
> 1432
> <
> https://github.com/apache/arrow/pull/8648/checks?check_run_id=1625423680#step:7:1432>
> };
> 1433
> <
> https://github.com/apache/arrow/pull/8648/checks?check_run_id=1625423680#step:7:1433>INFO:archery:Running
> Docker linter
> 1434
> <
> https://github.com/apache/arrow/pull/8648/checks?check_run_id=1625423680#step:7:1434>Error:
> `docker-compose --file /home/runner/work/arrow/arrow/docker-compose.yml run
> --rm ubuntu-lint` exited with a non-zero exit code 1, see the process log
> above.
>
> There is no C++ or reasonbly C++-related error. Does anyone know why the
> error happen?
>
> I also would like to ask about table equality when some columns are
> float/double. In this case do we have some built-in epsilon so that 0.6 ==
> 0.6 work? Right now I have separate tests for these types that looks like
> the following and they are pretty clumsy:
>
> EXPECT_TRUE(outputTable->schema()->Equals(*(table->schema())));
> EXPECT_TRUE(outputTable->column(0)
> ->chunk(0)
> ->Slice(0, numRows / 2)
> ->ApproxEquals(table->column(0)->chunk(1)));
> EXPECT_TRUE(outputTable->column(0)
> ->chunk(0)
> ->Slice(numRows / 2, numRows / 2)
> ->ApproxEquals(table->column(0)->chunk(3)));
> EXPECT_TRUE(outputTable->column(1)
> ->chunk(0)
> ->Slice(0, numRows / 2)
> ->ApproxEquals(table->column(1)->chunk(1)));
> EXPECT_TRUE(outputTable->column(1)
> ->chunk(0)
> ->Slice(numRows / 2, numRows / 2)
> ->ApproxEquals(table->column(1)->chunk(3)));
>
> Thanks,
> Ying