You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2022/03/04 14:19:51 UTC
[GitHub] [arrow-rs] alamb opened a new issue #1398: Integration Test is failing on master branch
alamb opened a new issue #1398:
URL: https://github.com/apache/arrow-rs/issues/1398
**Describe the bug**
The "Integration Test" CI test is failing on master. For example:
https://github.com/apache/arrow-rs/runs/5421632152?check_suite_focus=true
It appears to have started in c9470279dffd81190725205e57022d68b7d24ad6 (though I don't think that PR had anything to do with it)
<img width="1353" alt="Screen Shot 2022-03-04 at 9 18 07 AM" src="https://user-images.githubusercontent.com/490673/156779611-516dbf07-72ad-4884-b791-192c84adc089.png">
**To Reproduce**
Run integration test on master
**Expected behavior**
I expect it to pass
**Additional context**
The integration test pulls from https://github.com/apache/arrow so perhaps something upstream changed.
I think we need to sort out what is wrong with this test prior to the 10.0.0 release
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] alamb closed issue #1398: Integration Test is failing on master branch
Posted by GitBox <gi...@apache.org>.
alamb closed issue #1398:
URL: https://github.com/apache/arrow-rs/issues/1398
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] alamb commented on issue #1398: Integration Test is failing on master branch
Posted by GitBox <gi...@apache.org>.
alamb commented on issue #1398:
URL: https://github.com/apache/arrow-rs/issues/1398#issuecomment-1059297731
Thanks @xudong963 👍
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] alamb edited a comment on issue #1398: Integration Test is failing on master branch
Posted by GitBox <gi...@apache.org>.
alamb edited a comment on issue #1398:
URL: https://github.com/apache/arrow-rs/issues/1398#issuecomment-1059592336
I tried backing out https://github.com/apache/arrow/commit/2462492389a8f2ca286c481852c84ba1f0d0eff9 (the first build that failed in arrow) from my local environment and it did not help
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] alamb commented on issue #1398: Integration Test is failing on master branch
Posted by GitBox <gi...@apache.org>.
alamb commented on issue #1398:
URL: https://github.com/apache/arrow-rs/issues/1398#issuecomment-1059592336
I tried backing out https://github.com/apache/arrow/commit/2462492389a8f2ca286c481852c84ba1f0d0eff9 from my local environment and it did not help
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] alamb edited a comment on issue #1398: Integration Test is failing on master branch
Posted by GitBox <gi...@apache.org>.
alamb edited a comment on issue #1398:
URL: https://github.com/apache/arrow-rs/issues/1398#issuecomment-1059589935
@matthewmturner -- I agree with your assesment.
Interesting the integration tests appear to fail inhttps://github.com/apache/arrow/ as well with the same errors:
https://github.com/apache/arrow/runs/5427260660?check_suite_focus=true
I am looking into that as well
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] xudong963 commented on issue #1398: Integration Test is failing on master branch
Posted by GitBox <gi...@apache.org>.
xudong963 commented on issue #1398:
URL: https://github.com/apache/arrow-rs/issues/1398#issuecomment-1059243064
Maybe related to https://github.com/hyperium/tonic/issues/887
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] lidavidm commented on issue #1398: Integration Test is failing on master branch
Posted by GitBox <gi...@apache.org>.
lidavidm commented on issue #1398:
URL: https://github.com/apache/arrow-rs/issues/1398#issuecomment-1059610103
I believe the :scheme pseudo-header is supposed to be 'http' or 'https', not 'grpc': https://grpc.github.io/grpc/cpp/md_doc__p_r_o_t_o_c_o_l-_h_t_t_p2.html
It sounds like Tonic is sending `:scheme = grpc` for some reason, then gRPC in C++ is sending a RST_STREAM back to Rust.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] jorgecarleitao commented on issue #1398: Integration Test is failing on master branch
Posted by GitBox <gi...@apache.org>.
jorgecarleitao commented on issue #1398:
URL: https://github.com/apache/arrow-rs/issues/1398#issuecomment-1059503668
fwiw arrow2 is also failing on those two tests and we did not change anything in tonic, flight, IPC etc that could have caused this. I am investigating it as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] alamb edited a comment on issue #1398: Integration Test is failing on master branch
Posted by GitBox <gi...@apache.org>.
alamb edited a comment on issue #1398:
URL: https://github.com/apache/arrow-rs/issues/1398#issuecomment-1059551674
FYI @jorgecarleitao
I have been able to reproduce this locally.
Here are some notes I have in case that is helpful:
```
# check out arrow
# install archery:
cd arrow
pip install -e dev/archery[docker]
# link arrow-rs to arrow/rust
ln -s ../arrow-rs rust
```
```
# build cpp binaries
cd arrow/cpp
mkdir build
cd build
cmake -DARROW_BUILD_INTEGRATION=ON -DARROW_FLIGHT=ON --preset ninja-debug-minimal ..
cd ..
ninja
```
Then
```
# build rust:
cd ../arrow-rs
cargo build --all
```
# now from arrow directory run the tests:
```
archery integration --with-cpp=true --with-rust=true
```
Run individual test:
```
(arrow_dev) alamb@MacBook-Pro-2:~/Software/arrow/cpp/build$ /Users/alamb/Software/arrow/cpp/build/debug/flight-test-integration-server -port 49153
Run in rust:
/Users/alamb/Software/arrow/rust/target/debug/flight-test-integration-client --host localhost --port=49153 --path /var/folders/s3/h5hgj43j0bv83shtmz_t_w400000gn/T/arrow-integration-v7xz5285/generated_dictionary_unsigned.json
Repro with this:
cd /Users/alamb/Software/arrow-rs && RUST_LOG=debug RUST_BACKTRACE=1 CARGO_TARGET_DIR=/Users/alamb/Software/df-target cargo run --bin flight-test-integration-client -- --host localhost --port=49153 --path /var/folders/s3/h5hgj43j0bv83shtmz_t_w400000gn/T/arrow-integration-v7xz5285/generated_dictionary_unsigned.json
Results in this
E0304 16:33:43.055262000 123145352572928 hpack_parser.cc:1240] Error parsing metadata: error=invalid value key=:scheme value=grpc
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] alamb commented on issue #1398: Integration Test is failing on master branch
Posted by GitBox <gi...@apache.org>.
alamb commented on issue #1398:
URL: https://github.com/apache/arrow-rs/issues/1398#issuecomment-1059426068
I do not yet know what is going on, but wanted to post on my progress (I don't have all that much more time to spend today on this)
Last good run @ `Mar 3 9:04 AM EST`: https://github.com/apache/arrow-rs/commit/258f828f9fd5bdfb668cbab7005d6bff73184b8f
First failed run @ `Mar 3 12:47 PM EST`: https://github.com/apache/arrow-rs/commit/c9470279dffd81190725205e57022d68b7d24ad6
These commits PR from arrow appears to be in that time range, but are not obviously related to the tests
* https://github.com/apache/arrow/commit/ce46c1adf10654248b02c349210bb63ecc6828ad (update flight dependencies)
* https://github.com/apache/arrow/commit/9719eae66dcf38c966ae769215d27020a6dd5550
The error appears to be happening with C++ serving and rust requesting in two scenarios:
```
Testing file interval_mdn
Testing file duration
```
I am working to reproduce the error locally so I can debug further, but as I am not super familiar with the integration testing setup, it is slow going.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] alamb commented on issue #1398: Integration Test is failing on master branch
Posted by GitBox <gi...@apache.org>.
alamb commented on issue #1398:
URL: https://github.com/apache/arrow-rs/issues/1398#issuecomment-1059744816
Thanks @lidavidm -- that was a great hint! I think I have a workaround now https://github.com/apache/arrow-rs/pull/1402 and we'll see if that works
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] alamb commented on issue #1398: Integration Test is failing on master branch
Posted by GitBox <gi...@apache.org>.
alamb commented on issue #1398:
URL: https://github.com/apache/arrow-rs/issues/1398#issuecomment-1059594658
Even when I use arrow at https://github.com/apache/arrow/commit/e314d8d0d611c7f9ca7f2fbee174fcea3d0c66f2 which had a passing [Integration test](https://github.com/apache/arrow/runs/5408330491?check_suite_focus=true) my local test setup still fails.
This suggests that it is some change in third-party dependencies (either rust or c++) perhaps
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] alamb commented on issue #1398: Integration Test is failing on master branch
Posted by GitBox <gi...@apache.org>.
alamb commented on issue #1398:
URL: https://github.com/apache/arrow-rs/issues/1398#issuecomment-1059589935
@matthewmturner -- I agree with your assesment.
Interesting the integration tests appear to fail in apache as well with the same errors:
https://github.com/apache/arrow/runs/5427260660?check_suite_focus=true
I am looking into that as well
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] xudong963 commented on issue #1398: Integration Test is failing on master branch
Posted by GitBox <gi...@apache.org>.
xudong963 commented on issue #1398:
URL: https://github.com/apache/arrow-rs/issues/1398#issuecomment-1059257017
https://github.com/apache/arrow-rs/pull/864#issuecomment-951529603
When updated tonic to 6.0, I found the integration test also failed.
So I guess it's the problem of tonic @alamb
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] matthewmturner commented on issue #1398: Integration Test is failing on master branch
Posted by GitBox <gi...@apache.org>.
matthewmturner commented on issue #1398:
URL: https://github.com/apache/arrow-rs/issues/1398#issuecomment-1059587436
apologize if ive misunderstood any part of this or if i dont understand full intergration testing process - but given that both arrow and arrow2 started failing i am focusing on two scenarios:
1. Dependency that both arrow and arrow2 use which caused issue. @alamb and @jorgecarleitao are each looking into this - so i consider this "covered" in the sense that its being looked into.
2. Issue from C++ (content or dependency) that's impacted both. I know there was an email sent to dev email list mentioning the integration test fail but its not clear to me if anything beyond that has been raised on that side. Should we be raising something on that side?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] alamb commented on issue #1398: Integration Test is failing on master branch
Posted by GitBox <gi...@apache.org>.
alamb commented on issue #1398:
URL: https://github.com/apache/arrow-rs/issues/1398#issuecomment-1059563416
The output from rust is:
```
Error: Status { code: Unknown, message: "transport error", source: Some(tonic::transport::Error(Transport, hyper::Error(Http2, Error { kind: Reset(StreamId(1), INTERNAL_ERROR, Remote) }))) }
```
The output from C++ is
```
E0304 16:47:21.317255000 123145352572928 hpack_parser.cc:1240] Error parsing metadata: error=invalid value key=:scheme value=grpc
```
Which seems to suggest some sort of bug in parsing headers or something
However, since other tests run, maybe it is something related to the contents?
Given how it came on suddenly, smells like something related to a change in a dependency, but I haven't found one yet
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] alamb commented on issue #1398: Integration Test is failing on master branch
Posted by GitBox <gi...@apache.org>.
alamb commented on issue #1398:
URL: https://github.com/apache/arrow-rs/issues/1398#issuecomment-1059551674
FYI @jorgecarleitao
I have been able to reproduce this locally.
Here are some notes I have in case that is helpful:
```
# check out arrow
# install archery:
cd arrow
pip install -e dev/archery[docker]
# link arrow-rs to arrow/rs
ln -s ../arrow-rs rust
```
```
# build cpp binaries
cd arrow/cpp
mkdir build
cd build
cmake -DARROW_BUILD_INTEGRATION=ON -DARROW_FLIGHT=ON --preset ninja-debug-minimal ..
cd ..
ninja
```
Then
```
# build rust:
cd ../arrow-rs
cargo build --all
```
# now from arrow directory run the tests:
```
archery integration --with-cpp=true --with-rust=true
```
Run individual test:
```
(arrow_dev) alamb@MacBook-Pro-2:~/Software/arrow/cpp/build$ /Users/alamb/Software/arrow/cpp/build/debug/flight-test-integration-server -port 49153
Run in rust:
/Users/alamb/Software/arrow/rust/target/debug/flight-test-integration-client --host localhost --port=49153 --path /var/folders/s3/h5hgj43j0bv83shtmz_t_w400000gn/T/arrow-integration-v7xz5285/generated_dictionary_unsigned.json
Repro with this:
cd /Users/alamb/Software/arrow-rs && RUST_LOG=debug RUST_BACKTRACE=1 CARGO_TARGET_DIR=/Users/alamb/Software/df-target cargo run --bin flight-test-integration-client -- --host localhost --port=49153 --path /var/folders/s3/h5hgj43j0bv83shtmz_t_w400000gn/T/arrow-integration-v7xz5285/generated_dictionary_unsigned.json
Results in this
E0304 16:33:43.055262000 123145352572928 hpack_parser.cc:1240] Error parsing metadata: error=invalid value key=:scheme value=grpc
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org