You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "raulcd (via GitHub)" <gi...@apache.org> on 2023/10/19 10:46:07 UTC
[I] [Release][Go] Verification tasks for the Release Candidate fail on Go parquet encryption tests [arrow]
raulcd opened a new issue, #38345:
URL: https://github.com/apache/arrow/issues/38345
### Describe the bug, including details regarding any error messages, version, and platform.
We have created the RC 2 for the Apache Arrow 14.0.0 release and currently we are facing the following error on the release verification for the go sources:
```
/tmp/parquet-encryption-test-3157156869
--- FAIL: TestFileEncryptionDecryption (0.40s)
--- FAIL: TestFileEncryptionDecryption/TestDecryption (0.34s)
--- FAIL: TestFileEncryptionDecryption/TestDecryption/uniform_encryption.parquet.encrypted (0.07s)
--- FAIL: TestFileEncryptionDecryption/TestDecryption/uniform_encryption.parquet.encrypted/config_1 (0.00s)
encryption_read_config_test.go:217:
Error Trace: /tmp/arrow-14.0.0.Rm87k/apache-arrow-14.0.0/go/parquet/encryption_read_config_test.go:217
/tmp/arrow-14.0.0.Rm87k/apache-arrow-14.0.0/go/parquet/encryption_read_config_test.go:406
/tmp/arrow-14.0.0.Rm87k/apache-arrow-14.0.0/go/parquet/encryption_read_config_test.go:430
/tmp/arrow-14.0.0.Rm87k/apache-arrow-14.0.0/go/parquet/encryption_read_config_test.go:468
/tmp/arrow-14.0.0.Rm87k/go/gopath/pkg/mod/github.com/stretchr/testify@v1.8.4/suite/suite.go:112
Error: Not equal:
expected: int(0)
actual : int64(50)
Test: TestFileEncryptionDecryption/TestDecryption/uniform_encryption.parquet.encrypted/config_1
--- FAIL: TestFileEncryptionDecryption/TestDecryption/uniform_encryption.parquet.encrypted/config_3 (0.01s)
encryption_read_config_test.go:217:
Error Trace: /tmp/arrow-14.0.0.Rm87k/apache-arrow-14.0.0/go/parquet/encryption_read_config_test.go:217
/tmp/arrow-14.0.0.Rm87k/apache-arrow-14.0.0/go/parquet/encryption_read_config_test.go:406
/tmp/arrow-14.0.0.Rm87k/apache-arrow-14.0.0/go/parquet/encryption_read_config_test.go:430
/tmp/arrow-14.0.0.Rm87k/apache-arrow-14.0.0/go/parquet/encryption_read_config_test.go:468
/tmp/arrow-14.0.0.Rm87k/go/gopath/pkg/mod/github.com/stretchr/testify@v1.8.4/suite/suite.go:112
Error: Not equal:
expected: int(0)
actual : int64(50)
Test: TestFileEncryptionDecryption/TestDecryption/uniform_encryption.parquet.encrypted/config_3
--- FAIL: TestFileEncryptionDecryption/TestDecryption/encrypt_columns_and_footer.parquet.encrypted (0.06s)
--- FAIL: TestFileEncryptionDecryption/TestDecryption/encrypt_columns_and_footer.parquet.encrypted/config_1 (0.01s)
encryption_read_config_test.go:217:
Error Trace: /tmp/arrow-14.0.0.Rm87k/apache-arrow-14.0.0/go/parquet/encryption_read_config_test.go:217
/tmp/arrow-14.0.0.Rm87k/apache-arrow-14.0.0/go/parquet/encryption_read_config_test.go:406
/tmp/arrow-14.0.0.Rm87k/apache-arrow-14.0.0/go/parquet/encryption_read_config_test.go:430
/tmp/arrow-14.0.0.Rm87k/apache-arrow-14.0.0/go/parquet/encryption_read_config_test.go:468
/tmp/arrow-14.0.0.Rm87k/go/gopath/pkg/mod/github.com/stretchr/testify@v1.8.4/suite/suite.go:112
Error: Not equal:
expected: int(0)
actual : int64(50)
Test: TestFileEncryptionDecryption/TestDecryption/encrypt_columns_and_footer.parquet.encrypted/config_1
--- FAIL: TestFileEncryptionDecryption/TestDecryption/encrypt_columns_and_footer.parquet.encrypted/config_3 (0.01s)
encryption_read_config_test.go:217:
Error Trace: /tmp/arrow-14.0.0.Rm87k/apache-arrow-14.0.0/go/parquet/encryption_read_config_test.go:217
/tmp/arrow-14.0.0.Rm87k/apache-arrow-14.0.0/go/parquet/encryption_read_config_test.go:406
/tmp/arrow-14.0.0.Rm87k/apache-arrow-14.0.0/go/parquet/encryption_read_config_test.go:430
/tmp/arrow-14.0.0.Rm87k/apache-arrow-14.0.0/go/parquet/encryption_read_config_test.go:468
/tmp/arrow-14.0.0.Rm87k/go/gopath/pkg/mod/github.com/stretchr/testify@v1.8.4/suite/suite.go:112
Error: Not equal:
expected: int(0)
actual : int64(50)
Test: TestFileEncryptionDecryption/TestDecryption/encrypt_columns_and_footer.parquet.encrypted/config_3
--- FAIL: TestFileEncryptionDecryption/TestDecryption/encrypt_columns_plaintext_footer.parquet.encrypted (0.08s)
--- FAIL: TestFileEncryptionDecryption/TestDecryption/encrypt_columns_plaintext_footer.parquet.encrypted/config_1 (0.01s)
encryption_read_config_test.go:217:
Error Trace: /tmp/arrow-14.0.0.Rm87k/apache-arrow-14.0.0/go/parquet/encryption_read_config_test.go:217
/tmp/arrow-14.0.0.Rm87k/apache-arrow-14.0.0/go/parquet/encryption_read_config_test.go:406
/tmp/arrow-14.0.0.Rm87k/apache-arrow-14.0.0/go/parquet/encryption_read_config_test.go:430
/tmp/arrow-14.0.0.Rm87k/apache-arrow-14.0.0/go/parquet/encryption_read_config_test.go:468
/tmp/arrow-14.0.0.Rm87k/go/gopath/pkg/mod/github.com/stretchr/testify@v1.8.4/suite/suite.go:112
Error: Not equal:
expected: int(0)
actual : int64(50)
Test: TestFileEncryptionDecryption/TestDecryption/encrypt_columns_plaintext_footer.parquet.encrypted/config_1
--- FAIL: TestFileEncryptionDecryption/TestDecryption/encrypt_columns_plaintext_footer.parquet.encrypted/config_3 (0.00s)
encryption_read_config_test.go:217:
Error Trace: /tmp/arrow-14.0.0.Rm87k/apache-arrow-14.0.0/go/parquet/encryption_read_config_test.go:217
/tmp/arrow-14.0.0.Rm87k/apache-arrow-14.0.0/go/parquet/encryption_read_config_test.go:406
/tmp/arrow-14.0.0.Rm87k/apache-arrow-14.0.0/go/parquet/encryption_read_config_test.go:430
/tmp/arrow-14.0.0.Rm87k/apache-arrow-14.0.0/go/parquet/encryption_read_config_test.go:468
/tmp/arrow-14.0.0.Rm87k/go/gopath/pkg/mod/github.com/stretchr/testify@v1.8.4/suite/suite.go:112
Error: Not equal:
expected: int(0)
actual : int64(50)
Test: TestFileEncryptionDecryption/TestDecryption/encrypt_columns_plaintext_footer.parquet.encrypted/config_3
--- FAIL: TestFileEncryptionDecryption/TestDecryption/encrypt_columns_plaintext_footer.parquet.encrypted/config_4 (0.00s)
encryption_read_config_test.go:217:
Error Trace: /tmp/arrow-14.0.0.Rm87k/apache-arrow-14.0.0/go/parquet/encryption_read_config_test.go:217
/tmp/arrow-14.0.0.Rm87k/apache-arrow-14.0.0/go/parquet/encryption_read_config_test.go:406
/tmp/arrow-14.0.0.Rm87k/apache-arrow-14.0.0/go/parquet/encryption_read_config_test.go:430
/tmp/arrow-14.0.0.Rm87k/apache-arrow-14.0.0/go/parquet/encryption_read_config_test.go:468
/tmp/arrow-14.0.0.Rm87k/go/gopath/pkg/mod/github.com/stretchr/testify@v1.8.4/suite/suite.go:112
Error: Not equal:
expected: int(0)
actual : int64(50)
Test: TestFileEncryptionDecryption/TestDecryption/encrypt_columns_plaintext_footer.parquet.encrypted/config_4
--- FAIL: TestFileEncryptionDecryption/TestDecryption/encrypt_columns_and_footer_aad.parquet.encrypted (0.06s)
--- FAIL: TestFileEncryptionDecryption/TestDecryption/encrypt_columns_and_footer_aad.parquet.encrypted/config_1 (0.00s)
encryption_read_config_test.go:217:
Error Trace: /tmp/arrow-14.0.0.Rm87k/apache-arrow-14.0.0/go/parquet/encryption_read_config_test.go:217
/tmp/arrow-14.0.0.Rm87k/apache-arrow-14.0.0/go/parquet/encryption_read_config_test.go:406
/tmp/arrow-14.0.0.Rm87k/apache-arrow-14.0.0/go/parquet/encryption_read_config_test.go:430
/tmp/arrow-14.0.0.Rm87k/apache-arrow-14.0.0/go/parquet/encryption_read_config_test.go:468
/tmp/arrow-14.0.0.Rm87k/go/gopath/pkg/mod/github.com/stretchr/testify@v1.8.4/suite/suite.go:112
Error: Not equal:
expected: int(0)
actual : int64(50)
Test: TestFileEncryptionDecryption/TestDecryption/encrypt_columns_and_footer_aad.parquet.encrypted/config_1
--- FAIL: TestFileEncryptionDecryption/TestDecryption/encrypt_columns_and_footer_aad.parquet.encrypted/config_2 (0.00s)
encryption_read_config_test.go:217:
Error Trace: /tmp/arrow-14.0.0.Rm87k/apache-arrow-14.0.0/go/parquet/encryption_read_config_test.go:217
/tmp/arrow-14.0.0.Rm87k/apache-arrow-14.0.0/go/parquet/encryption_read_config_test.go:406
/tmp/arrow-14.0.0.Rm87k/apache-arrow-14.0.0/go/parquet/encryption_read_config_test.go:430
/tmp/arrow-14.0.0.Rm87k/apache-arrow-14.0.0/go/parquet/encryption_read_config_test.go:468
/tmp/arrow-14.0.0.Rm87k/go/gopath/pkg/mod/github.com/stretchr/testify@v1.8.4/suite/suite.go:112
Error: Not equal:
expected: int(0)
actual : int64(50)
Test: TestFileEncryptionDecryption/TestDecryption/encrypt_columns_and_footer_aad.parquet.encrypted/config_2
--- FAIL: TestFileEncryptionDecryption/TestDecryption/encrypt_columns_and_footer_aad.parquet.encrypted/config_3 (0.00s)
encryption_read_config_test.go:217:
Error Trace: /tmp/arrow-14.0.0.Rm87k/apache-arrow-14.0.0/go/parquet/encryption_read_config_test.go:217
/tmp/arrow-14.0.0.Rm87k/apache-arrow-14.0.0/go/parquet/encryption_read_config_test.go:406
/tmp/arrow-14.0.0.Rm87k/apache-arrow-14.0.0/go/parquet/encryption_read_config_test.go:430
/tmp/arrow-14.0.0.Rm87k/apache-arrow-14.0.0/go/parquet/encryption_read_config_test.go:468
/tmp/arrow-14.0.0.Rm87k/go/gopath/pkg/mod/github.com/stretchr/testify@v1.8.4/suite/suite.go:112
Error: Not equal:
expected: int(0)
actual : int64(50)
Test: TestFileEncryptionDecryption/TestDecryption/encrypt_columns_and_footer_aad.parquet.encrypted/config_3
--- FAIL: TestFileEncryptionDecryption/TestDecryption/encrypt_columns_and_footer_disable_aad_storage.parquet.encrypted (0.02s)
--- FAIL: TestFileEncryptionDecryption/TestDecryption/encrypt_columns_and_footer_disable_aad_storage.parquet.encrypted/config_2 (0.00s)
encryption_read_config_test.go:217:
Error Trace: /tmp/arrow-14.0.0.Rm87k/apache-arrow-14.0.0/go/parquet/encryption_read_config_test.go:217
/tmp/arrow-14.0.0.Rm87k/apache-arrow-14.0.0/go/parquet/encryption_read_config_test.go:406
/tmp/arrow-14.0.0.Rm87k/apache-arrow-14.0.0/go/parquet/encryption_read_config_test.go:430
/tmp/arrow-14.0.0.Rm87k/apache-arrow-14.0.0/go/parquet/encryption_read_config_test.go:468
/tmp/arrow-14.0.0.Rm87k/go/gopath/pkg/mod/github.com/stretchr/testify@v1.8.4/suite/suite.go:112
Error: Not equal:
expected: int(0)
actual : int64(50)
Test: TestFileEncryptionDecryption/TestDecryption/encrypt_columns_and_footer_disable_aad_storage.parquet.encrypted/config_2
--- FAIL: TestFileEncryptionDecryption/TestDecryption/encrypt_columns_and_footer_ctr.parquet.encrypted (0.05s)
--- FAIL: TestFileEncryptionDecryption/TestDecryption/encrypt_columns_and_footer_ctr.parquet.encrypted/config_1 (0.00s)
encryption_read_config_test.go:217:
Error Trace: /tmp/arrow-14.0.0.Rm87k/apache-arrow-14.0.0/go/parquet/encryption_read_config_test.go:217
/tmp/arrow-14.0.0.Rm87k/apache-arrow-14.0.0/go/parquet/encryption_read_config_test.go:406
/tmp/arrow-14.0.0.Rm87k/apache-arrow-14.0.0/go/parquet/encryption_read_config_test.go:430
/tmp/arrow-14.0.0.Rm87k/apache-arrow-14.0.0/go/parquet/encryption_read_config_test.go:468
/tmp/arrow-14.0.0.Rm87k/go/gopath/pkg/mod/github.com/stretchr/testify@v1.8.4/suite/suite.go:112
Error: Not equal:
expected: int(0)
actual : int64(50)
Test: TestFileEncryptionDecryption/TestDecryption/encrypt_columns_and_footer_ctr.parquet.encrypted/config_1
--- FAIL: TestFileEncryptionDecryption/TestDecryption/encrypt_columns_and_footer_ctr.parquet.encrypted/config_3 (0.01s)
encryption_read_config_test.go:217:
Error Trace: /tmp/arrow-14.0.0.Rm87k/apache-arrow-14.0.0/go/parquet/encryption_read_config_test.go:217
/tmp/arrow-14.0.0.Rm87k/apache-arrow-14.0.0/go/parquet/encryption_read_config_test.go:406
/tmp/arrow-14.0.0.Rm87k/apache-arrow-14.0.0/go/parquet/encryption_read_config_test.go:430
/tmp/arrow-14.0.0.Rm87k/apache-arrow-14.0.0/go/parquet/encryption_read_config_test.go:468
/tmp/arrow-14.0.0.Rm87k/go/gopath/pkg/mod/github.com/stretchr/testify@v1.8.4/suite/suite.go:112
Error: Not equal:
expected: int(0)
actual : int64(50)
Test: TestFileEncryptionDecryption/TestDecryption/encrypt_columns_and_footer_ctr.parquet.encrypted/config_3
/tmp/parquet-encryption-test-3429813239
FAIL
FAIL github.com/apache/arrow/go/v14/parquet 0.464s
```
This is reproducible locally with:
```
archery docker run -e VERIFY_VERSION="14.0.0" -e VERIFY_RC="2" -e TEST_DEFAULT=0 -e TEST_GO=1 almalinux-verify-rc
```
We didn't noticed before because this doesn't fail with non-official releases (local source):
```
archery docker run -e VERIFY_VERSION="" -e VERIFY_RC="" -e TEST_DEFAULT=0 -e TEST_GO=1 almalinux-verify-rc
```
See error here: https://github.com/ursacomputing/crossbow/actions/runs/6572466559/job/17853633830
And workflow file: https://github.com/ursacomputing/crossbow/actions/runs/6572466559/workflow
Successful one on maintenance branch where we test from local: https://github.com/ursacomputing/crossbow/actions/runs/6572711355/job/17854324601
And workflow file: https://github.com/ursacomputing/crossbow/actions/runs/6572711355/workflow
### Component(s)
Go, Release
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Release][Go] Verification tasks for the Release Candidate fail on Go parquet encryption tests [arrow]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on issue #38345:
URL: https://github.com/apache/arrow/issues/38345#issuecomment-1771617282
I was able to reproduce the failure locally by manually and forcibly updating the `parquet-testing` submodule. It looks like a recent update there wrote some (or all) of the files so that the boolean columns used RLE encoding instead of plain encoding which the Go impl doesn't seem to support at the moment. Gonna take a quick look because it shouldn't be *too* difficult to add and test to confirm that it will work for the verification tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Release][Go] Verification tasks for the Release Candidate fail on Go parquet encryption tests [arrow]
Posted by "raulcd (via GitHub)" <gi...@apache.org>.
raulcd closed issue #38345: [Release][Go] Verification tasks for the Release Candidate fail on Go parquet encryption tests
URL: https://github.com/apache/arrow/issues/38345
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Release][Go] Verification tasks for the Release Candidate fail on Go parquet encryption tests [arrow]
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on issue #38345:
URL: https://github.com/apache/arrow/issues/38345#issuecomment-1771671690
How about this?
```diff
diff --git a/dev/release/verify-release-candidate.sh b/dev/release/verify-release-candidate.sh
index 0c6ac075b..287c557fb 100755
--- a/dev/release/verify-release-candidate.sh
+++ b/dev/release/verify-release-candidate.sh
@@ -959,12 +959,26 @@ ensure_source_directory() {
fi
fi
- # Ensure that the testing repositories are cloned
- if [ ! -d "${ARROW_SOURCE_DIR}/testing/data" ]; then
- git clone https://github.com/apache/arrow-testing.git ${ARROW_SOURCE_DIR}/testing
+ # Ensure that the testing repositories are prepared
+ if [ ! -d ${ARROW_SOURCE_DIR}/testing/data ]; then
+ if [ -d ${SOURCE_DIR}/../../testing/data ]; then
+ cp -a ${SOURCE_DIR}/../../testing/ ${ARROW_SOURCE_DIR}/
+ else
+ git clone \
+ https://github.com/apache/arrow-testing.git \
+ ${ARROW_SOURCE_DIR}/testing
+ fi
fi
- if [ ! -d "${ARROW_SOURCE_DIR}/cpp/submodules/parquet-testing/data" ]; then
- git clone https://github.com/apache/parquet-testing.git ${ARROW_SOURCE_DIR}/cpp/submodules/parquet-testing
+ if [ ! -d ${ARROW_SOURCE_DIR}/cpp/submodules/parquet-testing/data ]; then
+ if [ -d ${SOURCE_DIR}/../../cpp/submodules/parquet-testing/data ]; then
+ cp -a \
+ ${SOURCE_DIR}/../../cpp/submodules/parquet-testing/ \
+ ${ARROW_SOURCE_DIR}/cpp/submodules/
+ else
+ git clone \
+ https://github.com/apache/parquet-testing.git \
+ ${ARROW_SOURCE_DIR}/cpp/submodules/parquet-testing
+ fi
fi
export ARROW_TEST_DATA=$ARROW_SOURCE_DIR/testing/data
```
(I think that this is not a blocker.)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Release][Go] Verification tasks for the Release Candidate fail on Go parquet encryption tests [arrow]
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on issue #38345:
URL: https://github.com/apache/arrow/issues/38345#issuecomment-1771622406
I also think that https://github.com/apache/parquet-testing/pull/39 is related.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Release][Go] Verification tasks for the Release Candidate fail on Go parquet encryption tests [arrow]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on issue #38345:
URL: https://github.com/apache/arrow/issues/38345#issuecomment-1771742873
That's definitely a viable solution, i also just put together a PR to implement the Boolean RLE encoding...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Release][Go] Verification tasks for the Release Candidate fail on Go parquet encryption tests [arrow]
Posted by "raulcd (via GitHub)" <gi...@apache.org>.
raulcd commented on issue #38345:
URL: https://github.com/apache/arrow/issues/38345#issuecomment-1770554806
@zeroshade @kou I am investigating but I am really confused on why this only fails if we are verifying the Release from an official release but it is passing if we run the verification tasks from a local checkout. As reported on the issue this is reproducible locally if we use:
`archery docker run -e VERIFY_VERSION="14.0.0" -e VERIFY_RC="2" -e TEST_DEFAULT=0 -e TEST_GO=1 almalinux-verify-rc` but is successful if we use `archery docker run -e VERIFY_VERSION="" -e VERIFY_RC="" -e TEST_DEFAULT=0 -e TEST_GO=1 almalinux-verify-rc` with the RC branch
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org