You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2022/05/13 10:05:51 UTC
[GitHub] [arrow] raulcd opened a new pull request, #13147: ARROW-16548: [Python] Add pytest.mark.parquet to all tests under tests/parquet package
raulcd opened a new pull request, #13147:
URL: https://github.com/apache/arrow/pull/13147
If we built arrow and pyarrow without PARQUET and tried to run the PARQUET tests. The first test being executed was not correctly marked.
```
> /home/raulcd/open_source/arrow/python/pyarrow/tests/conftest.py(244)pytest_runtest_setup()
-> for mark in item.iter_markers():
(Pdb) item
<Function test_parquet_invalid_version>
(Pdb) [x for x in item.iter_markers()]
[]
(Pdb)
```
Meaning the test was not correctly skipped. This was found on the minimal builds PR: https://github.com/apache/arrow/pull/13113 on this job failures: https://github.com/ursacomputing/crossbow/runs/6407176338?check_suite_focus=true
All tests under the parquet package are on the structure:
- parquet
├── test_basic.py
├── test_compliant_nested_type.py
├── test_dataset.py
├── test_data_types.py
├── test_datetime.py
├── test_encryption.py
├── test_metadata.py
├── test_pandas.py
├── test_parquet_file.py
└── test_parquet_writer.py
The implementation marks all the individual tests that are on this structure with the parquet dataset mark correctly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #13147: ARROW-16548: [Python] Add pytest.mark.parquet to all tests under tests/parquet package
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #13147:
URL: https://github.com/apache/arrow/pull/13147#issuecomment-1125876416
https://issues.apache.org/jira/browse/ARROW-16548
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #13147: ARROW-16548: [Python] Add pytest.mark.parquet to all tests under tests/parquet package
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #13147:
URL: https://github.com/apache/arrow/pull/13147#issuecomment-1125876445
:warning: Ticket **has not been started in JIRA**, please click 'Start Progress'.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] raulcd commented on pull request #13147: ARROW-16548: [Python] Add pytest.mark.parquet to all tests under tests/parquet package
Posted by GitBox <gi...@apache.org>.
raulcd commented on PR #13147:
URL: https://github.com/apache/arrow/pull/13147#issuecomment-1125893203
@jorisvandenbossche other solutions that fix this issue could be:
* Add the `pytestmark = pytest.mark.parquet` to each test file individually instead of to the common file
* Add the `pytestmark` import (even when we are not using it) on each individual test file (`from pyarrow.tests.parquet.common import pytestmark`)
I was going in a rabbit hole on importlib and load_module but I don't think is worth the effort to spend more time on this having three possible solutions.
One nice thing is I had to go back to [a talk I gave at PyCon Spain](https://github.com/raulcd/import_magic/blob/master/magic_import_v2.ipynb) on how import works :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jorisvandenbossche commented on a diff in pull request #13147: ARROW-16548: [Python] Add pytest.mark.parquet to all tests under tests/parquet package
Posted by GitBox <gi...@apache.org>.
jorisvandenbossche commented on code in PR #13147:
URL: https://github.com/apache/arrow/pull/13147#discussion_r875953997
##########
python/pyarrow/tests/parquet/test_encryption.py:
##########
@@ -38,7 +38,9 @@
# Marks all of the tests in this module
# Ignore these with pytest ... -m 'not parquet_encryption'
+# Ignore these with pytest ... -m 'not parquet'
pytestmark = pytest.mark.parquet_encryption
+pytestmark = pytest.mark.parquet
Review Comment:
Ah, we can't assign two marks this way (it is redefining the same variable), it should be something like `pytestmark = [..., ...]` I think
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] ursabot commented on pull request #13147: ARROW-16548: [Python] Add pytest.mark.parquet to all tests under tests/parquet package
Posted by GitBox <gi...@apache.org>.
ursabot commented on PR #13147:
URL: https://github.com/apache/arrow/pull/13147#issuecomment-1129037603
Benchmark runs are scheduled for baseline = 52a051b1f01669647d8f4d16d7de30c10265dc51 and contender = c032290b9ea2699ce29f4fa26e6826911e13fcca. c032290b9ea2699ce29f4fa26e6826911e13fcca is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Finished :arrow_down:0.0% :arrow_up:0.0%] [ec2-t3-xlarge-us-east-2](https://conbench.ursa.dev/compare/runs/42f123755c914c3f8cfe6d152f75a9c0...fa08991e35424cb7a6e21757bcafcd10/)
[Finished :arrow_down:0.08% :arrow_up:0.0%] [test-mac-arm](https://conbench.ursa.dev/compare/runs/300f257d53144ae0a8c30279f52970c6...4f95789754a24c6e992f12429f13049a/)
[Finished :arrow_down:0.36% :arrow_up:0.0%] [ursa-i9-9960x](https://conbench.ursa.dev/compare/runs/e4ba7303a61941f6952819c649a6a485...d7dd40c9646f43f2a852a9b44fa1e345/)
[Finished :arrow_down:0.67% :arrow_up:0.0%] [ursa-thinkcentre-m75q](https://conbench.ursa.dev/compare/runs/fa646987e15f4f63953f533431b34366...365143f837834c96ad5a985dd4fc477c/)
Buildkite builds:
[Finished] [`c032290b` ec2-t3-xlarge-us-east-2](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ec2-t3-xlarge-us-east-2/builds/774)
[Finished] [`c032290b` test-mac-arm](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-test-mac-arm/builds/771)
[Finished] [`c032290b` ursa-i9-9960x](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ursa-i9-9960x/builds/761)
[Finished] [`c032290b` ursa-thinkcentre-m75q](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ursa-thinkcentre-m75q/builds/777)
[Finished] [`52a051b1` ec2-t3-xlarge-us-east-2](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ec2-t3-xlarge-us-east-2/builds/773)
[Finished] [`52a051b1` test-mac-arm](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-test-mac-arm/builds/770)
[Finished] [`52a051b1` ursa-i9-9960x](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ursa-i9-9960x/builds/760)
[Finished] [`52a051b1` ursa-thinkcentre-m75q](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ursa-thinkcentre-m75q/builds/776)
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python, R. Runs only benchmarks with cloud = True
test-mac-arm: Supported benchmark langs: C++, Python, R
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jorisvandenbossche closed pull request #13147: ARROW-16548: [Python] Add pytest.mark.parquet to all tests under tests/parquet package
Posted by GitBox <gi...@apache.org>.
jorisvandenbossche closed pull request #13147: ARROW-16548: [Python] Add pytest.mark.parquet to all tests under tests/parquet package
URL: https://github.com/apache/arrow/pull/13147
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] raulcd commented on pull request #13147: ARROW-16548: [Python] Add pytest.mark.parquet to all tests under tests/parquet package
Posted by GitBox <gi...@apache.org>.
raulcd commented on PR #13147:
URL: https://github.com/apache/arrow/pull/13147#issuecomment-1126027469
Thanks @jorisvandenbossche . I was able to reproduce the issue on each one of the test files individually. This solution solves the test failures when PARQUET is not enabled.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jorisvandenbossche commented on pull request #13147: ARROW-16548: [Python] Add pytest.mark.parquet to all tests under tests/parquet package
Posted by GitBox <gi...@apache.org>.
jorisvandenbossche commented on PR #13147:
URL: https://github.com/apache/arrow/pull/13147#issuecomment-1125936799
Hmm, I would actually have expected that the `pytestmark = ..` in the `tests/parquet/__init__.py` would ensure this applies that mark to all the tests in that directory (and not the one in common.py).
But if that seems to not work correctly in practice, I think adding `pytestmark = pytest.mark.parquet` to every file might be the most "low tech" solution?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org