You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "Michael-J-Ward (via GitHub)" <gi...@apache.org> on 2024/08/12 18:31:25 UTC
[I] Docs: Running unit-tests against a release candidate [datafusion-python]
Michael-J-Ward opened a new issue, #804:
URL: https://github.com/apache/datafusion-python/issues/804
## Goal: Document how to run the unit-tests against a release candidate
Previously, I had simply checked out the repo at the release tag, built `datafusion-python` with `maturin develop`, and ran the tests.
This morning, I tried running the unit-tests strictly against the released candidate of `datafusion-python` hosted on `test.pypi.org`.
The original attempt failed in a surprising way, so I'll share what I tried and found.
Please don't hesitate to correct me or suggest a better way to do something.
---
## Setting up the attempt to verify
```
# clone a fresh repo - Important to start fresh instead of in your existing repo
git clone https://github.com/apache/datafusion-python.git
cd datafusion-python
# checkout the release commit
git fetch --tags
git checkout 40.0.0-rc1
# create the env
python3 -m venv venv
source venv/bin/activate
# install release candidate
pip install --extra-index-url https://test.pypi.org/simple/ datafusion==40.0.0
pip install pytest
```
## Problem 1) The location of the tests causes it to look in the `datafusion` source code instead of the installed `datafusion` package
Notice, the error originates from our local repo code `python/datafusion/__init__.py:29` instead of the installed candidate `venv/lib/python3.11/site-packages/datafusion/__init__.py:29`
```
❯ pytest python/datafusion/tests/test_wrapper_coverage.py
ImportError while loading conftest '/home/mike/workspace/tmp/datafusion-python/python/datafusion/tests/conftest.py'.
python/datafusion/__init__.py:29: in <module>
from .context import (
python/datafusion/context.py:22: in <module>
from ._internal import SessionConfig as SessionConfigInternal
E ModuleNotFoundError: No module named 'datafusion._internal'
```
This is why the setup starts with cloning a fresh repo instead of re-using your existing `datafusion-python` clone. Your local clone almost certainly has a `./python/datafusion/_internal.abi3.so` created from a previous `maturin develop` run - this doesn't get cleaned up with `cargo clean`, and `maturin` doesn't appear to have a `clean` command that removes it.
This was the source of my original "surprising" test failure. My `_internal.abi3.so` was from my work on #802, which made it appear as if the candidate-release had been compiled against the wrong datafusion version.
## Problem 2) `typing_extensions` is not installed as a dependency
After moving the tests repo, running the test uses the installed `datafusion` package (`venv/lib/python3.11/site-packages/datafusion/__init__.py`), but `typing_extensions` is not installed as a dependency when we ran `pip install --extra-index-url https://test.pypi.org/simple/ datafusion==40.0.0`.
```
❯ mv python/datafusion/tests python-tests
❯ pytest python-tests/test_wrapper_coverage.py
ImportError while loading conftest '/home/mike/workspace/tmp/datafusion-python/python-tests/conftest.py'.
python-tests/conftest.py:19: in <module>
from datafusion import SessionContext
venv/lib/python3.11/site-packages/datafusion/__init__.py:29: in <module>
from .context import (
venv/lib/python3.11/site-packages/datafusion/context.py:30: in <module>
from datafusion.dataframe import DataFrame
venv/lib/python3.11/site-packages/datafusion/dataframe.py:26: in <module>
from typing_extensions import deprecated
E ModuleNotFoundError: No module named 'typing_extensions'
```
So we install the dependency, re-run the test and now it's green.
```
❯ pip install typing_extensions
Collecting typing_extensions
Using cached typing_extensions-4.12.2-py3-none-any.whl.metadata (3.0 kB)
Using cached typing_extensions-4.12.2-py3-none-any.whl (37 kB)
Installing collected packages: typing_extensions
Successfully installed typing_extensions-4.12.2
❯ pytest python-tests/test_wrapper_coverage.py
========================================================================================== test session starts ===========================================================================================
platform linux -- Python 3.11.9, pytest-8.3.2, pluggy-1.5.0
rootdir: /home/mike/workspace/tmp/datafusion-python
configfile: pyproject.toml
collected 1 item
python-tests/test_wrapper_coverage.py . [100%]
=========================================================================================== 1 passed in 0.04s ============================================================================================
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Docs: Running unit-tests against a release candidate [datafusion-python]
Posted by "Michael-J-Ward (via GitHub)" <gi...@apache.org>.
Michael-J-Ward commented on issue #804:
URL: https://github.com/apache/datafusion-python/issues/804#issuecomment-2284802424
Lastly, because our tests use relative imports for test-helpers, we need to make the test directory a proper package and we can run the entire test suite
```
mv python-tests python_tests
touch python_tests/__init__.py
pytest python_tests/
```
```
❯ pytest python_tests/
========================================================================================== test session starts ===========================================================================================
platform linux -- Python 3.11.9, pytest-8.3.2, pluggy-1.5.0
rootdir: /home/mike/workspace/tmp/datafusion-python
configfile: pyproject.toml
collected 338 items
python_tests/test_aggregation.py ......................... [ 7%]
python_tests/test_catalog.py . [ 7%]
python_tests/test_config.py ... [ 8%]
python_tests/test_context.py .................................... [ 19%]
python_tests/test_dataframe.py .............................................................................................. [ 47%]
python_tests/test_expr.py ....... [ 49%]
python_tests/test_functions.py ..................................................................................................................... [ 83%]
python_tests/test_imports.py ..... [ 85%]
python_tests/test_indexing.py .. [ 85%]
python_tests/test_input.py . [ 86%]
python_tests/test_sql.py ......................XxXXXxxxx........ [ 97%]
python_tests/test_store.py . [ 97%]
python_tests/test_substrait.py ... [ 98%]
python_tests/test_udaf.py s.. [ 99%]
python_tests/test_wrapper_coverage.py . [100%]
========================================================================== 328 passed, 1 skipped, 5 xfailed, 4 xpassed in 0.70s ==========================================================================
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org