You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2021/01/06 16:35:04 UTC
[GitHub] [arrow] arw2019 opened a new pull request #9113: ARROW-11147: [CI][Python] pin pandas=0.25.3 for dask-master
arw2019 opened a new pull request #9113:
URL: https://github.com/apache/arrow/pull/9113
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jorisvandenbossche closed pull request #9113: ARROW-11147: [CI][Python] Remove pandas=0.25.3 pin for dask-latest
Posted by GitBox <gi...@apache.org>.
jorisvandenbossche closed pull request #9113:
URL: https://github.com/apache/arrow/pull/9113
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jorisvandenbossche commented on pull request #9113: ARROW-11147: [CI][Python] pin pandas=0.25.3 for dask-master
Posted by GitBox <gi...@apache.org>.
jorisvandenbossche commented on pull request #9113:
URL: https://github.com/apache/arrow/pull/9113#issuecomment-766842198
@github-actions crossbow submit test-conda-python-3.8-dask-master test-conda-python-3.7-dask-latest
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #9113: ARROW-11147: [CI][Python] pin pandas=0.25.3 for dask-master
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #9113:
URL: https://github.com/apache/arrow/pull/9113#issuecomment-756127437
Revision: 6a79a83a77ecc95efc8f7c83d4e5aff0e2de5490
Submitted crossbow builds: [ursa-labs/crossbow @ actions-830](https://github.com/ursa-labs/crossbow/branches/all?query=actions-830)
|Task|Status|
|----|------|
|test-conda-python-3.7-dask-latest|[](https://github.com/ursa-labs/crossbow/actions?query=branch:actions-830-github-test-conda-python-3.7-dask-latest)|
|test-conda-python-3.8-dask-master|[](https://github.com/ursa-labs/crossbow/actions?query=branch:actions-830-github-test-conda-python-3.8-dask-master)|
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jorisvandenbossche commented on pull request #9113: ARROW-11147: [CI][Python] pin pandas=0.25.3 for dask-master
Posted by GitBox <gi...@apache.org>.
jorisvandenbossche commented on pull request #9113:
URL: https://github.com/apache/arrow/pull/9113#issuecomment-756615870
> tried deleting the last line of the dockerfile for now, will look more tomorrow
Yeah, but then I think we are back where we started: dask is failing with pandas 1.2.0.
Personally, if this turns out more difficult as expected, I wouldn't put too much more effort in it, I should rather try to finish https://github.com/dask/dask/pull/6996/ (so dask's tests pass with pandas 1.2.0)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jorisvandenbossche commented on pull request #9113: ARROW-11147: [CI][Python] Remove pandas=0.25.3 pin for dask-latest
Posted by GitBox <gi...@apache.org>.
jorisvandenbossche commented on pull request #9113:
URL: https://github.com/apache/arrow/pull/9113#issuecomment-766842929
The tests are now fixed on the dask side, so normally removing the pin altogether should now be fine (and updated the title to refect this)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jorisvandenbossche commented on pull request #9113: ARROW-11147: [CI][Python] Remove pandas=0.25.3 pin for dask-latest
Posted by GitBox <gi...@apache.org>.
jorisvandenbossche commented on pull request #9113:
URL: https://github.com/apache/arrow/pull/9113#issuecomment-766957931
Thanks @arw2019 !
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] arw2019 commented on pull request #9113: ARROW-11147: [CI][Python] pin pandas=0.25.3 for dask-master
Posted by GitBox <gi...@apache.org>.
arw2019 commented on pull request #9113:
URL: https://github.com/apache/arrow/pull/9113#issuecomment-756236583
dask-master still failing but for a different reason (looks pandas related)
```
/
Traceback (most recent call last):
File "/opt/conda/envs/arrow/lib/python3.8/site-packages/dask/dataframe/__init__.py", line 2, in <module>
from .core import (
File "/opt/conda/envs/arrow/lib/python3.8/site-packages/dask/dataframe/core.py", line 10, in <module>
import pandas as pd
File "/opt/conda/envs/arrow/lib/python3.8/site-packages/pandas/__init__.py", line 55, in <module>
from pandas.core.api import (
File "/opt/conda/envs/arrow/lib/python3.8/site-packages/pandas/core/api.py", line 5, in <module>
from pandas.core.arrays.integer import (
File "/opt/conda/envs/arrow/lib/python3.8/site-packages/pandas/core/arrays/__init__.py", line 13, in <module>
from .sparse import SparseArray # noqa: F401
File "/opt/conda/envs/arrow/lib/python3.8/site-packages/pandas/core/arrays/sparse/__init__.py", line 3, in <module>
from pandas.core.arrays.sparse.accessor import SparseAccessor, SparseFrameAccessor
File "/opt/conda/envs/arrow/lib/python3.8/site-packages/pandas/core/arrays/sparse/accessor.py", line 10, in <module>
from pandas.core.arrays.sparse.array import SparseArray
File "/opt/conda/envs/arrow/lib/python3.8/site-packages/pandas/core/arrays/sparse/array.py", line 20, in <module>
from pandas.core.dtypes.cast import (
ImportError: cannot import name 'maybe_box_datetimelike' from 'pandas.core.dtypes.cast' (/opt/conda/envs/arrow/lib/python3.8/site-packages/pandas/core/dtypes/cast.py)
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/opt/conda/envs/arrow/lib/python3.8/site-packages/dask/dataframe/__init__.py", line 57, in <module>
raise ImportError(msg) from e
ImportError: Dask dataframe requirements are not installed.
Please either conda or pip install as follows:
conda install dask # either conda install
python -m pip install "dask[dataframe]" --upgrade # or python -m pip install
Error: `docker-compose --file /home/runner/work/crossbow/crossbow/arrow/docker-compose.yml run --rm -e SETUPTOOLS_SCM_PRETEND_VERSION=2.1.0.dev543 conda-python-dask` exited with a non-zero exit code 1, see the process log above.
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jorisvandenbossche commented on pull request #9113: ARROW-11147: [CI][Python] pin pandas=0.25.3 for dask-master
Posted by GitBox <gi...@apache.org>.
jorisvandenbossche commented on pull request #9113:
URL: https://github.com/apache/arrow/pull/9113#issuecomment-756615870
> tried deleting the last line of the dockerfile for now, will look more tomorrow
Yeah, but then I think we are back where we started: dask is failing with pandas 1.2.0.
Personally, if this turns out more difficult as expected, I wouldn't put too much more effort in it, I should rather try to finish https://github.com/dask/dask/pull/6996/ (so dask's tests pass with pandas 1.2.0)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #9113: ARROW-11147: [CI][Python] pin pandas=0.25.3 for dask-master
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #9113:
URL: https://github.com/apache/arrow/pull/9113#issuecomment-756217688
Revision: 6a79a83a77ecc95efc8f7c83d4e5aff0e2de5490
Submitted crossbow builds: [ursa-labs/crossbow @ actions-834](https://github.com/ursa-labs/crossbow/branches/all?query=actions-834)
|Task|Status|
|----|------|
|test-conda-python-3.7-dask-latest|[](https://github.com/ursa-labs/crossbow/actions?query=branch:actions-834-github-test-conda-python-3.7-dask-latest)|
|test-conda-python-3.8-dask-master|[](https://github.com/ursa-labs/crossbow/actions?query=branch:actions-834-github-test-conda-python-3.8-dask-master)|
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] arw2019 commented on a change in pull request #9113: ARROW-11147: [CI][Python] pin pandas=0.25.3 for dask-master
Posted by GitBox <gi...@apache.org>.
arw2019 commented on a change in pull request #9113:
URL: https://github.com/apache/arrow/pull/9113#discussion_r553451810
##########
File path: ci/docker/conda-python-dask.dockerfile
##########
@@ -25,4 +25,4 @@ COPY ci/scripts/install_dask.sh /arrow/ci/scripts/
RUN /arrow/ci/scripts/install_dask.sh ${dask}
# The Spark tests currently break with pandas >= 1.0
Review comment:
xref https://github.com/apache/arrow/pull/9115#discussion_r553045953 we should be able to bump the pandas version to >= 1.0 in case that helps with the build failure
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #9113: ARROW-11147: [CI][Python] Remove pandas=0.25.3 pin for dask-latest
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #9113:
URL: https://github.com/apache/arrow/pull/9113#issuecomment-766842718
Revision: 872e082c4de957c1db82c0ea0f94c4009c866494
Submitted crossbow builds: [ursacomputing/crossbow @ actions-45](https://github.com/ursacomputing/crossbow/branches/all?query=actions-45)
|Task|Status|
|----|------|
|test-conda-python-3.7-dask-latest|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-45-github-test-conda-python-3.7-dask-latest)|
|test-conda-python-3.8-dask-master|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-45-github-test-conda-python-3.8-dask-master)|
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jorisvandenbossche commented on a change in pull request #9113: ARROW-11147: [CI][Python] pin pandas=0.25.3 for dask-master
Posted by GitBox <gi...@apache.org>.
jorisvandenbossche commented on a change in pull request #9113:
URL: https://github.com/apache/arrow/pull/9113#discussion_r553559574
##########
File path: ci/docker/conda-python-dask.dockerfile
##########
@@ -25,4 +25,4 @@ COPY ci/scripts/install_dask.sh /arrow/ci/scripts/
RUN /arrow/ci/scripts/install_dask.sh ${dask}
# The Spark tests currently break with pandas >= 1.0
Review comment:
yeah, you could try pandas 1.0.0, because I think both 1.1 and 1.2 will give failures with dask at the moment (my open PR has fixes for both)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jorisvandenbossche commented on pull request #9113: ARROW-11147: [CI][Python] pin pandas=0.25.3 for dask-master
Posted by GitBox <gi...@apache.org>.
jorisvandenbossche commented on pull request #9113:
URL: https://github.com/apache/arrow/pull/9113#issuecomment-756118996
@github-actions crossbow submit test-conda-python-3.8-dask-master test-conda-python-3.7-dask-latest
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #9113: ARROW-11147: [CI][Python] Remove pandas=0.25.3 pin for dask-latest
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #9113:
URL: https://github.com/apache/arrow/pull/9113#issuecomment-766842718
Revision: 872e082c4de957c1db82c0ea0f94c4009c866494
Submitted crossbow builds: [ursacomputing/crossbow @ actions-45](https://github.com/ursacomputing/crossbow/branches/all?query=actions-45)
|Task|Status|
|----|------|
|test-conda-python-3.7-dask-latest|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-45-github-test-conda-python-3.7-dask-latest)|
|test-conda-python-3.8-dask-master|[](https://github.com/ursacomputing/crossbow/actions?query=branch:actions-45-github-test-conda-python-3.8-dask-master)|
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jorisvandenbossche commented on pull request #9113: ARROW-11147: [CI][Python] pin pandas=0.25.3 for dask-master
Posted by GitBox <gi...@apache.org>.
jorisvandenbossche commented on pull request #9113:
URL: https://github.com/apache/arrow/pull/9113#issuecomment-755430432
@github-actions crossbow submit test-conda-python-3.8-dask-master test-conda-python-3.7-dask-latest
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jorisvandenbossche commented on pull request #9113: ARROW-11147: [CI][Python] pin pandas=0.25.3 for dask-master
Posted by GitBox <gi...@apache.org>.
jorisvandenbossche commented on pull request #9113:
URL: https://github.com/apache/arrow/pull/9113#issuecomment-766842198
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] arw2019 commented on pull request #9113: ARROW-11147: [CI][Python] pin pandas=0.25.3 for dask-master
Posted by GitBox <gi...@apache.org>.
arw2019 commented on pull request #9113:
URL: https://github.com/apache/arrow/pull/9113#issuecomment-755869838
@github-actions crossbow submit test-conda-python-3.8-dask-master
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jorisvandenbossche commented on pull request #9113: ARROW-11147: [CI][Python] pin pandas=0.25.3 for dask-master
Posted by GitBox <gi...@apache.org>.
jorisvandenbossche commented on pull request #9113:
URL: https://github.com/apache/arrow/pull/9113#issuecomment-755413336
@github-actions crossbow submit test-conda-python-3.7-dask-master test-conda-python-3.7-dask-latest
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jorisvandenbossche commented on pull request #9113: ARROW-11147: [CI][Python] pin pandas=0.25.3 for dask-master
Posted by GitBox <gi...@apache.org>.
jorisvandenbossche commented on pull request #9113:
URL: https://github.com/apache/arrow/pull/9113#issuecomment-756348342
Hmm, but that seems an installation issues (it's just the pandas import that is failing)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jorisvandenbossche commented on pull request #9113: ARROW-11147: [CI][Python] pin pandas=0.25.3 for dask-master
Posted by GitBox <gi...@apache.org>.
jorisvandenbossche commented on pull request #9113:
URL: https://github.com/apache/arrow/pull/9113#issuecomment-756216841
@github-actions crossbow submit test-conda-python-3.8-dask-master test-conda-python-3.7-dask-latest
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jorisvandenbossche closed pull request #9113: ARROW-11147: [CI][Python] Remove pandas=0.25.3 pin for dask-latest
Posted by GitBox <gi...@apache.org>.
jorisvandenbossche closed pull request #9113:
URL: https://github.com/apache/arrow/pull/9113
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] arw2019 commented on pull request #9113: ARROW-11147: [CI][Python] pin pandas=0.25.3 for dask-master
Posted by GitBox <gi...@apache.org>.
arw2019 commented on pull request #9113:
URL: https://github.com/apache/arrow/pull/9113#issuecomment-755411788
cc @jorisvandenbossche
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] arw2019 commented on pull request #9113: ARROW-11147: [CI][Python] pin pandas=0.25.3 for dask-master
Posted by GitBox <gi...@apache.org>.
arw2019 commented on pull request #9113:
URL: https://github.com/apache/arrow/pull/9113#issuecomment-756567261
> So first pandas 1.2.0 gets installed because of `pip install https://github.com/dask/dask/archive/master.tar.gz#egg=dask[dataframe]`, and then conda installs pandas 0.25.3, but the subsequent `conda list` still shows pandas 1.2.0, but is fishy ..
>
> So maybe pandas should be installed before calling the `install_dask.sh` script in ci/docker/conda-python-dask.dockerfile to avoid such problems?
I'm not sure why we install pandas after we install dask (I haven't used dask but AFAIU pandas is a dependency). I tried deleting the last line of the dockerfile for now, will look more tomorrow
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #9113: ARROW-11147: [CI][Python] pin pandas=0.25.3 for dask-master
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #9113:
URL: https://github.com/apache/arrow/pull/9113#issuecomment-755471367
https://issues.apache.org/jira/browse/ARROW-11147
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jorisvandenbossche commented on pull request #9113: ARROW-11147: [CI][Python] pin pandas=0.25.3 for dask-master
Posted by GitBox <gi...@apache.org>.
jorisvandenbossche commented on pull request #9113:
URL: https://github.com/apache/arrow/pull/9113#issuecomment-756351707
So first pandas 1.2.0 gets installed because of `pip install https://github.com/dask/dask/archive/master.tar.gz#egg=dask[dataframe]`, and then conda installs pandas 0.25.3, but the subsequent `conda list` still shows pandas 1.2.0, but is fishy ..
So maybe pandas should be installed before calling the `install_dask.sh` script in ci/docker/conda-python-dask.dockerfile to avoid such problems?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jorisvandenbossche commented on pull request #9113: ARROW-11147: [CI][Python] pin pandas=0.25.3 for dask-master
Posted by GitBox <gi...@apache.org>.
jorisvandenbossche commented on pull request #9113:
URL: https://github.com/apache/arrow/pull/9113#issuecomment-755684431
@github-actions crossbow submit test-conda-python-3.8-dask-master
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jorisvandenbossche commented on pull request #9113: ARROW-11147: [CI][Python] pin pandas=0.25.3 for dask-master
Posted by GitBox <gi...@apache.org>.
jorisvandenbossche commented on pull request #9113:
URL: https://github.com/apache/arrow/pull/9113#issuecomment-756118906
There was a permission error with the github actions that processes the bot yesterday, should be fixed again. Let's try once more ;)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org