You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "jorisvandenbossche (via GitHub)" <gi...@apache.org> on 2023/10/18 07:53:51 UTC
[PR] GH-35081: [Python] Remove usage of pandas internals DatetimeTZBlock [arrow]
jorisvandenbossche opened a new pull request, #38321:
URL: https://github.com/apache/arrow/pull/38321
### Rationale for this change
This usage probably stems from a long time ago that it was required to specify the Block type, but nowadays (and tested with pandas 1.0) it's good enough to just specify the dtype, and thus cutting down on our usage of internal pandas objects.
Part of #35081
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-35081: [Python] Remove usage of pandas internals DatetimeTZBlock [arrow]
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on PR #38321:
URL: https://github.com/apache/arrow/pull/38321#issuecomment-1767945494
So it seems that pandas doesn't preserve the unit in the DatetimeArray constructor:
```
In [4]: arr = np.array([1, 2, 3], dtype="datetime64[s]")
In [5]: dtype = pd.DatetimeTZDtype("s", tz="Europe/Brussels")
In [6]: pd.arrays.DatetimeArray(arr, dtype)
Out[6]:
<DatetimeArray>
['1970-01-01 01:00:01+01:00', '1970-01-01 01:00:02+01:00',
'1970-01-01 01:00:03+01:00']
Length: 3, dtype: datetime64[ns, Europe/Brussels]
```
This seems fixed on the last pandas release 2.1, so will have to keep the older code using internals for the older pandas versions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-35081: [Python] Remove usage of pandas internals DatetimeTZBlock [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #38321:
URL: https://github.com/apache/arrow/pull/38321#issuecomment-1767888826
:warning: GitHub issue #35081 **has been automatically assigned in GitHub** to PR creator.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-35081: [Python] Remove usage of pandas internals DatetimeTZBlock [arrow]
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on PR #38321:
URL: https://github.com/apache/arrow/pull/38321#issuecomment-1768438205
@github-actions crossbow submit -g python
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-38341: [Python] Remove usage of pandas internals DatetimeTZBlock [arrow]
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche merged PR #38321:
URL: https://github.com/apache/arrow/pull/38321
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-38341: [Python] Remove usage of pandas internals DatetimeTZBlock [arrow]
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on PR #38321:
URL: https://github.com/apache/arrow/pull/38321#issuecomment-1836595619
@github-actions crossbow submit -g python
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-35081: [Python] Remove usage of pandas internals DatetimeTZBlock [arrow]
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on PR #38321:
URL: https://github.com/apache/arrow/pull/38321#issuecomment-1767896579
@github-actions crossbow submit -g python
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-35081: [Python] Remove usage of pandas internals DatetimeTZBlock [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #38321:
URL: https://github.com/apache/arrow/pull/38321#issuecomment-1768444705
Revision: a13301452631306736838e913969bd2e71981252
Submitted crossbow builds: [ursacomputing/crossbow @ actions-eda4ccd5da](https://github.com/ursacomputing/crossbow/branches/all?query=actions-eda4ccd5da)
|Task|Status|
|----|------|
|test-conda-python-3.10|[](https://github.com/ursacomputing/crossbow/actions/runs/6561482916/job/17821265861)|
|test-conda-python-3.10-cython2|[](https://github.com/ursacomputing/crossbow/actions/runs/6561480955/job/17821259398)|
|test-conda-python-3.10-hdfs-2.9.2|[](https://github.com/ursacomputing/crossbow/actions/runs/6561480533/job/17821258295)|
|test-conda-python-3.10-hdfs-3.2.1|[](https://github.com/ursacomputing/crossbow/actions/runs/6561474165/job/17821237540)|
|test-conda-python-3.10-pandas-latest|[](https://github.com/ursacomputing/crossbow/actions/runs/6561483720/job/17821268649)|
|test-conda-python-3.10-pandas-nightly|[](https://github.com/ursacomputing/crossbow/actions/runs/6561482232/job/17821263228)|
|test-conda-python-3.10-spark-v3.5.0|[](https://github.com/ursacomputing/crossbow/actions/runs/6561478128/job/17821250675)|
|test-conda-python-3.10-substrait|[](https://github.com/ursacomputing/crossbow/actions/runs/6561482565/job/17821264615)|
|test-conda-python-3.11|[](https://github.com/ursacomputing/crossbow/actions/runs/6561475168/job/17821241282)|
|test-conda-python-3.11-dask-latest|[](https://github.com/ursacomputing/crossbow/actions/runs/6561480141/job/17821257028)|
|test-conda-python-3.11-dask-upstream_devel|[](https://github.com/ursacomputing/crossbow/actions/runs/6561475770/job/17821243347)|
|test-conda-python-3.11-hypothesis|[](https://github.com/ursacomputing/crossbow/actions/runs/6561481441/job/17821260767)|
|test-conda-python-3.11-pandas-upstream_devel|[](https://github.com/ursacomputing/crossbow/actions/runs/6561477421/job/17821248642)|
|test-conda-python-3.11-spark-master|[](https://github.com/ursacomputing/crossbow/actions/runs/6561475911/job/17821243892)|
|test-conda-python-3.12|[](https://github.com/ursacomputing/crossbow/actions/runs/6561481760/job/17821262168)|
|test-conda-python-3.8|[](https://github.com/ursacomputing/crossbow/actions/runs/6561474889/job/17821240377)|
|test-conda-python-3.8-pandas-1.0|[](https://github.com/ursacomputing/crossbow/actions/runs/6561476730/job/17821246175)|
|test-conda-python-3.8-spark-v3.5.0|[](https://github.com/ursacomputing/crossbow/actions/runs/6561477825/job/17821249713)|
|test-conda-python-3.9|[](https://github.com/ursacomputing/crossbow/actions/runs/6561478525/job/17821251999)|
|test-conda-python-3.9-pandas-latest|[](https://github.com/ursacomputing/crossbow/actions/runs/6561476350/job/17821245372)|
|test-cuda-python|[](https://github.com/ursacomputing/crossbow/actions/runs/6561479341/job/17821254679)|
|test-debian-11-python-3|[](https://github.com/ursacomputing/crossbow/runs/17821255418)|
|test-fedora-35-python-3|[](https://github.com/ursacomputing/crossbow/runs/17821249836)|
|test-ubuntu-20.04-python-3|[](https://github.com/ursacomputing/crossbow/runs/17821241199)|
|test-ubuntu-22.04-python-3|[](https://github.com/ursacomputing/crossbow/actions/runs/6561483310/job/17821267424)|
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-38341: [Python] Remove usage of pandas internals DatetimeTZBlock [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #38321:
URL: https://github.com/apache/arrow/pull/38321#issuecomment-1836599673
Revision: dfcfa22a181acdf3c04c1abe0d3c27a9603ca954
Submitted crossbow builds: [ursacomputing/crossbow @ actions-6fbf28b83d](https://github.com/ursacomputing/crossbow/branches/all?query=actions-6fbf28b83d)
|Task|Status|
|----|------|
|test-conda-python-3.10|[](https://github.com/ursacomputing/crossbow/actions/runs/7063870899/job/19230726310)|
|test-conda-python-3.10-cython2|[](https://github.com/ursacomputing/crossbow/actions/runs/7063875300/job/19230740626)|
|test-conda-python-3.10-hdfs-2.9.2|[](https://github.com/ursacomputing/crossbow/actions/runs/7063869925/job/19230723585)|
|test-conda-python-3.10-hdfs-3.2.1|[](https://github.com/ursacomputing/crossbow/actions/runs/7063871731/job/19230729252)|
|test-conda-python-3.10-pandas-latest|[](https://github.com/ursacomputing/crossbow/actions/runs/7063876850/job/19230744202)|
|test-conda-python-3.10-pandas-nightly|[](https://github.com/ursacomputing/crossbow/actions/runs/7063875911/job/19230742040)|
|test-conda-python-3.10-spark-v3.5.0|[](https://github.com/ursacomputing/crossbow/actions/runs/7063872632/job/19230732655)|
|test-conda-python-3.10-substrait|[](https://github.com/ursacomputing/crossbow/actions/runs/7063873652/job/19230735942)|
|test-conda-python-3.11|[](https://github.com/ursacomputing/crossbow/actions/runs/7063874624/job/19230738755)|
|test-conda-python-3.11-dask-latest|[](https://github.com/ursacomputing/crossbow/actions/runs/7063871256/job/19230727351)|
|test-conda-python-3.11-dask-upstream_devel|[](https://github.com/ursacomputing/crossbow/actions/runs/7063872521/job/19230732291)|
|test-conda-python-3.11-hypothesis|[](https://github.com/ursacomputing/crossbow/actions/runs/7063871685/job/19230728985)|
|test-conda-python-3.11-pandas-upstream_devel|[](https://github.com/ursacomputing/crossbow/actions/runs/7063874162/job/19230737581)|
|test-conda-python-3.11-spark-master|[](https://github.com/ursacomputing/crossbow/actions/runs/7063870670/job/19230725572)|
|test-conda-python-3.12|[](https://github.com/ursacomputing/crossbow/actions/runs/7063872870/job/19230733304)|
|test-conda-python-3.8|[](https://github.com/ursacomputing/crossbow/actions/runs/7063870349/job/19230724570)|
|test-conda-python-3.8-pandas-1.0|[](https://github.com/ursacomputing/crossbow/actions/runs/7063874563/job/19230738518)|
|test-conda-python-3.8-spark-v3.5.0|[](https://github.com/ursacomputing/crossbow/actions/runs/7063875742/job/19230741517)|
|test-conda-python-3.9|[](https://github.com/ursacomputing/crossbow/actions/runs/7063873302/job/19230734729)|
|test-conda-python-3.9-pandas-latest|[](https://github.com/ursacomputing/crossbow/actions/runs/7063872131/job/19230730900)|
|test-cuda-python|[](https://github.com/ursacomputing/crossbow/actions/runs/7063877138/job/19230745162)|
|test-debian-11-python-3|[](https://github.com/ursacomputing/crossbow/runs/19230724129)|
|test-fedora-38-python-3|[](https://github.com/ursacomputing/crossbow/runs/19230725261)|
|test-ubuntu-20.04-python-3|[](https://github.com/ursacomputing/crossbow/runs/19230736196)|
|test-ubuntu-22.04-python-3|[](https://github.com/ursacomputing/crossbow/actions/runs/7063876600/job/19230743729)|
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-38341: [Python] Remove usage of pandas internals DatetimeTZBlock [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #38321:
URL: https://github.com/apache/arrow/pull/38321#issuecomment-1770243023
:warning: GitHub issue #38341 **has been automatically assigned in GitHub** to PR creator.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-38341: [Python] Remove usage of pandas internals DatetimeTZBlock [arrow]
Posted by "conbench-apache-arrow[bot] (via GitHub)" <gi...@apache.org>.
conbench-apache-arrow[bot] commented on PR #38321:
URL: https://github.com/apache/arrow/pull/38321#issuecomment-1881705079
After merging your PR, Conbench analyzed the 6 benchmarking runs that have been run so far on merge-commit 6b93c4a0e8cb5110c6c4d3746f4e8bb0a8b76ec8.
There were no benchmark performance regressions. 🎉
The [full Conbench report](https://github.com/apache/arrow/runs/20275279777) has more details. It also includes information about 1 possible false positive for unstable benchmarks that are known to sometimes produce them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-35081: [Python] Remove usage of pandas internals DatetimeTZBlock [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #38321:
URL: https://github.com/apache/arrow/pull/38321#issuecomment-1767901862
Revision: 9c3fdf96a7c2d609e9dc0b4b42cf42cc703ef3b8
Submitted crossbow builds: [ursacomputing/crossbow @ actions-68809493b5](https://github.com/ursacomputing/crossbow/branches/all?query=actions-68809493b5)
|Task|Status|
|----|------|
|test-conda-python-3.10|[](https://github.com/ursacomputing/crossbow/actions/runs/6557940051/job/17810511743)|
|test-conda-python-3.10-cython2|[](https://github.com/ursacomputing/crossbow/actions/runs/6557941072/job/17810514885)|
|test-conda-python-3.10-hdfs-2.9.2|[](https://github.com/ursacomputing/crossbow/actions/runs/6557939707/job/17810510862)|
|test-conda-python-3.10-hdfs-3.2.1|[](https://github.com/ursacomputing/crossbow/actions/runs/6557937365/job/17810502468)|
|test-conda-python-3.10-pandas-latest|[](https://github.com/ursacomputing/crossbow/actions/runs/6557937922/job/17810504794)|
|test-conda-python-3.10-pandas-nightly|[](https://github.com/ursacomputing/crossbow/actions/runs/6557940481/job/17810512476)|
|test-conda-python-3.10-spark-v3.5.0|[](https://github.com/ursacomputing/crossbow/actions/runs/6557936723/job/17810500057)|
|test-conda-python-3.10-substrait|[](https://github.com/ursacomputing/crossbow/actions/runs/6557941927/job/17810516171)|
|test-conda-python-3.11|[](https://github.com/ursacomputing/crossbow/actions/runs/6557938920/job/17810508187)|
|test-conda-python-3.11-dask-latest|[](https://github.com/ursacomputing/crossbow/actions/runs/6557936966/job/17810500676)|
|test-conda-python-3.11-dask-upstream_devel|[](https://github.com/ursacomputing/crossbow/actions/runs/6557935573/job/17810496376)|
|test-conda-python-3.11-hypothesis|[](https://github.com/ursacomputing/crossbow/actions/runs/6557936505/job/17810499353)|
|test-conda-python-3.11-pandas-upstream_devel|[](https://github.com/ursacomputing/crossbow/actions/runs/6557937233/job/17810501910)|
|test-conda-python-3.11-spark-master|[](https://github.com/ursacomputing/crossbow/actions/runs/6557941531/job/17810515561)|
|test-conda-python-3.12|[](https://github.com/ursacomputing/crossbow/actions/runs/6557938177/job/17810505388)|
|test-conda-python-3.8|[](https://github.com/ursacomputing/crossbow/actions/runs/6557935684/job/17810496389)|
|test-conda-python-3.8-pandas-1.0|[](https://github.com/ursacomputing/crossbow/actions/runs/6557942325/job/17810516900)|
|test-conda-python-3.8-spark-v3.5.0|[](https://github.com/ursacomputing/crossbow/actions/runs/6557936112/job/17810497713)|
|test-conda-python-3.9|[](https://github.com/ursacomputing/crossbow/actions/runs/6557938439/job/17810506238)|
|test-conda-python-3.9-pandas-latest|[](https://github.com/ursacomputing/crossbow/actions/runs/6557939405/job/17810509478)|
|test-cuda-python|[](https://github.com/ursacomputing/crossbow/actions/runs/6557937575/job/17810503130)|
|test-debian-11-python-3|[](https://github.com/ursacomputing/crossbow/runs/17810501269)|
|test-fedora-35-python-3|[](https://github.com/ursacomputing/crossbow/runs/17810505366)|
|test-ubuntu-20.04-python-3|[](https://github.com/ursacomputing/crossbow/runs/17810515276)|
|test-ubuntu-22.04-python-3|[](https://github.com/ursacomputing/crossbow/actions/runs/6557938625/job/17810506880)|
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org