You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "PratyushD21 (via GitHub)" <gi...@apache.org> on 2026/02/10 03:30:39 UTC

[PR] PYARROW: map date32/date64 to datetime64[s] in to_pandas_dtype [arrow]

PratyushD21 opened a new pull request, #49210:
URL: https://github.com/apache/arrow/pull/49210

   ### Rationale for this change
   
   Fixes #49168
   
   ### What changes are included in this PR?
   
   As per the issue raised in [[Python] Consider pa.date32/64.to_pandas_dtype() returning datetime64[s] instead of datetime64[ms]](https://github.com/apache/arrow/issues/49168), it was observed that no `ms` timestamp was supported in [parquet](https://parquet.apache.org/docs/file-format/types/logicaltypes/). To rectify that, the `pa.Date32/64.to_pandas_dtype` will return `datetime64[s]`.
   
   python/pyarrow/types.pxi
   python/pyarrow/tests/test_schema.py
   
   ### Are these changes tested?
   Yes, the schema test file `python/pyarrow/tests/test_schema.py` has been updated to include this change.
   
   ### Are there any user-facing changes?
   Yes, the return types `pa.date64/32().to_pandas_dtype()` should return `datetime64[s]`.
   
   **This PR includes breaking changes to public APIs.** (If there are any breaking changes to public APIs, please explain which changes are breaking. If not, you can remove this.)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] PYARROW: map date32/date64 to datetime64[s] in to_pandas_dtype [arrow]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #49210:
URL: https://github.com/apache/arrow/pull/49210#issuecomment-3875120553

   <!--
     Licensed to the Apache Software Foundation (ASF) under one
     or more contributor license agreements.  See the NOTICE file
     distributed with this work for additional information
     regarding copyright ownership.  The ASF licenses this file
     to you under the Apache License, Version 2.0 (the
     "License"); you may not use this file except in compliance
     with the License.  You may obtain a copy of the License at
   
       http://www.apache.org/licenses/LICENSE-2.0
   
     Unless required by applicable law or agreed to in writing,
     software distributed under the License is distributed on an
     "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
     KIND, either express or implied.  See the License for the
     specific language governing permissions and limitations
     under the License.
   -->
   
   Thanks for opening a pull request!
   
   If this is not a [minor PR](https://github.com/apache/arrow/blob/main/CONTRIBUTING.md#Minor-Fixes). Could you open an issue for this pull request on GitHub? https://github.com/apache/arrow/issues/new/choose
   
   Opening GitHub issues ahead of time contributes to the [Openness](http://theapacheway.com/open/#:~:text=Openness%20allows%20new%20users%20the,must%20happen%20in%20the%20open.) of the Apache Arrow project.
   
   Then could you also rename the pull request title in the following format?
   
       GH-${GITHUB_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}
   
   or
   
       MINOR: [${COMPONENT}] ${SUMMARY}
   
   See also:
   
     * [Other pull requests](https://github.com/apache/arrow/pulls/)
     * [Contribution Guidelines - Contributing Overview](https://arrow.apache.org/docs/developers/overview.html)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-49168: [Python] map date32/date64 to datetime64[s] in to_pandas_dtype [arrow]

Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on PR #49210:
URL: https://github.com/apache/arrow/pull/49210#issuecomment-3883390376

   I don't think we should change this for `to_pandas_dtype()` if not changing it for `to_pandas()` itself? (i.e. both should ideally match?) 
   (or, if we want to change this, we should probably change `to_pandas()` as well? Or what is the exact rationale or use case for wanting to change `to_pandas_dtype()`)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-49168: [Python] map date32/date64 to datetime64[s] in to_pandas_dtype [arrow]

Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on PR #49210:
URL: https://github.com/apache/arrow/pull/49210#issuecomment-3886023319

   > `datetime64[s]` makes more sense to me compared to `datetime64[ms]` because of second resolution's capacity to represent a larger amount of dates that a `date32/64` might represent.
   
   For `date32[day]` that indeed makes sense, but for `date64[ms]`, using `datetime64[ms]` ensures that the conversion is more likely to be zero-copy 
   (and I think that that point we maybe took date32 to use ms as well for consistency, since a conversion for that type is needed anyhow)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] PYARROW: map date32/date64 to datetime64[s] in to_pandas_dtype [arrow]

Posted by "alippai (via GitHub)" <gi...@apache.org>.
alippai commented on PR #49210:
URL: https://github.com/apache/arrow/pull/49210#issuecomment-3875831186

   What about datetime64[D], wouldn’t that be more appropriate?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-49168: [Python] map date32/date64 to datetime64[s] in to_pandas_dtype [arrow]

Posted by "PratyushD21 (via GitHub)" <gi...@apache.org>.
PratyushD21 commented on PR #49210:
URL: https://github.com/apache/arrow/pull/49210#issuecomment-3887439673

   Hi @mroeschke @jorisvandenbossche, if I understand correctly, we would like to perform the same change for `to_pandas()` method as well, or are we planning to keep current `datetime[ms]` logic? Thank you for your inputs.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] PYARROW: map date32/date64 to datetime64[s] in to_pandas_dtype [arrow]

Posted by "PratyushD21 (via GitHub)" <gi...@apache.org>.
PratyushD21 commented on PR #49210:
URL: https://github.com/apache/arrow/pull/49210#issuecomment-3877487484

   Hi @alippai, it seems that this approach of including time in the date32/64 was included in the #35656 discussion which has been merged already. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] PYARROW: map date32/date64 to datetime64[s] in to_pandas_dtype [arrow]

Posted by "PratyushD21 (via GitHub)" <gi...@apache.org>.
PratyushD21 commented on PR #49210:
URL: https://github.com/apache/arrow/pull/49210#issuecomment-3880919497

   @rok would it be possible to re-trigger the CI? I have added additional check for pandas<=2. Now, the CIs should pass.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] PYARROW: map date32/date64 to datetime64[s] in to_pandas_dtype [arrow]

Posted by "alippai (via GitHub)" <gi...@apache.org>.
alippai commented on PR #49210:
URL: https://github.com/apache/arrow/pull/49210#issuecomment-3877986858

   @PratyushD21 good point, thanks. @mroeschke will know it the best, I remember pandas also changed how it handles days since 2023.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] PYARROW: map date32/date64 to datetime64[s] in to_pandas_dtype [arrow]

Posted by "PratyushD21 (via GitHub)" <gi...@apache.org>.
PratyushD21 commented on PR #49210:
URL: https://github.com/apache/arrow/pull/49210#issuecomment-3880631565

   Hi @mroeschke, I built from Source, and ran the test suite (especially the parquet tests). The tests have passed.
   ` python -m pytest pyarrow ` and `python -m pytest -q pyarrow/tests/parquet`. Exit status was 0. Should I push an empty commit to trigger CI in Arrow? 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-49168: [Python] map date32/date64 to datetime64[s] in to_pandas_dtype [arrow]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #49210:
URL: https://github.com/apache/arrow/pull/49210#issuecomment-3883200042

   :warning: GitHub issue #49168 **has been automatically assigned in GitHub** to PR creator.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] PYARROW: map date32/date64 to datetime64[s] in to_pandas_dtype [arrow]

Posted by "mroeschke (via GitHub)" <gi...@apache.org>.
mroeschke commented on PR #49210:
URL: https://github.com/apache/arrow/pull/49210#issuecomment-3879681811

   > What about datetime64[D], wouldn’t that be more appropriate?
   
   Yeah pandas does not natively support storing `datetime64[D]`, `[s]` is the largest resolution supported.
   
   The changes look good, but I don't have permissions to run the CI in Arrow. You may want to build Arrow/PyArrow from source and run the test suite to see if other tests passes, especially the parquet tests based on the history in #35656


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] PYARROW: map date32/date64 to datetime64[s] in to_pandas_dtype [arrow]

Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on PR #49210:
URL: https://github.com/apache/arrow/pull/49210#issuecomment-3880686619

   I kicked off the CI, ping me if you need to start it again.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-49168: [Python] map date32/date64 to datetime64[s] in to_pandas_dtype [arrow]

Posted by "mroeschke (via GitHub)" <gi...@apache.org>.
mroeschke commented on PR #49210:
URL: https://github.com/apache/arrow/pull/49210#issuecomment-3885821437

   > I don't think we should change this for to_pandas_dtype() if not changing it for to_pandas() itself? (i.e. both should ideally match?)
   
   Ah sure, I think there should be consistency in both APIs.
   
   For `pyarrow.DataType.to_pandas_dtype()` I would interpret "Return the equivalent NumPy / Pandas dtype" as mentioned in the docs as the "closest representable" type for types with no equivalence like `date` types; therefore; `datetime64[s]` makes more sense to me compared to `datetime64[ms]` because of second resolution's capacity to represent a larger amount of dates that a `date32/64` might represent. This was at least our expectation in [cuDF](https://github.com/rapidsai/cudf/pull/21369)
   
   Conversely it appears `ms` was chosen in https://github.com/apache/arrow/pull/35656 due to parquet compatibility IIUC? And if so, does that dependence between pandas ops and parquet still exist today?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] GH-49168: [Python] map date32/date64 to datetime64[s] in to_pandas_dtype [arrow]

Posted by "mroeschke (via GitHub)" <gi...@apache.org>.
mroeschke commented on PR #49210:
URL: https://github.com/apache/arrow/pull/49210#issuecomment-3886065927

   > ensures that the conversion is more likely to be zero-copy
   
   OK that's reasonable. I understand if ultimately the change is not made; we just found it unexpected in cuDF


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org