You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "raulcd (via GitHub)" <gi...@apache.org> on 2026/07/10 07:06:06 UTC
[PR] GH-50464: [C++][Python] Simplify arrow_to_pandas DateOffset handling for nanoseconds/milliseconds [arrow]
raulcd opened a new pull request, #50465:
URL: https://github.com/apache/arrow/pull/50465
## TBD
### Rationale for this change
### What changes are included in this PR?
### Are these changes tested?
### Are there any user-facing changes?
**This PR includes breaking changes to public APIs.** (If there are any breaking changes to public APIs, please explain which changes are breaking. If not, you can remove this.)
**This PR contains a "Critical Fix".** (If the changes fix either (a) a security vulnerability, (b) a bug that caused incorrect or invalid data to be produced, or (c) a bug that causes a crash (even when the API contract is upheld), please provide explanation. If not, you can remove this.)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-50464: [C++][Python] Simplify arrow_to_pandas DateOffset handling for nanoseconds/milliseconds [arrow]
Posted by "raulcd (via GitHub)" <gi...@apache.org>.
raulcd commented on code in PR #50465:
URL: https://github.com/apache/arrow/pull/50465#discussion_r3629138801
##########
python/pyarrow/src/arrow/python/arrow_to_pandas.cc:
##########
@@ -1292,24 +1292,11 @@ struct ObjectWriterVisitor {
auto to_date_offset = [&](const MonthDayNanoIntervalType::MonthDayNanos& interval,
PyObject** out) {
ARROW_DCHECK(internal::BorrowPandasDataOffsetType() != nullptr);
- // DateOffset objects do not add nanoseconds component to pd.Timestamp.
- // as of Pandas 1.3.3
- // (https://github.com/pandas-dev/pandas/issues/43892).
- // So convert microseconds and remainder to preserve data
- // but give users more expected results.
- int64_t microseconds = interval.nanoseconds / 1000;
- int64_t nanoseconds;
- if (interval.nanoseconds >= 0) {
- nanoseconds = interval.nanoseconds % 1000;
- } else {
- nanoseconds = -((-interval.nanoseconds) % 1000);
- }
PyDict_SetItemString(kwargs.obj(), "months", PyLong_FromLong(interval.months));
PyDict_SetItemString(kwargs.obj(), "days", PyLong_FromLong(interval.days));
- PyDict_SetItemString(kwargs.obj(), "microseconds",
- PyLong_FromLongLong(microseconds));
- PyDict_SetItemString(kwargs.obj(), "nanoseconds", PyLong_FromLongLong(nanoseconds));
+ PyDict_SetItemString(kwargs.obj(), "nanoseconds",
+ PyLong_FromLongLong(interval.nanoseconds));
Review Comment:
Thanks @pitrou, applied the fix. I am unsure whether all `RETURN_IF_PYERROR` checks are necessary or one after all `PyDict_SetItemString` is enough as I don't think `PyErr_Clear` is exercised but I suppose better safe than sorry. Is there a way to know which CPython methods from the C-API clear PyErr?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-50464: [C++][Python] Simplify arrow_to_pandas DateOffset handling for nanoseconds/milliseconds [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #50465:
URL: https://github.com/apache/arrow/pull/50465#discussion_r3629267559
##########
python/pyarrow/src/arrow/python/arrow_to_pandas.cc:
##########
@@ -1292,24 +1292,11 @@ struct ObjectWriterVisitor {
auto to_date_offset = [&](const MonthDayNanoIntervalType::MonthDayNanos& interval,
PyObject** out) {
ARROW_DCHECK(internal::BorrowPandasDataOffsetType() != nullptr);
- // DateOffset objects do not add nanoseconds component to pd.Timestamp.
- // as of Pandas 1.3.3
- // (https://github.com/pandas-dev/pandas/issues/43892).
- // So convert microseconds and remainder to preserve data
- // but give users more expected results.
- int64_t microseconds = interval.nanoseconds / 1000;
- int64_t nanoseconds;
- if (interval.nanoseconds >= 0) {
- nanoseconds = interval.nanoseconds % 1000;
- } else {
- nanoseconds = -((-interval.nanoseconds) % 1000);
- }
PyDict_SetItemString(kwargs.obj(), "months", PyLong_FromLong(interval.months));
PyDict_SetItemString(kwargs.obj(), "days", PyLong_FromLong(interval.days));
- PyDict_SetItemString(kwargs.obj(), "microseconds",
- PyLong_FromLongLong(microseconds));
- PyDict_SetItemString(kwargs.obj(), "nanoseconds", PyLong_FromLongLong(nanoseconds));
+ PyDict_SetItemString(kwargs.obj(), "nanoseconds",
+ PyLong_FromLongLong(interval.nanoseconds));
Review Comment:
Normally, C API functions shouldn't clear the error indicator. You should still call `RETURN_IF_PYERROR` after creating all the OwnedRefs, I think, lest one of them may be NULL.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-50464: [C++][Python] Simplify arrow_to_pandas DateOffset handling for nanoseconds/milliseconds [arrow]
Posted by "raulcd (via GitHub)" <gi...@apache.org>.
raulcd merged PR #50465:
URL: https://github.com/apache/arrow/pull/50465
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-50464: [C++][Python] Simplify arrow_to_pandas DateOffset handling for nanoseconds/milliseconds [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #50465:
URL: https://github.com/apache/arrow/pull/50465#issuecomment-4932857066
:warning: GitHub issue #50464 **has been automatically assigned in GitHub** to PR creator.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-50464: [C++][Python] Simplify arrow_to_pandas DateOffset handling for nanoseconds/milliseconds [arrow]
Posted by "conbench-apache-arrow[bot] (via GitHub)" <gi...@apache.org>.
conbench-apache-arrow[bot] commented on PR #50465:
URL: https://github.com/apache/arrow/pull/50465#issuecomment-5076868643
After merging your PR, Conbench analyzed the 4 benchmarking runs that have been run so far on merge-commit 6287272f39b8ebef02ca3ec8cf51650e4b240231.
There were no benchmark performance regressions. 🎉
The [full Conbench report](https://github.com/apache/arrow/runs/89641295077) has more details. It also includes information about 3 possible false positives for unstable benchmarks that are known to sometimes produce them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-50464: [C++][Python] Simplify arrow_to_pandas DateOffset handling for nanoseconds/milliseconds [arrow]
Posted by "raulcd (via GitHub)" <gi...@apache.org>.
raulcd commented on PR #50465:
URL: https://github.com/apache/arrow/pull/50465#issuecomment-4935812687
@jorisvandenbossche would there be any reason to keep roundtriping microseconds on `DateOffset(months=1, days=1, microseconds=1, nanoseconds=1)` when we don't do it for `years, seconds, minutes, hours, weeks`?
We can simplify a little here with this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-50464: [C++][Python] Simplify arrow_to_pandas DateOffset handling for nanoseconds/milliseconds [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #50465:
URL: https://github.com/apache/arrow/pull/50465#discussion_r3629295435
##########
python/pyarrow/src/arrow/python/arrow_to_pandas.cc:
##########
@@ -1292,24 +1292,11 @@ struct ObjectWriterVisitor {
auto to_date_offset = [&](const MonthDayNanoIntervalType::MonthDayNanos& interval,
PyObject** out) {
ARROW_DCHECK(internal::BorrowPandasDataOffsetType() != nullptr);
- // DateOffset objects do not add nanoseconds component to pd.Timestamp.
- // as of Pandas 1.3.3
- // (https://github.com/pandas-dev/pandas/issues/43892).
- // So convert microseconds and remainder to preserve data
- // but give users more expected results.
- int64_t microseconds = interval.nanoseconds / 1000;
- int64_t nanoseconds;
- if (interval.nanoseconds >= 0) {
- nanoseconds = interval.nanoseconds % 1000;
- } else {
- nanoseconds = -((-interval.nanoseconds) % 1000);
- }
PyDict_SetItemString(kwargs.obj(), "months", PyLong_FromLong(interval.months));
PyDict_SetItemString(kwargs.obj(), "days", PyLong_FromLong(interval.days));
- PyDict_SetItemString(kwargs.obj(), "microseconds",
- PyLong_FromLongLong(microseconds));
- PyDict_SetItemString(kwargs.obj(), "nanoseconds", PyLong_FromLongLong(nanoseconds));
+ PyDict_SetItemString(kwargs.obj(), "nanoseconds",
+ PyLong_FromLongLong(interval.nanoseconds));
Review Comment:
No. But you need only one `RETURN_IF_PYERROR` after all the `PyDict_SetItemString`, not after each one of them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-50464: [C++][Python] Simplify arrow_to_pandas DateOffset handling for nanoseconds/milliseconds [arrow]
Posted by "raulcd (via GitHub)" <gi...@apache.org>.
raulcd commented on code in PR #50465:
URL: https://github.com/apache/arrow/pull/50465#discussion_r3629282480
##########
python/pyarrow/src/arrow/python/arrow_to_pandas.cc:
##########
@@ -1292,24 +1292,11 @@ struct ObjectWriterVisitor {
auto to_date_offset = [&](const MonthDayNanoIntervalType::MonthDayNanos& interval,
PyObject** out) {
ARROW_DCHECK(internal::BorrowPandasDataOffsetType() != nullptr);
- // DateOffset objects do not add nanoseconds component to pd.Timestamp.
- // as of Pandas 1.3.3
- // (https://github.com/pandas-dev/pandas/issues/43892).
- // So convert microseconds and remainder to preserve data
- // but give users more expected results.
- int64_t microseconds = interval.nanoseconds / 1000;
- int64_t nanoseconds;
- if (interval.nanoseconds >= 0) {
- nanoseconds = interval.nanoseconds % 1000;
- } else {
- nanoseconds = -((-interval.nanoseconds) % 1000);
- }
PyDict_SetItemString(kwargs.obj(), "months", PyLong_FromLong(interval.months));
PyDict_SetItemString(kwargs.obj(), "days", PyLong_FromLong(interval.days));
- PyDict_SetItemString(kwargs.obj(), "microseconds",
- PyLong_FromLongLong(microseconds));
- PyDict_SetItemString(kwargs.obj(), "nanoseconds", PyLong_FromLongLong(nanoseconds));
+ PyDict_SetItemString(kwargs.obj(), "nanoseconds",
+ PyLong_FromLongLong(interval.nanoseconds));
Review Comment:
I think this is what I've done, have I missed any?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-50464: [C++][Python] Simplify arrow_to_pandas DateOffset handling for nanoseconds/milliseconds [arrow]
Posted by "raulcd (via GitHub)" <gi...@apache.org>.
raulcd commented on PR #50465:
URL: https://github.com/apache/arrow/pull/50465#issuecomment-5056130934
I am going to merge this. This is currently not only simplifying part of the custom logic but fixing existing reference leaks.
If there are concerns about the UX change in the future we can discuss on a follow up.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-50464: [C++][Python] Simplify arrow_to_pandas DateOffset handling for nanoseconds/milliseconds [arrow]
Posted by "raulcd (via GitHub)" <gi...@apache.org>.
raulcd commented on code in PR #50465:
URL: https://github.com/apache/arrow/pull/50465#discussion_r3629316465
##########
python/pyarrow/src/arrow/python/arrow_to_pandas.cc:
##########
@@ -1292,24 +1292,11 @@ struct ObjectWriterVisitor {
auto to_date_offset = [&](const MonthDayNanoIntervalType::MonthDayNanos& interval,
PyObject** out) {
ARROW_DCHECK(internal::BorrowPandasDataOffsetType() != nullptr);
- // DateOffset objects do not add nanoseconds component to pd.Timestamp.
- // as of Pandas 1.3.3
- // (https://github.com/pandas-dev/pandas/issues/43892).
- // So convert microseconds and remainder to preserve data
- // but give users more expected results.
- int64_t microseconds = interval.nanoseconds / 1000;
- int64_t nanoseconds;
- if (interval.nanoseconds >= 0) {
- nanoseconds = interval.nanoseconds % 1000;
- } else {
- nanoseconds = -((-interval.nanoseconds) % 1000);
- }
PyDict_SetItemString(kwargs.obj(), "months", PyLong_FromLong(interval.months));
PyDict_SetItemString(kwargs.obj(), "days", PyLong_FromLong(interval.days));
- PyDict_SetItemString(kwargs.obj(), "microseconds",
- PyLong_FromLongLong(microseconds));
- PyDict_SetItemString(kwargs.obj(), "nanoseconds", PyLong_FromLongLong(nanoseconds));
+ PyDict_SetItemString(kwargs.obj(), "nanoseconds",
+ PyLong_FromLongLong(interval.nanoseconds));
Review Comment:
ah! I understand you now. Yes the single `RETURN_IF_PYERROR` after all `PyDict_SetItemString` is what I wasn't sure. Thanks for confirming.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-50464: [C++][Python] Simplify arrow_to_pandas DateOffset handling for nanoseconds/milliseconds [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #50465:
URL: https://github.com/apache/arrow/pull/50465#discussion_r3628616609
##########
python/pyarrow/src/arrow/python/arrow_to_pandas.cc:
##########
@@ -1292,24 +1292,11 @@ struct ObjectWriterVisitor {
auto to_date_offset = [&](const MonthDayNanoIntervalType::MonthDayNanos& interval,
PyObject** out) {
ARROW_DCHECK(internal::BorrowPandasDataOffsetType() != nullptr);
- // DateOffset objects do not add nanoseconds component to pd.Timestamp.
- // as of Pandas 1.3.3
- // (https://github.com/pandas-dev/pandas/issues/43892).
- // So convert microseconds and remainder to preserve data
- // but give users more expected results.
- int64_t microseconds = interval.nanoseconds / 1000;
- int64_t nanoseconds;
- if (interval.nanoseconds >= 0) {
- nanoseconds = interval.nanoseconds % 1000;
- } else {
- nanoseconds = -((-interval.nanoseconds) % 1000);
- }
PyDict_SetItemString(kwargs.obj(), "months", PyLong_FromLong(interval.months));
PyDict_SetItemString(kwargs.obj(), "days", PyLong_FromLong(interval.days));
- PyDict_SetItemString(kwargs.obj(), "microseconds",
- PyLong_FromLongLong(microseconds));
- PyDict_SetItemString(kwargs.obj(), "nanoseconds", PyLong_FromLongLong(nanoseconds));
+ PyDict_SetItemString(kwargs.obj(), "nanoseconds",
+ PyLong_FromLongLong(interval.nanoseconds));
Review Comment:
I think this Copilot comment is right @raulcd , though obviously it already applied to the existing code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-50464: [C++][Python] Simplify arrow_to_pandas DateOffset handling for nanoseconds/milliseconds [arrow]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #50465:
URL: https://github.com/apache/arrow/pull/50465#discussion_r3559277519
##########
python/pyarrow/src/arrow/python/arrow_to_pandas.cc:
##########
@@ -1292,24 +1292,11 @@ struct ObjectWriterVisitor {
auto to_date_offset = [&](const MonthDayNanoIntervalType::MonthDayNanos& interval,
PyObject** out) {
ARROW_DCHECK(internal::BorrowPandasDataOffsetType() != nullptr);
- // DateOffset objects do not add nanoseconds component to pd.Timestamp.
- // as of Pandas 1.3.3
- // (https://github.com/pandas-dev/pandas/issues/43892).
- // So convert microseconds and remainder to preserve data
- // but give users more expected results.
- int64_t microseconds = interval.nanoseconds / 1000;
- int64_t nanoseconds;
- if (interval.nanoseconds >= 0) {
- nanoseconds = interval.nanoseconds % 1000;
- } else {
- nanoseconds = -((-interval.nanoseconds) % 1000);
- }
PyDict_SetItemString(kwargs.obj(), "months", PyLong_FromLong(interval.months));
PyDict_SetItemString(kwargs.obj(), "days", PyLong_FromLong(interval.days));
- PyDict_SetItemString(kwargs.obj(), "microseconds",
- PyLong_FromLongLong(microseconds));
- PyDict_SetItemString(kwargs.obj(), "nanoseconds", PyLong_FromLongLong(nanoseconds));
+ PyDict_SetItemString(kwargs.obj(), "nanoseconds",
+ PyLong_FromLongLong(interval.nanoseconds));
Review Comment:
The PyLong objects created by PyLong_FromLong / PyLong_FromLongLong are new references; passing them directly into PyDict_SetItemString without a DECREF (or an owning wrapper) leaks one reference per element. Wrap these temporaries in OwnedRef (or DECREF after insertion) and check for PyErr after setting items.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org