You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2021/10/04 06:46:11 UTC
[GitHub] [arrow] emkornfield opened a new pull request #11302: ARROW-13806: [C++][Python] Add support for new Interval Type
emkornfield opened a new pull request #11302:
URL: https://github.com/apache/arrow/pull/11302
- Refactored ObjectWriter helpers from arrow_to_pandas, so they can be
used for plain python types as well (generalized the lowest level so
it can work on both PyObject** and an adapter for PyList.
- Add DateOffset to static pandas imports
- Tried to start laying out code in a way to use C++ for Array.to_pylist
(feel free to comment).
Support importing from timeinterval, relativedelta and DateOffset types
(this is actually mostly duck types, the one complication is that
relativedelta has a property weeks that is automatically calculated, so
some type checking is necessary).
Open questions:
- Should we be more strict on duck typing imports? I chose generalism
over performance here (rechecking non-present attributes, etc)?
- Is the new arrow_to_python.h desirable (I think this can be easily
extended for other types)?
- My python is rusty and Python C-API even more so, please don't assume
I know exactly what I'm doing :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on a change in pull request #11302: ARROW-13806: [C++][Python] Add support for new Interval Type
Posted by GitBox <gi...@apache.org>.
emkornfield commented on a change in pull request #11302:
URL: https://github.com/apache/arrow/pull/11302#discussion_r722776848
##########
File path: cpp/src/arrow/python/arrow_to_python.h
##########
@@ -0,0 +1,59 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+// Functions for converting between pandas's NumPy-based data representation
Review comment:
removed for now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on pull request #11302: ARROW-13806: [C++][Python] Add support for new Interval Type
Posted by GitBox <gi...@apache.org>.
emkornfield commented on pull request #11302:
URL: https://github.com/apache/arrow/pull/11302#issuecomment-933193094
CC @tswast @jorisvandenbossche
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield edited a comment on pull request #11302: ARROW-13806: [C++][Python] Add support for new MonthDayNano Interval Type
Posted by GitBox <gi...@apache.org>.
emkornfield edited a comment on pull request #11302:
URL: https://github.com/apache/arrow/pull/11302#issuecomment-936299002
@pitrou is changing the signature and docs to be specific to MonthDayNanoInterval
in this module OK for this PR or do you have another suggestion on
organization? Move it to datetime.h?
On Wednesday, October 6, 2021, Antoine Pitrou ***@***.***>
wrote:
> ***@***.**** commented on this pull request.
> ------------------------------
>
> In cpp/src/arrow/python/arrow_to_python.cc
> <https://github.com/apache/arrow/pull/11302#discussion_r723139253>:
>
> > + }
> +
> + PyListAssigner& operator+=(int64_t offset) {
> + current_index_ += offset;
> + return *this;
> + }
> +
> + private:
> + PyObject* list_;
> + int64_t current_index_ = 0;
> +};
> +
> +} // namespace
> +
> +Result<PyObject*> ArrowToPython::ToPyList(const Array& array) {
> + RETURN_NOT_OK(CheckInterval(*array.type()));
>
> I see. I'd rather see a comprehensive refactor in another PR than a stub
> like this, which may end up abandoned.
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/apache/arrow/pull/11302#discussion_r723139253>, or
> unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AEIKYDWFIVHB3GUOHI2735TUFQV2RANCNFSM5FITG4GA>
> .
> Triage notifications on the go with GitHub Mobile for iOS
> <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675>
> or Android
> <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.
>
>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on a change in pull request #11302: ARROW-13806: [C++][Python] Add support for new Interval Type
Posted by GitBox <gi...@apache.org>.
emkornfield commented on a change in pull request #11302:
URL: https://github.com/apache/arrow/pull/11302#discussion_r722910082
##########
File path: cpp/src/arrow/python/arrow_to_python.h
##########
@@ -0,0 +1,59 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+// Functions for converting between pandas's NumPy-based data representation
+// and Arrow data structures
+
+#pragma once
+
+#include "arrow/python/common.h"
+#include "arrow/python/platform.h"
+
+namespace arrow {
+
+class Array;
+struct Scalar;
+
+namespace py {
+
+/// \brief Utility class for converting Arrow to Python obects. A class instead
+///
+/// A class is chosen because in the future some amount of state will be
+/// (e.g. imported python classes), doing this one lazily will be helpful
+/// and having members present avoids static C++ variables.
+class ARROW_PYTHON_EXPORT ArrowToPython {
+ public:
+ /// \brief Converts the given Array to a PyList object. Returns NULL if there
+ /// is an error converting the Array. The list elements are the same ones
+ /// generated via ToLogical()
+ ///
+ /// N.B. This has limited type support. ARROW-12976 tracks extending the
+ /// implementation.
+ Result<PyObject*> ToPyList(const Array& array);
+
+ /// \brief Converts the given Scalar the type that is closest to its arrow
+ /// representation.
+ ///
+ /// For instance timestamp would be translated to a integer representing an
+ // offset from the unix epoch.
+ ///
+ /// N.B. This has limited type support. ARROW-12976 tracks full implementation.
+ Result<PyObject*> ToPrimitive(const Scalar& scalar);
Review comment:
After thinking about this some more, I don't think it is worth porting over the the "value" property code, only the as_py code. I'm going to rename this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on a change in pull request #11302: ARROW-13806: [C++][Python] Add support for new Interval Type
Posted by GitBox <gi...@apache.org>.
emkornfield commented on a change in pull request #11302:
URL: https://github.com/apache/arrow/pull/11302#discussion_r722429167
##########
File path: cpp/src/arrow/python/arrow_to_python.h
##########
@@ -0,0 +1,59 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+// Functions for converting between pandas's NumPy-based data representation
+// and Arrow data structures
+
+#pragma once
+
+#include "arrow/python/common.h"
+#include "arrow/python/platform.h"
+
+namespace arrow {
+
+class Array;
+struct Scalar;
+
+namespace py {
+
+/// \brief Utility class for converting Arrow to Python obects. A class instead
+///
+/// A class is chosen because in the future some amount of state will be
+/// (e.g. imported python classes), doing this one lazily will be helpful
+/// and having members present avoids static C++ variables.
+class ARROW_PYTHON_EXPORT ArrowToPython {
+ public:
+ /// \brief Converts the given Array to a PyList object. Returns NULL if there
+ /// is an error converting the Array. The list elements are the same ones
+ /// generated via ToLogical()
+ ///
+ /// N.B. This has limited type support. ARROW-12976 tracks extending the
+ /// implementation.
+ Result<PyObject*> ToPyList(const Array& array);
+
+ /// \brief Converts the given Scalar the type that is closest to its arrow
+ /// representation.
+ ///
+ /// For instance timestamp would be translated to a integer representing an
+ // offset from the unix epoch.
Review comment:
Scalars seem to have two notions. a property value which corresponds to the primitive and "as_py" which does the conversion. Previously I had ToLogical() method which was meant to be the analogue of as_py, and I expect this method would be added back as part of ARROW-12976. I'm open to naming recommendations. ToLogical was removed because for this interval type the two end up being the same.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on a change in pull request #11302: ARROW-13806: [C++][Python] Add support for new Interval Type
Posted by GitBox <gi...@apache.org>.
emkornfield commented on a change in pull request #11302:
URL: https://github.com/apache/arrow/pull/11302#discussion_r721894598
##########
File path: python/pyarrow/tests/test_array.py
##########
@@ -2162,6 +2163,101 @@ def test_array_from_numpy_ascii():
assert arrow_arr.equals(expected)
+@pytest.mark.nopandas
Review comment:
removed. before this was testing the different return of to_pylist.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on a change in pull request #11302: ARROW-13806: [C++][Python] Add support for new Interval Type
Posted by GitBox <gi...@apache.org>.
emkornfield commented on a change in pull request #11302:
URL: https://github.com/apache/arrow/pull/11302#discussion_r721891111
##########
File path: cpp/src/arrow/python/python_to_arrow.cc
##########
@@ -68,6 +69,95 @@ using internal::MakeConverter;
namespace py {
+enum class MonthDayNanoField { kMonths, kWeeksAndDays, kDaysOnly, kNanoseconds };
+
+template <MonthDayNanoField field>
+struct MonthDayNanoTraits;
+
+struct MonthDayNanoAttrData {
+ const char* name;
+ const int64_t multiplier;
+};
+
+template <>
+struct MonthDayNanoTraits<MonthDayNanoField::kMonths> {
+ using c_type = int32_t;
+ static constexpr char name[] = "months";
+ static const MonthDayNanoAttrData attrs[];
+};
+
+const MonthDayNanoAttrData MonthDayNanoTraits<MonthDayNanoField::kMonths>::attrs[] = {
+ {"years", 1}, {"months", /*months_in_year=*/12}, {nullptr, 0}};
+
+template <>
+struct MonthDayNanoTraits<MonthDayNanoField::kWeeksAndDays> {
+ using c_type = int32_t;
+ static constexpr char name[] = "days";
+ static const MonthDayNanoAttrData attrs[];
+};
+
+const MonthDayNanoAttrData MonthDayNanoTraits<MonthDayNanoField::kWeeksAndDays>::attrs[] =
+ {{"weeks", 1}, {"days", /*days_in_week=*/7}, {nullptr, 0}};
+
+template <>
+struct MonthDayNanoTraits<MonthDayNanoField::kDaysOnly> {
+ using c_type = int32_t;
+ static constexpr char name[] = "days";
+ static const MonthDayNanoAttrData attrs[];
+};
+
+const MonthDayNanoAttrData MonthDayNanoTraits<MonthDayNanoField::kDaysOnly>::attrs[] = {
+ {"days", 1}, {nullptr, 0}};
+
+template <>
+struct MonthDayNanoTraits<MonthDayNanoField::kNanoseconds> {
+ using c_type = int64_t;
+ static constexpr char name[] = "nanoseconds";
+ static const MonthDayNanoAttrData attrs[];
+};
+
+const MonthDayNanoAttrData MonthDayNanoTraits<MonthDayNanoField::kNanoseconds>::attrs[] =
+ {{"hours", 1},
+ {"minutes", /*minutes_in_hours=*/60},
+ {"seconds", /*seconds_in_minute=*/60},
+ {"milliseconds", /*milliseconds_in_seconds*/ 1000},
+ {"microseconds", /*microseconds_in_millseconds=*/1000},
+ {"nanoseconds", /*nanoseconds_in_microseconds=*/1000},
+ {nullptr, 0}};
+
+template <MonthDayNanoField field>
+struct PopulateMonthDayNano {
+ using Traits = MonthDayNanoTraits<field>;
Review comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on a change in pull request #11302: ARROW-13806: [C++][Python] Add support for new Interval Type
Posted by GitBox <gi...@apache.org>.
emkornfield commented on a change in pull request #11302:
URL: https://github.com/apache/arrow/pull/11302#discussion_r721892098
##########
File path: cpp/src/arrow/python/python_to_arrow.cc
##########
@@ -68,6 +69,95 @@ using internal::MakeConverter;
namespace py {
+enum class MonthDayNanoField { kMonths, kWeeksAndDays, kDaysOnly, kNanoseconds };
+
+template <MonthDayNanoField field>
+struct MonthDayNanoTraits;
+
+struct MonthDayNanoAttrData {
+ const char* name;
+ const int64_t multiplier;
+};
+
+template <>
+struct MonthDayNanoTraits<MonthDayNanoField::kMonths> {
+ using c_type = int32_t;
+ static constexpr char name[] = "months";
+ static const MonthDayNanoAttrData attrs[];
+};
+
+const MonthDayNanoAttrData MonthDayNanoTraits<MonthDayNanoField::kMonths>::attrs[] = {
+ {"years", 1}, {"months", /*months_in_year=*/12}, {nullptr, 0}};
+
+template <>
+struct MonthDayNanoTraits<MonthDayNanoField::kWeeksAndDays> {
+ using c_type = int32_t;
+ static constexpr char name[] = "days";
+ static const MonthDayNanoAttrData attrs[];
+};
+
+const MonthDayNanoAttrData MonthDayNanoTraits<MonthDayNanoField::kWeeksAndDays>::attrs[] =
+ {{"weeks", 1}, {"days", /*days_in_week=*/7}, {nullptr, 0}};
+
+template <>
+struct MonthDayNanoTraits<MonthDayNanoField::kDaysOnly> {
+ using c_type = int32_t;
+ static constexpr char name[] = "days";
+ static const MonthDayNanoAttrData attrs[];
+};
+
+const MonthDayNanoAttrData MonthDayNanoTraits<MonthDayNanoField::kDaysOnly>::attrs[] = {
+ {"days", 1}, {nullptr, 0}};
+
+template <>
+struct MonthDayNanoTraits<MonthDayNanoField::kNanoseconds> {
+ using c_type = int64_t;
+ static constexpr char name[] = "nanoseconds";
+ static const MonthDayNanoAttrData attrs[];
+};
+
+const MonthDayNanoAttrData MonthDayNanoTraits<MonthDayNanoField::kNanoseconds>::attrs[] =
+ {{"hours", 1},
+ {"minutes", /*minutes_in_hours=*/60},
+ {"seconds", /*seconds_in_minute=*/60},
+ {"milliseconds", /*milliseconds_in_seconds*/ 1000},
+ {"microseconds", /*microseconds_in_millseconds=*/1000},
+ {"nanoseconds", /*nanoseconds_in_microseconds=*/1000},
+ {nullptr, 0}};
+
+template <MonthDayNanoField field>
+struct PopulateMonthDayNano {
+ using Traits = MonthDayNanoTraits<field>;
+ inline static Status Field(PyObject* obj,
+ typename MonthDayNanoTraits<field>::c_type* out,
+ bool* found_attrs) {
+ *out = 0;
+ for (const MonthDayNanoAttrData* attr = &Traits::attrs[0]; attr->multiplier != 0;
+ ++attr) {
+ if (attr->multiplier != 1 &&
+ ::arrow::internal::MultiplyWithOverflow(
+ static_cast<typename Traits::c_type>(attr->multiplier), *out, out)) {
+ return Status::Invalid("Overflow on: ", (attr - 1)->name);
+ }
+ if (PyObject_HasAttrString(obj, attr->name)) {
+ OwnedRef field_value(PyObject_GetAttrString(obj, attr->name));
+ RETURN_IF_PYERROR();
+ *found_attrs = true;
+ if (field_value.obj() == Py_None) {
Review comment:
i think I mean null here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on a change in pull request #11302: ARROW-13806: [C++][Python] Add support for new Interval Type
Posted by GitBox <gi...@apache.org>.
emkornfield commented on a change in pull request #11302:
URL: https://github.com/apache/arrow/pull/11302#discussion_r721909647
##########
File path: cpp/src/arrow/python/python_to_arrow.cc
##########
@@ -68,6 +69,95 @@ using internal::MakeConverter;
namespace py {
+enum class MonthDayNanoField { kMonths, kWeeksAndDays, kDaysOnly, kNanoseconds };
Review comment:
After moving code around i was able to move the new ones to private. I'll play around with adding some more.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on a change in pull request #11302: ARROW-13806: [C++][Python] Add support for new Interval Type
Posted by GitBox <gi...@apache.org>.
emkornfield commented on a change in pull request #11302:
URL: https://github.com/apache/arrow/pull/11302#discussion_r722427632
##########
File path: cpp/src/arrow/python/arrow_to_python.h
##########
@@ -0,0 +1,59 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+// Functions for converting between pandas's NumPy-based data representation
Review comment:
I'll update the comment, I removed the numpy/pandas stuff for now, not sure if it will be required here or of pandas/numpy can continue to live in arrow_to_pandas as part of ARROW-12976
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on a change in pull request #11302: ARROW-13806: [C++][Python] Add support for new Interval Type
Posted by GitBox <gi...@apache.org>.
emkornfield commented on a change in pull request #11302:
URL: https://github.com/apache/arrow/pull/11302#discussion_r721897842
##########
File path: python/pyarrow/tests/test_array.py
##########
@@ -2162,6 +2163,101 @@ def test_array_from_numpy_ascii():
assert arrow_arr.equals(expected)
+@pytest.mark.nopandas
+def test_interval_array_from_timedelta():
+ data = [
+ None,
+ datetime.timedelta(days=1, seconds=1, microseconds=1,
+ milliseconds=1, minutes=1, hours=1, weeks=1)]
+
+ # From timedelta (explicit type required)
+ arr = pa.array(data, pa.month_day_nano_interval())
+ assert isinstance(arr, pa.MonthDayNanoIntervalArray)
+ assert arr.type == pa.month_day_nano_interval()
+ expected_list = [
+ None,
+ pa.MonthDayNanoTuple([0, 8,
+ (datetime.timedelta(seconds=1, microseconds=1,
+ milliseconds=1, minutes=1,
+ hours=1) //
+ datetime.timedelta(microseconds=1)) * 1000])]
+ expected = pa.array(expected_list)
+ assert arr.equals(expected)
+ assert arr.to_pylist() == expected_list
+
+# dateutil is dependency of pandas
+
+
+@pytest.mark.pandas
+def test_interval_array_from_relativedelta():
+ from dateutil.relativedelta import relativedelta
+ from pandas.tseries.offsets import DateOffset
+ data = [
+ None,
+ relativedelta(years=1, months=1,
+ days=1, seconds=1, microseconds=1,
+ minutes=1, hours=1, weeks=1, leapdays=1)]
+ # Note leapdays are ignored.
Review comment:
added to pa.array(...) doc string.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on a change in pull request #11302: ARROW-13806: [C++][Python] Add support for new Interval Type
Posted by GitBox <gi...@apache.org>.
emkornfield commented on a change in pull request #11302:
URL: https://github.com/apache/arrow/pull/11302#discussion_r721892098
##########
File path: cpp/src/arrow/python/python_to_arrow.cc
##########
@@ -68,6 +69,95 @@ using internal::MakeConverter;
namespace py {
+enum class MonthDayNanoField { kMonths, kWeeksAndDays, kDaysOnly, kNanoseconds };
+
+template <MonthDayNanoField field>
+struct MonthDayNanoTraits;
+
+struct MonthDayNanoAttrData {
+ const char* name;
+ const int64_t multiplier;
+};
+
+template <>
+struct MonthDayNanoTraits<MonthDayNanoField::kMonths> {
+ using c_type = int32_t;
+ static constexpr char name[] = "months";
+ static const MonthDayNanoAttrData attrs[];
+};
+
+const MonthDayNanoAttrData MonthDayNanoTraits<MonthDayNanoField::kMonths>::attrs[] = {
+ {"years", 1}, {"months", /*months_in_year=*/12}, {nullptr, 0}};
+
+template <>
+struct MonthDayNanoTraits<MonthDayNanoField::kWeeksAndDays> {
+ using c_type = int32_t;
+ static constexpr char name[] = "days";
+ static const MonthDayNanoAttrData attrs[];
+};
+
+const MonthDayNanoAttrData MonthDayNanoTraits<MonthDayNanoField::kWeeksAndDays>::attrs[] =
+ {{"weeks", 1}, {"days", /*days_in_week=*/7}, {nullptr, 0}};
+
+template <>
+struct MonthDayNanoTraits<MonthDayNanoField::kDaysOnly> {
+ using c_type = int32_t;
+ static constexpr char name[] = "days";
+ static const MonthDayNanoAttrData attrs[];
+};
+
+const MonthDayNanoAttrData MonthDayNanoTraits<MonthDayNanoField::kDaysOnly>::attrs[] = {
+ {"days", 1}, {nullptr, 0}};
+
+template <>
+struct MonthDayNanoTraits<MonthDayNanoField::kNanoseconds> {
+ using c_type = int64_t;
+ static constexpr char name[] = "nanoseconds";
+ static const MonthDayNanoAttrData attrs[];
+};
+
+const MonthDayNanoAttrData MonthDayNanoTraits<MonthDayNanoField::kNanoseconds>::attrs[] =
+ {{"hours", 1},
+ {"minutes", /*minutes_in_hours=*/60},
+ {"seconds", /*seconds_in_minute=*/60},
+ {"milliseconds", /*milliseconds_in_seconds*/ 1000},
+ {"microseconds", /*microseconds_in_millseconds=*/1000},
+ {"nanoseconds", /*nanoseconds_in_microseconds=*/1000},
+ {nullptr, 0}};
+
+template <MonthDayNanoField field>
+struct PopulateMonthDayNano {
+ using Traits = MonthDayNanoTraits<field>;
+ inline static Status Field(PyObject* obj,
+ typename MonthDayNanoTraits<field>::c_type* out,
+ bool* found_attrs) {
+ *out = 0;
+ for (const MonthDayNanoAttrData* attr = &Traits::attrs[0]; attr->multiplier != 0;
+ ++attr) {
+ if (attr->multiplier != 1 &&
+ ::arrow::internal::MultiplyWithOverflow(
+ static_cast<typename Traits::c_type>(attr->multiplier), *out, out)) {
+ return Status::Invalid("Overflow on: ", (attr - 1)->name);
+ }
+ if (PyObject_HasAttrString(obj, attr->name)) {
+ OwnedRef field_value(PyObject_GetAttrString(obj, attr->name));
+ RETURN_IF_PYERROR();
+ *found_attrs = true;
+ if (field_value.obj() == Py_None) {
Review comment:
i thought it might have been a valid value. remove this check.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] westonpace commented on a change in pull request #11302: ARROW-13806: [C++][Python] Add support for new MonthDayNano Interval Type
Posted by GitBox <gi...@apache.org>.
westonpace commented on a change in pull request #11302:
URL: https://github.com/apache/arrow/pull/11302#discussion_r723744352
##########
File path: python/pyarrow/types.pxi
##########
@@ -2176,6 +2176,14 @@ def duration(unit):
return out
+def month_day_nano_interval():
+ """
+ Create instance of an interval representing months, days and nanoseconds
+ between two dates.
Review comment:
```suggestion
A type for intervals representing months, days and nanoseconds
between two dates.
```
Nit: Primitive types are cached so no instances are created (usually) and the wording here sounds like it is creating intervals and not types.
##########
File path: python/pyarrow/array.pxi
##########
@@ -163,6 +163,12 @@ def array(object obj, type=None, mask=None, size=None, from_pandas=None,
representation). Timezone-naive data will be implicitly interpreted as
UTC.
+ Pandas's DateOffsets and dateutil.relativedelta.relativedelta are by
+ default converted as MonthDayNanoIntervalArray. relativedelta leapday's
+ are ignored as are all absolute fields on both objects. datetime.timedelta
+ can also be converted to MonthDayNanoIntervalArray but require passing
+ MonthDayNanoIntervalType explicitly.
Review comment:
```suggestion
can also be converted to MonthDayNanoIntervalArray but this requires
passing MonthDayNanoIntervalType explicitly.
```
##########
File path: cpp/src/arrow/python/datetime.cc
##########
@@ -71,6 +74,26 @@ bool MatchFixedOffset(const std::string& tz, util::string_view* sign,
return iter == (tz.data() + tz.size());
}
+static PyTypeObject MonthDayNanoTupleType = {};
+
+constexpr char* NonConst(const char* st) {
+ // Hack for python versions < 3.7 where members of PyStruct members
+ // where non-const (C++ doesn't like assigning string literals to these types)
+ return const_cast<char*>(st);
Review comment:
Do we still support python versions < 3.7? I thought we stopped shipping binary wheels for these versions but maybe we still support building from source.
##########
File path: python/pyarrow/array.pxi
##########
@@ -163,6 +163,12 @@ def array(object obj, type=None, mask=None, size=None, from_pandas=None,
representation). Timezone-naive data will be implicitly interpreted as
UTC.
+ Pandas's DateOffsets and dateutil.relativedelta.relativedelta are by
+ default converted as MonthDayNanoIntervalArray. relativedelta leapday's
Review comment:
```suggestion
default converted as MonthDayNanoIntervalArray. relativedelta leapdays
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou closed pull request #11302: ARROW-13806: [C++][Python] Add support for new MonthDayNano Interval Type
Posted by GitBox <gi...@apache.org>.
pitrou closed pull request #11302:
URL: https://github.com/apache/arrow/pull/11302
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on a change in pull request #11302: ARROW-13806: [C++][Python] Add support for new Interval Type
Posted by GitBox <gi...@apache.org>.
pitrou commented on a change in pull request #11302:
URL: https://github.com/apache/arrow/pull/11302#discussion_r721506360
##########
File path: cpp/src/arrow/python/datetime.cc
##########
@@ -71,6 +71,19 @@ bool MatchFixedOffset(const std::string& tz, util::string_view* sign,
return iter == (tz.data() + tz.size());
}
+static PyTypeObject MonthDayNanoTupleType = {0, 0};
+
+static PyStructSequence_Field MonthDayNanoField[] = {
+ {"months", "The number of months in the interval"},
+ {"days", "The number days in the interval"},
+ {"nanoseconds", "The number of nanoseconds in the interval"},
+ {nullptr, nullptr}};
+
+static PyStructSequence_Desc MonthDayNanoTupleDesc = {
+ "MonthDayNanoTuple", "A interval consistent of months, days and nanoseconds.",
Review comment:
What does "consistent" mean here?
##########
File path: cpp/src/arrow/python/datetime.cc
##########
@@ -270,6 +283,18 @@ static inline Status PyDate_convert_int(int64_t val, const DateUnit unit, int64_
return Status::OK();
}
+PyObject* NewMonthDayNanoTupleType() {
+ if (MonthDayNanoTupleType.tp_name == nullptr) {
+ if (PyStructSequence_InitType2(&MonthDayNanoTupleType, &MonthDayNanoTupleDesc) != 0) {
+ Py_FatalError("Could not initialize MonthDayNanoTuple");
+ }
+ }
+ Py_INCREF(&MonthDayNanoTupleType);
+ return (PyObject*)&MonthDayNanoTupleType;
+}
+
+PyTypeObject* BorrowMonthDayNanoTupleType() { return &MonthDayNanoTupleType; }
Review comment:
This doesn't seem used?
##########
File path: python/pyarrow/scalar.pxi
##########
@@ -514,6 +514,32 @@ cdef class DurationScalar(Scalar):
return datetime.timedelta(microseconds=sp.value // 1000)
+cdef class MonthDayNanoIntervalScalar(Scalar):
+ """
+ Concrete class for month, day, nanosecond scalars.
+ """
+
+ @property
+ def value(self):
+ """
+ Returns this value pyarrow.MonthDayNanoTuple.
+ """
+ cdef PyObject* val
+ val = GetResultValue(ARROW_TO_PYTHON.ToPrimitive(
+ (deref(self.wrapped.get()))))
+ return PyObject_to_object(val)
+
+ def as_py(self):
+ """
+ Return this value as a Pandas DateOffset instance if Pandas is present
+ otherwise as a named tuple containing months days and nanoseconds.
Review comment:
I'm a bit concerned that this makes user code fragile: depending on whether Pandas is installed or not, you'll get an object with a different API.
Instead, perhaps always return a `pa.MonthDayNano`, but add a `MonthDayNano.to_pandas()` method to produce a Pandas DateOffset?
##########
File path: cpp/src/arrow/python/python_to_arrow.cc
##########
@@ -68,6 +69,95 @@ using internal::MakeConverter;
namespace py {
+enum class MonthDayNanoField { kMonths, kWeeksAndDays, kDaysOnly, kNanoseconds };
+
+template <MonthDayNanoField field>
+struct MonthDayNanoTraits;
+
+struct MonthDayNanoAttrData {
+ const char* name;
+ const int64_t multiplier;
+};
+
+template <>
+struct MonthDayNanoTraits<MonthDayNanoField::kMonths> {
+ using c_type = int32_t;
+ static constexpr char name[] = "months";
+ static const MonthDayNanoAttrData attrs[];
+};
+
+const MonthDayNanoAttrData MonthDayNanoTraits<MonthDayNanoField::kMonths>::attrs[] = {
+ {"years", 1}, {"months", /*months_in_year=*/12}, {nullptr, 0}};
+
+template <>
+struct MonthDayNanoTraits<MonthDayNanoField::kWeeksAndDays> {
+ using c_type = int32_t;
+ static constexpr char name[] = "days";
+ static const MonthDayNanoAttrData attrs[];
+};
+
+const MonthDayNanoAttrData MonthDayNanoTraits<MonthDayNanoField::kWeeksAndDays>::attrs[] =
+ {{"weeks", 1}, {"days", /*days_in_week=*/7}, {nullptr, 0}};
+
+template <>
+struct MonthDayNanoTraits<MonthDayNanoField::kDaysOnly> {
+ using c_type = int32_t;
+ static constexpr char name[] = "days";
+ static const MonthDayNanoAttrData attrs[];
+};
+
+const MonthDayNanoAttrData MonthDayNanoTraits<MonthDayNanoField::kDaysOnly>::attrs[] = {
+ {"days", 1}, {nullptr, 0}};
+
+template <>
+struct MonthDayNanoTraits<MonthDayNanoField::kNanoseconds> {
+ using c_type = int64_t;
+ static constexpr char name[] = "nanoseconds";
+ static const MonthDayNanoAttrData attrs[];
+};
+
+const MonthDayNanoAttrData MonthDayNanoTraits<MonthDayNanoField::kNanoseconds>::attrs[] =
+ {{"hours", 1},
+ {"minutes", /*minutes_in_hours=*/60},
+ {"seconds", /*seconds_in_minute=*/60},
+ {"milliseconds", /*milliseconds_in_seconds*/ 1000},
+ {"microseconds", /*microseconds_in_millseconds=*/1000},
+ {"nanoseconds", /*nanoseconds_in_microseconds=*/1000},
+ {nullptr, 0}};
+
+template <MonthDayNanoField field>
+struct PopulateMonthDayNano {
+ using Traits = MonthDayNanoTraits<field>;
+ inline static Status Field(PyObject* obj,
+ typename MonthDayNanoTraits<field>::c_type* out,
+ bool* found_attrs) {
+ *out = 0;
+ for (const MonthDayNanoAttrData* attr = &Traits::attrs[0]; attr->multiplier != 0;
+ ++attr) {
+ if (attr->multiplier != 1 &&
+ ::arrow::internal::MultiplyWithOverflow(
+ static_cast<typename Traits::c_type>(attr->multiplier), *out, out)) {
+ return Status::Invalid("Overflow on: ", (attr - 1)->name);
+ }
+ if (PyObject_HasAttrString(obj, attr->name)) {
+ OwnedRef field_value(PyObject_GetAttrString(obj, attr->name));
Review comment:
If you want to avoid a double hash lookup, you can simply call `PyObject_GetAttrString` and catch the `AttributeError` on failure.
##########
File path: cpp/src/arrow/python/datetime.cc
##########
@@ -450,6 +475,19 @@ Result<std::string> TzinfoToString(PyObject* tzinfo) {
return PyTZInfo_utcoffset_hhmm(tzinfo);
}
+Result<PyObject*> MonthDayNanoIntervalToNamedTuple(
Review comment:
Since NULL is return an error, you don't need to wrap the return value in a `Result<>`.
##########
File path: cpp/src/arrow/python/CMakeLists.txt
##########
@@ -28,6 +28,7 @@ add_dependencies(arrow_python-all arrow_python arrow_python-tests)
set(ARROW_PYTHON_SRCS
arrow_to_pandas.cc
+ arrow_to_python.cc
Review comment:
Looks like you forgot to add this file?
##########
File path: python/pyarrow/tests/test_array.py
##########
@@ -2162,6 +2163,101 @@ def test_array_from_numpy_ascii():
assert arrow_arr.equals(expected)
+@pytest.mark.nopandas
Review comment:
"nopandas" is for tests that fail if Pandas is installed, I think.
##########
File path: python/pyarrow/array.pxi
##########
@@ -1508,6 +1508,30 @@ cdef class DurationArray(NumericArray):
Concrete class for Arrow arrays of duration data type.
"""
+cdef class MonthDayNanoIntervalArray(Array):
Review comment:
Nit, but please let's remain consistent about spacing between classes.
##########
File path: python/pyarrow/lib.pyx
##########
@@ -37,6 +37,10 @@ arrow_init_numpy()
# (used from some of our C++ code, see e.g. ARROW-5260)
import_pyarrow()
+cdef libarrow.ArrowToPython ARROW_TO_PYTHON
+
+MonthDayNanoTuple = NewMonthDayNanoTupleType()
Review comment:
Nit, but I would simply call it `MonthDayNano`.
##########
File path: python/pyarrow/array.pxi
##########
@@ -1508,6 +1508,30 @@ cdef class DurationArray(NumericArray):
Concrete class for Arrow arrays of duration data type.
"""
+cdef class MonthDayNanoIntervalArray(Array):
+ """
+ Concrete class for Arrow arrays of interval[MonthDayNano] type.
+ """
+
+ def to_pylist(self):
+ """
+ Convert to a list of native Python objects.
+
+ is installed the objects will be
+ pd.tseries.offsets.DateOffset objects. Otherwise they are
+ pyarrow.MonthDayNanoTuple objects.
Review comment:
Same remark as for the corresponding scalar class. Also, it seems the docstring is a bit garbled?
##########
File path: python/pyarrow/lib.pyx
##########
@@ -37,6 +37,10 @@ arrow_init_numpy()
# (used from some of our C++ code, see e.g. ARROW-5260)
import_pyarrow()
+cdef libarrow.ArrowToPython ARROW_TO_PYTHON
Review comment:
This seems rather cryptic. What does this do? Is this a globally defined converter?
(is initialization costly?)
##########
File path: cpp/src/arrow/python/python_to_arrow.cc
##########
@@ -68,6 +69,95 @@ using internal::MakeConverter;
namespace py {
+enum class MonthDayNanoField { kMonths, kWeeksAndDays, kDaysOnly, kNanoseconds };
Review comment:
Since we are at it, can you perhaps enclose all private APIs in the anonymous namespace?
##########
File path: python/pyarrow/tests/test_array.py
##########
@@ -2162,6 +2163,101 @@ def test_array_from_numpy_ascii():
assert arrow_arr.equals(expected)
+@pytest.mark.nopandas
+def test_interval_array_from_timedelta():
+ data = [
+ None,
+ datetime.timedelta(days=1, seconds=1, microseconds=1,
+ milliseconds=1, minutes=1, hours=1, weeks=1)]
+
+ # From timedelta (explicit type required)
+ arr = pa.array(data, pa.month_day_nano_interval())
+ assert isinstance(arr, pa.MonthDayNanoIntervalArray)
+ assert arr.type == pa.month_day_nano_interval()
+ expected_list = [
+ None,
+ pa.MonthDayNanoTuple([0, 8,
Review comment:
Can you test with a non-zero number of months?
##########
File path: cpp/src/arrow/python/python_to_arrow.cc
##########
@@ -68,6 +69,95 @@ using internal::MakeConverter;
namespace py {
+enum class MonthDayNanoField { kMonths, kWeeksAndDays, kDaysOnly, kNanoseconds };
+
+template <MonthDayNanoField field>
+struct MonthDayNanoTraits;
+
+struct MonthDayNanoAttrData {
+ const char* name;
+ const int64_t multiplier;
+};
+
+template <>
+struct MonthDayNanoTraits<MonthDayNanoField::kMonths> {
+ using c_type = int32_t;
+ static constexpr char name[] = "months";
+ static const MonthDayNanoAttrData attrs[];
+};
+
+const MonthDayNanoAttrData MonthDayNanoTraits<MonthDayNanoField::kMonths>::attrs[] = {
+ {"years", 1}, {"months", /*months_in_year=*/12}, {nullptr, 0}};
+
+template <>
+struct MonthDayNanoTraits<MonthDayNanoField::kWeeksAndDays> {
+ using c_type = int32_t;
+ static constexpr char name[] = "days";
+ static const MonthDayNanoAttrData attrs[];
+};
+
+const MonthDayNanoAttrData MonthDayNanoTraits<MonthDayNanoField::kWeeksAndDays>::attrs[] =
+ {{"weeks", 1}, {"days", /*days_in_week=*/7}, {nullptr, 0}};
+
+template <>
+struct MonthDayNanoTraits<MonthDayNanoField::kDaysOnly> {
+ using c_type = int32_t;
+ static constexpr char name[] = "days";
+ static const MonthDayNanoAttrData attrs[];
+};
+
+const MonthDayNanoAttrData MonthDayNanoTraits<MonthDayNanoField::kDaysOnly>::attrs[] = {
+ {"days", 1}, {nullptr, 0}};
+
+template <>
+struct MonthDayNanoTraits<MonthDayNanoField::kNanoseconds> {
+ using c_type = int64_t;
+ static constexpr char name[] = "nanoseconds";
+ static const MonthDayNanoAttrData attrs[];
+};
+
+const MonthDayNanoAttrData MonthDayNanoTraits<MonthDayNanoField::kNanoseconds>::attrs[] =
+ {{"hours", 1},
+ {"minutes", /*minutes_in_hours=*/60},
+ {"seconds", /*seconds_in_minute=*/60},
+ {"milliseconds", /*milliseconds_in_seconds*/ 1000},
+ {"microseconds", /*microseconds_in_millseconds=*/1000},
+ {"nanoseconds", /*nanoseconds_in_microseconds=*/1000},
+ {nullptr, 0}};
+
+template <MonthDayNanoField field>
+struct PopulateMonthDayNano {
+ using Traits = MonthDayNanoTraits<field>;
Review comment:
Add a `using c_type = ...` as well?
##########
File path: cpp/src/arrow/python/python_to_arrow.cc
##########
@@ -68,6 +69,95 @@ using internal::MakeConverter;
namespace py {
+enum class MonthDayNanoField { kMonths, kWeeksAndDays, kDaysOnly, kNanoseconds };
+
+template <MonthDayNanoField field>
+struct MonthDayNanoTraits;
+
+struct MonthDayNanoAttrData {
+ const char* name;
+ const int64_t multiplier;
+};
+
+template <>
+struct MonthDayNanoTraits<MonthDayNanoField::kMonths> {
+ using c_type = int32_t;
+ static constexpr char name[] = "months";
+ static const MonthDayNanoAttrData attrs[];
+};
+
+const MonthDayNanoAttrData MonthDayNanoTraits<MonthDayNanoField::kMonths>::attrs[] = {
+ {"years", 1}, {"months", /*months_in_year=*/12}, {nullptr, 0}};
+
+template <>
+struct MonthDayNanoTraits<MonthDayNanoField::kWeeksAndDays> {
+ using c_type = int32_t;
+ static constexpr char name[] = "days";
+ static const MonthDayNanoAttrData attrs[];
+};
+
+const MonthDayNanoAttrData MonthDayNanoTraits<MonthDayNanoField::kWeeksAndDays>::attrs[] =
+ {{"weeks", 1}, {"days", /*days_in_week=*/7}, {nullptr, 0}};
+
+template <>
+struct MonthDayNanoTraits<MonthDayNanoField::kDaysOnly> {
+ using c_type = int32_t;
+ static constexpr char name[] = "days";
+ static const MonthDayNanoAttrData attrs[];
+};
+
+const MonthDayNanoAttrData MonthDayNanoTraits<MonthDayNanoField::kDaysOnly>::attrs[] = {
+ {"days", 1}, {nullptr, 0}};
+
+template <>
+struct MonthDayNanoTraits<MonthDayNanoField::kNanoseconds> {
+ using c_type = int64_t;
+ static constexpr char name[] = "nanoseconds";
+ static const MonthDayNanoAttrData attrs[];
+};
+
+const MonthDayNanoAttrData MonthDayNanoTraits<MonthDayNanoField::kNanoseconds>::attrs[] =
+ {{"hours", 1},
+ {"minutes", /*minutes_in_hours=*/60},
+ {"seconds", /*seconds_in_minute=*/60},
+ {"milliseconds", /*milliseconds_in_seconds*/ 1000},
+ {"microseconds", /*microseconds_in_millseconds=*/1000},
+ {"nanoseconds", /*nanoseconds_in_microseconds=*/1000},
+ {nullptr, 0}};
+
+template <MonthDayNanoField field>
+struct PopulateMonthDayNano {
+ using Traits = MonthDayNanoTraits<field>;
+ inline static Status Field(PyObject* obj,
+ typename MonthDayNanoTraits<field>::c_type* out,
+ bool* found_attrs) {
+ *out = 0;
+ for (const MonthDayNanoAttrData* attr = &Traits::attrs[0]; attr->multiplier != 0;
+ ++attr) {
+ if (attr->multiplier != 1 &&
+ ::arrow::internal::MultiplyWithOverflow(
+ static_cast<typename Traits::c_type>(attr->multiplier), *out, out)) {
+ return Status::Invalid("Overflow on: ", (attr - 1)->name);
+ }
+ if (PyObject_HasAttrString(obj, attr->name)) {
+ OwnedRef field_value(PyObject_GetAttrString(obj, attr->name));
+ RETURN_IF_PYERROR();
+ *found_attrs = true;
+ if (field_value.obj() == Py_None) {
Review comment:
What does `None` mean in this context?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jorisvandenbossche commented on a change in pull request #11302: ARROW-13806: [C++][Python] Add support for new MonthDayNano Interval Type
Posted by GitBox <gi...@apache.org>.
jorisvandenbossche commented on a change in pull request #11302:
URL: https://github.com/apache/arrow/pull/11302#discussion_r723034268
##########
File path: python/pyarrow/scalar.pxi
##########
@@ -514,6 +514,32 @@ cdef class DurationScalar(Scalar):
return datetime.timedelta(microseconds=sp.value // 1000)
+cdef class MonthDayNanoIntervalScalar(Scalar):
+ """
+ Concrete class for month, day, nanosecond scalars.
+ """
+
+ @property
+ def value(self):
+ """
+ Returns this value pyarrow.MonthDayNanoTuple.
+ """
+ cdef PyObject* val
+ val = GetResultValue(ARROW_TO_PYTHON.ToPrimitive(
+ (deref(self.wrapped.get()))))
+ return PyObject_to_object(val)
+
+ def as_py(self):
+ """
+ Return this value as a Pandas DateOffset instance if Pandas is present
+ otherwise as a named tuple containing months days and nanoseconds.
Review comment:
Yes, I also agree returning a consistent type here is best (for Timestamp there is the complication of datetime.datetime not supporting nanoseconds ..). As Weston mentions, we can see later if we want to add some option to get a more pandas native object.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on a change in pull request #11302: ARROW-13806: [C++][Python] Add support for new MonthDayNano Interval Type
Posted by GitBox <gi...@apache.org>.
emkornfield commented on a change in pull request #11302:
URL: https://github.com/apache/arrow/pull/11302#discussion_r723688517
##########
File path: cpp/src/arrow/python/helpers.cc
##########
@@ -321,6 +323,14 @@ void InitPandasStaticData() {
pandas_NA = ref.obj();
}
+ // Import DateOffset type
+ OwnedRef offsets;
+ if (internal::ImportModule("pandas.tseries.offsets", &offsets).ok()) {
Review comment:
done in next commit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on a change in pull request #11302: ARROW-13806: [C++][Python] Add support for new Interval Type
Posted by GitBox <gi...@apache.org>.
emkornfield commented on a change in pull request #11302:
URL: https://github.com/apache/arrow/pull/11302#discussion_r722778098
##########
File path: python/pyarrow/tests/test_scalars.py
##########
@@ -364,6 +367,14 @@ def test_duration_nanos_nopandas():
arr[0].as_py()
+def test_month_day_nano_interval():
+ triple = pa.MonthDayNano([3600, 3600, 3600])
Review comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on a change in pull request #11302: ARROW-13806: [C++][Python] Add support for new Interval Type
Posted by GitBox <gi...@apache.org>.
pitrou commented on a change in pull request #11302:
URL: https://github.com/apache/arrow/pull/11302#discussion_r721539304
##########
File path: python/pyarrow/scalar.pxi
##########
@@ -514,6 +514,32 @@ cdef class DurationScalar(Scalar):
return datetime.timedelta(microseconds=sp.value // 1000)
+cdef class MonthDayNanoIntervalScalar(Scalar):
+ """
+ Concrete class for month, day, nanosecond scalars.
+ """
+
+ @property
+ def value(self):
+ """
+ Returns this value pyarrow.MonthDayNanoTuple.
+ """
+ cdef PyObject* val
+ val = GetResultValue(ARROW_TO_PYTHON.ToPrimitive(
+ (deref(self.wrapped.get()))))
+ return PyObject_to_object(val)
+
+ def as_py(self):
+ """
+ Return this value as a Pandas DateOffset instance if Pandas is present
+ otherwise as a named tuple containing months days and nanoseconds.
Review comment:
Hmm, it's a bit unfortunate for timestamps, but the decision there is understandable: returning a `datetime.datetime` loses precision. Also, `pandas.Timestamp` is probably API-compatible with `datetime.datetime`.
You don't lose any precision by returning a `MonthDayNano`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on a change in pull request #11302: ARROW-13806: [C++][Python] Add support for new Interval Type
Posted by GitBox <gi...@apache.org>.
emkornfield commented on a change in pull request #11302:
URL: https://github.com/apache/arrow/pull/11302#discussion_r721893527
##########
File path: python/pyarrow/array.pxi
##########
@@ -1508,6 +1508,30 @@ cdef class DurationArray(NumericArray):
Concrete class for Arrow arrays of duration data type.
"""
+cdef class MonthDayNanoIntervalArray(Array):
Review comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on a change in pull request #11302: ARROW-13806: [C++][Python] Add support for new Interval Type
Posted by GitBox <gi...@apache.org>.
emkornfield commented on a change in pull request #11302:
URL: https://github.com/apache/arrow/pull/11302#discussion_r721891998
##########
File path: cpp/src/arrow/python/python_to_arrow.cc
##########
@@ -68,6 +69,95 @@ using internal::MakeConverter;
namespace py {
+enum class MonthDayNanoField { kMonths, kWeeksAndDays, kDaysOnly, kNanoseconds };
+
+template <MonthDayNanoField field>
+struct MonthDayNanoTraits;
+
+struct MonthDayNanoAttrData {
+ const char* name;
+ const int64_t multiplier;
+};
+
+template <>
+struct MonthDayNanoTraits<MonthDayNanoField::kMonths> {
+ using c_type = int32_t;
+ static constexpr char name[] = "months";
+ static const MonthDayNanoAttrData attrs[];
+};
+
+const MonthDayNanoAttrData MonthDayNanoTraits<MonthDayNanoField::kMonths>::attrs[] = {
+ {"years", 1}, {"months", /*months_in_year=*/12}, {nullptr, 0}};
+
+template <>
+struct MonthDayNanoTraits<MonthDayNanoField::kWeeksAndDays> {
+ using c_type = int32_t;
+ static constexpr char name[] = "days";
+ static const MonthDayNanoAttrData attrs[];
+};
+
+const MonthDayNanoAttrData MonthDayNanoTraits<MonthDayNanoField::kWeeksAndDays>::attrs[] =
+ {{"weeks", 1}, {"days", /*days_in_week=*/7}, {nullptr, 0}};
+
+template <>
+struct MonthDayNanoTraits<MonthDayNanoField::kDaysOnly> {
+ using c_type = int32_t;
+ static constexpr char name[] = "days";
+ static const MonthDayNanoAttrData attrs[];
+};
+
+const MonthDayNanoAttrData MonthDayNanoTraits<MonthDayNanoField::kDaysOnly>::attrs[] = {
+ {"days", 1}, {nullptr, 0}};
+
+template <>
+struct MonthDayNanoTraits<MonthDayNanoField::kNanoseconds> {
+ using c_type = int64_t;
+ static constexpr char name[] = "nanoseconds";
+ static const MonthDayNanoAttrData attrs[];
+};
+
+const MonthDayNanoAttrData MonthDayNanoTraits<MonthDayNanoField::kNanoseconds>::attrs[] =
+ {{"hours", 1},
+ {"minutes", /*minutes_in_hours=*/60},
+ {"seconds", /*seconds_in_minute=*/60},
+ {"milliseconds", /*milliseconds_in_seconds*/ 1000},
+ {"microseconds", /*microseconds_in_millseconds=*/1000},
+ {"nanoseconds", /*nanoseconds_in_microseconds=*/1000},
+ {nullptr, 0}};
+
+template <MonthDayNanoField field>
+struct PopulateMonthDayNano {
+ using Traits = MonthDayNanoTraits<field>;
+ inline static Status Field(PyObject* obj,
+ typename MonthDayNanoTraits<field>::c_type* out,
+ bool* found_attrs) {
+ *out = 0;
+ for (const MonthDayNanoAttrData* attr = &Traits::attrs[0]; attr->multiplier != 0;
+ ++attr) {
+ if (attr->multiplier != 1 &&
+ ::arrow::internal::MultiplyWithOverflow(
+ static_cast<typename Traits::c_type>(attr->multiplier), *out, out)) {
+ return Status::Invalid("Overflow on: ", (attr - 1)->name);
+ }
+ if (PyObject_HasAttrString(obj, attr->name)) {
+ OwnedRef field_value(PyObject_GetAttrString(obj, attr->name));
Review comment:
Good point. Done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on a change in pull request #11302: ARROW-13806: [C++][Python] Add support for new Interval Type
Posted by GitBox <gi...@apache.org>.
emkornfield commented on a change in pull request #11302:
URL: https://github.com/apache/arrow/pull/11302#discussion_r721878423
##########
File path: python/pyarrow/scalar.pxi
##########
@@ -514,6 +514,32 @@ cdef class DurationScalar(Scalar):
return datetime.timedelta(microseconds=sp.value // 1000)
+cdef class MonthDayNanoIntervalScalar(Scalar):
+ """
+ Concrete class for month, day, nanosecond scalars.
+ """
+
+ @property
+ def value(self):
+ """
+ Returns this value pyarrow.MonthDayNanoTuple.
+ """
+ cdef PyObject* val
+ val = GetResultValue(ARROW_TO_PYTHON.ToPrimitive(
+ (deref(self.wrapped.get()))))
+ return PyObject_to_object(val)
+
+ def as_py(self):
+ """
+ Return this value as a Pandas DateOffset instance if Pandas is present
+ otherwise as a named tuple containing months days and nanoseconds.
Review comment:
Going to change. This actually simplifies the code a lot.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on a change in pull request #11302: ARROW-13806: [C++][Python] Add support for new Interval Type
Posted by GitBox <gi...@apache.org>.
emkornfield commented on a change in pull request #11302:
URL: https://github.com/apache/arrow/pull/11302#discussion_r721869652
##########
File path: cpp/src/arrow/python/arrow_to_python.h
##########
@@ -0,0 +1,161 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+// Functions for converting between pandas's NumPy-based data representation
+// and Arrow data structures
+
+#pragma once
+
+#include "arrow/chunked_array.h"
+#include "arrow/python/common.h"
+#include "arrow/python/platform.h"
+#include "arrow/util/hashing.h"
+
+namespace arrow {
+
+class Array;
+class Scalar;
+
+namespace py {
+
+struct ArrowToPythonObjectOptions {
+ MemoryPool* pool = default_memory_pool();
+ bool deduplicate_objects = false;
+};
+
+class ARROW_PYTHON_EXPORT ArrowToPython {
+ public:
+ // \brief Converts the given Array to a PyList object. Returns NULL if there
+ // is an error converting the Array. The list elements are the same ones
+ // generated via ToLogical()
+ //
+ // N.B. This has limited type support. ARROW-12976 tracks extending the implementation.
+ Result<PyObject*> ToPyList(const Array& array);
+
+ // Populates out_objects with the result of converting the array values
+ // to python objects. The same logic as ToLogical().
+ //
+ // N.B. Not all types are supported. ARROW-12976 tracks extending the implementation.
+ Status ToNumpyObjectArray(const ArrowToPythonObjectOptions& options,
+ const ChunkedArray& array, PyObject** out_objects);
+
+ // \brief Converts the given Scalar to a python object that best corresponds
+ // with the Scalar's type.
+ //
+ // For example timestamp[ms] is translated into datetime.datetime.
+ //
+ // N.B. This has limited type support. ARROW-12976 tracks extending the implementation.
+ Result<PyObject*> ToLogical(const Scalar& scalar);
+
+ // \brief Converts the given Scalar the type that is closed to its arrow
+ // representation.
+ //
+ // For instance timestamp would be translated to a integer representing an
+ // offset from the unix epoch.
+ //
+ // Returns nullptr on error.
+ //
+ // GIL must be health when calling this method.
+ // N.B. This has limited type support. ARROW-12976 tracks full implementation.
+ Result<PyObject*> ToPrimitive(const Scalar& scalar);
+
+ private:
+ Status Init();
+};
+
+namespace internal {
+// TODO(ARROW-12976): See if we can refactor Pandas ObjectWriter logic
+// to the .cc file and move this there as well if we can.
+
+// Generic Array -> PyObject** converter that handles object deduplication, if
+// requested
+template <typename ArrayType, typename WriteValue, typename Assigner>
+inline Status WriteArrayObjects(const ArrayType& arr, WriteValue&& write_func,
+ Assigner out_values) {
+ // TODO(ARROW-12976): Use visitor here?
+ const bool has_nulls = arr.null_count() > 0;
+ for (int64_t i = 0; i < arr.length(); ++i) {
+ if (has_nulls && arr.IsNull(i)) {
+ Py_INCREF(Py_None);
+ *out_values = Py_None;
+ } else {
+ RETURN_NOT_OK(write_func(arr.GetView(i), out_values));
+ }
+ ++out_values;
+ }
Review comment:
this was moved from existing code base, I don't know but would prefer to handle the TODO above use Visitor in general.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on a change in pull request #11302: ARROW-13806: [C++][Python] Add support for new MonthDayNano Interval Type
Posted by GitBox <gi...@apache.org>.
emkornfield commented on a change in pull request #11302:
URL: https://github.com/apache/arrow/pull/11302#discussion_r723849509
##########
File path: python/pyarrow/types.pxi
##########
@@ -2176,6 +2176,14 @@ def duration(unit):
return out
+def month_day_nano_interval():
+ """
+ Create instance of an interval representing months, days and nanoseconds
+ between two dates.
Review comment:
seems good to me. I liked you other wording also, but we should probably do a general cleanup.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on pull request #11302: ARROW-13806: [C++][Python] Add support for new MonthDayNano Interval Type
Posted by GitBox <gi...@apache.org>.
emkornfield commented on pull request #11302:
URL: https://github.com/apache/arrow/pull/11302#issuecomment-936299002
Is changing the signature and docs to be specific to MonthDayNanoInterval
in this module OK for this PR or do you have another suggestion on
organization? (Move to datetime.h)?
On Wednesday, October 6, 2021, Antoine Pitrou ***@***.***>
wrote:
> ***@***.**** commented on this pull request.
> ------------------------------
>
> In cpp/src/arrow/python/arrow_to_python.cc
> <https://github.com/apache/arrow/pull/11302#discussion_r723139253>:
>
> > + }
> +
> + PyListAssigner& operator+=(int64_t offset) {
> + current_index_ += offset;
> + return *this;
> + }
> +
> + private:
> + PyObject* list_;
> + int64_t current_index_ = 0;
> +};
> +
> +} // namespace
> +
> +Result<PyObject*> ArrowToPython::ToPyList(const Array& array) {
> + RETURN_NOT_OK(CheckInterval(*array.type()));
>
> I see. I'd rather see a comprehensive refactor in another PR than a stub
> like this, which may end up abandoned.
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/apache/arrow/pull/11302#discussion_r723139253>, or
> unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AEIKYDWFIVHB3GUOHI2735TUFQV2RANCNFSM5FITG4GA>
> .
> Triage notifications on the go with GitHub Mobile for iOS
> <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675>
> or Android
> <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.
>
>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on a change in pull request #11302: ARROW-13806: [C++][Python] Add support for new MonthDayNano Interval Type
Posted by GitBox <gi...@apache.org>.
emkornfield commented on a change in pull request #11302:
URL: https://github.com/apache/arrow/pull/11302#discussion_r723837221
##########
File path: python/pyarrow/types.pxi
##########
@@ -2176,6 +2176,14 @@ def duration(unit):
return out
+def month_day_nano_interval():
+ """
+ Create instance of an interval representing months, days and nanoseconds
+ between two dates.
Review comment:
this language is consistent with other methods in the file.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on a change in pull request #11302: ARROW-13806: [C++][Python] Add support for new MonthDayNano Interval Type
Posted by GitBox <gi...@apache.org>.
emkornfield commented on a change in pull request #11302:
URL: https://github.com/apache/arrow/pull/11302#discussion_r723836536
##########
File path: cpp/src/arrow/python/datetime.cc
##########
@@ -71,6 +74,26 @@ bool MatchFixedOffset(const std::string& tz, util::string_view* sign,
return iter == (tz.data() + tz.size());
}
+static PyTypeObject MonthDayNanoTupleType = {};
+
+constexpr char* NonConst(const char* st) {
+ // Hack for python versions < 3.7 where members of PyStruct members
+ // where non-const (C++ doesn't like assigning string literals to these types)
+ return const_cast<char*>(st);
Review comment:
we do, I asked that we ddidn't drop it last release. I would also ask that we don't drop it this release (i.e. keep it through its full python support cycle. There are a number of consumers of pyarrow that try to keep support for python versions until they dropped it which is end of this year). This was caught because we run it in CI.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on pull request #11302: ARROW-13806: [C++][Python] Add support for new Interval Type
Posted by GitBox <gi...@apache.org>.
emkornfield commented on pull request #11302:
URL: https://github.com/apache/arrow/pull/11302#issuecomment-933662628
@pitrou thanks for the feedback, I'll address the ones I haven't commented on. I think the most substantive one is whether return types should be different with or without Pandas. Happy to go with whatever you and @jorisvandenbossche think is best.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on a change in pull request #11302: ARROW-13806: [C++][Python] Add support for new Interval Type
Posted by GitBox <gi...@apache.org>.
pitrou commented on a change in pull request #11302:
URL: https://github.com/apache/arrow/pull/11302#discussion_r721545539
##########
File path: python/pyarrow/lib.pyx
##########
@@ -37,6 +37,10 @@ arrow_init_numpy()
# (used from some of our C++ code, see e.g. ARROW-5260)
import_pyarrow()
+cdef libarrow.ArrowToPython ARROW_TO_PYTHON
Review comment:
I'm ok with having a wrapper object, however we shouldn't import Pandas unconditionally at startup as that's quite costly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on a change in pull request #11302: ARROW-13806: [C++][Python] Add support for new Interval Type
Posted by GitBox <gi...@apache.org>.
emkornfield commented on a change in pull request #11302:
URL: https://github.com/apache/arrow/pull/11302#discussion_r721535058
##########
File path: cpp/src/arrow/python/python_to_arrow.cc
##########
@@ -68,6 +69,95 @@ using internal::MakeConverter;
namespace py {
+enum class MonthDayNanoField { kMonths, kWeeksAndDays, kDaysOnly, kNanoseconds };
Review comment:
i had these in an anonymous namespace originally and got obscure linking errors. I can try it again to see if I get different results.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on a change in pull request #11302: ARROW-13806: [C++][Python] Add support for new Interval Type
Posted by GitBox <gi...@apache.org>.
emkornfield commented on a change in pull request #11302:
URL: https://github.com/apache/arrow/pull/11302#discussion_r722425733
##########
File path: cpp/src/arrow/python/arrow_to_pandas.cc
##########
@@ -1097,6 +1081,40 @@ struct ObjectWriterVisitor {
return Status::OK();
}
+ template <typename Type>
+ enable_if_t<std::is_same<Type, MonthDayNanoIntervalType>::value, Status> Visit(
+ const Type& type) {
+ OwnedRef args(PyTuple_New(0));
+ OwnedRef kwargs(PyDict_New());
+ RETURN_IF_PYERROR();
+ auto to_date_offset = [&](const MonthDayNanoIntervalType::MonthDayNanos& interval,
+ PyObject** out) {
+ DCHECK(internal::BorrowPandasDataOffsetType() != nullptr);
+ // TimeDelta objects do not add nanoseconds component to timestamp.
+ // so convert microseconds and remainder to preserve data
+ // but give users more expected results.
Review comment:
I'll try to clarify but there are actually two bugs:
https://github.com/pandas-dev/pandas/issues/36589
https://github.com/pandas-dev/pandas/issues/43892
This is meant to mitigate the latter. I'll link the second issue here.
CC @tswast as an FYI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] westonpace commented on a change in pull request #11302: ARROW-13806: [C++][Python] Add support for new MonthDayNano Interval Type
Posted by GitBox <gi...@apache.org>.
westonpace commented on a change in pull request #11302:
URL: https://github.com/apache/arrow/pull/11302#discussion_r723842177
##########
File path: python/pyarrow/types.pxi
##########
@@ -2176,6 +2176,14 @@ def duration(unit):
return out
+def month_day_nano_interval():
+ """
+ Create instance of an interval representing months, days and nanoseconds
+ between two dates.
Review comment:
```suggestion
Create instance of an interval type representing months, days and nanoseconds
between two dates.
```
Fair enough. How about a more subtle suggestion then.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on a change in pull request #11302: ARROW-13806: [C++][Python] Add support for new Interval Type
Posted by GitBox <gi...@apache.org>.
emkornfield commented on a change in pull request #11302:
URL: https://github.com/apache/arrow/pull/11302#discussion_r721889488
##########
File path: cpp/src/arrow/python/datetime.cc
##########
@@ -270,6 +283,18 @@ static inline Status PyDate_convert_int(int64_t val, const DateUnit unit, int64_
return Status::OK();
}
+PyObject* NewMonthDayNanoTupleType() {
+ if (MonthDayNanoTupleType.tp_name == nullptr) {
+ if (PyStructSequence_InitType2(&MonthDayNanoTupleType, &MonthDayNanoTupleDesc) != 0) {
+ Py_FatalError("Could not initialize MonthDayNanoTuple");
+ }
+ }
+ Py_INCREF(&MonthDayNanoTupleType);
+ return (PyObject*)&MonthDayNanoTupleType;
+}
+
+PyTypeObject* BorrowMonthDayNanoTupleType() { return &MonthDayNanoTupleType; }
Review comment:
yep, left over from a prior design. removed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jorisvandenbossche commented on a change in pull request #11302: ARROW-13806: [C++][Python] Add support for new MonthDayNano Interval Type
Posted by GitBox <gi...@apache.org>.
jorisvandenbossche commented on a change in pull request #11302:
URL: https://github.com/apache/arrow/pull/11302#discussion_r723001439
##########
File path: cpp/src/arrow/python/helpers.cc
##########
@@ -321,6 +323,14 @@ void InitPandasStaticData() {
pandas_NA = ref.obj();
}
+ // Import DateOffset type
+ OwnedRef offsets;
+ if (internal::ImportModule("pandas.tseries.offsets", &offsets).ok()) {
Review comment:
It seems that DateOffset if available in the top-level pandas namespace, in which case I would import it from there
##########
File path: cpp/src/arrow/python/datetime.h
##########
@@ -178,6 +182,17 @@ Result<PyObject*> StringToTzinfo(const std::string& tz);
ARROW_PYTHON_EXPORT
Result<std::string> TzinfoToString(PyObject* pytzinfo);
+/// Converts MonthDayNano to a python dictionary.
+///
+/// Returns a named tuple (pyarrow.MonthDayNanoTuple) containing attributes
Review comment:
```suggestion
/// Returns a named tuple (pyarrow.MonthDayNano) containing attributes
```
##########
File path: python/pyarrow/array.pxi
##########
@@ -163,6 +163,12 @@ def array(object obj, type=None, mask=None, size=None, from_pandas=None,
representation). Timezone-naive data will be implicitly interpreted as
UTC.
+ Pandas's DateOffsets and dateutil.relativedelta.relativedelta are by
+ default converted as MonthDayNanoIntervalArray. relativedelta leapday's
+ are ignored as are all absolute fields on both objects. datetime.timedelta
+ can also be converted to MonthDayNanoIntervalArray but require passing
+ MonthDayIntervalType explicitly.
Review comment:
```suggestion
MonthDayNanoIntervalType explicitly.
```
##########
File path: cpp/src/arrow/python/datetime.h
##########
@@ -178,6 +182,17 @@ Result<PyObject*> StringToTzinfo(const std::string& tz);
ARROW_PYTHON_EXPORT
Result<std::string> TzinfoToString(PyObject* pytzinfo);
+/// Converts MonthDayNano to a python dictionary.
Review comment:
```suggestion
/// Converts MonthDayNano to a python namedtuple.
```
##########
File path: python/pyarrow/tests/test_pandas.py
##########
@@ -1495,6 +1495,19 @@ def test_timedeltas_nulls(self):
expected_schema=schema,
)
+ def test_month_day_nano_interval(self):
+ from pandas.tseries.offsets import DateOffset
+ df = pd.DataFrame({
+ 'date_offset': [None,
+ DateOffset(days=3600, months=3600, microseconds=3,
+ nanoseconds=600)]
+ })
+ field = pa.field('date_offset', pa.month_day_nano_interval())
+ schema = pa.schema([field])
Review comment:
```suggestion
schema = pa.schema([('date_offset', pa.month_day_nano_interval()])
```
##########
File path: cpp/src/arrow/python/arrow_to_python_internal.h
##########
@@ -0,0 +1,52 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+// Functions for converting between pandas's NumPy-based data representation
+// and Arrow data structures
Review comment:
I think this comment is a left-over from copying another file?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] westonpace commented on a change in pull request #11302: ARROW-13806: [C++][Python] Add support for new Interval Type
Posted by GitBox <gi...@apache.org>.
westonpace commented on a change in pull request #11302:
URL: https://github.com/apache/arrow/pull/11302#discussion_r721820721
##########
File path: python/pyarrow/tests/test_array.py
##########
@@ -2162,6 +2163,101 @@ def test_array_from_numpy_ascii():
assert arrow_arr.equals(expected)
+@pytest.mark.nopandas
+def test_interval_array_from_timedelta():
+ data = [
+ None,
+ datetime.timedelta(days=1, seconds=1, microseconds=1,
+ milliseconds=1, minutes=1, hours=1, weeks=1)]
+
+ # From timedelta (explicit type required)
+ arr = pa.array(data, pa.month_day_nano_interval())
+ assert isinstance(arr, pa.MonthDayNanoIntervalArray)
+ assert arr.type == pa.month_day_nano_interval()
+ expected_list = [
+ None,
+ pa.MonthDayNanoTuple([0, 8,
+ (datetime.timedelta(seconds=1, microseconds=1,
+ milliseconds=1, minutes=1,
+ hours=1) //
+ datetime.timedelta(microseconds=1)) * 1000])]
+ expected = pa.array(expected_list)
+ assert arr.equals(expected)
+ assert arr.to_pylist() == expected_list
+
+# dateutil is dependency of pandas
+
+
+@pytest.mark.pandas
+def test_interval_array_from_relativedelta():
+ from dateutil.relativedelta import relativedelta
+ from pandas.tseries.offsets import DateOffset
+ data = [
+ None,
+ relativedelta(years=1, months=1,
+ days=1, seconds=1, microseconds=1,
+ minutes=1, hours=1, weeks=1, leapdays=1)]
+ # Note leapdays are ignored.
Review comment:
Nit: Can we mention this in at least one place in a docstring.
##########
File path: cpp/src/arrow/python/arrow_to_python.h
##########
@@ -0,0 +1,161 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+// Functions for converting between pandas's NumPy-based data representation
+// and Arrow data structures
+
+#pragma once
+
+#include "arrow/chunked_array.h"
+#include "arrow/python/common.h"
+#include "arrow/python/platform.h"
+#include "arrow/util/hashing.h"
+
+namespace arrow {
+
+class Array;
+class Scalar;
+
+namespace py {
+
+struct ArrowToPythonObjectOptions {
+ MemoryPool* pool = default_memory_pool();
+ bool deduplicate_objects = false;
+};
+
+class ARROW_PYTHON_EXPORT ArrowToPython {
+ public:
+ // \brief Converts the given Array to a PyList object. Returns NULL if there
+ // is an error converting the Array. The list elements are the same ones
+ // generated via ToLogical()
+ //
+ // N.B. This has limited type support. ARROW-12976 tracks extending the implementation.
+ Result<PyObject*> ToPyList(const Array& array);
+
+ // Populates out_objects with the result of converting the array values
+ // to python objects. The same logic as ToLogical().
+ //
+ // N.B. Not all types are supported. ARROW-12976 tracks extending the implementation.
+ Status ToNumpyObjectArray(const ArrowToPythonObjectOptions& options,
+ const ChunkedArray& array, PyObject** out_objects);
+
+ // \brief Converts the given Scalar to a python object that best corresponds
+ // with the Scalar's type.
+ //
+ // For example timestamp[ms] is translated into datetime.datetime.
+ //
+ // N.B. This has limited type support. ARROW-12976 tracks extending the implementation.
+ Result<PyObject*> ToLogical(const Scalar& scalar);
+
+ // \brief Converts the given Scalar the type that is closed to its arrow
+ // representation.
+ //
+ // For instance timestamp would be translated to a integer representing an
+ // offset from the unix epoch.
+ //
+ // Returns nullptr on error.
+ //
+ // GIL must be health when calling this method.
+ // N.B. This has limited type support. ARROW-12976 tracks full implementation.
+ Result<PyObject*> ToPrimitive(const Scalar& scalar);
+
+ private:
+ Status Init();
+};
+
+namespace internal {
+// TODO(ARROW-12976): See if we can refactor Pandas ObjectWriter logic
+// to the .cc file and move this there as well if we can.
+
+// Generic Array -> PyObject** converter that handles object deduplication, if
+// requested
+template <typename ArrayType, typename WriteValue, typename Assigner>
+inline Status WriteArrayObjects(const ArrayType& arr, WriteValue&& write_func,
+ Assigner out_values) {
+ // TODO(ARROW-12976): Use visitor here?
+ const bool has_nulls = arr.null_count() > 0;
+ for (int64_t i = 0; i < arr.length(); ++i) {
+ if (has_nulls && arr.IsNull(i)) {
+ Py_INCREF(Py_None);
+ *out_values = Py_None;
+ } else {
+ RETURN_NOT_OK(write_func(arr.GetView(i), out_values));
+ }
+ ++out_values;
+ }
Review comment:
Is this `has_nulls` enabling some kind of compiler optimization? From a naive read it doesn't look like it is providing any value.
##########
File path: python/pyarrow/scalar.pxi
##########
@@ -514,6 +514,32 @@ cdef class DurationScalar(Scalar):
return datetime.timedelta(microseconds=sp.value // 1000)
+cdef class MonthDayNanoIntervalScalar(Scalar):
+ """
+ Concrete class for month, day, nanosecond scalars.
+ """
+
+ @property
+ def value(self):
+ """
+ Returns this value pyarrow.MonthDayNanoTuple.
Review comment:
```suggestion
Returns this value as a pyarrow.MonthDayNanoTuple.
```
##########
File path: python/pyarrow/types.pxi
##########
@@ -2154,6 +2154,14 @@ def duration(unit):
return out
+def month_day_nano_interval():
+ """
+ Create instance of a interval representing the time between two calendar
+ instances represented as a triple of months, days and nanoseconds.
+ """
+ return primitive_type(_Type_INTERVAL_MONTH_DAY_NANO)
Review comment:
Nit: I would avoid "the time between two calendar instances" because a duration/Joda-interval is traditionally defined as "the time between two instants". I haven't seen intervals / periods defined in this way before (possibly because calendars could change).
Also, instead of `a triple of months` (which might make a python user think 3-tuple) could we say "represented by values for months, days, and nanoseconds"? Or, if we want to be real precise, "represented by a signed 32 bit integer of months, a signed 32 bit integer of days, and a signed 64 bit integer of nanoseconds".
##########
File path: python/pyarrow/scalar.pxi
##########
@@ -514,6 +514,32 @@ cdef class DurationScalar(Scalar):
return datetime.timedelta(microseconds=sp.value // 1000)
+cdef class MonthDayNanoIntervalScalar(Scalar):
+ """
+ Concrete class for month, day, nanosecond scalars.
+ """
+
+ @property
+ def value(self):
+ """
+ Returns this value pyarrow.MonthDayNanoTuple.
+ """
+ cdef PyObject* val
+ val = GetResultValue(ARROW_TO_PYTHON.ToPrimitive(
+ (deref(self.wrapped.get()))))
+ return PyObject_to_object(val)
+
+ def as_py(self):
+ """
+ Return this value as a Pandas DateOffset instance if Pandas is present
+ otherwise as a named tuple containing months days and nanoseconds.
Review comment:
I'm going to agree with @pitrou . With pyarrow we even have cases where users aren't even really aware they are using pyarrow. I could see a circumstance where they install their application on some new machine or environment and suddenly start getting errors. Maybe someday we can add a `use_pandas_types` option to `as_py`, `to_pylist`, and `to_pydict`. That way we can error if the expected library is not available instead of silently changing the type.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on pull request #11302: ARROW-13806: [C++][Python] Add support for new Interval Type
Posted by GitBox <gi...@apache.org>.
emkornfield commented on pull request #11302:
URL: https://github.com/apache/arrow/pull/11302#issuecomment-935071524
> I find it rather unwelcome that this adds an abstraction ("ArrowToPython") that takes care of a single type. I'm not sure if that refactor is potentially useful, but we should have a single facility for conversion to Python objects, not a bunch of unrelated ones.
Mentioned this elsewhere but I was trying to prep layout for ARROW-12976 is this is undesirable, I can make these simple functions for now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on a change in pull request #11302: ARROW-13806: [C++][Python] Add support for new Interval Type
Posted by GitBox <gi...@apache.org>.
emkornfield commented on a change in pull request #11302:
URL: https://github.com/apache/arrow/pull/11302#discussion_r722776638
##########
File path: cpp/src/arrow/python/arrow_to_python.h
##########
@@ -0,0 +1,59 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+// Functions for converting between pandas's NumPy-based data representation
+// and Arrow data structures
+
+#pragma once
+
+#include "arrow/python/common.h"
+#include "arrow/python/platform.h"
+
+namespace arrow {
+
+class Array;
+struct Scalar;
+
+namespace py {
+
+/// \brief Utility class for converting Arrow to Python obects. A class instead
Review comment:
yeah, covered below.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on a change in pull request #11302: ARROW-13806: [C++][Python] Add support for new Interval Type
Posted by GitBox <gi...@apache.org>.
emkornfield commented on a change in pull request #11302:
URL: https://github.com/apache/arrow/pull/11302#discussion_r721901404
##########
File path: python/pyarrow/array.pxi
##########
@@ -1508,6 +1508,30 @@ cdef class DurationArray(NumericArray):
Concrete class for Arrow arrays of duration data type.
"""
+cdef class MonthDayNanoIntervalArray(Array):
+ """
+ Concrete class for Arrow arrays of interval[MonthDayNano] type.
+ """
+
+ def to_pylist(self):
+ """
+ Convert to a list of native Python objects.
+
+ is installed the objects will be
+ pd.tseries.offsets.DateOffset objects. Otherwise they are
+ pyarrow.MonthDayNanoTuple objects.
Review comment:
Should be fixed now (once I push latest set of commits.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on pull request #11302: ARROW-13806: [C++][Python] Add support for new MonthDayNano Interval Type
Posted by GitBox <gi...@apache.org>.
emkornfield commented on pull request #11302:
URL: https://github.com/apache/arrow/pull/11302#issuecomment-937291121
@jorisvandenbossche thanks for the feedback. Please see responses inline
> Looks good! Added a few inline comments. And some additional non-inline comments:
>
> * Can you also add it to `test_types.py` (eg adding to `get_all_types()` will automatically run some tests)
It looks like this has been refactored. Please let me know if I did it correctly.
>
> I can also push myself for those (trivial) changes if you want.
>
I think I've addressed all the feedback here, let me know if I've missed something.
> One other question (but doesn't need to be handled here):
>
> * Should we allow creating an interval array from plain tuples instead of only from the MonthDayNano named tuple?
Possibly, I think we can add this if users request it later. I don't think there is a need to do it here because tuples already get inferred as lists.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on pull request #11302: ARROW-13806: [C++][Python] Add support for new MonthDayNano Interval Type
Posted by GitBox <gi...@apache.org>.
emkornfield commented on pull request #11302:
URL: https://github.com/apache/arrow/pull/11302#issuecomment-937287047
> @emkornfield It can live somewhere with the other Python helpers IMHO.
Consolidated in datetime.h/datetime.cc.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on a change in pull request #11302: ARROW-13806: [C++][Python] Add support for new MonthDayNano Interval Type
Posted by GitBox <gi...@apache.org>.
emkornfield commented on a change in pull request #11302:
URL: https://github.com/apache/arrow/pull/11302#discussion_r723687949
##########
File path: cpp/src/arrow/python/arrow_to_python_internal.h
##########
@@ -0,0 +1,52 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+// Functions for converting between pandas's NumPy-based data representation
+// and Arrow data structures
Review comment:
yeah. removed on next commit.
##########
File path: cpp/src/arrow/python/helpers.cc
##########
@@ -321,6 +323,14 @@ void InitPandasStaticData() {
pandas_NA = ref.obj();
}
+ // Import DateOffset type
+ OwnedRef offsets;
+ if (internal::ImportModule("pandas.tseries.offsets", &offsets).ok()) {
Review comment:
done in next commit.
##########
File path: cpp/src/arrow/python/datetime.cc
##########
@@ -71,6 +74,26 @@ bool MatchFixedOffset(const std::string& tz, util::string_view* sign,
return iter == (tz.data() + tz.size());
}
+static PyTypeObject MonthDayNanoTupleType = {};
+
+constexpr char* NonConst(const char* st) {
+ // Hack for python versions < 3.7 where members of PyStruct members
+ // where non-const (C++ doesn't like assigning string literals to these types)
+ return const_cast<char*>(st);
Review comment:
we do, I asked that we ddidn't drop it last release. I would also ask that we don't drop it this release (i.e. keep it through its full python support cycle. There are a number of consumers of pyarrow that try to keep support for python versions until they dropped it which is end of this year). This was caught because we run it in CI.
##########
File path: python/pyarrow/types.pxi
##########
@@ -2176,6 +2176,14 @@ def duration(unit):
return out
+def month_day_nano_interval():
+ """
+ Create instance of an interval representing months, days and nanoseconds
+ between two dates.
Review comment:
this language is consistent with other methods in the file.
##########
File path: python/pyarrow/types.pxi
##########
@@ -2176,6 +2176,14 @@ def duration(unit):
return out
+def month_day_nano_interval():
+ """
+ Create instance of an interval representing months, days and nanoseconds
+ between two dates.
Review comment:
seems good to me. I liked you other wording also, but we should probably do a general cleanup.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] westonpace commented on a change in pull request #11302: ARROW-13806: [C++][Python] Add support for new MonthDayNano Interval Type
Posted by GitBox <gi...@apache.org>.
westonpace commented on a change in pull request #11302:
URL: https://github.com/apache/arrow/pull/11302#discussion_r723744352
##########
File path: python/pyarrow/types.pxi
##########
@@ -2176,6 +2176,14 @@ def duration(unit):
return out
+def month_day_nano_interval():
+ """
+ Create instance of an interval representing months, days and nanoseconds
+ between two dates.
Review comment:
```suggestion
A type for intervals representing months, days and nanoseconds
between two dates.
```
Nit: Primitive types are cached so no instances are created (usually) and the wording here sounds like it is creating intervals and not types.
##########
File path: python/pyarrow/array.pxi
##########
@@ -163,6 +163,12 @@ def array(object obj, type=None, mask=None, size=None, from_pandas=None,
representation). Timezone-naive data will be implicitly interpreted as
UTC.
+ Pandas's DateOffsets and dateutil.relativedelta.relativedelta are by
+ default converted as MonthDayNanoIntervalArray. relativedelta leapday's
+ are ignored as are all absolute fields on both objects. datetime.timedelta
+ can also be converted to MonthDayNanoIntervalArray but require passing
+ MonthDayNanoIntervalType explicitly.
Review comment:
```suggestion
can also be converted to MonthDayNanoIntervalArray but this requires
passing MonthDayNanoIntervalType explicitly.
```
##########
File path: cpp/src/arrow/python/datetime.cc
##########
@@ -71,6 +74,26 @@ bool MatchFixedOffset(const std::string& tz, util::string_view* sign,
return iter == (tz.data() + tz.size());
}
+static PyTypeObject MonthDayNanoTupleType = {};
+
+constexpr char* NonConst(const char* st) {
+ // Hack for python versions < 3.7 where members of PyStruct members
+ // where non-const (C++ doesn't like assigning string literals to these types)
+ return const_cast<char*>(st);
Review comment:
Do we still support python versions < 3.7? I thought we stopped shipping binary wheels for these versions but maybe we still support building from source.
##########
File path: python/pyarrow/array.pxi
##########
@@ -163,6 +163,12 @@ def array(object obj, type=None, mask=None, size=None, from_pandas=None,
representation). Timezone-naive data will be implicitly interpreted as
UTC.
+ Pandas's DateOffsets and dateutil.relativedelta.relativedelta are by
+ default converted as MonthDayNanoIntervalArray. relativedelta leapday's
Review comment:
```suggestion
default converted as MonthDayNanoIntervalArray. relativedelta leapdays
```
##########
File path: python/pyarrow/types.pxi
##########
@@ -2176,6 +2176,14 @@ def duration(unit):
return out
+def month_day_nano_interval():
+ """
+ Create instance of an interval representing months, days and nanoseconds
+ between two dates.
Review comment:
```suggestion
Create instance of an interval type representing months, days and nanoseconds
between two dates.
```
Fair enough. How about a more subtle suggestion then.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on pull request #11302: ARROW-13806: [C++][Python] Add support for new MonthDayNano Interval Type
Posted by GitBox <gi...@apache.org>.
pitrou commented on pull request #11302:
URL: https://github.com/apache/arrow/pull/11302#issuecomment-936347367
@emkornfield It can live somewhere with the other Python helpers IMHO.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on a change in pull request #11302: ARROW-13806: [C++][Python] Add support for new Interval Type
Posted by GitBox <gi...@apache.org>.
emkornfield commented on a change in pull request #11302:
URL: https://github.com/apache/arrow/pull/11302#discussion_r721901835
##########
File path: cpp/src/arrow/python/datetime.cc
##########
@@ -450,6 +475,19 @@ Result<std::string> TzinfoToString(PyObject* tzinfo) {
return PyTZInfo_utcoffset_hhmm(tzinfo);
}
+Result<PyObject*> MonthDayNanoIntervalToNamedTuple(
Review comment:
There are a few other places that return Result<PyObject*> is there any guidance on when to use that and when to use pure PyObject*?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on a change in pull request #11302: ARROW-13806: [C++][Python] Add support for new Interval Type
Posted by GitBox <gi...@apache.org>.
emkornfield commented on a change in pull request #11302:
URL: https://github.com/apache/arrow/pull/11302#discussion_r721901597
##########
File path: cpp/src/arrow/python/datetime.cc
##########
@@ -450,6 +475,19 @@ Result<std::string> TzinfoToString(PyObject* tzinfo) {
return PyTZInfo_utcoffset_hhmm(tzinfo);
}
+Result<PyObject*> MonthDayNanoIntervalToNamedTuple(
Review comment:
done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on pull request #11302: ARROW-13806: [C++][Python] Add support for new Interval Type
Posted by GitBox <gi...@apache.org>.
emkornfield commented on pull request #11302:
URL: https://github.com/apache/arrow/pull/11302#issuecomment-934089985
@pitrou @westonpace thanks for the reviews I think I've addressed all of the comments.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #11302: ARROW-13806: [C++][Python] Add support for new Interval Type
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #11302:
URL: https://github.com/apache/arrow/pull/11302#issuecomment-933192894
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on a change in pull request #11302: ARROW-13806: [C++][Python] Add support for new Interval Type
Posted by GitBox <gi...@apache.org>.
emkornfield commented on a change in pull request #11302:
URL: https://github.com/apache/arrow/pull/11302#discussion_r722775423
##########
File path: cpp/src/arrow/python/arrow_to_pandas.cc
##########
@@ -1097,6 +1081,40 @@ struct ObjectWriterVisitor {
return Status::OK();
}
+ template <typename Type>
+ enable_if_t<std::is_same<Type, MonthDayNanoIntervalType>::value, Status> Visit(
+ const Type& type) {
+ OwnedRef args(PyTuple_New(0));
+ OwnedRef kwargs(PyDict_New());
+ RETURN_IF_PYERROR();
+ auto to_date_offset = [&](const MonthDayNanoIntervalType::MonthDayNanos& interval,
+ PyObject** out) {
+ DCHECK(internal::BorrowPandasDataOffsetType() != nullptr);
+ // TimeDelta objects do not add nanoseconds component to timestamp.
+ // so convert microseconds and remainder to preserve data
+ // but give users more expected results.
Review comment:
Rephrased (had the wrong time should have been DateOffset) and referenced the bug.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on pull request #11302: ARROW-13806: [C++][Python] Add support for new MonthDayNano Interval Type
Posted by GitBox <gi...@apache.org>.
emkornfield commented on pull request #11302:
URL: https://github.com/apache/arrow/pull/11302#issuecomment-937287047
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on a change in pull request #11302: ARROW-13806: [C++][Python] Add support for new Interval Type
Posted by GitBox <gi...@apache.org>.
emkornfield commented on a change in pull request #11302:
URL: https://github.com/apache/arrow/pull/11302#discussion_r721900859
##########
File path: python/pyarrow/types.pxi
##########
@@ -2154,6 +2154,14 @@ def duration(unit):
return out
+def month_day_nano_interval():
+ """
+ Create instance of a interval representing the time between two calendar
+ instances represented as a triple of months, days and nanoseconds.
+ """
+ return primitive_type(_Type_INTERVAL_MONTH_DAY_NANO)
Review comment:
copied language from type.h
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on a change in pull request #11302: ARROW-13806: [C++][Python] Add support for new Interval Type
Posted by GitBox <gi...@apache.org>.
emkornfield commented on a change in pull request #11302:
URL: https://github.com/apache/arrow/pull/11302#discussion_r721534167
##########
File path: cpp/src/arrow/python/CMakeLists.txt
##########
@@ -28,6 +28,7 @@ add_dependencies(arrow_python-all arrow_python arrow_python-tests)
set(ARROW_PYTHON_SRCS
arrow_to_pandas.cc
+ arrow_to_python.cc
Review comment:
added.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on a change in pull request #11302: ARROW-13806: [C++][Python] Add support for new MonthDayNano Interval Type
Posted by GitBox <gi...@apache.org>.
pitrou commented on a change in pull request #11302:
URL: https://github.com/apache/arrow/pull/11302#discussion_r723139253
##########
File path: cpp/src/arrow/python/arrow_to_python.cc
##########
@@ -0,0 +1,107 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+// Functions for converting between pandas's NumPy-based data representation
+// and Arrow data structures
+
+#include "arrow/python/arrow_to_python.h"
+
+#include "arrow/python/arrow_to_python_internal.h"
+#include "arrow/python/datetime.h"
+#include "arrow/python/helpers.h"
+#include "arrow/result_internal.h"
+#include "arrow/scalar.h"
+
+namespace arrow {
+namespace py {
+namespace {
+
+Status CheckInterval(const DataType& datatype) {
+ if (datatype.id() != Type::INTERVAL_MONTH_DAY_NANO) {
+ return Status::NotImplemented(
+ "Only MonthDayIntervalNanoIntervalType supported. Provided.",
+ datatype.ToString());
+ }
+ return Status::OK();
+}
+
+// Wrapper around a Python list object that mimics dereference and assignment
+// operations.
+struct PyListAssigner {
+ public:
+ explicit PyListAssigner(PyObject* list) : list_(list) { DCHECK(PyList_Check(list_)); }
+
+ PyListAssigner& operator*() { return *this; }
+
+ void operator=(PyObject* obj) {
+ if (ARROW_PREDICT_FALSE(PyList_SetItem(list_, current_index_, obj) == -1)) {
+ Py_FatalError("list did not have the correct preallocated size.");
+ }
+ }
+
+ PyListAssigner& operator++() {
+ current_index_++;
+ return *this;
+ }
+
+ PyListAssigner& operator+=(int64_t offset) {
+ current_index_ += offset;
+ return *this;
+ }
+
+ private:
+ PyObject* list_;
+ int64_t current_index_ = 0;
+};
+
+} // namespace
+
+Result<PyObject*> ArrowToPython::ToPyList(const Array& array) {
+ RETURN_NOT_OK(CheckInterval(*array.type()));
Review comment:
I see. I'd rather see a comprehensive refactor in another PR than a stub like this, which may end up abandoned.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on a change in pull request #11302: ARROW-13806: [C++][Python] Add support for new Interval Type
Posted by GitBox <gi...@apache.org>.
emkornfield commented on a change in pull request #11302:
URL: https://github.com/apache/arrow/pull/11302#discussion_r721895788
##########
File path: python/pyarrow/tests/test_array.py
##########
@@ -2162,6 +2163,101 @@ def test_array_from_numpy_ascii():
assert arrow_arr.equals(expected)
+@pytest.mark.nopandas
+def test_interval_array_from_timedelta():
+ data = [
+ None,
+ datetime.timedelta(days=1, seconds=1, microseconds=1,
+ milliseconds=1, minutes=1, hours=1, weeks=1)]
+
+ # From timedelta (explicit type required)
+ arr = pa.array(data, pa.month_day_nano_interval())
+ assert isinstance(arr, pa.MonthDayNanoIntervalArray)
+ assert arr.type == pa.month_day_nano_interval()
+ expected_list = [
+ None,
+ pa.MonthDayNanoTuple([0, 8,
Review comment:
I might be missing something but it doesn't appear timedelta has a months or years field, which would be what is required to have a non-zero months fields (otherwise days is sufficient).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on a change in pull request #11302: ARROW-13806: [C++][Python] Add support for new Interval Type
Posted by GitBox <gi...@apache.org>.
emkornfield commented on a change in pull request #11302:
URL: https://github.com/apache/arrow/pull/11302#discussion_r721893684
##########
File path: python/pyarrow/lib.pyx
##########
@@ -37,6 +37,10 @@ arrow_init_numpy()
# (used from some of our C++ code, see e.g. ARROW-5260)
import_pyarrow()
+cdef libarrow.ArrowToPython ARROW_TO_PYTHON
+
+MonthDayNanoTuple = NewMonthDayNanoTupleType()
Review comment:
Done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on a change in pull request #11302: ARROW-13806: [C++][Python] Add support for new Interval Type
Posted by GitBox <gi...@apache.org>.
emkornfield commented on a change in pull request #11302:
URL: https://github.com/apache/arrow/pull/11302#discussion_r721889385
##########
File path: cpp/src/arrow/python/datetime.cc
##########
@@ -71,6 +71,19 @@ bool MatchFixedOffset(const std::string& tz, util::string_view* sign,
return iter == (tz.data() + tz.size());
}
+static PyTypeObject MonthDayNanoTupleType = {0, 0};
+
+static PyStructSequence_Field MonthDayNanoField[] = {
+ {"months", "The number of months in the interval"},
+ {"days", "The number days in the interval"},
+ {"nanoseconds", "The number of nanoseconds in the interval"},
+ {nullptr, nullptr}};
+
+static PyStructSequence_Desc MonthDayNanoTupleDesc = {
+ "MonthDayNanoTuple", "A interval consistent of months, days and nanoseconds.",
Review comment:
meant to say consisting
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on a change in pull request #11302: ARROW-13806: [C++][Python] Add support for new Interval Type
Posted by GitBox <gi...@apache.org>.
emkornfield commented on a change in pull request #11302:
URL: https://github.com/apache/arrow/pull/11302#discussion_r721537234
##########
File path: python/pyarrow/scalar.pxi
##########
@@ -514,6 +514,32 @@ cdef class DurationScalar(Scalar):
return datetime.timedelta(microseconds=sp.value // 1000)
+cdef class MonthDayNanoIntervalScalar(Scalar):
+ """
+ Concrete class for month, day, nanosecond scalars.
+ """
+
+ @property
+ def value(self):
+ """
+ Returns this value pyarrow.MonthDayNanoTuple.
+ """
+ cdef PyObject* val
+ val = GetResultValue(ARROW_TO_PYTHON.ToPrimitive(
+ (deref(self.wrapped.get()))))
+ return PyObject_to_object(val)
+
+ def as_py(self):
+ """
+ Return this value as a Pandas DateOffset instance if Pandas is present
+ otherwise as a named tuple containing months days and nanoseconds.
Review comment:
I kind of agree, I modeled this after [Timestamps](https://github.com/apache/arrow/blob/master/python/pyarrow/scalar.pxi#L382) which also changes behavior. Happy to simplify this though if that is desired (along with corresponding to_pylist()) logic.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on a change in pull request #11302: ARROW-13806: [C++][Python] Add support for new Interval Type
Posted by GitBox <gi...@apache.org>.
pitrou commented on a change in pull request #11302:
URL: https://github.com/apache/arrow/pull/11302#discussion_r722137618
##########
File path: cpp/src/arrow/python/arrow_to_python.h
##########
@@ -0,0 +1,59 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+// Functions for converting between pandas's NumPy-based data representation
+// and Arrow data structures
+
+#pragma once
+
+#include "arrow/python/common.h"
+#include "arrow/python/platform.h"
+
+namespace arrow {
+
+class Array;
+struct Scalar;
+
+namespace py {
+
+/// \brief Utility class for converting Arrow to Python obects. A class instead
Review comment:
Trailing unfinished sentence here?
##########
File path: python/pyarrow/tests/test_scalars.py
##########
@@ -364,6 +367,14 @@ def test_duration_nanos_nopandas():
arr[0].as_py()
+def test_month_day_nano_interval():
+ triple = pa.MonthDayNano([3600, 3600, 3600])
Review comment:
Please use different values for components here.
##########
File path: cpp/src/arrow/python/arrow_to_python.cc
##########
@@ -0,0 +1,107 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+// Functions for converting between pandas's NumPy-based data representation
+// and Arrow data structures
+
+#include "arrow/python/arrow_to_python.h"
+
+#include "arrow/python/arrow_to_python_internal.h"
+#include "arrow/python/datetime.h"
+#include "arrow/python/helpers.h"
+#include "arrow/result_internal.h"
+#include "arrow/scalar.h"
+
+namespace arrow {
+namespace py {
+namespace {
+
+Status CheckInterval(const DataType& datatype) {
+ if (datatype.id() != Type::INTERVAL_MONTH_DAY_NANO) {
+ return Status::NotImplemented(
+ "Only MonthDayIntervalNanoIntervalType supported. Provided.",
+ datatype.ToString());
+ }
+ return Status::OK();
+}
+
+// Wrapper around a Python list object that mimics dereference and assignment
+// operations.
+struct PyListAssigner {
+ public:
+ explicit PyListAssigner(PyObject* list) : list_(list) { DCHECK(PyList_Check(list_)); }
+
+ PyListAssigner& operator*() { return *this; }
+
+ void operator=(PyObject* obj) {
+ if (ARROW_PREDICT_FALSE(PyList_SetItem(list_, current_index_, obj) == -1)) {
+ Py_FatalError("list did not have the correct preallocated size.");
+ }
+ }
+
+ PyListAssigner& operator++() {
+ current_index_++;
+ return *this;
+ }
+
+ PyListAssigner& operator+=(int64_t offset) {
+ current_index_ += offset;
+ return *this;
+ }
+
+ private:
+ PyObject* list_;
+ int64_t current_index_ = 0;
+};
+
+} // namespace
+
+Result<PyObject*> ArrowToPython::ToPyList(const Array& array) {
+ RETURN_NOT_OK(CheckInterval(*array.type()));
Review comment:
So these methods only work for month_day_nano_interval? That seems like a weird API choice.
##########
File path: cpp/src/arrow/python/arrow_to_python.h
##########
@@ -0,0 +1,59 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+// Functions for converting between pandas's NumPy-based data representation
+// and Arrow data structures
+
+#pragma once
+
+#include "arrow/python/common.h"
+#include "arrow/python/platform.h"
+
+namespace arrow {
+
+class Array;
+struct Scalar;
+
+namespace py {
+
+/// \brief Utility class for converting Arrow to Python obects. A class instead
+///
+/// A class is chosen because in the future some amount of state will be
+/// (e.g. imported python classes), doing this one lazily will be helpful
+/// and having members present avoids static C++ variables.
+class ARROW_PYTHON_EXPORT ArrowToPython {
+ public:
+ /// \brief Converts the given Array to a PyList object. Returns NULL if there
+ /// is an error converting the Array. The list elements are the same ones
+ /// generated via ToLogical()
+ ///
+ /// N.B. This has limited type support. ARROW-12976 tracks extending the
+ /// implementation.
+ Result<PyObject*> ToPyList(const Array& array);
+
+ /// \brief Converts the given Scalar the type that is closest to its arrow
+ /// representation.
+ ///
+ /// For instance timestamp would be translated to a integer representing an
+ // offset from the unix epoch.
+ ///
+ /// N.B. This has limited type support. ARROW-12976 tracks full implementation.
+ Result<PyObject*> ToPrimitive(const Scalar& scalar);
Review comment:
Why isn't this called "ToPyObject"? Is it only for primitive types?
##########
File path: cpp/src/arrow/python/arrow_to_python.h
##########
@@ -0,0 +1,59 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+// Functions for converting between pandas's NumPy-based data representation
Review comment:
Do these functions have anything to do with Pandas?
##########
File path: python/pyarrow/array.pxi
##########
@@ -163,6 +163,12 @@ def array(object obj, type=None, mask=None, size=None, from_pandas=None,
representation). Timezone-naive data will be implicitly interpreted as
UTC.
+ Pandas's DateOffsets and dateutil.relativedelta.relativedetla are by
Review comment:
"delta"
##########
File path: cpp/src/arrow/python/arrow_to_python.h
##########
@@ -0,0 +1,59 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+// Functions for converting between pandas's NumPy-based data representation
+// and Arrow data structures
+
+#pragma once
+
+#include "arrow/python/common.h"
+#include "arrow/python/platform.h"
+
+namespace arrow {
+
+class Array;
+struct Scalar;
+
+namespace py {
+
+/// \brief Utility class for converting Arrow to Python obects. A class instead
+///
+/// A class is chosen because in the future some amount of state will be
+/// (e.g. imported python classes), doing this one lazily will be helpful
+/// and having members present avoids static C++ variables.
+class ARROW_PYTHON_EXPORT ArrowToPython {
+ public:
+ /// \brief Converts the given Array to a PyList object. Returns NULL if there
+ /// is an error converting the Array. The list elements are the same ones
+ /// generated via ToLogical()
Review comment:
What is ToLogical?
##########
File path: cpp/src/arrow/python/arrow_to_pandas.cc
##########
@@ -1097,6 +1081,40 @@ struct ObjectWriterVisitor {
return Status::OK();
}
+ template <typename Type>
+ enable_if_t<std::is_same<Type, MonthDayNanoIntervalType>::value, Status> Visit(
+ const Type& type) {
+ OwnedRef args(PyTuple_New(0));
+ OwnedRef kwargs(PyDict_New());
+ RETURN_IF_PYERROR();
+ auto to_date_offset = [&](const MonthDayNanoIntervalType::MonthDayNanos& interval,
+ PyObject** out) {
+ DCHECK(internal::BorrowPandasDataOffsetType() != nullptr);
+ // TimeDelta objects do not add nanoseconds component to timestamp.
+ // so convert microseconds and remainder to preserve data
+ // but give users more expected results.
Review comment:
Code looks ok, but I don't understand this comment. What do you mean with "do not add nanoseconds component to timestamp"?
##########
File path: cpp/src/arrow/python/arrow_to_python.h
##########
@@ -0,0 +1,59 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+// Functions for converting between pandas's NumPy-based data representation
+// and Arrow data structures
+
+#pragma once
+
+#include "arrow/python/common.h"
+#include "arrow/python/platform.h"
+
+namespace arrow {
+
+class Array;
+struct Scalar;
+
+namespace py {
+
+/// \brief Utility class for converting Arrow to Python obects. A class instead
+///
+/// A class is chosen because in the future some amount of state will be
+/// (e.g. imported python classes), doing this one lazily will be helpful
+/// and having members present avoids static C++ variables.
+class ARROW_PYTHON_EXPORT ArrowToPython {
+ public:
+ /// \brief Converts the given Array to a PyList object. Returns NULL if there
+ /// is an error converting the Array. The list elements are the same ones
+ /// generated via ToLogical()
+ ///
+ /// N.B. This has limited type support. ARROW-12976 tracks extending the
+ /// implementation.
+ Result<PyObject*> ToPyList(const Array& array);
+
+ /// \brief Converts the given Scalar the type that is closest to its arrow
+ /// representation.
+ ///
+ /// For instance timestamp would be translated to a integer representing an
+ // offset from the unix epoch.
Review comment:
Would it? I thought a timestamp would be converted to a datetime.
##########
File path: python/pyarrow/array.pxi
##########
@@ -163,6 +163,12 @@ def array(object obj, type=None, mask=None, size=None, from_pandas=None,
representation). Timezone-naive data will be implicitly interpreted as
UTC.
+ Pandas's DateOffsets and dateutil.relativedelta.relativedetla are by
+ default converted as MonthDayNanoIntervalArray. relativedelta leapday's
+ are ignored as are all absolute fields on both objects. datetime.timedelta
+ also be converted to MonthDayNanoIntervalArray but require passing
Review comment:
"also be"?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on a change in pull request #11302: ARROW-13806: [C++][Python] Add support for new Interval Type
Posted by GitBox <gi...@apache.org>.
emkornfield commented on a change in pull request #11302:
URL: https://github.com/apache/arrow/pull/11302#discussion_r721901271
##########
File path: python/pyarrow/lib.pyx
##########
@@ -37,6 +37,10 @@ arrow_init_numpy()
# (used from some of our C++ code, see e.g. ARROW-5260)
import_pyarrow()
+cdef libarrow.ArrowToPython ARROW_TO_PYTHON
Review comment:
It no longer needs to do any imports for this PR (pandas objects are left in pandas related classes). It was doing the import lazily on first method calls.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on a change in pull request #11302: ARROW-13806: [C++][Python] Add support for new Interval Type
Posted by GitBox <gi...@apache.org>.
emkornfield commented on a change in pull request #11302:
URL: https://github.com/apache/arrow/pull/11302#discussion_r722426368
##########
File path: cpp/src/arrow/python/arrow_to_python.cc
##########
@@ -0,0 +1,107 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+// Functions for converting between pandas's NumPy-based data representation
+// and Arrow data structures
+
+#include "arrow/python/arrow_to_python.h"
+
+#include "arrow/python/arrow_to_python_internal.h"
+#include "arrow/python/datetime.h"
+#include "arrow/python/helpers.h"
+#include "arrow/result_internal.h"
+#include "arrow/scalar.h"
+
+namespace arrow {
+namespace py {
+namespace {
+
+Status CheckInterval(const DataType& datatype) {
+ if (datatype.id() != Type::INTERVAL_MONTH_DAY_NANO) {
+ return Status::NotImplemented(
+ "Only MonthDayIntervalNanoIntervalType supported. Provided.",
+ datatype.ToString());
+ }
+ return Status::OK();
+}
+
+// Wrapper around a Python list object that mimics dereference and assignment
+// operations.
+struct PyListAssigner {
+ public:
+ explicit PyListAssigner(PyObject* list) : list_(list) { DCHECK(PyList_Check(list_)); }
+
+ PyListAssigner& operator*() { return *this; }
+
+ void operator=(PyObject* obj) {
+ if (ARROW_PREDICT_FALSE(PyList_SetItem(list_, current_index_, obj) == -1)) {
+ Py_FatalError("list did not have the correct preallocated size.");
+ }
+ }
+
+ PyListAssigner& operator++() {
+ current_index_++;
+ return *this;
+ }
+
+ PyListAssigner& operator+=(int64_t offset) {
+ current_index_ += offset;
+ return *this;
+ }
+
+ private:
+ PyObject* list_;
+ int64_t current_index_ = 0;
+};
+
+} // namespace
+
+Result<PyObject*> ArrowToPython::ToPyList(const Array& array) {
+ RETURN_NOT_OK(CheckInterval(*array.type()));
Review comment:
The intent was to generalize them for other types (see TODO JIRAs in the header for supporting more types).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on a change in pull request #11302: ARROW-13806: [C++][Python] Add support for new Interval Type
Posted by GitBox <gi...@apache.org>.
emkornfield commented on a change in pull request #11302:
URL: https://github.com/apache/arrow/pull/11302#discussion_r721543618
##########
File path: python/pyarrow/lib.pyx
##########
@@ -37,6 +37,10 @@ arrow_init_numpy()
# (used from some of our C++ code, see e.g. ARROW-5260)
import_pyarrow()
+cdef libarrow.ArrowToPython ARROW_TO_PYTHON
Review comment:
yes, globally defined converter. This will likely change based on the outcome of simplying "as_py" but before it required getting references to Pandas types. In the long run we likely need to import a few others (timestamp, timedelta, decimal, etc). It seems like for Pandas we used C++ static variables. It felt cleaner to me make it an explicit object. Again don't feel too strongly here and if we eliminate the immediate need to share Pandas conversion logic this could go away. Do you have a preference on the pattern here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou closed pull request #11302: ARROW-13806: [C++][Python] Add support for new MonthDayNano Interval Type
Posted by GitBox <gi...@apache.org>.
pitrou closed pull request #11302:
URL: https://github.com/apache/arrow/pull/11302
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on a change in pull request #11302: ARROW-13806: [C++][Python] Add support for new Interval Type
Posted by GitBox <gi...@apache.org>.
emkornfield commented on a change in pull request #11302:
URL: https://github.com/apache/arrow/pull/11302#discussion_r722429634
##########
File path: cpp/src/arrow/python/arrow_to_python.h
##########
@@ -0,0 +1,59 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+// Functions for converting between pandas's NumPy-based data representation
+// and Arrow data structures
+
+#pragma once
+
+#include "arrow/python/common.h"
+#include "arrow/python/platform.h"
+
+namespace arrow {
+
+class Array;
+struct Scalar;
+
+namespace py {
+
+/// \brief Utility class for converting Arrow to Python obects. A class instead
+///
+/// A class is chosen because in the future some amount of state will be
+/// (e.g. imported python classes), doing this one lazily will be helpful
+/// and having members present avoids static C++ variables.
+class ARROW_PYTHON_EXPORT ArrowToPython {
+ public:
+ /// \brief Converts the given Array to a PyList object. Returns NULL if there
+ /// is an error converting the Array. The list elements are the same ones
+ /// generated via ToLogical()
+ ///
+ /// N.B. This has limited type support. ARROW-12976 tracks extending the
+ /// implementation.
+ Result<PyObject*> ToPyList(const Array& array);
+
+ /// \brief Converts the given Scalar the type that is closest to its arrow
+ /// representation.
+ ///
+ /// For instance timestamp would be translated to a integer representing an
+ // offset from the unix epoch.
+ ///
+ /// N.B. This has limited type support. ARROW-12976 tracks full implementation.
+ Result<PyObject*> ToPrimitive(const Scalar& scalar);
Review comment:
See comment above for explanation, but I think ToPyObject isn't specific enough when it comes to ARROW-12976
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on a change in pull request #11302: ARROW-13806: [C++][Python] Add support for new Interval Type
Posted by GitBox <gi...@apache.org>.
emkornfield commented on a change in pull request #11302:
URL: https://github.com/apache/arrow/pull/11302#discussion_r722777741
##########
File path: python/pyarrow/array.pxi
##########
@@ -163,6 +163,12 @@ def array(object obj, type=None, mask=None, size=None, from_pandas=None,
representation). Timezone-naive data will be implicitly interpreted as
UTC.
+ Pandas's DateOffsets and dateutil.relativedelta.relativedetla are by
+ default converted as MonthDayNanoIntervalArray. relativedelta leapday's
+ are ignored as are all absolute fields on both objects. datetime.timedelta
+ also be converted to MonthDayNanoIntervalArray but require passing
Review comment:
add "can"
##########
File path: python/pyarrow/array.pxi
##########
@@ -163,6 +163,12 @@ def array(object obj, type=None, mask=None, size=None, from_pandas=None,
representation). Timezone-naive data will be implicitly interpreted as
UTC.
+ Pandas's DateOffsets and dateutil.relativedelta.relativedetla are by
Review comment:
fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on a change in pull request #11302: ARROW-13806: [C++][Python] Add support for new MonthDayNano Interval Type
Posted by GitBox <gi...@apache.org>.
pitrou commented on a change in pull request #11302:
URL: https://github.com/apache/arrow/pull/11302#discussion_r723140716
##########
File path: cpp/src/arrow/python/arrow_to_pandas.cc
##########
@@ -1097,6 +1081,40 @@ struct ObjectWriterVisitor {
return Status::OK();
}
+ template <typename Type>
+ enable_if_t<std::is_same<Type, MonthDayNanoIntervalType>::value, Status> Visit(
+ const Type& type) {
+ OwnedRef args(PyTuple_New(0));
+ OwnedRef kwargs(PyDict_New());
+ RETURN_IF_PYERROR();
+ auto to_date_offset = [&](const MonthDayNanoIntervalType::MonthDayNanos& interval,
+ PyObject** out) {
+ DCHECK(internal::BorrowPandasDataOffsetType() != nullptr);
+ // TimeDelta objects do not add nanoseconds component to timestamp.
+ // so convert microseconds and remainder to preserve data
+ // but give users more expected results.
Review comment:
Ah, thank you.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on a change in pull request #11302: ARROW-13806: [C++][Python] Add support for new MonthDayNano Interval Type
Posted by GitBox <gi...@apache.org>.
emkornfield commented on a change in pull request #11302:
URL: https://github.com/apache/arrow/pull/11302#discussion_r723687949
##########
File path: cpp/src/arrow/python/arrow_to_python_internal.h
##########
@@ -0,0 +1,52 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+// Functions for converting between pandas's NumPy-based data representation
+// and Arrow data structures
Review comment:
yeah. removed on next commit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org