You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2021/08/24 00:40:53 UTC
[GitHub] [arrow] kainoa21 opened a new pull request #10981: ARROW-13696: [Python] Support for MapType with Fields
kainoa21 opened a new pull request #10981:
URL: https://github.com/apache/arrow/pull/10981
Adds support for passing Field types for the key and item when creating MapType data types.
This also enforces that the key field is non-nullable per the spec (https://github.com/apache/arrow/blob/master/format/Schema.fbs#L118)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jorisvandenbossche commented on pull request #10981: ARROW-13696: [Python] Support for MapType with Fields
Posted by GitBox <gi...@apache.org>.
jorisvandenbossche commented on pull request #10981:
URL: https://github.com/apache/arrow/pull/10981#issuecomment-906443055
Thanks @kainoa21
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jorisvandenbossche commented on a change in pull request #10981: ARROW-13696: [Python] Support for MapType with Fields
Posted by GitBox <gi...@apache.org>.
jorisvandenbossche commented on a change in pull request #10981:
URL: https://github.com/apache/arrow/pull/10981#discussion_r694842266
##########
File path: python/pyarrow/types.pxi
##########
@@ -2349,12 +2363,23 @@ cpdef MapType map_(key_type, item_type, keys_sorted=False):
map_type : DataType
"""
cdef:
- DataType _key_type = ensure_type(key_type, allow_none=False)
- DataType _item_type = ensure_type(item_type, allow_none=False)
+ Field _key_field
+ Field _item_field
shared_ptr[CDataType] map_type
MapType out = MapType.__new__(MapType)
- map_type.reset(new CMapType(_key_type.sp_type, _item_type.sp_type,
+ if isinstance(key_type, Field):
+ _key_field = key_type
Review comment:
I was quickly trying this out, and it seems we crash if the field is not exactly named "key" ànd nullable=False (-> https://issues.apache.org/jira/browse/ARROW-13735). The field name is something we shouldn't be strict about (since the specification says this can be named differently), but so for nullability I think we should check this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jorisvandenbossche commented on a change in pull request #10981: ARROW-13696: [Python] Support for MapType with Fields
Posted by GitBox <gi...@apache.org>.
jorisvandenbossche commented on a change in pull request #10981:
URL: https://github.com/apache/arrow/pull/10981#discussion_r694699866
##########
File path: python/pyarrow/types.pxi
##########
@@ -2349,12 +2363,23 @@ cpdef MapType map_(key_type, item_type, keys_sorted=False):
map_type : DataType
"""
cdef:
- DataType _key_type = ensure_type(key_type, allow_none=False)
- DataType _item_type = ensure_type(item_type, allow_none=False)
+ Field _key_field
+ Field _item_field
shared_ptr[CDataType] map_type
MapType out = MapType.__new__(MapType)
- map_type.reset(new CMapType(_key_type.sp_type, _item_type.sp_type,
+ if isinstance(key_type, Field):
+ _key_field = key_type
Review comment:
Should we check here that the nullability is correct?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kainoa21 commented on a change in pull request #10981: ARROW-13696: [Python] Support for MapType with Fields
Posted by GitBox <gi...@apache.org>.
kainoa21 commented on a change in pull request #10981:
URL: https://github.com/apache/arrow/pull/10981#discussion_r694942284
##########
File path: python/pyarrow/types.pxi
##########
@@ -2349,12 +2363,23 @@ cpdef MapType map_(key_type, item_type, keys_sorted=False):
map_type : DataType
"""
cdef:
- DataType _key_type = ensure_type(key_type, allow_none=False)
- DataType _item_type = ensure_type(item_type, allow_none=False)
+ Field _key_field
+ Field _item_field
shared_ptr[CDataType] map_type
MapType out = MapType.__new__(MapType)
- map_type.reset(new CMapType(_key_type.sp_type, _item_type.sp_type,
+ if isinstance(key_type, Field):
+ _key_field = key_type
Review comment:
I wasn't able to reproduce the segfault from this [issue](https://issues.apache.org/jira/browse/ARROW-13735).
```
>>> t = pa.map_(pa.field("name", "string", nullable=False), "int64")
>>> pa.array([[('a', 1), ('b', 2)], [('c', 3)]], type=t)
<pyarrow.lib.MapArray object at 0x12d150220>
[
keys:
[
"a",
"b"
]
values:
[
1,
2
],
keys:
[
"c"
]
values:
[
3
]
]
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kainoa21 commented on a change in pull request #10981: ARROW-13696: [Python] Support for MapType with Fields
Posted by GitBox <gi...@apache.org>.
kainoa21 commented on a change in pull request #10981:
URL: https://github.com/apache/arrow/pull/10981#discussion_r694940645
##########
File path: python/pyarrow/types.pxi
##########
@@ -2349,12 +2363,23 @@ cpdef MapType map_(key_type, item_type, keys_sorted=False):
map_type : DataType
"""
cdef:
- DataType _key_type = ensure_type(key_type, allow_none=False)
- DataType _item_type = ensure_type(item_type, allow_none=False)
+ Field _key_field
+ Field _item_field
shared_ptr[CDataType] map_type
MapType out = MapType.__new__(MapType)
- map_type.reset(new CMapType(_key_type.sp_type, _item_type.sp_type,
+ if isinstance(key_type, Field):
+ _key_field = key_type
Review comment:
Added a commit to ensure the nullability for a passed field is correct.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #10981: ARROW-13696: [Python] Support for MapType with Fields
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #10981:
URL: https://github.com/apache/arrow/pull/10981#issuecomment-904230918
https://issues.apache.org/jira/browse/ARROW-13696
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jorisvandenbossche closed pull request #10981: ARROW-13696: [Python] Support for MapType with Fields
Posted by GitBox <gi...@apache.org>.
jorisvandenbossche closed pull request #10981:
URL: https://github.com/apache/arrow/pull/10981
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org