You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2022/09/08 21:35:23 UTC
[GitHub] [arrow] jrbourbeau opened a new pull request, #14080: ARROW-16838: [Python] Improve schema inference for pandas indexes with extension dtypes
jrbourbeau opened a new pull request, #14080:
URL: https://github.com/apache/arrow/pull/14080
Possible fix for https://issues.apache.org/jira/browse/ARROW-16838. `pd.Index` objects don't have a `.head` method, while `pd.DataFrame`, `pd.Series`, and `pd.Index` all support indexing with `[:0]` to return a empty object of the same type.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jorisvandenbossche commented on pull request #14080: ARROW-16838: [Python] Improve schema inference for pandas indexes with extension dtypes
Posted by GitBox <gi...@apache.org>.
jorisvandenbossche commented on PR #14080:
URL: https://github.com/apache/arrow/pull/14080#issuecomment-1253336104
No, looks all good, thanks for the update! (I was just away for a few days)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm commented on pull request #14080: ARROW-16838: [Python] Improve schema inference for pandas indexes with extension dtypes
Posted by GitBox <gi...@apache.org>.
lidavidm commented on PR #14080:
URL: https://github.com/apache/arrow/pull/14080#issuecomment-1242152090
Travis: that does tend to be a little flaky.
JIRA: you need to be signed in to JIRA (it's basically asking you to make sure the ticket is assigned to yourself)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] ursabot commented on pull request #14080: ARROW-16838: [Python] Improve schema inference for pandas indexes with extension dtypes
Posted by GitBox <gi...@apache.org>.
ursabot commented on PR #14080:
URL: https://github.com/apache/arrow/pull/14080#issuecomment-1253513701
['Python', 'R'] benchmarks have high level of regressions.
[ursa-i9-9960x](https://conbench.ursa.dev/compare/runs/3e5ca46982444a10b702f63939d08ed9...e9ed3f6dc3c24c9a8c95cfc38c444973/)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jrbourbeau commented on pull request #14080: ARROW-16838: [Python] Improve schema inference for pandas indexes with extension dtypes
Posted by GitBox <gi...@apache.org>.
jrbourbeau commented on PR #14080:
URL: https://github.com/apache/arrow/pull/14080#issuecomment-1253924516
Thanks @jorisvandenbossche!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jrbourbeau commented on pull request #14080: ARROW-16838: [Python] Improve schema inference for pandas indexes with extension dtypes
Posted by GitBox <gi...@apache.org>.
jrbourbeau commented on PR #14080:
URL: https://github.com/apache/arrow/pull/14080#issuecomment-1241303154
> ⚠️ Ticket has not been started in JIRA, please click 'Start Progress'.
I'm not quite sure what to do here. I don't see a "Start Progress" link here or over in the corresponding JIRA issue. Can somebody clarify?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] ursabot commented on pull request #14080: ARROW-16838: [Python] Improve schema inference for pandas indexes with extension dtypes
Posted by GitBox <gi...@apache.org>.
ursabot commented on PR #14080:
URL: https://github.com/apache/arrow/pull/14080#issuecomment-1253513444
Benchmark runs are scheduled for baseline = 8ad5e59803dda03ff5b829ae1635bbe301a1a4e4 and contender = afd3c40a42aa330de61860257a8a09b3c2fe93c8. afd3c40a42aa330de61860257a8a09b3c2fe93c8 is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Finished :arrow_down:0.0% :arrow_up:0.0%] [ec2-t3-xlarge-us-east-2](https://conbench.ursa.dev/compare/runs/d704aedde88841848d9dbe3a2e60f5f3...1952b9d27a57426198a0cf1e28a9cf3c/)
[Failed :arrow_down:0.24% :arrow_up:0.0%] [test-mac-arm](https://conbench.ursa.dev/compare/runs/3b7b19f20ccb4369928988d80dab1997...9070088ba5a845f08e841facae0c2281/)
[Failed :arrow_down:0.56% :arrow_up:0.0%] [ursa-i9-9960x](https://conbench.ursa.dev/compare/runs/3e5ca46982444a10b702f63939d08ed9...e9ed3f6dc3c24c9a8c95cfc38c444973/)
[Finished :arrow_down:0.14% :arrow_up:0.04%] [ursa-thinkcentre-m75q](https://conbench.ursa.dev/compare/runs/9860d66ea5bf456995bd082b64eb355f...d018dd8e3fde4fd289400fb6c815faf0/)
Buildkite builds:
[Finished] [`afd3c40a` ec2-t3-xlarge-us-east-2](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ec2-t3-xlarge-us-east-2/builds/1530)
[Failed] [`afd3c40a` test-mac-arm](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-test-mac-arm/builds/1547)
[Failed] [`afd3c40a` ursa-i9-9960x](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ursa-i9-9960x/builds/1530)
[Finished] [`afd3c40a` ursa-thinkcentre-m75q](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ursa-thinkcentre-m75q/builds/1544)
[Finished] [`8ad5e598` ec2-t3-xlarge-us-east-2](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ec2-t3-xlarge-us-east-2/builds/1529)
[Failed] [`8ad5e598` test-mac-arm](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-test-mac-arm/builds/1546)
[Failed] [`8ad5e598` ursa-i9-9960x](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ursa-i9-9960x/builds/1529)
[Finished] [`8ad5e598` ursa-thinkcentre-m75q](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ursa-thinkcentre-m75q/builds/1543)
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python, R. Runs only benchmarks with cloud = True
test-mac-arm: Supported benchmark langs: C++, Python, R
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jrbourbeau commented on a diff in pull request #14080: ARROW-16838: [Python] Improve schema inference for pandas indexes with extension dtypes
Posted by GitBox <gi...@apache.org>.
jrbourbeau commented on code in PR #14080:
URL: https://github.com/apache/arrow/pull/14080#discussion_r970916713
##########
python/pyarrow/pandas_compat.py:
##########
@@ -541,7 +541,7 @@ def dataframe_to_types(df, preserve_index, columns=None):
if _pandas_api.is_categorical(values):
type_ = pa.array(c, from_pandas=True).type
elif _pandas_api.is_extension_array_dtype(values):
- type_ = pa.array(c.head(0), from_pandas=True).type
+ type_ = pa.array(c[:0], from_pandas=True).type
Review Comment:
Ah, good catch. Just updated to avoid that deprecation with the `if isinstance(c, Series)` check
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jrbourbeau commented on pull request #14080: ARROW-16838: [Python] Improve schema inference for pandas indexes with extension dtypes
Posted by GitBox <gi...@apache.org>.
jrbourbeau commented on PR #14080:
URL: https://github.com/apache/arrow/pull/14080#issuecomment-1247447396
The Travis CI failures is, I think, unrelated to the changes in this PR. @jorisvandenbossche let me know if there are any other changes you'd like to see
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jorisvandenbossche merged pull request #14080: ARROW-16838: [Python] Improve schema inference for pandas indexes with extension dtypes
Posted by GitBox <gi...@apache.org>.
jorisvandenbossche merged PR #14080:
URL: https://github.com/apache/arrow/pull/14080
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jrbourbeau commented on pull request #14080: ARROW-16838: [Python] Improve schema inference for pandas indexes with extension dtypes
Posted by GitBox <gi...@apache.org>.
jrbourbeau commented on PR #14080:
URL: https://github.com/apache/arrow/pull/14080#issuecomment-1253065162
Just wanted to check in here -- are there any additional comment, question, or concerns on this PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jrbourbeau commented on pull request #14080: ARROW-16838: [Python] Improve schema inference for pandas indexes with extension dtypes
Posted by GitBox <gi...@apache.org>.
jrbourbeau commented on PR #14080:
URL: https://github.com/apache/arrow/pull/14080#issuecomment-1242140257
Hrm it looks like the failing Travis CI build happened when building CXX objects -- so presumably unrelated to the changes in this PR. Not sure if this is a known issue or not
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jrbourbeau commented on pull request #14080: ARROW-16838: [Python] Improve schema inference for pandas indexes with extension dtypes
Posted by GitBox <gi...@apache.org>.
jrbourbeau commented on PR #14080:
URL: https://github.com/apache/arrow/pull/14080#issuecomment-1247019401
Hrm, the linting errors in CI appear to be unrelated to the changes in this PR. Maybe https://github.com/apache/arrow/pull/14113 fixes them? Merging `main` to see if that helps
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #14080: ARROW-16838: [Python] Improve schema inference for pandas indexes with extension dtypes
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #14080:
URL: https://github.com/apache/arrow/pull/14080#issuecomment-1241258128
https://issues.apache.org/jira/browse/ARROW-16838
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jorisvandenbossche commented on a diff in pull request #14080: ARROW-16838: [Python] Improve schema inference for pandas indexes with extension dtypes
Posted by GitBox <gi...@apache.org>.
jorisvandenbossche commented on code in PR #14080:
URL: https://github.com/apache/arrow/pull/14080#discussion_r970737215
##########
python/pyarrow/pandas_compat.py:
##########
@@ -541,7 +541,7 @@ def dataframe_to_types(df, preserve_index, columns=None):
if _pandas_api.is_categorical(values):
type_ = pa.array(c, from_pandas=True).type
elif _pandas_api.is_extension_array_dtype(values):
- type_ = pa.array(c.head(0), from_pandas=True).type
+ type_ = pa.array(c[:0], from_pandas=True).type
Review Comment:
> I am thinking that we could maybe also use values[:0] instead?
But, that then will loose some information in case of tz-aware datetimes I think.
Maybe a "dumb" `c.head(0) if isinstance(c, Series) else c[:0]` might be the easiest after all.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm commented on pull request #14080: ARROW-16838: [Python] Improve schema inference for pandas indexes with extension dtypes
Posted by GitBox <gi...@apache.org>.
lidavidm commented on PR #14080:
URL: https://github.com/apache/arrow/pull/14080#issuecomment-1246655730
@jorisvandenbossche are you perhaps able to take a look here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jorisvandenbossche commented on a diff in pull request #14080: ARROW-16838: [Python] Improve schema inference for pandas indexes with extension dtypes
Posted by GitBox <gi...@apache.org>.
jorisvandenbossche commented on code in PR #14080:
URL: https://github.com/apache/arrow/pull/14080#discussion_r970732387
##########
python/pyarrow/pandas_compat.py:
##########
@@ -541,7 +541,7 @@ def dataframe_to_types(df, preserve_index, columns=None):
if _pandas_api.is_categorical(values):
type_ = pa.array(c, from_pandas=True).type
elif _pandas_api.is_extension_array_dtype(values):
- type_ = pa.array(c.head(0), from_pandas=True).type
+ type_ = pa.array(c[:0], from_pandas=True).type
Review Comment:
So for the case that `c` is a Series and not Index, this [:0] currently is always positional (so that's fine). But to be sure I was just checking on pandas main, and this now gives a warning that this will change in the future:
```
In [9]: s = pd.Series([1, 2], index=[1, 2])
In [10]: s[:0]
<ipython-input-10-48e1b387a0b4>:1: FutureWarning: The behavior of `series[i:j]` with an integer-dtype index is deprecated. In a future version, this will be treated as *label-based* indexing, consistent with e.g. `series[i]` lookups. To retain the old behavior, use `series.iloc[i:j]`. To get the future behavior, use `series.loc[i:j]`.
s[:0]
Out[10]: Series([], dtype: int64)
```
So we should avoid running into this warning.
I am thinking that we could maybe also use `values[:0]` instead?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] jrbourbeau commented on pull request #14080: ARROW-16838: [Python] Improve schema inference for pandas indexes with extension dtypes
Posted by GitBox <gi...@apache.org>.
jrbourbeau commented on PR #14080:
URL: https://github.com/apache/arrow/pull/14080#issuecomment-1242184118
Gotcha -- thanks @lidavidm
I've gone ahead and assigned the JIRA issue to me 👍
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #14080: ARROW-16838: [Python] Improve schema inference for pandas indexes with extension dtypes
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #14080:
URL: https://github.com/apache/arrow/pull/14080#issuecomment-1241258153
:warning: Ticket **has not been started in JIRA**, please click 'Start Progress'.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org