You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "caneff (via GitHub)" <gi...@apache.org> on 2023/09/20 15:58:17 UTC
[GitHub] [beam] caneff opened a new issue, #28559: [Bug]: .xs() method on DeferredSeries and DeferredDataFrame doesn't work right when returning single items
caneff opened a new issue, #28559:
URL: https://github.com/apache/beam/issues/28559
### What happened?
I've been upgrading dataframe to be compliant with Pandas 2, and a new doctest in Pandas 2 revealed this bug that currently exists in Pandas 1.
Whenever xs will yield a result that is a single item, it will either fail or have the wrong outcome compared to pandas xs. For examples, I have a draft PR with failing tests [here](https://github.com/caneff/beam/pull/1)
There are two different types of failures. There are the cases where we drop dimensionality (like returning a scalar for ser.xs() or a series for df.xs()) when pick exactly one element for a single Index series/dataframe. Then we get concatenation failures when trying to combine everything back together, because we are expecting to concat series/frame elements (for ser.xs/frame.xs respectively) but instead recieve a scalar/series because pandas has done the dimension reduction.
The other type of failure is with a MultiIndex, where we call df.xs() with a tuple the same length as the number of dimensions of the MultiIndex, and only a single item is returned. We still get a df but it looks wrong and has the wrong shape.
Both types of failure examples are in my link above.
### Issue Priority
Priority: 2 (default / most bugs should be filed as P2)
### Issue Components
- [X] Component: Python SDK
- [ ] Component: Java SDK
- [ ] Component: Go SDK
- [ ] Component: Typescript SDK
- [ ] Component: IO connector
- [ ] Component: Beam examples
- [ ] Component: Beam playground
- [ ] Component: Beam katas
- [ ] Component: Website
- [ ] Component: Spark Runner
- [ ] Component: Flink Runner
- [ ] Component: Samza Runner
- [ ] Component: Twister2 Runner
- [ ] Component: Hazelcast Jet Runner
- [ ] Component: Google Cloud Dataflow Runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: Some methods on DeferredSeries and DeferredDataFrame don't work right when returning single items [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #28559:
URL: https://github.com/apache/beam/issues/28559#issuecomment-5104513272
This issue has been marked as stale due to 150 days of inactivity. It will be closed in 30 days if no further activity occurs. If you think that’s incorrect or this issue still needs to be addressed, please simply write any comment. If closed, you can reopen the issue at any time. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: Some methods on DeferredSeries and DeferredDataFrame don't work right when returning single items [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #28559: [Bug]: Some methods on DeferredSeries and DeferredDataFrame don't work right when returning single items
URL: https://github.com/apache/beam/issues/28559
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: Some methods on DeferredSeries and DeferredDataFrame don't work right when returning single items [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #28559:
URL: https://github.com/apache/beam/issues/28559#issuecomment-5441975910
This issue has been closed due to lack of activity. If you think that is incorrect, you can reopen the issue at any time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: .xs() method on DeferredSeries and DeferredDataFrame doesn't work right when returning single items [beam]
Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on issue #28559:
URL: https://github.com/apache/beam/issues/28559#issuecomment-2087785335
Seeing a similar error on a different operation:
```
from apache_beam.dataframe.doctests import teststring
test = """
With a DataFrame
>>> df = pd.DataFrame({'a': [1, 2, 2, 3], 'b': [3, 4, 4, 4]},
... index=['cat', 'dog', 'dog', 'mouse'])
>>> df
a b
cat 1 3
dog 2 4
dog 2 4
mouse 3 4
>>> df.kurt()
a 1.5
b 4.0
dtype: float64
Passes on Beam with axis=1
>>> df = pd.DataFrame({'a': [1, 2], 'b': [3, 4], 'c': [3, 4], 'd': [1, 2]},
... index=['cat', 'dog'])
>>> df.kurt(axis=1)
cat -6.0
dog -6.0
dtype: float64
Fails on Beam with axis=None
>>> df.kurt(axis=None).round(6)
-0.988693
"""
teststring(test)
```
```
AttributeError: '_DeferredScalar' object has no attribute 'round'
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org