You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Apache Spark (Jira)" <ji...@apache.org> on 2021/12/07 03:47:00 UTC
[jira] [Commented] (SPARK-37553) pandas.DataFrame.pivot_table KeyError on underscore in name
[ https://issues.apache.org/jira/browse/SPARK-37553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17454356#comment-17454356 ]
Apache Spark commented on SPARK-37553:
--------------------------------------
User 'oeuf' has created a pull request for this issue:
https://github.com/apache/spark/pull/34812
> pandas.DataFrame.pivot_table KeyError on underscore in name
> -----------------------------------------------------------
>
> Key: SPARK-37553
> URL: https://issues.apache.org/jira/browse/SPARK-37553
> Project: Spark
> Issue Type: Bug
> Components: PySpark
> Affects Versions: 3.2.0
> Reporter: Nicholas Gustafson
> Priority: Minor
>
> The method [pyspark.pandas.frame.DataFrame.pivot_table|https://github.com/apache/spark/blob/master/python/pyspark/pandas/frame.py#L5841-L6134] will {{KeyError}} when the {{values}} kwarg has length greater than 1. There are two cases where this occurs.
> First, when the columns for {{values}} contains an underscore({{{}_{}}}) and it splits the name [here|https://github.com/apache/spark/blob/master/python/pyspark/pandas/frame.py#L6064-L6067].
> Minimal reproduction of the bug in {{values}} column names (using IPython):
> {code:python}
> >>> import numpy as np
> >>> import pandas as pd
> >>> from pyspark import pandas as ps
> >>> pdf = pd.DataFrame(
> {
> "a": [4, 2, 3, 4, 8, 6],
> "b_b": [1, 2, 2, 4, 2, 4],
> "e": [10, 20, 20, 40, 20, 40],
> "c": [1, 2, 9, 4, 7, 4],
> "d": [-1, -2, -3, -4, -5, -6],
> },
> index=np.random.rand(6),
> )
> >>> psdf = ps.from_pandas(pdf)
> >>> psdf.pivot_table(index=["c"], columns="a", values=["b_b", "e"])
> ---------------------------------------------------------------------------
> KeyError Traceback (most recent call last)
> <ipython-input-8-32d5bb0e1166> in <module>
> ----> 1 psdf.pivot_table(index=["c"], columns="a", values=["b_b", "e"])
> ~/.pyenv/versions/3.7.9/envs/venv37/lib/python3.7/site-packages/pyspark/pandas/frame.py in pivot_table(self, values, index, columns, aggfunc, fill_value)
> 6053 column_labels = [
> 6054 tuple(list(column_name_to_index[name.split("_")[1]]) + [name.split("_")[0]])
> -> 6055 for name in data_columns
> 6056 ]
> 6057 column_label_names = (
> ~/.pyenv/versions/3.7.9/envs/venv37/lib/python3.7/site-packages/pyspark/pandas/frame.py in <listcomp>(.0)
> 6053 column_labels = [
> 6054 tuple(list(column_name_to_index[name.split("_")[1]]) + [name.split("_")[0]])
> -> 6055 for name in data_columns
> 6056 ]
> 6057 column_label_names = (
> KeyError: 'b'
> {code}
> Second, when elements of the {{columns}} are strings containing underscores {{_}} we get a similar {{KeyError}} due to [this line|https://github.com/apache/spark/blob/master/python/pyspark/pandas/frame.py#L6058].
> Here is a basic reproduction of the bug (in IPython):
> {code:python}
> >>> import numpy as np
> >>> import pandas as pd
> >>> from pyspark import pandas as ps
> >>> pdf = pd.DataFrame(
> {
> "a": ["4_4", "2_2", "3_3", "4_4", "8_8", "6_6"],
> "b": [1, 2, 2, 4, 2, 4],
> "e": [10, 20, 20, 40, 20, 40],
> "c": [1, 2, 9, 4, 7, 4],
> "d": [-1, -2, -3, -4, -5, -6],
> },
> index=np.random.rand(6),
> )
> >>> psdf = ps.from_pandas(pdf)
> >>> psdf.pivot_table(index=["c"], columns="a", values=["b", "e"])
> KeyError Traceback (most recent call last)
> <ipython-input-10-787c0fabea99> in <module>
> 1 psdf = ps.from_pandas(pdf)
> ----> 2 psdf.pivot_table(index=["c"], columns="a", values=["b", "e"])
> ~/.pyenv/versions/3.7.9/envs/venv37/lib/python3.7/site-packages/pyspark/pandas/frame.py in pivot_table(self, values, index, columns, aggfunc, fill_value)
> 6053 column_labels = [
> 6054 tuple(list(column_name_to_index[name.split("_")[1]]) + [name.split("_")[0]])
> -> 6055 for name in data_columns
> 6056 ]
> 6057 column_label_names = (
> ~/.pyenv/versions/3.7.9/envs/venv37/lib/python3.7/site-packages/pyspark/pandas/frame.py in <listcomp>(.0)
> 6053 column_labels = [
> 6054 tuple(list(column_name_to_index[name.split("_")[1]]) + [name.split("_")[0]])
> -> 6055 for name in data_columns
> 6056 ]
> 6057 column_label_names = (
> KeyError: '2'
> {code}
> Based on instructions in the [PySpark Contributing Guide|https://spark.apache.org/docs/latest/api/python/development/contributing.html], I forked the repo, created a new branch, and added an (example) fix + new unit tests for the code changes. Since I have never contributed to this project before, I have submitted a Draft Pull Request (instead of a normal pull request).
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org