You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Yuming Wang (Jira)" <ji...@apache.org> on 2023/04/16 12:34:00 UTC
[jira] [Commented] (SPARK-43154) Pyspark 3.4 fails when running "pivot" function on a dataframe using the values arguement
[ https://issues.apache.org/jira/browse/SPARK-43154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17712774#comment-17712774 ]
Yuming Wang commented on SPARK-43154:
-------------------------------------
cc [~gurwls223]
> Pyspark 3.4 fails when running "pivot" function on a dataframe using the values arguement
> -----------------------------------------------------------------------------------------
>
> Key: SPARK-43154
> URL: https://issues.apache.org/jira/browse/SPARK-43154
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 3.4.0
> Reporter: Ofri Kleinfeld
> Priority: Major
>
> After upgrading to Pyspark version 3.4.0 from version 3.3.2, workflows using the "pivot" function fails with the following error:
>
> `File "/usr/local/lib/python3.8/site-packages/pyspark/sql/group.py", line 512, in pivot
> [847|https://gitlab.booking.com/core/content-intelligence/content-ml/-/jobs/140674703#L847] jgd = self._jgd.pivot(pivot_col, values)
> [848|https://gitlab.booking.com/core/content-intelligence/content-ml/-/jobs/140674703#L848] File "/usr/local/lib/python3.8/site-packages/py4j/java_gateway.py", line 1322, in __call__
> [849|https://gitlab.booking.com/core/content-intelligence/content-ml/-/jobs/140674703#L849] return_value = get_return_value(
> [850|https://gitlab.booking.com/core/content-intelligence/content-ml/-/jobs/140674703#L850] File "/usr/local/lib/python3.8/site-packages/pyspark/errors/exceptions/captured.py", line 169, in deco
> [851|https://gitlab.booking.com/core/content-intelligence/content-ml/-/jobs/140674703#L851] return f(*a, **kw)
> [852|https://gitlab.booking.com/core/content-intelligence/content-ml/-/jobs/140674703#L852] File "/usr/local/lib/python3.8/site-packages/py4j/protocol.py", line 330, in get_return_value
> [853|https://gitlab.booking.com/core/content-intelligence/content-ml/-/jobs/140674703#L853] raise Py4JError(
> [854|https://gitlab.booking.com/core/content-intelligence/content-ml/-/jobs/140674703#L854]py4j.protocol.Py4JError: An error occurred while calling o999.pivot. Trace:
> [855|https://gitlab.booking.com/core/content-intelligence/content-ml/-/jobs/140674703#L855]py4j.Py4JException: Method pivot([class java.lang.String, class [Ljava.lang.String;]) does not exist
> [856|https://gitlab.booking.com/core/content-intelligence/content-ml/-/jobs/140674703#L856] at py4j.reflection.ReflectionEngine.getMethod(ReflectionEngine.java:321)
> [857|https://gitlab.booking.com/core/content-intelligence/content-ml/-/jobs/140674703#L857] at py4j.reflection.ReflectionEngine.getMethod(ReflectionEngine.java:329)
> [858|https://gitlab.booking.com/core/content-intelligence/content-ml/-/jobs/140674703#L858] at py4j.Gateway.invoke(Gateway.java:274)
> [859|https://gitlab.booking.com/core/content-intelligence/content-ml/-/jobs/140674703#L859] at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
> [860|https://gitlab.booking.com/core/content-intelligence/content-ml/-/jobs/140674703#L860] at py4j.commands.CallCommand.execute(CallCommand.java:79)
> [861|https://gitlab.booking.com/core/content-intelligence/content-ml/-/jobs/140674703#L861] at py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:182)
> [862|https://gitlab.booking.com/core/content-intelligence/content-ml/-/jobs/140674703#L862] at py4j.ClientServerConnection.run(ClientServerConnection.java:106)
> [863|https://gitlab.booking.com/core/content-intelligence/content-ml/-/jobs/140674703#L863] at java.base/java.lang.Thread.run(Thread.java:829)`
>
> My workflow calls the "pivot" method using the "values" argument.
> I didn't have the chance to try calling pivot without it to see if the error occurs also when omitting the "values" argument.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org