You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Siying Dong (Jira)" <ji...@apache.org> on 2023/07/18 00:32:00 UTC
[jira] [Updated] (SPARK-44464) Fix applyInPandasWithStatePythonRunner to output rows that have Null as first column value
[ https://issues.apache.org/jira/browse/SPARK-44464?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Siying Dong updated SPARK-44464:
--------------------------------
Description: The current implementation of {{ApplyInPandasWithStatePythonRunner}} cannot deal with outputs where the first column of the row is {{{}null{}}}, as it cannot distinguish the case where the column is null, or the field is filled as the number of data records are smaller than state records. It causes incorrect results for the former case. (was: The current implementation of {{ApplyInPandasWithStatePythonRunner}} cannot deal with outputs where the first column of the row is {{{}null{}}}, as it cannot distinguish the case where the column is null, or the field is filled as the number of data records are smaller than state records.)
> Fix applyInPandasWithStatePythonRunner to output rows that have Null as first column value
> ------------------------------------------------------------------------------------------
>
> Key: SPARK-44464
> URL: https://issues.apache.org/jira/browse/SPARK-44464
> Project: Spark
> Issue Type: Bug
> Components: Structured Streaming
> Affects Versions: 3.3.3
> Reporter: Siying Dong
> Priority: Major
>
> The current implementation of {{ApplyInPandasWithStatePythonRunner}} cannot deal with outputs where the first column of the row is {{{}null{}}}, as it cannot distinguish the case where the column is null, or the field is filled as the number of data records are smaller than state records. It causes incorrect results for the former case.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org