You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Jungtaek Lim (Jira)" <ji...@apache.org> on 2024/12/02 00:54:00 UTC

[jira] [Resolved] (SPARK-50451) mapPartitions fails when called in streaming mode from Python

     [ https://issues.apache.org/jira/browse/SPARK-50451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jungtaek Lim resolved SPARK-50451.
----------------------------------
    Resolution: Not A Problem

structured streaming is designed to restrict the source and sink, which is not satisfied if the query is going under RDD API. It's probably a bad error message for users, but it's by intention.

You can do most of things you could do with RDD without RDD in DataFrame API - mapPartitions without calling rdd should give iterator to you and you can do with iterator most of the same you do with RDD.

> mapPartitions fails when called in streaming mode from Python
> -------------------------------------------------------------
>
>                 Key: SPARK-50451
>                 URL: https://issues.apache.org/jira/browse/SPARK-50451
>             Project: Spark
>          Issue Type: Bug
>          Components: Structured Streaming
>    Affects Versions: 3.5.0
>         Environment: OS: AL2023
> JVM: openjdk 22.0.2 2024-07-16
> OpenJDK Runtime Environment Corretto-22.0.2.9.1 (build 22.0.2+9-FR)
> OpenJDK 64-Bit Server VM Corretto-22.0.2.9.1 (build 22.0.2+9-FR, mixed mode, sharing)
>            Reporter: Alberto Andreotti
>            Priority: Major
>
> Calling the mapPartitions API in Python like this,
>  
> mapped_rdd = dataset.rdd.mapPartitions(process_partitions)
>  
> in Streaming mode results in the following error,
>  
> AnalysisException: Queries with streaming sources must be executed with writeStream.start();
> the same API works if I call it from Scala, or if I execute outside Streaming Mode.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org