You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Hyukjin Kwon (JIRA)" <ji...@apache.org> on 2019/03/22 02:56:00 UTC

[jira] [Commented] (SPARK-27234) Continuous Streaming does not support python UDFs

    [ https://issues.apache.org/jira/browse/SPARK-27234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16798654#comment-16798654 ] 

Hyukjin Kwon commented on SPARK-27234:
--------------------------------------

What's {{makeReply}} and output from the codes?

> Continuous Streaming does not support python UDFs
> -------------------------------------------------
>
>                 Key: SPARK-27234
>                 URL: https://issues.apache.org/jira/browse/SPARK-27234
>             Project: Spark
>          Issue Type: Bug
>          Components: Structured Streaming
>    Affects Versions: 2.4.0
>         Environment: Azure Databricks 5.1
>            Reporter: Mark Hamilton
>            Priority: Major
>
> Heres a repro
> get_p_eng = udf(lambda p: "foo")
> df = spark \
>  .readStream \
>  .format("rate") \
>  .load()\
>  .withColumn("p_eng", get_p_eng(col("value")))\
>  .makeReply("p_eng")
> df.isStreaming # Returns True for DataFrames that have streaming sources
> df.printSchema
> # Read all the csv files written atomically in a directory
> df.writeStream.format("console").trigger(continuous="1 second").start()



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org