You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Hyukjin Kwon (JIRA)" <ji...@apache.org> on 2019/03/22 02:56:00 UTC
[jira] [Commented] (SPARK-27234) Continuous Streaming does not
support python UDFs
[ https://issues.apache.org/jira/browse/SPARK-27234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16798654#comment-16798654 ]
Hyukjin Kwon commented on SPARK-27234:
--------------------------------------
What's {{makeReply}} and output from the codes?
> Continuous Streaming does not support python UDFs
> -------------------------------------------------
>
> Key: SPARK-27234
> URL: https://issues.apache.org/jira/browse/SPARK-27234
> Project: Spark
> Issue Type: Bug
> Components: Structured Streaming
> Affects Versions: 2.4.0
> Environment: Azure Databricks 5.1
> Reporter: Mark Hamilton
> Priority: Major
>
> Heres a repro
> get_p_eng = udf(lambda p: "foo")
> df = spark \
> .readStream \
> .format("rate") \
> .load()\
> .withColumn("p_eng", get_p_eng(col("value")))\
> .makeReply("p_eng")
> df.isStreaming # Returns True for DataFrames that have streaming sources
> df.printSchema
> # Read all the csv files written atomically in a directory
> df.writeStream.format("console").trigger(continuous="1 second").start()
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org