You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@kudu.apache.org by "Attila Zsolt Piros (JIRA)" <ji...@apache.org> on 2018/08/13 10:28:00 UTC

[jira] [Created] (KUDU-2539) Supporting Spark Streaming DataFrame in KuduContext

Attila Zsolt Piros created KUDU-2539:
----------------------------------------

             Summary: Supporting Spark Streaming DataFrame in KuduContext
                 Key: KUDU-2539
                 URL: https://issues.apache.org/jira/browse/KUDU-2539
             Project: Kudu
          Issue Type: Improvement
          Components: spark
    Affects Versions: 1.8.0
            Reporter: Attila Zsolt Piros


Currently KuduContext does not support Spark Streaming DataFrame. The problem comes from a foreachPartition call which in case of spark streaming is an unsupported operation, like foreach: 

[unsupported operations in streaming|https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html#unsupported-operations]

I have created a small example app with a custom Kudu sink which can be used for testing:

[kudu custom sink and example app|https://github.com/attilapiros/kudu_custom_sink]

The patch fixing this issue is also available for kudu-spark, so soon a gerrit review can be expected with the solution.  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)