You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "Raymond Xu (Jira)" <ji...@apache.org> on 2022/08/20 23:15:00 UTC

[jira] [Closed] (HUDI-4374) Support BULK_INSERT row-writing on streaming Dataset/DataFrame

     [ https://issues.apache.org/jira/browse/HUDI-4374?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Raymond Xu closed HUDI-4374.
----------------------------
    Fix Version/s: 0.12.0
                       (was: 0.13.0)
       Resolution: Fixed

> Support BULK_INSERT row-writing on streaming Dataset/DataFrame 
> ---------------------------------------------------------------
>
>                 Key: HUDI-4374
>                 URL: https://issues.apache.org/jira/browse/HUDI-4374
>             Project: Apache Hudi
>          Issue Type: Task
>          Components: spark, writer-core
>            Reporter: Sagar Sumit
>            Assignee: Sagar Sumit
>            Priority: Major
>              Labels: pull-request-available, streaming
>             Fix For: 0.12.0
>
>
> With structured streaming setup, when Hudi table is written from a streaming source, then HoodieStreamingSink calls HoodieSparkSqlWriter.write(). If BULK_INSERT operation type is set, then HoodieSparkSqlWriter.write() internally calls HoodieSparkSqlWriter.bulkInsertAsRow() which does a simple df.write.format("hudi").options(...).save(). The 'write' call does not work on streaming Dataset/DataFrame.
> {code:java}
> org.apache.spark.sql.AnalysisException: 'write' can not be called on streaming Dataset/DataFrame
>     at org.apache.spark.sql.catalyst.analysis.package$AnalysisErrorAt.failAnalysis(package.scala:42)
>     at org.apache.spark.sql.Dataset.write(Dataset.scala:3377)
>     at org.apache.hudi.HoodieSparkSqlWriter$.bulkInsertAsRow(HoodieSparkSqlWriter.scala:557)
>     at org.apache.hudi.HoodieSparkSqlWriter$.write(HoodieSparkSqlWriter.scala:178)
>     at org.apache.hudi.HoodieStreamingSink.$anonfun$addBatch$2(HoodieStreamingSink.scala:91)
>     at scala.util.Try$.apply(Try.scala:213)
>     at org.apache.hudi.HoodieStreamingSink.$anonfun$addBatch$1(HoodieStreamingSink.scala:90)
>     at org.apache.hudi.HoodieStreamingSink.retry(HoodieStreamingSink.scala:166)
>     at org.apache.hudi.HoodieStreamingSink.addBatch(HoodieStreamingSink.scala:89) {code}
> Bulk insert can still be done by not going via the row-writing path. But, we need to fix the HoodieStreamingSink to support bulk insert via row-writing.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)