You are viewing a plain text version of this content. The canonical link for it is here.

Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2022/07/28 10:59:10 UTC

[GitHub] [beam] damo1991 opened a new issue, #22492: SparkStructureStreamingRunner issue due to spark3.*

damo1991 opened a new issue, #22492:
URL: https://github.com/apache/beam/issues/22492

   We are getting below error on using spark 3 and above version:
   
   java.lang.NoClassDefFoundError: org/apache/spark/sql/sources/v2/DataSourceV2
   
   This error is due to SparkStructuredStreamingRunner internally using DataSourceV2 api from spark 3.* this class is not found, as it is tightly coupled with DataSourceV2 class we are facing this issue:
   
   Please see below code snippet for the same.
   
   public class DatasetSourceBatch implements DataSourceV2, ReadSupport {
   
   @Override
   public DataSourceReader createReader(DataSourceOptions options)
   
   { return new DatasetReader<>(options); }


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org