You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2021/08/28 03:07:27 UTC

[GitHub] [hudi] melin opened a new issue #3554: [SUPPORT] Support Apache Spark 3.1

melin opened a new issue #3554:
URL: https://github.com/apache/hudi/issues/3554


   Apache Spark 3.1 has been released for a while and will hopefully be able to support it
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] borasy commented on issue #3554: [SUPPORT] Support Apache Spark 3.1

Posted by GitBox <gi...@apache.org>.
borasy commented on issue #3554:
URL: https://github.com/apache/hudi/issues/3554#issuecomment-909078185


   hudi 0.9 doesn't support Spark3.1.2 fully. More specifically inserting data with SparkSQL doesn't work. With Spark3.0.3, it works.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] xushiyan commented on issue #3554: [SUPPORT] Support Apache Spark 3.1

Posted by GitBox <gi...@apache.org>.
xushiyan commented on issue #3554:
URL: https://github.com/apache/hudi/issues/3554#issuecomment-917554170


   Moved to JIRA
   https://issues.apache.org/jira/browse/HUDI-1869


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] melin commented on issue #3554: [SUPPORT] Support Apache Spark 3.1

Posted by GitBox <gi...@apache.org>.
melin commented on issue #3554:
URL: https://github.com/apache/hudi/issues/3554#issuecomment-910186444


   /Spark3Adapter.scala:70: error: wrong number of arguments for pattern org.apache.spark.sql.catalyst.plans.logical.InsertIntoStatement(table: org.apache.spark.sql.catalyst.plans.logical.LogicalPlan,partitionSpec: Map[String,Option[String]],userSpecifiedCols: Seq[String],query: org.apache.spark.sql.catalyst.plans.logical.LogicalPlan,overwrite: Boolean,ifPartitionNotExists: Boolean)
    
   
   [ERROR] /Users/melin/Documents/codes/bigdata/hudi/hudi-sync/hudi-hive-sync/src/test/java/org/apache/hudi/hive/TestParquet2SparkSchemaUtils.java:[39,41] 无法将类 org.apache.spark.sql.execution.SparkSqlParser中的构造器 SparkSqlParser应型;
     需要: 没有参数
     找到: org.apache.spark.sql.internal.SQLConf
     原因: 实际参数列表和形式参数列表长度不同
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] nsivabalan commented on issue #3554: [SUPPORT] Support Apache Spark 3.1

Posted by GitBox <gi...@apache.org>.
nsivabalan commented on issue #3554:
URL: https://github.com/apache/hudi/issues/3554#issuecomment-917432982


   @pengzhiwei2018 : feel free to close out this issue if we have a tracking jira. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] khiem291 commented on issue #3554: [SUPPORT] Support Apache Spark 3.1

Posted by GitBox <gi...@apache.org>.
khiem291 commented on issue #3554:
URL: https://github.com/apache/hudi/issues/3554#issuecomment-907734386


   I'm working on building POC with hudi 0.8 and Spark3.1.2. Is there any issue with spark 3.1 @melin ?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] khiem291 edited a comment on issue #3554: [SUPPORT] Support Apache Spark 3.1

Posted by GitBox <gi...@apache.org>.
khiem291 edited a comment on issue #3554:
URL: https://github.com/apache/hudi/issues/3554#issuecomment-907734386


   I'm working on building POC with hudi 0.8 and Spark3.1.2. It work with Deltastream and HudiWriter 
   Is there any issue with hudi and spark 3.1 @melin ?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] kenny-shopback removed a comment on issue #3554: [SUPPORT] Support Apache Spark 3.1

Posted by GitBox <gi...@apache.org>.
kenny-shopback removed a comment on issue #3554:
URL: https://github.com/apache/hudi/issues/3554#issuecomment-907749695


   @findczk , https://github.com/apache/hudi/issues/2641


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] kenny-shopback commented on issue #3554: [SUPPORT] Support Apache Spark 3.1

Posted by GitBox <gi...@apache.org>.
kenny-shopback commented on issue #3554:
URL: https://github.com/apache/hudi/issues/3554#issuecomment-907749695


   @findczk , https://github.com/apache/hudi/issues/2641


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] melin edited a comment on issue #3554: [SUPPORT] Support Apache Spark 3.1

Posted by GitBox <gi...@apache.org>.
melin edited a comment on issue #3554:
URL: https://github.com/apache/hudi/issues/3554#issuecomment-910186444


   SPARK-32976 Support column list in INSERT statement
   ```
   /Spark3Adapter.scala:70: error: wrong number of arguments for pattern org.apache.spark.sql.catalyst.plans.logical.InsertIntoStatement(table: org.apache.spark.sql.catalyst.plans.logical.LogicalPlan,partitionSpec: Map[String,Option[String]],userSpecifiedCols: Seq[String],query: org.apache.spark.sql.catalyst.plans.logical.LogicalPlan,overwrite: Boolean,ifPartitionNotExists: Boolean)
   ```
   
   [ERROR] /Users/melin/Documents/codes/bigdata/hudi/hudi-sync/hudi-hive-sync/src/test/java/org/apache/hudi/hive/TestParquet2SparkSchemaUtils.java:[39,41] 无法将类 org.apache.spark.sql.execution.SparkSqlParser中的构造器 SparkSqlParser应型;
     需要: 没有参数
     找到: org.apache.spark.sql.internal.SQLConf
     原因: 实际参数列表和形式参数列表长度不同
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] khiem291 commented on issue #3554: [SUPPORT] Support Apache Spark 3.1

Posted by GitBox <gi...@apache.org>.
khiem291 commented on issue #3554:
URL: https://github.com/apache/hudi/issues/3554#issuecomment-907749868


   @findczk , https://github.com/apache/hudi/issues/2641


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] borasy commented on issue #3554: [SUPPORT] Support Apache Spark 3.1

Posted by GitBox <gi...@apache.org>.
borasy commented on issue #3554:
URL: https://github.com/apache/hudi/issues/3554#issuecomment-909078185


   hudi 0.9 doesn't support Spark3.1.2 fully. More specifically inserting data with SparkSQL doesn't work. With Spark3.0.3, it works.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] borasy commented on issue #3554: [SUPPORT] Support Apache Spark 3.1

Posted by GitBox <gi...@apache.org>.
borasy commented on issue #3554:
URL: https://github.com/apache/hudi/issues/3554#issuecomment-909078185


   hudi 0.9 doesn't support Spark3.1.2 fully. More specifically inserting data with SparkSQL doesn't work. With Spark3.0.3, it works.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] vinothchandar commented on issue #3554: [SUPPORT] Support Apache Spark 3.1

Posted by GitBox <gi...@apache.org>.
vinothchandar commented on issue #3554:
URL: https://github.com/apache/hudi/issues/3554#issuecomment-910391795


   cc @pengzhiwei2018 could you please comment on this?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] khiem291 edited a comment on issue #3554: [SUPPORT] Support Apache Spark 3.1

Posted by GitBox <gi...@apache.org>.
khiem291 edited a comment on issue #3554:
URL: https://github.com/apache/hudi/issues/3554#issuecomment-907734386


   I'm working on building POC with hudi 0.8 and Spark3.1.2. Is there any issue with hudi and spark 3.1 @melin ?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] melin edited a comment on issue #3554: [SUPPORT] Support Apache Spark 3.1

Posted by GitBox <gi...@apache.org>.
melin edited a comment on issue #3554:
URL: https://github.com/apache/hudi/issues/3554#issuecomment-910186444


   SPARK-32976 Support column list in INSERT statement
   ```/Spark3Adapter.scala:70: error: wrong number of arguments for pattern org.apache.spark.sql.catalyst.plans.logical.InsertIntoStatement(table: org.apache.spark.sql.catalyst.plans.logical.LogicalPlan,partitionSpec: Map[String,Option[String]],userSpecifiedCols: Seq[String],query: org.apache.spark.sql.catalyst.plans.logical.LogicalPlan,overwrite: Boolean,ifPartitionNotExists: Boolean)
   ```
   
   [ERROR] /Users/melin/Documents/codes/bigdata/hudi/hudi-sync/hudi-hive-sync/src/test/java/org/apache/hudi/hive/TestParquet2SparkSchemaUtils.java:[39,41] 无法将类 org.apache.spark.sql.execution.SparkSqlParser中的构造器 SparkSqlParser应型;
     需要: 没有参数
     找到: org.apache.spark.sql.internal.SQLConf
     原因: 实际参数列表和形式参数列表长度不同
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] pengzhiwei2018 commented on issue #3554: [SUPPORT] Support Apache Spark 3.1

Posted by GitBox <gi...@apache.org>.
pengzhiwei2018 commented on issue #3554:
URL: https://github.com/apache/hudi/issues/3554#issuecomment-911051419


   Yes, spark 3.1 has many changes to the class for spark sql. I have submit a PR to adapter 3.1 https://github.com/apache/hudi/pull/3580


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] findczk commented on issue #3554: [SUPPORT] Support Apache Spark 3.1

Posted by GitBox <gi...@apache.org>.
findczk commented on issue #3554:
URL: https://github.com/apache/hudi/issues/3554#issuecomment-907739489


   
   > I'm working on building POC with hudi 0.8 and Spark3.1.2. Is there any issue with spark 3.1 @melin ?
   Does it support spark3.0.1? When I set it to spark3.0.1, the compilation is unsuccessful, because the default is spark3.0.0, spark3.0.1 has many methods to change
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [hudi] xushiyan closed issue #3554: [SUPPORT] Support Apache Spark 3.1

Posted by GitBox <gi...@apache.org>.
xushiyan closed issue #3554:
URL: https://github.com/apache/hudi/issues/3554


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org