You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2020/06/15 16:33:33 UTC

[GitHub] [iceberg] vvysotskyi opened a new issue #1114: Add support for spark-sql INSERT INTO command

vvysotskyi opened a new issue #1114:
URL: https://github.com/apache/iceberg/issues/1114


   The current version of the Iceberg project successfully queries the data using spark-sql functionality. But it fails when running `insert into` queries.
   
   The following test
   ```
     @Test
     public void testSqlInsert() throws IOException {
       File parent = temp.newFolder(format.toString());
       File location = new File(parent, "test");
   
       HadoopTables tables = new HadoopTables(CONF);
       PartitionSpec spec = PartitionSpec.builderFor(SCHEMA).identity("data").build();
       Table table = tables.create(SCHEMA, spec, location.toString());
   
       table.refresh();
   
       spark.read()
           .format("iceberg")
           .load(location.toString())
           .createOrReplaceTempView("test_table");
   
       spark.sql("insert into test_table values(4, 'd')");
     }
   ```
   fails with the error:
   ```
   org.apache.spark.sql.AnalysisException: unresolved operator 'InsertIntoTable RelationV2 iceberg[id#0, data#1] (Options: [path=/tmp/junit4419829961099417997/PARQUET/test,paths=[]]), false, false;;
   'InsertIntoTable RelationV2 iceberg[id#0, data#1] (Options: [path=/tmp/junit4419829961099417997/PARQUET/test,paths=[]]), false, false
   +- LocalRelation [col1#4, col2#5]
   
   
   	at org.apache.spark.sql.catalyst.analysis.CheckAnalysis$class.failAnalysis(CheckAnalysis.scala:43)
   	at org.apache.spark.sql.catalyst.analysis.Analyzer.failAnalysis(Analyzer.scala:95)
   	at org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$checkAnalysis$3.apply(CheckAnalysis.scala:431)
   	at org.apache.spark.sql.catalyst.analysis.CheckAnalysis$$anonfun$checkAnalysis$3.apply(CheckAnalysis.scala:430)
   	at org.apache.spark.sql.catalyst.trees.TreeNode.foreachUp(TreeNode.scala:126)
   	at org.apache.spark.sql.catalyst.analysis.CheckAnalysis$class.checkAnalysis(CheckAnalysis.scala:430)
   	at org.apache.spark.sql.catalyst.analysis.Analyzer.checkAnalysis(Analyzer.scala:95)
   	at org.apache.spark.sql.catalyst.analysis.Analyzer$$anonfun$executeAndCheck$1.apply(Analyzer.scala:108)
   	at org.apache.spark.sql.catalyst.analysis.Analyzer$$anonfun$executeAndCheck$1.apply(Analyzer.scala:105)
   	at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper$.markInAnalyzer(AnalysisHelper.scala:201)
   	at org.apache.spark.sql.catalyst.analysis.Analyzer.executeAndCheck(Analyzer.scala:105)
   	at org.apache.spark.sql.execution.QueryExecution.analyzed$lzycompute(QueryExecution.scala:58)
   	at org.apache.spark.sql.execution.QueryExecution.analyzed(QueryExecution.scala:56)
   	at org.apache.spark.sql.execution.QueryExecution.assertAnalyzed(QueryExecution.scala:48)
   	at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:78)
   	at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:642)
   	at org.apache.iceberg.spark.source.TestSparkDataWrite.testSqlInsert(TestSparkDataWrite.java:251)
   	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   	at java.lang.reflect.Method.invoke(Method.java:498)
   	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
   	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
   	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
   	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
   	at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
   	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
   	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
   	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
   	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
   	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
   	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
   	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
   	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
   	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
   	at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
   	at org.junit.runners.Suite.runChild(Suite.java:128)
   	at org.junit.runners.Suite.runChild(Suite.java:27)
   	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
   	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
   	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
   	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
   	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
   	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
   	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
   	at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
   	at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
   	at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
   	at com.intellij.rt.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:33)
   	at com.intellij.rt.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:230)
   	at com.intellij.rt.junit.JUnitStarter.main(JUnitStarter.java:58)
   ```


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] vvysotskyi commented on issue #1114: Add support for spark-sql INSERT INTO command

Posted by GitBox <gi...@apache.org>.
vvysotskyi commented on issue #1114:
URL: https://github.com/apache/iceberg/issues/1114#issuecomment-663831580


   Great, thanks!


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] GrigorievNick commented on issue #1114: Add support for spark-sql INSERT INTO command

Posted by GitBox <gi...@apache.org>.
GrigorievNick commented on issue #1114:
URL: https://github.com/apache/iceberg/issues/1114#issuecomment-658112956


   @vvysotskyi It's here.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] yiguolei commented on issue #1114: Add support for spark-sql INSERT INTO command

Posted by GitBox <gi...@apache.org>.
yiguolei commented on issue #1114:
URL: https://github.com/apache/iceberg/issues/1114#issuecomment-647369713


   Do you use spark-3 branch?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] yiguolei commented on issue #1114: Add support for spark-sql INSERT INTO command

Posted by GitBox <gi...@apache.org>.
yiguolei commented on issue #1114:
URL: https://github.com/apache/iceberg/issues/1114#issuecomment-650737185


   @rdblue when will 0.9.0 be released?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] rdblue commented on issue #1114: Add support for spark-sql INSERT INTO command

Posted by GitBox <gi...@apache.org>.
rdblue commented on issue #1114:
URL: https://github.com/apache/iceberg/issues/1114#issuecomment-650468284


   This support is coming with the 0.9.0 release for Spark 3.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] rdblue closed issue #1114: Add support for spark-sql INSERT INTO command

Posted by GitBox <gi...@apache.org>.
rdblue closed issue #1114:
URL: https://github.com/apache/iceberg/issues/1114


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] rdblue commented on issue #1114: Add support for spark-sql INSERT INTO command

Posted by GitBox <gi...@apache.org>.
rdblue commented on issue #1114:
URL: https://github.com/apache/iceberg/issues/1114#issuecomment-663728548


   This was released in 0.9.0.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] rdblue commented on issue #1114: Add support for spark-sql INSERT INTO command

Posted by GitBox <gi...@apache.org>.
rdblue commented on issue #1114:
URL: https://github.com/apache/iceberg/issues/1114#issuecomment-651286889


   I'm aiming to get it out in the next few weeks, now that Spark 3 has been released.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org