[GitHub] [iceberg] t0ma-sz commented on issue #5565: [Spark] Cannot append to Glue table - StorageDescriptor#InputFormat cannot be null for table

[GitBox <gi...@apache.org>]

t0ma-sz commented on issue #5565:
URL: https://github.com/apache/iceberg/issues/5565#issuecomment-1218476324

   > > ```
   > > val tableExists = sparkSession.catalog.tableExists(s"$dbName.$tableName")
   > > ```
   > 
   > 
   > I think this is happening because you haven't specified the catalog name in identifier of write to and neither made my_catalog as default ....
   > 
   > 
   > can you please try once with 3 part identifier here i.e sparkSession.catalog.tableExists(s"$catalog.$dbName.$tableName")
   > 
   > 
   > ref : A ticket stating similar issue : [#2202 (comment)](https://github.com/apache/iceberg/issues/2202#issuecomment-786301938)
   
   Thanks for your input. This line is not creating the issue, it is caused by DataFrameWriterV2.append() method.
   `df.writeTo(s"$catalog.dbName.$tableName")
           .append`
   
   Anyway I changed the line you asked, including catalog name in the tableExists method, but I received another error. The error is not there, when I downgraded to Iceberg 0.13.0
   
   Error trace, when I used iceberg 0.14.0 and included catalog name in 
   
   `sparkSession.catalog.tableExists(s"$catalog.$dbName.$tableName")`
   
   ```
   2022-08-17 12:56:03,376 ERROR [main] glue.ProcessLauncher (Logging.scala:logError(94)): Exception in User Class
   org.apache.spark.sql.catalyst.parser.ParseException: 
   mismatched input '.' expecting {<EOF>, '-'}(line 1, pos 43)
   
   == SQL ==
   glue_catalog.mydb.mytable
   -----------------^^^
   
   	at org.apache.spark.sql.catalyst.parser.ParseException.withCommand(ParseDriver.scala:255)
   	at org.apache.spark.sql.catalyst.parser.AbstractSqlParser.parse(ParseDriver.scala:124)
   	at org.apache.spark.sql.execution.SparkSqlParser.parse(SparkSqlParser.scala:49)
   	at org.apache.spark.sql.catalyst.parser.AbstractSqlParser.parseTableIdentifier(ParseDriver.scala:48)
   	at org.apache.spark.sql.catalyst.parser.extensions.IcebergSparkSqlExtensionsParser.parseTableIdentifier(IcebergSparkSqlExtensionsParser.scala:73)
   	at org.apache.spark.sql.internal.CatalogImpl.tableExists(CatalogImpl.scala:251)
   	at Main$.main(Main.scala:78)
   	at Main.main(Main.scala)
   	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   	at java.lang.reflect.Method.invoke(Method.java:498)
   	at com.amazonaws.services.glue.SparkProcessLauncherPlugin.invoke(ProcessLauncher.scala:48)
   	at com.amazonaws.services.glue.SparkProcessLauncherPlugin.invoke$(ProcessLauncher.scala:48)
   	at com.amazonaws.services.glue.ProcessLauncher$$anon$1.invoke(ProcessLauncher.scala:78)
   	at com.amazonaws.services.glue.ProcessLauncher.launch(ProcessLauncher.scala:143)
   	at com.amazonaws.services.glue.ProcessLauncher$.main(ProcessLauncher.scala:30)
   	at com.amazonaws.services.glue.ProcessLauncher.main(ProcessLauncher.scala)
   ``` 
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org