[GitHub] [iceberg] zhangxia1030 opened a new issue #3705: Explore spark struct streaming write iceberg and synchronize to hive Metastore

[GitBox <gi...@apache.org>]

zhangxia1030 opened a new issue #3705:
URL: https://github.com/apache/iceberg/issues/3705


   1. build sparksession
    ```
   val spark = SparkSession
         .builder()
         .config("spark.sql.extensions", "org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions")
         .config("spark.sql.catalog.spark_catalog", "org.apache.iceberg.spark.SparkSessionCatalog")
         .config("spark.sql.catalog.spark_catalog.type", "hive")
         .config(SQLConf.PARTITION_OVERWRITE_MODE.key, "dynamic")
         .config("spark.hadoop.hive.metastore.uris", "thrift://ip:port")
         .config("hive.exec.dynamic.partition", "true")
         .config("hive.exec.dynamic.partition.mode", "nonstrict")
         .appName("test-iceberg")
         .master("local[*]")
         .enableHiveSupport()
         .getOrCreate()
   ```
   2. use hive catalog
   ```
   val v2catlog = spark.sessionState.catalogManager.v2SessionCatalog.asInstanceOf[SparkSessionCatalog[SparkCatalog]]
   v2catlog.createTable(id, schema, Spark3Util.toTransforms(spec.build()), immutableMap)
   
   ```
   
   error log:
   ```
   Caused by: org.apache.hadoop.ipc.RemoteException: File does not exist: /path/metadata/version-hint.text
   	at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:71)
   	at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:61)
   ```
   
   **Spark 3: HadoopTables create table, that not sync hive metastore?**
   
   3.   use hadoop catalog
   ```
   val tables = new HadoopTables(spark.sparkContext.hadoopConfiguration)
   tables.create(schema, spec, sort, immutableMap, path)
   ```
   **spark struct streaming works normally, but I need to synchronize the meta information to hive metastroe, Is there any way?**


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org