You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "machadoluiz (via GitHub)" <gi...@apache.org> on 2024/09/10 19:10:37 UTC

[I] [SUPPORT] Comments for partition columns not synced to Hive/AWS Glue [hudi]

machadoluiz opened a new issue, #11922:
URL: https://github.com/apache/hudi/issues/11922

   **Describe the problem you faced**
   
   When using Apache Hudi to sync tables with Hive/AWS Glue using the `hoodie.datasource.hive_sync.sync_comment = true` option, comments for non-partition columns are synced correctly. However, the comments for partition fields are not being synced. This leads to partition columns in the schema missing their associated comments when viewed in AWS Glue Data Catalog.
   
   **To Reproduce**
   
   Steps to reproduce the behavior:
   
   1. Create a Hudi table in PySpark with both non-partition and partition columns.
   2. Define metadata comments for all columns, including the partition column.
   3. Write the table to Hudi with Hive sync enabled, using the option hoodie.datasource.hive_sync.sync_comment = true.
   
   Such as:
   ```Python
   from pyspark.sql import SparkSession
   from pyspark.sql.types import StructType, StructField, StringType, IntegerType
   
   spark = SparkSession.builder \
       .appName("HudiTableWithComments") \
       .config("spark.serializer", "org.apache.spark.serializer.KryoSerializer") \
       .config("spark.sql.catalog.spark_catalog", "org.apache.spark.sql.hudi.catalog.HoodieCatalog") \
       .config("spark.sql.extensions", "org.apache.spark.sql.hudi.HoodieSparkSessionExtension") \
       .config("spark.kryo.registrator", "org.apache.spark.HoodieSparkKryoRegistrar") \
       .getOrCreate()
   
   schema = StructType([
       StructField("id", StringType(), nullable=False, metadata={"comment": "Unique identifier"}),
       StructField("name", StringType(), nullable=True, metadata={"comment": "Name of the person"}),
       StructField("age", IntegerType(), nullable=True, metadata={"comment": "Age of the person"}),
       StructField("date", StringType(), nullable=True, metadata={"comment": "Partition field, date of entry"})  # Partition field
   ])
   
   data = [
       ("1", "John", 30, "2023-01-01"),
       ("2", "Jane", 25, "2023-01-01"),
       ("3", "Bob", 35, "2023-01-02")
   ]
   
   df = spark.createDataFrame(data, schema=schema)
   
   hudi_options = {
       'hoodie.table.name': 'hudi_table_with_comments',
       "hoodie.datasource.write.table.name": "hudi_table_with_comments",
       'hoodie.datasource.write.operation': 'insert',
       'hoodie.datasource.write.partitionpath.field': 'date',
       'hoodie.datasource.write.precombine.field': 'id',
       'hoodie.datasource.write.recordkey.field': 'id',
       "hoodie.datasource.write.schema.allow.auto.evolution.column.drop": "true",
       'hoodie.datasource.hive_sync.database': 'default',
       'hoodie.datasource.hive_sync.enable': 'true',
       'hoodie.datasource.hive_sync.partition_extractor_class': 'org.apache.hudi.hive.MultiPartKeysValueExtractor',
       'hoodie.datasource.hive_sync.partition_fields': 'date',
       'hoodie.datasource.hive_sync.sync_comment': 'true',
       'hoodie.datasource.hive_sync.table': 'hudi_table_with_comments',
       "hoodie.datasource.hive_sync.mode": "hms",
       "hoodie.datasource.hive_sync.support_timestamp": "true",
       "hoodie.datasource.hive_sync.use_jdbc": "false",
       "hoodie.merge.allow.duplicate.on.inserts": "false",
       "hoodie.schema.on.read.enable": "true",
   }
   
   df.write.format("hudi").options(**hudi_options).mode("append").save("...")
   ```
   
   After running it, you'll find that comments for non-partition columns (id, name, age) are correctly synced, but the partition field (date) does not have its comment.
   
   **Expected behavior**
   
   That comments for both non-partition and partition columns should be synced. The partition field should also have its comment displayed in the table schema.
   
   **Environment Description**
   
   * Hudi version : 0.15.0
   
   * Spark version : 3.5.1
   
   * Hive version : 3.1.3
   
   * Hadoop version : 3.3.6
   
   * Storage (HDFS/S3/GCS..) : S3
   
   * Running on Docker? (yes/no) : no
   
   
   **Additional context**
   
   I am running it in a AWS EMR cluster (version 7.2.0).
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [SUPPORT] Comments for partition columns not synced to Hive/AWS Glue [hudi]

Posted by "deepakpanda93 (via GitHub)" <gi...@apache.org>.
deepakpanda93 closed issue #11922: [SUPPORT] Comments for partition columns not synced to Hive/AWS Glue
URL: https://github.com/apache/hudi/issues/11922


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [SUPPORT] Comments for partition columns not synced to Hive/AWS Glue [hudi]

Posted by "rangareddy (via GitHub)" <gi...@apache.org>.
rangareddy commented on issue #11922:
URL: https://github.com/apache/hudi/issues/11922#issuecomment-2577605489

   **Hive Table Create Statement:**
   
   ```sql
   +----------------------------------------------------+
   |                   createtab_stmt                   |
   +----------------------------------------------------+
   | CREATE EXTERNAL TABLE `hive_table_with_comments`(  |
   |   `id` string COMMENT 'Unique identifier',         |
   |   `name` string COMMENT 'Name of the person',      |
   |   `age` int COMMENT 'Age of the person',           |
   |   `date` string COMMENT 'Partition field, date of entry') |
   | ROW FORMAT SERDE                                   |
   |   'org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe'  |
   | WITH SERDEPROPERTIES (                             |
   |   'path'='s3a://warehouse/hive_table_with_comments')  |
   | STORED AS INPUTFORMAT                              |
   |   'org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat'  |
   | OUTPUTFORMAT                                       |
   |   'org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat' |
   | LOCATION                                           |
   |   's3a://warehouse/hive_table_with_comments'       |
   | TBLPROPERTIES (                                    |
   |   'TRANSLATED_TO_EXTERNAL'='TRUE',                 |
   |   'external.table.purge'='TRUE',                   |
   |   'spark.sql.create.version'='3.5.3',              |
   |   'spark.sql.sources.provider'='parquet',          |
   |   'spark.sql.sources.schema'='{"type":"struct","fields":[{"name":"id","type":"string","nullable":true,"metadata":{"comment":"Unique identifier"}},{"name":"name","type":"string","nullable":true,"metadata":{"comment":"Name of the person"}},{"name":"age","type":"integer","nullable":true,"metadata":{"comment":"Age of the person"}},{"name":"date","type":"string","nullable":true,"metadata":{"comment":"Partition field, date of entry"}}]}',  |
   |   'transient_lastDdlTime'='1736340771')            |
   +----------------------------------------------------+
   22 rows selected (0.208 seconds)
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [SUPPORT] Comments for partition columns not synced to Hive/AWS Glue [hudi]

Posted by "rangareddy (via GitHub)" <gi...@apache.org>.
rangareddy commented on issue #11922:
URL: https://github.com/apache/hudi/issues/11922#issuecomment-2577626538

   Created a Hudi Issue - https://issues.apache.org/jira/browse/HUDI-8843


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org