[jira] [Updated] (HUDI-7349) Spark structured streaming didnt work after upgrade from hudi 0.11 to 0.13

["voon (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HUDI-7349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

voon updated HUDI-7349:
-----------------------
    Fix Version/s: 1.1.0
                       (was: 1.0.2)

> Spark structured streaming didnt work after upgrade from hudi 0.11 to 0.13
> --------------------------------------------------------------------------
>
>                 Key: HUDI-7349
>                 URL: https://issues.apache.org/jira/browse/HUDI-7349
>             Project: Apache Hudi
>          Issue Type: Bug
>          Components: spark, spark-sql
>    Affects Versions: 0.13.0
>         Environment: Environment Description
> Hudi version : 0.13.0
> Amazon EMR version : emr-6.11.1
> Spark version : 3.3.2
> Hive version : 3.1.3
>  
> Hadoop version : 3.3.3 
> Storage (HDFS/S3/GCS..): S3
> Cluster manager : yarn
>            Reporter: Haitham Eltaweel
>            Priority: Major
>             Fix For: 1.1.0
>
>
> We have Spark structured streaming job writing data in hudi format. After we made an upgrade from hudi 0.11.0 to hudi 0.13.0, the streaming app doesn't write data to existing hudi table. The streaming app started successfully, triggered listing job but didn't trigger any other job to compact, clean , write data , etc. No errors in Spark UI nor Stdout/Stderr logs. When running the streaming application to write to new s3 location (hudie table), everything works fine.  We use append output mode and 30 seconds trigger processing time. 
> Here are hudi configurations used (confiscated some values with xxx): 
> 'hoodie.datasource.write.table.type': 'MERGE_ON_READ',
> 'hoodie.datasource.write.keygenerator.class': 'org.apache.hudi.keygen.CustomKeyGenerator',
> 'hoodie.datasource.write.precombine.field': 'xxx',
> 'hoodie.datasource.write.partitionpath.field': 'xxx:SIMPLE',
> 'hoodie.embed.timeline.server': False,
> 'hoodie.index.type': 'BLOOM',
> 'hoodie.parquet.compression.codec': 'snappy',
> 'hoodie.clean.async': True,
> 'hoodie.clean.max.commits': 5,
> 'hoodie.parquet.max.file.size': 125829120,
> 'hoodie.parquet.small.file.limit': 104857600,
> 'hoodie.parquet.block.size': 125829120,
> 'hoodie.metadata.enable': True,
> 'hoodie.metadata.validate': True,
> 'hoodie.datasource.write.hive_style_partitioning': True,
> 'hoodie.datasource.hive_sync.support_timestamp': True,
> 'hoodie.datasource.hive_sync.jdbcurl': "xxx",
> 'hoodie.datasource.hive_sync.username': 'xxx',
> 'hoodie.datasource.hive_sync.password': 'xxx',
> 'hoodie.datasource.hive_sync.partition_fields': 'xxx',
> 'hoodie.datasource.hive_sync.enable': True,
> 'hoodie.datasource.hive_sync.partition_extractor_class': 'org.apache.hudi.hive.MultiPartKeysValueExtractor',
> 'hoodie.avro.schema.external.transformation': True,
> 'hoodie.avro.schema.validate': True,
> 'hoodie.table.name', 'xxx'
> 'hoodie.datasource.write.table.name', 'xxx'
> 'hoodie.datasource.write.recordkey.field', 'xxx'
> 'hoodie.datasource.hive_sync.database', 'xxx'
> 'hoodie.datasource.hive_sync.table', 'xxx'
> 'hoodie.datasource.write.operation', 'upsert'



--
This message was sent by Atlassian Jira
(v8.20.10#820010)