[GitHub] [hudi] glory9211 commented on issue #6107: [SUPPORT] RO table did not get updated while RT table did

[GitBox <gi...@apache.org>]

glory9211 commented on issue #6107:
URL: https://github.com/apache/hudi/issues/6107#issuecomment-1195603690

   > > looks like something to do with meta sync where RO is not getting sync'ed. please provide scripts and configs for reproducing then we can help from there.
   > 
   > please find below configuration which we are using currently
   > 
   > hudi_options = { 'hoodie.datasource.write.table.type': self._write_table_type, 'hoodie.table.name': self._table_name, 'hoodie.datasource.write.recordkey.field': self._record_key, 'hoodie.datasource.write.partitionpath.field': self._partition_field, 'hoodie.datasource.write.precombine.field': self._combine_key, 'hoodie.datasource.write.keygenerator.class': 'org.apache.hudi.keygen.ComplexKeyGenerator', 'hoodie.parquet.max.file.size': "20971520", 'hoodie.datasource.hive_sync.enable': 'true', 'hoodie.datasource.hive_sync.table': self._table_name.lower(), 'hoodie.datasource.hive_sync.partition_fields': self._partition_field, 'hoodie.datasource.hive_sync.partition_extractor_class': 'org.apache.hudi.hive.MultiPartKeysValueExtractor', 'hoodie.datasource.hive_sync.database': self._hive_database.lower(), 'hoodie.datasource.write.hive_style_partitioning': 'true', 'hoodie.datasource.hive_sync.mode': 'hms', 'hoodie.datasource.hive_sync.support_timestamp': 'true' }
   
   
   As mentioned by @KnightChess RT and RO tables are synced when you run compaction on Hudi MOR tables.
   i.e. The Delta (Avro) Files are merged into the Parquet Files. In HUDI
   COW Tables == Data in Parquet Files
   MOR Tables == Data in Avro + Parquet Files
   
   You can read the configs from the docs [here](https://hudi.apache.org/docs/configurations/)
   
   Some sample configs you should provide
   
   ```
   ## Compaction
       'hoodie.compact.inline.max.delta.seconds' : 60,
       'hoodie.compact.inline.max.delta.commits' : 4,
       'hoodie.compact.inline.trigger.strategy' : 'NUM_OR_TIME',
       'hoodie.compact.inline' : True,
       'hoodie.datasource.compaction.async.enable' : True,
   
   ```
   
   This will trigger compaction after every 60 seconds or 4 delta commits for a streaming job.
   Read more about what is compaction in Hudi [here](https://hudi.apache.org/docs/compaction)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org