[GitHub] [hudi] ksrihari93 commented on issue #5823: Hoodie Deltastreamer Job unable to fecth data from kafka topic from starting offset available

[GitBox <gi...@apache.org>]

ksrihari93 commented on issue #5823:
URL: https://github.com/apache/hudi/issues/5823#issuecomment-1155450860

   Hi @nsivabalan ,
   
   
   
   #base properties
   
   hoodie.insert.shuffle.parallelism=50
   hoodie.bulkinsert.shuffle.parallelism=200
   hoodie.embed.timeline.server=true
   hoodie.filesystem.view.type=EMBEDDED_KV_STORE
   hoodie.compact.inline=false
   hoodie.bulkinsert.sort.mode=none
   
   
   #cleaner properties
   hoodie.cleaner.policy=KEEP_LATEST_FILE_VERSIONS
   hoodie.cleaner.fileversions.retained=60
   hoodie.clean.async=true
   
   #archival
   hoodie.keep.min.commits=12
   hoodie.keep.max.commits=15
   
   #datasource properties
   hoodie.deltastreamer.schemaprovider.registry.url=
   hoodie.datasource.write.recordkey.field=
   hoodie.deltastreamer.source.kafka.topic=
   hoodie.datasource.write.keygenerator.class=org.apache.hudi.keygen.CustomKeyGenerator
   hoodie.datasource.write.partitionpath.field=timestamp:TIMESTAMP
   hoodie.deltastreamer.kafka.source.maxEvents=600000000
   hoodie.deltastreamer.keygen.timebased.timestamp.type=EPOCHMILLISECONDS
   hoodie.deltastreamer.keygen.timebased.input.timezone=UTC
   hoodie.deltastreamer.keygen.timebased.output.timezone=UTC
   hoodie.deltastreamer.keygen.timebased.output.dateformat='dt='yyyy-MM-dd
   hoodie.clustering.async.enabled=true
   hoodie.clustering.plan.strategy.target.file.max.bytes=3000000000
   hoodie.clustering.plan.strategy.small.file.limit=200000001
   hoodie.clustering.async.max.commits=1
   hoodie.clustering.plan.strategy.max.num.groups=10
   oodie.clustering.plan.strategy.max.bytes.per.group=9000000000
   
   
   #kafka props
   bootstrap.servers=
   group.id=hudi-lpe
   auto.offset.reset=(As is said above when i passed earliest it got failed) ,so no other choice to recover  i have passed latest
   hoodie.deltastreamer.source.kafka.checkpoint.type=timestamp
   
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org