deltastreamer group.id Noeffectaftersetting

["965147871@qq.com" <96...@qq.com>]

hello, all When using deltastreamer to use kafka data, I want to specify group.id, but the problem encountered is that after specifying it, I cannot find it on the kafka side. I found that there are no groups under my theme. why is it like this? I also manually set enable.auto.commit = true at the same time, but it didn't seem to work. In kafkaUtils.scala fixKafkaParams this method,
   kafkaParams.put (ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, false: java.lang.Boolean), 
which is forced to be rewritten as false. I think it is one of the reasons why the group cannot be found, but when it is not automatically submitted, it is usually chosen to be submitted manually. , I did not feel this phenomenon. 


Please help answer  
thanks
liujinhui  


965147871@qq.com

Re: deltastreamer group.id Noeffectaftersetting

[Vinoth Chandar <vi...@apache.org>]

DeltaStreamer actually just uses the same mechanism as Spark Streaming to
manage offsets. So wondering if you see the same behavior with a plain
spark streaming job. ?

It manages the offset checkpoints manually by itself within the hoodie
commit metadata, to do exactly once ingestion of data..

On Wed, Mar 18, 2020 at 3:07 AM 965147871@qq.com <96...@qq.com> wrote:

>
> hello, all When using deltastreamer to use kafka data, I want to specify
> group.id, but the problem encountered is that after specifying it, I
> cannot find it on the kafka side. I found that there are no groups under my
> theme. why is it like this? I also manually set enable.auto.commit = true
> at the same time, but it didn't seem to work. In kafkaUtils.scala
> fixKafkaParams this method,
>    kafkaParams.put (ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, false:
> java.lang.Boolean),
> which is forced to be rewritten as false. I think it is one of the reasons
> why the group cannot be found, but when it is not automatically submitted,
> it is usually chosen to be submitted manually. , I did not feel this
> phenomenon.
>
>
> Please help answer
> thanks
> liujinhui
>
>
> 965147871@qq.com
>