You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@beam.apache.org by Eduardo Soldera <ed...@arquivei.com.br> on 2018/09/11 19:09:58 UTC

Problem with KafkaIO

Hi,
We have a Apache Beam pipeline running in Google Dataflow using KafkaIO.
Suddenly the pipeline stop fetching Kafka messages at all, as our other
workers from other pipelines continued to get Kafka messages.

At the moment it stopped we got these messages:

I  [Consumer clientId=consumer-1, groupId=genericPipe] Error sending
fetch request (sessionId=1396189203, epoch=2431598) to node 3:
org.apache.kafka.common.errors.DisconnectException.
I  [Consumer clientId=consumer-1, groupId=genericPipe] Group
coordinator 10.0.52.70:9093 (id: 2147483646 rack: null) is unavailable
or invalid, will attempt rediscovery
I  [Consumer clientId=consumer-1, groupId=genericPipe] Discovered
group coordinator 10.0.52.70:9093 (id: 2147483646 rack: null)

And then the pipeline stopped reading the messages.

This is the KafkaIO setup  we have:

KafkaIO.read[String,String]()
  .withBootstrapServers(server)
  .withTopic(topic)
  .withKeyDeserializer(classOf[StringDeserializer])
  .withValueDeserializer(classOf[StringDeserializer])
  .updateConsumerProperties(properties)
  .commitOffsetsInFinalize()
  .withoutMetadata()

 Any help will be much appreciated.

Best regards,
-- 
Eduardo Soldera Garcia
Data Engineer
(16) 3509-5555 | www.arquivei.com.br
<https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
[image: Arquivei.com.br – Inteligência em Notas Fiscais]
<https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
[image: Google seleciona Arquivei para imersão e mentoria no Vale do
Silício]
<https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad>
<https://www.facebook.com/arquivei>
<https://www.linkedin.com/company/arquivei>
<https://www.youtube.com/watch?v=sSUUKxbXnxk>

Re: Problem with KafkaIO

Posted by Raghu Angadi <ra...@google.com>.
Thanks for confirming the fix.

On Mon, Oct 22, 2018 at 7:48 AM Eduardo Soldera <
eduardo.soldera@arquivei.com.br> wrote:

> Hi Raghu, Dataflow throws an exception if Kafka fails now and it recovers
> after Kafka is available.
>
> Regards
>
> Em sex, 19 de out de 2018 às 14:01, Raghu Angadi <ra...@google.com>
> escreveu:
>
>> On Fri, Oct 19, 2018 at 6:54 AM Eduardo Soldera <
>> eduardo.soldera@arquivei.com.br> wrote:
>>
>>> Hi Raghu, just a quick update. We were waiting for Spotify's Scio to
>>> update to Beam 2.7. We've just deployed the pipeling sucessfully. Just for
>>> letting you know, I tried to use the workaround code snipped, but Dataflow
>>> wouldn't recover after a Kafka unavailability.
>>>
>>
>> Thanks for the update. The workaround helps only if KafkaClient itself
>> can recover when try to read again. I guess some of those exceptions are
>> are not recoverable.
>>
>> Please let us know how the actual fix works.
>>
>> Thanks.
>> Raghu.
>>
>>
>>>
>>> Thanks for your help.
>>>
>>> Regards
>>>
>>> Em qua, 19 de set de 2018 às 15:37, Raghu Angadi <ra...@google.com>
>>> escreveu:
>>>
>>>>
>>>>
>>>> On Wed, Sep 19, 2018 at 11:24 AM Juan Carlos Garcia <
>>>> jcgarciam@gmail.com> wrote:
>>>>
>>>>> Sorry I hit the send button to fast... The error occurs in the worker.
>>>>>
>>>>
>>>> Np. Just one more comment on it: it is a very important
>>>> design/correctness decision to for runner to decide how to handle
>>>> persistent errors in a streaming pipeline. Dataflow keeps failing since
>>>> there is no solution to restart a pipeline from scratch without losing
>>>> exactly-once guarantees. It lets user decide if the pipeline needs to be
>>>> 'upgraded'.
>>>>
>>>> Raghu.
>>>>
>>>>>
>>>>> Juan Carlos Garcia <jc...@gmail.com> schrieb am Mi., 19. Sep.
>>>>> 2018, 20:22:
>>>>>
>>>>>> Sorry for hijacking the thread, we are running Spark on top of Yarn,
>>>>>> yarn retries multiple times until it reachs it max attempt and then gives
>>>>>> up.
>>>>>>
>>>>>> Raghu Angadi <ra...@google.com> schrieb am Mi., 19. Sep. 2018,
>>>>>> 18:58:
>>>>>>
>>>>>>> On Wed, Sep 19, 2018 at 7:26 AM Juan Carlos Garcia <
>>>>>>> jcgarciam@gmail.com> wrote:
>>>>>>>
>>>>>>>> Don't know if its related, but we have seen our pipeline dying
>>>>>>>> (using SparkRunner) when there is problem with Kafka  (network
>>>>>>>> interruptions), errors like:
>>>>>>>> org.apache.kafka.common.errors.TimeoutException: Timeout expired while
>>>>>>>> fetching topic metadata
>>>>>>>>
>>>>>>>> Maybe this will fix it as well, thanks Raghu for the hint about
>>>>>>>> *withConsumerFactoryFn.*
>>>>>>>>
>>>>>>>
>>>>>>> Wouldn't that be retried by the SparkRunner if it happens on the
>>>>>>> worker? or does it happen while launching the pipeline on the client?
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Wed, Sep 19, 2018 at 3:29 PM Eduardo Soldera <
>>>>>>>> eduardo.soldera@arquivei.com.br> wrote:
>>>>>>>>
>>>>>>>>> Hi Raghu, thank you.
>>>>>>>>>
>>>>>>>>> I'm not sure though what to pass as an argument:
>>>>>>>>>
>>>>>>>>> KafkaIO.read[String,String]()
>>>>>>>>>   .withBootstrapServers(server)
>>>>>>>>>   .withTopic(topic)
>>>>>>>>>   .withKeyDeserializer(classOf[StringDeserializer])
>>>>>>>>>   .withValueDeserializer(classOf[StringDeserializer])
>>>>>>>>>   .withConsumerFactoryFn(new KafkaExecutor.ConsumerFactoryFn(????????????????))
>>>>>>>>>   .updateConsumerProperties(properties)
>>>>>>>>>   .commitOffsetsInFinalize()
>>>>>>>>>   .withoutMetadata()
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Regards
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Em ter, 18 de set de 2018 às 21:15, Raghu Angadi <
>>>>>>>>> rangadi@google.com> escreveu:
>>>>>>>>>
>>>>>>>>>> Hi Eduardo,
>>>>>>>>>>
>>>>>>>>>> There another work around you can try without having to wait for
>>>>>>>>>> 2.7.0 release: Use a wrapper to catch exception from KafkaConsumer#poll()
>>>>>>>>>> and pass the wrapper to withConsumerFactoryFn() for KafkIO reader [1].
>>>>>>>>>>
>>>>>>>>>> Using something like (such a wrapper is used in KafkasIO tests
>>>>>>>>>> [2]):
>>>>>>>>>> private static class ConsumerFactoryFn
>>>>>>>>>>                 implements SerializableFunction<Map<String,
>>>>>>>>>> Object>, Consumer<byte[], byte[]>> {
>>>>>>>>>>   @Override
>>>>>>>>>>     public Consumer<byte[], byte[]> apply(Map<String, Object>
>>>>>>>>>> config) {
>>>>>>>>>>       return new KafkaConsumer(config) {
>>>>>>>>>>           @Override
>>>>>>>>>>           public ConsumerRecords<K, V> poll(long timeout) {
>>>>>>>>>>               // work around for BEAM-5375
>>>>>>>>>>               while (true) {
>>>>>>>>>>                   try {
>>>>>>>>>>                     return super.poll(timeout);
>>>>>>>>>>                  } catch (Exception e) {
>>>>>>>>>>                     // LOG & sleep for sec
>>>>>>>>>>                 }
>>>>>>>>>>           }
>>>>>>>>>>        }
>>>>>>>>>>     }
>>>>>>>>>> }
>>>>>>>>>>
>>>>>>>>>> [1]:
>>>>>>>>>> https://github.com/apache/beam/blob/release-2.4.0/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaIO.java#L417
>>>>>>>>>> [2]:
>>>>>>>>>> https://github.com/apache/beam/blob/release-2.4.0/sdks/java/io/kafka/src/test/java/org/apache/beam/sdk/io/kafka/KafkaIOTest.java#L261
>>>>>>>>>>
>>>>>>>>>> On Tue, Sep 18, 2018 at 5:49 AM Eduardo Soldera <
>>>>>>>>>> eduardo.soldera@arquivei.com.br> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi Raghu, we're not sure how long the network was down.
>>>>>>>>>>> According to the logs no longer than one minute. A 30 second shutdown would
>>>>>>>>>>> work for the tests.
>>>>>>>>>>>
>>>>>>>>>>> Regards
>>>>>>>>>>>
>>>>>>>>>>> Em sex, 14 de set de 2018 às 21:41, Raghu Angadi <
>>>>>>>>>>> rangadi@google.com> escreveu:
>>>>>>>>>>>
>>>>>>>>>>>> Thanks. I could repro myself as well. How long was the network
>>>>>>>>>>>> down?
>>>>>>>>>>>>
>>>>>>>>>>>> Trying to get the fix into 2.7 RC2.
>>>>>>>>>>>>
>>>>>>>>>>>> On Fri, Sep 14, 2018 at 12:25 PM Eduardo Soldera <
>>>>>>>>>>>> eduardo.soldera@arquivei.com.br> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Just to make myself clear, I'm not sure how to use the patch
>>>>>>>>>>>>> but if you could send us some guidance would be great.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Em sex, 14 de set de 2018 às 16:24, Eduardo Soldera <
>>>>>>>>>>>>> eduardo.soldera@arquivei.com.br> escreveu:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi Raghu, yes, it is feasible, would you do that for us? I'm
>>>>>>>>>>>>>> not sure how we'd use the patch. We're using SBT and Spotify's Scio with
>>>>>>>>>>>>>> Scala.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Em sex, 14 de set de 2018 às 16:07, Raghu Angadi <
>>>>>>>>>>>>>> rangadi@google.com> escreveu:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Is is feasible for you to verify the fix in your dev job? I
>>>>>>>>>>>>>>> can make a patch against Beam 2.4 branch if you like.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Raghu.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Fri, Sep 14, 2018 at 11:14 AM Eduardo Soldera <
>>>>>>>>>>>>>>> eduardo.soldera@arquivei.com.br> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hi Raghu, thank you very much for the pull request.
>>>>>>>>>>>>>>>> We'll wait for the 2.7 Beam release.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Regards!
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Em qui, 13 de set de 2018 às 18:19, Raghu Angadi <
>>>>>>>>>>>>>>>> rangadi@google.com> escreveu:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Fix: https://github.com/apache/beam/pull/6391
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Wed, Sep 12, 2018 at 3:30 PM Raghu Angadi <
>>>>>>>>>>>>>>>>> rangadi@google.com> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Filed BEAM-5375
>>>>>>>>>>>>>>>>>> <https://issues.apache.org/jira/browse/BEAM-5375>. I
>>>>>>>>>>>>>>>>>> will fix it later this week.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On Wed, Sep 12, 2018 at 12:16 PM Raghu Angadi <
>>>>>>>>>>>>>>>>>> rangadi@google.com> wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> On Wed, Sep 12, 2018 at 12:11 PM Raghu Angadi <
>>>>>>>>>>>>>>>>>>> rangadi@google.com> wrote:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Thanks for the job id, I looked at the worker logs
>>>>>>>>>>>>>>>>>>>> (following usual support oncall access protocol that provides temporary
>>>>>>>>>>>>>>>>>>>> access to things like logs in GCP):
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Root issue looks like consumerPollLoop() mentioned
>>>>>>>>>>>>>>>>>>>> earlier needs to handle unchecked exception. In your case it is clear that
>>>>>>>>>>>>>>>>>>>> poll thread exited with a runtime exception. The reader does not check for
>>>>>>>>>>>>>>>>>>>> it and continues to wait for poll thread to enqueue messages. A fix should
>>>>>>>>>>>>>>>>>>>> result in an IOException for read from the source. The runners will handle
>>>>>>>>>>>>>>>>>>>> that appropriately after that.  I will file a jira.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> https://github.com/apache/beam/blob/master/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaUnboundedReader.java#L678
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Ignore the link.. was pasted here by mistake.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> From the logs (with a comment below each one):
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>    - 2018-09-12 06:13:07.345 PDT Reader-0: reading
>>>>>>>>>>>>>>>>>>>>    from kafka_topic-0 starting at offset 2
>>>>>>>>>>>>>>>>>>>>       - Implies the reader is initialized and poll
>>>>>>>>>>>>>>>>>>>>       thread is started.
>>>>>>>>>>>>>>>>>>>>    - 2018-09-12 06:13:07.780 PDT Reader-0: first
>>>>>>>>>>>>>>>>>>>>    record offset 2
>>>>>>>>>>>>>>>>>>>>       - The reader actually got a message received by
>>>>>>>>>>>>>>>>>>>>       the poll thread from Kafka.
>>>>>>>>>>>>>>>>>>>>    - 2018-09-12 06:16:48.771 PDT Reader-0: exception
>>>>>>>>>>>>>>>>>>>>    while fetching latest offset for partition kafka_topic-0. will be retried.
>>>>>>>>>>>>>>>>>>>>       - This must have happened around the time when
>>>>>>>>>>>>>>>>>>>>       network was disrupted. This is from. Actual log is from another periodic
>>>>>>>>>>>>>>>>>>>>       task that fetches latest offsets for partitions.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> The poll thread must have died around the time network
>>>>>>>>>>>>>>>>>>>> was disrupted.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> The following log comes from kafka client itself and is
>>>>>>>>>>>>>>>>>>>> printed every second when KafkaIO fetches latest offset. This log seems to
>>>>>>>>>>>>>>>>>>>> be added in recent versions. It is probably an unintentional log. I don't
>>>>>>>>>>>>>>>>>>>> think there is any better to fetch latest offsets than how KafkaIO does
>>>>>>>>>>>>>>>>>>>> now. This is logged inside consumer.position() called at [1].
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>    - 2018-09-12 06:13:11.786 PDT [Consumer
>>>>>>>>>>>>>>>>>>>>    clientId=consumer-2,
>>>>>>>>>>>>>>>>>>>>    groupId=Reader-0_offset_consumer_1735388161_genericPipe] Resetting offset
>>>>>>>>>>>>>>>>>>>>    for partition com.arquivei.dataeng.andre-0 to offset 3.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> [1]:
>>>>>>>>>>>>>>>>>>>> https://github.com/apache/beam/blob/master/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaUnboundedReader.java#L678
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> This 'Resetting offset' is harmless, but is quite
>>>>>>>>>>>>>>>>>>> annoying to see in the worker logs. One way to avoid is to set kafka
>>>>>>>>>>>>>>>>>>> consumer's log level to WARNING. Ideally KafkaIO itself should do something
>>>>>>>>>>>>>>>>>>> to avoid it without user option.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> On Wed, Sep 12, 2018 at 10:27 AM Eduardo Soldera <
>>>>>>>>>>>>>>>>>>>> eduardo.soldera@arquivei.com.br> wrote:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Hi Raghu! The job_id of our dev job is
>>>>>>>>>>>>>>>>>>>>> 2018-09-12_06_11_48-5600553605191377866.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Thanks!
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Em qua, 12 de set de 2018 às 14:18, Raghu Angadi <
>>>>>>>>>>>>>>>>>>>>> rangadi@google.com> escreveu:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Thanks for debugging.
>>>>>>>>>>>>>>>>>>>>>> Can you provide the job_id of your dev job? The
>>>>>>>>>>>>>>>>>>>>>> stacktrace shows that there is no thread running 'consumerPollLoop()' which
>>>>>>>>>>>>>>>>>>>>>> can explain stuck reader. You will likely find a logs at line 594 & 587
>>>>>>>>>>>>>>>>>>>>>> [1].  Dataflow caches its readers and DirectRunner may not. That can
>>>>>>>>>>>>>>>>>>>>>> explain DirectRunner resume reads. The expectation in KafkaIO is that Kafka
>>>>>>>>>>>>>>>>>>>>>> client library takes care of retrying in case of connection problems (as
>>>>>>>>>>>>>>>>>>>>>> documented). It is possible that in some cases poll() throws and we need to
>>>>>>>>>>>>>>>>>>>>>> restart the client in KafkaIO.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> [1]:
>>>>>>>>>>>>>>>>>>>>>> https://github.com/apache/beam/blob/master/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaUnboundedReader.java#L594
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> On Wed, Sep 12, 2018 at 9:59 AM Eduardo Soldera <
>>>>>>>>>>>>>>>>>>>>>> eduardo.soldera@arquivei.com.br> wrote:
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Hi Raghu, thanks for your help.
>>>>>>>>>>>>>>>>>>>>>>> Just answering your previous question, the following
>>>>>>>>>>>>>>>>>>>>>>> logs were the same as before the error, as if the pipeline were still
>>>>>>>>>>>>>>>>>>>>>>> getting the messages, for example:
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> (...)
>>>>>>>>>>>>>>>>>>>>>>> Resetting offset for partition
>>>>>>>>>>>>>>>>>>>>>>> com.arquivei.dataeng.andre-0 to offset 10.
>>>>>>>>>>>>>>>>>>>>>>> Resetting offset for partition
>>>>>>>>>>>>>>>>>>>>>>> com.arquivei.dataeng.andre-0 to offset 15.
>>>>>>>>>>>>>>>>>>>>>>> ERROR
>>>>>>>>>>>>>>>>>>>>>>> Resetting offset for partition
>>>>>>>>>>>>>>>>>>>>>>> com.arquivei.dataeng.andre-0 to offset 22.
>>>>>>>>>>>>>>>>>>>>>>> Resetting offset for partition
>>>>>>>>>>>>>>>>>>>>>>> com.arquivei.dataeng.andre-0 to offset 30.
>>>>>>>>>>>>>>>>>>>>>>> (...)
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> But when checking the Kafka Consumer Group, the
>>>>>>>>>>>>>>>>>>>>>>> current offset stays at 15, the commited offset from the last processed
>>>>>>>>>>>>>>>>>>>>>>> message, before the error.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> We'll file a bug, but we could now reproduce the
>>>>>>>>>>>>>>>>>>>>>>> issue in a Dev scenario.
>>>>>>>>>>>>>>>>>>>>>>> We started the same pipeline using the direct
>>>>>>>>>>>>>>>>>>>>>>> runner, without Google Dataflow. We blocked the Kafka Broker network and
>>>>>>>>>>>>>>>>>>>>>>> the same error was thrown. Then we unblocked the network and the pipeline
>>>>>>>>>>>>>>>>>>>>>>> was able to successfully process the subsequent messages.
>>>>>>>>>>>>>>>>>>>>>>> When we started the same pipeline in the Dataflow
>>>>>>>>>>>>>>>>>>>>>>> runner and did the same test, the same problem from our production scenario
>>>>>>>>>>>>>>>>>>>>>>> happened, Dataflow couldn't process the new messages. Unfortunately, we've
>>>>>>>>>>>>>>>>>>>>>>> stopped the dataflow job in production, but the problematic dev job is
>>>>>>>>>>>>>>>>>>>>>>> still running and the log file of the VM is attached. Thank you very much.
>>>>>>>>>>>>>>>>>>>>>>> Best regards
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Em ter, 11 de set de 2018 às 18:28, Raghu Angadi <
>>>>>>>>>>>>>>>>>>>>>>> rangadi@google.com> escreveu:
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Specifically, I am interested if you have any
>>>>>>>>>>>>>>>>>>>>>>>> thread running 'consumerPollLoop()' [1]. There should always be one (if a
>>>>>>>>>>>>>>>>>>>>>>>> worker is assigned one of the partitions). It is possible that KafkaClient
>>>>>>>>>>>>>>>>>>>>>>>> itself is hasn't recovered from the group coordinator error (though
>>>>>>>>>>>>>>>>>>>>>>>> unlikely).
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> https://github.com/apache/beam/blob/master/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaUnboundedReader.java#L570
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Sep 11, 2018 at 12:31 PM Raghu Angadi <
>>>>>>>>>>>>>>>>>>>>>>>> rangadi@google.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Hi Eduardo,
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> In case of any error, the pipeline should keep on
>>>>>>>>>>>>>>>>>>>>>>>>> trying to fetch. I don't know about this particular error. Do you see any
>>>>>>>>>>>>>>>>>>>>>>>>> others afterwards in the log?
>>>>>>>>>>>>>>>>>>>>>>>>> Couple of things you could try if the logs are not
>>>>>>>>>>>>>>>>>>>>>>>>> useful :
>>>>>>>>>>>>>>>>>>>>>>>>>  - login to one of the VMs and get stacktrace of
>>>>>>>>>>>>>>>>>>>>>>>>> java worker (look for a container called java-streaming)
>>>>>>>>>>>>>>>>>>>>>>>>>  - file a support bug or stackoverflow question
>>>>>>>>>>>>>>>>>>>>>>>>> with jobid so that Dataflow oncall can take a look.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Raghu.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Sep 11, 2018 at 12:10 PM Eduardo Soldera <
>>>>>>>>>>>>>>>>>>>>>>>>> eduardo.soldera@arquivei.com.br> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>>>>>>>>>>> We have a Apache Beam pipeline running in Google
>>>>>>>>>>>>>>>>>>>>>>>>>> Dataflow using KafkaIO. Suddenly the pipeline stop fetching Kafka messages
>>>>>>>>>>>>>>>>>>>>>>>>>> at all, as our other workers from other pipelines continued to get Kafka
>>>>>>>>>>>>>>>>>>>>>>>>>> messages.
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> At the moment it stopped we got these messages:
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> I  [Consumer clientId=consumer-1, groupId=genericPipe] Error sending fetch request (sessionId=1396189203, epoch=2431598) to node 3: org.apache.kafka.common.errors.DisconnectException.
>>>>>>>>>>>>>>>>>>>>>>>>>> I  [Consumer clientId=consumer-1, groupId=genericPipe] Group coordinator 10.0.52.70:9093 (id: 2147483646 rack: null) is unavailable or invalid, will attempt rediscovery
>>>>>>>>>>>>>>>>>>>>>>>>>> I  [Consumer clientId=consumer-1, groupId=genericPipe] Discovered group coordinator 10.0.52.70:9093 (id: 2147483646 rack: null)
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> And then the pipeline stopped reading the
>>>>>>>>>>>>>>>>>>>>>>>>>> messages.
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> This is the KafkaIO setup  we have:
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> KafkaIO.read[String,String]()
>>>>>>>>>>>>>>>>>>>>>>>>>>   .withBootstrapServers(server)
>>>>>>>>>>>>>>>>>>>>>>>>>>   .withTopic(topic)
>>>>>>>>>>>>>>>>>>>>>>>>>>   .withKeyDeserializer(classOf[StringDeserializer])
>>>>>>>>>>>>>>>>>>>>>>>>>>   .withValueDeserializer(classOf[StringDeserializer])
>>>>>>>>>>>>>>>>>>>>>>>>>>   .updateConsumerProperties(properties)
>>>>>>>>>>>>>>>>>>>>>>>>>>   .commitOffsetsInFinalize()
>>>>>>>>>>>>>>>>>>>>>>>>>>   .withoutMetadata()
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>  Any help will be much appreciated.
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> Best regards,
>>>>>>>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>>>>>> Eduardo Soldera Garcia
>>>>>>>>>>>>>>>>>>>>>>>>>> Data Engineer
>>>>>>>>>>>>>>>>>>>>>>>>>> (16) 3509-5555 | www.arquivei.com.br
>>>>>>>>>>>>>>>>>>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>>>>>>>>>>>>>>>>>>>> [image: Arquivei.com.br – Inteligência em Notas
>>>>>>>>>>>>>>>>>>>>>>>>>> Fiscais]
>>>>>>>>>>>>>>>>>>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>>>>>>>>>>>>>>>>>>>> [image: Google seleciona Arquivei para imersão e
>>>>>>>>>>>>>>>>>>>>>>>>>> mentoria no Vale do Silício]
>>>>>>>>>>>>>>>>>>>>>>>>>> <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad>
>>>>>>>>>>>>>>>>>>>>>>>>>> <https://www.facebook.com/arquivei>
>>>>>>>>>>>>>>>>>>>>>>>>>> <https://www.linkedin.com/company/arquivei>
>>>>>>>>>>>>>>>>>>>>>>>>>> <https://www.youtube.com/watch?v=sSUUKxbXnxk>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>>> Eduardo Soldera Garcia
>>>>>>>>>>>>>>>>>>>>>>> Data Engineer
>>>>>>>>>>>>>>>>>>>>>>> (16) 3509-5555 | www.arquivei.com.br
>>>>>>>>>>>>>>>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>>>>>>>>>>>>>>>>> [image: Arquivei.com.br – Inteligência em Notas
>>>>>>>>>>>>>>>>>>>>>>> Fiscais]
>>>>>>>>>>>>>>>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>>>>>>>>>>>>>>>>> [image: Google seleciona Arquivei para imersão e
>>>>>>>>>>>>>>>>>>>>>>> mentoria no Vale do Silício]
>>>>>>>>>>>>>>>>>>>>>>> <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad>
>>>>>>>>>>>>>>>>>>>>>>> <https://www.facebook.com/arquivei>
>>>>>>>>>>>>>>>>>>>>>>> <https://www.linkedin.com/company/arquivei>
>>>>>>>>>>>>>>>>>>>>>>> <https://www.youtube.com/watch?v=sSUUKxbXnxk>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>> Eduardo Soldera Garcia
>>>>>>>>>>>>>>>>>>>>> Data Engineer
>>>>>>>>>>>>>>>>>>>>> (16) 3509-5555 | www.arquivei.com.br
>>>>>>>>>>>>>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>>>>>>>>>>>>>>> [image: Arquivei.com.br – Inteligência em Notas
>>>>>>>>>>>>>>>>>>>>> Fiscais]
>>>>>>>>>>>>>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>>>>>>>>>>>>>>> [image: Google seleciona Arquivei para imersão e
>>>>>>>>>>>>>>>>>>>>> mentoria no Vale do Silício]
>>>>>>>>>>>>>>>>>>>>> <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad>
>>>>>>>>>>>>>>>>>>>>> <https://www.facebook.com/arquivei>
>>>>>>>>>>>>>>>>>>>>> <https://www.linkedin.com/company/arquivei>
>>>>>>>>>>>>>>>>>>>>> <https://www.youtube.com/watch?v=sSUUKxbXnxk>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>> Eduardo Soldera Garcia
>>>>>>>>>>>>>>>> Data Engineer
>>>>>>>>>>>>>>>> (16) 3509-5555 | www.arquivei.com.br
>>>>>>>>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>>>>>>>>>> [image: Arquivei.com.br – Inteligência em Notas Fiscais]
>>>>>>>>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>>>>>>>>>> [image: Google seleciona Arquivei para imersão e mentoria
>>>>>>>>>>>>>>>> no Vale do Silício]
>>>>>>>>>>>>>>>> <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad>
>>>>>>>>>>>>>>>> <https://www.facebook.com/arquivei>
>>>>>>>>>>>>>>>> <https://www.linkedin.com/company/arquivei>
>>>>>>>>>>>>>>>> <https://www.youtube.com/watch?v=sSUUKxbXnxk>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> --
>>>>>>>>>>>>>> Eduardo Soldera Garcia
>>>>>>>>>>>>>> Data Engineer
>>>>>>>>>>>>>> (16) 3509-5555 | www.arquivei.com.br
>>>>>>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>>>>>>>> [image: Arquivei.com.br – Inteligência em Notas Fiscais]
>>>>>>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>>>>>>>> [image: Google seleciona Arquivei para imersão e mentoria no
>>>>>>>>>>>>>> Vale do Silício]
>>>>>>>>>>>>>> <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad>
>>>>>>>>>>>>>> <https://www.facebook.com/arquivei>
>>>>>>>>>>>>>> <https://www.linkedin.com/company/arquivei>
>>>>>>>>>>>>>> <https://www.youtube.com/watch?v=sSUUKxbXnxk>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>>> Eduardo Soldera Garcia
>>>>>>>>>>>>> Data Engineer
>>>>>>>>>>>>> (16) 3509-5555 | www.arquivei.com.br
>>>>>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>>>>>>> [image: Arquivei.com.br – Inteligência em Notas Fiscais]
>>>>>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>>>>>>> [image: Google seleciona Arquivei para imersão e mentoria no
>>>>>>>>>>>>> Vale do Silício]
>>>>>>>>>>>>> <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad>
>>>>>>>>>>>>> <https://www.facebook.com/arquivei>
>>>>>>>>>>>>> <https://www.linkedin.com/company/arquivei>
>>>>>>>>>>>>> <https://www.youtube.com/watch?v=sSUUKxbXnxk>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Eduardo Soldera Garcia
>>>>>>>>>>> Data Engineer
>>>>>>>>>>> (16) 3509-5555 | www.arquivei.com.br
>>>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>>>>> [image: Arquivei.com.br – Inteligência em Notas Fiscais]
>>>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>>>>> [image: Google seleciona Arquivei para imersão e mentoria no
>>>>>>>>>>> Vale do Silício]
>>>>>>>>>>> <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad>
>>>>>>>>>>> <https://www.facebook.com/arquivei>
>>>>>>>>>>> <https://www.linkedin.com/company/arquivei>
>>>>>>>>>>> <https://www.youtube.com/watch?v=sSUUKxbXnxk>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Eduardo Soldera Garcia
>>>>>>>>> Data Engineer
>>>>>>>>> (16) 3509-5555 | www.arquivei.com.br
>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>>> [image: Arquivei.com.br – Inteligência em Notas Fiscais]
>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>>> [image: Google seleciona Arquivei para imersão e mentoria no Vale
>>>>>>>>> do Silício]
>>>>>>>>> <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad>
>>>>>>>>> <https://www.facebook.com/arquivei>
>>>>>>>>> <https://www.linkedin.com/company/arquivei>
>>>>>>>>> <https://www.youtube.com/watch?v=sSUUKxbXnxk>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>>
>>>>>>>> JC
>>>>>>>>
>>>>>>>>
>>>
>>> --
>>> Eduardo Soldera Garcia
>>> Data Engineer
>>> (16) 3509-5555 | www.arquivei.com.br
>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>> [image: Arquivei.com.br – Inteligência em Notas Fiscais]
>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>> [image: Google seleciona Arquivei para imersão e mentoria no Vale do
>>> Silício]
>>> <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad>
>>> <https://www.facebook.com/arquivei>
>>> <https://www.linkedin.com/company/arquivei>
>>> <https://www.youtube.com/watch?v=sSUUKxbXnxk>
>>>
>>
>
> --
> Eduardo Soldera Garcia
> Data Engineer
> (16) 3509-5555 | www.arquivei.com.br
> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
> [image: Arquivei.com.br – Inteligência em Notas Fiscais]
> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
> [image: Google seleciona Arquivei para imersão e mentoria no Vale do
> Silício]
> <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad>
> <https://www.facebook.com/arquivei>
> <https://www.linkedin.com/company/arquivei>
> <https://www.youtube.com/watch?v=sSUUKxbXnxk>
>

Re: Problem with KafkaIO

Posted by Eduardo Soldera <ed...@arquivei.com.br>.
Hi Raghu, Dataflow throws an exception if Kafka fails now and it recovers
after Kafka is available.

Regards

Em sex, 19 de out de 2018 às 14:01, Raghu Angadi <ra...@google.com>
escreveu:

> On Fri, Oct 19, 2018 at 6:54 AM Eduardo Soldera <
> eduardo.soldera@arquivei.com.br> wrote:
>
>> Hi Raghu, just a quick update. We were waiting for Spotify's Scio to
>> update to Beam 2.7. We've just deployed the pipeling sucessfully. Just for
>> letting you know, I tried to use the workaround code snipped, but Dataflow
>> wouldn't recover after a Kafka unavailability.
>>
>
> Thanks for the update. The workaround helps only if KafkaClient itself can
> recover when try to read again. I guess some of those exceptions are are
> not recoverable.
>
> Please let us know how the actual fix works.
>
> Thanks.
> Raghu.
>
>
>>
>> Thanks for your help.
>>
>> Regards
>>
>> Em qua, 19 de set de 2018 às 15:37, Raghu Angadi <ra...@google.com>
>> escreveu:
>>
>>>
>>>
>>> On Wed, Sep 19, 2018 at 11:24 AM Juan Carlos Garcia <jc...@gmail.com>
>>> wrote:
>>>
>>>> Sorry I hit the send button to fast... The error occurs in the worker.
>>>>
>>>
>>> Np. Just one more comment on it: it is a very important
>>> design/correctness decision to for runner to decide how to handle
>>> persistent errors in a streaming pipeline. Dataflow keeps failing since
>>> there is no solution to restart a pipeline from scratch without losing
>>> exactly-once guarantees. It lets user decide if the pipeline needs to be
>>> 'upgraded'.
>>>
>>> Raghu.
>>>
>>>>
>>>> Juan Carlos Garcia <jc...@gmail.com> schrieb am Mi., 19. Sep.
>>>> 2018, 20:22:
>>>>
>>>>> Sorry for hijacking the thread, we are running Spark on top of Yarn,
>>>>> yarn retries multiple times until it reachs it max attempt and then gives
>>>>> up.
>>>>>
>>>>> Raghu Angadi <ra...@google.com> schrieb am Mi., 19. Sep. 2018,
>>>>> 18:58:
>>>>>
>>>>>> On Wed, Sep 19, 2018 at 7:26 AM Juan Carlos Garcia <
>>>>>> jcgarciam@gmail.com> wrote:
>>>>>>
>>>>>>> Don't know if its related, but we have seen our pipeline dying
>>>>>>> (using SparkRunner) when there is problem with Kafka  (network
>>>>>>> interruptions), errors like:
>>>>>>> org.apache.kafka.common.errors.TimeoutException: Timeout expired while
>>>>>>> fetching topic metadata
>>>>>>>
>>>>>>> Maybe this will fix it as well, thanks Raghu for the hint about
>>>>>>> *withConsumerFactoryFn.*
>>>>>>>
>>>>>>
>>>>>> Wouldn't that be retried by the SparkRunner if it happens on the
>>>>>> worker? or does it happen while launching the pipeline on the client?
>>>>>>
>>>>>>
>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Wed, Sep 19, 2018 at 3:29 PM Eduardo Soldera <
>>>>>>> eduardo.soldera@arquivei.com.br> wrote:
>>>>>>>
>>>>>>>> Hi Raghu, thank you.
>>>>>>>>
>>>>>>>> I'm not sure though what to pass as an argument:
>>>>>>>>
>>>>>>>> KafkaIO.read[String,String]()
>>>>>>>>   .withBootstrapServers(server)
>>>>>>>>   .withTopic(topic)
>>>>>>>>   .withKeyDeserializer(classOf[StringDeserializer])
>>>>>>>>   .withValueDeserializer(classOf[StringDeserializer])
>>>>>>>>   .withConsumerFactoryFn(new KafkaExecutor.ConsumerFactoryFn(????????????????))
>>>>>>>>   .updateConsumerProperties(properties)
>>>>>>>>   .commitOffsetsInFinalize()
>>>>>>>>   .withoutMetadata()
>>>>>>>>
>>>>>>>>
>>>>>>>> Regards
>>>>>>>>
>>>>>>>>
>>>>>>>> Em ter, 18 de set de 2018 às 21:15, Raghu Angadi <
>>>>>>>> rangadi@google.com> escreveu:
>>>>>>>>
>>>>>>>>> Hi Eduardo,
>>>>>>>>>
>>>>>>>>> There another work around you can try without having to wait for
>>>>>>>>> 2.7.0 release: Use a wrapper to catch exception from KafkaConsumer#poll()
>>>>>>>>> and pass the wrapper to withConsumerFactoryFn() for KafkIO reader [1].
>>>>>>>>>
>>>>>>>>> Using something like (such a wrapper is used in KafkasIO tests
>>>>>>>>> [2]):
>>>>>>>>> private static class ConsumerFactoryFn
>>>>>>>>>                 implements SerializableFunction<Map<String,
>>>>>>>>> Object>, Consumer<byte[], byte[]>> {
>>>>>>>>>   @Override
>>>>>>>>>     public Consumer<byte[], byte[]> apply(Map<String, Object>
>>>>>>>>> config) {
>>>>>>>>>       return new KafkaConsumer(config) {
>>>>>>>>>           @Override
>>>>>>>>>           public ConsumerRecords<K, V> poll(long timeout) {
>>>>>>>>>               // work around for BEAM-5375
>>>>>>>>>               while (true) {
>>>>>>>>>                   try {
>>>>>>>>>                     return super.poll(timeout);
>>>>>>>>>                  } catch (Exception e) {
>>>>>>>>>                     // LOG & sleep for sec
>>>>>>>>>                 }
>>>>>>>>>           }
>>>>>>>>>        }
>>>>>>>>>     }
>>>>>>>>> }
>>>>>>>>>
>>>>>>>>> [1]:
>>>>>>>>> https://github.com/apache/beam/blob/release-2.4.0/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaIO.java#L417
>>>>>>>>> [2]:
>>>>>>>>> https://github.com/apache/beam/blob/release-2.4.0/sdks/java/io/kafka/src/test/java/org/apache/beam/sdk/io/kafka/KafkaIOTest.java#L261
>>>>>>>>>
>>>>>>>>> On Tue, Sep 18, 2018 at 5:49 AM Eduardo Soldera <
>>>>>>>>> eduardo.soldera@arquivei.com.br> wrote:
>>>>>>>>>
>>>>>>>>>> Hi Raghu, we're not sure how long the network was down. According
>>>>>>>>>> to the logs no longer than one minute. A 30 second shutdown would work for
>>>>>>>>>> the tests.
>>>>>>>>>>
>>>>>>>>>> Regards
>>>>>>>>>>
>>>>>>>>>> Em sex, 14 de set de 2018 às 21:41, Raghu Angadi <
>>>>>>>>>> rangadi@google.com> escreveu:
>>>>>>>>>>
>>>>>>>>>>> Thanks. I could repro myself as well. How long was the network
>>>>>>>>>>> down?
>>>>>>>>>>>
>>>>>>>>>>> Trying to get the fix into 2.7 RC2.
>>>>>>>>>>>
>>>>>>>>>>> On Fri, Sep 14, 2018 at 12:25 PM Eduardo Soldera <
>>>>>>>>>>> eduardo.soldera@arquivei.com.br> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Just to make myself clear, I'm not sure how to use the patch
>>>>>>>>>>>> but if you could send us some guidance would be great.
>>>>>>>>>>>>
>>>>>>>>>>>> Em sex, 14 de set de 2018 às 16:24, Eduardo Soldera <
>>>>>>>>>>>> eduardo.soldera@arquivei.com.br> escreveu:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi Raghu, yes, it is feasible, would you do that for us? I'm
>>>>>>>>>>>>> not sure how we'd use the patch. We're using SBT and Spotify's Scio with
>>>>>>>>>>>>> Scala.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks
>>>>>>>>>>>>>
>>>>>>>>>>>>> Em sex, 14 de set de 2018 às 16:07, Raghu Angadi <
>>>>>>>>>>>>> rangadi@google.com> escreveu:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Is is feasible for you to verify the fix in your dev job? I
>>>>>>>>>>>>>> can make a patch against Beam 2.4 branch if you like.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Raghu.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Fri, Sep 14, 2018 at 11:14 AM Eduardo Soldera <
>>>>>>>>>>>>>> eduardo.soldera@arquivei.com.br> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi Raghu, thank you very much for the pull request.
>>>>>>>>>>>>>>> We'll wait for the 2.7 Beam release.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Regards!
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Em qui, 13 de set de 2018 às 18:19, Raghu Angadi <
>>>>>>>>>>>>>>> rangadi@google.com> escreveu:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Fix: https://github.com/apache/beam/pull/6391
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Wed, Sep 12, 2018 at 3:30 PM Raghu Angadi <
>>>>>>>>>>>>>>>> rangadi@google.com> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Filed BEAM-5375
>>>>>>>>>>>>>>>>> <https://issues.apache.org/jira/browse/BEAM-5375>. I will
>>>>>>>>>>>>>>>>> fix it later this week.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Wed, Sep 12, 2018 at 12:16 PM Raghu Angadi <
>>>>>>>>>>>>>>>>> rangadi@google.com> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On Wed, Sep 12, 2018 at 12:11 PM Raghu Angadi <
>>>>>>>>>>>>>>>>>> rangadi@google.com> wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Thanks for the job id, I looked at the worker logs
>>>>>>>>>>>>>>>>>>> (following usual support oncall access protocol that provides temporary
>>>>>>>>>>>>>>>>>>> access to things like logs in GCP):
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Root issue looks like consumerPollLoop() mentioned
>>>>>>>>>>>>>>>>>>> earlier needs to handle unchecked exception. In your case it is clear that
>>>>>>>>>>>>>>>>>>> poll thread exited with a runtime exception. The reader does not check for
>>>>>>>>>>>>>>>>>>> it and continues to wait for poll thread to enqueue messages. A fix should
>>>>>>>>>>>>>>>>>>> result in an IOException for read from the source. The runners will handle
>>>>>>>>>>>>>>>>>>> that appropriately after that.  I will file a jira.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> https://github.com/apache/beam/blob/master/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaUnboundedReader.java#L678
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Ignore the link.. was pasted here by mistake.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> From the logs (with a comment below each one):
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>    - 2018-09-12 06:13:07.345 PDT Reader-0: reading from
>>>>>>>>>>>>>>>>>>>    kafka_topic-0 starting at offset 2
>>>>>>>>>>>>>>>>>>>       - Implies the reader is initialized and poll
>>>>>>>>>>>>>>>>>>>       thread is started.
>>>>>>>>>>>>>>>>>>>    - 2018-09-12 06:13:07.780 PDT Reader-0: first record
>>>>>>>>>>>>>>>>>>>    offset 2
>>>>>>>>>>>>>>>>>>>       - The reader actually got a message received by
>>>>>>>>>>>>>>>>>>>       the poll thread from Kafka.
>>>>>>>>>>>>>>>>>>>    - 2018-09-12 06:16:48.771 PDT Reader-0: exception
>>>>>>>>>>>>>>>>>>>    while fetching latest offset for partition kafka_topic-0. will be retried.
>>>>>>>>>>>>>>>>>>>       - This must have happened around the time when
>>>>>>>>>>>>>>>>>>>       network was disrupted. This is from. Actual log is from another periodic
>>>>>>>>>>>>>>>>>>>       task that fetches latest offsets for partitions.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> The poll thread must have died around the time network
>>>>>>>>>>>>>>>>>>> was disrupted.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> The following log comes from kafka client itself and is
>>>>>>>>>>>>>>>>>>> printed every second when KafkaIO fetches latest offset. This log seems to
>>>>>>>>>>>>>>>>>>> be added in recent versions. It is probably an unintentional log. I don't
>>>>>>>>>>>>>>>>>>> think there is any better to fetch latest offsets than how KafkaIO does
>>>>>>>>>>>>>>>>>>> now. This is logged inside consumer.position() called at [1].
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>    - 2018-09-12 06:13:11.786 PDT [Consumer
>>>>>>>>>>>>>>>>>>>    clientId=consumer-2,
>>>>>>>>>>>>>>>>>>>    groupId=Reader-0_offset_consumer_1735388161_genericPipe] Resetting offset
>>>>>>>>>>>>>>>>>>>    for partition com.arquivei.dataeng.andre-0 to offset 3.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> [1]:
>>>>>>>>>>>>>>>>>>> https://github.com/apache/beam/blob/master/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaUnboundedReader.java#L678
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> This 'Resetting offset' is harmless, but is quite
>>>>>>>>>>>>>>>>>> annoying to see in the worker logs. One way to avoid is to set kafka
>>>>>>>>>>>>>>>>>> consumer's log level to WARNING. Ideally KafkaIO itself should do something
>>>>>>>>>>>>>>>>>> to avoid it without user option.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> On Wed, Sep 12, 2018 at 10:27 AM Eduardo Soldera <
>>>>>>>>>>>>>>>>>>> eduardo.soldera@arquivei.com.br> wrote:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Hi Raghu! The job_id of our dev job is
>>>>>>>>>>>>>>>>>>>> 2018-09-12_06_11_48-5600553605191377866.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Thanks!
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Em qua, 12 de set de 2018 às 14:18, Raghu Angadi <
>>>>>>>>>>>>>>>>>>>> rangadi@google.com> escreveu:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Thanks for debugging.
>>>>>>>>>>>>>>>>>>>>> Can you provide the job_id of your dev job? The
>>>>>>>>>>>>>>>>>>>>> stacktrace shows that there is no thread running 'consumerPollLoop()' which
>>>>>>>>>>>>>>>>>>>>> can explain stuck reader. You will likely find a logs at line 594 & 587
>>>>>>>>>>>>>>>>>>>>> [1].  Dataflow caches its readers and DirectRunner may not. That can
>>>>>>>>>>>>>>>>>>>>> explain DirectRunner resume reads. The expectation in KafkaIO is that Kafka
>>>>>>>>>>>>>>>>>>>>> client library takes care of retrying in case of connection problems (as
>>>>>>>>>>>>>>>>>>>>> documented). It is possible that in some cases poll() throws and we need to
>>>>>>>>>>>>>>>>>>>>> restart the client in KafkaIO.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> [1]:
>>>>>>>>>>>>>>>>>>>>> https://github.com/apache/beam/blob/master/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaUnboundedReader.java#L594
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> On Wed, Sep 12, 2018 at 9:59 AM Eduardo Soldera <
>>>>>>>>>>>>>>>>>>>>> eduardo.soldera@arquivei.com.br> wrote:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Hi Raghu, thanks for your help.
>>>>>>>>>>>>>>>>>>>>>> Just answering your previous question, the following
>>>>>>>>>>>>>>>>>>>>>> logs were the same as before the error, as if the pipeline were still
>>>>>>>>>>>>>>>>>>>>>> getting the messages, for example:
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> (...)
>>>>>>>>>>>>>>>>>>>>>> Resetting offset for partition
>>>>>>>>>>>>>>>>>>>>>> com.arquivei.dataeng.andre-0 to offset 10.
>>>>>>>>>>>>>>>>>>>>>> Resetting offset for partition
>>>>>>>>>>>>>>>>>>>>>> com.arquivei.dataeng.andre-0 to offset 15.
>>>>>>>>>>>>>>>>>>>>>> ERROR
>>>>>>>>>>>>>>>>>>>>>> Resetting offset for partition
>>>>>>>>>>>>>>>>>>>>>> com.arquivei.dataeng.andre-0 to offset 22.
>>>>>>>>>>>>>>>>>>>>>> Resetting offset for partition
>>>>>>>>>>>>>>>>>>>>>> com.arquivei.dataeng.andre-0 to offset 30.
>>>>>>>>>>>>>>>>>>>>>> (...)
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> But when checking the Kafka Consumer Group, the
>>>>>>>>>>>>>>>>>>>>>> current offset stays at 15, the commited offset from the last processed
>>>>>>>>>>>>>>>>>>>>>> message, before the error.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> We'll file a bug, but we could now reproduce the
>>>>>>>>>>>>>>>>>>>>>> issue in a Dev scenario.
>>>>>>>>>>>>>>>>>>>>>> We started the same pipeline using the direct runner,
>>>>>>>>>>>>>>>>>>>>>> without Google Dataflow. We blocked the Kafka Broker network and the same
>>>>>>>>>>>>>>>>>>>>>> error was thrown. Then we unblocked the network and the pipeline was able
>>>>>>>>>>>>>>>>>>>>>> to successfully process the subsequent messages.
>>>>>>>>>>>>>>>>>>>>>> When we started the same pipeline in the Dataflow
>>>>>>>>>>>>>>>>>>>>>> runner and did the same test, the same problem from our production scenario
>>>>>>>>>>>>>>>>>>>>>> happened, Dataflow couldn't process the new messages. Unfortunately, we've
>>>>>>>>>>>>>>>>>>>>>> stopped the dataflow job in production, but the problematic dev job is
>>>>>>>>>>>>>>>>>>>>>> still running and the log file of the VM is attached. Thank you very much.
>>>>>>>>>>>>>>>>>>>>>> Best regards
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Em ter, 11 de set de 2018 às 18:28, Raghu Angadi <
>>>>>>>>>>>>>>>>>>>>>> rangadi@google.com> escreveu:
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Specifically, I am interested if you have any thread
>>>>>>>>>>>>>>>>>>>>>>> running 'consumerPollLoop()' [1]. There should always be one (if a worker
>>>>>>>>>>>>>>>>>>>>>>> is assigned one of the partitions). It is possible that KafkaClient itself
>>>>>>>>>>>>>>>>>>>>>>> is hasn't recovered from the group coordinator error (though unlikely).
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> https://github.com/apache/beam/blob/master/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaUnboundedReader.java#L570
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> On Tue, Sep 11, 2018 at 12:31 PM Raghu Angadi <
>>>>>>>>>>>>>>>>>>>>>>> rangadi@google.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Hi Eduardo,
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> In case of any error, the pipeline should keep on
>>>>>>>>>>>>>>>>>>>>>>>> trying to fetch. I don't know about this particular error. Do you see any
>>>>>>>>>>>>>>>>>>>>>>>> others afterwards in the log?
>>>>>>>>>>>>>>>>>>>>>>>> Couple of things you could try if the logs are not
>>>>>>>>>>>>>>>>>>>>>>>> useful :
>>>>>>>>>>>>>>>>>>>>>>>>  - login to one of the VMs and get stacktrace of
>>>>>>>>>>>>>>>>>>>>>>>> java worker (look for a container called java-streaming)
>>>>>>>>>>>>>>>>>>>>>>>>  - file a support bug or stackoverflow question
>>>>>>>>>>>>>>>>>>>>>>>> with jobid so that Dataflow oncall can take a look.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Raghu.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Sep 11, 2018 at 12:10 PM Eduardo Soldera <
>>>>>>>>>>>>>>>>>>>>>>>> eduardo.soldera@arquivei.com.br> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>>>>>>>>>> We have a Apache Beam pipeline running in Google
>>>>>>>>>>>>>>>>>>>>>>>>> Dataflow using KafkaIO. Suddenly the pipeline stop fetching Kafka messages
>>>>>>>>>>>>>>>>>>>>>>>>> at all, as our other workers from other pipelines continued to get Kafka
>>>>>>>>>>>>>>>>>>>>>>>>> messages.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> At the moment it stopped we got these messages:
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> I  [Consumer clientId=consumer-1, groupId=genericPipe] Error sending fetch request (sessionId=1396189203, epoch=2431598) to node 3: org.apache.kafka.common.errors.DisconnectException.
>>>>>>>>>>>>>>>>>>>>>>>>> I  [Consumer clientId=consumer-1, groupId=genericPipe] Group coordinator 10.0.52.70:9093 (id: 2147483646 rack: null) is unavailable or invalid, will attempt rediscovery
>>>>>>>>>>>>>>>>>>>>>>>>> I  [Consumer clientId=consumer-1, groupId=genericPipe] Discovered group coordinator 10.0.52.70:9093 (id: 2147483646 rack: null)
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> And then the pipeline stopped reading the messages.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> This is the KafkaIO setup  we have:
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> KafkaIO.read[String,String]()
>>>>>>>>>>>>>>>>>>>>>>>>>   .withBootstrapServers(server)
>>>>>>>>>>>>>>>>>>>>>>>>>   .withTopic(topic)
>>>>>>>>>>>>>>>>>>>>>>>>>   .withKeyDeserializer(classOf[StringDeserializer])
>>>>>>>>>>>>>>>>>>>>>>>>>   .withValueDeserializer(classOf[StringDeserializer])
>>>>>>>>>>>>>>>>>>>>>>>>>   .updateConsumerProperties(properties)
>>>>>>>>>>>>>>>>>>>>>>>>>   .commitOffsetsInFinalize()
>>>>>>>>>>>>>>>>>>>>>>>>>   .withoutMetadata()
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>  Any help will be much appreciated.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Best regards,
>>>>>>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>>>>> Eduardo Soldera Garcia
>>>>>>>>>>>>>>>>>>>>>>>>> Data Engineer
>>>>>>>>>>>>>>>>>>>>>>>>> (16) 3509-5555 | www.arquivei.com.br
>>>>>>>>>>>>>>>>>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>>>>>>>>>>>>>>>>>>> [image: Arquivei.com.br – Inteligência em Notas
>>>>>>>>>>>>>>>>>>>>>>>>> Fiscais]
>>>>>>>>>>>>>>>>>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>>>>>>>>>>>>>>>>>>> [image: Google seleciona Arquivei para imersão e
>>>>>>>>>>>>>>>>>>>>>>>>> mentoria no Vale do Silício]
>>>>>>>>>>>>>>>>>>>>>>>>> <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad>
>>>>>>>>>>>>>>>>>>>>>>>>> <https://www.facebook.com/arquivei>
>>>>>>>>>>>>>>>>>>>>>>>>> <https://www.linkedin.com/company/arquivei>
>>>>>>>>>>>>>>>>>>>>>>>>> <https://www.youtube.com/watch?v=sSUUKxbXnxk>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>> Eduardo Soldera Garcia
>>>>>>>>>>>>>>>>>>>>>> Data Engineer
>>>>>>>>>>>>>>>>>>>>>> (16) 3509-5555 | www.arquivei.com.br
>>>>>>>>>>>>>>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>>>>>>>>>>>>>>>> [image: Arquivei.com.br – Inteligência em Notas
>>>>>>>>>>>>>>>>>>>>>> Fiscais]
>>>>>>>>>>>>>>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>>>>>>>>>>>>>>>> [image: Google seleciona Arquivei para imersão e
>>>>>>>>>>>>>>>>>>>>>> mentoria no Vale do Silício]
>>>>>>>>>>>>>>>>>>>>>> <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad>
>>>>>>>>>>>>>>>>>>>>>> <https://www.facebook.com/arquivei>
>>>>>>>>>>>>>>>>>>>>>> <https://www.linkedin.com/company/arquivei>
>>>>>>>>>>>>>>>>>>>>>> <https://www.youtube.com/watch?v=sSUUKxbXnxk>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>> Eduardo Soldera Garcia
>>>>>>>>>>>>>>>>>>>> Data Engineer
>>>>>>>>>>>>>>>>>>>> (16) 3509-5555 | www.arquivei.com.br
>>>>>>>>>>>>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>>>>>>>>>>>>>> [image: Arquivei.com.br – Inteligência em Notas Fiscais]
>>>>>>>>>>>>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>>>>>>>>>>>>>> [image: Google seleciona Arquivei para imersão e
>>>>>>>>>>>>>>>>>>>> mentoria no Vale do Silício]
>>>>>>>>>>>>>>>>>>>> <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad>
>>>>>>>>>>>>>>>>>>>> <https://www.facebook.com/arquivei>
>>>>>>>>>>>>>>>>>>>> <https://www.linkedin.com/company/arquivei>
>>>>>>>>>>>>>>>>>>>> <https://www.youtube.com/watch?v=sSUUKxbXnxk>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>> Eduardo Soldera Garcia
>>>>>>>>>>>>>>> Data Engineer
>>>>>>>>>>>>>>> (16) 3509-5555 | www.arquivei.com.br
>>>>>>>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>>>>>>>>> [image: Arquivei.com.br – Inteligência em Notas Fiscais]
>>>>>>>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>>>>>>>>> [image: Google seleciona Arquivei para imersão e mentoria no
>>>>>>>>>>>>>>> Vale do Silício]
>>>>>>>>>>>>>>> <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad>
>>>>>>>>>>>>>>> <https://www.facebook.com/arquivei>
>>>>>>>>>>>>>>> <https://www.linkedin.com/company/arquivei>
>>>>>>>>>>>>>>> <https://www.youtube.com/watch?v=sSUUKxbXnxk>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>>> Eduardo Soldera Garcia
>>>>>>>>>>>>> Data Engineer
>>>>>>>>>>>>> (16) 3509-5555 | www.arquivei.com.br
>>>>>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>>>>>>> [image: Arquivei.com.br – Inteligência em Notas Fiscais]
>>>>>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>>>>>>> [image: Google seleciona Arquivei para imersão e mentoria no
>>>>>>>>>>>>> Vale do Silício]
>>>>>>>>>>>>> <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad>
>>>>>>>>>>>>> <https://www.facebook.com/arquivei>
>>>>>>>>>>>>> <https://www.linkedin.com/company/arquivei>
>>>>>>>>>>>>> <https://www.youtube.com/watch?v=sSUUKxbXnxk>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> Eduardo Soldera Garcia
>>>>>>>>>>>> Data Engineer
>>>>>>>>>>>> (16) 3509-5555 | www.arquivei.com.br
>>>>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>>>>>> [image: Arquivei.com.br – Inteligência em Notas Fiscais]
>>>>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>>>>>> [image: Google seleciona Arquivei para imersão e mentoria no
>>>>>>>>>>>> Vale do Silício]
>>>>>>>>>>>> <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad>
>>>>>>>>>>>> <https://www.facebook.com/arquivei>
>>>>>>>>>>>> <https://www.linkedin.com/company/arquivei>
>>>>>>>>>>>> <https://www.youtube.com/watch?v=sSUUKxbXnxk>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Eduardo Soldera Garcia
>>>>>>>>>> Data Engineer
>>>>>>>>>> (16) 3509-5555 | www.arquivei.com.br
>>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>>>> [image: Arquivei.com.br – Inteligência em Notas Fiscais]
>>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>>>> [image: Google seleciona Arquivei para imersão e mentoria no Vale
>>>>>>>>>> do Silício]
>>>>>>>>>> <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad>
>>>>>>>>>> <https://www.facebook.com/arquivei>
>>>>>>>>>> <https://www.linkedin.com/company/arquivei>
>>>>>>>>>> <https://www.youtube.com/watch?v=sSUUKxbXnxk>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Eduardo Soldera Garcia
>>>>>>>> Data Engineer
>>>>>>>> (16) 3509-5555 | www.arquivei.com.br
>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>> [image: Arquivei.com.br – Inteligência em Notas Fiscais]
>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>> [image: Google seleciona Arquivei para imersão e mentoria no Vale
>>>>>>>> do Silício]
>>>>>>>> <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad>
>>>>>>>> <https://www.facebook.com/arquivei>
>>>>>>>> <https://www.linkedin.com/company/arquivei>
>>>>>>>> <https://www.youtube.com/watch?v=sSUUKxbXnxk>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>>
>>>>>>> JC
>>>>>>>
>>>>>>>
>>
>> --
>> Eduardo Soldera Garcia
>> Data Engineer
>> (16) 3509-5555 | www.arquivei.com.br
>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>> [image: Arquivei.com.br – Inteligência em Notas Fiscais]
>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>> [image: Google seleciona Arquivei para imersão e mentoria no Vale do
>> Silício]
>> <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad>
>> <https://www.facebook.com/arquivei>
>> <https://www.linkedin.com/company/arquivei>
>> <https://www.youtube.com/watch?v=sSUUKxbXnxk>
>>
>

-- 
Eduardo Soldera Garcia
Data Engineer
(16) 3509-5555 | www.arquivei.com.br
<https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
[image: Arquivei.com.br – Inteligência em Notas Fiscais]
<https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
[image: Google seleciona Arquivei para imersão e mentoria no Vale do
Silício]
<https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad>
<https://www.facebook.com/arquivei>
<https://www.linkedin.com/company/arquivei>
<https://www.youtube.com/watch?v=sSUUKxbXnxk>

Re: Problem with KafkaIO

Posted by Raghu Angadi <ra...@google.com>.
On Fri, Oct 19, 2018 at 6:54 AM Eduardo Soldera <
eduardo.soldera@arquivei.com.br> wrote:

> Hi Raghu, just a quick update. We were waiting for Spotify's Scio to
> update to Beam 2.7. We've just deployed the pipeling sucessfully. Just for
> letting you know, I tried to use the workaround code snipped, but Dataflow
> wouldn't recover after a Kafka unavailability.
>

Thanks for the update. The workaround helps only if KafkaClient itself can
recover when try to read again. I guess some of those exceptions are are
not recoverable.

Please let us know how the actual fix works.

Thanks.
Raghu.


>
> Thanks for your help.
>
> Regards
>
> Em qua, 19 de set de 2018 às 15:37, Raghu Angadi <ra...@google.com>
> escreveu:
>
>>
>>
>> On Wed, Sep 19, 2018 at 11:24 AM Juan Carlos Garcia <jc...@gmail.com>
>> wrote:
>>
>>> Sorry I hit the send button to fast... The error occurs in the worker.
>>>
>>
>> Np. Just one more comment on it: it is a very important
>> design/correctness decision to for runner to decide how to handle
>> persistent errors in a streaming pipeline. Dataflow keeps failing since
>> there is no solution to restart a pipeline from scratch without losing
>> exactly-once guarantees. It lets user decide if the pipeline needs to be
>> 'upgraded'.
>>
>> Raghu.
>>
>>>
>>> Juan Carlos Garcia <jc...@gmail.com> schrieb am Mi., 19. Sep. 2018,
>>> 20:22:
>>>
>>>> Sorry for hijacking the thread, we are running Spark on top of Yarn,
>>>> yarn retries multiple times until it reachs it max attempt and then gives
>>>> up.
>>>>
>>>> Raghu Angadi <ra...@google.com> schrieb am Mi., 19. Sep. 2018, 18:58:
>>>>
>>>>> On Wed, Sep 19, 2018 at 7:26 AM Juan Carlos Garcia <
>>>>> jcgarciam@gmail.com> wrote:
>>>>>
>>>>>> Don't know if its related, but we have seen our pipeline dying (using
>>>>>> SparkRunner) when there is problem with Kafka  (network interruptions),
>>>>>> errors like:  org.apache.kafka.common.errors.TimeoutException: Timeout
>>>>>> expired while fetching topic metadata
>>>>>>
>>>>>> Maybe this will fix it as well, thanks Raghu for the hint about
>>>>>> *withConsumerFactoryFn.*
>>>>>>
>>>>>
>>>>> Wouldn't that be retried by the SparkRunner if it happens on the
>>>>> worker? or does it happen while launching the pipeline on the client?
>>>>>
>>>>>
>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Wed, Sep 19, 2018 at 3:29 PM Eduardo Soldera <
>>>>>> eduardo.soldera@arquivei.com.br> wrote:
>>>>>>
>>>>>>> Hi Raghu, thank you.
>>>>>>>
>>>>>>> I'm not sure though what to pass as an argument:
>>>>>>>
>>>>>>> KafkaIO.read[String,String]()
>>>>>>>   .withBootstrapServers(server)
>>>>>>>   .withTopic(topic)
>>>>>>>   .withKeyDeserializer(classOf[StringDeserializer])
>>>>>>>   .withValueDeserializer(classOf[StringDeserializer])
>>>>>>>   .withConsumerFactoryFn(new KafkaExecutor.ConsumerFactoryFn(????????????????))
>>>>>>>   .updateConsumerProperties(properties)
>>>>>>>   .commitOffsetsInFinalize()
>>>>>>>   .withoutMetadata()
>>>>>>>
>>>>>>>
>>>>>>> Regards
>>>>>>>
>>>>>>>
>>>>>>> Em ter, 18 de set de 2018 às 21:15, Raghu Angadi <ra...@google.com>
>>>>>>> escreveu:
>>>>>>>
>>>>>>>> Hi Eduardo,
>>>>>>>>
>>>>>>>> There another work around you can try without having to wait for
>>>>>>>> 2.7.0 release: Use a wrapper to catch exception from KafkaConsumer#poll()
>>>>>>>> and pass the wrapper to withConsumerFactoryFn() for KafkIO reader [1].
>>>>>>>>
>>>>>>>> Using something like (such a wrapper is used in KafkasIO tests
>>>>>>>> [2]):
>>>>>>>> private static class ConsumerFactoryFn
>>>>>>>>                 implements SerializableFunction<Map<String,
>>>>>>>> Object>, Consumer<byte[], byte[]>> {
>>>>>>>>   @Override
>>>>>>>>     public Consumer<byte[], byte[]> apply(Map<String, Object>
>>>>>>>> config) {
>>>>>>>>       return new KafkaConsumer(config) {
>>>>>>>>           @Override
>>>>>>>>           public ConsumerRecords<K, V> poll(long timeout) {
>>>>>>>>               // work around for BEAM-5375
>>>>>>>>               while (true) {
>>>>>>>>                   try {
>>>>>>>>                     return super.poll(timeout);
>>>>>>>>                  } catch (Exception e) {
>>>>>>>>                     // LOG & sleep for sec
>>>>>>>>                 }
>>>>>>>>           }
>>>>>>>>        }
>>>>>>>>     }
>>>>>>>> }
>>>>>>>>
>>>>>>>> [1]:
>>>>>>>> https://github.com/apache/beam/blob/release-2.4.0/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaIO.java#L417
>>>>>>>> [2]:
>>>>>>>> https://github.com/apache/beam/blob/release-2.4.0/sdks/java/io/kafka/src/test/java/org/apache/beam/sdk/io/kafka/KafkaIOTest.java#L261
>>>>>>>>
>>>>>>>> On Tue, Sep 18, 2018 at 5:49 AM Eduardo Soldera <
>>>>>>>> eduardo.soldera@arquivei.com.br> wrote:
>>>>>>>>
>>>>>>>>> Hi Raghu, we're not sure how long the network was down. According
>>>>>>>>> to the logs no longer than one minute. A 30 second shutdown would work for
>>>>>>>>> the tests.
>>>>>>>>>
>>>>>>>>> Regards
>>>>>>>>>
>>>>>>>>> Em sex, 14 de set de 2018 às 21:41, Raghu Angadi <
>>>>>>>>> rangadi@google.com> escreveu:
>>>>>>>>>
>>>>>>>>>> Thanks. I could repro myself as well. How long was the network
>>>>>>>>>> down?
>>>>>>>>>>
>>>>>>>>>> Trying to get the fix into 2.7 RC2.
>>>>>>>>>>
>>>>>>>>>> On Fri, Sep 14, 2018 at 12:25 PM Eduardo Soldera <
>>>>>>>>>> eduardo.soldera@arquivei.com.br> wrote:
>>>>>>>>>>
>>>>>>>>>>> Just to make myself clear, I'm not sure how to use the patch but
>>>>>>>>>>> if you could send us some guidance would be great.
>>>>>>>>>>>
>>>>>>>>>>> Em sex, 14 de set de 2018 às 16:24, Eduardo Soldera <
>>>>>>>>>>> eduardo.soldera@arquivei.com.br> escreveu:
>>>>>>>>>>>
>>>>>>>>>>>> Hi Raghu, yes, it is feasible, would you do that for us? I'm
>>>>>>>>>>>> not sure how we'd use the patch. We're using SBT and Spotify's Scio with
>>>>>>>>>>>> Scala.
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks
>>>>>>>>>>>>
>>>>>>>>>>>> Em sex, 14 de set de 2018 às 16:07, Raghu Angadi <
>>>>>>>>>>>> rangadi@google.com> escreveu:
>>>>>>>>>>>>
>>>>>>>>>>>>> Is is feasible for you to verify the fix in your dev job? I
>>>>>>>>>>>>> can make a patch against Beam 2.4 branch if you like.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Raghu.
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Fri, Sep 14, 2018 at 11:14 AM Eduardo Soldera <
>>>>>>>>>>>>> eduardo.soldera@arquivei.com.br> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi Raghu, thank you very much for the pull request.
>>>>>>>>>>>>>> We'll wait for the 2.7 Beam release.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Regards!
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Em qui, 13 de set de 2018 às 18:19, Raghu Angadi <
>>>>>>>>>>>>>> rangadi@google.com> escreveu:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Fix: https://github.com/apache/beam/pull/6391
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Wed, Sep 12, 2018 at 3:30 PM Raghu Angadi <
>>>>>>>>>>>>>>> rangadi@google.com> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Filed BEAM-5375
>>>>>>>>>>>>>>>> <https://issues.apache.org/jira/browse/BEAM-5375>. I will
>>>>>>>>>>>>>>>> fix it later this week.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Wed, Sep 12, 2018 at 12:16 PM Raghu Angadi <
>>>>>>>>>>>>>>>> rangadi@google.com> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Wed, Sep 12, 2018 at 12:11 PM Raghu Angadi <
>>>>>>>>>>>>>>>>> rangadi@google.com> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Thanks for the job id, I looked at the worker logs
>>>>>>>>>>>>>>>>>> (following usual support oncall access protocol that provides temporary
>>>>>>>>>>>>>>>>>> access to things like logs in GCP):
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Root issue looks like consumerPollLoop() mentioned
>>>>>>>>>>>>>>>>>> earlier needs to handle unchecked exception. In your case it is clear that
>>>>>>>>>>>>>>>>>> poll thread exited with a runtime exception. The reader does not check for
>>>>>>>>>>>>>>>>>> it and continues to wait for poll thread to enqueue messages. A fix should
>>>>>>>>>>>>>>>>>> result in an IOException for read from the source. The runners will handle
>>>>>>>>>>>>>>>>>> that appropriately after that.  I will file a jira.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> https://github.com/apache/beam/blob/master/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaUnboundedReader.java#L678
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Ignore the link.. was pasted here by mistake.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> From the logs (with a comment below each one):
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>    - 2018-09-12 06:13:07.345 PDT Reader-0: reading from
>>>>>>>>>>>>>>>>>>    kafka_topic-0 starting at offset 2
>>>>>>>>>>>>>>>>>>       - Implies the reader is initialized and poll
>>>>>>>>>>>>>>>>>>       thread is started.
>>>>>>>>>>>>>>>>>>    - 2018-09-12 06:13:07.780 PDT Reader-0: first record
>>>>>>>>>>>>>>>>>>    offset 2
>>>>>>>>>>>>>>>>>>       - The reader actually got a message received by
>>>>>>>>>>>>>>>>>>       the poll thread from Kafka.
>>>>>>>>>>>>>>>>>>    - 2018-09-12 06:16:48.771 PDT Reader-0: exception
>>>>>>>>>>>>>>>>>>    while fetching latest offset for partition kafka_topic-0. will be retried.
>>>>>>>>>>>>>>>>>>       - This must have happened around the time when
>>>>>>>>>>>>>>>>>>       network was disrupted. This is from. Actual log is from another periodic
>>>>>>>>>>>>>>>>>>       task that fetches latest offsets for partitions.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> The poll thread must have died around the time network
>>>>>>>>>>>>>>>>>> was disrupted.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> The following log comes from kafka client itself and is
>>>>>>>>>>>>>>>>>> printed every second when KafkaIO fetches latest offset. This log seems to
>>>>>>>>>>>>>>>>>> be added in recent versions. It is probably an unintentional log. I don't
>>>>>>>>>>>>>>>>>> think there is any better to fetch latest offsets than how KafkaIO does
>>>>>>>>>>>>>>>>>> now. This is logged inside consumer.position() called at [1].
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>    - 2018-09-12 06:13:11.786 PDT [Consumer
>>>>>>>>>>>>>>>>>>    clientId=consumer-2,
>>>>>>>>>>>>>>>>>>    groupId=Reader-0_offset_consumer_1735388161_genericPipe] Resetting offset
>>>>>>>>>>>>>>>>>>    for partition com.arquivei.dataeng.andre-0 to offset 3.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> [1]:
>>>>>>>>>>>>>>>>>> https://github.com/apache/beam/blob/master/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaUnboundedReader.java#L678
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> This 'Resetting offset' is harmless, but is quite annoying
>>>>>>>>>>>>>>>>> to see in the worker logs. One way to avoid is to set kafka consumer's log
>>>>>>>>>>>>>>>>> level to WARNING. Ideally KafkaIO itself should do something to avoid it
>>>>>>>>>>>>>>>>> without user option.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On Wed, Sep 12, 2018 at 10:27 AM Eduardo Soldera <
>>>>>>>>>>>>>>>>>> eduardo.soldera@arquivei.com.br> wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Hi Raghu! The job_id of our dev job is
>>>>>>>>>>>>>>>>>>> 2018-09-12_06_11_48-5600553605191377866.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Thanks!
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Em qua, 12 de set de 2018 às 14:18, Raghu Angadi <
>>>>>>>>>>>>>>>>>>> rangadi@google.com> escreveu:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Thanks for debugging.
>>>>>>>>>>>>>>>>>>>> Can you provide the job_id of your dev job? The
>>>>>>>>>>>>>>>>>>>> stacktrace shows that there is no thread running 'consumerPollLoop()' which
>>>>>>>>>>>>>>>>>>>> can explain stuck reader. You will likely find a logs at line 594 & 587
>>>>>>>>>>>>>>>>>>>> [1].  Dataflow caches its readers and DirectRunner may not. That can
>>>>>>>>>>>>>>>>>>>> explain DirectRunner resume reads. The expectation in KafkaIO is that Kafka
>>>>>>>>>>>>>>>>>>>> client library takes care of retrying in case of connection problems (as
>>>>>>>>>>>>>>>>>>>> documented). It is possible that in some cases poll() throws and we need to
>>>>>>>>>>>>>>>>>>>> restart the client in KafkaIO.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> [1]:
>>>>>>>>>>>>>>>>>>>> https://github.com/apache/beam/blob/master/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaUnboundedReader.java#L594
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> On Wed, Sep 12, 2018 at 9:59 AM Eduardo Soldera <
>>>>>>>>>>>>>>>>>>>> eduardo.soldera@arquivei.com.br> wrote:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Hi Raghu, thanks for your help.
>>>>>>>>>>>>>>>>>>>>> Just answering your previous question, the following
>>>>>>>>>>>>>>>>>>>>> logs were the same as before the error, as if the pipeline were still
>>>>>>>>>>>>>>>>>>>>> getting the messages, for example:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> (...)
>>>>>>>>>>>>>>>>>>>>> Resetting offset for partition
>>>>>>>>>>>>>>>>>>>>> com.arquivei.dataeng.andre-0 to offset 10.
>>>>>>>>>>>>>>>>>>>>> Resetting offset for partition
>>>>>>>>>>>>>>>>>>>>> com.arquivei.dataeng.andre-0 to offset 15.
>>>>>>>>>>>>>>>>>>>>> ERROR
>>>>>>>>>>>>>>>>>>>>> Resetting offset for partition
>>>>>>>>>>>>>>>>>>>>> com.arquivei.dataeng.andre-0 to offset 22.
>>>>>>>>>>>>>>>>>>>>> Resetting offset for partition
>>>>>>>>>>>>>>>>>>>>> com.arquivei.dataeng.andre-0 to offset 30.
>>>>>>>>>>>>>>>>>>>>> (...)
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> But when checking the Kafka Consumer Group, the
>>>>>>>>>>>>>>>>>>>>> current offset stays at 15, the commited offset from the last processed
>>>>>>>>>>>>>>>>>>>>> message, before the error.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> We'll file a bug, but we could now reproduce the issue
>>>>>>>>>>>>>>>>>>>>> in a Dev scenario.
>>>>>>>>>>>>>>>>>>>>> We started the same pipeline using the direct runner,
>>>>>>>>>>>>>>>>>>>>> without Google Dataflow. We blocked the Kafka Broker network and the same
>>>>>>>>>>>>>>>>>>>>> error was thrown. Then we unblocked the network and the pipeline was able
>>>>>>>>>>>>>>>>>>>>> to successfully process the subsequent messages.
>>>>>>>>>>>>>>>>>>>>> When we started the same pipeline in the Dataflow
>>>>>>>>>>>>>>>>>>>>> runner and did the same test, the same problem from our production scenario
>>>>>>>>>>>>>>>>>>>>> happened, Dataflow couldn't process the new messages. Unfortunately, we've
>>>>>>>>>>>>>>>>>>>>> stopped the dataflow job in production, but the problematic dev job is
>>>>>>>>>>>>>>>>>>>>> still running and the log file of the VM is attached. Thank you very much.
>>>>>>>>>>>>>>>>>>>>> Best regards
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Em ter, 11 de set de 2018 às 18:28, Raghu Angadi <
>>>>>>>>>>>>>>>>>>>>> rangadi@google.com> escreveu:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Specifically, I am interested if you have any thread
>>>>>>>>>>>>>>>>>>>>>> running 'consumerPollLoop()' [1]. There should always be one (if a worker
>>>>>>>>>>>>>>>>>>>>>> is assigned one of the partitions). It is possible that KafkaClient itself
>>>>>>>>>>>>>>>>>>>>>> is hasn't recovered from the group coordinator error (though unlikely).
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> https://github.com/apache/beam/blob/master/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaUnboundedReader.java#L570
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> On Tue, Sep 11, 2018 at 12:31 PM Raghu Angadi <
>>>>>>>>>>>>>>>>>>>>>> rangadi@google.com> wrote:
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Hi Eduardo,
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> In case of any error, the pipeline should keep on
>>>>>>>>>>>>>>>>>>>>>>> trying to fetch. I don't know about this particular error. Do you see any
>>>>>>>>>>>>>>>>>>>>>>> others afterwards in the log?
>>>>>>>>>>>>>>>>>>>>>>> Couple of things you could try if the logs are not
>>>>>>>>>>>>>>>>>>>>>>> useful :
>>>>>>>>>>>>>>>>>>>>>>>  - login to one of the VMs and get stacktrace of
>>>>>>>>>>>>>>>>>>>>>>> java worker (look for a container called java-streaming)
>>>>>>>>>>>>>>>>>>>>>>>  - file a support bug or stackoverflow question with
>>>>>>>>>>>>>>>>>>>>>>> jobid so that Dataflow oncall can take a look.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Raghu.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> On Tue, Sep 11, 2018 at 12:10 PM Eduardo Soldera <
>>>>>>>>>>>>>>>>>>>>>>> eduardo.soldera@arquivei.com.br> wrote:
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>>>>>>>>> We have a Apache Beam pipeline running in Google
>>>>>>>>>>>>>>>>>>>>>>>> Dataflow using KafkaIO. Suddenly the pipeline stop fetching Kafka messages
>>>>>>>>>>>>>>>>>>>>>>>> at all, as our other workers from other pipelines continued to get Kafka
>>>>>>>>>>>>>>>>>>>>>>>> messages.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> At the moment it stopped we got these messages:
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> I  [Consumer clientId=consumer-1, groupId=genericPipe] Error sending fetch request (sessionId=1396189203, epoch=2431598) to node 3: org.apache.kafka.common.errors.DisconnectException.
>>>>>>>>>>>>>>>>>>>>>>>> I  [Consumer clientId=consumer-1, groupId=genericPipe] Group coordinator 10.0.52.70:9093 (id: 2147483646 rack: null) is unavailable or invalid, will attempt rediscovery
>>>>>>>>>>>>>>>>>>>>>>>> I  [Consumer clientId=consumer-1, groupId=genericPipe] Discovered group coordinator 10.0.52.70:9093 (id: 2147483646 rack: null)
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> And then the pipeline stopped reading the messages.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> This is the KafkaIO setup  we have:
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> KafkaIO.read[String,String]()
>>>>>>>>>>>>>>>>>>>>>>>>   .withBootstrapServers(server)
>>>>>>>>>>>>>>>>>>>>>>>>   .withTopic(topic)
>>>>>>>>>>>>>>>>>>>>>>>>   .withKeyDeserializer(classOf[StringDeserializer])
>>>>>>>>>>>>>>>>>>>>>>>>   .withValueDeserializer(classOf[StringDeserializer])
>>>>>>>>>>>>>>>>>>>>>>>>   .updateConsumerProperties(properties)
>>>>>>>>>>>>>>>>>>>>>>>>   .commitOffsetsInFinalize()
>>>>>>>>>>>>>>>>>>>>>>>>   .withoutMetadata()
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>  Any help will be much appreciated.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Best regards,
>>>>>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>>>> Eduardo Soldera Garcia
>>>>>>>>>>>>>>>>>>>>>>>> Data Engineer
>>>>>>>>>>>>>>>>>>>>>>>> (16) 3509-5555 | www.arquivei.com.br
>>>>>>>>>>>>>>>>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>>>>>>>>>>>>>>>>>> [image: Arquivei.com.br – Inteligência em Notas
>>>>>>>>>>>>>>>>>>>>>>>> Fiscais]
>>>>>>>>>>>>>>>>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>>>>>>>>>>>>>>>>>> [image: Google seleciona Arquivei para imersão e
>>>>>>>>>>>>>>>>>>>>>>>> mentoria no Vale do Silício]
>>>>>>>>>>>>>>>>>>>>>>>> <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad>
>>>>>>>>>>>>>>>>>>>>>>>> <https://www.facebook.com/arquivei>
>>>>>>>>>>>>>>>>>>>>>>>> <https://www.linkedin.com/company/arquivei>
>>>>>>>>>>>>>>>>>>>>>>>> <https://www.youtube.com/watch?v=sSUUKxbXnxk>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>> Eduardo Soldera Garcia
>>>>>>>>>>>>>>>>>>>>> Data Engineer
>>>>>>>>>>>>>>>>>>>>> (16) 3509-5555 | www.arquivei.com.br
>>>>>>>>>>>>>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>>>>>>>>>>>>>>> [image: Arquivei.com.br – Inteligência em Notas
>>>>>>>>>>>>>>>>>>>>> Fiscais]
>>>>>>>>>>>>>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>>>>>>>>>>>>>>> [image: Google seleciona Arquivei para imersão e
>>>>>>>>>>>>>>>>>>>>> mentoria no Vale do Silício]
>>>>>>>>>>>>>>>>>>>>> <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad>
>>>>>>>>>>>>>>>>>>>>> <https://www.facebook.com/arquivei>
>>>>>>>>>>>>>>>>>>>>> <https://www.linkedin.com/company/arquivei>
>>>>>>>>>>>>>>>>>>>>> <https://www.youtube.com/watch?v=sSUUKxbXnxk>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>> Eduardo Soldera Garcia
>>>>>>>>>>>>>>>>>>> Data Engineer
>>>>>>>>>>>>>>>>>>> (16) 3509-5555 | www.arquivei.com.br
>>>>>>>>>>>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>>>>>>>>>>>>> [image: Arquivei.com.br – Inteligência em Notas Fiscais]
>>>>>>>>>>>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>>>>>>>>>>>>> [image: Google seleciona Arquivei para imersão e
>>>>>>>>>>>>>>>>>>> mentoria no Vale do Silício]
>>>>>>>>>>>>>>>>>>> <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad>
>>>>>>>>>>>>>>>>>>> <https://www.facebook.com/arquivei>
>>>>>>>>>>>>>>>>>>> <https://www.linkedin.com/company/arquivei>
>>>>>>>>>>>>>>>>>>> <https://www.youtube.com/watch?v=sSUUKxbXnxk>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> --
>>>>>>>>>>>>>> Eduardo Soldera Garcia
>>>>>>>>>>>>>> Data Engineer
>>>>>>>>>>>>>> (16) 3509-5555 | www.arquivei.com.br
>>>>>>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>>>>>>>> [image: Arquivei.com.br – Inteligência em Notas Fiscais]
>>>>>>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>>>>>>>> [image: Google seleciona Arquivei para imersão e mentoria no
>>>>>>>>>>>>>> Vale do Silício]
>>>>>>>>>>>>>> <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad>
>>>>>>>>>>>>>> <https://www.facebook.com/arquivei>
>>>>>>>>>>>>>> <https://www.linkedin.com/company/arquivei>
>>>>>>>>>>>>>> <https://www.youtube.com/watch?v=sSUUKxbXnxk>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> Eduardo Soldera Garcia
>>>>>>>>>>>> Data Engineer
>>>>>>>>>>>> (16) 3509-5555 | www.arquivei.com.br
>>>>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>>>>>> [image: Arquivei.com.br – Inteligência em Notas Fiscais]
>>>>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>>>>>> [image: Google seleciona Arquivei para imersão e mentoria no
>>>>>>>>>>>> Vale do Silício]
>>>>>>>>>>>> <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad>
>>>>>>>>>>>> <https://www.facebook.com/arquivei>
>>>>>>>>>>>> <https://www.linkedin.com/company/arquivei>
>>>>>>>>>>>> <https://www.youtube.com/watch?v=sSUUKxbXnxk>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Eduardo Soldera Garcia
>>>>>>>>>>> Data Engineer
>>>>>>>>>>> (16) 3509-5555 | www.arquivei.com.br
>>>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>>>>> [image: Arquivei.com.br – Inteligência em Notas Fiscais]
>>>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>>>>> [image: Google seleciona Arquivei para imersão e mentoria no
>>>>>>>>>>> Vale do Silício]
>>>>>>>>>>> <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad>
>>>>>>>>>>> <https://www.facebook.com/arquivei>
>>>>>>>>>>> <https://www.linkedin.com/company/arquivei>
>>>>>>>>>>> <https://www.youtube.com/watch?v=sSUUKxbXnxk>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Eduardo Soldera Garcia
>>>>>>>>> Data Engineer
>>>>>>>>> (16) 3509-5555 | www.arquivei.com.br
>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>>> [image: Arquivei.com.br – Inteligência em Notas Fiscais]
>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>>> [image: Google seleciona Arquivei para imersão e mentoria no Vale
>>>>>>>>> do Silício]
>>>>>>>>> <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad>
>>>>>>>>> <https://www.facebook.com/arquivei>
>>>>>>>>> <https://www.linkedin.com/company/arquivei>
>>>>>>>>> <https://www.youtube.com/watch?v=sSUUKxbXnxk>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Eduardo Soldera Garcia
>>>>>>> Data Engineer
>>>>>>> (16) 3509-5555 | www.arquivei.com.br
>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>> [image: Arquivei.com.br – Inteligência em Notas Fiscais]
>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>> [image: Google seleciona Arquivei para imersão e mentoria no Vale do
>>>>>>> Silício]
>>>>>>> <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad>
>>>>>>> <https://www.facebook.com/arquivei>
>>>>>>> <https://www.linkedin.com/company/arquivei>
>>>>>>> <https://www.youtube.com/watch?v=sSUUKxbXnxk>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>>
>>>>>> JC
>>>>>>
>>>>>>
>
> --
> Eduardo Soldera Garcia
> Data Engineer
> (16) 3509-5555 | www.arquivei.com.br
> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
> [image: Arquivei.com.br – Inteligência em Notas Fiscais]
> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
> [image: Google seleciona Arquivei para imersão e mentoria no Vale do
> Silício]
> <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad>
> <https://www.facebook.com/arquivei>
> <https://www.linkedin.com/company/arquivei>
> <https://www.youtube.com/watch?v=sSUUKxbXnxk>
>

Re: Problem with KafkaIO

Posted by Eduardo Soldera <ed...@arquivei.com.br>.
Hi Raghu, just a quick update. We were waiting for Spotify's Scio to update
to Beam 2.7. We've just deployed the pipeling sucessfully. Just for letting
you know, I tried to use the workaround code snipped, but Dataflow wouldn't
recover after a Kafka unavailability.

Thanks for your help.

Regards

Em qua, 19 de set de 2018 às 15:37, Raghu Angadi <ra...@google.com>
escreveu:

>
>
> On Wed, Sep 19, 2018 at 11:24 AM Juan Carlos Garcia <jc...@gmail.com>
> wrote:
>
>> Sorry I hit the send button to fast... The error occurs in the worker.
>>
>
> Np. Just one more comment on it: it is a very important design/correctness
> decision to for runner to decide how to handle persistent errors in a
> streaming pipeline. Dataflow keeps failing since there is no solution to
> restart a pipeline from scratch without losing exactly-once guarantees. It
> lets user decide if the pipeline needs to be 'upgraded'.
>
> Raghu.
>
>>
>> Juan Carlos Garcia <jc...@gmail.com> schrieb am Mi., 19. Sep. 2018,
>> 20:22:
>>
>>> Sorry for hijacking the thread, we are running Spark on top of Yarn,
>>> yarn retries multiple times until it reachs it max attempt and then gives
>>> up.
>>>
>>> Raghu Angadi <ra...@google.com> schrieb am Mi., 19. Sep. 2018, 18:58:
>>>
>>>> On Wed, Sep 19, 2018 at 7:26 AM Juan Carlos Garcia <jc...@gmail.com>
>>>> wrote:
>>>>
>>>>> Don't know if its related, but we have seen our pipeline dying (using
>>>>> SparkRunner) when there is problem with Kafka  (network interruptions),
>>>>> errors like:  org.apache.kafka.common.errors.TimeoutException: Timeout
>>>>> expired while fetching topic metadata
>>>>>
>>>>> Maybe this will fix it as well, thanks Raghu for the hint about
>>>>> *withConsumerFactoryFn.*
>>>>>
>>>>
>>>> Wouldn't that be retried by the SparkRunner if it happens on the
>>>> worker? or does it happen while launching the pipeline on the client?
>>>>
>>>>
>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Wed, Sep 19, 2018 at 3:29 PM Eduardo Soldera <
>>>>> eduardo.soldera@arquivei.com.br> wrote:
>>>>>
>>>>>> Hi Raghu, thank you.
>>>>>>
>>>>>> I'm not sure though what to pass as an argument:
>>>>>>
>>>>>> KafkaIO.read[String,String]()
>>>>>>   .withBootstrapServers(server)
>>>>>>   .withTopic(topic)
>>>>>>   .withKeyDeserializer(classOf[StringDeserializer])
>>>>>>   .withValueDeserializer(classOf[StringDeserializer])
>>>>>>   .withConsumerFactoryFn(new KafkaExecutor.ConsumerFactoryFn(????????????????))
>>>>>>   .updateConsumerProperties(properties)
>>>>>>   .commitOffsetsInFinalize()
>>>>>>   .withoutMetadata()
>>>>>>
>>>>>>
>>>>>> Regards
>>>>>>
>>>>>>
>>>>>> Em ter, 18 de set de 2018 às 21:15, Raghu Angadi <ra...@google.com>
>>>>>> escreveu:
>>>>>>
>>>>>>> Hi Eduardo,
>>>>>>>
>>>>>>> There another work around you can try without having to wait for
>>>>>>> 2.7.0 release: Use a wrapper to catch exception from KafkaConsumer#poll()
>>>>>>> and pass the wrapper to withConsumerFactoryFn() for KafkIO reader [1].
>>>>>>>
>>>>>>> Using something like (such a wrapper is used in KafkasIO tests [2]):
>>>>>>> private static class ConsumerFactoryFn
>>>>>>>                 implements SerializableFunction<Map<String, Object>,
>>>>>>> Consumer<byte[], byte[]>> {
>>>>>>>   @Override
>>>>>>>     public Consumer<byte[], byte[]> apply(Map<String, Object>
>>>>>>> config) {
>>>>>>>       return new KafkaConsumer(config) {
>>>>>>>           @Override
>>>>>>>           public ConsumerRecords<K, V> poll(long timeout) {
>>>>>>>               // work around for BEAM-5375
>>>>>>>               while (true) {
>>>>>>>                   try {
>>>>>>>                     return super.poll(timeout);
>>>>>>>                  } catch (Exception e) {
>>>>>>>                     // LOG & sleep for sec
>>>>>>>                 }
>>>>>>>           }
>>>>>>>        }
>>>>>>>     }
>>>>>>> }
>>>>>>>
>>>>>>> [1]:
>>>>>>> https://github.com/apache/beam/blob/release-2.4.0/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaIO.java#L417
>>>>>>> [2]:
>>>>>>> https://github.com/apache/beam/blob/release-2.4.0/sdks/java/io/kafka/src/test/java/org/apache/beam/sdk/io/kafka/KafkaIOTest.java#L261
>>>>>>>
>>>>>>> On Tue, Sep 18, 2018 at 5:49 AM Eduardo Soldera <
>>>>>>> eduardo.soldera@arquivei.com.br> wrote:
>>>>>>>
>>>>>>>> Hi Raghu, we're not sure how long the network was down. According
>>>>>>>> to the logs no longer than one minute. A 30 second shutdown would work for
>>>>>>>> the tests.
>>>>>>>>
>>>>>>>> Regards
>>>>>>>>
>>>>>>>> Em sex, 14 de set de 2018 às 21:41, Raghu Angadi <
>>>>>>>> rangadi@google.com> escreveu:
>>>>>>>>
>>>>>>>>> Thanks. I could repro myself as well. How long was the network
>>>>>>>>> down?
>>>>>>>>>
>>>>>>>>> Trying to get the fix into 2.7 RC2.
>>>>>>>>>
>>>>>>>>> On Fri, Sep 14, 2018 at 12:25 PM Eduardo Soldera <
>>>>>>>>> eduardo.soldera@arquivei.com.br> wrote:
>>>>>>>>>
>>>>>>>>>> Just to make myself clear, I'm not sure how to use the patch but
>>>>>>>>>> if you could send us some guidance would be great.
>>>>>>>>>>
>>>>>>>>>> Em sex, 14 de set de 2018 às 16:24, Eduardo Soldera <
>>>>>>>>>> eduardo.soldera@arquivei.com.br> escreveu:
>>>>>>>>>>
>>>>>>>>>>> Hi Raghu, yes, it is feasible, would you do that for us? I'm not
>>>>>>>>>>> sure how we'd use the patch. We're using SBT and Spotify's Scio with Scala.
>>>>>>>>>>>
>>>>>>>>>>> Thanks
>>>>>>>>>>>
>>>>>>>>>>> Em sex, 14 de set de 2018 às 16:07, Raghu Angadi <
>>>>>>>>>>> rangadi@google.com> escreveu:
>>>>>>>>>>>
>>>>>>>>>>>> Is is feasible for you to verify the fix in your dev job? I can
>>>>>>>>>>>> make a patch against Beam 2.4 branch if you like.
>>>>>>>>>>>>
>>>>>>>>>>>> Raghu.
>>>>>>>>>>>>
>>>>>>>>>>>> On Fri, Sep 14, 2018 at 11:14 AM Eduardo Soldera <
>>>>>>>>>>>> eduardo.soldera@arquivei.com.br> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi Raghu, thank you very much for the pull request.
>>>>>>>>>>>>> We'll wait for the 2.7 Beam release.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Regards!
>>>>>>>>>>>>>
>>>>>>>>>>>>> Em qui, 13 de set de 2018 às 18:19, Raghu Angadi <
>>>>>>>>>>>>> rangadi@google.com> escreveu:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Fix: https://github.com/apache/beam/pull/6391
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Wed, Sep 12, 2018 at 3:30 PM Raghu Angadi <
>>>>>>>>>>>>>> rangadi@google.com> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Filed BEAM-5375
>>>>>>>>>>>>>>> <https://issues.apache.org/jira/browse/BEAM-5375>. I will
>>>>>>>>>>>>>>> fix it later this week.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Wed, Sep 12, 2018 at 12:16 PM Raghu Angadi <
>>>>>>>>>>>>>>> rangadi@google.com> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Wed, Sep 12, 2018 at 12:11 PM Raghu Angadi <
>>>>>>>>>>>>>>>> rangadi@google.com> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Thanks for the job id, I looked at the worker logs
>>>>>>>>>>>>>>>>> (following usual support oncall access protocol that provides temporary
>>>>>>>>>>>>>>>>> access to things like logs in GCP):
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Root issue looks like consumerPollLoop() mentioned earlier
>>>>>>>>>>>>>>>>> needs to handle unchecked exception. In your case it is clear that poll
>>>>>>>>>>>>>>>>> thread exited with a runtime exception. The reader does not check for it
>>>>>>>>>>>>>>>>> and continues to wait for poll thread to enqueue messages. A fix should
>>>>>>>>>>>>>>>>> result in an IOException for read from the source. The runners will handle
>>>>>>>>>>>>>>>>> that appropriately after that.  I will file a jira.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> https://github.com/apache/beam/blob/master/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaUnboundedReader.java#L678
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Ignore the link.. was pasted here by mistake.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> From the logs (with a comment below each one):
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>    - 2018-09-12 06:13:07.345 PDT Reader-0: reading from
>>>>>>>>>>>>>>>>>    kafka_topic-0 starting at offset 2
>>>>>>>>>>>>>>>>>       - Implies the reader is initialized and poll thread
>>>>>>>>>>>>>>>>>       is started.
>>>>>>>>>>>>>>>>>    - 2018-09-12 06:13:07.780 PDT Reader-0: first record
>>>>>>>>>>>>>>>>>    offset 2
>>>>>>>>>>>>>>>>>       - The reader actually got a message received by the
>>>>>>>>>>>>>>>>>       poll thread from Kafka.
>>>>>>>>>>>>>>>>>    - 2018-09-12 06:16:48.771 PDT Reader-0: exception
>>>>>>>>>>>>>>>>>    while fetching latest offset for partition kafka_topic-0. will be retried.
>>>>>>>>>>>>>>>>>       - This must have happened around the time when
>>>>>>>>>>>>>>>>>       network was disrupted. This is from. Actual log is from another periodic
>>>>>>>>>>>>>>>>>       task that fetches latest offsets for partitions.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> The poll thread must have died around the time network was
>>>>>>>>>>>>>>>>> disrupted.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> The following log comes from kafka client itself and is
>>>>>>>>>>>>>>>>> printed every second when KafkaIO fetches latest offset. This log seems to
>>>>>>>>>>>>>>>>> be added in recent versions. It is probably an unintentional log. I don't
>>>>>>>>>>>>>>>>> think there is any better to fetch latest offsets than how KafkaIO does
>>>>>>>>>>>>>>>>> now. This is logged inside consumer.position() called at [1].
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>    - 2018-09-12 06:13:11.786 PDT [Consumer
>>>>>>>>>>>>>>>>>    clientId=consumer-2,
>>>>>>>>>>>>>>>>>    groupId=Reader-0_offset_consumer_1735388161_genericPipe] Resetting offset
>>>>>>>>>>>>>>>>>    for partition com.arquivei.dataeng.andre-0 to offset 3.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> [1]:
>>>>>>>>>>>>>>>>> https://github.com/apache/beam/blob/master/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaUnboundedReader.java#L678
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> This 'Resetting offset' is harmless, but is quite annoying
>>>>>>>>>>>>>>>> to see in the worker logs. One way to avoid is to set kafka consumer's log
>>>>>>>>>>>>>>>> level to WARNING. Ideally KafkaIO itself should do something to avoid it
>>>>>>>>>>>>>>>> without user option.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Wed, Sep 12, 2018 at 10:27 AM Eduardo Soldera <
>>>>>>>>>>>>>>>>> eduardo.soldera@arquivei.com.br> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Hi Raghu! The job_id of our dev job is
>>>>>>>>>>>>>>>>>> 2018-09-12_06_11_48-5600553605191377866.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Thanks!
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Em qua, 12 de set de 2018 às 14:18, Raghu Angadi <
>>>>>>>>>>>>>>>>>> rangadi@google.com> escreveu:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Thanks for debugging.
>>>>>>>>>>>>>>>>>>> Can you provide the job_id of your dev job? The
>>>>>>>>>>>>>>>>>>> stacktrace shows that there is no thread running 'consumerPollLoop()' which
>>>>>>>>>>>>>>>>>>> can explain stuck reader. You will likely find a logs at line 594 & 587
>>>>>>>>>>>>>>>>>>> [1].  Dataflow caches its readers and DirectRunner may not. That can
>>>>>>>>>>>>>>>>>>> explain DirectRunner resume reads. The expectation in KafkaIO is that Kafka
>>>>>>>>>>>>>>>>>>> client library takes care of retrying in case of connection problems (as
>>>>>>>>>>>>>>>>>>> documented). It is possible that in some cases poll() throws and we need to
>>>>>>>>>>>>>>>>>>> restart the client in KafkaIO.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> [1]:
>>>>>>>>>>>>>>>>>>> https://github.com/apache/beam/blob/master/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaUnboundedReader.java#L594
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> On Wed, Sep 12, 2018 at 9:59 AM Eduardo Soldera <
>>>>>>>>>>>>>>>>>>> eduardo.soldera@arquivei.com.br> wrote:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Hi Raghu, thanks for your help.
>>>>>>>>>>>>>>>>>>>> Just answering your previous question, the following
>>>>>>>>>>>>>>>>>>>> logs were the same as before the error, as if the pipeline were still
>>>>>>>>>>>>>>>>>>>> getting the messages, for example:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> (...)
>>>>>>>>>>>>>>>>>>>> Resetting offset for partition
>>>>>>>>>>>>>>>>>>>> com.arquivei.dataeng.andre-0 to offset 10.
>>>>>>>>>>>>>>>>>>>> Resetting offset for partition
>>>>>>>>>>>>>>>>>>>> com.arquivei.dataeng.andre-0 to offset 15.
>>>>>>>>>>>>>>>>>>>> ERROR
>>>>>>>>>>>>>>>>>>>> Resetting offset for partition
>>>>>>>>>>>>>>>>>>>> com.arquivei.dataeng.andre-0 to offset 22.
>>>>>>>>>>>>>>>>>>>> Resetting offset for partition
>>>>>>>>>>>>>>>>>>>> com.arquivei.dataeng.andre-0 to offset 30.
>>>>>>>>>>>>>>>>>>>> (...)
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> But when checking the Kafka Consumer Group, the current
>>>>>>>>>>>>>>>>>>>> offset stays at 15, the commited offset from the last processed message,
>>>>>>>>>>>>>>>>>>>> before the error.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> We'll file a bug, but we could now reproduce the issue
>>>>>>>>>>>>>>>>>>>> in a Dev scenario.
>>>>>>>>>>>>>>>>>>>> We started the same pipeline using the direct runner,
>>>>>>>>>>>>>>>>>>>> without Google Dataflow. We blocked the Kafka Broker network and the same
>>>>>>>>>>>>>>>>>>>> error was thrown. Then we unblocked the network and the pipeline was able
>>>>>>>>>>>>>>>>>>>> to successfully process the subsequent messages.
>>>>>>>>>>>>>>>>>>>> When we started the same pipeline in the Dataflow
>>>>>>>>>>>>>>>>>>>> runner and did the same test, the same problem from our production scenario
>>>>>>>>>>>>>>>>>>>> happened, Dataflow couldn't process the new messages. Unfortunately, we've
>>>>>>>>>>>>>>>>>>>> stopped the dataflow job in production, but the problematic dev job is
>>>>>>>>>>>>>>>>>>>> still running and the log file of the VM is attached. Thank you very much.
>>>>>>>>>>>>>>>>>>>> Best regards
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Em ter, 11 de set de 2018 às 18:28, Raghu Angadi <
>>>>>>>>>>>>>>>>>>>> rangadi@google.com> escreveu:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Specifically, I am interested if you have any thread
>>>>>>>>>>>>>>>>>>>>> running 'consumerPollLoop()' [1]. There should always be one (if a worker
>>>>>>>>>>>>>>>>>>>>> is assigned one of the partitions). It is possible that KafkaClient itself
>>>>>>>>>>>>>>>>>>>>> is hasn't recovered from the group coordinator error (though unlikely).
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> https://github.com/apache/beam/blob/master/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaUnboundedReader.java#L570
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> On Tue, Sep 11, 2018 at 12:31 PM Raghu Angadi <
>>>>>>>>>>>>>>>>>>>>> rangadi@google.com> wrote:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Hi Eduardo,
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> In case of any error, the pipeline should keep on
>>>>>>>>>>>>>>>>>>>>>> trying to fetch. I don't know about this particular error. Do you see any
>>>>>>>>>>>>>>>>>>>>>> others afterwards in the log?
>>>>>>>>>>>>>>>>>>>>>> Couple of things you could try if the logs are not
>>>>>>>>>>>>>>>>>>>>>> useful :
>>>>>>>>>>>>>>>>>>>>>>  - login to one of the VMs and get stacktrace of java
>>>>>>>>>>>>>>>>>>>>>> worker (look for a container called java-streaming)
>>>>>>>>>>>>>>>>>>>>>>  - file a support bug or stackoverflow question with
>>>>>>>>>>>>>>>>>>>>>> jobid so that Dataflow oncall can take a look.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Raghu.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> On Tue, Sep 11, 2018 at 12:10 PM Eduardo Soldera <
>>>>>>>>>>>>>>>>>>>>>> eduardo.soldera@arquivei.com.br> wrote:
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>>>>>>>> We have a Apache Beam pipeline running in Google
>>>>>>>>>>>>>>>>>>>>>>> Dataflow using KafkaIO. Suddenly the pipeline stop fetching Kafka messages
>>>>>>>>>>>>>>>>>>>>>>> at all, as our other workers from other pipelines continued to get Kafka
>>>>>>>>>>>>>>>>>>>>>>> messages.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> At the moment it stopped we got these messages:
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> I  [Consumer clientId=consumer-1, groupId=genericPipe] Error sending fetch request (sessionId=1396189203, epoch=2431598) to node 3: org.apache.kafka.common.errors.DisconnectException.
>>>>>>>>>>>>>>>>>>>>>>> I  [Consumer clientId=consumer-1, groupId=genericPipe] Group coordinator 10.0.52.70:9093 (id: 2147483646 rack: null) is unavailable or invalid, will attempt rediscovery
>>>>>>>>>>>>>>>>>>>>>>> I  [Consumer clientId=consumer-1, groupId=genericPipe] Discovered group coordinator 10.0.52.70:9093 (id: 2147483646 rack: null)
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> And then the pipeline stopped reading the messages.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> This is the KafkaIO setup  we have:
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> KafkaIO.read[String,String]()
>>>>>>>>>>>>>>>>>>>>>>>   .withBootstrapServers(server)
>>>>>>>>>>>>>>>>>>>>>>>   .withTopic(topic)
>>>>>>>>>>>>>>>>>>>>>>>   .withKeyDeserializer(classOf[StringDeserializer])
>>>>>>>>>>>>>>>>>>>>>>>   .withValueDeserializer(classOf[StringDeserializer])
>>>>>>>>>>>>>>>>>>>>>>>   .updateConsumerProperties(properties)
>>>>>>>>>>>>>>>>>>>>>>>   .commitOffsetsInFinalize()
>>>>>>>>>>>>>>>>>>>>>>>   .withoutMetadata()
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>  Any help will be much appreciated.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Best regards,
>>>>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>>> Eduardo Soldera Garcia
>>>>>>>>>>>>>>>>>>>>>>> Data Engineer
>>>>>>>>>>>>>>>>>>>>>>> (16) 3509-5555 | www.arquivei.com.br
>>>>>>>>>>>>>>>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>>>>>>>>>>>>>>>>> [image: Arquivei.com.br – Inteligência em Notas
>>>>>>>>>>>>>>>>>>>>>>> Fiscais]
>>>>>>>>>>>>>>>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>>>>>>>>>>>>>>>>> [image: Google seleciona Arquivei para imersão e
>>>>>>>>>>>>>>>>>>>>>>> mentoria no Vale do Silício]
>>>>>>>>>>>>>>>>>>>>>>> <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad>
>>>>>>>>>>>>>>>>>>>>>>> <https://www.facebook.com/arquivei>
>>>>>>>>>>>>>>>>>>>>>>> <https://www.linkedin.com/company/arquivei>
>>>>>>>>>>>>>>>>>>>>>>> <https://www.youtube.com/watch?v=sSUUKxbXnxk>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>> Eduardo Soldera Garcia
>>>>>>>>>>>>>>>>>>>> Data Engineer
>>>>>>>>>>>>>>>>>>>> (16) 3509-5555 | www.arquivei.com.br
>>>>>>>>>>>>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>>>>>>>>>>>>>> [image: Arquivei.com.br – Inteligência em Notas Fiscais]
>>>>>>>>>>>>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>>>>>>>>>>>>>> [image: Google seleciona Arquivei para imersão e
>>>>>>>>>>>>>>>>>>>> mentoria no Vale do Silício]
>>>>>>>>>>>>>>>>>>>> <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad>
>>>>>>>>>>>>>>>>>>>> <https://www.facebook.com/arquivei>
>>>>>>>>>>>>>>>>>>>> <https://www.linkedin.com/company/arquivei>
>>>>>>>>>>>>>>>>>>>> <https://www.youtube.com/watch?v=sSUUKxbXnxk>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>> Eduardo Soldera Garcia
>>>>>>>>>>>>>>>>>> Data Engineer
>>>>>>>>>>>>>>>>>> (16) 3509-5555 | www.arquivei.com.br
>>>>>>>>>>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>>>>>>>>>>>> [image: Arquivei.com.br – Inteligência em Notas Fiscais]
>>>>>>>>>>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>>>>>>>>>>>> [image: Google seleciona Arquivei para imersão e mentoria
>>>>>>>>>>>>>>>>>> no Vale do Silício]
>>>>>>>>>>>>>>>>>> <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad>
>>>>>>>>>>>>>>>>>> <https://www.facebook.com/arquivei>
>>>>>>>>>>>>>>>>>> <https://www.linkedin.com/company/arquivei>
>>>>>>>>>>>>>>>>>> <https://www.youtube.com/watch?v=sSUUKxbXnxk>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>>> Eduardo Soldera Garcia
>>>>>>>>>>>>> Data Engineer
>>>>>>>>>>>>> (16) 3509-5555 | www.arquivei.com.br
>>>>>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>>>>>>> [image: Arquivei.com.br – Inteligência em Notas Fiscais]
>>>>>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>>>>>>> [image: Google seleciona Arquivei para imersão e mentoria no
>>>>>>>>>>>>> Vale do Silício]
>>>>>>>>>>>>> <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad>
>>>>>>>>>>>>> <https://www.facebook.com/arquivei>
>>>>>>>>>>>>> <https://www.linkedin.com/company/arquivei>
>>>>>>>>>>>>> <https://www.youtube.com/watch?v=sSUUKxbXnxk>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Eduardo Soldera Garcia
>>>>>>>>>>> Data Engineer
>>>>>>>>>>> (16) 3509-5555 | www.arquivei.com.br
>>>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>>>>> [image: Arquivei.com.br – Inteligência em Notas Fiscais]
>>>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>>>>> [image: Google seleciona Arquivei para imersão e mentoria no
>>>>>>>>>>> Vale do Silício]
>>>>>>>>>>> <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad>
>>>>>>>>>>> <https://www.facebook.com/arquivei>
>>>>>>>>>>> <https://www.linkedin.com/company/arquivei>
>>>>>>>>>>> <https://www.youtube.com/watch?v=sSUUKxbXnxk>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Eduardo Soldera Garcia
>>>>>>>>>> Data Engineer
>>>>>>>>>> (16) 3509-5555 | www.arquivei.com.br
>>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>>>> [image: Arquivei.com.br – Inteligência em Notas Fiscais]
>>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>>>> [image: Google seleciona Arquivei para imersão e mentoria no Vale
>>>>>>>>>> do Silício]
>>>>>>>>>> <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad>
>>>>>>>>>> <https://www.facebook.com/arquivei>
>>>>>>>>>> <https://www.linkedin.com/company/arquivei>
>>>>>>>>>> <https://www.youtube.com/watch?v=sSUUKxbXnxk>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Eduardo Soldera Garcia
>>>>>>>> Data Engineer
>>>>>>>> (16) 3509-5555 | www.arquivei.com.br
>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>> [image: Arquivei.com.br – Inteligência em Notas Fiscais]
>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>> [image: Google seleciona Arquivei para imersão e mentoria no Vale
>>>>>>>> do Silício]
>>>>>>>> <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad>
>>>>>>>> <https://www.facebook.com/arquivei>
>>>>>>>> <https://www.linkedin.com/company/arquivei>
>>>>>>>> <https://www.youtube.com/watch?v=sSUUKxbXnxk>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>> --
>>>>>> Eduardo Soldera Garcia
>>>>>> Data Engineer
>>>>>> (16) 3509-5555 | www.arquivei.com.br
>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>> [image: Arquivei.com.br – Inteligência em Notas Fiscais]
>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>> [image: Google seleciona Arquivei para imersão e mentoria no Vale do
>>>>>> Silício]
>>>>>> <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad>
>>>>>> <https://www.facebook.com/arquivei>
>>>>>> <https://www.linkedin.com/company/arquivei>
>>>>>> <https://www.youtube.com/watch?v=sSUUKxbXnxk>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>>
>>>>> JC
>>>>>
>>>>>

-- 
Eduardo Soldera Garcia
Data Engineer
(16) 3509-5555 | www.arquivei.com.br
<https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
[image: Arquivei.com.br – Inteligência em Notas Fiscais]
<https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
[image: Google seleciona Arquivei para imersão e mentoria no Vale do
Silício]
<https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad>
<https://www.facebook.com/arquivei>
<https://www.linkedin.com/company/arquivei>
<https://www.youtube.com/watch?v=sSUUKxbXnxk>

Re: Problem with KafkaIO

Posted by Raghu Angadi <ra...@google.com>.
On Wed, Sep 19, 2018 at 11:24 AM Juan Carlos Garcia <jc...@gmail.com>
wrote:

> Sorry I hit the send button to fast... The error occurs in the worker.
>

Np. Just one more comment on it: it is a very important design/correctness
decision to for runner to decide how to handle persistent errors in a
streaming pipeline. Dataflow keeps failing since there is no solution to
restart a pipeline from scratch without losing exactly-once guarantees. It
lets user decide if the pipeline needs to be 'upgraded'.

Raghu.

>
> Juan Carlos Garcia <jc...@gmail.com> schrieb am Mi., 19. Sep. 2018,
> 20:22:
>
>> Sorry for hijacking the thread, we are running Spark on top of Yarn, yarn
>> retries multiple times until it reachs it max attempt and then gives up.
>>
>> Raghu Angadi <ra...@google.com> schrieb am Mi., 19. Sep. 2018, 18:58:
>>
>>> On Wed, Sep 19, 2018 at 7:26 AM Juan Carlos Garcia <jc...@gmail.com>
>>> wrote:
>>>
>>>> Don't know if its related, but we have seen our pipeline dying (using
>>>> SparkRunner) when there is problem with Kafka  (network interruptions),
>>>> errors like:  org.apache.kafka.common.errors.TimeoutException: Timeout
>>>> expired while fetching topic metadata
>>>>
>>>> Maybe this will fix it as well, thanks Raghu for the hint about
>>>> *withConsumerFactoryFn.*
>>>>
>>>
>>> Wouldn't that be retried by the SparkRunner if it happens on the worker?
>>> or does it happen while launching the pipeline on the client?
>>>
>>>
>>>
>>>>
>>>>
>>>>
>>>> On Wed, Sep 19, 2018 at 3:29 PM Eduardo Soldera <
>>>> eduardo.soldera@arquivei.com.br> wrote:
>>>>
>>>>> Hi Raghu, thank you.
>>>>>
>>>>> I'm not sure though what to pass as an argument:
>>>>>
>>>>> KafkaIO.read[String,String]()
>>>>>   .withBootstrapServers(server)
>>>>>   .withTopic(topic)
>>>>>   .withKeyDeserializer(classOf[StringDeserializer])
>>>>>   .withValueDeserializer(classOf[StringDeserializer])
>>>>>   .withConsumerFactoryFn(new KafkaExecutor.ConsumerFactoryFn(????????????????))
>>>>>   .updateConsumerProperties(properties)
>>>>>   .commitOffsetsInFinalize()
>>>>>   .withoutMetadata()
>>>>>
>>>>>
>>>>> Regards
>>>>>
>>>>>
>>>>> Em ter, 18 de set de 2018 às 21:15, Raghu Angadi <ra...@google.com>
>>>>> escreveu:
>>>>>
>>>>>> Hi Eduardo,
>>>>>>
>>>>>> There another work around you can try without having to wait for
>>>>>> 2.7.0 release: Use a wrapper to catch exception from KafkaConsumer#poll()
>>>>>> and pass the wrapper to withConsumerFactoryFn() for KafkIO reader [1].
>>>>>>
>>>>>> Using something like (such a wrapper is used in KafkasIO tests [2]):
>>>>>> private static class ConsumerFactoryFn
>>>>>>                 implements SerializableFunction<Map<String, Object>,
>>>>>> Consumer<byte[], byte[]>> {
>>>>>>   @Override
>>>>>>     public Consumer<byte[], byte[]> apply(Map<String, Object> config)
>>>>>> {
>>>>>>       return new KafkaConsumer(config) {
>>>>>>           @Override
>>>>>>           public ConsumerRecords<K, V> poll(long timeout) {
>>>>>>               // work around for BEAM-5375
>>>>>>               while (true) {
>>>>>>                   try {
>>>>>>                     return super.poll(timeout);
>>>>>>                  } catch (Exception e) {
>>>>>>                     // LOG & sleep for sec
>>>>>>                 }
>>>>>>           }
>>>>>>        }
>>>>>>     }
>>>>>> }
>>>>>>
>>>>>> [1]:
>>>>>> https://github.com/apache/beam/blob/release-2.4.0/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaIO.java#L417
>>>>>> [2]:
>>>>>> https://github.com/apache/beam/blob/release-2.4.0/sdks/java/io/kafka/src/test/java/org/apache/beam/sdk/io/kafka/KafkaIOTest.java#L261
>>>>>>
>>>>>> On Tue, Sep 18, 2018 at 5:49 AM Eduardo Soldera <
>>>>>> eduardo.soldera@arquivei.com.br> wrote:
>>>>>>
>>>>>>> Hi Raghu, we're not sure how long the network was down. According to
>>>>>>> the logs no longer than one minute. A 30 second shutdown would work for the
>>>>>>> tests.
>>>>>>>
>>>>>>> Regards
>>>>>>>
>>>>>>> Em sex, 14 de set de 2018 às 21:41, Raghu Angadi <ra...@google.com>
>>>>>>> escreveu:
>>>>>>>
>>>>>>>> Thanks. I could repro myself as well. How long was the network
>>>>>>>> down?
>>>>>>>>
>>>>>>>> Trying to get the fix into 2.7 RC2.
>>>>>>>>
>>>>>>>> On Fri, Sep 14, 2018 at 12:25 PM Eduardo Soldera <
>>>>>>>> eduardo.soldera@arquivei.com.br> wrote:
>>>>>>>>
>>>>>>>>> Just to make myself clear, I'm not sure how to use the patch but
>>>>>>>>> if you could send us some guidance would be great.
>>>>>>>>>
>>>>>>>>> Em sex, 14 de set de 2018 às 16:24, Eduardo Soldera <
>>>>>>>>> eduardo.soldera@arquivei.com.br> escreveu:
>>>>>>>>>
>>>>>>>>>> Hi Raghu, yes, it is feasible, would you do that for us? I'm not
>>>>>>>>>> sure how we'd use the patch. We're using SBT and Spotify's Scio with Scala.
>>>>>>>>>>
>>>>>>>>>> Thanks
>>>>>>>>>>
>>>>>>>>>> Em sex, 14 de set de 2018 às 16:07, Raghu Angadi <
>>>>>>>>>> rangadi@google.com> escreveu:
>>>>>>>>>>
>>>>>>>>>>> Is is feasible for you to verify the fix in your dev job? I can
>>>>>>>>>>> make a patch against Beam 2.4 branch if you like.
>>>>>>>>>>>
>>>>>>>>>>> Raghu.
>>>>>>>>>>>
>>>>>>>>>>> On Fri, Sep 14, 2018 at 11:14 AM Eduardo Soldera <
>>>>>>>>>>> eduardo.soldera@arquivei.com.br> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi Raghu, thank you very much for the pull request.
>>>>>>>>>>>> We'll wait for the 2.7 Beam release.
>>>>>>>>>>>>
>>>>>>>>>>>> Regards!
>>>>>>>>>>>>
>>>>>>>>>>>> Em qui, 13 de set de 2018 às 18:19, Raghu Angadi <
>>>>>>>>>>>> rangadi@google.com> escreveu:
>>>>>>>>>>>>
>>>>>>>>>>>>> Fix: https://github.com/apache/beam/pull/6391
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Wed, Sep 12, 2018 at 3:30 PM Raghu Angadi <
>>>>>>>>>>>>> rangadi@google.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Filed BEAM-5375
>>>>>>>>>>>>>> <https://issues.apache.org/jira/browse/BEAM-5375>. I will
>>>>>>>>>>>>>> fix it later this week.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Wed, Sep 12, 2018 at 12:16 PM Raghu Angadi <
>>>>>>>>>>>>>> rangadi@google.com> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Wed, Sep 12, 2018 at 12:11 PM Raghu Angadi <
>>>>>>>>>>>>>>> rangadi@google.com> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Thanks for the job id, I looked at the worker logs
>>>>>>>>>>>>>>>> (following usual support oncall access protocol that provides temporary
>>>>>>>>>>>>>>>> access to things like logs in GCP):
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Root issue looks like consumerPollLoop() mentioned earlier
>>>>>>>>>>>>>>>> needs to handle unchecked exception. In your case it is clear that poll
>>>>>>>>>>>>>>>> thread exited with a runtime exception. The reader does not check for it
>>>>>>>>>>>>>>>> and continues to wait for poll thread to enqueue messages. A fix should
>>>>>>>>>>>>>>>> result in an IOException for read from the source. The runners will handle
>>>>>>>>>>>>>>>> that appropriately after that.  I will file a jira.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> https://github.com/apache/beam/blob/master/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaUnboundedReader.java#L678
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Ignore the link.. was pasted here by mistake.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> From the logs (with a comment below each one):
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>    - 2018-09-12 06:13:07.345 PDT Reader-0: reading from
>>>>>>>>>>>>>>>>    kafka_topic-0 starting at offset 2
>>>>>>>>>>>>>>>>       - Implies the reader is initialized and poll thread
>>>>>>>>>>>>>>>>       is started.
>>>>>>>>>>>>>>>>    - 2018-09-12 06:13:07.780 PDT Reader-0: first record
>>>>>>>>>>>>>>>>    offset 2
>>>>>>>>>>>>>>>>       - The reader actually got a message received by the
>>>>>>>>>>>>>>>>       poll thread from Kafka.
>>>>>>>>>>>>>>>>    - 2018-09-12 06:16:48.771 PDT Reader-0: exception while
>>>>>>>>>>>>>>>>    fetching latest offset for partition kafka_topic-0. will be retried.
>>>>>>>>>>>>>>>>       - This must have happened around the time when
>>>>>>>>>>>>>>>>       network was disrupted. This is from. Actual log is from another periodic
>>>>>>>>>>>>>>>>       task that fetches latest offsets for partitions.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> The poll thread must have died around the time network was
>>>>>>>>>>>>>>>> disrupted.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> The following log comes from kafka client itself and is
>>>>>>>>>>>>>>>> printed every second when KafkaIO fetches latest offset. This log seems to
>>>>>>>>>>>>>>>> be added in recent versions. It is probably an unintentional log. I don't
>>>>>>>>>>>>>>>> think there is any better to fetch latest offsets than how KafkaIO does
>>>>>>>>>>>>>>>> now. This is logged inside consumer.position() called at [1].
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>    - 2018-09-12 06:13:11.786 PDT [Consumer
>>>>>>>>>>>>>>>>    clientId=consumer-2,
>>>>>>>>>>>>>>>>    groupId=Reader-0_offset_consumer_1735388161_genericPipe] Resetting offset
>>>>>>>>>>>>>>>>    for partition com.arquivei.dataeng.andre-0 to offset 3.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> [1]:
>>>>>>>>>>>>>>>> https://github.com/apache/beam/blob/master/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaUnboundedReader.java#L678
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> This 'Resetting offset' is harmless, but is quite annoying
>>>>>>>>>>>>>>> to see in the worker logs. One way to avoid is to set kafka consumer's log
>>>>>>>>>>>>>>> level to WARNING. Ideally KafkaIO itself should do something to avoid it
>>>>>>>>>>>>>>> without user option.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Wed, Sep 12, 2018 at 10:27 AM Eduardo Soldera <
>>>>>>>>>>>>>>>> eduardo.soldera@arquivei.com.br> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Hi Raghu! The job_id of our dev job is
>>>>>>>>>>>>>>>>> 2018-09-12_06_11_48-5600553605191377866.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Thanks!
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Em qua, 12 de set de 2018 às 14:18, Raghu Angadi <
>>>>>>>>>>>>>>>>> rangadi@google.com> escreveu:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Thanks for debugging.
>>>>>>>>>>>>>>>>>> Can you provide the job_id of your dev job? The
>>>>>>>>>>>>>>>>>> stacktrace shows that there is no thread running 'consumerPollLoop()' which
>>>>>>>>>>>>>>>>>> can explain stuck reader. You will likely find a logs at line 594 & 587
>>>>>>>>>>>>>>>>>> [1].  Dataflow caches its readers and DirectRunner may not. That can
>>>>>>>>>>>>>>>>>> explain DirectRunner resume reads. The expectation in KafkaIO is that Kafka
>>>>>>>>>>>>>>>>>> client library takes care of retrying in case of connection problems (as
>>>>>>>>>>>>>>>>>> documented). It is possible that in some cases poll() throws and we need to
>>>>>>>>>>>>>>>>>> restart the client in KafkaIO.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> [1]:
>>>>>>>>>>>>>>>>>> https://github.com/apache/beam/blob/master/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaUnboundedReader.java#L594
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On Wed, Sep 12, 2018 at 9:59 AM Eduardo Soldera <
>>>>>>>>>>>>>>>>>> eduardo.soldera@arquivei.com.br> wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Hi Raghu, thanks for your help.
>>>>>>>>>>>>>>>>>>> Just answering your previous question, the following
>>>>>>>>>>>>>>>>>>> logs were the same as before the error, as if the pipeline were still
>>>>>>>>>>>>>>>>>>> getting the messages, for example:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> (...)
>>>>>>>>>>>>>>>>>>> Resetting offset for partition
>>>>>>>>>>>>>>>>>>> com.arquivei.dataeng.andre-0 to offset 10.
>>>>>>>>>>>>>>>>>>> Resetting offset for partition
>>>>>>>>>>>>>>>>>>> com.arquivei.dataeng.andre-0 to offset 15.
>>>>>>>>>>>>>>>>>>> ERROR
>>>>>>>>>>>>>>>>>>> Resetting offset for partition
>>>>>>>>>>>>>>>>>>> com.arquivei.dataeng.andre-0 to offset 22.
>>>>>>>>>>>>>>>>>>> Resetting offset for partition
>>>>>>>>>>>>>>>>>>> com.arquivei.dataeng.andre-0 to offset 30.
>>>>>>>>>>>>>>>>>>> (...)
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> But when checking the Kafka Consumer Group, the current
>>>>>>>>>>>>>>>>>>> offset stays at 15, the commited offset from the last processed message,
>>>>>>>>>>>>>>>>>>> before the error.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> We'll file a bug, but we could now reproduce the issue
>>>>>>>>>>>>>>>>>>> in a Dev scenario.
>>>>>>>>>>>>>>>>>>> We started the same pipeline using the direct runner,
>>>>>>>>>>>>>>>>>>> without Google Dataflow. We blocked the Kafka Broker network and the same
>>>>>>>>>>>>>>>>>>> error was thrown. Then we unblocked the network and the pipeline was able
>>>>>>>>>>>>>>>>>>> to successfully process the subsequent messages.
>>>>>>>>>>>>>>>>>>> When we started the same pipeline in the Dataflow runner
>>>>>>>>>>>>>>>>>>> and did the same test, the same problem from our production scenario
>>>>>>>>>>>>>>>>>>> happened, Dataflow couldn't process the new messages. Unfortunately, we've
>>>>>>>>>>>>>>>>>>> stopped the dataflow job in production, but the problematic dev job is
>>>>>>>>>>>>>>>>>>> still running and the log file of the VM is attached. Thank you very much.
>>>>>>>>>>>>>>>>>>> Best regards
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Em ter, 11 de set de 2018 às 18:28, Raghu Angadi <
>>>>>>>>>>>>>>>>>>> rangadi@google.com> escreveu:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Specifically, I am interested if you have any thread
>>>>>>>>>>>>>>>>>>>> running 'consumerPollLoop()' [1]. There should always be one (if a worker
>>>>>>>>>>>>>>>>>>>> is assigned one of the partitions). It is possible that KafkaClient itself
>>>>>>>>>>>>>>>>>>>> is hasn't recovered from the group coordinator error (though unlikely).
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> https://github.com/apache/beam/blob/master/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaUnboundedReader.java#L570
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> On Tue, Sep 11, 2018 at 12:31 PM Raghu Angadi <
>>>>>>>>>>>>>>>>>>>> rangadi@google.com> wrote:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Hi Eduardo,
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> In case of any error, the pipeline should keep on
>>>>>>>>>>>>>>>>>>>>> trying to fetch. I don't know about this particular error. Do you see any
>>>>>>>>>>>>>>>>>>>>> others afterwards in the log?
>>>>>>>>>>>>>>>>>>>>> Couple of things you could try if the logs are not
>>>>>>>>>>>>>>>>>>>>> useful :
>>>>>>>>>>>>>>>>>>>>>  - login to one of the VMs and get stacktrace of java
>>>>>>>>>>>>>>>>>>>>> worker (look for a container called java-streaming)
>>>>>>>>>>>>>>>>>>>>>  - file a support bug or stackoverflow question with
>>>>>>>>>>>>>>>>>>>>> jobid so that Dataflow oncall can take a look.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Raghu.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> On Tue, Sep 11, 2018 at 12:10 PM Eduardo Soldera <
>>>>>>>>>>>>>>>>>>>>> eduardo.soldera@arquivei.com.br> wrote:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>>>>>>> We have a Apache Beam pipeline running in Google
>>>>>>>>>>>>>>>>>>>>>> Dataflow using KafkaIO. Suddenly the pipeline stop fetching Kafka messages
>>>>>>>>>>>>>>>>>>>>>> at all, as our other workers from other pipelines continued to get Kafka
>>>>>>>>>>>>>>>>>>>>>> messages.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> At the moment it stopped we got these messages:
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> I  [Consumer clientId=consumer-1, groupId=genericPipe] Error sending fetch request (sessionId=1396189203, epoch=2431598) to node 3: org.apache.kafka.common.errors.DisconnectException.
>>>>>>>>>>>>>>>>>>>>>> I  [Consumer clientId=consumer-1, groupId=genericPipe] Group coordinator 10.0.52.70:9093 (id: 2147483646 rack: null) is unavailable or invalid, will attempt rediscovery
>>>>>>>>>>>>>>>>>>>>>> I  [Consumer clientId=consumer-1, groupId=genericPipe] Discovered group coordinator 10.0.52.70:9093 (id: 2147483646 rack: null)
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> And then the pipeline stopped reading the messages.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> This is the KafkaIO setup  we have:
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> KafkaIO.read[String,String]()
>>>>>>>>>>>>>>>>>>>>>>   .withBootstrapServers(server)
>>>>>>>>>>>>>>>>>>>>>>   .withTopic(topic)
>>>>>>>>>>>>>>>>>>>>>>   .withKeyDeserializer(classOf[StringDeserializer])
>>>>>>>>>>>>>>>>>>>>>>   .withValueDeserializer(classOf[StringDeserializer])
>>>>>>>>>>>>>>>>>>>>>>   .updateConsumerProperties(properties)
>>>>>>>>>>>>>>>>>>>>>>   .commitOffsetsInFinalize()
>>>>>>>>>>>>>>>>>>>>>>   .withoutMetadata()
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>  Any help will be much appreciated.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Best regards,
>>>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>>> Eduardo Soldera Garcia
>>>>>>>>>>>>>>>>>>>>>> Data Engineer
>>>>>>>>>>>>>>>>>>>>>> (16) 3509-5555 | www.arquivei.com.br
>>>>>>>>>>>>>>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>>>>>>>>>>>>>>>> [image: Arquivei.com.br – Inteligência em Notas
>>>>>>>>>>>>>>>>>>>>>> Fiscais]
>>>>>>>>>>>>>>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>>>>>>>>>>>>>>>> [image: Google seleciona Arquivei para imersão e
>>>>>>>>>>>>>>>>>>>>>> mentoria no Vale do Silício]
>>>>>>>>>>>>>>>>>>>>>> <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad>
>>>>>>>>>>>>>>>>>>>>>> <https://www.facebook.com/arquivei>
>>>>>>>>>>>>>>>>>>>>>> <https://www.linkedin.com/company/arquivei>
>>>>>>>>>>>>>>>>>>>>>> <https://www.youtube.com/watch?v=sSUUKxbXnxk>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>> Eduardo Soldera Garcia
>>>>>>>>>>>>>>>>>>> Data Engineer
>>>>>>>>>>>>>>>>>>> (16) 3509-5555 | www.arquivei.com.br
>>>>>>>>>>>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>>>>>>>>>>>>> [image: Arquivei.com.br – Inteligência em Notas Fiscais]
>>>>>>>>>>>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>>>>>>>>>>>>> [image: Google seleciona Arquivei para imersão e
>>>>>>>>>>>>>>>>>>> mentoria no Vale do Silício]
>>>>>>>>>>>>>>>>>>> <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad>
>>>>>>>>>>>>>>>>>>> <https://www.facebook.com/arquivei>
>>>>>>>>>>>>>>>>>>> <https://www.linkedin.com/company/arquivei>
>>>>>>>>>>>>>>>>>>> <https://www.youtube.com/watch?v=sSUUKxbXnxk>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>> Eduardo Soldera Garcia
>>>>>>>>>>>>>>>>> Data Engineer
>>>>>>>>>>>>>>>>> (16) 3509-5555 | www.arquivei.com.br
>>>>>>>>>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>>>>>>>>>>> [image: Arquivei.com.br – Inteligência em Notas Fiscais]
>>>>>>>>>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>>>>>>>>>>> [image: Google seleciona Arquivei para imersão e mentoria
>>>>>>>>>>>>>>>>> no Vale do Silício]
>>>>>>>>>>>>>>>>> <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad>
>>>>>>>>>>>>>>>>> <https://www.facebook.com/arquivei>
>>>>>>>>>>>>>>>>> <https://www.linkedin.com/company/arquivei>
>>>>>>>>>>>>>>>>> <https://www.youtube.com/watch?v=sSUUKxbXnxk>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> Eduardo Soldera Garcia
>>>>>>>>>>>> Data Engineer
>>>>>>>>>>>> (16) 3509-5555 | www.arquivei.com.br
>>>>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>>>>>> [image: Arquivei.com.br – Inteligência em Notas Fiscais]
>>>>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>>>>>> [image: Google seleciona Arquivei para imersão e mentoria no
>>>>>>>>>>>> Vale do Silício]
>>>>>>>>>>>> <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad>
>>>>>>>>>>>> <https://www.facebook.com/arquivei>
>>>>>>>>>>>> <https://www.linkedin.com/company/arquivei>
>>>>>>>>>>>> <https://www.youtube.com/watch?v=sSUUKxbXnxk>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Eduardo Soldera Garcia
>>>>>>>>>> Data Engineer
>>>>>>>>>> (16) 3509-5555 | www.arquivei.com.br
>>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>>>> [image: Arquivei.com.br – Inteligência em Notas Fiscais]
>>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>>>> [image: Google seleciona Arquivei para imersão e mentoria no Vale
>>>>>>>>>> do Silício]
>>>>>>>>>> <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad>
>>>>>>>>>> <https://www.facebook.com/arquivei>
>>>>>>>>>> <https://www.linkedin.com/company/arquivei>
>>>>>>>>>> <https://www.youtube.com/watch?v=sSUUKxbXnxk>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Eduardo Soldera Garcia
>>>>>>>>> Data Engineer
>>>>>>>>> (16) 3509-5555 | www.arquivei.com.br
>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>>> [image: Arquivei.com.br – Inteligência em Notas Fiscais]
>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>>> [image: Google seleciona Arquivei para imersão e mentoria no Vale
>>>>>>>>> do Silício]
>>>>>>>>> <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad>
>>>>>>>>> <https://www.facebook.com/arquivei>
>>>>>>>>> <https://www.linkedin.com/company/arquivei>
>>>>>>>>> <https://www.youtube.com/watch?v=sSUUKxbXnxk>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Eduardo Soldera Garcia
>>>>>>> Data Engineer
>>>>>>> (16) 3509-5555 | www.arquivei.com.br
>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>> [image: Arquivei.com.br – Inteligência em Notas Fiscais]
>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>> [image: Google seleciona Arquivei para imersão e mentoria no Vale do
>>>>>>> Silício]
>>>>>>> <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad>
>>>>>>> <https://www.facebook.com/arquivei>
>>>>>>> <https://www.linkedin.com/company/arquivei>
>>>>>>> <https://www.youtube.com/watch?v=sSUUKxbXnxk>
>>>>>>>
>>>>>>
>>>>>
>>>>> --
>>>>> Eduardo Soldera Garcia
>>>>> Data Engineer
>>>>> (16) 3509-5555 | www.arquivei.com.br
>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>> [image: Arquivei.com.br – Inteligência em Notas Fiscais]
>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>> [image: Google seleciona Arquivei para imersão e mentoria no Vale do
>>>>> Silício]
>>>>> <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad>
>>>>> <https://www.facebook.com/arquivei>
>>>>> <https://www.linkedin.com/company/arquivei>
>>>>> <https://www.youtube.com/watch?v=sSUUKxbXnxk>
>>>>>
>>>>
>>>>
>>>> --
>>>>
>>>> JC
>>>>
>>>>

Re: Problem with KafkaIO

Posted by Juan Carlos Garcia <jc...@gmail.com>.
Sorry I hit the send button to fast... The error occurs in the worker.

Juan Carlos Garcia <jc...@gmail.com> schrieb am Mi., 19. Sep. 2018,
20:22:

> Sorry for hijacking the thread, we are running Spark on top of Yarn, yarn
> retries multiple times until it reachs it max attempt and then gives up.
>
> Raghu Angadi <ra...@google.com> schrieb am Mi., 19. Sep. 2018, 18:58:
>
>> On Wed, Sep 19, 2018 at 7:26 AM Juan Carlos Garcia <jc...@gmail.com>
>> wrote:
>>
>>> Don't know if its related, but we have seen our pipeline dying (using
>>> SparkRunner) when there is problem with Kafka  (network interruptions),
>>> errors like:  org.apache.kafka.common.errors.TimeoutException: Timeout
>>> expired while fetching topic metadata
>>>
>>> Maybe this will fix it as well, thanks Raghu for the hint about
>>> *withConsumerFactoryFn.*
>>>
>>
>> Wouldn't that be retried by the SparkRunner if it happens on the worker?
>> or does it happen while launching the pipeline on the client?
>>
>>
>>
>>>
>>>
>>>
>>> On Wed, Sep 19, 2018 at 3:29 PM Eduardo Soldera <
>>> eduardo.soldera@arquivei.com.br> wrote:
>>>
>>>> Hi Raghu, thank you.
>>>>
>>>> I'm not sure though what to pass as an argument:
>>>>
>>>> KafkaIO.read[String,String]()
>>>>   .withBootstrapServers(server)
>>>>   .withTopic(topic)
>>>>   .withKeyDeserializer(classOf[StringDeserializer])
>>>>   .withValueDeserializer(classOf[StringDeserializer])
>>>>   .withConsumerFactoryFn(new KafkaExecutor.ConsumerFactoryFn(????????????????))
>>>>   .updateConsumerProperties(properties)
>>>>   .commitOffsetsInFinalize()
>>>>   .withoutMetadata()
>>>>
>>>>
>>>> Regards
>>>>
>>>>
>>>> Em ter, 18 de set de 2018 às 21:15, Raghu Angadi <ra...@google.com>
>>>> escreveu:
>>>>
>>>>> Hi Eduardo,
>>>>>
>>>>> There another work around you can try without having to wait for 2.7.0
>>>>> release: Use a wrapper to catch exception from KafkaConsumer#poll() and
>>>>> pass the wrapper to withConsumerFactoryFn() for KafkIO reader [1].
>>>>>
>>>>> Using something like (such a wrapper is used in KafkasIO tests [2]):
>>>>> private static class ConsumerFactoryFn
>>>>>                 implements SerializableFunction<Map<String, Object>,
>>>>> Consumer<byte[], byte[]>> {
>>>>>   @Override
>>>>>     public Consumer<byte[], byte[]> apply(Map<String, Object> config) {
>>>>>       return new KafkaConsumer(config) {
>>>>>           @Override
>>>>>           public ConsumerRecords<K, V> poll(long timeout) {
>>>>>               // work around for BEAM-5375
>>>>>               while (true) {
>>>>>                   try {
>>>>>                     return super.poll(timeout);
>>>>>                  } catch (Exception e) {
>>>>>                     // LOG & sleep for sec
>>>>>                 }
>>>>>           }
>>>>>        }
>>>>>     }
>>>>> }
>>>>>
>>>>> [1]:
>>>>> https://github.com/apache/beam/blob/release-2.4.0/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaIO.java#L417
>>>>> [2]:
>>>>> https://github.com/apache/beam/blob/release-2.4.0/sdks/java/io/kafka/src/test/java/org/apache/beam/sdk/io/kafka/KafkaIOTest.java#L261
>>>>>
>>>>> On Tue, Sep 18, 2018 at 5:49 AM Eduardo Soldera <
>>>>> eduardo.soldera@arquivei.com.br> wrote:
>>>>>
>>>>>> Hi Raghu, we're not sure how long the network was down. According to
>>>>>> the logs no longer than one minute. A 30 second shutdown would work for the
>>>>>> tests.
>>>>>>
>>>>>> Regards
>>>>>>
>>>>>> Em sex, 14 de set de 2018 às 21:41, Raghu Angadi <ra...@google.com>
>>>>>> escreveu:
>>>>>>
>>>>>>> Thanks. I could repro myself as well. How long was the network down?
>>>>>>>
>>>>>>> Trying to get the fix into 2.7 RC2.
>>>>>>>
>>>>>>> On Fri, Sep 14, 2018 at 12:25 PM Eduardo Soldera <
>>>>>>> eduardo.soldera@arquivei.com.br> wrote:
>>>>>>>
>>>>>>>> Just to make myself clear, I'm not sure how to use the patch but if
>>>>>>>> you could send us some guidance would be great.
>>>>>>>>
>>>>>>>> Em sex, 14 de set de 2018 às 16:24, Eduardo Soldera <
>>>>>>>> eduardo.soldera@arquivei.com.br> escreveu:
>>>>>>>>
>>>>>>>>> Hi Raghu, yes, it is feasible, would you do that for us? I'm not
>>>>>>>>> sure how we'd use the patch. We're using SBT and Spotify's Scio with Scala.
>>>>>>>>>
>>>>>>>>> Thanks
>>>>>>>>>
>>>>>>>>> Em sex, 14 de set de 2018 às 16:07, Raghu Angadi <
>>>>>>>>> rangadi@google.com> escreveu:
>>>>>>>>>
>>>>>>>>>> Is is feasible for you to verify the fix in your dev job? I can
>>>>>>>>>> make a patch against Beam 2.4 branch if you like.
>>>>>>>>>>
>>>>>>>>>> Raghu.
>>>>>>>>>>
>>>>>>>>>> On Fri, Sep 14, 2018 at 11:14 AM Eduardo Soldera <
>>>>>>>>>> eduardo.soldera@arquivei.com.br> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi Raghu, thank you very much for the pull request.
>>>>>>>>>>> We'll wait for the 2.7 Beam release.
>>>>>>>>>>>
>>>>>>>>>>> Regards!
>>>>>>>>>>>
>>>>>>>>>>> Em qui, 13 de set de 2018 às 18:19, Raghu Angadi <
>>>>>>>>>>> rangadi@google.com> escreveu:
>>>>>>>>>>>
>>>>>>>>>>>> Fix: https://github.com/apache/beam/pull/6391
>>>>>>>>>>>>
>>>>>>>>>>>> On Wed, Sep 12, 2018 at 3:30 PM Raghu Angadi <
>>>>>>>>>>>> rangadi@google.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Filed BEAM-5375
>>>>>>>>>>>>> <https://issues.apache.org/jira/browse/BEAM-5375>. I will fix
>>>>>>>>>>>>> it later this week.
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Wed, Sep 12, 2018 at 12:16 PM Raghu Angadi <
>>>>>>>>>>>>> rangadi@google.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Wed, Sep 12, 2018 at 12:11 PM Raghu Angadi <
>>>>>>>>>>>>>> rangadi@google.com> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks for the job id, I looked at the worker logs
>>>>>>>>>>>>>>> (following usual support oncall access protocol that provides temporary
>>>>>>>>>>>>>>> access to things like logs in GCP):
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Root issue looks like consumerPollLoop() mentioned earlier
>>>>>>>>>>>>>>> needs to handle unchecked exception. In your case it is clear that poll
>>>>>>>>>>>>>>> thread exited with a runtime exception. The reader does not check for it
>>>>>>>>>>>>>>> and continues to wait for poll thread to enqueue messages. A fix should
>>>>>>>>>>>>>>> result in an IOException for read from the source. The runners will handle
>>>>>>>>>>>>>>> that appropriately after that.  I will file a jira.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> https://github.com/apache/beam/blob/master/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaUnboundedReader.java#L678
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Ignore the link.. was pasted here by mistake.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> From the logs (with a comment below each one):
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>    - 2018-09-12 06:13:07.345 PDT Reader-0: reading from
>>>>>>>>>>>>>>>    kafka_topic-0 starting at offset 2
>>>>>>>>>>>>>>>       - Implies the reader is initialized and poll thread
>>>>>>>>>>>>>>>       is started.
>>>>>>>>>>>>>>>    - 2018-09-12 06:13:07.780 PDT Reader-0: first record
>>>>>>>>>>>>>>>    offset 2
>>>>>>>>>>>>>>>       - The reader actually got a message received by the
>>>>>>>>>>>>>>>       poll thread from Kafka.
>>>>>>>>>>>>>>>    - 2018-09-12 06:16:48.771 PDT Reader-0: exception while
>>>>>>>>>>>>>>>    fetching latest offset for partition kafka_topic-0. will be retried.
>>>>>>>>>>>>>>>       - This must have happened around the time when
>>>>>>>>>>>>>>>       network was disrupted. This is from. Actual log is from another periodic
>>>>>>>>>>>>>>>       task that fetches latest offsets for partitions.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> The poll thread must have died around the time network was
>>>>>>>>>>>>>>> disrupted.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> The following log comes from kafka client itself and is
>>>>>>>>>>>>>>> printed every second when KafkaIO fetches latest offset. This log seems to
>>>>>>>>>>>>>>> be added in recent versions. It is probably an unintentional log. I don't
>>>>>>>>>>>>>>> think there is any better to fetch latest offsets than how KafkaIO does
>>>>>>>>>>>>>>> now. This is logged inside consumer.position() called at [1].
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>    - 2018-09-12 06:13:11.786 PDT [Consumer
>>>>>>>>>>>>>>>    clientId=consumer-2,
>>>>>>>>>>>>>>>    groupId=Reader-0_offset_consumer_1735388161_genericPipe] Resetting offset
>>>>>>>>>>>>>>>    for partition com.arquivei.dataeng.andre-0 to offset 3.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> [1]:
>>>>>>>>>>>>>>> https://github.com/apache/beam/blob/master/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaUnboundedReader.java#L678
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> This 'Resetting offset' is harmless, but is quite annoying to
>>>>>>>>>>>>>> see in the worker logs. One way to avoid is to set kafka consumer's log
>>>>>>>>>>>>>> level to WARNING. Ideally KafkaIO itself should do something to avoid it
>>>>>>>>>>>>>> without user option.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Wed, Sep 12, 2018 at 10:27 AM Eduardo Soldera <
>>>>>>>>>>>>>>> eduardo.soldera@arquivei.com.br> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hi Raghu! The job_id of our dev job is
>>>>>>>>>>>>>>>> 2018-09-12_06_11_48-5600553605191377866.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Thanks!
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Em qua, 12 de set de 2018 às 14:18, Raghu Angadi <
>>>>>>>>>>>>>>>> rangadi@google.com> escreveu:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Thanks for debugging.
>>>>>>>>>>>>>>>>> Can you provide the job_id of your dev job? The stacktrace
>>>>>>>>>>>>>>>>> shows that there is no thread running 'consumerPollLoop()' which can
>>>>>>>>>>>>>>>>> explain stuck reader. You will likely find a logs at line 594 & 587 [1].
>>>>>>>>>>>>>>>>> Dataflow caches its readers and DirectRunner may not. That can explain
>>>>>>>>>>>>>>>>> DirectRunner resume reads. The expectation in KafkaIO is that Kafka client
>>>>>>>>>>>>>>>>> library takes care of retrying in case of connection problems (as
>>>>>>>>>>>>>>>>> documented). It is possible that in some cases poll() throws and we need to
>>>>>>>>>>>>>>>>> restart the client in KafkaIO.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> [1]:
>>>>>>>>>>>>>>>>> https://github.com/apache/beam/blob/master/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaUnboundedReader.java#L594
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Wed, Sep 12, 2018 at 9:59 AM Eduardo Soldera <
>>>>>>>>>>>>>>>>> eduardo.soldera@arquivei.com.br> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Hi Raghu, thanks for your help.
>>>>>>>>>>>>>>>>>> Just answering your previous question, the following logs
>>>>>>>>>>>>>>>>>> were the same as before the error, as if the pipeline were still getting
>>>>>>>>>>>>>>>>>> the messages, for example:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> (...)
>>>>>>>>>>>>>>>>>> Resetting offset for partition
>>>>>>>>>>>>>>>>>> com.arquivei.dataeng.andre-0 to offset 10.
>>>>>>>>>>>>>>>>>> Resetting offset for partition
>>>>>>>>>>>>>>>>>> com.arquivei.dataeng.andre-0 to offset 15.
>>>>>>>>>>>>>>>>>> ERROR
>>>>>>>>>>>>>>>>>> Resetting offset for partition
>>>>>>>>>>>>>>>>>> com.arquivei.dataeng.andre-0 to offset 22.
>>>>>>>>>>>>>>>>>> Resetting offset for partition
>>>>>>>>>>>>>>>>>> com.arquivei.dataeng.andre-0 to offset 30.
>>>>>>>>>>>>>>>>>> (...)
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> But when checking the Kafka Consumer Group, the current
>>>>>>>>>>>>>>>>>> offset stays at 15, the commited offset from the last processed message,
>>>>>>>>>>>>>>>>>> before the error.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> We'll file a bug, but we could now reproduce the issue in
>>>>>>>>>>>>>>>>>> a Dev scenario.
>>>>>>>>>>>>>>>>>> We started the same pipeline using the direct runner,
>>>>>>>>>>>>>>>>>> without Google Dataflow. We blocked the Kafka Broker network and the same
>>>>>>>>>>>>>>>>>> error was thrown. Then we unblocked the network and the pipeline was able
>>>>>>>>>>>>>>>>>> to successfully process the subsequent messages.
>>>>>>>>>>>>>>>>>> When we started the same pipeline in the Dataflow runner
>>>>>>>>>>>>>>>>>> and did the same test, the same problem from our production scenario
>>>>>>>>>>>>>>>>>> happened, Dataflow couldn't process the new messages. Unfortunately, we've
>>>>>>>>>>>>>>>>>> stopped the dataflow job in production, but the problematic dev job is
>>>>>>>>>>>>>>>>>> still running and the log file of the VM is attached. Thank you very much.
>>>>>>>>>>>>>>>>>> Best regards
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Em ter, 11 de set de 2018 às 18:28, Raghu Angadi <
>>>>>>>>>>>>>>>>>> rangadi@google.com> escreveu:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Specifically, I am interested if you have any thread
>>>>>>>>>>>>>>>>>>> running 'consumerPollLoop()' [1]. There should always be one (if a worker
>>>>>>>>>>>>>>>>>>> is assigned one of the partitions). It is possible that KafkaClient itself
>>>>>>>>>>>>>>>>>>> is hasn't recovered from the group coordinator error (though unlikely).
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> https://github.com/apache/beam/blob/master/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaUnboundedReader.java#L570
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> On Tue, Sep 11, 2018 at 12:31 PM Raghu Angadi <
>>>>>>>>>>>>>>>>>>> rangadi@google.com> wrote:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Hi Eduardo,
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> In case of any error, the pipeline should keep on
>>>>>>>>>>>>>>>>>>>> trying to fetch. I don't know about this particular error. Do you see any
>>>>>>>>>>>>>>>>>>>> others afterwards in the log?
>>>>>>>>>>>>>>>>>>>> Couple of things you could try if the logs are not
>>>>>>>>>>>>>>>>>>>> useful :
>>>>>>>>>>>>>>>>>>>>  - login to one of the VMs and get stacktrace of java
>>>>>>>>>>>>>>>>>>>> worker (look for a container called java-streaming)
>>>>>>>>>>>>>>>>>>>>  - file a support bug or stackoverflow question with
>>>>>>>>>>>>>>>>>>>> jobid so that Dataflow oncall can take a look.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Raghu.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> On Tue, Sep 11, 2018 at 12:10 PM Eduardo Soldera <
>>>>>>>>>>>>>>>>>>>> eduardo.soldera@arquivei.com.br> wrote:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>>>>>> We have a Apache Beam pipeline running in Google
>>>>>>>>>>>>>>>>>>>>> Dataflow using KafkaIO. Suddenly the pipeline stop fetching Kafka messages
>>>>>>>>>>>>>>>>>>>>> at all, as our other workers from other pipelines continued to get Kafka
>>>>>>>>>>>>>>>>>>>>> messages.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> At the moment it stopped we got these messages:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> I  [Consumer clientId=consumer-1, groupId=genericPipe] Error sending fetch request (sessionId=1396189203, epoch=2431598) to node 3: org.apache.kafka.common.errors.DisconnectException.
>>>>>>>>>>>>>>>>>>>>> I  [Consumer clientId=consumer-1, groupId=genericPipe] Group coordinator 10.0.52.70:9093 (id: 2147483646 rack: null) is unavailable or invalid, will attempt rediscovery
>>>>>>>>>>>>>>>>>>>>> I  [Consumer clientId=consumer-1, groupId=genericPipe] Discovered group coordinator 10.0.52.70:9093 (id: 2147483646 rack: null)
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> And then the pipeline stopped reading the messages.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> This is the KafkaIO setup  we have:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> KafkaIO.read[String,String]()
>>>>>>>>>>>>>>>>>>>>>   .withBootstrapServers(server)
>>>>>>>>>>>>>>>>>>>>>   .withTopic(topic)
>>>>>>>>>>>>>>>>>>>>>   .withKeyDeserializer(classOf[StringDeserializer])
>>>>>>>>>>>>>>>>>>>>>   .withValueDeserializer(classOf[StringDeserializer])
>>>>>>>>>>>>>>>>>>>>>   .updateConsumerProperties(properties)
>>>>>>>>>>>>>>>>>>>>>   .commitOffsetsInFinalize()
>>>>>>>>>>>>>>>>>>>>>   .withoutMetadata()
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>  Any help will be much appreciated.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Best regards,
>>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>> Eduardo Soldera Garcia
>>>>>>>>>>>>>>>>>>>>> Data Engineer
>>>>>>>>>>>>>>>>>>>>> (16) 3509-5555 | www.arquivei.com.br
>>>>>>>>>>>>>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>>>>>>>>>>>>>>> [image: Arquivei.com.br – Inteligência em Notas
>>>>>>>>>>>>>>>>>>>>> Fiscais]
>>>>>>>>>>>>>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>>>>>>>>>>>>>>> [image: Google seleciona Arquivei para imersão e
>>>>>>>>>>>>>>>>>>>>> mentoria no Vale do Silício]
>>>>>>>>>>>>>>>>>>>>> <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad>
>>>>>>>>>>>>>>>>>>>>> <https://www.facebook.com/arquivei>
>>>>>>>>>>>>>>>>>>>>> <https://www.linkedin.com/company/arquivei>
>>>>>>>>>>>>>>>>>>>>> <https://www.youtube.com/watch?v=sSUUKxbXnxk>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>> Eduardo Soldera Garcia
>>>>>>>>>>>>>>>>>> Data Engineer
>>>>>>>>>>>>>>>>>> (16) 3509-5555 | www.arquivei.com.br
>>>>>>>>>>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>>>>>>>>>>>> [image: Arquivei.com.br – Inteligência em Notas Fiscais]
>>>>>>>>>>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>>>>>>>>>>>> [image: Google seleciona Arquivei para imersão e mentoria
>>>>>>>>>>>>>>>>>> no Vale do Silício]
>>>>>>>>>>>>>>>>>> <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad>
>>>>>>>>>>>>>>>>>> <https://www.facebook.com/arquivei>
>>>>>>>>>>>>>>>>>> <https://www.linkedin.com/company/arquivei>
>>>>>>>>>>>>>>>>>> <https://www.youtube.com/watch?v=sSUUKxbXnxk>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>> Eduardo Soldera Garcia
>>>>>>>>>>>>>>>> Data Engineer
>>>>>>>>>>>>>>>> (16) 3509-5555 | www.arquivei.com.br
>>>>>>>>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>>>>>>>>>> [image: Arquivei.com.br – Inteligência em Notas Fiscais]
>>>>>>>>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>>>>>>>>>> [image: Google seleciona Arquivei para imersão e mentoria
>>>>>>>>>>>>>>>> no Vale do Silício]
>>>>>>>>>>>>>>>> <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad>
>>>>>>>>>>>>>>>> <https://www.facebook.com/arquivei>
>>>>>>>>>>>>>>>> <https://www.linkedin.com/company/arquivei>
>>>>>>>>>>>>>>>> <https://www.youtube.com/watch?v=sSUUKxbXnxk>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Eduardo Soldera Garcia
>>>>>>>>>>> Data Engineer
>>>>>>>>>>> (16) 3509-5555 | www.arquivei.com.br
>>>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>>>>> [image: Arquivei.com.br – Inteligência em Notas Fiscais]
>>>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>>>>> [image: Google seleciona Arquivei para imersão e mentoria no
>>>>>>>>>>> Vale do Silício]
>>>>>>>>>>> <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad>
>>>>>>>>>>> <https://www.facebook.com/arquivei>
>>>>>>>>>>> <https://www.linkedin.com/company/arquivei>
>>>>>>>>>>> <https://www.youtube.com/watch?v=sSUUKxbXnxk>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Eduardo Soldera Garcia
>>>>>>>>> Data Engineer
>>>>>>>>> (16) 3509-5555 | www.arquivei.com.br
>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>>> [image: Arquivei.com.br – Inteligência em Notas Fiscais]
>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>>> [image: Google seleciona Arquivei para imersão e mentoria no Vale
>>>>>>>>> do Silício]
>>>>>>>>> <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad>
>>>>>>>>> <https://www.facebook.com/arquivei>
>>>>>>>>> <https://www.linkedin.com/company/arquivei>
>>>>>>>>> <https://www.youtube.com/watch?v=sSUUKxbXnxk>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Eduardo Soldera Garcia
>>>>>>>> Data Engineer
>>>>>>>> (16) 3509-5555 | www.arquivei.com.br
>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>> [image: Arquivei.com.br – Inteligência em Notas Fiscais]
>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>> [image: Google seleciona Arquivei para imersão e mentoria no Vale
>>>>>>>> do Silício]
>>>>>>>> <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad>
>>>>>>>> <https://www.facebook.com/arquivei>
>>>>>>>> <https://www.linkedin.com/company/arquivei>
>>>>>>>> <https://www.youtube.com/watch?v=sSUUKxbXnxk>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>> --
>>>>>> Eduardo Soldera Garcia
>>>>>> Data Engineer
>>>>>> (16) 3509-5555 | www.arquivei.com.br
>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>> [image: Arquivei.com.br – Inteligência em Notas Fiscais]
>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>> [image: Google seleciona Arquivei para imersão e mentoria no Vale do
>>>>>> Silício]
>>>>>> <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad>
>>>>>> <https://www.facebook.com/arquivei>
>>>>>> <https://www.linkedin.com/company/arquivei>
>>>>>> <https://www.youtube.com/watch?v=sSUUKxbXnxk>
>>>>>>
>>>>>
>>>>
>>>> --
>>>> Eduardo Soldera Garcia
>>>> Data Engineer
>>>> (16) 3509-5555 | www.arquivei.com.br
>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>> [image: Arquivei.com.br – Inteligência em Notas Fiscais]
>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>> [image: Google seleciona Arquivei para imersão e mentoria no Vale do
>>>> Silício]
>>>> <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad>
>>>> <https://www.facebook.com/arquivei>
>>>> <https://www.linkedin.com/company/arquivei>
>>>> <https://www.youtube.com/watch?v=sSUUKxbXnxk>
>>>>
>>>
>>>
>>> --
>>>
>>> JC
>>>
>>>

Re: Problem with KafkaIO

Posted by Juan Carlos Garcia <jc...@gmail.com>.
Sorry for hijacking the thread, we are running Spark on top of Yarn, yarn
retries multiple times until it reachs it max attempt and then gives up.

Raghu Angadi <ra...@google.com> schrieb am Mi., 19. Sep. 2018, 18:58:

> On Wed, Sep 19, 2018 at 7:26 AM Juan Carlos Garcia <jc...@gmail.com>
> wrote:
>
>> Don't know if its related, but we have seen our pipeline dying (using
>> SparkRunner) when there is problem with Kafka  (network interruptions),
>> errors like:  org.apache.kafka.common.errors.TimeoutException: Timeout
>> expired while fetching topic metadata
>>
>> Maybe this will fix it as well, thanks Raghu for the hint about
>> *withConsumerFactoryFn.*
>>
>
> Wouldn't that be retried by the SparkRunner if it happens on the worker?
> or does it happen while launching the pipeline on the client?
>
>
>
>>
>>
>>
>> On Wed, Sep 19, 2018 at 3:29 PM Eduardo Soldera <
>> eduardo.soldera@arquivei.com.br> wrote:
>>
>>> Hi Raghu, thank you.
>>>
>>> I'm not sure though what to pass as an argument:
>>>
>>> KafkaIO.read[String,String]()
>>>   .withBootstrapServers(server)
>>>   .withTopic(topic)
>>>   .withKeyDeserializer(classOf[StringDeserializer])
>>>   .withValueDeserializer(classOf[StringDeserializer])
>>>   .withConsumerFactoryFn(new KafkaExecutor.ConsumerFactoryFn(????????????????))
>>>   .updateConsumerProperties(properties)
>>>   .commitOffsetsInFinalize()
>>>   .withoutMetadata()
>>>
>>>
>>> Regards
>>>
>>>
>>> Em ter, 18 de set de 2018 às 21:15, Raghu Angadi <ra...@google.com>
>>> escreveu:
>>>
>>>> Hi Eduardo,
>>>>
>>>> There another work around you can try without having to wait for 2.7.0
>>>> release: Use a wrapper to catch exception from KafkaConsumer#poll() and
>>>> pass the wrapper to withConsumerFactoryFn() for KafkIO reader [1].
>>>>
>>>> Using something like (such a wrapper is used in KafkasIO tests [2]):
>>>> private static class ConsumerFactoryFn
>>>>                 implements SerializableFunction<Map<String, Object>,
>>>> Consumer<byte[], byte[]>> {
>>>>   @Override
>>>>     public Consumer<byte[], byte[]> apply(Map<String, Object> config) {
>>>>       return new KafkaConsumer(config) {
>>>>           @Override
>>>>           public ConsumerRecords<K, V> poll(long timeout) {
>>>>               // work around for BEAM-5375
>>>>               while (true) {
>>>>                   try {
>>>>                     return super.poll(timeout);
>>>>                  } catch (Exception e) {
>>>>                     // LOG & sleep for sec
>>>>                 }
>>>>           }
>>>>        }
>>>>     }
>>>> }
>>>>
>>>> [1]:
>>>> https://github.com/apache/beam/blob/release-2.4.0/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaIO.java#L417
>>>> [2]:
>>>> https://github.com/apache/beam/blob/release-2.4.0/sdks/java/io/kafka/src/test/java/org/apache/beam/sdk/io/kafka/KafkaIOTest.java#L261
>>>>
>>>> On Tue, Sep 18, 2018 at 5:49 AM Eduardo Soldera <
>>>> eduardo.soldera@arquivei.com.br> wrote:
>>>>
>>>>> Hi Raghu, we're not sure how long the network was down. According to
>>>>> the logs no longer than one minute. A 30 second shutdown would work for the
>>>>> tests.
>>>>>
>>>>> Regards
>>>>>
>>>>> Em sex, 14 de set de 2018 às 21:41, Raghu Angadi <ra...@google.com>
>>>>> escreveu:
>>>>>
>>>>>> Thanks. I could repro myself as well. How long was the network down?
>>>>>>
>>>>>> Trying to get the fix into 2.7 RC2.
>>>>>>
>>>>>> On Fri, Sep 14, 2018 at 12:25 PM Eduardo Soldera <
>>>>>> eduardo.soldera@arquivei.com.br> wrote:
>>>>>>
>>>>>>> Just to make myself clear, I'm not sure how to use the patch but if
>>>>>>> you could send us some guidance would be great.
>>>>>>>
>>>>>>> Em sex, 14 de set de 2018 às 16:24, Eduardo Soldera <
>>>>>>> eduardo.soldera@arquivei.com.br> escreveu:
>>>>>>>
>>>>>>>> Hi Raghu, yes, it is feasible, would you do that for us? I'm not
>>>>>>>> sure how we'd use the patch. We're using SBT and Spotify's Scio with Scala.
>>>>>>>>
>>>>>>>> Thanks
>>>>>>>>
>>>>>>>> Em sex, 14 de set de 2018 às 16:07, Raghu Angadi <
>>>>>>>> rangadi@google.com> escreveu:
>>>>>>>>
>>>>>>>>> Is is feasible for you to verify the fix in your dev job? I can
>>>>>>>>> make a patch against Beam 2.4 branch if you like.
>>>>>>>>>
>>>>>>>>> Raghu.
>>>>>>>>>
>>>>>>>>> On Fri, Sep 14, 2018 at 11:14 AM Eduardo Soldera <
>>>>>>>>> eduardo.soldera@arquivei.com.br> wrote:
>>>>>>>>>
>>>>>>>>>> Hi Raghu, thank you very much for the pull request.
>>>>>>>>>> We'll wait for the 2.7 Beam release.
>>>>>>>>>>
>>>>>>>>>> Regards!
>>>>>>>>>>
>>>>>>>>>> Em qui, 13 de set de 2018 às 18:19, Raghu Angadi <
>>>>>>>>>> rangadi@google.com> escreveu:
>>>>>>>>>>
>>>>>>>>>>> Fix: https://github.com/apache/beam/pull/6391
>>>>>>>>>>>
>>>>>>>>>>> On Wed, Sep 12, 2018 at 3:30 PM Raghu Angadi <ra...@google.com>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Filed BEAM-5375
>>>>>>>>>>>> <https://issues.apache.org/jira/browse/BEAM-5375>. I will fix
>>>>>>>>>>>> it later this week.
>>>>>>>>>>>>
>>>>>>>>>>>> On Wed, Sep 12, 2018 at 12:16 PM Raghu Angadi <
>>>>>>>>>>>> rangadi@google.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Wed, Sep 12, 2018 at 12:11 PM Raghu Angadi <
>>>>>>>>>>>>> rangadi@google.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks for the job id, I looked at the worker logs (following
>>>>>>>>>>>>>> usual support oncall access protocol that provides temporary access to
>>>>>>>>>>>>>> things like logs in GCP):
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Root issue looks like consumerPollLoop() mentioned earlier
>>>>>>>>>>>>>> needs to handle unchecked exception. In your case it is clear that poll
>>>>>>>>>>>>>> thread exited with a runtime exception. The reader does not check for it
>>>>>>>>>>>>>> and continues to wait for poll thread to enqueue messages. A fix should
>>>>>>>>>>>>>> result in an IOException for read from the source. The runners will handle
>>>>>>>>>>>>>> that appropriately after that.  I will file a jira.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> https://github.com/apache/beam/blob/master/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaUnboundedReader.java#L678
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Ignore the link.. was pasted here by mistake.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> From the logs (with a comment below each one):
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>    - 2018-09-12 06:13:07.345 PDT Reader-0: reading from
>>>>>>>>>>>>>>    kafka_topic-0 starting at offset 2
>>>>>>>>>>>>>>       - Implies the reader is initialized and poll thread is
>>>>>>>>>>>>>>       started.
>>>>>>>>>>>>>>    - 2018-09-12 06:13:07.780 PDT Reader-0: first record
>>>>>>>>>>>>>>    offset 2
>>>>>>>>>>>>>>       - The reader actually got a message received by the
>>>>>>>>>>>>>>       poll thread from Kafka.
>>>>>>>>>>>>>>    - 2018-09-12 06:16:48.771 PDT Reader-0: exception while
>>>>>>>>>>>>>>    fetching latest offset for partition kafka_topic-0. will be retried.
>>>>>>>>>>>>>>       - This must have happened around the time when network
>>>>>>>>>>>>>>       was disrupted. This is from. Actual log is from another periodic task that
>>>>>>>>>>>>>>       fetches latest offsets for partitions.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> The poll thread must have died around the time network was
>>>>>>>>>>>>>> disrupted.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> The following log comes from kafka client itself and is
>>>>>>>>>>>>>> printed every second when KafkaIO fetches latest offset. This log seems to
>>>>>>>>>>>>>> be added in recent versions. It is probably an unintentional log. I don't
>>>>>>>>>>>>>> think there is any better to fetch latest offsets than how KafkaIO does
>>>>>>>>>>>>>> now. This is logged inside consumer.position() called at [1].
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>    - 2018-09-12 06:13:11.786 PDT [Consumer
>>>>>>>>>>>>>>    clientId=consumer-2,
>>>>>>>>>>>>>>    groupId=Reader-0_offset_consumer_1735388161_genericPipe] Resetting offset
>>>>>>>>>>>>>>    for partition com.arquivei.dataeng.andre-0 to offset 3.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> [1]:
>>>>>>>>>>>>>> https://github.com/apache/beam/blob/master/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaUnboundedReader.java#L678
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> This 'Resetting offset' is harmless, but is quite annoying to
>>>>>>>>>>>>> see in the worker logs. One way to avoid is to set kafka consumer's log
>>>>>>>>>>>>> level to WARNING. Ideally KafkaIO itself should do something to avoid it
>>>>>>>>>>>>> without user option.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Wed, Sep 12, 2018 at 10:27 AM Eduardo Soldera <
>>>>>>>>>>>>>> eduardo.soldera@arquivei.com.br> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi Raghu! The job_id of our dev job is
>>>>>>>>>>>>>>> 2018-09-12_06_11_48-5600553605191377866.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks!
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Em qua, 12 de set de 2018 às 14:18, Raghu Angadi <
>>>>>>>>>>>>>>> rangadi@google.com> escreveu:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Thanks for debugging.
>>>>>>>>>>>>>>>> Can you provide the job_id of your dev job? The stacktrace
>>>>>>>>>>>>>>>> shows that there is no thread running 'consumerPollLoop()' which can
>>>>>>>>>>>>>>>> explain stuck reader. You will likely find a logs at line 594 & 587 [1].
>>>>>>>>>>>>>>>> Dataflow caches its readers and DirectRunner may not. That can explain
>>>>>>>>>>>>>>>> DirectRunner resume reads. The expectation in KafkaIO is that Kafka client
>>>>>>>>>>>>>>>> library takes care of retrying in case of connection problems (as
>>>>>>>>>>>>>>>> documented). It is possible that in some cases poll() throws and we need to
>>>>>>>>>>>>>>>> restart the client in KafkaIO.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> [1]:
>>>>>>>>>>>>>>>> https://github.com/apache/beam/blob/master/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaUnboundedReader.java#L594
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Wed, Sep 12, 2018 at 9:59 AM Eduardo Soldera <
>>>>>>>>>>>>>>>> eduardo.soldera@arquivei.com.br> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Hi Raghu, thanks for your help.
>>>>>>>>>>>>>>>>> Just answering your previous question, the following logs
>>>>>>>>>>>>>>>>> were the same as before the error, as if the pipeline were still getting
>>>>>>>>>>>>>>>>> the messages, for example:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> (...)
>>>>>>>>>>>>>>>>> Resetting offset for partition
>>>>>>>>>>>>>>>>> com.arquivei.dataeng.andre-0 to offset 10.
>>>>>>>>>>>>>>>>> Resetting offset for partition
>>>>>>>>>>>>>>>>> com.arquivei.dataeng.andre-0 to offset 15.
>>>>>>>>>>>>>>>>> ERROR
>>>>>>>>>>>>>>>>> Resetting offset for partition
>>>>>>>>>>>>>>>>> com.arquivei.dataeng.andre-0 to offset 22.
>>>>>>>>>>>>>>>>> Resetting offset for partition
>>>>>>>>>>>>>>>>> com.arquivei.dataeng.andre-0 to offset 30.
>>>>>>>>>>>>>>>>> (...)
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> But when checking the Kafka Consumer Group, the current
>>>>>>>>>>>>>>>>> offset stays at 15, the commited offset from the last processed message,
>>>>>>>>>>>>>>>>> before the error.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> We'll file a bug, but we could now reproduce the issue in
>>>>>>>>>>>>>>>>> a Dev scenario.
>>>>>>>>>>>>>>>>> We started the same pipeline using the direct runner,
>>>>>>>>>>>>>>>>> without Google Dataflow. We blocked the Kafka Broker network and the same
>>>>>>>>>>>>>>>>> error was thrown. Then we unblocked the network and the pipeline was able
>>>>>>>>>>>>>>>>> to successfully process the subsequent messages.
>>>>>>>>>>>>>>>>> When we started the same pipeline in the Dataflow runner
>>>>>>>>>>>>>>>>> and did the same test, the same problem from our production scenario
>>>>>>>>>>>>>>>>> happened, Dataflow couldn't process the new messages. Unfortunately, we've
>>>>>>>>>>>>>>>>> stopped the dataflow job in production, but the problematic dev job is
>>>>>>>>>>>>>>>>> still running and the log file of the VM is attached. Thank you very much.
>>>>>>>>>>>>>>>>> Best regards
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Em ter, 11 de set de 2018 às 18:28, Raghu Angadi <
>>>>>>>>>>>>>>>>> rangadi@google.com> escreveu:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Specifically, I am interested if you have any thread
>>>>>>>>>>>>>>>>>> running 'consumerPollLoop()' [1]. There should always be one (if a worker
>>>>>>>>>>>>>>>>>> is assigned one of the partitions). It is possible that KafkaClient itself
>>>>>>>>>>>>>>>>>> is hasn't recovered from the group coordinator error (though unlikely).
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> https://github.com/apache/beam/blob/master/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaUnboundedReader.java#L570
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On Tue, Sep 11, 2018 at 12:31 PM Raghu Angadi <
>>>>>>>>>>>>>>>>>> rangadi@google.com> wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Hi Eduardo,
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> In case of any error, the pipeline should keep on trying
>>>>>>>>>>>>>>>>>>> to fetch. I don't know about this particular error. Do you see any others
>>>>>>>>>>>>>>>>>>> afterwards in the log?
>>>>>>>>>>>>>>>>>>> Couple of things you could try if the logs are not
>>>>>>>>>>>>>>>>>>> useful :
>>>>>>>>>>>>>>>>>>>  - login to one of the VMs and get stacktrace of java
>>>>>>>>>>>>>>>>>>> worker (look for a container called java-streaming)
>>>>>>>>>>>>>>>>>>>  - file a support bug or stackoverflow question with
>>>>>>>>>>>>>>>>>>> jobid so that Dataflow oncall can take a look.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Raghu.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> On Tue, Sep 11, 2018 at 12:10 PM Eduardo Soldera <
>>>>>>>>>>>>>>>>>>> eduardo.soldera@arquivei.com.br> wrote:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>>>>> We have a Apache Beam pipeline running in Google
>>>>>>>>>>>>>>>>>>>> Dataflow using KafkaIO. Suddenly the pipeline stop fetching Kafka messages
>>>>>>>>>>>>>>>>>>>> at all, as our other workers from other pipelines continued to get Kafka
>>>>>>>>>>>>>>>>>>>> messages.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> At the moment it stopped we got these messages:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> I  [Consumer clientId=consumer-1, groupId=genericPipe] Error sending fetch request (sessionId=1396189203, epoch=2431598) to node 3: org.apache.kafka.common.errors.DisconnectException.
>>>>>>>>>>>>>>>>>>>> I  [Consumer clientId=consumer-1, groupId=genericPipe] Group coordinator 10.0.52.70:9093 (id: 2147483646 rack: null) is unavailable or invalid, will attempt rediscovery
>>>>>>>>>>>>>>>>>>>> I  [Consumer clientId=consumer-1, groupId=genericPipe] Discovered group coordinator 10.0.52.70:9093 (id: 2147483646 rack: null)
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> And then the pipeline stopped reading the messages.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> This is the KafkaIO setup  we have:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> KafkaIO.read[String,String]()
>>>>>>>>>>>>>>>>>>>>   .withBootstrapServers(server)
>>>>>>>>>>>>>>>>>>>>   .withTopic(topic)
>>>>>>>>>>>>>>>>>>>>   .withKeyDeserializer(classOf[StringDeserializer])
>>>>>>>>>>>>>>>>>>>>   .withValueDeserializer(classOf[StringDeserializer])
>>>>>>>>>>>>>>>>>>>>   .updateConsumerProperties(properties)
>>>>>>>>>>>>>>>>>>>>   .commitOffsetsInFinalize()
>>>>>>>>>>>>>>>>>>>>   .withoutMetadata()
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>  Any help will be much appreciated.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Best regards,
>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>> Eduardo Soldera Garcia
>>>>>>>>>>>>>>>>>>>> Data Engineer
>>>>>>>>>>>>>>>>>>>> (16) 3509-5555 | www.arquivei.com.br
>>>>>>>>>>>>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>>>>>>>>>>>>>> [image: Arquivei.com.br – Inteligência em Notas Fiscais]
>>>>>>>>>>>>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>>>>>>>>>>>>>> [image: Google seleciona Arquivei para imersão e
>>>>>>>>>>>>>>>>>>>> mentoria no Vale do Silício]
>>>>>>>>>>>>>>>>>>>> <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad>
>>>>>>>>>>>>>>>>>>>> <https://www.facebook.com/arquivei>
>>>>>>>>>>>>>>>>>>>> <https://www.linkedin.com/company/arquivei>
>>>>>>>>>>>>>>>>>>>> <https://www.youtube.com/watch?v=sSUUKxbXnxk>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>> Eduardo Soldera Garcia
>>>>>>>>>>>>>>>>> Data Engineer
>>>>>>>>>>>>>>>>> (16) 3509-5555 | www.arquivei.com.br
>>>>>>>>>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>>>>>>>>>>> [image: Arquivei.com.br – Inteligência em Notas Fiscais]
>>>>>>>>>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>>>>>>>>>>> [image: Google seleciona Arquivei para imersão e mentoria
>>>>>>>>>>>>>>>>> no Vale do Silício]
>>>>>>>>>>>>>>>>> <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad>
>>>>>>>>>>>>>>>>> <https://www.facebook.com/arquivei>
>>>>>>>>>>>>>>>>> <https://www.linkedin.com/company/arquivei>
>>>>>>>>>>>>>>>>> <https://www.youtube.com/watch?v=sSUUKxbXnxk>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>> Eduardo Soldera Garcia
>>>>>>>>>>>>>>> Data Engineer
>>>>>>>>>>>>>>> (16) 3509-5555 | www.arquivei.com.br
>>>>>>>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>>>>>>>>> [image: Arquivei.com.br – Inteligência em Notas Fiscais]
>>>>>>>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>>>>>>>>> [image: Google seleciona Arquivei para imersão e mentoria no
>>>>>>>>>>>>>>> Vale do Silício]
>>>>>>>>>>>>>>> <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad>
>>>>>>>>>>>>>>> <https://www.facebook.com/arquivei>
>>>>>>>>>>>>>>> <https://www.linkedin.com/company/arquivei>
>>>>>>>>>>>>>>> <https://www.youtube.com/watch?v=sSUUKxbXnxk>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Eduardo Soldera Garcia
>>>>>>>>>> Data Engineer
>>>>>>>>>> (16) 3509-5555 | www.arquivei.com.br
>>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>>>> [image: Arquivei.com.br – Inteligência em Notas Fiscais]
>>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>>>> [image: Google seleciona Arquivei para imersão e mentoria no Vale
>>>>>>>>>> do Silício]
>>>>>>>>>> <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad>
>>>>>>>>>> <https://www.facebook.com/arquivei>
>>>>>>>>>> <https://www.linkedin.com/company/arquivei>
>>>>>>>>>> <https://www.youtube.com/watch?v=sSUUKxbXnxk>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Eduardo Soldera Garcia
>>>>>>>> Data Engineer
>>>>>>>> (16) 3509-5555 | www.arquivei.com.br
>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>> [image: Arquivei.com.br – Inteligência em Notas Fiscais]
>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>> [image: Google seleciona Arquivei para imersão e mentoria no Vale
>>>>>>>> do Silício]
>>>>>>>> <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad>
>>>>>>>> <https://www.facebook.com/arquivei>
>>>>>>>> <https://www.linkedin.com/company/arquivei>
>>>>>>>> <https://www.youtube.com/watch?v=sSUUKxbXnxk>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Eduardo Soldera Garcia
>>>>>>> Data Engineer
>>>>>>> (16) 3509-5555 | www.arquivei.com.br
>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>> [image: Arquivei.com.br – Inteligência em Notas Fiscais]
>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>> [image: Google seleciona Arquivei para imersão e mentoria no Vale do
>>>>>>> Silício]
>>>>>>> <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad>
>>>>>>> <https://www.facebook.com/arquivei>
>>>>>>> <https://www.linkedin.com/company/arquivei>
>>>>>>> <https://www.youtube.com/watch?v=sSUUKxbXnxk>
>>>>>>>
>>>>>>
>>>>>
>>>>> --
>>>>> Eduardo Soldera Garcia
>>>>> Data Engineer
>>>>> (16) 3509-5555 | www.arquivei.com.br
>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>> [image: Arquivei.com.br – Inteligência em Notas Fiscais]
>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>> [image: Google seleciona Arquivei para imersão e mentoria no Vale do
>>>>> Silício]
>>>>> <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad>
>>>>> <https://www.facebook.com/arquivei>
>>>>> <https://www.linkedin.com/company/arquivei>
>>>>> <https://www.youtube.com/watch?v=sSUUKxbXnxk>
>>>>>
>>>>
>>>
>>> --
>>> Eduardo Soldera Garcia
>>> Data Engineer
>>> (16) 3509-5555 | www.arquivei.com.br
>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>> [image: Arquivei.com.br – Inteligência em Notas Fiscais]
>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>> [image: Google seleciona Arquivei para imersão e mentoria no Vale do
>>> Silício]
>>> <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad>
>>> <https://www.facebook.com/arquivei>
>>> <https://www.linkedin.com/company/arquivei>
>>> <https://www.youtube.com/watch?v=sSUUKxbXnxk>
>>>
>>
>>
>> --
>>
>> JC
>>
>>

Re: Problem with KafkaIO

Posted by Raghu Angadi <ra...@google.com>.
On Wed, Sep 19, 2018 at 7:26 AM Juan Carlos Garcia <jc...@gmail.com>
wrote:

> Don't know if its related, but we have seen our pipeline dying (using
> SparkRunner) when there is problem with Kafka  (network interruptions),
> errors like:  org.apache.kafka.common.errors.TimeoutException: Timeout
> expired while fetching topic metadata
>
> Maybe this will fix it as well, thanks Raghu for the hint about
> *withConsumerFactoryFn.*
>

Wouldn't that be retried by the SparkRunner if it happens on the worker? or
does it happen while launching the pipeline on the client?



>
>
>
> On Wed, Sep 19, 2018 at 3:29 PM Eduardo Soldera <
> eduardo.soldera@arquivei.com.br> wrote:
>
>> Hi Raghu, thank you.
>>
>> I'm not sure though what to pass as an argument:
>>
>> KafkaIO.read[String,String]()
>>   .withBootstrapServers(server)
>>   .withTopic(topic)
>>   .withKeyDeserializer(classOf[StringDeserializer])
>>   .withValueDeserializer(classOf[StringDeserializer])
>>   .withConsumerFactoryFn(new KafkaExecutor.ConsumerFactoryFn(????????????????))
>>   .updateConsumerProperties(properties)
>>   .commitOffsetsInFinalize()
>>   .withoutMetadata()
>>
>>
>> Regards
>>
>>
>> Em ter, 18 de set de 2018 às 21:15, Raghu Angadi <ra...@google.com>
>> escreveu:
>>
>>> Hi Eduardo,
>>>
>>> There another work around you can try without having to wait for 2.7.0
>>> release: Use a wrapper to catch exception from KafkaConsumer#poll() and
>>> pass the wrapper to withConsumerFactoryFn() for KafkIO reader [1].
>>>
>>> Using something like (such a wrapper is used in KafkasIO tests [2]):
>>> private static class ConsumerFactoryFn
>>>                 implements SerializableFunction<Map<String, Object>,
>>> Consumer<byte[], byte[]>> {
>>>   @Override
>>>     public Consumer<byte[], byte[]> apply(Map<String, Object> config) {
>>>       return new KafkaConsumer(config) {
>>>           @Override
>>>           public ConsumerRecords<K, V> poll(long timeout) {
>>>               // work around for BEAM-5375
>>>               while (true) {
>>>                   try {
>>>                     return super.poll(timeout);
>>>                  } catch (Exception e) {
>>>                     // LOG & sleep for sec
>>>                 }
>>>           }
>>>        }
>>>     }
>>> }
>>>
>>> [1]:
>>> https://github.com/apache/beam/blob/release-2.4.0/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaIO.java#L417
>>> [2]:
>>> https://github.com/apache/beam/blob/release-2.4.0/sdks/java/io/kafka/src/test/java/org/apache/beam/sdk/io/kafka/KafkaIOTest.java#L261
>>>
>>> On Tue, Sep 18, 2018 at 5:49 AM Eduardo Soldera <
>>> eduardo.soldera@arquivei.com.br> wrote:
>>>
>>>> Hi Raghu, we're not sure how long the network was down. According to
>>>> the logs no longer than one minute. A 30 second shutdown would work for the
>>>> tests.
>>>>
>>>> Regards
>>>>
>>>> Em sex, 14 de set de 2018 às 21:41, Raghu Angadi <ra...@google.com>
>>>> escreveu:
>>>>
>>>>> Thanks. I could repro myself as well. How long was the network down?
>>>>>
>>>>> Trying to get the fix into 2.7 RC2.
>>>>>
>>>>> On Fri, Sep 14, 2018 at 12:25 PM Eduardo Soldera <
>>>>> eduardo.soldera@arquivei.com.br> wrote:
>>>>>
>>>>>> Just to make myself clear, I'm not sure how to use the patch but if
>>>>>> you could send us some guidance would be great.
>>>>>>
>>>>>> Em sex, 14 de set de 2018 às 16:24, Eduardo Soldera <
>>>>>> eduardo.soldera@arquivei.com.br> escreveu:
>>>>>>
>>>>>>> Hi Raghu, yes, it is feasible, would you do that for us? I'm not
>>>>>>> sure how we'd use the patch. We're using SBT and Spotify's Scio with Scala.
>>>>>>>
>>>>>>> Thanks
>>>>>>>
>>>>>>> Em sex, 14 de set de 2018 às 16:07, Raghu Angadi <ra...@google.com>
>>>>>>> escreveu:
>>>>>>>
>>>>>>>> Is is feasible for you to verify the fix in your dev job? I can
>>>>>>>> make a patch against Beam 2.4 branch if you like.
>>>>>>>>
>>>>>>>> Raghu.
>>>>>>>>
>>>>>>>> On Fri, Sep 14, 2018 at 11:14 AM Eduardo Soldera <
>>>>>>>> eduardo.soldera@arquivei.com.br> wrote:
>>>>>>>>
>>>>>>>>> Hi Raghu, thank you very much for the pull request.
>>>>>>>>> We'll wait for the 2.7 Beam release.
>>>>>>>>>
>>>>>>>>> Regards!
>>>>>>>>>
>>>>>>>>> Em qui, 13 de set de 2018 às 18:19, Raghu Angadi <
>>>>>>>>> rangadi@google.com> escreveu:
>>>>>>>>>
>>>>>>>>>> Fix: https://github.com/apache/beam/pull/6391
>>>>>>>>>>
>>>>>>>>>> On Wed, Sep 12, 2018 at 3:30 PM Raghu Angadi <ra...@google.com>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Filed BEAM-5375
>>>>>>>>>>> <https://issues.apache.org/jira/browse/BEAM-5375>. I will fix
>>>>>>>>>>> it later this week.
>>>>>>>>>>>
>>>>>>>>>>> On Wed, Sep 12, 2018 at 12:16 PM Raghu Angadi <
>>>>>>>>>>> rangadi@google.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Wed, Sep 12, 2018 at 12:11 PM Raghu Angadi <
>>>>>>>>>>>> rangadi@google.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks for the job id, I looked at the worker logs (following
>>>>>>>>>>>>> usual support oncall access protocol that provides temporary access to
>>>>>>>>>>>>> things like logs in GCP):
>>>>>>>>>>>>>
>>>>>>>>>>>>> Root issue looks like consumerPollLoop() mentioned earlier
>>>>>>>>>>>>> needs to handle unchecked exception. In your case it is clear that poll
>>>>>>>>>>>>> thread exited with a runtime exception. The reader does not check for it
>>>>>>>>>>>>> and continues to wait for poll thread to enqueue messages. A fix should
>>>>>>>>>>>>> result in an IOException for read from the source. The runners will handle
>>>>>>>>>>>>> that appropriately after that.  I will file a jira.
>>>>>>>>>>>>>
>>>>>>>>>>>>> https://github.com/apache/beam/blob/master/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaUnboundedReader.java#L678
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Ignore the link.. was pasted here by mistake.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> From the logs (with a comment below each one):
>>>>>>>>>>>>>
>>>>>>>>>>>>>    - 2018-09-12 06:13:07.345 PDT Reader-0: reading from
>>>>>>>>>>>>>    kafka_topic-0 starting at offset 2
>>>>>>>>>>>>>       - Implies the reader is initialized and poll thread is
>>>>>>>>>>>>>       started.
>>>>>>>>>>>>>    - 2018-09-12 06:13:07.780 PDT Reader-0: first record
>>>>>>>>>>>>>    offset 2
>>>>>>>>>>>>>       - The reader actually got a message received by the
>>>>>>>>>>>>>       poll thread from Kafka.
>>>>>>>>>>>>>    - 2018-09-12 06:16:48.771 PDT Reader-0: exception while
>>>>>>>>>>>>>    fetching latest offset for partition kafka_topic-0. will be retried.
>>>>>>>>>>>>>       - This must have happened around the time when network
>>>>>>>>>>>>>       was disrupted. This is from. Actual log is from another periodic task that
>>>>>>>>>>>>>       fetches latest offsets for partitions.
>>>>>>>>>>>>>
>>>>>>>>>>>>> The poll thread must have died around the time network was
>>>>>>>>>>>>> disrupted.
>>>>>>>>>>>>>
>>>>>>>>>>>>> The following log comes from kafka client itself and is
>>>>>>>>>>>>> printed every second when KafkaIO fetches latest offset. This log seems to
>>>>>>>>>>>>> be added in recent versions. It is probably an unintentional log. I don't
>>>>>>>>>>>>> think there is any better to fetch latest offsets than how KafkaIO does
>>>>>>>>>>>>> now. This is logged inside consumer.position() called at [1].
>>>>>>>>>>>>>
>>>>>>>>>>>>>    - 2018-09-12 06:13:11.786 PDT [Consumer
>>>>>>>>>>>>>    clientId=consumer-2,
>>>>>>>>>>>>>    groupId=Reader-0_offset_consumer_1735388161_genericPipe] Resetting offset
>>>>>>>>>>>>>    for partition com.arquivei.dataeng.andre-0 to offset 3.
>>>>>>>>>>>>>
>>>>>>>>>>>>> [1]:
>>>>>>>>>>>>> https://github.com/apache/beam/blob/master/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaUnboundedReader.java#L678
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> This 'Resetting offset' is harmless, but is quite annoying to
>>>>>>>>>>>> see in the worker logs. One way to avoid is to set kafka consumer's log
>>>>>>>>>>>> level to WARNING. Ideally KafkaIO itself should do something to avoid it
>>>>>>>>>>>> without user option.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>> On Wed, Sep 12, 2018 at 10:27 AM Eduardo Soldera <
>>>>>>>>>>>>> eduardo.soldera@arquivei.com.br> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi Raghu! The job_id of our dev job is
>>>>>>>>>>>>>> 2018-09-12_06_11_48-5600553605191377866.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks!
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Em qua, 12 de set de 2018 às 14:18, Raghu Angadi <
>>>>>>>>>>>>>> rangadi@google.com> escreveu:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks for debugging.
>>>>>>>>>>>>>>> Can you provide the job_id of your dev job? The stacktrace
>>>>>>>>>>>>>>> shows that there is no thread running 'consumerPollLoop()' which can
>>>>>>>>>>>>>>> explain stuck reader. You will likely find a logs at line 594 & 587 [1].
>>>>>>>>>>>>>>> Dataflow caches its readers and DirectRunner may not. That can explain
>>>>>>>>>>>>>>> DirectRunner resume reads. The expectation in KafkaIO is that Kafka client
>>>>>>>>>>>>>>> library takes care of retrying in case of connection problems (as
>>>>>>>>>>>>>>> documented). It is possible that in some cases poll() throws and we need to
>>>>>>>>>>>>>>> restart the client in KafkaIO.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> [1]:
>>>>>>>>>>>>>>> https://github.com/apache/beam/blob/master/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaUnboundedReader.java#L594
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Wed, Sep 12, 2018 at 9:59 AM Eduardo Soldera <
>>>>>>>>>>>>>>> eduardo.soldera@arquivei.com.br> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hi Raghu, thanks for your help.
>>>>>>>>>>>>>>>> Just answering your previous question, the following logs
>>>>>>>>>>>>>>>> were the same as before the error, as if the pipeline were still getting
>>>>>>>>>>>>>>>> the messages, for example:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> (...)
>>>>>>>>>>>>>>>> Resetting offset for partition com.arquivei.dataeng.andre-0
>>>>>>>>>>>>>>>> to offset 10.
>>>>>>>>>>>>>>>> Resetting offset for partition com.arquivei.dataeng.andre-0
>>>>>>>>>>>>>>>> to offset 15.
>>>>>>>>>>>>>>>> ERROR
>>>>>>>>>>>>>>>> Resetting offset for partition com.arquivei.dataeng.andre-0
>>>>>>>>>>>>>>>> to offset 22.
>>>>>>>>>>>>>>>> Resetting offset for partition com.arquivei.dataeng.andre-0
>>>>>>>>>>>>>>>> to offset 30.
>>>>>>>>>>>>>>>> (...)
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> But when checking the Kafka Consumer Group, the current
>>>>>>>>>>>>>>>> offset stays at 15, the commited offset from the last processed message,
>>>>>>>>>>>>>>>> before the error.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> We'll file a bug, but we could now reproduce the issue in a
>>>>>>>>>>>>>>>> Dev scenario.
>>>>>>>>>>>>>>>> We started the same pipeline using the direct runner,
>>>>>>>>>>>>>>>> without Google Dataflow. We blocked the Kafka Broker network and the same
>>>>>>>>>>>>>>>> error was thrown. Then we unblocked the network and the pipeline was able
>>>>>>>>>>>>>>>> to successfully process the subsequent messages.
>>>>>>>>>>>>>>>> When we started the same pipeline in the Dataflow runner
>>>>>>>>>>>>>>>> and did the same test, the same problem from our production scenario
>>>>>>>>>>>>>>>> happened, Dataflow couldn't process the new messages. Unfortunately, we've
>>>>>>>>>>>>>>>> stopped the dataflow job in production, but the problematic dev job is
>>>>>>>>>>>>>>>> still running and the log file of the VM is attached. Thank you very much.
>>>>>>>>>>>>>>>> Best regards
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Em ter, 11 de set de 2018 às 18:28, Raghu Angadi <
>>>>>>>>>>>>>>>> rangadi@google.com> escreveu:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Specifically, I am interested if you have any thread
>>>>>>>>>>>>>>>>> running 'consumerPollLoop()' [1]. There should always be one (if a worker
>>>>>>>>>>>>>>>>> is assigned one of the partitions). It is possible that KafkaClient itself
>>>>>>>>>>>>>>>>> is hasn't recovered from the group coordinator error (though unlikely).
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> https://github.com/apache/beam/blob/master/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaUnboundedReader.java#L570
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Tue, Sep 11, 2018 at 12:31 PM Raghu Angadi <
>>>>>>>>>>>>>>>>> rangadi@google.com> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Hi Eduardo,
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> In case of any error, the pipeline should keep on trying
>>>>>>>>>>>>>>>>>> to fetch. I don't know about this particular error. Do you see any others
>>>>>>>>>>>>>>>>>> afterwards in the log?
>>>>>>>>>>>>>>>>>> Couple of things you could try if the logs are not useful
>>>>>>>>>>>>>>>>>> :
>>>>>>>>>>>>>>>>>>  - login to one of the VMs and get stacktrace of java
>>>>>>>>>>>>>>>>>> worker (look for a container called java-streaming)
>>>>>>>>>>>>>>>>>>  - file a support bug or stackoverflow question with
>>>>>>>>>>>>>>>>>> jobid so that Dataflow oncall can take a look.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Raghu.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On Tue, Sep 11, 2018 at 12:10 PM Eduardo Soldera <
>>>>>>>>>>>>>>>>>> eduardo.soldera@arquivei.com.br> wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>>>> We have a Apache Beam pipeline running in Google
>>>>>>>>>>>>>>>>>>> Dataflow using KafkaIO. Suddenly the pipeline stop fetching Kafka messages
>>>>>>>>>>>>>>>>>>> at all, as our other workers from other pipelines continued to get Kafka
>>>>>>>>>>>>>>>>>>> messages.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> At the moment it stopped we got these messages:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> I  [Consumer clientId=consumer-1, groupId=genericPipe] Error sending fetch request (sessionId=1396189203, epoch=2431598) to node 3: org.apache.kafka.common.errors.DisconnectException.
>>>>>>>>>>>>>>>>>>> I  [Consumer clientId=consumer-1, groupId=genericPipe] Group coordinator 10.0.52.70:9093 (id: 2147483646 rack: null) is unavailable or invalid, will attempt rediscovery
>>>>>>>>>>>>>>>>>>> I  [Consumer clientId=consumer-1, groupId=genericPipe] Discovered group coordinator 10.0.52.70:9093 (id: 2147483646 rack: null)
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> And then the pipeline stopped reading the messages.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> This is the KafkaIO setup  we have:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> KafkaIO.read[String,String]()
>>>>>>>>>>>>>>>>>>>   .withBootstrapServers(server)
>>>>>>>>>>>>>>>>>>>   .withTopic(topic)
>>>>>>>>>>>>>>>>>>>   .withKeyDeserializer(classOf[StringDeserializer])
>>>>>>>>>>>>>>>>>>>   .withValueDeserializer(classOf[StringDeserializer])
>>>>>>>>>>>>>>>>>>>   .updateConsumerProperties(properties)
>>>>>>>>>>>>>>>>>>>   .commitOffsetsInFinalize()
>>>>>>>>>>>>>>>>>>>   .withoutMetadata()
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>  Any help will be much appreciated.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Best regards,
>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>> Eduardo Soldera Garcia
>>>>>>>>>>>>>>>>>>> Data Engineer
>>>>>>>>>>>>>>>>>>> (16) 3509-5555 | www.arquivei.com.br
>>>>>>>>>>>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>>>>>>>>>>>>> [image: Arquivei.com.br – Inteligência em Notas Fiscais]
>>>>>>>>>>>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>>>>>>>>>>>>> [image: Google seleciona Arquivei para imersão e
>>>>>>>>>>>>>>>>>>> mentoria no Vale do Silício]
>>>>>>>>>>>>>>>>>>> <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad>
>>>>>>>>>>>>>>>>>>> <https://www.facebook.com/arquivei>
>>>>>>>>>>>>>>>>>>> <https://www.linkedin.com/company/arquivei>
>>>>>>>>>>>>>>>>>>> <https://www.youtube.com/watch?v=sSUUKxbXnxk>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>> Eduardo Soldera Garcia
>>>>>>>>>>>>>>>> Data Engineer
>>>>>>>>>>>>>>>> (16) 3509-5555 | www.arquivei.com.br
>>>>>>>>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>>>>>>>>>> [image: Arquivei.com.br – Inteligência em Notas Fiscais]
>>>>>>>>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>>>>>>>>>> [image: Google seleciona Arquivei para imersão e mentoria
>>>>>>>>>>>>>>>> no Vale do Silício]
>>>>>>>>>>>>>>>> <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad>
>>>>>>>>>>>>>>>> <https://www.facebook.com/arquivei>
>>>>>>>>>>>>>>>> <https://www.linkedin.com/company/arquivei>
>>>>>>>>>>>>>>>> <https://www.youtube.com/watch?v=sSUUKxbXnxk>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> --
>>>>>>>>>>>>>> Eduardo Soldera Garcia
>>>>>>>>>>>>>> Data Engineer
>>>>>>>>>>>>>> (16) 3509-5555 | www.arquivei.com.br
>>>>>>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>>>>>>>> [image: Arquivei.com.br – Inteligência em Notas Fiscais]
>>>>>>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>>>>>>>> [image: Google seleciona Arquivei para imersão e mentoria no
>>>>>>>>>>>>>> Vale do Silício]
>>>>>>>>>>>>>> <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad>
>>>>>>>>>>>>>> <https://www.facebook.com/arquivei>
>>>>>>>>>>>>>> <https://www.linkedin.com/company/arquivei>
>>>>>>>>>>>>>> <https://www.youtube.com/watch?v=sSUUKxbXnxk>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Eduardo Soldera Garcia
>>>>>>>>> Data Engineer
>>>>>>>>> (16) 3509-5555 | www.arquivei.com.br
>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>>> [image: Arquivei.com.br – Inteligência em Notas Fiscais]
>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>>> [image: Google seleciona Arquivei para imersão e mentoria no Vale
>>>>>>>>> do Silício]
>>>>>>>>> <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad>
>>>>>>>>> <https://www.facebook.com/arquivei>
>>>>>>>>> <https://www.linkedin.com/company/arquivei>
>>>>>>>>> <https://www.youtube.com/watch?v=sSUUKxbXnxk>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Eduardo Soldera Garcia
>>>>>>> Data Engineer
>>>>>>> (16) 3509-5555 | www.arquivei.com.br
>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>> [image: Arquivei.com.br – Inteligência em Notas Fiscais]
>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>> [image: Google seleciona Arquivei para imersão e mentoria no Vale do
>>>>>>> Silício]
>>>>>>> <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad>
>>>>>>> <https://www.facebook.com/arquivei>
>>>>>>> <https://www.linkedin.com/company/arquivei>
>>>>>>> <https://www.youtube.com/watch?v=sSUUKxbXnxk>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Eduardo Soldera Garcia
>>>>>> Data Engineer
>>>>>> (16) 3509-5555 | www.arquivei.com.br
>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>> [image: Arquivei.com.br – Inteligência em Notas Fiscais]
>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>> [image: Google seleciona Arquivei para imersão e mentoria no Vale do
>>>>>> Silício]
>>>>>> <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad>
>>>>>> <https://www.facebook.com/arquivei>
>>>>>> <https://www.linkedin.com/company/arquivei>
>>>>>> <https://www.youtube.com/watch?v=sSUUKxbXnxk>
>>>>>>
>>>>>
>>>>
>>>> --
>>>> Eduardo Soldera Garcia
>>>> Data Engineer
>>>> (16) 3509-5555 | www.arquivei.com.br
>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>> [image: Arquivei.com.br – Inteligência em Notas Fiscais]
>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>> [image: Google seleciona Arquivei para imersão e mentoria no Vale do
>>>> Silício]
>>>> <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad>
>>>> <https://www.facebook.com/arquivei>
>>>> <https://www.linkedin.com/company/arquivei>
>>>> <https://www.youtube.com/watch?v=sSUUKxbXnxk>
>>>>
>>>
>>
>> --
>> Eduardo Soldera Garcia
>> Data Engineer
>> (16) 3509-5555 | www.arquivei.com.br
>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>> [image: Arquivei.com.br – Inteligência em Notas Fiscais]
>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>> [image: Google seleciona Arquivei para imersão e mentoria no Vale do
>> Silício]
>> <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad>
>> <https://www.facebook.com/arquivei>
>> <https://www.linkedin.com/company/arquivei>
>> <https://www.youtube.com/watch?v=sSUUKxbXnxk>
>>
>
>
> --
>
> JC
>
>

Re: Problem with KafkaIO

Posted by Juan Carlos Garcia <jc...@gmail.com>.
Don't know if its related, but we have seen our pipeline dying (using
SparkRunner) when there is problem with Kafka  (network interruptions),
errors like:  org.apache.kafka.common.errors.TimeoutException: Timeout
expired while fetching topic metadata

Maybe this will fix it as well, thanks Raghu for the hint about
*withConsumerFactoryFn.*




On Wed, Sep 19, 2018 at 3:29 PM Eduardo Soldera <
eduardo.soldera@arquivei.com.br> wrote:

> Hi Raghu, thank you.
>
> I'm not sure though what to pass as an argument:
>
> KafkaIO.read[String,String]()
>   .withBootstrapServers(server)
>   .withTopic(topic)
>   .withKeyDeserializer(classOf[StringDeserializer])
>   .withValueDeserializer(classOf[StringDeserializer])
>   .withConsumerFactoryFn(new KafkaExecutor.ConsumerFactoryFn(????????????????))
>   .updateConsumerProperties(properties)
>   .commitOffsetsInFinalize()
>   .withoutMetadata()
>
>
> Regards
>
>
> Em ter, 18 de set de 2018 às 21:15, Raghu Angadi <ra...@google.com>
> escreveu:
>
>> Hi Eduardo,
>>
>> There another work around you can try without having to wait for 2.7.0
>> release: Use a wrapper to catch exception from KafkaConsumer#poll() and
>> pass the wrapper to withConsumerFactoryFn() for KafkIO reader [1].
>>
>> Using something like (such a wrapper is used in KafkasIO tests [2]):
>> private static class ConsumerFactoryFn
>>                 implements SerializableFunction<Map<String, Object>,
>> Consumer<byte[], byte[]>> {
>>   @Override
>>     public Consumer<byte[], byte[]> apply(Map<String, Object> config) {
>>       return new KafkaConsumer(config) {
>>           @Override
>>           public ConsumerRecords<K, V> poll(long timeout) {
>>               // work around for BEAM-5375
>>               while (true) {
>>                   try {
>>                     return super.poll(timeout);
>>                  } catch (Exception e) {
>>                     // LOG & sleep for sec
>>                 }
>>           }
>>        }
>>     }
>> }
>>
>> [1]:
>> https://github.com/apache/beam/blob/release-2.4.0/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaIO.java#L417
>> [2]:
>> https://github.com/apache/beam/blob/release-2.4.0/sdks/java/io/kafka/src/test/java/org/apache/beam/sdk/io/kafka/KafkaIOTest.java#L261
>>
>> On Tue, Sep 18, 2018 at 5:49 AM Eduardo Soldera <
>> eduardo.soldera@arquivei.com.br> wrote:
>>
>>> Hi Raghu, we're not sure how long the network was down. According to the
>>> logs no longer than one minute. A 30 second shutdown would work for the
>>> tests.
>>>
>>> Regards
>>>
>>> Em sex, 14 de set de 2018 às 21:41, Raghu Angadi <ra...@google.com>
>>> escreveu:
>>>
>>>> Thanks. I could repro myself as well. How long was the network down?
>>>>
>>>> Trying to get the fix into 2.7 RC2.
>>>>
>>>> On Fri, Sep 14, 2018 at 12:25 PM Eduardo Soldera <
>>>> eduardo.soldera@arquivei.com.br> wrote:
>>>>
>>>>> Just to make myself clear, I'm not sure how to use the patch but if
>>>>> you could send us some guidance would be great.
>>>>>
>>>>> Em sex, 14 de set de 2018 às 16:24, Eduardo Soldera <
>>>>> eduardo.soldera@arquivei.com.br> escreveu:
>>>>>
>>>>>> Hi Raghu, yes, it is feasible, would you do that for us? I'm not sure
>>>>>> how we'd use the patch. We're using SBT and Spotify's Scio with Scala.
>>>>>>
>>>>>> Thanks
>>>>>>
>>>>>> Em sex, 14 de set de 2018 às 16:07, Raghu Angadi <ra...@google.com>
>>>>>> escreveu:
>>>>>>
>>>>>>> Is is feasible for you to verify the fix in your dev job? I can make
>>>>>>> a patch against Beam 2.4 branch if you like.
>>>>>>>
>>>>>>> Raghu.
>>>>>>>
>>>>>>> On Fri, Sep 14, 2018 at 11:14 AM Eduardo Soldera <
>>>>>>> eduardo.soldera@arquivei.com.br> wrote:
>>>>>>>
>>>>>>>> Hi Raghu, thank you very much for the pull request.
>>>>>>>> We'll wait for the 2.7 Beam release.
>>>>>>>>
>>>>>>>> Regards!
>>>>>>>>
>>>>>>>> Em qui, 13 de set de 2018 às 18:19, Raghu Angadi <
>>>>>>>> rangadi@google.com> escreveu:
>>>>>>>>
>>>>>>>>> Fix: https://github.com/apache/beam/pull/6391
>>>>>>>>>
>>>>>>>>> On Wed, Sep 12, 2018 at 3:30 PM Raghu Angadi <ra...@google.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Filed BEAM-5375 <https://issues.apache.org/jira/browse/BEAM-5375>.
>>>>>>>>>> I will fix it later this week.
>>>>>>>>>>
>>>>>>>>>> On Wed, Sep 12, 2018 at 12:16 PM Raghu Angadi <ra...@google.com>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Wed, Sep 12, 2018 at 12:11 PM Raghu Angadi <
>>>>>>>>>>> rangadi@google.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Thanks for the job id, I looked at the worker logs (following
>>>>>>>>>>>> usual support oncall access protocol that provides temporary access to
>>>>>>>>>>>> things like logs in GCP):
>>>>>>>>>>>>
>>>>>>>>>>>> Root issue looks like consumerPollLoop() mentioned earlier
>>>>>>>>>>>> needs to handle unchecked exception. In your case it is clear that poll
>>>>>>>>>>>> thread exited with a runtime exception. The reader does not check for it
>>>>>>>>>>>> and continues to wait for poll thread to enqueue messages. A fix should
>>>>>>>>>>>> result in an IOException for read from the source. The runners will handle
>>>>>>>>>>>> that appropriately after that.  I will file a jira.
>>>>>>>>>>>>
>>>>>>>>>>>> https://github.com/apache/beam/blob/master/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaUnboundedReader.java#L678
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Ignore the link.. was pasted here by mistake.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> From the logs (with a comment below each one):
>>>>>>>>>>>>
>>>>>>>>>>>>    - 2018-09-12 06:13:07.345 PDT Reader-0: reading from
>>>>>>>>>>>>    kafka_topic-0 starting at offset 2
>>>>>>>>>>>>       - Implies the reader is initialized and poll thread is
>>>>>>>>>>>>       started.
>>>>>>>>>>>>    - 2018-09-12 06:13:07.780 PDT Reader-0: first record offset
>>>>>>>>>>>>    2
>>>>>>>>>>>>       - The reader actually got a message received by the poll
>>>>>>>>>>>>       thread from Kafka.
>>>>>>>>>>>>    - 2018-09-12 06:16:48.771 PDT Reader-0: exception while
>>>>>>>>>>>>    fetching latest offset for partition kafka_topic-0. will be retried.
>>>>>>>>>>>>       - This must have happened around the time when network
>>>>>>>>>>>>       was disrupted. This is from. Actual log is from another periodic task that
>>>>>>>>>>>>       fetches latest offsets for partitions.
>>>>>>>>>>>>
>>>>>>>>>>>> The poll thread must have died around the time network was
>>>>>>>>>>>> disrupted.
>>>>>>>>>>>>
>>>>>>>>>>>> The following log comes from kafka client itself and is printed
>>>>>>>>>>>> every second when KafkaIO fetches latest offset. This log seems to be added
>>>>>>>>>>>> in recent versions. It is probably an unintentional log. I don't think
>>>>>>>>>>>> there is any better to fetch latest offsets than how KafkaIO does now. This
>>>>>>>>>>>> is logged inside consumer.position() called at [1].
>>>>>>>>>>>>
>>>>>>>>>>>>    - 2018-09-12 06:13:11.786 PDT [Consumer
>>>>>>>>>>>>    clientId=consumer-2,
>>>>>>>>>>>>    groupId=Reader-0_offset_consumer_1735388161_genericPipe] Resetting offset
>>>>>>>>>>>>    for partition com.arquivei.dataeng.andre-0 to offset 3.
>>>>>>>>>>>>
>>>>>>>>>>>> [1]:
>>>>>>>>>>>> https://github.com/apache/beam/blob/master/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaUnboundedReader.java#L678
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> This 'Resetting offset' is harmless, but is quite annoying to
>>>>>>>>>>> see in the worker logs. One way to avoid is to set kafka consumer's log
>>>>>>>>>>> level to WARNING. Ideally KafkaIO itself should do something to avoid it
>>>>>>>>>>> without user option.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>> On Wed, Sep 12, 2018 at 10:27 AM Eduardo Soldera <
>>>>>>>>>>>> eduardo.soldera@arquivei.com.br> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi Raghu! The job_id of our dev job is
>>>>>>>>>>>>> 2018-09-12_06_11_48-5600553605191377866.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks!
>>>>>>>>>>>>>
>>>>>>>>>>>>> Em qua, 12 de set de 2018 às 14:18, Raghu Angadi <
>>>>>>>>>>>>> rangadi@google.com> escreveu:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks for debugging.
>>>>>>>>>>>>>> Can you provide the job_id of your dev job? The stacktrace
>>>>>>>>>>>>>> shows that there is no thread running 'consumerPollLoop()' which can
>>>>>>>>>>>>>> explain stuck reader. You will likely find a logs at line 594 & 587 [1].
>>>>>>>>>>>>>> Dataflow caches its readers and DirectRunner may not. That can explain
>>>>>>>>>>>>>> DirectRunner resume reads. The expectation in KafkaIO is that Kafka client
>>>>>>>>>>>>>> library takes care of retrying in case of connection problems (as
>>>>>>>>>>>>>> documented). It is possible that in some cases poll() throws and we need to
>>>>>>>>>>>>>> restart the client in KafkaIO.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> [1]:
>>>>>>>>>>>>>> https://github.com/apache/beam/blob/master/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaUnboundedReader.java#L594
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Wed, Sep 12, 2018 at 9:59 AM Eduardo Soldera <
>>>>>>>>>>>>>> eduardo.soldera@arquivei.com.br> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi Raghu, thanks for your help.
>>>>>>>>>>>>>>> Just answering your previous question, the following logs
>>>>>>>>>>>>>>> were the same as before the error, as if the pipeline were still getting
>>>>>>>>>>>>>>> the messages, for example:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> (...)
>>>>>>>>>>>>>>> Resetting offset for partition com.arquivei.dataeng.andre-0
>>>>>>>>>>>>>>> to offset 10.
>>>>>>>>>>>>>>> Resetting offset for partition com.arquivei.dataeng.andre-0
>>>>>>>>>>>>>>> to offset 15.
>>>>>>>>>>>>>>> ERROR
>>>>>>>>>>>>>>> Resetting offset for partition com.arquivei.dataeng.andre-0
>>>>>>>>>>>>>>> to offset 22.
>>>>>>>>>>>>>>> Resetting offset for partition com.arquivei.dataeng.andre-0
>>>>>>>>>>>>>>> to offset 30.
>>>>>>>>>>>>>>> (...)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> But when checking the Kafka Consumer Group, the current
>>>>>>>>>>>>>>> offset stays at 15, the commited offset from the last processed message,
>>>>>>>>>>>>>>> before the error.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> We'll file a bug, but we could now reproduce the issue in a
>>>>>>>>>>>>>>> Dev scenario.
>>>>>>>>>>>>>>> We started the same pipeline using the direct runner,
>>>>>>>>>>>>>>> without Google Dataflow. We blocked the Kafka Broker network and the same
>>>>>>>>>>>>>>> error was thrown. Then we unblocked the network and the pipeline was able
>>>>>>>>>>>>>>> to successfully process the subsequent messages.
>>>>>>>>>>>>>>> When we started the same pipeline in the Dataflow runner and
>>>>>>>>>>>>>>> did the same test, the same problem from our production scenario happened,
>>>>>>>>>>>>>>> Dataflow couldn't process the new messages. Unfortunately, we've stopped
>>>>>>>>>>>>>>> the dataflow job in production, but the problematic dev job is still
>>>>>>>>>>>>>>> running and the log file of the VM is attached. Thank you very much.
>>>>>>>>>>>>>>> Best regards
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Em ter, 11 de set de 2018 às 18:28, Raghu Angadi <
>>>>>>>>>>>>>>> rangadi@google.com> escreveu:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Specifically, I am interested if you have any thread
>>>>>>>>>>>>>>>> running 'consumerPollLoop()' [1]. There should always be one (if a worker
>>>>>>>>>>>>>>>> is assigned one of the partitions). It is possible that KafkaClient itself
>>>>>>>>>>>>>>>> is hasn't recovered from the group coordinator error (though unlikely).
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> https://github.com/apache/beam/blob/master/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaUnboundedReader.java#L570
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Tue, Sep 11, 2018 at 12:31 PM Raghu Angadi <
>>>>>>>>>>>>>>>> rangadi@google.com> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Hi Eduardo,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> In case of any error, the pipeline should keep on trying
>>>>>>>>>>>>>>>>> to fetch. I don't know about this particular error. Do you see any others
>>>>>>>>>>>>>>>>> afterwards in the log?
>>>>>>>>>>>>>>>>> Couple of things you could try if the logs are not useful
>>>>>>>>>>>>>>>>> :
>>>>>>>>>>>>>>>>>  - login to one of the VMs and get stacktrace of java
>>>>>>>>>>>>>>>>> worker (look for a container called java-streaming)
>>>>>>>>>>>>>>>>>  - file a support bug or stackoverflow question with jobid
>>>>>>>>>>>>>>>>> so that Dataflow oncall can take a look.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Raghu.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Tue, Sep 11, 2018 at 12:10 PM Eduardo Soldera <
>>>>>>>>>>>>>>>>> eduardo.soldera@arquivei.com.br> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>>> We have a Apache Beam pipeline running in Google Dataflow
>>>>>>>>>>>>>>>>>> using KafkaIO. Suddenly the pipeline stop fetching Kafka messages at all,
>>>>>>>>>>>>>>>>>> as our other workers from other pipelines continued to get Kafka messages.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> At the moment it stopped we got these messages:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I  [Consumer clientId=consumer-1, groupId=genericPipe] Error sending fetch request (sessionId=1396189203, epoch=2431598) to node 3: org.apache.kafka.common.errors.DisconnectException.
>>>>>>>>>>>>>>>>>> I  [Consumer clientId=consumer-1, groupId=genericPipe] Group coordinator 10.0.52.70:9093 (id: 2147483646 rack: null) is unavailable or invalid, will attempt rediscovery
>>>>>>>>>>>>>>>>>> I  [Consumer clientId=consumer-1, groupId=genericPipe] Discovered group coordinator 10.0.52.70:9093 (id: 2147483646 rack: null)
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> And then the pipeline stopped reading the messages.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> This is the KafkaIO setup  we have:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> KafkaIO.read[String,String]()
>>>>>>>>>>>>>>>>>>   .withBootstrapServers(server)
>>>>>>>>>>>>>>>>>>   .withTopic(topic)
>>>>>>>>>>>>>>>>>>   .withKeyDeserializer(classOf[StringDeserializer])
>>>>>>>>>>>>>>>>>>   .withValueDeserializer(classOf[StringDeserializer])
>>>>>>>>>>>>>>>>>>   .updateConsumerProperties(properties)
>>>>>>>>>>>>>>>>>>   .commitOffsetsInFinalize()
>>>>>>>>>>>>>>>>>>   .withoutMetadata()
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>  Any help will be much appreciated.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Best regards,
>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>> Eduardo Soldera Garcia
>>>>>>>>>>>>>>>>>> Data Engineer
>>>>>>>>>>>>>>>>>> (16) 3509-5555 | www.arquivei.com.br
>>>>>>>>>>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>>>>>>>>>>>> [image: Arquivei.com.br – Inteligência em Notas Fiscais]
>>>>>>>>>>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>>>>>>>>>>>> [image: Google seleciona Arquivei para imersão e mentoria
>>>>>>>>>>>>>>>>>> no Vale do Silício]
>>>>>>>>>>>>>>>>>> <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad>
>>>>>>>>>>>>>>>>>> <https://www.facebook.com/arquivei>
>>>>>>>>>>>>>>>>>> <https://www.linkedin.com/company/arquivei>
>>>>>>>>>>>>>>>>>> <https://www.youtube.com/watch?v=sSUUKxbXnxk>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>> Eduardo Soldera Garcia
>>>>>>>>>>>>>>> Data Engineer
>>>>>>>>>>>>>>> (16) 3509-5555 | www.arquivei.com.br
>>>>>>>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>>>>>>>>> [image: Arquivei.com.br – Inteligência em Notas Fiscais]
>>>>>>>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>>>>>>>>> [image: Google seleciona Arquivei para imersão e mentoria no
>>>>>>>>>>>>>>> Vale do Silício]
>>>>>>>>>>>>>>> <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad>
>>>>>>>>>>>>>>> <https://www.facebook.com/arquivei>
>>>>>>>>>>>>>>> <https://www.linkedin.com/company/arquivei>
>>>>>>>>>>>>>>> <https://www.youtube.com/watch?v=sSUUKxbXnxk>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>>> Eduardo Soldera Garcia
>>>>>>>>>>>>> Data Engineer
>>>>>>>>>>>>> (16) 3509-5555 | www.arquivei.com.br
>>>>>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>>>>>>> [image: Arquivei.com.br – Inteligência em Notas Fiscais]
>>>>>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>>>>>>> [image: Google seleciona Arquivei para imersão e mentoria no
>>>>>>>>>>>>> Vale do Silício]
>>>>>>>>>>>>> <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad>
>>>>>>>>>>>>> <https://www.facebook.com/arquivei>
>>>>>>>>>>>>> <https://www.linkedin.com/company/arquivei>
>>>>>>>>>>>>> <https://www.youtube.com/watch?v=sSUUKxbXnxk>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Eduardo Soldera Garcia
>>>>>>>> Data Engineer
>>>>>>>> (16) 3509-5555 | www.arquivei.com.br
>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>> [image: Arquivei.com.br – Inteligência em Notas Fiscais]
>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>> [image: Google seleciona Arquivei para imersão e mentoria no Vale
>>>>>>>> do Silício]
>>>>>>>> <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad>
>>>>>>>> <https://www.facebook.com/arquivei>
>>>>>>>> <https://www.linkedin.com/company/arquivei>
>>>>>>>> <https://www.youtube.com/watch?v=sSUUKxbXnxk>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>> --
>>>>>> Eduardo Soldera Garcia
>>>>>> Data Engineer
>>>>>> (16) 3509-5555 | www.arquivei.com.br
>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>> [image: Arquivei.com.br – Inteligência em Notas Fiscais]
>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>> [image: Google seleciona Arquivei para imersão e mentoria no Vale do
>>>>>> Silício]
>>>>>> <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad>
>>>>>> <https://www.facebook.com/arquivei>
>>>>>> <https://www.linkedin.com/company/arquivei>
>>>>>> <https://www.youtube.com/watch?v=sSUUKxbXnxk>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Eduardo Soldera Garcia
>>>>> Data Engineer
>>>>> (16) 3509-5555 | www.arquivei.com.br
>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>> [image: Arquivei.com.br – Inteligência em Notas Fiscais]
>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>> [image: Google seleciona Arquivei para imersão e mentoria no Vale do
>>>>> Silício]
>>>>> <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad>
>>>>> <https://www.facebook.com/arquivei>
>>>>> <https://www.linkedin.com/company/arquivei>
>>>>> <https://www.youtube.com/watch?v=sSUUKxbXnxk>
>>>>>
>>>>
>>>
>>> --
>>> Eduardo Soldera Garcia
>>> Data Engineer
>>> (16) 3509-5555 | www.arquivei.com.br
>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>> [image: Arquivei.com.br – Inteligência em Notas Fiscais]
>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>> [image: Google seleciona Arquivei para imersão e mentoria no Vale do
>>> Silício]
>>> <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad>
>>> <https://www.facebook.com/arquivei>
>>> <https://www.linkedin.com/company/arquivei>
>>> <https://www.youtube.com/watch?v=sSUUKxbXnxk>
>>>
>>
>
> --
> Eduardo Soldera Garcia
> Data Engineer
> (16) 3509-5555 | www.arquivei.com.br
> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
> [image: Arquivei.com.br – Inteligência em Notas Fiscais]
> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
> [image: Google seleciona Arquivei para imersão e mentoria no Vale do
> Silício]
> <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad>
> <https://www.facebook.com/arquivei>
> <https://www.linkedin.com/company/arquivei>
> <https://www.youtube.com/watch?v=sSUUKxbXnxk>
>


-- 

JC

Re: Problem with KafkaIO

Posted by Raghu Angadi <ra...@google.com>.
You would pass a SerializableFunction (with whatever args required to
create the object of this functor, if any). E.g. ConsumerFactoryFn()
defined in my pseudo code below does not need to any args. So it would be :
------------
KafkaIO.read()
   ...
   .withConsumerFactoryFn(new ConsumerFactoryFn()).
-----------






On Wed, Sep 19, 2018 at 6:29 AM Eduardo Soldera <
eduardo.soldera@arquivei.com.br> wrote:

> Hi Raghu, thank you.
>
> I'm not sure though what to pass as an argument:
>
> KafkaIO.read[String,String]()
>   .withBootstrapServers(server)
>   .withTopic(topic)
>   .withKeyDeserializer(classOf[StringDeserializer])
>   .withValueDeserializer(classOf[StringDeserializer])
>   .withConsumerFactoryFn(new KafkaExecutor.ConsumerFactoryFn(????????????????))
>   .updateConsumerProperties(properties)
>   .commitOffsetsInFinalize()
>   .withoutMetadata()
>
>
> Regards
>
>
> Em ter, 18 de set de 2018 às 21:15, Raghu Angadi <ra...@google.com>
> escreveu:
>
>> Hi Eduardo,
>>
>> There another work around you can try without having to wait for 2.7.0
>> release: Use a wrapper to catch exception from KafkaConsumer#poll() and
>> pass the wrapper to withConsumerFactoryFn() for KafkIO reader [1].
>>
>> Using something like (such a wrapper is used in KafkasIO tests [2]):
>> private static class ConsumerFactoryFn
>>                 implements SerializableFunction<Map<String, Object>,
>> Consumer<byte[], byte[]>> {
>>   @Override
>>     public Consumer<byte[], byte[]> apply(Map<String, Object> config) {
>>       return new KafkaConsumer(config) {
>>           @Override
>>           public ConsumerRecords<K, V> poll(long timeout) {
>>               // work around for BEAM-5375
>>               while (true) {
>>                   try {
>>                     return super.poll(timeout);
>>                  } catch (Exception e) {
>>                     // LOG & sleep for sec
>>                 }
>>           }
>>        }
>>     }
>> }
>>
>> [1]:
>> https://github.com/apache/beam/blob/release-2.4.0/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaIO.java#L417
>> [2]:
>> https://github.com/apache/beam/blob/release-2.4.0/sdks/java/io/kafka/src/test/java/org/apache/beam/sdk/io/kafka/KafkaIOTest.java#L261
>>
>> On Tue, Sep 18, 2018 at 5:49 AM Eduardo Soldera <
>> eduardo.soldera@arquivei.com.br> wrote:
>>
>>> Hi Raghu, we're not sure how long the network was down. According to the
>>> logs no longer than one minute. A 30 second shutdown would work for the
>>> tests.
>>>
>>> Regards
>>>
>>> Em sex, 14 de set de 2018 às 21:41, Raghu Angadi <ra...@google.com>
>>> escreveu:
>>>
>>>> Thanks. I could repro myself as well. How long was the network down?
>>>>
>>>> Trying to get the fix into 2.7 RC2.
>>>>
>>>> On Fri, Sep 14, 2018 at 12:25 PM Eduardo Soldera <
>>>> eduardo.soldera@arquivei.com.br> wrote:
>>>>
>>>>> Just to make myself clear, I'm not sure how to use the patch but if
>>>>> you could send us some guidance would be great.
>>>>>
>>>>> Em sex, 14 de set de 2018 às 16:24, Eduardo Soldera <
>>>>> eduardo.soldera@arquivei.com.br> escreveu:
>>>>>
>>>>>> Hi Raghu, yes, it is feasible, would you do that for us? I'm not sure
>>>>>> how we'd use the patch. We're using SBT and Spotify's Scio with Scala.
>>>>>>
>>>>>> Thanks
>>>>>>
>>>>>> Em sex, 14 de set de 2018 às 16:07, Raghu Angadi <ra...@google.com>
>>>>>> escreveu:
>>>>>>
>>>>>>> Is is feasible for you to verify the fix in your dev job? I can make
>>>>>>> a patch against Beam 2.4 branch if you like.
>>>>>>>
>>>>>>> Raghu.
>>>>>>>
>>>>>>> On Fri, Sep 14, 2018 at 11:14 AM Eduardo Soldera <
>>>>>>> eduardo.soldera@arquivei.com.br> wrote:
>>>>>>>
>>>>>>>> Hi Raghu, thank you very much for the pull request.
>>>>>>>> We'll wait for the 2.7 Beam release.
>>>>>>>>
>>>>>>>> Regards!
>>>>>>>>
>>>>>>>> Em qui, 13 de set de 2018 às 18:19, Raghu Angadi <
>>>>>>>> rangadi@google.com> escreveu:
>>>>>>>>
>>>>>>>>> Fix: https://github.com/apache/beam/pull/6391
>>>>>>>>>
>>>>>>>>> On Wed, Sep 12, 2018 at 3:30 PM Raghu Angadi <ra...@google.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Filed BEAM-5375 <https://issues.apache.org/jira/browse/BEAM-5375>.
>>>>>>>>>> I will fix it later this week.
>>>>>>>>>>
>>>>>>>>>> On Wed, Sep 12, 2018 at 12:16 PM Raghu Angadi <ra...@google.com>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Wed, Sep 12, 2018 at 12:11 PM Raghu Angadi <
>>>>>>>>>>> rangadi@google.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Thanks for the job id, I looked at the worker logs (following
>>>>>>>>>>>> usual support oncall access protocol that provides temporary access to
>>>>>>>>>>>> things like logs in GCP):
>>>>>>>>>>>>
>>>>>>>>>>>> Root issue looks like consumerPollLoop() mentioned earlier
>>>>>>>>>>>> needs to handle unchecked exception. In your case it is clear that poll
>>>>>>>>>>>> thread exited with a runtime exception. The reader does not check for it
>>>>>>>>>>>> and continues to wait for poll thread to enqueue messages. A fix should
>>>>>>>>>>>> result in an IOException for read from the source. The runners will handle
>>>>>>>>>>>> that appropriately after that.  I will file a jira.
>>>>>>>>>>>>
>>>>>>>>>>>> https://github.com/apache/beam/blob/master/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaUnboundedReader.java#L678
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Ignore the link.. was pasted here by mistake.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> From the logs (with a comment below each one):
>>>>>>>>>>>>
>>>>>>>>>>>>    - 2018-09-12 06:13:07.345 PDT Reader-0: reading from
>>>>>>>>>>>>    kafka_topic-0 starting at offset 2
>>>>>>>>>>>>       - Implies the reader is initialized and poll thread is
>>>>>>>>>>>>       started.
>>>>>>>>>>>>    - 2018-09-12 06:13:07.780 PDT Reader-0: first record offset
>>>>>>>>>>>>    2
>>>>>>>>>>>>       - The reader actually got a message received by the poll
>>>>>>>>>>>>       thread from Kafka.
>>>>>>>>>>>>    - 2018-09-12 06:16:48.771 PDT Reader-0: exception while
>>>>>>>>>>>>    fetching latest offset for partition kafka_topic-0. will be retried.
>>>>>>>>>>>>       - This must have happened around the time when network
>>>>>>>>>>>>       was disrupted. This is from. Actual log is from another periodic task that
>>>>>>>>>>>>       fetches latest offsets for partitions.
>>>>>>>>>>>>
>>>>>>>>>>>> The poll thread must have died around the time network was
>>>>>>>>>>>> disrupted.
>>>>>>>>>>>>
>>>>>>>>>>>> The following log comes from kafka client itself and is printed
>>>>>>>>>>>> every second when KafkaIO fetches latest offset. This log seems to be added
>>>>>>>>>>>> in recent versions. It is probably an unintentional log. I don't think
>>>>>>>>>>>> there is any better to fetch latest offsets than how KafkaIO does now. This
>>>>>>>>>>>> is logged inside consumer.position() called at [1].
>>>>>>>>>>>>
>>>>>>>>>>>>    - 2018-09-12 06:13:11.786 PDT [Consumer
>>>>>>>>>>>>    clientId=consumer-2,
>>>>>>>>>>>>    groupId=Reader-0_offset_consumer_1735388161_genericPipe] Resetting offset
>>>>>>>>>>>>    for partition com.arquivei.dataeng.andre-0 to offset 3.
>>>>>>>>>>>>
>>>>>>>>>>>> [1]:
>>>>>>>>>>>> https://github.com/apache/beam/blob/master/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaUnboundedReader.java#L678
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> This 'Resetting offset' is harmless, but is quite annoying to
>>>>>>>>>>> see in the worker logs. One way to avoid is to set kafka consumer's log
>>>>>>>>>>> level to WARNING. Ideally KafkaIO itself should do something to avoid it
>>>>>>>>>>> without user option.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>> On Wed, Sep 12, 2018 at 10:27 AM Eduardo Soldera <
>>>>>>>>>>>> eduardo.soldera@arquivei.com.br> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi Raghu! The job_id of our dev job is
>>>>>>>>>>>>> 2018-09-12_06_11_48-5600553605191377866.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks!
>>>>>>>>>>>>>
>>>>>>>>>>>>> Em qua, 12 de set de 2018 às 14:18, Raghu Angadi <
>>>>>>>>>>>>> rangadi@google.com> escreveu:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks for debugging.
>>>>>>>>>>>>>> Can you provide the job_id of your dev job? The stacktrace
>>>>>>>>>>>>>> shows that there is no thread running 'consumerPollLoop()' which can
>>>>>>>>>>>>>> explain stuck reader. You will likely find a logs at line 594 & 587 [1].
>>>>>>>>>>>>>> Dataflow caches its readers and DirectRunner may not. That can explain
>>>>>>>>>>>>>> DirectRunner resume reads. The expectation in KafkaIO is that Kafka client
>>>>>>>>>>>>>> library takes care of retrying in case of connection problems (as
>>>>>>>>>>>>>> documented). It is possible that in some cases poll() throws and we need to
>>>>>>>>>>>>>> restart the client in KafkaIO.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> [1]:
>>>>>>>>>>>>>> https://github.com/apache/beam/blob/master/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaUnboundedReader.java#L594
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Wed, Sep 12, 2018 at 9:59 AM Eduardo Soldera <
>>>>>>>>>>>>>> eduardo.soldera@arquivei.com.br> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi Raghu, thanks for your help.
>>>>>>>>>>>>>>> Just answering your previous question, the following logs
>>>>>>>>>>>>>>> were the same as before the error, as if the pipeline were still getting
>>>>>>>>>>>>>>> the messages, for example:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> (...)
>>>>>>>>>>>>>>> Resetting offset for partition com.arquivei.dataeng.andre-0
>>>>>>>>>>>>>>> to offset 10.
>>>>>>>>>>>>>>> Resetting offset for partition com.arquivei.dataeng.andre-0
>>>>>>>>>>>>>>> to offset 15.
>>>>>>>>>>>>>>> ERROR
>>>>>>>>>>>>>>> Resetting offset for partition com.arquivei.dataeng.andre-0
>>>>>>>>>>>>>>> to offset 22.
>>>>>>>>>>>>>>> Resetting offset for partition com.arquivei.dataeng.andre-0
>>>>>>>>>>>>>>> to offset 30.
>>>>>>>>>>>>>>> (...)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> But when checking the Kafka Consumer Group, the current
>>>>>>>>>>>>>>> offset stays at 15, the commited offset from the last processed message,
>>>>>>>>>>>>>>> before the error.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> We'll file a bug, but we could now reproduce the issue in a
>>>>>>>>>>>>>>> Dev scenario.
>>>>>>>>>>>>>>> We started the same pipeline using the direct runner,
>>>>>>>>>>>>>>> without Google Dataflow. We blocked the Kafka Broker network and the same
>>>>>>>>>>>>>>> error was thrown. Then we unblocked the network and the pipeline was able
>>>>>>>>>>>>>>> to successfully process the subsequent messages.
>>>>>>>>>>>>>>> When we started the same pipeline in the Dataflow runner and
>>>>>>>>>>>>>>> did the same test, the same problem from our production scenario happened,
>>>>>>>>>>>>>>> Dataflow couldn't process the new messages. Unfortunately, we've stopped
>>>>>>>>>>>>>>> the dataflow job in production, but the problematic dev job is still
>>>>>>>>>>>>>>> running and the log file of the VM is attached. Thank you very much.
>>>>>>>>>>>>>>> Best regards
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Em ter, 11 de set de 2018 às 18:28, Raghu Angadi <
>>>>>>>>>>>>>>> rangadi@google.com> escreveu:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Specifically, I am interested if you have any thread
>>>>>>>>>>>>>>>> running 'consumerPollLoop()' [1]. There should always be one (if a worker
>>>>>>>>>>>>>>>> is assigned one of the partitions). It is possible that KafkaClient itself
>>>>>>>>>>>>>>>> is hasn't recovered from the group coordinator error (though unlikely).
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> https://github.com/apache/beam/blob/master/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaUnboundedReader.java#L570
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Tue, Sep 11, 2018 at 12:31 PM Raghu Angadi <
>>>>>>>>>>>>>>>> rangadi@google.com> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Hi Eduardo,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> In case of any error, the pipeline should keep on trying
>>>>>>>>>>>>>>>>> to fetch. I don't know about this particular error. Do you see any others
>>>>>>>>>>>>>>>>> afterwards in the log?
>>>>>>>>>>>>>>>>> Couple of things you could try if the logs are not useful
>>>>>>>>>>>>>>>>> :
>>>>>>>>>>>>>>>>>  - login to one of the VMs and get stacktrace of java
>>>>>>>>>>>>>>>>> worker (look for a container called java-streaming)
>>>>>>>>>>>>>>>>>  - file a support bug or stackoverflow question with jobid
>>>>>>>>>>>>>>>>> so that Dataflow oncall can take a look.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Raghu.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Tue, Sep 11, 2018 at 12:10 PM Eduardo Soldera <
>>>>>>>>>>>>>>>>> eduardo.soldera@arquivei.com.br> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>>> We have a Apache Beam pipeline running in Google Dataflow
>>>>>>>>>>>>>>>>>> using KafkaIO. Suddenly the pipeline stop fetching Kafka messages at all,
>>>>>>>>>>>>>>>>>> as our other workers from other pipelines continued to get Kafka messages.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> At the moment it stopped we got these messages:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I  [Consumer clientId=consumer-1, groupId=genericPipe] Error sending fetch request (sessionId=1396189203, epoch=2431598) to node 3: org.apache.kafka.common.errors.DisconnectException.
>>>>>>>>>>>>>>>>>> I  [Consumer clientId=consumer-1, groupId=genericPipe] Group coordinator 10.0.52.70:9093 (id: 2147483646 rack: null) is unavailable or invalid, will attempt rediscovery
>>>>>>>>>>>>>>>>>> I  [Consumer clientId=consumer-1, groupId=genericPipe] Discovered group coordinator 10.0.52.70:9093 (id: 2147483646 rack: null)
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> And then the pipeline stopped reading the messages.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> This is the KafkaIO setup  we have:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> KafkaIO.read[String,String]()
>>>>>>>>>>>>>>>>>>   .withBootstrapServers(server)
>>>>>>>>>>>>>>>>>>   .withTopic(topic)
>>>>>>>>>>>>>>>>>>   .withKeyDeserializer(classOf[StringDeserializer])
>>>>>>>>>>>>>>>>>>   .withValueDeserializer(classOf[StringDeserializer])
>>>>>>>>>>>>>>>>>>   .updateConsumerProperties(properties)
>>>>>>>>>>>>>>>>>>   .commitOffsetsInFinalize()
>>>>>>>>>>>>>>>>>>   .withoutMetadata()
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>  Any help will be much appreciated.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Best regards,
>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>> Eduardo Soldera Garcia
>>>>>>>>>>>>>>>>>> Data Engineer
>>>>>>>>>>>>>>>>>> (16) 3509-5555 | www.arquivei.com.br
>>>>>>>>>>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>>>>>>>>>>>> [image: Arquivei.com.br – Inteligência em Notas Fiscais]
>>>>>>>>>>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>>>>>>>>>>>> [image: Google seleciona Arquivei para imersão e mentoria
>>>>>>>>>>>>>>>>>> no Vale do Silício]
>>>>>>>>>>>>>>>>>> <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad>
>>>>>>>>>>>>>>>>>> <https://www.facebook.com/arquivei>
>>>>>>>>>>>>>>>>>> <https://www.linkedin.com/company/arquivei>
>>>>>>>>>>>>>>>>>> <https://www.youtube.com/watch?v=sSUUKxbXnxk>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>> Eduardo Soldera Garcia
>>>>>>>>>>>>>>> Data Engineer
>>>>>>>>>>>>>>> (16) 3509-5555 | www.arquivei.com.br
>>>>>>>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>>>>>>>>> [image: Arquivei.com.br – Inteligência em Notas Fiscais]
>>>>>>>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>>>>>>>>> [image: Google seleciona Arquivei para imersão e mentoria no
>>>>>>>>>>>>>>> Vale do Silício]
>>>>>>>>>>>>>>> <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad>
>>>>>>>>>>>>>>> <https://www.facebook.com/arquivei>
>>>>>>>>>>>>>>> <https://www.linkedin.com/company/arquivei>
>>>>>>>>>>>>>>> <https://www.youtube.com/watch?v=sSUUKxbXnxk>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>>> Eduardo Soldera Garcia
>>>>>>>>>>>>> Data Engineer
>>>>>>>>>>>>> (16) 3509-5555 | www.arquivei.com.br
>>>>>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>>>>>>> [image: Arquivei.com.br – Inteligência em Notas Fiscais]
>>>>>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>>>>>>> [image: Google seleciona Arquivei para imersão e mentoria no
>>>>>>>>>>>>> Vale do Silício]
>>>>>>>>>>>>> <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad>
>>>>>>>>>>>>> <https://www.facebook.com/arquivei>
>>>>>>>>>>>>> <https://www.linkedin.com/company/arquivei>
>>>>>>>>>>>>> <https://www.youtube.com/watch?v=sSUUKxbXnxk>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Eduardo Soldera Garcia
>>>>>>>> Data Engineer
>>>>>>>> (16) 3509-5555 | www.arquivei.com.br
>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>> [image: Arquivei.com.br – Inteligência em Notas Fiscais]
>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>> [image: Google seleciona Arquivei para imersão e mentoria no Vale
>>>>>>>> do Silício]
>>>>>>>> <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad>
>>>>>>>> <https://www.facebook.com/arquivei>
>>>>>>>> <https://www.linkedin.com/company/arquivei>
>>>>>>>> <https://www.youtube.com/watch?v=sSUUKxbXnxk>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>> --
>>>>>> Eduardo Soldera Garcia
>>>>>> Data Engineer
>>>>>> (16) 3509-5555 | www.arquivei.com.br
>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>> [image: Arquivei.com.br – Inteligência em Notas Fiscais]
>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>> [image: Google seleciona Arquivei para imersão e mentoria no Vale do
>>>>>> Silício]
>>>>>> <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad>
>>>>>> <https://www.facebook.com/arquivei>
>>>>>> <https://www.linkedin.com/company/arquivei>
>>>>>> <https://www.youtube.com/watch?v=sSUUKxbXnxk>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Eduardo Soldera Garcia
>>>>> Data Engineer
>>>>> (16) 3509-5555 | www.arquivei.com.br
>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>> [image: Arquivei.com.br – Inteligência em Notas Fiscais]
>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>> [image: Google seleciona Arquivei para imersão e mentoria no Vale do
>>>>> Silício]
>>>>> <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad>
>>>>> <https://www.facebook.com/arquivei>
>>>>> <https://www.linkedin.com/company/arquivei>
>>>>> <https://www.youtube.com/watch?v=sSUUKxbXnxk>
>>>>>
>>>>
>>>
>>> --
>>> Eduardo Soldera Garcia
>>> Data Engineer
>>> (16) 3509-5555 | www.arquivei.com.br
>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>> [image: Arquivei.com.br – Inteligência em Notas Fiscais]
>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>> [image: Google seleciona Arquivei para imersão e mentoria no Vale do
>>> Silício]
>>> <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad>
>>> <https://www.facebook.com/arquivei>
>>> <https://www.linkedin.com/company/arquivei>
>>> <https://www.youtube.com/watch?v=sSUUKxbXnxk>
>>>
>>
>
> --
> Eduardo Soldera Garcia
> Data Engineer
> (16) 3509-5555 | www.arquivei.com.br
> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
> [image: Arquivei.com.br – Inteligência em Notas Fiscais]
> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
> [image: Google seleciona Arquivei para imersão e mentoria no Vale do
> Silício]
> <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad>
> <https://www.facebook.com/arquivei>
> <https://www.linkedin.com/company/arquivei>
> <https://www.youtube.com/watch?v=sSUUKxbXnxk>
>

Re: Problem with KafkaIO

Posted by Eduardo Soldera <ed...@arquivei.com.br>.
Hi Raghu, thank you.

I'm not sure though what to pass as an argument:

KafkaIO.read[String,String]()
  .withBootstrapServers(server)
  .withTopic(topic)
  .withKeyDeserializer(classOf[StringDeserializer])
  .withValueDeserializer(classOf[StringDeserializer])
  .withConsumerFactoryFn(new KafkaExecutor.ConsumerFactoryFn(????????????????))
  .updateConsumerProperties(properties)
  .commitOffsetsInFinalize()
  .withoutMetadata()


Regards


Em ter, 18 de set de 2018 às 21:15, Raghu Angadi <ra...@google.com>
escreveu:

> Hi Eduardo,
>
> There another work around you can try without having to wait for 2.7.0
> release: Use a wrapper to catch exception from KafkaConsumer#poll() and
> pass the wrapper to withConsumerFactoryFn() for KafkIO reader [1].
>
> Using something like (such a wrapper is used in KafkasIO tests [2]):
> private static class ConsumerFactoryFn
>                 implements SerializableFunction<Map<String, Object>,
> Consumer<byte[], byte[]>> {
>   @Override
>     public Consumer<byte[], byte[]> apply(Map<String, Object> config) {
>       return new KafkaConsumer(config) {
>           @Override
>           public ConsumerRecords<K, V> poll(long timeout) {
>               // work around for BEAM-5375
>               while (true) {
>                   try {
>                     return super.poll(timeout);
>                  } catch (Exception e) {
>                     // LOG & sleep for sec
>                 }
>           }
>        }
>     }
> }
>
> [1]:
> https://github.com/apache/beam/blob/release-2.4.0/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaIO.java#L417
> [2]:
> https://github.com/apache/beam/blob/release-2.4.0/sdks/java/io/kafka/src/test/java/org/apache/beam/sdk/io/kafka/KafkaIOTest.java#L261
>
> On Tue, Sep 18, 2018 at 5:49 AM Eduardo Soldera <
> eduardo.soldera@arquivei.com.br> wrote:
>
>> Hi Raghu, we're not sure how long the network was down. According to the
>> logs no longer than one minute. A 30 second shutdown would work for the
>> tests.
>>
>> Regards
>>
>> Em sex, 14 de set de 2018 às 21:41, Raghu Angadi <ra...@google.com>
>> escreveu:
>>
>>> Thanks. I could repro myself as well. How long was the network down?
>>>
>>> Trying to get the fix into 2.7 RC2.
>>>
>>> On Fri, Sep 14, 2018 at 12:25 PM Eduardo Soldera <
>>> eduardo.soldera@arquivei.com.br> wrote:
>>>
>>>> Just to make myself clear, I'm not sure how to use the patch but if you
>>>> could send us some guidance would be great.
>>>>
>>>> Em sex, 14 de set de 2018 às 16:24, Eduardo Soldera <
>>>> eduardo.soldera@arquivei.com.br> escreveu:
>>>>
>>>>> Hi Raghu, yes, it is feasible, would you do that for us? I'm not sure
>>>>> how we'd use the patch. We're using SBT and Spotify's Scio with Scala.
>>>>>
>>>>> Thanks
>>>>>
>>>>> Em sex, 14 de set de 2018 às 16:07, Raghu Angadi <ra...@google.com>
>>>>> escreveu:
>>>>>
>>>>>> Is is feasible for you to verify the fix in your dev job? I can make
>>>>>> a patch against Beam 2.4 branch if you like.
>>>>>>
>>>>>> Raghu.
>>>>>>
>>>>>> On Fri, Sep 14, 2018 at 11:14 AM Eduardo Soldera <
>>>>>> eduardo.soldera@arquivei.com.br> wrote:
>>>>>>
>>>>>>> Hi Raghu, thank you very much for the pull request.
>>>>>>> We'll wait for the 2.7 Beam release.
>>>>>>>
>>>>>>> Regards!
>>>>>>>
>>>>>>> Em qui, 13 de set de 2018 às 18:19, Raghu Angadi <ra...@google.com>
>>>>>>> escreveu:
>>>>>>>
>>>>>>>> Fix: https://github.com/apache/beam/pull/6391
>>>>>>>>
>>>>>>>> On Wed, Sep 12, 2018 at 3:30 PM Raghu Angadi <ra...@google.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Filed BEAM-5375 <https://issues.apache.org/jira/browse/BEAM-5375>.
>>>>>>>>> I will fix it later this week.
>>>>>>>>>
>>>>>>>>> On Wed, Sep 12, 2018 at 12:16 PM Raghu Angadi <ra...@google.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Wed, Sep 12, 2018 at 12:11 PM Raghu Angadi <ra...@google.com>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Thanks for the job id, I looked at the worker logs (following
>>>>>>>>>>> usual support oncall access protocol that provides temporary access to
>>>>>>>>>>> things like logs in GCP):
>>>>>>>>>>>
>>>>>>>>>>> Root issue looks like consumerPollLoop() mentioned earlier needs
>>>>>>>>>>> to handle unchecked exception. In your case it is clear that poll thread
>>>>>>>>>>> exited with a runtime exception. The reader does not check for it and
>>>>>>>>>>> continues to wait for poll thread to enqueue messages. A fix should result
>>>>>>>>>>> in an IOException for read from the source. The runners will handle that
>>>>>>>>>>> appropriately after that.  I will file a jira.
>>>>>>>>>>>
>>>>>>>>>>> https://github.com/apache/beam/blob/master/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaUnboundedReader.java#L678
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Ignore the link.. was pasted here by mistake.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> From the logs (with a comment below each one):
>>>>>>>>>>>
>>>>>>>>>>>    - 2018-09-12 06:13:07.345 PDT Reader-0: reading from
>>>>>>>>>>>    kafka_topic-0 starting at offset 2
>>>>>>>>>>>       - Implies the reader is initialized and poll thread is
>>>>>>>>>>>       started.
>>>>>>>>>>>    - 2018-09-12 06:13:07.780 PDT Reader-0: first record offset 2
>>>>>>>>>>>       - The reader actually got a message received by the poll
>>>>>>>>>>>       thread from Kafka.
>>>>>>>>>>>    - 2018-09-12 06:16:48.771 PDT Reader-0: exception while
>>>>>>>>>>>    fetching latest offset for partition kafka_topic-0. will be retried.
>>>>>>>>>>>       - This must have happened around the time when network
>>>>>>>>>>>       was disrupted. This is from. Actual log is from another periodic task that
>>>>>>>>>>>       fetches latest offsets for partitions.
>>>>>>>>>>>
>>>>>>>>>>> The poll thread must have died around the time network was
>>>>>>>>>>> disrupted.
>>>>>>>>>>>
>>>>>>>>>>> The following log comes from kafka client itself and is printed
>>>>>>>>>>> every second when KafkaIO fetches latest offset. This log seems to be added
>>>>>>>>>>> in recent versions. It is probably an unintentional log. I don't think
>>>>>>>>>>> there is any better to fetch latest offsets than how KafkaIO does now. This
>>>>>>>>>>> is logged inside consumer.position() called at [1].
>>>>>>>>>>>
>>>>>>>>>>>    - 2018-09-12 06:13:11.786 PDT [Consumer clientId=consumer-2,
>>>>>>>>>>>    groupId=Reader-0_offset_consumer_1735388161_genericPipe] Resetting offset
>>>>>>>>>>>    for partition com.arquivei.dataeng.andre-0 to offset 3.
>>>>>>>>>>>
>>>>>>>>>>> [1]:
>>>>>>>>>>> https://github.com/apache/beam/blob/master/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaUnboundedReader.java#L678
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> This 'Resetting offset' is harmless, but is quite annoying to see
>>>>>>>>>> in the worker logs. One way to avoid is to set kafka consumer's log level
>>>>>>>>>> to WARNING. Ideally KafkaIO itself should do something to avoid it without
>>>>>>>>>> user option.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> On Wed, Sep 12, 2018 at 10:27 AM Eduardo Soldera <
>>>>>>>>>>> eduardo.soldera@arquivei.com.br> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi Raghu! The job_id of our dev job is
>>>>>>>>>>>> 2018-09-12_06_11_48-5600553605191377866.
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks!
>>>>>>>>>>>>
>>>>>>>>>>>> Em qua, 12 de set de 2018 às 14:18, Raghu Angadi <
>>>>>>>>>>>> rangadi@google.com> escreveu:
>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks for debugging.
>>>>>>>>>>>>> Can you provide the job_id of your dev job? The stacktrace
>>>>>>>>>>>>> shows that there is no thread running 'consumerPollLoop()' which can
>>>>>>>>>>>>> explain stuck reader. You will likely find a logs at line 594 & 587 [1].
>>>>>>>>>>>>> Dataflow caches its readers and DirectRunner may not. That can explain
>>>>>>>>>>>>> DirectRunner resume reads. The expectation in KafkaIO is that Kafka client
>>>>>>>>>>>>> library takes care of retrying in case of connection problems (as
>>>>>>>>>>>>> documented). It is possible that in some cases poll() throws and we need to
>>>>>>>>>>>>> restart the client in KafkaIO.
>>>>>>>>>>>>>
>>>>>>>>>>>>> [1]:
>>>>>>>>>>>>> https://github.com/apache/beam/blob/master/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaUnboundedReader.java#L594
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Wed, Sep 12, 2018 at 9:59 AM Eduardo Soldera <
>>>>>>>>>>>>> eduardo.soldera@arquivei.com.br> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi Raghu, thanks for your help.
>>>>>>>>>>>>>> Just answering your previous question, the following logs
>>>>>>>>>>>>>> were the same as before the error, as if the pipeline were still getting
>>>>>>>>>>>>>> the messages, for example:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> (...)
>>>>>>>>>>>>>> Resetting offset for partition com.arquivei.dataeng.andre-0
>>>>>>>>>>>>>> to offset 10.
>>>>>>>>>>>>>> Resetting offset for partition com.arquivei.dataeng.andre-0
>>>>>>>>>>>>>> to offset 15.
>>>>>>>>>>>>>> ERROR
>>>>>>>>>>>>>> Resetting offset for partition com.arquivei.dataeng.andre-0
>>>>>>>>>>>>>> to offset 22.
>>>>>>>>>>>>>> Resetting offset for partition com.arquivei.dataeng.andre-0
>>>>>>>>>>>>>> to offset 30.
>>>>>>>>>>>>>> (...)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> But when checking the Kafka Consumer Group, the current
>>>>>>>>>>>>>> offset stays at 15, the commited offset from the last processed message,
>>>>>>>>>>>>>> before the error.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> We'll file a bug, but we could now reproduce the issue in a
>>>>>>>>>>>>>> Dev scenario.
>>>>>>>>>>>>>> We started the same pipeline using the direct runner, without
>>>>>>>>>>>>>> Google Dataflow. We blocked the Kafka Broker network and the same error was
>>>>>>>>>>>>>> thrown. Then we unblocked the network and the pipeline was able to
>>>>>>>>>>>>>> successfully process the subsequent messages.
>>>>>>>>>>>>>> When we started the same pipeline in the Dataflow runner and
>>>>>>>>>>>>>> did the same test, the same problem from our production scenario happened,
>>>>>>>>>>>>>> Dataflow couldn't process the new messages. Unfortunately, we've stopped
>>>>>>>>>>>>>> the dataflow job in production, but the problematic dev job is still
>>>>>>>>>>>>>> running and the log file of the VM is attached. Thank you very much.
>>>>>>>>>>>>>> Best regards
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Em ter, 11 de set de 2018 às 18:28, Raghu Angadi <
>>>>>>>>>>>>>> rangadi@google.com> escreveu:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Specifically, I am interested if you have any thread running
>>>>>>>>>>>>>>> 'consumerPollLoop()' [1]. There should always be one (if a worker is
>>>>>>>>>>>>>>> assigned one of the partitions). It is possible that KafkaClient itself is
>>>>>>>>>>>>>>> hasn't recovered from the group coordinator error (though unlikely).
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> https://github.com/apache/beam/blob/master/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaUnboundedReader.java#L570
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Tue, Sep 11, 2018 at 12:31 PM Raghu Angadi <
>>>>>>>>>>>>>>> rangadi@google.com> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hi Eduardo,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> In case of any error, the pipeline should keep on trying to
>>>>>>>>>>>>>>>> fetch. I don't know about this particular error. Do you see any others
>>>>>>>>>>>>>>>> afterwards in the log?
>>>>>>>>>>>>>>>> Couple of things you could try if the logs are not useful :
>>>>>>>>>>>>>>>>  - login to one of the VMs and get stacktrace of java
>>>>>>>>>>>>>>>> worker (look for a container called java-streaming)
>>>>>>>>>>>>>>>>  - file a support bug or stackoverflow question with jobid
>>>>>>>>>>>>>>>> so that Dataflow oncall can take a look.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Raghu.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Tue, Sep 11, 2018 at 12:10 PM Eduardo Soldera <
>>>>>>>>>>>>>>>> eduardo.soldera@arquivei.com.br> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>> We have a Apache Beam pipeline running in Google Dataflow
>>>>>>>>>>>>>>>>> using KafkaIO. Suddenly the pipeline stop fetching Kafka messages at all,
>>>>>>>>>>>>>>>>> as our other workers from other pipelines continued to get Kafka messages.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> At the moment it stopped we got these messages:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I  [Consumer clientId=consumer-1, groupId=genericPipe] Error sending fetch request (sessionId=1396189203, epoch=2431598) to node 3: org.apache.kafka.common.errors.DisconnectException.
>>>>>>>>>>>>>>>>> I  [Consumer clientId=consumer-1, groupId=genericPipe] Group coordinator 10.0.52.70:9093 (id: 2147483646 rack: null) is unavailable or invalid, will attempt rediscovery
>>>>>>>>>>>>>>>>> I  [Consumer clientId=consumer-1, groupId=genericPipe] Discovered group coordinator 10.0.52.70:9093 (id: 2147483646 rack: null)
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> And then the pipeline stopped reading the messages.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> This is the KafkaIO setup  we have:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> KafkaIO.read[String,String]()
>>>>>>>>>>>>>>>>>   .withBootstrapServers(server)
>>>>>>>>>>>>>>>>>   .withTopic(topic)
>>>>>>>>>>>>>>>>>   .withKeyDeserializer(classOf[StringDeserializer])
>>>>>>>>>>>>>>>>>   .withValueDeserializer(classOf[StringDeserializer])
>>>>>>>>>>>>>>>>>   .updateConsumerProperties(properties)
>>>>>>>>>>>>>>>>>   .commitOffsetsInFinalize()
>>>>>>>>>>>>>>>>>   .withoutMetadata()
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>  Any help will be much appreciated.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Best regards,
>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>> Eduardo Soldera Garcia
>>>>>>>>>>>>>>>>> Data Engineer
>>>>>>>>>>>>>>>>> (16) 3509-5555 | www.arquivei.com.br
>>>>>>>>>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>>>>>>>>>>> [image: Arquivei.com.br – Inteligência em Notas Fiscais]
>>>>>>>>>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>>>>>>>>>>> [image: Google seleciona Arquivei para imersão e mentoria
>>>>>>>>>>>>>>>>> no Vale do Silício]
>>>>>>>>>>>>>>>>> <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad>
>>>>>>>>>>>>>>>>> <https://www.facebook.com/arquivei>
>>>>>>>>>>>>>>>>> <https://www.linkedin.com/company/arquivei>
>>>>>>>>>>>>>>>>> <https://www.youtube.com/watch?v=sSUUKxbXnxk>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> --
>>>>>>>>>>>>>> Eduardo Soldera Garcia
>>>>>>>>>>>>>> Data Engineer
>>>>>>>>>>>>>> (16) 3509-5555 | www.arquivei.com.br
>>>>>>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>>>>>>>> [image: Arquivei.com.br – Inteligência em Notas Fiscais]
>>>>>>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>>>>>>>> [image: Google seleciona Arquivei para imersão e mentoria no
>>>>>>>>>>>>>> Vale do Silício]
>>>>>>>>>>>>>> <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad>
>>>>>>>>>>>>>> <https://www.facebook.com/arquivei>
>>>>>>>>>>>>>> <https://www.linkedin.com/company/arquivei>
>>>>>>>>>>>>>> <https://www.youtube.com/watch?v=sSUUKxbXnxk>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> Eduardo Soldera Garcia
>>>>>>>>>>>> Data Engineer
>>>>>>>>>>>> (16) 3509-5555 | www.arquivei.com.br
>>>>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>>>>>> [image: Arquivei.com.br – Inteligência em Notas Fiscais]
>>>>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>>>>>> [image: Google seleciona Arquivei para imersão e mentoria no
>>>>>>>>>>>> Vale do Silício]
>>>>>>>>>>>> <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad>
>>>>>>>>>>>> <https://www.facebook.com/arquivei>
>>>>>>>>>>>> <https://www.linkedin.com/company/arquivei>
>>>>>>>>>>>> <https://www.youtube.com/watch?v=sSUUKxbXnxk>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Eduardo Soldera Garcia
>>>>>>> Data Engineer
>>>>>>> (16) 3509-5555 | www.arquivei.com.br
>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>> [image: Arquivei.com.br – Inteligência em Notas Fiscais]
>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>> [image: Google seleciona Arquivei para imersão e mentoria no Vale do
>>>>>>> Silício]
>>>>>>> <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad>
>>>>>>> <https://www.facebook.com/arquivei>
>>>>>>> <https://www.linkedin.com/company/arquivei>
>>>>>>> <https://www.youtube.com/watch?v=sSUUKxbXnxk>
>>>>>>>
>>>>>>
>>>>>
>>>>> --
>>>>> Eduardo Soldera Garcia
>>>>> Data Engineer
>>>>> (16) 3509-5555 | www.arquivei.com.br
>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>> [image: Arquivei.com.br – Inteligência em Notas Fiscais]
>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>> [image: Google seleciona Arquivei para imersão e mentoria no Vale do
>>>>> Silício]
>>>>> <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad>
>>>>> <https://www.facebook.com/arquivei>
>>>>> <https://www.linkedin.com/company/arquivei>
>>>>> <https://www.youtube.com/watch?v=sSUUKxbXnxk>
>>>>>
>>>>
>>>>
>>>> --
>>>> Eduardo Soldera Garcia
>>>> Data Engineer
>>>> (16) 3509-5555 | www.arquivei.com.br
>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>> [image: Arquivei.com.br – Inteligência em Notas Fiscais]
>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>> [image: Google seleciona Arquivei para imersão e mentoria no Vale do
>>>> Silício]
>>>> <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad>
>>>> <https://www.facebook.com/arquivei>
>>>> <https://www.linkedin.com/company/arquivei>
>>>> <https://www.youtube.com/watch?v=sSUUKxbXnxk>
>>>>
>>>
>>
>> --
>> Eduardo Soldera Garcia
>> Data Engineer
>> (16) 3509-5555 | www.arquivei.com.br
>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>> [image: Arquivei.com.br – Inteligência em Notas Fiscais]
>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>> [image: Google seleciona Arquivei para imersão e mentoria no Vale do
>> Silício]
>> <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad>
>> <https://www.facebook.com/arquivei>
>> <https://www.linkedin.com/company/arquivei>
>> <https://www.youtube.com/watch?v=sSUUKxbXnxk>
>>
>

-- 
Eduardo Soldera Garcia
Data Engineer
(16) 3509-5555 | www.arquivei.com.br
<https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
[image: Arquivei.com.br – Inteligência em Notas Fiscais]
<https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
[image: Google seleciona Arquivei para imersão e mentoria no Vale do
Silício]
<https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad>
<https://www.facebook.com/arquivei>
<https://www.linkedin.com/company/arquivei>
<https://www.youtube.com/watch?v=sSUUKxbXnxk>

Re: Problem with KafkaIO

Posted by Raghu Angadi <ra...@google.com>.
Hi Eduardo,

There another work around you can try without having to wait for 2.7.0
release: Use a wrapper to catch exception from KafkaConsumer#poll() and
pass the wrapper to withConsumerFactoryFn() for KafkIO reader [1].

Using something like (such a wrapper is used in KafkasIO tests [2]):
private static class ConsumerFactoryFn
                implements SerializableFunction<Map<String, Object>,
Consumer<byte[], byte[]>> {
  @Override
    public Consumer<byte[], byte[]> apply(Map<String, Object> config) {
      return new KafkaConsumer(config) {
          @Override
          public ConsumerRecords<K, V> poll(long timeout) {
              // work around for BEAM-5375
              while (true) {
                  try {
                    return super.poll(timeout);
                 } catch (Exception e) {
                    // LOG & sleep for sec
                }
          }
       }
    }
}

[1]:
https://github.com/apache/beam/blob/release-2.4.0/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaIO.java#L417
[2]:
https://github.com/apache/beam/blob/release-2.4.0/sdks/java/io/kafka/src/test/java/org/apache/beam/sdk/io/kafka/KafkaIOTest.java#L261

On Tue, Sep 18, 2018 at 5:49 AM Eduardo Soldera <
eduardo.soldera@arquivei.com.br> wrote:

> Hi Raghu, we're not sure how long the network was down. According to the
> logs no longer than one minute. A 30 second shutdown would work for the
> tests.
>
> Regards
>
> Em sex, 14 de set de 2018 às 21:41, Raghu Angadi <ra...@google.com>
> escreveu:
>
>> Thanks. I could repro myself as well. How long was the network down?
>>
>> Trying to get the fix into 2.7 RC2.
>>
>> On Fri, Sep 14, 2018 at 12:25 PM Eduardo Soldera <
>> eduardo.soldera@arquivei.com.br> wrote:
>>
>>> Just to make myself clear, I'm not sure how to use the patch but if you
>>> could send us some guidance would be great.
>>>
>>> Em sex, 14 de set de 2018 às 16:24, Eduardo Soldera <
>>> eduardo.soldera@arquivei.com.br> escreveu:
>>>
>>>> Hi Raghu, yes, it is feasible, would you do that for us? I'm not sure
>>>> how we'd use the patch. We're using SBT and Spotify's Scio with Scala.
>>>>
>>>> Thanks
>>>>
>>>> Em sex, 14 de set de 2018 às 16:07, Raghu Angadi <ra...@google.com>
>>>> escreveu:
>>>>
>>>>> Is is feasible for you to verify the fix in your dev job? I can make a
>>>>> patch against Beam 2.4 branch if you like.
>>>>>
>>>>> Raghu.
>>>>>
>>>>> On Fri, Sep 14, 2018 at 11:14 AM Eduardo Soldera <
>>>>> eduardo.soldera@arquivei.com.br> wrote:
>>>>>
>>>>>> Hi Raghu, thank you very much for the pull request.
>>>>>> We'll wait for the 2.7 Beam release.
>>>>>>
>>>>>> Regards!
>>>>>>
>>>>>> Em qui, 13 de set de 2018 às 18:19, Raghu Angadi <ra...@google.com>
>>>>>> escreveu:
>>>>>>
>>>>>>> Fix: https://github.com/apache/beam/pull/6391
>>>>>>>
>>>>>>> On Wed, Sep 12, 2018 at 3:30 PM Raghu Angadi <ra...@google.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Filed BEAM-5375 <https://issues.apache.org/jira/browse/BEAM-5375>.
>>>>>>>> I will fix it later this week.
>>>>>>>>
>>>>>>>> On Wed, Sep 12, 2018 at 12:16 PM Raghu Angadi <ra...@google.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Wed, Sep 12, 2018 at 12:11 PM Raghu Angadi <ra...@google.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Thanks for the job id, I looked at the worker logs (following
>>>>>>>>>> usual support oncall access protocol that provides temporary access to
>>>>>>>>>> things like logs in GCP):
>>>>>>>>>>
>>>>>>>>>> Root issue looks like consumerPollLoop() mentioned earlier needs
>>>>>>>>>> to handle unchecked exception. In your case it is clear that poll thread
>>>>>>>>>> exited with a runtime exception. The reader does not check for it and
>>>>>>>>>> continues to wait for poll thread to enqueue messages. A fix should result
>>>>>>>>>> in an IOException for read from the source. The runners will handle that
>>>>>>>>>> appropriately after that.  I will file a jira.
>>>>>>>>>>
>>>>>>>>>> https://github.com/apache/beam/blob/master/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaUnboundedReader.java#L678
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Ignore the link.. was pasted here by mistake.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> From the logs (with a comment below each one):
>>>>>>>>>>
>>>>>>>>>>    - 2018-09-12 06:13:07.345 PDT Reader-0: reading from
>>>>>>>>>>    kafka_topic-0 starting at offset 2
>>>>>>>>>>       - Implies the reader is initialized and poll thread is
>>>>>>>>>>       started.
>>>>>>>>>>    - 2018-09-12 06:13:07.780 PDT Reader-0: first record offset 2
>>>>>>>>>>       - The reader actually got a message received by the poll
>>>>>>>>>>       thread from Kafka.
>>>>>>>>>>    - 2018-09-12 06:16:48.771 PDT Reader-0: exception while
>>>>>>>>>>    fetching latest offset for partition kafka_topic-0. will be retried.
>>>>>>>>>>       - This must have happened around the time when network was
>>>>>>>>>>       disrupted. This is from. Actual log is from another periodic task that
>>>>>>>>>>       fetches latest offsets for partitions.
>>>>>>>>>>
>>>>>>>>>> The poll thread must have died around the time network was
>>>>>>>>>> disrupted.
>>>>>>>>>>
>>>>>>>>>> The following log comes from kafka client itself and is printed
>>>>>>>>>> every second when KafkaIO fetches latest offset. This log seems to be added
>>>>>>>>>> in recent versions. It is probably an unintentional log. I don't think
>>>>>>>>>> there is any better to fetch latest offsets than how KafkaIO does now. This
>>>>>>>>>> is logged inside consumer.position() called at [1].
>>>>>>>>>>
>>>>>>>>>>    - 2018-09-12 06:13:11.786 PDT [Consumer clientId=consumer-2,
>>>>>>>>>>    groupId=Reader-0_offset_consumer_1735388161_genericPipe] Resetting offset
>>>>>>>>>>    for partition com.arquivei.dataeng.andre-0 to offset 3.
>>>>>>>>>>
>>>>>>>>>> [1]:
>>>>>>>>>> https://github.com/apache/beam/blob/master/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaUnboundedReader.java#L678
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> This 'Resetting offset' is harmless, but is quite annoying to see
>>>>>>>>> in the worker logs. One way to avoid is to set kafka consumer's log level
>>>>>>>>> to WARNING. Ideally KafkaIO itself should do something to avoid it without
>>>>>>>>> user option.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> On Wed, Sep 12, 2018 at 10:27 AM Eduardo Soldera <
>>>>>>>>>> eduardo.soldera@arquivei.com.br> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi Raghu! The job_id of our dev job is
>>>>>>>>>>> 2018-09-12_06_11_48-5600553605191377866.
>>>>>>>>>>>
>>>>>>>>>>> Thanks!
>>>>>>>>>>>
>>>>>>>>>>> Em qua, 12 de set de 2018 às 14:18, Raghu Angadi <
>>>>>>>>>>> rangadi@google.com> escreveu:
>>>>>>>>>>>
>>>>>>>>>>>> Thanks for debugging.
>>>>>>>>>>>> Can you provide the job_id of your dev job? The stacktrace
>>>>>>>>>>>> shows that there is no thread running 'consumerPollLoop()' which can
>>>>>>>>>>>> explain stuck reader. You will likely find a logs at line 594 & 587 [1].
>>>>>>>>>>>> Dataflow caches its readers and DirectRunner may not. That can explain
>>>>>>>>>>>> DirectRunner resume reads. The expectation in KafkaIO is that Kafka client
>>>>>>>>>>>> library takes care of retrying in case of connection problems (as
>>>>>>>>>>>> documented). It is possible that in some cases poll() throws and we need to
>>>>>>>>>>>> restart the client in KafkaIO.
>>>>>>>>>>>>
>>>>>>>>>>>> [1]:
>>>>>>>>>>>> https://github.com/apache/beam/blob/master/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaUnboundedReader.java#L594
>>>>>>>>>>>>
>>>>>>>>>>>> On Wed, Sep 12, 2018 at 9:59 AM Eduardo Soldera <
>>>>>>>>>>>> eduardo.soldera@arquivei.com.br> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi Raghu, thanks for your help.
>>>>>>>>>>>>> Just answering your previous question, the following logs were
>>>>>>>>>>>>> the same as before the error, as if the pipeline were still getting the
>>>>>>>>>>>>> messages, for example:
>>>>>>>>>>>>>
>>>>>>>>>>>>> (...)
>>>>>>>>>>>>> Resetting offset for partition com.arquivei.dataeng.andre-0 to
>>>>>>>>>>>>> offset 10.
>>>>>>>>>>>>> Resetting offset for partition com.arquivei.dataeng.andre-0 to
>>>>>>>>>>>>> offset 15.
>>>>>>>>>>>>> ERROR
>>>>>>>>>>>>> Resetting offset for partition com.arquivei.dataeng.andre-0 to
>>>>>>>>>>>>> offset 22.
>>>>>>>>>>>>> Resetting offset for partition com.arquivei.dataeng.andre-0 to
>>>>>>>>>>>>> offset 30.
>>>>>>>>>>>>> (...)
>>>>>>>>>>>>>
>>>>>>>>>>>>> But when checking the Kafka Consumer Group, the current offset
>>>>>>>>>>>>> stays at 15, the commited offset from the last processed message, before
>>>>>>>>>>>>> the error.
>>>>>>>>>>>>>
>>>>>>>>>>>>> We'll file a bug, but we could now reproduce the issue in a
>>>>>>>>>>>>> Dev scenario.
>>>>>>>>>>>>> We started the same pipeline using the direct runner, without
>>>>>>>>>>>>> Google Dataflow. We blocked the Kafka Broker network and the same error was
>>>>>>>>>>>>> thrown. Then we unblocked the network and the pipeline was able to
>>>>>>>>>>>>> successfully process the subsequent messages.
>>>>>>>>>>>>> When we started the same pipeline in the Dataflow runner and
>>>>>>>>>>>>> did the same test, the same problem from our production scenario happened,
>>>>>>>>>>>>> Dataflow couldn't process the new messages. Unfortunately, we've stopped
>>>>>>>>>>>>> the dataflow job in production, but the problematic dev job is still
>>>>>>>>>>>>> running and the log file of the VM is attached. Thank you very much.
>>>>>>>>>>>>> Best regards
>>>>>>>>>>>>>
>>>>>>>>>>>>> Em ter, 11 de set de 2018 às 18:28, Raghu Angadi <
>>>>>>>>>>>>> rangadi@google.com> escreveu:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Specifically, I am interested if you have any thread running
>>>>>>>>>>>>>> 'consumerPollLoop()' [1]. There should always be one (if a worker is
>>>>>>>>>>>>>> assigned one of the partitions). It is possible that KafkaClient itself is
>>>>>>>>>>>>>> hasn't recovered from the group coordinator error (though unlikely).
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> https://github.com/apache/beam/blob/master/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaUnboundedReader.java#L570
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Tue, Sep 11, 2018 at 12:31 PM Raghu Angadi <
>>>>>>>>>>>>>> rangadi@google.com> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi Eduardo,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> In case of any error, the pipeline should keep on trying to
>>>>>>>>>>>>>>> fetch. I don't know about this particular error. Do you see any others
>>>>>>>>>>>>>>> afterwards in the log?
>>>>>>>>>>>>>>> Couple of things you could try if the logs are not useful :
>>>>>>>>>>>>>>>  - login to one of the VMs and get stacktrace of java worker
>>>>>>>>>>>>>>> (look for a container called java-streaming)
>>>>>>>>>>>>>>>  - file a support bug or stackoverflow question with jobid
>>>>>>>>>>>>>>> so that Dataflow oncall can take a look.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Raghu.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Tue, Sep 11, 2018 at 12:10 PM Eduardo Soldera <
>>>>>>>>>>>>>>> eduardo.soldera@arquivei.com.br> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>> We have a Apache Beam pipeline running in Google Dataflow
>>>>>>>>>>>>>>>> using KafkaIO. Suddenly the pipeline stop fetching Kafka messages at all,
>>>>>>>>>>>>>>>> as our other workers from other pipelines continued to get Kafka messages.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> At the moment it stopped we got these messages:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I  [Consumer clientId=consumer-1, groupId=genericPipe] Error sending fetch request (sessionId=1396189203, epoch=2431598) to node 3: org.apache.kafka.common.errors.DisconnectException.
>>>>>>>>>>>>>>>> I  [Consumer clientId=consumer-1, groupId=genericPipe] Group coordinator 10.0.52.70:9093 (id: 2147483646 rack: null) is unavailable or invalid, will attempt rediscovery
>>>>>>>>>>>>>>>> I  [Consumer clientId=consumer-1, groupId=genericPipe] Discovered group coordinator 10.0.52.70:9093 (id: 2147483646 rack: null)
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> And then the pipeline stopped reading the messages.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> This is the KafkaIO setup  we have:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> KafkaIO.read[String,String]()
>>>>>>>>>>>>>>>>   .withBootstrapServers(server)
>>>>>>>>>>>>>>>>   .withTopic(topic)
>>>>>>>>>>>>>>>>   .withKeyDeserializer(classOf[StringDeserializer])
>>>>>>>>>>>>>>>>   .withValueDeserializer(classOf[StringDeserializer])
>>>>>>>>>>>>>>>>   .updateConsumerProperties(properties)
>>>>>>>>>>>>>>>>   .commitOffsetsInFinalize()
>>>>>>>>>>>>>>>>   .withoutMetadata()
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>  Any help will be much appreciated.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Best regards,
>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>> Eduardo Soldera Garcia
>>>>>>>>>>>>>>>> Data Engineer
>>>>>>>>>>>>>>>> (16) 3509-5555 | www.arquivei.com.br
>>>>>>>>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>>>>>>>>>> [image: Arquivei.com.br – Inteligência em Notas Fiscais]
>>>>>>>>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>>>>>>>>>> [image: Google seleciona Arquivei para imersão e mentoria
>>>>>>>>>>>>>>>> no Vale do Silício]
>>>>>>>>>>>>>>>> <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad>
>>>>>>>>>>>>>>>> <https://www.facebook.com/arquivei>
>>>>>>>>>>>>>>>> <https://www.linkedin.com/company/arquivei>
>>>>>>>>>>>>>>>> <https://www.youtube.com/watch?v=sSUUKxbXnxk>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>>> Eduardo Soldera Garcia
>>>>>>>>>>>>> Data Engineer
>>>>>>>>>>>>> (16) 3509-5555 | www.arquivei.com.br
>>>>>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>>>>>>> [image: Arquivei.com.br – Inteligência em Notas Fiscais]
>>>>>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>>>>>>> [image: Google seleciona Arquivei para imersão e mentoria no
>>>>>>>>>>>>> Vale do Silício]
>>>>>>>>>>>>> <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad>
>>>>>>>>>>>>> <https://www.facebook.com/arquivei>
>>>>>>>>>>>>> <https://www.linkedin.com/company/arquivei>
>>>>>>>>>>>>> <https://www.youtube.com/watch?v=sSUUKxbXnxk>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Eduardo Soldera Garcia
>>>>>>>>>>> Data Engineer
>>>>>>>>>>> (16) 3509-5555 | www.arquivei.com.br
>>>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>>>>> [image: Arquivei.com.br – Inteligência em Notas Fiscais]
>>>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>>>>> [image: Google seleciona Arquivei para imersão e mentoria no
>>>>>>>>>>> Vale do Silício]
>>>>>>>>>>> <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad>
>>>>>>>>>>> <https://www.facebook.com/arquivei>
>>>>>>>>>>> <https://www.linkedin.com/company/arquivei>
>>>>>>>>>>> <https://www.youtube.com/watch?v=sSUUKxbXnxk>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>
>>>>>> --
>>>>>> Eduardo Soldera Garcia
>>>>>> Data Engineer
>>>>>> (16) 3509-5555 | www.arquivei.com.br
>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>> [image: Arquivei.com.br – Inteligência em Notas Fiscais]
>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>> [image: Google seleciona Arquivei para imersão e mentoria no Vale do
>>>>>> Silício]
>>>>>> <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad>
>>>>>> <https://www.facebook.com/arquivei>
>>>>>> <https://www.linkedin.com/company/arquivei>
>>>>>> <https://www.youtube.com/watch?v=sSUUKxbXnxk>
>>>>>>
>>>>>
>>>>
>>>> --
>>>> Eduardo Soldera Garcia
>>>> Data Engineer
>>>> (16) 3509-5555 | www.arquivei.com.br
>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>> [image: Arquivei.com.br – Inteligência em Notas Fiscais]
>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>> [image: Google seleciona Arquivei para imersão e mentoria no Vale do
>>>> Silício]
>>>> <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad>
>>>> <https://www.facebook.com/arquivei>
>>>> <https://www.linkedin.com/company/arquivei>
>>>> <https://www.youtube.com/watch?v=sSUUKxbXnxk>
>>>>
>>>
>>>
>>> --
>>> Eduardo Soldera Garcia
>>> Data Engineer
>>> (16) 3509-5555 | www.arquivei.com.br
>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>> [image: Arquivei.com.br – Inteligência em Notas Fiscais]
>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>> [image: Google seleciona Arquivei para imersão e mentoria no Vale do
>>> Silício]
>>> <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad>
>>> <https://www.facebook.com/arquivei>
>>> <https://www.linkedin.com/company/arquivei>
>>> <https://www.youtube.com/watch?v=sSUUKxbXnxk>
>>>
>>
>
> --
> Eduardo Soldera Garcia
> Data Engineer
> (16) 3509-5555 | www.arquivei.com.br
> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
> [image: Arquivei.com.br – Inteligência em Notas Fiscais]
> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
> [image: Google seleciona Arquivei para imersão e mentoria no Vale do
> Silício]
> <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad>
> <https://www.facebook.com/arquivei>
> <https://www.linkedin.com/company/arquivei>
> <https://www.youtube.com/watch?v=sSUUKxbXnxk>
>

Re: Problem with KafkaIO

Posted by Eduardo Soldera <ed...@arquivei.com.br>.
Hi Raghu, we're not sure how long the network was down. According to the
logs no longer than one minute. A 30 second shutdown would work for the
tests.

Regards

Em sex, 14 de set de 2018 às 21:41, Raghu Angadi <ra...@google.com>
escreveu:

> Thanks. I could repro myself as well. How long was the network down?
>
> Trying to get the fix into 2.7 RC2.
>
> On Fri, Sep 14, 2018 at 12:25 PM Eduardo Soldera <
> eduardo.soldera@arquivei.com.br> wrote:
>
>> Just to make myself clear, I'm not sure how to use the patch but if you
>> could send us some guidance would be great.
>>
>> Em sex, 14 de set de 2018 às 16:24, Eduardo Soldera <
>> eduardo.soldera@arquivei.com.br> escreveu:
>>
>>> Hi Raghu, yes, it is feasible, would you do that for us? I'm not sure
>>> how we'd use the patch. We're using SBT and Spotify's Scio with Scala.
>>>
>>> Thanks
>>>
>>> Em sex, 14 de set de 2018 às 16:07, Raghu Angadi <ra...@google.com>
>>> escreveu:
>>>
>>>> Is is feasible for you to verify the fix in your dev job? I can make a
>>>> patch against Beam 2.4 branch if you like.
>>>>
>>>> Raghu.
>>>>
>>>> On Fri, Sep 14, 2018 at 11:14 AM Eduardo Soldera <
>>>> eduardo.soldera@arquivei.com.br> wrote:
>>>>
>>>>> Hi Raghu, thank you very much for the pull request.
>>>>> We'll wait for the 2.7 Beam release.
>>>>>
>>>>> Regards!
>>>>>
>>>>> Em qui, 13 de set de 2018 às 18:19, Raghu Angadi <ra...@google.com>
>>>>> escreveu:
>>>>>
>>>>>> Fix: https://github.com/apache/beam/pull/6391
>>>>>>
>>>>>> On Wed, Sep 12, 2018 at 3:30 PM Raghu Angadi <ra...@google.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Filed BEAM-5375 <https://issues.apache.org/jira/browse/BEAM-5375>.
>>>>>>> I will fix it later this week.
>>>>>>>
>>>>>>> On Wed, Sep 12, 2018 at 12:16 PM Raghu Angadi <ra...@google.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Wed, Sep 12, 2018 at 12:11 PM Raghu Angadi <ra...@google.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Thanks for the job id, I looked at the worker logs (following
>>>>>>>>> usual support oncall access protocol that provides temporary access to
>>>>>>>>> things like logs in GCP):
>>>>>>>>>
>>>>>>>>> Root issue looks like consumerPollLoop() mentioned earlier needs
>>>>>>>>> to handle unchecked exception. In your case it is clear that poll thread
>>>>>>>>> exited with a runtime exception. The reader does not check for it and
>>>>>>>>> continues to wait for poll thread to enqueue messages. A fix should result
>>>>>>>>> in an IOException for read from the source. The runners will handle that
>>>>>>>>> appropriately after that.  I will file a jira.
>>>>>>>>>
>>>>>>>>> https://github.com/apache/beam/blob/master/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaUnboundedReader.java#L678
>>>>>>>>>
>>>>>>>>
>>>>>>>> Ignore the link.. was pasted here by mistake.
>>>>>>>>
>>>>>>>>
>>>>>>>>>
>>>>>>>>> From the logs (with a comment below each one):
>>>>>>>>>
>>>>>>>>>    - 2018-09-12 06:13:07.345 PDT Reader-0: reading from
>>>>>>>>>    kafka_topic-0 starting at offset 2
>>>>>>>>>       - Implies the reader is initialized and poll thread is
>>>>>>>>>       started.
>>>>>>>>>    - 2018-09-12 06:13:07.780 PDT Reader-0: first record offset 2
>>>>>>>>>       - The reader actually got a message received by the poll
>>>>>>>>>       thread from Kafka.
>>>>>>>>>    - 2018-09-12 06:16:48.771 PDT Reader-0: exception while
>>>>>>>>>    fetching latest offset for partition kafka_topic-0. will be retried.
>>>>>>>>>       - This must have happened around the time when network was
>>>>>>>>>       disrupted. This is from. Actual log is from another periodic task that
>>>>>>>>>       fetches latest offsets for partitions.
>>>>>>>>>
>>>>>>>>> The poll thread must have died around the time network was
>>>>>>>>> disrupted.
>>>>>>>>>
>>>>>>>>> The following log comes from kafka client itself and is printed
>>>>>>>>> every second when KafkaIO fetches latest offset. This log seems to be added
>>>>>>>>> in recent versions. It is probably an unintentional log. I don't think
>>>>>>>>> there is any better to fetch latest offsets than how KafkaIO does now. This
>>>>>>>>> is logged inside consumer.position() called at [1].
>>>>>>>>>
>>>>>>>>>    - 2018-09-12 06:13:11.786 PDT [Consumer clientId=consumer-2,
>>>>>>>>>    groupId=Reader-0_offset_consumer_1735388161_genericPipe] Resetting offset
>>>>>>>>>    for partition com.arquivei.dataeng.andre-0 to offset 3.
>>>>>>>>>
>>>>>>>>> [1]:
>>>>>>>>> https://github.com/apache/beam/blob/master/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaUnboundedReader.java#L678
>>>>>>>>>
>>>>>>>>
>>>>>>>> This 'Resetting offset' is harmless, but is quite annoying to see
>>>>>>>> in the worker logs. One way to avoid is to set kafka consumer's log level
>>>>>>>> to WARNING. Ideally KafkaIO itself should do something to avoid it without
>>>>>>>> user option.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>> On Wed, Sep 12, 2018 at 10:27 AM Eduardo Soldera <
>>>>>>>>> eduardo.soldera@arquivei.com.br> wrote:
>>>>>>>>>
>>>>>>>>>> Hi Raghu! The job_id of our dev job is
>>>>>>>>>> 2018-09-12_06_11_48-5600553605191377866.
>>>>>>>>>>
>>>>>>>>>> Thanks!
>>>>>>>>>>
>>>>>>>>>> Em qua, 12 de set de 2018 às 14:18, Raghu Angadi <
>>>>>>>>>> rangadi@google.com> escreveu:
>>>>>>>>>>
>>>>>>>>>>> Thanks for debugging.
>>>>>>>>>>> Can you provide the job_id of your dev job? The stacktrace shows
>>>>>>>>>>> that there is no thread running 'consumerPollLoop()' which can explain
>>>>>>>>>>> stuck reader. You will likely find a logs at line 594 & 587 [1].  Dataflow
>>>>>>>>>>> caches its readers and DirectRunner may not. That can explain DirectRunner
>>>>>>>>>>> resume reads. The expectation in KafkaIO is that Kafka client library takes
>>>>>>>>>>> care of retrying in case of connection problems (as documented). It is
>>>>>>>>>>> possible that in some cases poll() throws and we need to restart the client
>>>>>>>>>>> in KafkaIO.
>>>>>>>>>>>
>>>>>>>>>>> [1]:
>>>>>>>>>>> https://github.com/apache/beam/blob/master/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaUnboundedReader.java#L594
>>>>>>>>>>>
>>>>>>>>>>> On Wed, Sep 12, 2018 at 9:59 AM Eduardo Soldera <
>>>>>>>>>>> eduardo.soldera@arquivei.com.br> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi Raghu, thanks for your help.
>>>>>>>>>>>> Just answering your previous question, the following logs were
>>>>>>>>>>>> the same as before the error, as if the pipeline were still getting the
>>>>>>>>>>>> messages, for example:
>>>>>>>>>>>>
>>>>>>>>>>>> (...)
>>>>>>>>>>>> Resetting offset for partition com.arquivei.dataeng.andre-0 to
>>>>>>>>>>>> offset 10.
>>>>>>>>>>>> Resetting offset for partition com.arquivei.dataeng.andre-0 to
>>>>>>>>>>>> offset 15.
>>>>>>>>>>>> ERROR
>>>>>>>>>>>> Resetting offset for partition com.arquivei.dataeng.andre-0 to
>>>>>>>>>>>> offset 22.
>>>>>>>>>>>> Resetting offset for partition com.arquivei.dataeng.andre-0 to
>>>>>>>>>>>> offset 30.
>>>>>>>>>>>> (...)
>>>>>>>>>>>>
>>>>>>>>>>>> But when checking the Kafka Consumer Group, the current offset
>>>>>>>>>>>> stays at 15, the commited offset from the last processed message, before
>>>>>>>>>>>> the error.
>>>>>>>>>>>>
>>>>>>>>>>>> We'll file a bug, but we could now reproduce the issue in a Dev
>>>>>>>>>>>> scenario.
>>>>>>>>>>>> We started the same pipeline using the direct runner, without
>>>>>>>>>>>> Google Dataflow. We blocked the Kafka Broker network and the same error was
>>>>>>>>>>>> thrown. Then we unblocked the network and the pipeline was able to
>>>>>>>>>>>> successfully process the subsequent messages.
>>>>>>>>>>>> When we started the same pipeline in the Dataflow runner and
>>>>>>>>>>>> did the same test, the same problem from our production scenario happened,
>>>>>>>>>>>> Dataflow couldn't process the new messages. Unfortunately, we've stopped
>>>>>>>>>>>> the dataflow job in production, but the problematic dev job is still
>>>>>>>>>>>> running and the log file of the VM is attached. Thank you very much.
>>>>>>>>>>>> Best regards
>>>>>>>>>>>>
>>>>>>>>>>>> Em ter, 11 de set de 2018 às 18:28, Raghu Angadi <
>>>>>>>>>>>> rangadi@google.com> escreveu:
>>>>>>>>>>>>
>>>>>>>>>>>>> Specifically, I am interested if you have any thread running
>>>>>>>>>>>>> 'consumerPollLoop()' [1]. There should always be one (if a worker is
>>>>>>>>>>>>> assigned one of the partitions). It is possible that KafkaClient itself is
>>>>>>>>>>>>> hasn't recovered from the group coordinator error (though unlikely).
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> https://github.com/apache/beam/blob/master/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaUnboundedReader.java#L570
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Tue, Sep 11, 2018 at 12:31 PM Raghu Angadi <
>>>>>>>>>>>>> rangadi@google.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi Eduardo,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> In case of any error, the pipeline should keep on trying to
>>>>>>>>>>>>>> fetch. I don't know about this particular error. Do you see any others
>>>>>>>>>>>>>> afterwards in the log?
>>>>>>>>>>>>>> Couple of things you could try if the logs are not useful :
>>>>>>>>>>>>>>  - login to one of the VMs and get stacktrace of java worker
>>>>>>>>>>>>>> (look for a container called java-streaming)
>>>>>>>>>>>>>>  - file a support bug or stackoverflow question with jobid so
>>>>>>>>>>>>>> that Dataflow oncall can take a look.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Raghu.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Tue, Sep 11, 2018 at 12:10 PM Eduardo Soldera <
>>>>>>>>>>>>>> eduardo.soldera@arquivei.com.br> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>> We have a Apache Beam pipeline running in Google Dataflow
>>>>>>>>>>>>>>> using KafkaIO. Suddenly the pipeline stop fetching Kafka messages at all,
>>>>>>>>>>>>>>> as our other workers from other pipelines continued to get Kafka messages.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> At the moment it stopped we got these messages:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I  [Consumer clientId=consumer-1, groupId=genericPipe] Error sending fetch request (sessionId=1396189203, epoch=2431598) to node 3: org.apache.kafka.common.errors.DisconnectException.
>>>>>>>>>>>>>>> I  [Consumer clientId=consumer-1, groupId=genericPipe] Group coordinator 10.0.52.70:9093 (id: 2147483646 rack: null) is unavailable or invalid, will attempt rediscovery
>>>>>>>>>>>>>>> I  [Consumer clientId=consumer-1, groupId=genericPipe] Discovered group coordinator 10.0.52.70:9093 (id: 2147483646 rack: null)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> And then the pipeline stopped reading the messages.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> This is the KafkaIO setup  we have:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> KafkaIO.read[String,String]()
>>>>>>>>>>>>>>>   .withBootstrapServers(server)
>>>>>>>>>>>>>>>   .withTopic(topic)
>>>>>>>>>>>>>>>   .withKeyDeserializer(classOf[StringDeserializer])
>>>>>>>>>>>>>>>   .withValueDeserializer(classOf[StringDeserializer])
>>>>>>>>>>>>>>>   .updateConsumerProperties(properties)
>>>>>>>>>>>>>>>   .commitOffsetsInFinalize()
>>>>>>>>>>>>>>>   .withoutMetadata()
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>  Any help will be much appreciated.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Best regards,
>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>> Eduardo Soldera Garcia
>>>>>>>>>>>>>>> Data Engineer
>>>>>>>>>>>>>>> (16) 3509-5555 | www.arquivei.com.br
>>>>>>>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>>>>>>>>> [image: Arquivei.com.br – Inteligência em Notas Fiscais]
>>>>>>>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>>>>>>>>> [image: Google seleciona Arquivei para imersão e mentoria no
>>>>>>>>>>>>>>> Vale do Silício]
>>>>>>>>>>>>>>> <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad>
>>>>>>>>>>>>>>> <https://www.facebook.com/arquivei>
>>>>>>>>>>>>>>> <https://www.linkedin.com/company/arquivei>
>>>>>>>>>>>>>>> <https://www.youtube.com/watch?v=sSUUKxbXnxk>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> Eduardo Soldera Garcia
>>>>>>>>>>>> Data Engineer
>>>>>>>>>>>> (16) 3509-5555 | www.arquivei.com.br
>>>>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>>>>>> [image: Arquivei.com.br – Inteligência em Notas Fiscais]
>>>>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>>>>>> [image: Google seleciona Arquivei para imersão e mentoria no
>>>>>>>>>>>> Vale do Silício]
>>>>>>>>>>>> <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad>
>>>>>>>>>>>> <https://www.facebook.com/arquivei>
>>>>>>>>>>>> <https://www.linkedin.com/company/arquivei>
>>>>>>>>>>>> <https://www.youtube.com/watch?v=sSUUKxbXnxk>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Eduardo Soldera Garcia
>>>>>>>>>> Data Engineer
>>>>>>>>>> (16) 3509-5555 | www.arquivei.com.br
>>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>>>> [image: Arquivei.com.br – Inteligência em Notas Fiscais]
>>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>>>> [image: Google seleciona Arquivei para imersão e mentoria no Vale
>>>>>>>>>> do Silício]
>>>>>>>>>> <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad>
>>>>>>>>>> <https://www.facebook.com/arquivei>
>>>>>>>>>> <https://www.linkedin.com/company/arquivei>
>>>>>>>>>> <https://www.youtube.com/watch?v=sSUUKxbXnxk>
>>>>>>>>>>
>>>>>>>>>
>>>>>
>>>>> --
>>>>> Eduardo Soldera Garcia
>>>>> Data Engineer
>>>>> (16) 3509-5555 | www.arquivei.com.br
>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>> [image: Arquivei.com.br – Inteligência em Notas Fiscais]
>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>> [image: Google seleciona Arquivei para imersão e mentoria no Vale do
>>>>> Silício]
>>>>> <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad>
>>>>> <https://www.facebook.com/arquivei>
>>>>> <https://www.linkedin.com/company/arquivei>
>>>>> <https://www.youtube.com/watch?v=sSUUKxbXnxk>
>>>>>
>>>>
>>>
>>> --
>>> Eduardo Soldera Garcia
>>> Data Engineer
>>> (16) 3509-5555 | www.arquivei.com.br
>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>> [image: Arquivei.com.br – Inteligência em Notas Fiscais]
>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>> [image: Google seleciona Arquivei para imersão e mentoria no Vale do
>>> Silício]
>>> <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad>
>>> <https://www.facebook.com/arquivei>
>>> <https://www.linkedin.com/company/arquivei>
>>> <https://www.youtube.com/watch?v=sSUUKxbXnxk>
>>>
>>
>>
>> --
>> Eduardo Soldera Garcia
>> Data Engineer
>> (16) 3509-5555 | www.arquivei.com.br
>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>> [image: Arquivei.com.br – Inteligência em Notas Fiscais]
>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>> [image: Google seleciona Arquivei para imersão e mentoria no Vale do
>> Silício]
>> <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad>
>> <https://www.facebook.com/arquivei>
>> <https://www.linkedin.com/company/arquivei>
>> <https://www.youtube.com/watch?v=sSUUKxbXnxk>
>>
>

-- 
Eduardo Soldera Garcia
Data Engineer
(16) 3509-5555 | www.arquivei.com.br
<https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
[image: Arquivei.com.br – Inteligência em Notas Fiscais]
<https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
[image: Google seleciona Arquivei para imersão e mentoria no Vale do
Silício]
<https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad>
<https://www.facebook.com/arquivei>
<https://www.linkedin.com/company/arquivei>
<https://www.youtube.com/watch?v=sSUUKxbXnxk>

Re: Problem with KafkaIO

Posted by Raghu Angadi <ra...@google.com>.
Thanks. I could repro myself as well. How long was the network down?

Trying to get the fix into 2.7 RC2.

On Fri, Sep 14, 2018 at 12:25 PM Eduardo Soldera <
eduardo.soldera@arquivei.com.br> wrote:

> Just to make myself clear, I'm not sure how to use the patch but if you
> could send us some guidance would be great.
>
> Em sex, 14 de set de 2018 às 16:24, Eduardo Soldera <
> eduardo.soldera@arquivei.com.br> escreveu:
>
>> Hi Raghu, yes, it is feasible, would you do that for us? I'm not sure how
>> we'd use the patch. We're using SBT and Spotify's Scio with Scala.
>>
>> Thanks
>>
>> Em sex, 14 de set de 2018 às 16:07, Raghu Angadi <ra...@google.com>
>> escreveu:
>>
>>> Is is feasible for you to verify the fix in your dev job? I can make a
>>> patch against Beam 2.4 branch if you like.
>>>
>>> Raghu.
>>>
>>> On Fri, Sep 14, 2018 at 11:14 AM Eduardo Soldera <
>>> eduardo.soldera@arquivei.com.br> wrote:
>>>
>>>> Hi Raghu, thank you very much for the pull request.
>>>> We'll wait for the 2.7 Beam release.
>>>>
>>>> Regards!
>>>>
>>>> Em qui, 13 de set de 2018 às 18:19, Raghu Angadi <ra...@google.com>
>>>> escreveu:
>>>>
>>>>> Fix: https://github.com/apache/beam/pull/6391
>>>>>
>>>>> On Wed, Sep 12, 2018 at 3:30 PM Raghu Angadi <ra...@google.com>
>>>>> wrote:
>>>>>
>>>>>> Filed BEAM-5375 <https://issues.apache.org/jira/browse/BEAM-5375>. I
>>>>>> will fix it later this week.
>>>>>>
>>>>>> On Wed, Sep 12, 2018 at 12:16 PM Raghu Angadi <ra...@google.com>
>>>>>> wrote:
>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Wed, Sep 12, 2018 at 12:11 PM Raghu Angadi <ra...@google.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Thanks for the job id, I looked at the worker logs (following usual
>>>>>>>> support oncall access protocol that provides temporary access to things
>>>>>>>> like logs in GCP):
>>>>>>>>
>>>>>>>> Root issue looks like consumerPollLoop() mentioned earlier needs to
>>>>>>>> handle unchecked exception. In your case it is clear that poll thread
>>>>>>>> exited with a runtime exception. The reader does not check for it and
>>>>>>>> continues to wait for poll thread to enqueue messages. A fix should result
>>>>>>>> in an IOException for read from the source. The runners will handle that
>>>>>>>> appropriately after that.  I will file a jira.
>>>>>>>>
>>>>>>>> https://github.com/apache/beam/blob/master/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaUnboundedReader.java#L678
>>>>>>>>
>>>>>>>
>>>>>>> Ignore the link.. was pasted here by mistake.
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>> From the logs (with a comment below each one):
>>>>>>>>
>>>>>>>>    - 2018-09-12 06:13:07.345 PDT Reader-0: reading from
>>>>>>>>    kafka_topic-0 starting at offset 2
>>>>>>>>       - Implies the reader is initialized and poll thread is
>>>>>>>>       started.
>>>>>>>>    - 2018-09-12 06:13:07.780 PDT Reader-0: first record offset 2
>>>>>>>>       - The reader actually got a message received by the poll
>>>>>>>>       thread from Kafka.
>>>>>>>>    - 2018-09-12 06:16:48.771 PDT Reader-0: exception while
>>>>>>>>    fetching latest offset for partition kafka_topic-0. will be retried.
>>>>>>>>       - This must have happened around the time when network was
>>>>>>>>       disrupted. This is from. Actual log is from another periodic task that
>>>>>>>>       fetches latest offsets for partitions.
>>>>>>>>
>>>>>>>> The poll thread must have died around the time network was
>>>>>>>> disrupted.
>>>>>>>>
>>>>>>>> The following log comes from kafka client itself and is printed
>>>>>>>> every second when KafkaIO fetches latest offset. This log seems to be added
>>>>>>>> in recent versions. It is probably an unintentional log. I don't think
>>>>>>>> there is any better to fetch latest offsets than how KafkaIO does now. This
>>>>>>>> is logged inside consumer.position() called at [1].
>>>>>>>>
>>>>>>>>    - 2018-09-12 06:13:11.786 PDT [Consumer clientId=consumer-2,
>>>>>>>>    groupId=Reader-0_offset_consumer_1735388161_genericPipe] Resetting offset
>>>>>>>>    for partition com.arquivei.dataeng.andre-0 to offset 3.
>>>>>>>>
>>>>>>>> [1]:
>>>>>>>> https://github.com/apache/beam/blob/master/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaUnboundedReader.java#L678
>>>>>>>>
>>>>>>>
>>>>>>> This 'Resetting offset' is harmless, but is quite annoying to see in
>>>>>>> the worker logs. One way to avoid is to set kafka consumer's log level to
>>>>>>> WARNING. Ideally KafkaIO itself should do something to avoid it without
>>>>>>> user option.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> On Wed, Sep 12, 2018 at 10:27 AM Eduardo Soldera <
>>>>>>>> eduardo.soldera@arquivei.com.br> wrote:
>>>>>>>>
>>>>>>>>> Hi Raghu! The job_id of our dev job is
>>>>>>>>> 2018-09-12_06_11_48-5600553605191377866.
>>>>>>>>>
>>>>>>>>> Thanks!
>>>>>>>>>
>>>>>>>>> Em qua, 12 de set de 2018 às 14:18, Raghu Angadi <
>>>>>>>>> rangadi@google.com> escreveu:
>>>>>>>>>
>>>>>>>>>> Thanks for debugging.
>>>>>>>>>> Can you provide the job_id of your dev job? The stacktrace shows
>>>>>>>>>> that there is no thread running 'consumerPollLoop()' which can explain
>>>>>>>>>> stuck reader. You will likely find a logs at line 594 & 587 [1].  Dataflow
>>>>>>>>>> caches its readers and DirectRunner may not. That can explain DirectRunner
>>>>>>>>>> resume reads. The expectation in KafkaIO is that Kafka client library takes
>>>>>>>>>> care of retrying in case of connection problems (as documented). It is
>>>>>>>>>> possible that in some cases poll() throws and we need to restart the client
>>>>>>>>>> in KafkaIO.
>>>>>>>>>>
>>>>>>>>>> [1]:
>>>>>>>>>> https://github.com/apache/beam/blob/master/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaUnboundedReader.java#L594
>>>>>>>>>>
>>>>>>>>>> On Wed, Sep 12, 2018 at 9:59 AM Eduardo Soldera <
>>>>>>>>>> eduardo.soldera@arquivei.com.br> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi Raghu, thanks for your help.
>>>>>>>>>>> Just answering your previous question, the following logs were
>>>>>>>>>>> the same as before the error, as if the pipeline were still getting the
>>>>>>>>>>> messages, for example:
>>>>>>>>>>>
>>>>>>>>>>> (...)
>>>>>>>>>>> Resetting offset for partition com.arquivei.dataeng.andre-0 to
>>>>>>>>>>> offset 10.
>>>>>>>>>>> Resetting offset for partition com.arquivei.dataeng.andre-0 to
>>>>>>>>>>> offset 15.
>>>>>>>>>>> ERROR
>>>>>>>>>>> Resetting offset for partition com.arquivei.dataeng.andre-0 to
>>>>>>>>>>> offset 22.
>>>>>>>>>>> Resetting offset for partition com.arquivei.dataeng.andre-0 to
>>>>>>>>>>> offset 30.
>>>>>>>>>>> (...)
>>>>>>>>>>>
>>>>>>>>>>> But when checking the Kafka Consumer Group, the current offset
>>>>>>>>>>> stays at 15, the commited offset from the last processed message, before
>>>>>>>>>>> the error.
>>>>>>>>>>>
>>>>>>>>>>> We'll file a bug, but we could now reproduce the issue in a Dev
>>>>>>>>>>> scenario.
>>>>>>>>>>> We started the same pipeline using the direct runner, without
>>>>>>>>>>> Google Dataflow. We blocked the Kafka Broker network and the same error was
>>>>>>>>>>> thrown. Then we unblocked the network and the pipeline was able to
>>>>>>>>>>> successfully process the subsequent messages.
>>>>>>>>>>> When we started the same pipeline in the Dataflow runner and did
>>>>>>>>>>> the same test, the same problem from our production scenario happened,
>>>>>>>>>>> Dataflow couldn't process the new messages. Unfortunately, we've stopped
>>>>>>>>>>> the dataflow job in production, but the problematic dev job is still
>>>>>>>>>>> running and the log file of the VM is attached. Thank you very much.
>>>>>>>>>>> Best regards
>>>>>>>>>>>
>>>>>>>>>>> Em ter, 11 de set de 2018 às 18:28, Raghu Angadi <
>>>>>>>>>>> rangadi@google.com> escreveu:
>>>>>>>>>>>
>>>>>>>>>>>> Specifically, I am interested if you have any thread running
>>>>>>>>>>>> 'consumerPollLoop()' [1]. There should always be one (if a worker is
>>>>>>>>>>>> assigned one of the partitions). It is possible that KafkaClient itself is
>>>>>>>>>>>> hasn't recovered from the group coordinator error (though unlikely).
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> https://github.com/apache/beam/blob/master/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaUnboundedReader.java#L570
>>>>>>>>>>>>
>>>>>>>>>>>> On Tue, Sep 11, 2018 at 12:31 PM Raghu Angadi <
>>>>>>>>>>>> rangadi@google.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi Eduardo,
>>>>>>>>>>>>>
>>>>>>>>>>>>> In case of any error, the pipeline should keep on trying to
>>>>>>>>>>>>> fetch. I don't know about this particular error. Do you see any others
>>>>>>>>>>>>> afterwards in the log?
>>>>>>>>>>>>> Couple of things you could try if the logs are not useful :
>>>>>>>>>>>>>  - login to one of the VMs and get stacktrace of java worker
>>>>>>>>>>>>> (look for a container called java-streaming)
>>>>>>>>>>>>>  - file a support bug or stackoverflow question with jobid so
>>>>>>>>>>>>> that Dataflow oncall can take a look.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Raghu.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Tue, Sep 11, 2018 at 12:10 PM Eduardo Soldera <
>>>>>>>>>>>>> eduardo.soldera@arquivei.com.br> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>> We have a Apache Beam pipeline running in Google Dataflow
>>>>>>>>>>>>>> using KafkaIO. Suddenly the pipeline stop fetching Kafka messages at all,
>>>>>>>>>>>>>> as our other workers from other pipelines continued to get Kafka messages.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> At the moment it stopped we got these messages:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I  [Consumer clientId=consumer-1, groupId=genericPipe] Error sending fetch request (sessionId=1396189203, epoch=2431598) to node 3: org.apache.kafka.common.errors.DisconnectException.
>>>>>>>>>>>>>> I  [Consumer clientId=consumer-1, groupId=genericPipe] Group coordinator 10.0.52.70:9093 (id: 2147483646 rack: null) is unavailable or invalid, will attempt rediscovery
>>>>>>>>>>>>>> I  [Consumer clientId=consumer-1, groupId=genericPipe] Discovered group coordinator 10.0.52.70:9093 (id: 2147483646 rack: null)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> And then the pipeline stopped reading the messages.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> This is the KafkaIO setup  we have:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> KafkaIO.read[String,String]()
>>>>>>>>>>>>>>   .withBootstrapServers(server)
>>>>>>>>>>>>>>   .withTopic(topic)
>>>>>>>>>>>>>>   .withKeyDeserializer(classOf[StringDeserializer])
>>>>>>>>>>>>>>   .withValueDeserializer(classOf[StringDeserializer])
>>>>>>>>>>>>>>   .updateConsumerProperties(properties)
>>>>>>>>>>>>>>   .commitOffsetsInFinalize()
>>>>>>>>>>>>>>   .withoutMetadata()
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>  Any help will be much appreciated.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Best regards,
>>>>>>>>>>>>>> --
>>>>>>>>>>>>>> Eduardo Soldera Garcia
>>>>>>>>>>>>>> Data Engineer
>>>>>>>>>>>>>> (16) 3509-5555 | www.arquivei.com.br
>>>>>>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>>>>>>>> [image: Arquivei.com.br – Inteligência em Notas Fiscais]
>>>>>>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>>>>>>>> [image: Google seleciona Arquivei para imersão e mentoria no
>>>>>>>>>>>>>> Vale do Silício]
>>>>>>>>>>>>>> <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad>
>>>>>>>>>>>>>> <https://www.facebook.com/arquivei>
>>>>>>>>>>>>>> <https://www.linkedin.com/company/arquivei>
>>>>>>>>>>>>>> <https://www.youtube.com/watch?v=sSUUKxbXnxk>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Eduardo Soldera Garcia
>>>>>>>>>>> Data Engineer
>>>>>>>>>>> (16) 3509-5555 | www.arquivei.com.br
>>>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>>>>> [image: Arquivei.com.br – Inteligência em Notas Fiscais]
>>>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>>>>> [image: Google seleciona Arquivei para imersão e mentoria no
>>>>>>>>>>> Vale do Silício]
>>>>>>>>>>> <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad>
>>>>>>>>>>> <https://www.facebook.com/arquivei>
>>>>>>>>>>> <https://www.linkedin.com/company/arquivei>
>>>>>>>>>>> <https://www.youtube.com/watch?v=sSUUKxbXnxk>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Eduardo Soldera Garcia
>>>>>>>>> Data Engineer
>>>>>>>>> (16) 3509-5555 | www.arquivei.com.br
>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>>> [image: Arquivei.com.br – Inteligência em Notas Fiscais]
>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>>> [image: Google seleciona Arquivei para imersão e mentoria no Vale
>>>>>>>>> do Silício]
>>>>>>>>> <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad>
>>>>>>>>> <https://www.facebook.com/arquivei>
>>>>>>>>> <https://www.linkedin.com/company/arquivei>
>>>>>>>>> <https://www.youtube.com/watch?v=sSUUKxbXnxk>
>>>>>>>>>
>>>>>>>>
>>>>
>>>> --
>>>> Eduardo Soldera Garcia
>>>> Data Engineer
>>>> (16) 3509-5555 | www.arquivei.com.br
>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>> [image: Arquivei.com.br – Inteligência em Notas Fiscais]
>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>> [image: Google seleciona Arquivei para imersão e mentoria no Vale do
>>>> Silício]
>>>> <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad>
>>>> <https://www.facebook.com/arquivei>
>>>> <https://www.linkedin.com/company/arquivei>
>>>> <https://www.youtube.com/watch?v=sSUUKxbXnxk>
>>>>
>>>
>>
>> --
>> Eduardo Soldera Garcia
>> Data Engineer
>> (16) 3509-5555 | www.arquivei.com.br
>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>> [image: Arquivei.com.br – Inteligência em Notas Fiscais]
>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>> [image: Google seleciona Arquivei para imersão e mentoria no Vale do
>> Silício]
>> <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad>
>> <https://www.facebook.com/arquivei>
>> <https://www.linkedin.com/company/arquivei>
>> <https://www.youtube.com/watch?v=sSUUKxbXnxk>
>>
>
>
> --
> Eduardo Soldera Garcia
> Data Engineer
> (16) 3509-5555 | www.arquivei.com.br
> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
> [image: Arquivei.com.br – Inteligência em Notas Fiscais]
> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
> [image: Google seleciona Arquivei para imersão e mentoria no Vale do
> Silício]
> <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad>
> <https://www.facebook.com/arquivei>
> <https://www.linkedin.com/company/arquivei>
> <https://www.youtube.com/watch?v=sSUUKxbXnxk>
>

Re: Problem with KafkaIO

Posted by Eduardo Soldera <ed...@arquivei.com.br>.
Just to make myself clear, I'm not sure how to use the patch but if you
could send us some guidance would be great.

Em sex, 14 de set de 2018 às 16:24, Eduardo Soldera <
eduardo.soldera@arquivei.com.br> escreveu:

> Hi Raghu, yes, it is feasible, would you do that for us? I'm not sure how
> we'd use the patch. We're using SBT and Spotify's Scio with Scala.
>
> Thanks
>
> Em sex, 14 de set de 2018 às 16:07, Raghu Angadi <ra...@google.com>
> escreveu:
>
>> Is is feasible for you to verify the fix in your dev job? I can make a
>> patch against Beam 2.4 branch if you like.
>>
>> Raghu.
>>
>> On Fri, Sep 14, 2018 at 11:14 AM Eduardo Soldera <
>> eduardo.soldera@arquivei.com.br> wrote:
>>
>>> Hi Raghu, thank you very much for the pull request.
>>> We'll wait for the 2.7 Beam release.
>>>
>>> Regards!
>>>
>>> Em qui, 13 de set de 2018 às 18:19, Raghu Angadi <ra...@google.com>
>>> escreveu:
>>>
>>>> Fix: https://github.com/apache/beam/pull/6391
>>>>
>>>> On Wed, Sep 12, 2018 at 3:30 PM Raghu Angadi <ra...@google.com>
>>>> wrote:
>>>>
>>>>> Filed BEAM-5375 <https://issues.apache.org/jira/browse/BEAM-5375>. I
>>>>> will fix it later this week.
>>>>>
>>>>> On Wed, Sep 12, 2018 at 12:16 PM Raghu Angadi <ra...@google.com>
>>>>> wrote:
>>>>>
>>>>>>
>>>>>>
>>>>>> On Wed, Sep 12, 2018 at 12:11 PM Raghu Angadi <ra...@google.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Thanks for the job id, I looked at the worker logs (following usual
>>>>>>> support oncall access protocol that provides temporary access to things
>>>>>>> like logs in GCP):
>>>>>>>
>>>>>>> Root issue looks like consumerPollLoop() mentioned earlier needs to
>>>>>>> handle unchecked exception. In your case it is clear that poll thread
>>>>>>> exited with a runtime exception. The reader does not check for it and
>>>>>>> continues to wait for poll thread to enqueue messages. A fix should result
>>>>>>> in an IOException for read from the source. The runners will handle that
>>>>>>> appropriately after that.  I will file a jira.
>>>>>>>
>>>>>>> https://github.com/apache/beam/blob/master/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaUnboundedReader.java#L678
>>>>>>>
>>>>>>
>>>>>> Ignore the link.. was pasted here by mistake.
>>>>>>
>>>>>>
>>>>>>>
>>>>>>> From the logs (with a comment below each one):
>>>>>>>
>>>>>>>    - 2018-09-12 06:13:07.345 PDT Reader-0: reading from
>>>>>>>    kafka_topic-0 starting at offset 2
>>>>>>>       - Implies the reader is initialized and poll thread is
>>>>>>>       started.
>>>>>>>    - 2018-09-12 06:13:07.780 PDT Reader-0: first record offset 2
>>>>>>>       - The reader actually got a message received by the poll
>>>>>>>       thread from Kafka.
>>>>>>>    - 2018-09-12 06:16:48.771 PDT Reader-0: exception while fetching
>>>>>>>    latest offset for partition kafka_topic-0. will be retried.
>>>>>>>       - This must have happened around the time when network was
>>>>>>>       disrupted. This is from. Actual log is from another periodic task that
>>>>>>>       fetches latest offsets for partitions.
>>>>>>>
>>>>>>> The poll thread must have died around the time network was
>>>>>>> disrupted.
>>>>>>>
>>>>>>> The following log comes from kafka client itself and is printed
>>>>>>> every second when KafkaIO fetches latest offset. This log seems to be added
>>>>>>> in recent versions. It is probably an unintentional log. I don't think
>>>>>>> there is any better to fetch latest offsets than how KafkaIO does now. This
>>>>>>> is logged inside consumer.position() called at [1].
>>>>>>>
>>>>>>>    - 2018-09-12 06:13:11.786 PDT [Consumer clientId=consumer-2,
>>>>>>>    groupId=Reader-0_offset_consumer_1735388161_genericPipe] Resetting offset
>>>>>>>    for partition com.arquivei.dataeng.andre-0 to offset 3.
>>>>>>>
>>>>>>> [1]:
>>>>>>> https://github.com/apache/beam/blob/master/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaUnboundedReader.java#L678
>>>>>>>
>>>>>>
>>>>>> This 'Resetting offset' is harmless, but is quite annoying to see in
>>>>>> the worker logs. One way to avoid is to set kafka consumer's log level to
>>>>>> WARNING. Ideally KafkaIO itself should do something to avoid it without
>>>>>> user option.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>> On Wed, Sep 12, 2018 at 10:27 AM Eduardo Soldera <
>>>>>>> eduardo.soldera@arquivei.com.br> wrote:
>>>>>>>
>>>>>>>> Hi Raghu! The job_id of our dev job is
>>>>>>>> 2018-09-12_06_11_48-5600553605191377866.
>>>>>>>>
>>>>>>>> Thanks!
>>>>>>>>
>>>>>>>> Em qua, 12 de set de 2018 às 14:18, Raghu Angadi <
>>>>>>>> rangadi@google.com> escreveu:
>>>>>>>>
>>>>>>>>> Thanks for debugging.
>>>>>>>>> Can you provide the job_id of your dev job? The stacktrace shows
>>>>>>>>> that there is no thread running 'consumerPollLoop()' which can explain
>>>>>>>>> stuck reader. You will likely find a logs at line 594 & 587 [1].  Dataflow
>>>>>>>>> caches its readers and DirectRunner may not. That can explain DirectRunner
>>>>>>>>> resume reads. The expectation in KafkaIO is that Kafka client library takes
>>>>>>>>> care of retrying in case of connection problems (as documented). It is
>>>>>>>>> possible that in some cases poll() throws and we need to restart the client
>>>>>>>>> in KafkaIO.
>>>>>>>>>
>>>>>>>>> [1]:
>>>>>>>>> https://github.com/apache/beam/blob/master/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaUnboundedReader.java#L594
>>>>>>>>>
>>>>>>>>> On Wed, Sep 12, 2018 at 9:59 AM Eduardo Soldera <
>>>>>>>>> eduardo.soldera@arquivei.com.br> wrote:
>>>>>>>>>
>>>>>>>>>> Hi Raghu, thanks for your help.
>>>>>>>>>> Just answering your previous question, the following logs were
>>>>>>>>>> the same as before the error, as if the pipeline were still getting the
>>>>>>>>>> messages, for example:
>>>>>>>>>>
>>>>>>>>>> (...)
>>>>>>>>>> Resetting offset for partition com.arquivei.dataeng.andre-0 to
>>>>>>>>>> offset 10.
>>>>>>>>>> Resetting offset for partition com.arquivei.dataeng.andre-0 to
>>>>>>>>>> offset 15.
>>>>>>>>>> ERROR
>>>>>>>>>> Resetting offset for partition com.arquivei.dataeng.andre-0 to
>>>>>>>>>> offset 22.
>>>>>>>>>> Resetting offset for partition com.arquivei.dataeng.andre-0 to
>>>>>>>>>> offset 30.
>>>>>>>>>> (...)
>>>>>>>>>>
>>>>>>>>>> But when checking the Kafka Consumer Group, the current offset
>>>>>>>>>> stays at 15, the commited offset from the last processed message, before
>>>>>>>>>> the error.
>>>>>>>>>>
>>>>>>>>>> We'll file a bug, but we could now reproduce the issue in a Dev
>>>>>>>>>> scenario.
>>>>>>>>>> We started the same pipeline using the direct runner, without
>>>>>>>>>> Google Dataflow. We blocked the Kafka Broker network and the same error was
>>>>>>>>>> thrown. Then we unblocked the network and the pipeline was able to
>>>>>>>>>> successfully process the subsequent messages.
>>>>>>>>>> When we started the same pipeline in the Dataflow runner and did
>>>>>>>>>> the same test, the same problem from our production scenario happened,
>>>>>>>>>> Dataflow couldn't process the new messages. Unfortunately, we've stopped
>>>>>>>>>> the dataflow job in production, but the problematic dev job is still
>>>>>>>>>> running and the log file of the VM is attached. Thank you very much.
>>>>>>>>>> Best regards
>>>>>>>>>>
>>>>>>>>>> Em ter, 11 de set de 2018 às 18:28, Raghu Angadi <
>>>>>>>>>> rangadi@google.com> escreveu:
>>>>>>>>>>
>>>>>>>>>>> Specifically, I am interested if you have any thread running
>>>>>>>>>>> 'consumerPollLoop()' [1]. There should always be one (if a worker is
>>>>>>>>>>> assigned one of the partitions). It is possible that KafkaClient itself is
>>>>>>>>>>> hasn't recovered from the group coordinator error (though unlikely).
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> https://github.com/apache/beam/blob/master/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaUnboundedReader.java#L570
>>>>>>>>>>>
>>>>>>>>>>> On Tue, Sep 11, 2018 at 12:31 PM Raghu Angadi <
>>>>>>>>>>> rangadi@google.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi Eduardo,
>>>>>>>>>>>>
>>>>>>>>>>>> In case of any error, the pipeline should keep on trying to
>>>>>>>>>>>> fetch. I don't know about this particular error. Do you see any others
>>>>>>>>>>>> afterwards in the log?
>>>>>>>>>>>> Couple of things you could try if the logs are not useful :
>>>>>>>>>>>>  - login to one of the VMs and get stacktrace of java worker
>>>>>>>>>>>> (look for a container called java-streaming)
>>>>>>>>>>>>  - file a support bug or stackoverflow question with jobid so
>>>>>>>>>>>> that Dataflow oncall can take a look.
>>>>>>>>>>>>
>>>>>>>>>>>> Raghu.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Tue, Sep 11, 2018 at 12:10 PM Eduardo Soldera <
>>>>>>>>>>>> eduardo.soldera@arquivei.com.br> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>> We have a Apache Beam pipeline running in Google Dataflow
>>>>>>>>>>>>> using KafkaIO. Suddenly the pipeline stop fetching Kafka messages at all,
>>>>>>>>>>>>> as our other workers from other pipelines continued to get Kafka messages.
>>>>>>>>>>>>>
>>>>>>>>>>>>> At the moment it stopped we got these messages:
>>>>>>>>>>>>>
>>>>>>>>>>>>> I  [Consumer clientId=consumer-1, groupId=genericPipe] Error sending fetch request (sessionId=1396189203, epoch=2431598) to node 3: org.apache.kafka.common.errors.DisconnectException.
>>>>>>>>>>>>> I  [Consumer clientId=consumer-1, groupId=genericPipe] Group coordinator 10.0.52.70:9093 (id: 2147483646 rack: null) is unavailable or invalid, will attempt rediscovery
>>>>>>>>>>>>> I  [Consumer clientId=consumer-1, groupId=genericPipe] Discovered group coordinator 10.0.52.70:9093 (id: 2147483646 rack: null)
>>>>>>>>>>>>>
>>>>>>>>>>>>> And then the pipeline stopped reading the messages.
>>>>>>>>>>>>>
>>>>>>>>>>>>> This is the KafkaIO setup  we have:
>>>>>>>>>>>>>
>>>>>>>>>>>>> KafkaIO.read[String,String]()
>>>>>>>>>>>>>   .withBootstrapServers(server)
>>>>>>>>>>>>>   .withTopic(topic)
>>>>>>>>>>>>>   .withKeyDeserializer(classOf[StringDeserializer])
>>>>>>>>>>>>>   .withValueDeserializer(classOf[StringDeserializer])
>>>>>>>>>>>>>   .updateConsumerProperties(properties)
>>>>>>>>>>>>>   .commitOffsetsInFinalize()
>>>>>>>>>>>>>   .withoutMetadata()
>>>>>>>>>>>>>
>>>>>>>>>>>>>  Any help will be much appreciated.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Best regards,
>>>>>>>>>>>>> --
>>>>>>>>>>>>> Eduardo Soldera Garcia
>>>>>>>>>>>>> Data Engineer
>>>>>>>>>>>>> (16) 3509-5555 | www.arquivei.com.br
>>>>>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>>>>>>> [image: Arquivei.com.br – Inteligência em Notas Fiscais]
>>>>>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>>>>>>> [image: Google seleciona Arquivei para imersão e mentoria no
>>>>>>>>>>>>> Vale do Silício]
>>>>>>>>>>>>> <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad>
>>>>>>>>>>>>> <https://www.facebook.com/arquivei>
>>>>>>>>>>>>> <https://www.linkedin.com/company/arquivei>
>>>>>>>>>>>>> <https://www.youtube.com/watch?v=sSUUKxbXnxk>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Eduardo Soldera Garcia
>>>>>>>>>> Data Engineer
>>>>>>>>>> (16) 3509-5555 | www.arquivei.com.br
>>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>>>> [image: Arquivei.com.br – Inteligência em Notas Fiscais]
>>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>>>> [image: Google seleciona Arquivei para imersão e mentoria no Vale
>>>>>>>>>> do Silício]
>>>>>>>>>> <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad>
>>>>>>>>>> <https://www.facebook.com/arquivei>
>>>>>>>>>> <https://www.linkedin.com/company/arquivei>
>>>>>>>>>> <https://www.youtube.com/watch?v=sSUUKxbXnxk>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Eduardo Soldera Garcia
>>>>>>>> Data Engineer
>>>>>>>> (16) 3509-5555 | www.arquivei.com.br
>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>> [image: Arquivei.com.br – Inteligência em Notas Fiscais]
>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>> [image: Google seleciona Arquivei para imersão e mentoria no Vale
>>>>>>>> do Silício]
>>>>>>>> <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad>
>>>>>>>> <https://www.facebook.com/arquivei>
>>>>>>>> <https://www.linkedin.com/company/arquivei>
>>>>>>>> <https://www.youtube.com/watch?v=sSUUKxbXnxk>
>>>>>>>>
>>>>>>>
>>>
>>> --
>>> Eduardo Soldera Garcia
>>> Data Engineer
>>> (16) 3509-5555 | www.arquivei.com.br
>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>> [image: Arquivei.com.br – Inteligência em Notas Fiscais]
>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>> [image: Google seleciona Arquivei para imersão e mentoria no Vale do
>>> Silício]
>>> <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad>
>>> <https://www.facebook.com/arquivei>
>>> <https://www.linkedin.com/company/arquivei>
>>> <https://www.youtube.com/watch?v=sSUUKxbXnxk>
>>>
>>
>
> --
> Eduardo Soldera Garcia
> Data Engineer
> (16) 3509-5555 | www.arquivei.com.br
> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
> [image: Arquivei.com.br – Inteligência em Notas Fiscais]
> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
> [image: Google seleciona Arquivei para imersão e mentoria no Vale do
> Silício]
> <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad>
> <https://www.facebook.com/arquivei>
> <https://www.linkedin.com/company/arquivei>
> <https://www.youtube.com/watch?v=sSUUKxbXnxk>
>


-- 
Eduardo Soldera Garcia
Data Engineer
(16) 3509-5555 | www.arquivei.com.br
<https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
[image: Arquivei.com.br – Inteligência em Notas Fiscais]
<https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
[image: Google seleciona Arquivei para imersão e mentoria no Vale do
Silício]
<https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad>
<https://www.facebook.com/arquivei>
<https://www.linkedin.com/company/arquivei>
<https://www.youtube.com/watch?v=sSUUKxbXnxk>

Re: Problem with KafkaIO

Posted by Eduardo Soldera <ed...@arquivei.com.br>.
Hi Raghu, yes, it is feasible, would you do that for us? I'm not sure how
we'd use the patch. We're using SBT and Spotify's Scio with Scala.

Thanks

Em sex, 14 de set de 2018 às 16:07, Raghu Angadi <ra...@google.com>
escreveu:

> Is is feasible for you to verify the fix in your dev job? I can make a
> patch against Beam 2.4 branch if you like.
>
> Raghu.
>
> On Fri, Sep 14, 2018 at 11:14 AM Eduardo Soldera <
> eduardo.soldera@arquivei.com.br> wrote:
>
>> Hi Raghu, thank you very much for the pull request.
>> We'll wait for the 2.7 Beam release.
>>
>> Regards!
>>
>> Em qui, 13 de set de 2018 às 18:19, Raghu Angadi <ra...@google.com>
>> escreveu:
>>
>>> Fix: https://github.com/apache/beam/pull/6391
>>>
>>> On Wed, Sep 12, 2018 at 3:30 PM Raghu Angadi <ra...@google.com> wrote:
>>>
>>>> Filed BEAM-5375 <https://issues.apache.org/jira/browse/BEAM-5375>. I
>>>> will fix it later this week.
>>>>
>>>> On Wed, Sep 12, 2018 at 12:16 PM Raghu Angadi <ra...@google.com>
>>>> wrote:
>>>>
>>>>>
>>>>>
>>>>> On Wed, Sep 12, 2018 at 12:11 PM Raghu Angadi <ra...@google.com>
>>>>> wrote:
>>>>>
>>>>>> Thanks for the job id, I looked at the worker logs (following usual
>>>>>> support oncall access protocol that provides temporary access to things
>>>>>> like logs in GCP):
>>>>>>
>>>>>> Root issue looks like consumerPollLoop() mentioned earlier needs to
>>>>>> handle unchecked exception. In your case it is clear that poll thread
>>>>>> exited with a runtime exception. The reader does not check for it and
>>>>>> continues to wait for poll thread to enqueue messages. A fix should result
>>>>>> in an IOException for read from the source. The runners will handle that
>>>>>> appropriately after that.  I will file a jira.
>>>>>>
>>>>>> https://github.com/apache/beam/blob/master/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaUnboundedReader.java#L678
>>>>>>
>>>>>
>>>>> Ignore the link.. was pasted here by mistake.
>>>>>
>>>>>
>>>>>>
>>>>>> From the logs (with a comment below each one):
>>>>>>
>>>>>>    - 2018-09-12 06:13:07.345 PDT Reader-0: reading from
>>>>>>    kafka_topic-0 starting at offset 2
>>>>>>       - Implies the reader is initialized and poll thread is
>>>>>>       started.
>>>>>>    - 2018-09-12 06:13:07.780 PDT Reader-0: first record offset 2
>>>>>>       - The reader actually got a message received by the poll
>>>>>>       thread from Kafka.
>>>>>>    - 2018-09-12 06:16:48.771 PDT Reader-0: exception while fetching
>>>>>>    latest offset for partition kafka_topic-0. will be retried.
>>>>>>       - This must have happened around the time when network was
>>>>>>       disrupted. This is from. Actual log is from another periodic task that
>>>>>>       fetches latest offsets for partitions.
>>>>>>
>>>>>> The poll thread must have died around the time network was disrupted.
>>>>>>
>>>>>> The following log comes from kafka client itself and is printed every
>>>>>> second when KafkaIO fetches latest offset. This log seems to be added in
>>>>>> recent versions. It is probably an unintentional log. I don't think there
>>>>>> is any better to fetch latest offsets than how KafkaIO does now. This is
>>>>>> logged inside consumer.position() called at [1].
>>>>>>
>>>>>>    - 2018-09-12 06:13:11.786 PDT [Consumer clientId=consumer-2,
>>>>>>    groupId=Reader-0_offset_consumer_1735388161_genericPipe] Resetting offset
>>>>>>    for partition com.arquivei.dataeng.andre-0 to offset 3.
>>>>>>
>>>>>> [1]:
>>>>>> https://github.com/apache/beam/blob/master/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaUnboundedReader.java#L678
>>>>>>
>>>>>
>>>>> This 'Resetting offset' is harmless, but is quite annoying to see in
>>>>> the worker logs. One way to avoid is to set kafka consumer's log level to
>>>>> WARNING. Ideally KafkaIO itself should do something to avoid it without
>>>>> user option.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>> On Wed, Sep 12, 2018 at 10:27 AM Eduardo Soldera <
>>>>>> eduardo.soldera@arquivei.com.br> wrote:
>>>>>>
>>>>>>> Hi Raghu! The job_id of our dev job is
>>>>>>> 2018-09-12_06_11_48-5600553605191377866.
>>>>>>>
>>>>>>> Thanks!
>>>>>>>
>>>>>>> Em qua, 12 de set de 2018 às 14:18, Raghu Angadi <ra...@google.com>
>>>>>>> escreveu:
>>>>>>>
>>>>>>>> Thanks for debugging.
>>>>>>>> Can you provide the job_id of your dev job? The stacktrace shows
>>>>>>>> that there is no thread running 'consumerPollLoop()' which can explain
>>>>>>>> stuck reader. You will likely find a logs at line 594 & 587 [1].  Dataflow
>>>>>>>> caches its readers and DirectRunner may not. That can explain DirectRunner
>>>>>>>> resume reads. The expectation in KafkaIO is that Kafka client library takes
>>>>>>>> care of retrying in case of connection problems (as documented). It is
>>>>>>>> possible that in some cases poll() throws and we need to restart the client
>>>>>>>> in KafkaIO.
>>>>>>>>
>>>>>>>> [1]:
>>>>>>>> https://github.com/apache/beam/blob/master/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaUnboundedReader.java#L594
>>>>>>>>
>>>>>>>> On Wed, Sep 12, 2018 at 9:59 AM Eduardo Soldera <
>>>>>>>> eduardo.soldera@arquivei.com.br> wrote:
>>>>>>>>
>>>>>>>>> Hi Raghu, thanks for your help.
>>>>>>>>> Just answering your previous question, the following logs were the
>>>>>>>>> same as before the error, as if the pipeline were still getting the
>>>>>>>>> messages, for example:
>>>>>>>>>
>>>>>>>>> (...)
>>>>>>>>> Resetting offset for partition com.arquivei.dataeng.andre-0 to
>>>>>>>>> offset 10.
>>>>>>>>> Resetting offset for partition com.arquivei.dataeng.andre-0 to
>>>>>>>>> offset 15.
>>>>>>>>> ERROR
>>>>>>>>> Resetting offset for partition com.arquivei.dataeng.andre-0 to
>>>>>>>>> offset 22.
>>>>>>>>> Resetting offset for partition com.arquivei.dataeng.andre-0 to
>>>>>>>>> offset 30.
>>>>>>>>> (...)
>>>>>>>>>
>>>>>>>>> But when checking the Kafka Consumer Group, the current offset
>>>>>>>>> stays at 15, the commited offset from the last processed message, before
>>>>>>>>> the error.
>>>>>>>>>
>>>>>>>>> We'll file a bug, but we could now reproduce the issue in a Dev
>>>>>>>>> scenario.
>>>>>>>>> We started the same pipeline using the direct runner, without
>>>>>>>>> Google Dataflow. We blocked the Kafka Broker network and the same error was
>>>>>>>>> thrown. Then we unblocked the network and the pipeline was able to
>>>>>>>>> successfully process the subsequent messages.
>>>>>>>>> When we started the same pipeline in the Dataflow runner and did
>>>>>>>>> the same test, the same problem from our production scenario happened,
>>>>>>>>> Dataflow couldn't process the new messages. Unfortunately, we've stopped
>>>>>>>>> the dataflow job in production, but the problematic dev job is still
>>>>>>>>> running and the log file of the VM is attached. Thank you very much.
>>>>>>>>> Best regards
>>>>>>>>>
>>>>>>>>> Em ter, 11 de set de 2018 às 18:28, Raghu Angadi <
>>>>>>>>> rangadi@google.com> escreveu:
>>>>>>>>>
>>>>>>>>>> Specifically, I am interested if you have any thread running
>>>>>>>>>> 'consumerPollLoop()' [1]. There should always be one (if a worker is
>>>>>>>>>> assigned one of the partitions). It is possible that KafkaClient itself is
>>>>>>>>>> hasn't recovered from the group coordinator error (though unlikely).
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> https://github.com/apache/beam/blob/master/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaUnboundedReader.java#L570
>>>>>>>>>>
>>>>>>>>>> On Tue, Sep 11, 2018 at 12:31 PM Raghu Angadi <ra...@google.com>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi Eduardo,
>>>>>>>>>>>
>>>>>>>>>>> In case of any error, the pipeline should keep on trying to
>>>>>>>>>>> fetch. I don't know about this particular error. Do you see any others
>>>>>>>>>>> afterwards in the log?
>>>>>>>>>>> Couple of things you could try if the logs are not useful :
>>>>>>>>>>>  - login to one of the VMs and get stacktrace of java worker
>>>>>>>>>>> (look for a container called java-streaming)
>>>>>>>>>>>  - file a support bug or stackoverflow question with jobid so
>>>>>>>>>>> that Dataflow oncall can take a look.
>>>>>>>>>>>
>>>>>>>>>>> Raghu.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Tue, Sep 11, 2018 at 12:10 PM Eduardo Soldera <
>>>>>>>>>>> eduardo.soldera@arquivei.com.br> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi,
>>>>>>>>>>>> We have a Apache Beam pipeline running in Google Dataflow using
>>>>>>>>>>>> KafkaIO. Suddenly the pipeline stop fetching Kafka messages at all, as our
>>>>>>>>>>>> other workers from other pipelines continued to get Kafka messages.
>>>>>>>>>>>>
>>>>>>>>>>>> At the moment it stopped we got these messages:
>>>>>>>>>>>>
>>>>>>>>>>>> I  [Consumer clientId=consumer-1, groupId=genericPipe] Error sending fetch request (sessionId=1396189203, epoch=2431598) to node 3: org.apache.kafka.common.errors.DisconnectException.
>>>>>>>>>>>> I  [Consumer clientId=consumer-1, groupId=genericPipe] Group coordinator 10.0.52.70:9093 (id: 2147483646 rack: null) is unavailable or invalid, will attempt rediscovery
>>>>>>>>>>>> I  [Consumer clientId=consumer-1, groupId=genericPipe] Discovered group coordinator 10.0.52.70:9093 (id: 2147483646 rack: null)
>>>>>>>>>>>>
>>>>>>>>>>>> And then the pipeline stopped reading the messages.
>>>>>>>>>>>>
>>>>>>>>>>>> This is the KafkaIO setup  we have:
>>>>>>>>>>>>
>>>>>>>>>>>> KafkaIO.read[String,String]()
>>>>>>>>>>>>   .withBootstrapServers(server)
>>>>>>>>>>>>   .withTopic(topic)
>>>>>>>>>>>>   .withKeyDeserializer(classOf[StringDeserializer])
>>>>>>>>>>>>   .withValueDeserializer(classOf[StringDeserializer])
>>>>>>>>>>>>   .updateConsumerProperties(properties)
>>>>>>>>>>>>   .commitOffsetsInFinalize()
>>>>>>>>>>>>   .withoutMetadata()
>>>>>>>>>>>>
>>>>>>>>>>>>  Any help will be much appreciated.
>>>>>>>>>>>>
>>>>>>>>>>>> Best regards,
>>>>>>>>>>>> --
>>>>>>>>>>>> Eduardo Soldera Garcia
>>>>>>>>>>>> Data Engineer
>>>>>>>>>>>> (16) 3509-5555 | www.arquivei.com.br
>>>>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>>>>>> [image: Arquivei.com.br – Inteligência em Notas Fiscais]
>>>>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>>>>>> [image: Google seleciona Arquivei para imersão e mentoria no
>>>>>>>>>>>> Vale do Silício]
>>>>>>>>>>>> <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad>
>>>>>>>>>>>> <https://www.facebook.com/arquivei>
>>>>>>>>>>>> <https://www.linkedin.com/company/arquivei>
>>>>>>>>>>>> <https://www.youtube.com/watch?v=sSUUKxbXnxk>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Eduardo Soldera Garcia
>>>>>>>>> Data Engineer
>>>>>>>>> (16) 3509-5555 | www.arquivei.com.br
>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>>> [image: Arquivei.com.br – Inteligência em Notas Fiscais]
>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>>> [image: Google seleciona Arquivei para imersão e mentoria no Vale
>>>>>>>>> do Silício]
>>>>>>>>> <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad>
>>>>>>>>> <https://www.facebook.com/arquivei>
>>>>>>>>> <https://www.linkedin.com/company/arquivei>
>>>>>>>>> <https://www.youtube.com/watch?v=sSUUKxbXnxk>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Eduardo Soldera Garcia
>>>>>>> Data Engineer
>>>>>>> (16) 3509-5555 | www.arquivei.com.br
>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>> [image: Arquivei.com.br – Inteligência em Notas Fiscais]
>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>> [image: Google seleciona Arquivei para imersão e mentoria no Vale do
>>>>>>> Silício]
>>>>>>> <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad>
>>>>>>> <https://www.facebook.com/arquivei>
>>>>>>> <https://www.linkedin.com/company/arquivei>
>>>>>>> <https://www.youtube.com/watch?v=sSUUKxbXnxk>
>>>>>>>
>>>>>>
>>
>> --
>> Eduardo Soldera Garcia
>> Data Engineer
>> (16) 3509-5555 | www.arquivei.com.br
>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>> [image: Arquivei.com.br – Inteligência em Notas Fiscais]
>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>> [image: Google seleciona Arquivei para imersão e mentoria no Vale do
>> Silício]
>> <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad>
>> <https://www.facebook.com/arquivei>
>> <https://www.linkedin.com/company/arquivei>
>> <https://www.youtube.com/watch?v=sSUUKxbXnxk>
>>
>

-- 
Eduardo Soldera Garcia
Data Engineer
(16) 3509-5555 | www.arquivei.com.br
<https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
[image: Arquivei.com.br – Inteligência em Notas Fiscais]
<https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
[image: Google seleciona Arquivei para imersão e mentoria no Vale do
Silício]
<https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad>
<https://www.facebook.com/arquivei>
<https://www.linkedin.com/company/arquivei>
<https://www.youtube.com/watch?v=sSUUKxbXnxk>

Re: Problem with KafkaIO

Posted by Raghu Angadi <ra...@google.com>.
Is is feasible for you to verify the fix in your dev job? I can make a
patch against Beam 2.4 branch if you like.

Raghu.

On Fri, Sep 14, 2018 at 11:14 AM Eduardo Soldera <
eduardo.soldera@arquivei.com.br> wrote:

> Hi Raghu, thank you very much for the pull request.
> We'll wait for the 2.7 Beam release.
>
> Regards!
>
> Em qui, 13 de set de 2018 às 18:19, Raghu Angadi <ra...@google.com>
> escreveu:
>
>> Fix: https://github.com/apache/beam/pull/6391
>>
>> On Wed, Sep 12, 2018 at 3:30 PM Raghu Angadi <ra...@google.com> wrote:
>>
>>> Filed BEAM-5375 <https://issues.apache.org/jira/browse/BEAM-5375>. I
>>> will fix it later this week.
>>>
>>> On Wed, Sep 12, 2018 at 12:16 PM Raghu Angadi <ra...@google.com>
>>> wrote:
>>>
>>>>
>>>>
>>>> On Wed, Sep 12, 2018 at 12:11 PM Raghu Angadi <ra...@google.com>
>>>> wrote:
>>>>
>>>>> Thanks for the job id, I looked at the worker logs (following usual
>>>>> support oncall access protocol that provides temporary access to things
>>>>> like logs in GCP):
>>>>>
>>>>> Root issue looks like consumerPollLoop() mentioned earlier needs to
>>>>> handle unchecked exception. In your case it is clear that poll thread
>>>>> exited with a runtime exception. The reader does not check for it and
>>>>> continues to wait for poll thread to enqueue messages. A fix should result
>>>>> in an IOException for read from the source. The runners will handle that
>>>>> appropriately after that.  I will file a jira.
>>>>>
>>>>> https://github.com/apache/beam/blob/master/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaUnboundedReader.java#L678
>>>>>
>>>>
>>>> Ignore the link.. was pasted here by mistake.
>>>>
>>>>
>>>>>
>>>>> From the logs (with a comment below each one):
>>>>>
>>>>>    - 2018-09-12 06:13:07.345 PDT Reader-0: reading from kafka_topic-0
>>>>>    starting at offset 2
>>>>>       - Implies the reader is initialized and poll thread is started.
>>>>>    - 2018-09-12 06:13:07.780 PDT Reader-0: first record offset 2
>>>>>       - The reader actually got a message received by the poll thread
>>>>>       from Kafka.
>>>>>    - 2018-09-12 06:16:48.771 PDT Reader-0: exception while fetching
>>>>>    latest offset for partition kafka_topic-0. will be retried.
>>>>>       - This must have happened around the time when network was
>>>>>       disrupted. This is from. Actual log is from another periodic task that
>>>>>       fetches latest offsets for partitions.
>>>>>
>>>>> The poll thread must have died around the time network was disrupted.
>>>>>
>>>>> The following log comes from kafka client itself and is printed every
>>>>> second when KafkaIO fetches latest offset. This log seems to be added in
>>>>> recent versions. It is probably an unintentional log. I don't think there
>>>>> is any better to fetch latest offsets than how KafkaIO does now. This is
>>>>> logged inside consumer.position() called at [1].
>>>>>
>>>>>    - 2018-09-12 06:13:11.786 PDT [Consumer clientId=consumer-2,
>>>>>    groupId=Reader-0_offset_consumer_1735388161_genericPipe] Resetting offset
>>>>>    for partition com.arquivei.dataeng.andre-0 to offset 3.
>>>>>
>>>>> [1]:
>>>>> https://github.com/apache/beam/blob/master/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaUnboundedReader.java#L678
>>>>>
>>>>
>>>> This 'Resetting offset' is harmless, but is quite annoying to see in
>>>> the worker logs. One way to avoid is to set kafka consumer's log level to
>>>> WARNING. Ideally KafkaIO itself should do something to avoid it without
>>>> user option.
>>>>
>>>>
>>>>
>>>>
>>>>> On Wed, Sep 12, 2018 at 10:27 AM Eduardo Soldera <
>>>>> eduardo.soldera@arquivei.com.br> wrote:
>>>>>
>>>>>> Hi Raghu! The job_id of our dev job is
>>>>>> 2018-09-12_06_11_48-5600553605191377866.
>>>>>>
>>>>>> Thanks!
>>>>>>
>>>>>> Em qua, 12 de set de 2018 às 14:18, Raghu Angadi <ra...@google.com>
>>>>>> escreveu:
>>>>>>
>>>>>>> Thanks for debugging.
>>>>>>> Can you provide the job_id of your dev job? The stacktrace shows
>>>>>>> that there is no thread running 'consumerPollLoop()' which can explain
>>>>>>> stuck reader. You will likely find a logs at line 594 & 587 [1].  Dataflow
>>>>>>> caches its readers and DirectRunner may not. That can explain DirectRunner
>>>>>>> resume reads. The expectation in KafkaIO is that Kafka client library takes
>>>>>>> care of retrying in case of connection problems (as documented). It is
>>>>>>> possible that in some cases poll() throws and we need to restart the client
>>>>>>> in KafkaIO.
>>>>>>>
>>>>>>> [1]:
>>>>>>> https://github.com/apache/beam/blob/master/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaUnboundedReader.java#L594
>>>>>>>
>>>>>>> On Wed, Sep 12, 2018 at 9:59 AM Eduardo Soldera <
>>>>>>> eduardo.soldera@arquivei.com.br> wrote:
>>>>>>>
>>>>>>>> Hi Raghu, thanks for your help.
>>>>>>>> Just answering your previous question, the following logs were the
>>>>>>>> same as before the error, as if the pipeline were still getting the
>>>>>>>> messages, for example:
>>>>>>>>
>>>>>>>> (...)
>>>>>>>> Resetting offset for partition com.arquivei.dataeng.andre-0 to
>>>>>>>> offset 10.
>>>>>>>> Resetting offset for partition com.arquivei.dataeng.andre-0 to
>>>>>>>> offset 15.
>>>>>>>> ERROR
>>>>>>>> Resetting offset for partition com.arquivei.dataeng.andre-0 to
>>>>>>>> offset 22.
>>>>>>>> Resetting offset for partition com.arquivei.dataeng.andre-0 to
>>>>>>>> offset 30.
>>>>>>>> (...)
>>>>>>>>
>>>>>>>> But when checking the Kafka Consumer Group, the current offset
>>>>>>>> stays at 15, the commited offset from the last processed message, before
>>>>>>>> the error.
>>>>>>>>
>>>>>>>> We'll file a bug, but we could now reproduce the issue in a Dev
>>>>>>>> scenario.
>>>>>>>> We started the same pipeline using the direct runner, without
>>>>>>>> Google Dataflow. We blocked the Kafka Broker network and the same error was
>>>>>>>> thrown. Then we unblocked the network and the pipeline was able to
>>>>>>>> successfully process the subsequent messages.
>>>>>>>> When we started the same pipeline in the Dataflow runner and did
>>>>>>>> the same test, the same problem from our production scenario happened,
>>>>>>>> Dataflow couldn't process the new messages. Unfortunately, we've stopped
>>>>>>>> the dataflow job in production, but the problematic dev job is still
>>>>>>>> running and the log file of the VM is attached. Thank you very much.
>>>>>>>> Best regards
>>>>>>>>
>>>>>>>> Em ter, 11 de set de 2018 às 18:28, Raghu Angadi <
>>>>>>>> rangadi@google.com> escreveu:
>>>>>>>>
>>>>>>>>> Specifically, I am interested if you have any thread running
>>>>>>>>> 'consumerPollLoop()' [1]. There should always be one (if a worker is
>>>>>>>>> assigned one of the partitions). It is possible that KafkaClient itself is
>>>>>>>>> hasn't recovered from the group coordinator error (though unlikely).
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> https://github.com/apache/beam/blob/master/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaUnboundedReader.java#L570
>>>>>>>>>
>>>>>>>>> On Tue, Sep 11, 2018 at 12:31 PM Raghu Angadi <ra...@google.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Hi Eduardo,
>>>>>>>>>>
>>>>>>>>>> In case of any error, the pipeline should keep on trying to
>>>>>>>>>> fetch. I don't know about this particular error. Do you see any others
>>>>>>>>>> afterwards in the log?
>>>>>>>>>> Couple of things you could try if the logs are not useful :
>>>>>>>>>>  - login to one of the VMs and get stacktrace of java worker
>>>>>>>>>> (look for a container called java-streaming)
>>>>>>>>>>  - file a support bug or stackoverflow question with jobid so
>>>>>>>>>> that Dataflow oncall can take a look.
>>>>>>>>>>
>>>>>>>>>> Raghu.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Tue, Sep 11, 2018 at 12:10 PM Eduardo Soldera <
>>>>>>>>>> eduardo.soldera@arquivei.com.br> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi,
>>>>>>>>>>> We have a Apache Beam pipeline running in Google Dataflow using
>>>>>>>>>>> KafkaIO. Suddenly the pipeline stop fetching Kafka messages at all, as our
>>>>>>>>>>> other workers from other pipelines continued to get Kafka messages.
>>>>>>>>>>>
>>>>>>>>>>> At the moment it stopped we got these messages:
>>>>>>>>>>>
>>>>>>>>>>> I  [Consumer clientId=consumer-1, groupId=genericPipe] Error sending fetch request (sessionId=1396189203, epoch=2431598) to node 3: org.apache.kafka.common.errors.DisconnectException.
>>>>>>>>>>> I  [Consumer clientId=consumer-1, groupId=genericPipe] Group coordinator 10.0.52.70:9093 (id: 2147483646 rack: null) is unavailable or invalid, will attempt rediscovery
>>>>>>>>>>> I  [Consumer clientId=consumer-1, groupId=genericPipe] Discovered group coordinator 10.0.52.70:9093 (id: 2147483646 rack: null)
>>>>>>>>>>>
>>>>>>>>>>> And then the pipeline stopped reading the messages.
>>>>>>>>>>>
>>>>>>>>>>> This is the KafkaIO setup  we have:
>>>>>>>>>>>
>>>>>>>>>>> KafkaIO.read[String,String]()
>>>>>>>>>>>   .withBootstrapServers(server)
>>>>>>>>>>>   .withTopic(topic)
>>>>>>>>>>>   .withKeyDeserializer(classOf[StringDeserializer])
>>>>>>>>>>>   .withValueDeserializer(classOf[StringDeserializer])
>>>>>>>>>>>   .updateConsumerProperties(properties)
>>>>>>>>>>>   .commitOffsetsInFinalize()
>>>>>>>>>>>   .withoutMetadata()
>>>>>>>>>>>
>>>>>>>>>>>  Any help will be much appreciated.
>>>>>>>>>>>
>>>>>>>>>>> Best regards,
>>>>>>>>>>> --
>>>>>>>>>>> Eduardo Soldera Garcia
>>>>>>>>>>> Data Engineer
>>>>>>>>>>> (16) 3509-5555 | www.arquivei.com.br
>>>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>>>>> [image: Arquivei.com.br – Inteligência em Notas Fiscais]
>>>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>>>>> [image: Google seleciona Arquivei para imersão e mentoria no
>>>>>>>>>>> Vale do Silício]
>>>>>>>>>>> <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad>
>>>>>>>>>>> <https://www.facebook.com/arquivei>
>>>>>>>>>>> <https://www.linkedin.com/company/arquivei>
>>>>>>>>>>> <https://www.youtube.com/watch?v=sSUUKxbXnxk>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Eduardo Soldera Garcia
>>>>>>>> Data Engineer
>>>>>>>> (16) 3509-5555 | www.arquivei.com.br
>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>> [image: Arquivei.com.br – Inteligência em Notas Fiscais]
>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>> [image: Google seleciona Arquivei para imersão e mentoria no Vale
>>>>>>>> do Silício]
>>>>>>>> <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad>
>>>>>>>> <https://www.facebook.com/arquivei>
>>>>>>>> <https://www.linkedin.com/company/arquivei>
>>>>>>>> <https://www.youtube.com/watch?v=sSUUKxbXnxk>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>> --
>>>>>> Eduardo Soldera Garcia
>>>>>> Data Engineer
>>>>>> (16) 3509-5555 | www.arquivei.com.br
>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>> [image: Arquivei.com.br – Inteligência em Notas Fiscais]
>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>> [image: Google seleciona Arquivei para imersão e mentoria no Vale do
>>>>>> Silício]
>>>>>> <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad>
>>>>>> <https://www.facebook.com/arquivei>
>>>>>> <https://www.linkedin.com/company/arquivei>
>>>>>> <https://www.youtube.com/watch?v=sSUUKxbXnxk>
>>>>>>
>>>>>
>
> --
> Eduardo Soldera Garcia
> Data Engineer
> (16) 3509-5555 | www.arquivei.com.br
> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
> [image: Arquivei.com.br – Inteligência em Notas Fiscais]
> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
> [image: Google seleciona Arquivei para imersão e mentoria no Vale do
> Silício]
> <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad>
> <https://www.facebook.com/arquivei>
> <https://www.linkedin.com/company/arquivei>
> <https://www.youtube.com/watch?v=sSUUKxbXnxk>
>

Re: Problem with KafkaIO

Posted by Eduardo Soldera <ed...@arquivei.com.br>.
Hi Raghu, thank you very much for the pull request.
We'll wait for the 2.7 Beam release.

Regards!

Em qui, 13 de set de 2018 às 18:19, Raghu Angadi <ra...@google.com>
escreveu:

> Fix: https://github.com/apache/beam/pull/6391
>
> On Wed, Sep 12, 2018 at 3:30 PM Raghu Angadi <ra...@google.com> wrote:
>
>> Filed BEAM-5375 <https://issues.apache.org/jira/browse/BEAM-5375>. I
>> will fix it later this week.
>>
>> On Wed, Sep 12, 2018 at 12:16 PM Raghu Angadi <ra...@google.com> wrote:
>>
>>>
>>>
>>> On Wed, Sep 12, 2018 at 12:11 PM Raghu Angadi <ra...@google.com>
>>> wrote:
>>>
>>>> Thanks for the job id, I looked at the worker logs (following usual
>>>> support oncall access protocol that provides temporary access to things
>>>> like logs in GCP):
>>>>
>>>> Root issue looks like consumerPollLoop() mentioned earlier needs to
>>>> handle unchecked exception. In your case it is clear that poll thread
>>>> exited with a runtime exception. The reader does not check for it and
>>>> continues to wait for poll thread to enqueue messages. A fix should result
>>>> in an IOException for read from the source. The runners will handle that
>>>> appropriately after that.  I will file a jira.
>>>>
>>>> https://github.com/apache/beam/blob/master/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaUnboundedReader.java#L678
>>>>
>>>
>>> Ignore the link.. was pasted here by mistake.
>>>
>>>
>>>>
>>>> From the logs (with a comment below each one):
>>>>
>>>>    - 2018-09-12 06:13:07.345 PDT Reader-0: reading from kafka_topic-0
>>>>    starting at offset 2
>>>>       - Implies the reader is initialized and poll thread is started.
>>>>    - 2018-09-12 06:13:07.780 PDT Reader-0: first record offset 2
>>>>       - The reader actually got a message received by the poll thread
>>>>       from Kafka.
>>>>    - 2018-09-12 06:16:48.771 PDT Reader-0: exception while fetching
>>>>    latest offset for partition kafka_topic-0. will be retried.
>>>>       - This must have happened around the time when network was
>>>>       disrupted. This is from. Actual log is from another periodic task that
>>>>       fetches latest offsets for partitions.
>>>>
>>>> The poll thread must have died around the time network was disrupted.
>>>>
>>>> The following log comes from kafka client itself and is printed every
>>>> second when KafkaIO fetches latest offset. This log seems to be added in
>>>> recent versions. It is probably an unintentional log. I don't think there
>>>> is any better to fetch latest offsets than how KafkaIO does now. This is
>>>> logged inside consumer.position() called at [1].
>>>>
>>>>    - 2018-09-12 06:13:11.786 PDT [Consumer clientId=consumer-2,
>>>>    groupId=Reader-0_offset_consumer_1735388161_genericPipe] Resetting offset
>>>>    for partition com.arquivei.dataeng.andre-0 to offset 3.
>>>>
>>>> [1]:
>>>> https://github.com/apache/beam/blob/master/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaUnboundedReader.java#L678
>>>>
>>>
>>> This 'Resetting offset' is harmless, but is quite annoying to see in the
>>> worker logs. One way to avoid is to set kafka consumer's log level to
>>> WARNING. Ideally KafkaIO itself should do something to avoid it without
>>> user option.
>>>
>>>
>>>
>>>
>>>> On Wed, Sep 12, 2018 at 10:27 AM Eduardo Soldera <
>>>> eduardo.soldera@arquivei.com.br> wrote:
>>>>
>>>>> Hi Raghu! The job_id of our dev job is
>>>>> 2018-09-12_06_11_48-5600553605191377866.
>>>>>
>>>>> Thanks!
>>>>>
>>>>> Em qua, 12 de set de 2018 às 14:18, Raghu Angadi <ra...@google.com>
>>>>> escreveu:
>>>>>
>>>>>> Thanks for debugging.
>>>>>> Can you provide the job_id of your dev job? The stacktrace shows that
>>>>>> there is no thread running 'consumerPollLoop()' which can explain stuck
>>>>>> reader. You will likely find a logs at line 594 & 587 [1].  Dataflow caches
>>>>>> its readers and DirectRunner may not. That can explain DirectRunner resume
>>>>>> reads. The expectation in KafkaIO is that Kafka client library takes care
>>>>>> of retrying in case of connection problems (as documented). It is possible
>>>>>> that in some cases poll() throws and we need to restart the client in
>>>>>> KafkaIO.
>>>>>>
>>>>>> [1]:
>>>>>> https://github.com/apache/beam/blob/master/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaUnboundedReader.java#L594
>>>>>>
>>>>>> On Wed, Sep 12, 2018 at 9:59 AM Eduardo Soldera <
>>>>>> eduardo.soldera@arquivei.com.br> wrote:
>>>>>>
>>>>>>> Hi Raghu, thanks for your help.
>>>>>>> Just answering your previous question, the following logs were the
>>>>>>> same as before the error, as if the pipeline were still getting the
>>>>>>> messages, for example:
>>>>>>>
>>>>>>> (...)
>>>>>>> Resetting offset for partition com.arquivei.dataeng.andre-0 to
>>>>>>> offset 10.
>>>>>>> Resetting offset for partition com.arquivei.dataeng.andre-0 to
>>>>>>> offset 15.
>>>>>>> ERROR
>>>>>>> Resetting offset for partition com.arquivei.dataeng.andre-0 to
>>>>>>> offset 22.
>>>>>>> Resetting offset for partition com.arquivei.dataeng.andre-0 to
>>>>>>> offset 30.
>>>>>>> (...)
>>>>>>>
>>>>>>> But when checking the Kafka Consumer Group, the current offset stays
>>>>>>> at 15, the commited offset from the last processed message, before the
>>>>>>> error.
>>>>>>>
>>>>>>> We'll file a bug, but we could now reproduce the issue in a Dev
>>>>>>> scenario.
>>>>>>> We started the same pipeline using the direct runner, without Google
>>>>>>> Dataflow. We blocked the Kafka Broker network and the same error was
>>>>>>> thrown. Then we unblocked the network and the pipeline was able to
>>>>>>> successfully process the subsequent messages.
>>>>>>> When we started the same pipeline in the Dataflow runner and did the
>>>>>>> same test, the same problem from our production scenario happened, Dataflow
>>>>>>> couldn't process the new messages. Unfortunately, we've stopped the
>>>>>>> dataflow job in production, but the problematic dev job is still running
>>>>>>> and the log file of the VM is attached. Thank you very much.
>>>>>>> Best regards
>>>>>>>
>>>>>>> Em ter, 11 de set de 2018 às 18:28, Raghu Angadi <ra...@google.com>
>>>>>>> escreveu:
>>>>>>>
>>>>>>>> Specifically, I am interested if you have any thread running
>>>>>>>> 'consumerPollLoop()' [1]. There should always be one (if a worker is
>>>>>>>> assigned one of the partitions). It is possible that KafkaClient itself is
>>>>>>>> hasn't recovered from the group coordinator error (though unlikely).
>>>>>>>>
>>>>>>>>
>>>>>>>> https://github.com/apache/beam/blob/master/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaUnboundedReader.java#L570
>>>>>>>>
>>>>>>>> On Tue, Sep 11, 2018 at 12:31 PM Raghu Angadi <ra...@google.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Hi Eduardo,
>>>>>>>>>
>>>>>>>>> In case of any error, the pipeline should keep on trying to fetch.
>>>>>>>>> I don't know about this particular error. Do you see any others afterwards
>>>>>>>>> in the log?
>>>>>>>>> Couple of things you could try if the logs are not useful :
>>>>>>>>>  - login to one of the VMs and get stacktrace of java worker (look
>>>>>>>>> for a container called java-streaming)
>>>>>>>>>  - file a support bug or stackoverflow question with jobid so that
>>>>>>>>> Dataflow oncall can take a look.
>>>>>>>>>
>>>>>>>>> Raghu.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Tue, Sep 11, 2018 at 12:10 PM Eduardo Soldera <
>>>>>>>>> eduardo.soldera@arquivei.com.br> wrote:
>>>>>>>>>
>>>>>>>>>> Hi,
>>>>>>>>>> We have a Apache Beam pipeline running in Google Dataflow using
>>>>>>>>>> KafkaIO. Suddenly the pipeline stop fetching Kafka messages at all, as our
>>>>>>>>>> other workers from other pipelines continued to get Kafka messages.
>>>>>>>>>>
>>>>>>>>>> At the moment it stopped we got these messages:
>>>>>>>>>>
>>>>>>>>>> I  [Consumer clientId=consumer-1, groupId=genericPipe] Error sending fetch request (sessionId=1396189203, epoch=2431598) to node 3: org.apache.kafka.common.errors.DisconnectException.
>>>>>>>>>> I  [Consumer clientId=consumer-1, groupId=genericPipe] Group coordinator 10.0.52.70:9093 (id: 2147483646 rack: null) is unavailable or invalid, will attempt rediscovery
>>>>>>>>>> I  [Consumer clientId=consumer-1, groupId=genericPipe] Discovered group coordinator 10.0.52.70:9093 (id: 2147483646 rack: null)
>>>>>>>>>>
>>>>>>>>>> And then the pipeline stopped reading the messages.
>>>>>>>>>>
>>>>>>>>>> This is the KafkaIO setup  we have:
>>>>>>>>>>
>>>>>>>>>> KafkaIO.read[String,String]()
>>>>>>>>>>   .withBootstrapServers(server)
>>>>>>>>>>   .withTopic(topic)
>>>>>>>>>>   .withKeyDeserializer(classOf[StringDeserializer])
>>>>>>>>>>   .withValueDeserializer(classOf[StringDeserializer])
>>>>>>>>>>   .updateConsumerProperties(properties)
>>>>>>>>>>   .commitOffsetsInFinalize()
>>>>>>>>>>   .withoutMetadata()
>>>>>>>>>>
>>>>>>>>>>  Any help will be much appreciated.
>>>>>>>>>>
>>>>>>>>>> Best regards,
>>>>>>>>>> --
>>>>>>>>>> Eduardo Soldera Garcia
>>>>>>>>>> Data Engineer
>>>>>>>>>> (16) 3509-5555 | www.arquivei.com.br
>>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>>>> [image: Arquivei.com.br – Inteligência em Notas Fiscais]
>>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>>>> [image: Google seleciona Arquivei para imersão e mentoria no Vale
>>>>>>>>>> do Silício]
>>>>>>>>>> <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad>
>>>>>>>>>> <https://www.facebook.com/arquivei>
>>>>>>>>>> <https://www.linkedin.com/company/arquivei>
>>>>>>>>>> <https://www.youtube.com/watch?v=sSUUKxbXnxk>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Eduardo Soldera Garcia
>>>>>>> Data Engineer
>>>>>>> (16) 3509-5555 | www.arquivei.com.br
>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>> [image: Arquivei.com.br – Inteligência em Notas Fiscais]
>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>> [image: Google seleciona Arquivei para imersão e mentoria no Vale do
>>>>>>> Silício]
>>>>>>> <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad>
>>>>>>> <https://www.facebook.com/arquivei>
>>>>>>> <https://www.linkedin.com/company/arquivei>
>>>>>>> <https://www.youtube.com/watch?v=sSUUKxbXnxk>
>>>>>>>
>>>>>>
>>>>>
>>>>> --
>>>>> Eduardo Soldera Garcia
>>>>> Data Engineer
>>>>> (16) 3509-5555 | www.arquivei.com.br
>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>> [image: Arquivei.com.br – Inteligência em Notas Fiscais]
>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>> [image: Google seleciona Arquivei para imersão e mentoria no Vale do
>>>>> Silício]
>>>>> <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad>
>>>>> <https://www.facebook.com/arquivei>
>>>>> <https://www.linkedin.com/company/arquivei>
>>>>> <https://www.youtube.com/watch?v=sSUUKxbXnxk>
>>>>>
>>>>

-- 
Eduardo Soldera Garcia
Data Engineer
(16) 3509-5555 | www.arquivei.com.br
<https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
[image: Arquivei.com.br – Inteligência em Notas Fiscais]
<https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
[image: Google seleciona Arquivei para imersão e mentoria no Vale do
Silício]
<https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad>
<https://www.facebook.com/arquivei>
<https://www.linkedin.com/company/arquivei>
<https://www.youtube.com/watch?v=sSUUKxbXnxk>

Re: Problem with KafkaIO

Posted by Raghu Angadi <ra...@google.com>.
Fix: https://github.com/apache/beam/pull/6391

On Wed, Sep 12, 2018 at 3:30 PM Raghu Angadi <ra...@google.com> wrote:

> Filed BEAM-5375 <https://issues.apache.org/jira/browse/BEAM-5375>. I will
> fix it later this week.
>
> On Wed, Sep 12, 2018 at 12:16 PM Raghu Angadi <ra...@google.com> wrote:
>
>>
>>
>> On Wed, Sep 12, 2018 at 12:11 PM Raghu Angadi <ra...@google.com> wrote:
>>
>>> Thanks for the job id, I looked at the worker logs (following usual
>>> support oncall access protocol that provides temporary access to things
>>> like logs in GCP):
>>>
>>> Root issue looks like consumerPollLoop() mentioned earlier needs to
>>> handle unchecked exception. In your case it is clear that poll thread
>>> exited with a runtime exception. The reader does not check for it and
>>> continues to wait for poll thread to enqueue messages. A fix should result
>>> in an IOException for read from the source. The runners will handle that
>>> appropriately after that.  I will file a jira.
>>>
>>> https://github.com/apache/beam/blob/master/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaUnboundedReader.java#L678
>>>
>>
>> Ignore the link.. was pasted here by mistake.
>>
>>
>>>
>>> From the logs (with a comment below each one):
>>>
>>>    - 2018-09-12 06:13:07.345 PDT Reader-0: reading from kafka_topic-0
>>>    starting at offset 2
>>>       - Implies the reader is initialized and poll thread is started.
>>>    - 2018-09-12 06:13:07.780 PDT Reader-0: first record offset 2
>>>       - The reader actually got a message received by the poll thread
>>>       from Kafka.
>>>    - 2018-09-12 06:16:48.771 PDT Reader-0: exception while fetching
>>>    latest offset for partition kafka_topic-0. will be retried.
>>>       - This must have happened around the time when network was
>>>       disrupted. This is from. Actual log is from another periodic task that
>>>       fetches latest offsets for partitions.
>>>
>>> The poll thread must have died around the time network was disrupted.
>>>
>>> The following log comes from kafka client itself and is printed every
>>> second when KafkaIO fetches latest offset. This log seems to be added in
>>> recent versions. It is probably an unintentional log. I don't think there
>>> is any better to fetch latest offsets than how KafkaIO does now. This is
>>> logged inside consumer.position() called at [1].
>>>
>>>    - 2018-09-12 06:13:11.786 PDT [Consumer clientId=consumer-2,
>>>    groupId=Reader-0_offset_consumer_1735388161_genericPipe] Resetting offset
>>>    for partition com.arquivei.dataeng.andre-0 to offset 3.
>>>
>>> [1]:
>>> https://github.com/apache/beam/blob/master/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaUnboundedReader.java#L678
>>>
>>
>> This 'Resetting offset' is harmless, but is quite annoying to see in the
>> worker logs. One way to avoid is to set kafka consumer's log level to
>> WARNING. Ideally KafkaIO itself should do something to avoid it without
>> user option.
>>
>>
>>
>>
>>> On Wed, Sep 12, 2018 at 10:27 AM Eduardo Soldera <
>>> eduardo.soldera@arquivei.com.br> wrote:
>>>
>>>> Hi Raghu! The job_id of our dev job is
>>>> 2018-09-12_06_11_48-5600553605191377866.
>>>>
>>>> Thanks!
>>>>
>>>> Em qua, 12 de set de 2018 às 14:18, Raghu Angadi <ra...@google.com>
>>>> escreveu:
>>>>
>>>>> Thanks for debugging.
>>>>> Can you provide the job_id of your dev job? The stacktrace shows that
>>>>> there is no thread running 'consumerPollLoop()' which can explain stuck
>>>>> reader. You will likely find a logs at line 594 & 587 [1].  Dataflow caches
>>>>> its readers and DirectRunner may not. That can explain DirectRunner resume
>>>>> reads. The expectation in KafkaIO is that Kafka client library takes care
>>>>> of retrying in case of connection problems (as documented). It is possible
>>>>> that in some cases poll() throws and we need to restart the client in
>>>>> KafkaIO.
>>>>>
>>>>> [1]:
>>>>> https://github.com/apache/beam/blob/master/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaUnboundedReader.java#L594
>>>>>
>>>>> On Wed, Sep 12, 2018 at 9:59 AM Eduardo Soldera <
>>>>> eduardo.soldera@arquivei.com.br> wrote:
>>>>>
>>>>>> Hi Raghu, thanks for your help.
>>>>>> Just answering your previous question, the following logs were the
>>>>>> same as before the error, as if the pipeline were still getting the
>>>>>> messages, for example:
>>>>>>
>>>>>> (...)
>>>>>> Resetting offset for partition com.arquivei.dataeng.andre-0 to offset
>>>>>> 10.
>>>>>> Resetting offset for partition com.arquivei.dataeng.andre-0 to offset
>>>>>> 15.
>>>>>> ERROR
>>>>>> Resetting offset for partition com.arquivei.dataeng.andre-0 to offset
>>>>>> 22.
>>>>>> Resetting offset for partition com.arquivei.dataeng.andre-0 to offset
>>>>>> 30.
>>>>>> (...)
>>>>>>
>>>>>> But when checking the Kafka Consumer Group, the current offset stays
>>>>>> at 15, the commited offset from the last processed message, before the
>>>>>> error.
>>>>>>
>>>>>> We'll file a bug, but we could now reproduce the issue in a Dev
>>>>>> scenario.
>>>>>> We started the same pipeline using the direct runner, without Google
>>>>>> Dataflow. We blocked the Kafka Broker network and the same error was
>>>>>> thrown. Then we unblocked the network and the pipeline was able to
>>>>>> successfully process the subsequent messages.
>>>>>> When we started the same pipeline in the Dataflow runner and did the
>>>>>> same test, the same problem from our production scenario happened, Dataflow
>>>>>> couldn't process the new messages. Unfortunately, we've stopped the
>>>>>> dataflow job in production, but the problematic dev job is still running
>>>>>> and the log file of the VM is attached. Thank you very much.
>>>>>> Best regards
>>>>>>
>>>>>> Em ter, 11 de set de 2018 às 18:28, Raghu Angadi <ra...@google.com>
>>>>>> escreveu:
>>>>>>
>>>>>>> Specifically, I am interested if you have any thread running
>>>>>>> 'consumerPollLoop()' [1]. There should always be one (if a worker is
>>>>>>> assigned one of the partitions). It is possible that KafkaClient itself is
>>>>>>> hasn't recovered from the group coordinator error (though unlikely).
>>>>>>>
>>>>>>>
>>>>>>> https://github.com/apache/beam/blob/master/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaUnboundedReader.java#L570
>>>>>>>
>>>>>>> On Tue, Sep 11, 2018 at 12:31 PM Raghu Angadi <ra...@google.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hi Eduardo,
>>>>>>>>
>>>>>>>> In case of any error, the pipeline should keep on trying to fetch.
>>>>>>>> I don't know about this particular error. Do you see any others afterwards
>>>>>>>> in the log?
>>>>>>>> Couple of things you could try if the logs are not useful :
>>>>>>>>  - login to one of the VMs and get stacktrace of java worker (look
>>>>>>>> for a container called java-streaming)
>>>>>>>>  - file a support bug or stackoverflow question with jobid so that
>>>>>>>> Dataflow oncall can take a look.
>>>>>>>>
>>>>>>>> Raghu.
>>>>>>>>
>>>>>>>>
>>>>>>>> On Tue, Sep 11, 2018 at 12:10 PM Eduardo Soldera <
>>>>>>>> eduardo.soldera@arquivei.com.br> wrote:
>>>>>>>>
>>>>>>>>> Hi,
>>>>>>>>> We have a Apache Beam pipeline running in Google Dataflow using
>>>>>>>>> KafkaIO. Suddenly the pipeline stop fetching Kafka messages at all, as our
>>>>>>>>> other workers from other pipelines continued to get Kafka messages.
>>>>>>>>>
>>>>>>>>> At the moment it stopped we got these messages:
>>>>>>>>>
>>>>>>>>> I  [Consumer clientId=consumer-1, groupId=genericPipe] Error sending fetch request (sessionId=1396189203, epoch=2431598) to node 3: org.apache.kafka.common.errors.DisconnectException.
>>>>>>>>> I  [Consumer clientId=consumer-1, groupId=genericPipe] Group coordinator 10.0.52.70:9093 (id: 2147483646 rack: null) is unavailable or invalid, will attempt rediscovery
>>>>>>>>> I  [Consumer clientId=consumer-1, groupId=genericPipe] Discovered group coordinator 10.0.52.70:9093 (id: 2147483646 rack: null)
>>>>>>>>>
>>>>>>>>> And then the pipeline stopped reading the messages.
>>>>>>>>>
>>>>>>>>> This is the KafkaIO setup  we have:
>>>>>>>>>
>>>>>>>>> KafkaIO.read[String,String]()
>>>>>>>>>   .withBootstrapServers(server)
>>>>>>>>>   .withTopic(topic)
>>>>>>>>>   .withKeyDeserializer(classOf[StringDeserializer])
>>>>>>>>>   .withValueDeserializer(classOf[StringDeserializer])
>>>>>>>>>   .updateConsumerProperties(properties)
>>>>>>>>>   .commitOffsetsInFinalize()
>>>>>>>>>   .withoutMetadata()
>>>>>>>>>
>>>>>>>>>  Any help will be much appreciated.
>>>>>>>>>
>>>>>>>>> Best regards,
>>>>>>>>> --
>>>>>>>>> Eduardo Soldera Garcia
>>>>>>>>> Data Engineer
>>>>>>>>> (16) 3509-5555 | www.arquivei.com.br
>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>>> [image: Arquivei.com.br – Inteligência em Notas Fiscais]
>>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>>> [image: Google seleciona Arquivei para imersão e mentoria no Vale
>>>>>>>>> do Silício]
>>>>>>>>> <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad>
>>>>>>>>> <https://www.facebook.com/arquivei>
>>>>>>>>> <https://www.linkedin.com/company/arquivei>
>>>>>>>>> <https://www.youtube.com/watch?v=sSUUKxbXnxk>
>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>>> --
>>>>>> Eduardo Soldera Garcia
>>>>>> Data Engineer
>>>>>> (16) 3509-5555 | www.arquivei.com.br
>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>> [image: Arquivei.com.br – Inteligência em Notas Fiscais]
>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>> [image: Google seleciona Arquivei para imersão e mentoria no Vale do
>>>>>> Silício]
>>>>>> <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad>
>>>>>> <https://www.facebook.com/arquivei>
>>>>>> <https://www.linkedin.com/company/arquivei>
>>>>>> <https://www.youtube.com/watch?v=sSUUKxbXnxk>
>>>>>>
>>>>>
>>>>
>>>> --
>>>> Eduardo Soldera Garcia
>>>> Data Engineer
>>>> (16) 3509-5555 | www.arquivei.com.br
>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>> [image: Arquivei.com.br – Inteligência em Notas Fiscais]
>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>> [image: Google seleciona Arquivei para imersão e mentoria no Vale do
>>>> Silício]
>>>> <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad>
>>>> <https://www.facebook.com/arquivei>
>>>> <https://www.linkedin.com/company/arquivei>
>>>> <https://www.youtube.com/watch?v=sSUUKxbXnxk>
>>>>
>>>

Re: Problem with KafkaIO

Posted by Raghu Angadi <ra...@google.com>.
Filed BEAM-5375 <https://issues.apache.org/jira/browse/BEAM-5375>. I will
fix it later this week.

On Wed, Sep 12, 2018 at 12:16 PM Raghu Angadi <ra...@google.com> wrote:

>
>
> On Wed, Sep 12, 2018 at 12:11 PM Raghu Angadi <ra...@google.com> wrote:
>
>> Thanks for the job id, I looked at the worker logs (following usual
>> support oncall access protocol that provides temporary access to things
>> like logs in GCP):
>>
>> Root issue looks like consumerPollLoop() mentioned earlier needs to
>> handle unchecked exception. In your case it is clear that poll thread
>> exited with a runtime exception. The reader does not check for it and
>> continues to wait for poll thread to enqueue messages. A fix should result
>> in an IOException for read from the source. The runners will handle that
>> appropriately after that.  I will file a jira.
>>
>> https://github.com/apache/beam/blob/master/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaUnboundedReader.java#L678
>>
>
> Ignore the link.. was pasted here by mistake.
>
>
>>
>> From the logs (with a comment below each one):
>>
>>    - 2018-09-12 06:13:07.345 PDT Reader-0: reading from kafka_topic-0
>>    starting at offset 2
>>       - Implies the reader is initialized and poll thread is started.
>>    - 2018-09-12 06:13:07.780 PDT Reader-0: first record offset 2
>>       - The reader actually got a message received by the poll thread
>>       from Kafka.
>>    - 2018-09-12 06:16:48.771 PDT Reader-0: exception while fetching
>>    latest offset for partition kafka_topic-0. will be retried.
>>       - This must have happened around the time when network was
>>       disrupted. This is from. Actual log is from another periodic task that
>>       fetches latest offsets for partitions.
>>
>> The poll thread must have died around the time network was disrupted.
>>
>> The following log comes from kafka client itself and is printed every
>> second when KafkaIO fetches latest offset. This log seems to be added in
>> recent versions. It is probably an unintentional log. I don't think there
>> is any better to fetch latest offsets than how KafkaIO does now. This is
>> logged inside consumer.position() called at [1].
>>
>>    - 2018-09-12 06:13:11.786 PDT [Consumer clientId=consumer-2,
>>    groupId=Reader-0_offset_consumer_1735388161_genericPipe] Resetting offset
>>    for partition com.arquivei.dataeng.andre-0 to offset 3.
>>
>> [1]:
>> https://github.com/apache/beam/blob/master/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaUnboundedReader.java#L678
>>
>
> This 'Resetting offset' is harmless, but is quite annoying to see in the
> worker logs. One way to avoid is to set kafka consumer's log level to
> WARNING. Ideally KafkaIO itself should do something to avoid it without
> user option.
>
>
>
>
>> On Wed, Sep 12, 2018 at 10:27 AM Eduardo Soldera <
>> eduardo.soldera@arquivei.com.br> wrote:
>>
>>> Hi Raghu! The job_id of our dev job is
>>> 2018-09-12_06_11_48-5600553605191377866.
>>>
>>> Thanks!
>>>
>>> Em qua, 12 de set de 2018 às 14:18, Raghu Angadi <ra...@google.com>
>>> escreveu:
>>>
>>>> Thanks for debugging.
>>>> Can you provide the job_id of your dev job? The stacktrace shows that
>>>> there is no thread running 'consumerPollLoop()' which can explain stuck
>>>> reader. You will likely find a logs at line 594 & 587 [1].  Dataflow caches
>>>> its readers and DirectRunner may not. That can explain DirectRunner resume
>>>> reads. The expectation in KafkaIO is that Kafka client library takes care
>>>> of retrying in case of connection problems (as documented). It is possible
>>>> that in some cases poll() throws and we need to restart the client in
>>>> KafkaIO.
>>>>
>>>> [1]:
>>>> https://github.com/apache/beam/blob/master/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaUnboundedReader.java#L594
>>>>
>>>> On Wed, Sep 12, 2018 at 9:59 AM Eduardo Soldera <
>>>> eduardo.soldera@arquivei.com.br> wrote:
>>>>
>>>>> Hi Raghu, thanks for your help.
>>>>> Just answering your previous question, the following logs were the
>>>>> same as before the error, as if the pipeline were still getting the
>>>>> messages, for example:
>>>>>
>>>>> (...)
>>>>> Resetting offset for partition com.arquivei.dataeng.andre-0 to offset
>>>>> 10.
>>>>> Resetting offset for partition com.arquivei.dataeng.andre-0 to offset
>>>>> 15.
>>>>> ERROR
>>>>> Resetting offset for partition com.arquivei.dataeng.andre-0 to offset
>>>>> 22.
>>>>> Resetting offset for partition com.arquivei.dataeng.andre-0 to offset
>>>>> 30.
>>>>> (...)
>>>>>
>>>>> But when checking the Kafka Consumer Group, the current offset stays
>>>>> at 15, the commited offset from the last processed message, before the
>>>>> error.
>>>>>
>>>>> We'll file a bug, but we could now reproduce the issue in a Dev
>>>>> scenario.
>>>>> We started the same pipeline using the direct runner, without Google
>>>>> Dataflow. We blocked the Kafka Broker network and the same error was
>>>>> thrown. Then we unblocked the network and the pipeline was able to
>>>>> successfully process the subsequent messages.
>>>>> When we started the same pipeline in the Dataflow runner and did the
>>>>> same test, the same problem from our production scenario happened, Dataflow
>>>>> couldn't process the new messages. Unfortunately, we've stopped the
>>>>> dataflow job in production, but the problematic dev job is still running
>>>>> and the log file of the VM is attached. Thank you very much.
>>>>> Best regards
>>>>>
>>>>> Em ter, 11 de set de 2018 às 18:28, Raghu Angadi <ra...@google.com>
>>>>> escreveu:
>>>>>
>>>>>> Specifically, I am interested if you have any thread running
>>>>>> 'consumerPollLoop()' [1]. There should always be one (if a worker is
>>>>>> assigned one of the partitions). It is possible that KafkaClient itself is
>>>>>> hasn't recovered from the group coordinator error (though unlikely).
>>>>>>
>>>>>>
>>>>>> https://github.com/apache/beam/blob/master/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaUnboundedReader.java#L570
>>>>>>
>>>>>> On Tue, Sep 11, 2018 at 12:31 PM Raghu Angadi <ra...@google.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi Eduardo,
>>>>>>>
>>>>>>> In case of any error, the pipeline should keep on trying to fetch. I
>>>>>>> don't know about this particular error. Do you see any others afterwards in
>>>>>>> the log?
>>>>>>> Couple of things you could try if the logs are not useful :
>>>>>>>  - login to one of the VMs and get stacktrace of java worker (look
>>>>>>> for a container called java-streaming)
>>>>>>>  - file a support bug or stackoverflow question with jobid so that
>>>>>>> Dataflow oncall can take a look.
>>>>>>>
>>>>>>> Raghu.
>>>>>>>
>>>>>>>
>>>>>>> On Tue, Sep 11, 2018 at 12:10 PM Eduardo Soldera <
>>>>>>> eduardo.soldera@arquivei.com.br> wrote:
>>>>>>>
>>>>>>>> Hi,
>>>>>>>> We have a Apache Beam pipeline running in Google Dataflow using
>>>>>>>> KafkaIO. Suddenly the pipeline stop fetching Kafka messages at all, as our
>>>>>>>> other workers from other pipelines continued to get Kafka messages.
>>>>>>>>
>>>>>>>> At the moment it stopped we got these messages:
>>>>>>>>
>>>>>>>> I  [Consumer clientId=consumer-1, groupId=genericPipe] Error sending fetch request (sessionId=1396189203, epoch=2431598) to node 3: org.apache.kafka.common.errors.DisconnectException.
>>>>>>>> I  [Consumer clientId=consumer-1, groupId=genericPipe] Group coordinator 10.0.52.70:9093 (id: 2147483646 rack: null) is unavailable or invalid, will attempt rediscovery
>>>>>>>> I  [Consumer clientId=consumer-1, groupId=genericPipe] Discovered group coordinator 10.0.52.70:9093 (id: 2147483646 rack: null)
>>>>>>>>
>>>>>>>> And then the pipeline stopped reading the messages.
>>>>>>>>
>>>>>>>> This is the KafkaIO setup  we have:
>>>>>>>>
>>>>>>>> KafkaIO.read[String,String]()
>>>>>>>>   .withBootstrapServers(server)
>>>>>>>>   .withTopic(topic)
>>>>>>>>   .withKeyDeserializer(classOf[StringDeserializer])
>>>>>>>>   .withValueDeserializer(classOf[StringDeserializer])
>>>>>>>>   .updateConsumerProperties(properties)
>>>>>>>>   .commitOffsetsInFinalize()
>>>>>>>>   .withoutMetadata()
>>>>>>>>
>>>>>>>>  Any help will be much appreciated.
>>>>>>>>
>>>>>>>> Best regards,
>>>>>>>> --
>>>>>>>> Eduardo Soldera Garcia
>>>>>>>> Data Engineer
>>>>>>>> (16) 3509-5555 | www.arquivei.com.br
>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>> [image: Arquivei.com.br – Inteligência em Notas Fiscais]
>>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>>> [image: Google seleciona Arquivei para imersão e mentoria no Vale
>>>>>>>> do Silício]
>>>>>>>> <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad>
>>>>>>>> <https://www.facebook.com/arquivei>
>>>>>>>> <https://www.linkedin.com/company/arquivei>
>>>>>>>> <https://www.youtube.com/watch?v=sSUUKxbXnxk>
>>>>>>>>
>>>>>>>
>>>>>
>>>>> --
>>>>> Eduardo Soldera Garcia
>>>>> Data Engineer
>>>>> (16) 3509-5555 | www.arquivei.com.br
>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>> [image: Arquivei.com.br – Inteligência em Notas Fiscais]
>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>> [image: Google seleciona Arquivei para imersão e mentoria no Vale do
>>>>> Silício]
>>>>> <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad>
>>>>> <https://www.facebook.com/arquivei>
>>>>> <https://www.linkedin.com/company/arquivei>
>>>>> <https://www.youtube.com/watch?v=sSUUKxbXnxk>
>>>>>
>>>>
>>>
>>> --
>>> Eduardo Soldera Garcia
>>> Data Engineer
>>> (16) 3509-5555 | www.arquivei.com.br
>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>> [image: Arquivei.com.br – Inteligência em Notas Fiscais]
>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>> [image: Google seleciona Arquivei para imersão e mentoria no Vale do
>>> Silício]
>>> <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad>
>>> <https://www.facebook.com/arquivei>
>>> <https://www.linkedin.com/company/arquivei>
>>> <https://www.youtube.com/watch?v=sSUUKxbXnxk>
>>>
>>

Re: Problem with KafkaIO

Posted by Raghu Angadi <ra...@google.com>.
On Wed, Sep 12, 2018 at 12:11 PM Raghu Angadi <ra...@google.com> wrote:

> Thanks for the job id, I looked at the worker logs (following usual
> support oncall access protocol that provides temporary access to things
> like logs in GCP):
>
> Root issue looks like consumerPollLoop() mentioned earlier needs to handle
> unchecked exception. In your case it is clear that poll thread exited with
> a runtime exception. The reader does not check for it and continues to wait
> for poll thread to enqueue messages. A fix should result in an IOException
> for read from the source. The runners will handle that appropriately after
> that.  I will file a jira.
>
> https://github.com/apache/beam/blob/master/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaUnboundedReader.java#L678
>

Ignore the link.. was pasted here by mistake.


>
> From the logs (with a comment below each one):
>
>    - 2018-09-12 06:13:07.345 PDT Reader-0: reading from kafka_topic-0
>    starting at offset 2
>       - Implies the reader is initialized and poll thread is started.
>    - 2018-09-12 06:13:07.780 PDT Reader-0: first record offset 2
>       - The reader actually got a message received by the poll thread
>       from Kafka.
>    - 2018-09-12 06:16:48.771 PDT Reader-0: exception while fetching
>    latest offset for partition kafka_topic-0. will be retried.
>       - This must have happened around the time when network was
>       disrupted. This is from. Actual log is from another periodic task that
>       fetches latest offsets for partitions.
>
> The poll thread must have died around the time network was disrupted.
>
> The following log comes from kafka client itself and is printed every
> second when KafkaIO fetches latest offset. This log seems to be added in
> recent versions. It is probably an unintentional log. I don't think there
> is any better to fetch latest offsets than how KafkaIO does now. This is
> logged inside consumer.position() called at [1].
>
>    - 2018-09-12 06:13:11.786 PDT [Consumer clientId=consumer-2,
>    groupId=Reader-0_offset_consumer_1735388161_genericPipe] Resetting offset
>    for partition com.arquivei.dataeng.andre-0 to offset 3.
>
> [1]:
> https://github.com/apache/beam/blob/master/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaUnboundedReader.java#L678
>

This 'Resetting offset' is harmless, but is quite annoying to see in the
worker logs. One way to avoid is to set kafka consumer's log level to
WARNING. Ideally KafkaIO itself should do something to avoid it without
user option.




> On Wed, Sep 12, 2018 at 10:27 AM Eduardo Soldera <
> eduardo.soldera@arquivei.com.br> wrote:
>
>> Hi Raghu! The job_id of our dev job is
>> 2018-09-12_06_11_48-5600553605191377866.
>>
>> Thanks!
>>
>> Em qua, 12 de set de 2018 às 14:18, Raghu Angadi <ra...@google.com>
>> escreveu:
>>
>>> Thanks for debugging.
>>> Can you provide the job_id of your dev job? The stacktrace shows that
>>> there is no thread running 'consumerPollLoop()' which can explain stuck
>>> reader. You will likely find a logs at line 594 & 587 [1].  Dataflow caches
>>> its readers and DirectRunner may not. That can explain DirectRunner resume
>>> reads. The expectation in KafkaIO is that Kafka client library takes care
>>> of retrying in case of connection problems (as documented). It is possible
>>> that in some cases poll() throws and we need to restart the client in
>>> KafkaIO.
>>>
>>> [1]:
>>> https://github.com/apache/beam/blob/master/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaUnboundedReader.java#L594
>>>
>>> On Wed, Sep 12, 2018 at 9:59 AM Eduardo Soldera <
>>> eduardo.soldera@arquivei.com.br> wrote:
>>>
>>>> Hi Raghu, thanks for your help.
>>>> Just answering your previous question, the following logs were the same
>>>> as before the error, as if the pipeline were still getting the messages,
>>>> for example:
>>>>
>>>> (...)
>>>> Resetting offset for partition com.arquivei.dataeng.andre-0 to offset
>>>> 10.
>>>> Resetting offset for partition com.arquivei.dataeng.andre-0 to offset
>>>> 15.
>>>> ERROR
>>>> Resetting offset for partition com.arquivei.dataeng.andre-0 to offset
>>>> 22.
>>>> Resetting offset for partition com.arquivei.dataeng.andre-0 to offset
>>>> 30.
>>>> (...)
>>>>
>>>> But when checking the Kafka Consumer Group, the current offset stays at
>>>> 15, the commited offset from the last processed message, before the error.
>>>>
>>>> We'll file a bug, but we could now reproduce the issue in a Dev
>>>> scenario.
>>>> We started the same pipeline using the direct runner, without Google
>>>> Dataflow. We blocked the Kafka Broker network and the same error was
>>>> thrown. Then we unblocked the network and the pipeline was able to
>>>> successfully process the subsequent messages.
>>>> When we started the same pipeline in the Dataflow runner and did the
>>>> same test, the same problem from our production scenario happened, Dataflow
>>>> couldn't process the new messages. Unfortunately, we've stopped the
>>>> dataflow job in production, but the problematic dev job is still running
>>>> and the log file of the VM is attached. Thank you very much.
>>>> Best regards
>>>>
>>>> Em ter, 11 de set de 2018 às 18:28, Raghu Angadi <ra...@google.com>
>>>> escreveu:
>>>>
>>>>> Specifically, I am interested if you have any thread running
>>>>> 'consumerPollLoop()' [1]. There should always be one (if a worker is
>>>>> assigned one of the partitions). It is possible that KafkaClient itself is
>>>>> hasn't recovered from the group coordinator error (though unlikely).
>>>>>
>>>>>
>>>>> https://github.com/apache/beam/blob/master/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaUnboundedReader.java#L570
>>>>>
>>>>> On Tue, Sep 11, 2018 at 12:31 PM Raghu Angadi <ra...@google.com>
>>>>> wrote:
>>>>>
>>>>>> Hi Eduardo,
>>>>>>
>>>>>> In case of any error, the pipeline should keep on trying to fetch. I
>>>>>> don't know about this particular error. Do you see any others afterwards in
>>>>>> the log?
>>>>>> Couple of things you could try if the logs are not useful :
>>>>>>  - login to one of the VMs and get stacktrace of java worker (look
>>>>>> for a container called java-streaming)
>>>>>>  - file a support bug or stackoverflow question with jobid so that
>>>>>> Dataflow oncall can take a look.
>>>>>>
>>>>>> Raghu.
>>>>>>
>>>>>>
>>>>>> On Tue, Sep 11, 2018 at 12:10 PM Eduardo Soldera <
>>>>>> eduardo.soldera@arquivei.com.br> wrote:
>>>>>>
>>>>>>> Hi,
>>>>>>> We have a Apache Beam pipeline running in Google Dataflow using
>>>>>>> KafkaIO. Suddenly the pipeline stop fetching Kafka messages at all, as our
>>>>>>> other workers from other pipelines continued to get Kafka messages.
>>>>>>>
>>>>>>> At the moment it stopped we got these messages:
>>>>>>>
>>>>>>> I  [Consumer clientId=consumer-1, groupId=genericPipe] Error sending fetch request (sessionId=1396189203, epoch=2431598) to node 3: org.apache.kafka.common.errors.DisconnectException.
>>>>>>> I  [Consumer clientId=consumer-1, groupId=genericPipe] Group coordinator 10.0.52.70:9093 (id: 2147483646 rack: null) is unavailable or invalid, will attempt rediscovery
>>>>>>> I  [Consumer clientId=consumer-1, groupId=genericPipe] Discovered group coordinator 10.0.52.70:9093 (id: 2147483646 rack: null)
>>>>>>>
>>>>>>> And then the pipeline stopped reading the messages.
>>>>>>>
>>>>>>> This is the KafkaIO setup  we have:
>>>>>>>
>>>>>>> KafkaIO.read[String,String]()
>>>>>>>   .withBootstrapServers(server)
>>>>>>>   .withTopic(topic)
>>>>>>>   .withKeyDeserializer(classOf[StringDeserializer])
>>>>>>>   .withValueDeserializer(classOf[StringDeserializer])
>>>>>>>   .updateConsumerProperties(properties)
>>>>>>>   .commitOffsetsInFinalize()
>>>>>>>   .withoutMetadata()
>>>>>>>
>>>>>>>  Any help will be much appreciated.
>>>>>>>
>>>>>>> Best regards,
>>>>>>> --
>>>>>>> Eduardo Soldera Garcia
>>>>>>> Data Engineer
>>>>>>> (16) 3509-5555 | www.arquivei.com.br
>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>> [image: Arquivei.com.br – Inteligência em Notas Fiscais]
>>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>>> [image: Google seleciona Arquivei para imersão e mentoria no Vale do
>>>>>>> Silício]
>>>>>>> <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad>
>>>>>>> <https://www.facebook.com/arquivei>
>>>>>>> <https://www.linkedin.com/company/arquivei>
>>>>>>> <https://www.youtube.com/watch?v=sSUUKxbXnxk>
>>>>>>>
>>>>>>
>>>>
>>>> --
>>>> Eduardo Soldera Garcia
>>>> Data Engineer
>>>> (16) 3509-5555 | www.arquivei.com.br
>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>> [image: Arquivei.com.br – Inteligência em Notas Fiscais]
>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>> [image: Google seleciona Arquivei para imersão e mentoria no Vale do
>>>> Silício]
>>>> <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad>
>>>> <https://www.facebook.com/arquivei>
>>>> <https://www.linkedin.com/company/arquivei>
>>>> <https://www.youtube.com/watch?v=sSUUKxbXnxk>
>>>>
>>>
>>
>> --
>> Eduardo Soldera Garcia
>> Data Engineer
>> (16) 3509-5555 | www.arquivei.com.br
>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>> [image: Arquivei.com.br – Inteligência em Notas Fiscais]
>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>> [image: Google seleciona Arquivei para imersão e mentoria no Vale do
>> Silício]
>> <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad>
>> <https://www.facebook.com/arquivei>
>> <https://www.linkedin.com/company/arquivei>
>> <https://www.youtube.com/watch?v=sSUUKxbXnxk>
>>
>

Re: Problem with KafkaIO

Posted by Raghu Angadi <ra...@google.com>.
Thanks for the job id, I looked at the worker logs (following usual support
oncall access protocol that provides temporary access to things like logs
in GCP):

Root issue looks like consumerPollLoop() mentioned earlier needs to handle
unchecked exception. In your case it is clear that poll thread exited with
a runtime exception. The reader does not check for it and continues to wait
for poll thread to enqueue messages. A fix should result in an IOException
for read from the source. The runners will handle that appropriately after
that.  I will file a jira.
https://github.com/apache/beam/blob/master/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaUnboundedReader.java#L678
From the logs (with a comment below each one):

   - 2018-09-12 06:13:07.345 PDT Reader-0: reading from kafka_topic-0
   starting at offset 2
      - Implies the reader is initialized and poll thread is started.
   - 2018-09-12 06:13:07.780 PDT Reader-0: first record offset 2
      - The reader actually got a message received by the poll thread from
      Kafka.
   - 2018-09-12 06:16:48.771 PDT Reader-0: exception while fetching latest
   offset for partition kafka_topic-0. will be retried.
      - This must have happened around the time when network was disrupted.
      This is from. Actual log is from another periodic task that
fetches latest
      offsets for partitions.

The poll thread must have died around the time network was disrupted.

The following log comes from kafka client itself and is printed every
second when KafkaIO fetches latest offset. This log seems to be added in
recent versions. It is probably an unintentional log. I don't think there
is any better to fetch latest offsets than how KafkaIO does now. This is
logged inside consumer.position() called at [1].

   - 2018-09-12 06:13:11.786 PDT [Consumer clientId=consumer-2,
   groupId=Reader-0_offset_consumer_1735388161_genericPipe] Resetting offset
   for partition com.arquivei.dataeng.andre-0 to offset 3.

[1]:
https://github.com/apache/beam/blob/master/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaUnboundedReader.java#L678

On Wed, Sep 12, 2018 at 10:27 AM Eduardo Soldera <
eduardo.soldera@arquivei.com.br> wrote:

> Hi Raghu! The job_id of our dev job is
> 2018-09-12_06_11_48-5600553605191377866.
>
> Thanks!
>
> Em qua, 12 de set de 2018 às 14:18, Raghu Angadi <ra...@google.com>
> escreveu:
>
>> Thanks for debugging.
>> Can you provide the job_id of your dev job? The stacktrace shows that
>> there is no thread running 'consumerPollLoop()' which can explain stuck
>> reader. You will likely find a logs at line 594 & 587 [1].  Dataflow caches
>> its readers and DirectRunner may not. That can explain DirectRunner resume
>> reads. The expectation in KafkaIO is that Kafka client library takes care
>> of retrying in case of connection problems (as documented). It is possible
>> that in some cases poll() throws and we need to restart the client in
>> KafkaIO.
>>
>> [1]:
>> https://github.com/apache/beam/blob/master/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaUnboundedReader.java#L594
>>
>> On Wed, Sep 12, 2018 at 9:59 AM Eduardo Soldera <
>> eduardo.soldera@arquivei.com.br> wrote:
>>
>>> Hi Raghu, thanks for your help.
>>> Just answering your previous question, the following logs were the same
>>> as before the error, as if the pipeline were still getting the messages,
>>> for example:
>>>
>>> (...)
>>> Resetting offset for partition com.arquivei.dataeng.andre-0 to offset 10.
>>> Resetting offset for partition com.arquivei.dataeng.andre-0 to offset 15.
>>> ERROR
>>> Resetting offset for partition com.arquivei.dataeng.andre-0 to offset 22.
>>> Resetting offset for partition com.arquivei.dataeng.andre-0 to offset 30.
>>> (...)
>>>
>>> But when checking the Kafka Consumer Group, the current offset stays at
>>> 15, the commited offset from the last processed message, before the error.
>>>
>>> We'll file a bug, but we could now reproduce the issue in a Dev scenario.
>>> We started the same pipeline using the direct runner, without Google
>>> Dataflow. We blocked the Kafka Broker network and the same error was
>>> thrown. Then we unblocked the network and the pipeline was able to
>>> successfully process the subsequent messages.
>>> When we started the same pipeline in the Dataflow runner and did the
>>> same test, the same problem from our production scenario happened, Dataflow
>>> couldn't process the new messages. Unfortunately, we've stopped the
>>> dataflow job in production, but the problematic dev job is still running
>>> and the log file of the VM is attached. Thank you very much.
>>> Best regards
>>>
>>> Em ter, 11 de set de 2018 às 18:28, Raghu Angadi <ra...@google.com>
>>> escreveu:
>>>
>>>> Specifically, I am interested if you have any thread running
>>>> 'consumerPollLoop()' [1]. There should always be one (if a worker is
>>>> assigned one of the partitions). It is possible that KafkaClient itself is
>>>> hasn't recovered from the group coordinator error (though unlikely).
>>>>
>>>>
>>>> https://github.com/apache/beam/blob/master/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaUnboundedReader.java#L570
>>>>
>>>> On Tue, Sep 11, 2018 at 12:31 PM Raghu Angadi <ra...@google.com>
>>>> wrote:
>>>>
>>>>> Hi Eduardo,
>>>>>
>>>>> In case of any error, the pipeline should keep on trying to fetch. I
>>>>> don't know about this particular error. Do you see any others afterwards in
>>>>> the log?
>>>>> Couple of things you could try if the logs are not useful :
>>>>>  - login to one of the VMs and get stacktrace of java worker (look for
>>>>> a container called java-streaming)
>>>>>  - file a support bug or stackoverflow question with jobid so that
>>>>> Dataflow oncall can take a look.
>>>>>
>>>>> Raghu.
>>>>>
>>>>>
>>>>> On Tue, Sep 11, 2018 at 12:10 PM Eduardo Soldera <
>>>>> eduardo.soldera@arquivei.com.br> wrote:
>>>>>
>>>>>> Hi,
>>>>>> We have a Apache Beam pipeline running in Google Dataflow using
>>>>>> KafkaIO. Suddenly the pipeline stop fetching Kafka messages at all, as our
>>>>>> other workers from other pipelines continued to get Kafka messages.
>>>>>>
>>>>>> At the moment it stopped we got these messages:
>>>>>>
>>>>>> I  [Consumer clientId=consumer-1, groupId=genericPipe] Error sending fetch request (sessionId=1396189203, epoch=2431598) to node 3: org.apache.kafka.common.errors.DisconnectException.
>>>>>> I  [Consumer clientId=consumer-1, groupId=genericPipe] Group coordinator 10.0.52.70:9093 (id: 2147483646 rack: null) is unavailable or invalid, will attempt rediscovery
>>>>>> I  [Consumer clientId=consumer-1, groupId=genericPipe] Discovered group coordinator 10.0.52.70:9093 (id: 2147483646 rack: null)
>>>>>>
>>>>>> And then the pipeline stopped reading the messages.
>>>>>>
>>>>>> This is the KafkaIO setup  we have:
>>>>>>
>>>>>> KafkaIO.read[String,String]()
>>>>>>   .withBootstrapServers(server)
>>>>>>   .withTopic(topic)
>>>>>>   .withKeyDeserializer(classOf[StringDeserializer])
>>>>>>   .withValueDeserializer(classOf[StringDeserializer])
>>>>>>   .updateConsumerProperties(properties)
>>>>>>   .commitOffsetsInFinalize()
>>>>>>   .withoutMetadata()
>>>>>>
>>>>>>  Any help will be much appreciated.
>>>>>>
>>>>>> Best regards,
>>>>>> --
>>>>>> Eduardo Soldera Garcia
>>>>>> Data Engineer
>>>>>> (16) 3509-5555 | www.arquivei.com.br
>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>> [image: Arquivei.com.br – Inteligência em Notas Fiscais]
>>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>>> [image: Google seleciona Arquivei para imersão e mentoria no Vale do
>>>>>> Silício]
>>>>>> <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad>
>>>>>> <https://www.facebook.com/arquivei>
>>>>>> <https://www.linkedin.com/company/arquivei>
>>>>>> <https://www.youtube.com/watch?v=sSUUKxbXnxk>
>>>>>>
>>>>>
>>>
>>> --
>>> Eduardo Soldera Garcia
>>> Data Engineer
>>> (16) 3509-5555 | www.arquivei.com.br
>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>> [image: Arquivei.com.br – Inteligência em Notas Fiscais]
>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>> [image: Google seleciona Arquivei para imersão e mentoria no Vale do
>>> Silício]
>>> <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad>
>>> <https://www.facebook.com/arquivei>
>>> <https://www.linkedin.com/company/arquivei>
>>> <https://www.youtube.com/watch?v=sSUUKxbXnxk>
>>>
>>
>
> --
> Eduardo Soldera Garcia
> Data Engineer
> (16) 3509-5555 | www.arquivei.com.br
> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
> [image: Arquivei.com.br – Inteligência em Notas Fiscais]
> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
> [image: Google seleciona Arquivei para imersão e mentoria no Vale do
> Silício]
> <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad>
> <https://www.facebook.com/arquivei>
> <https://www.linkedin.com/company/arquivei>
> <https://www.youtube.com/watch?v=sSUUKxbXnxk>
>

Re: Problem with KafkaIO

Posted by Eduardo Soldera <ed...@arquivei.com.br>.
Hi Raghu! The job_id of our dev job is
2018-09-12_06_11_48-5600553605191377866.

Thanks!

Em qua, 12 de set de 2018 às 14:18, Raghu Angadi <ra...@google.com>
escreveu:

> Thanks for debugging.
> Can you provide the job_id of your dev job? The stacktrace shows that
> there is no thread running 'consumerPollLoop()' which can explain stuck
> reader. You will likely find a logs at line 594 & 587 [1].  Dataflow caches
> its readers and DirectRunner may not. That can explain DirectRunner resume
> reads. The expectation in KafkaIO is that Kafka client library takes care
> of retrying in case of connection problems (as documented). It is possible
> that in some cases poll() throws and we need to restart the client in
> KafkaIO.
>
> [1]:
> https://github.com/apache/beam/blob/master/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaUnboundedReader.java#L594
>
> On Wed, Sep 12, 2018 at 9:59 AM Eduardo Soldera <
> eduardo.soldera@arquivei.com.br> wrote:
>
>> Hi Raghu, thanks for your help.
>> Just answering your previous question, the following logs were the same
>> as before the error, as if the pipeline were still getting the messages,
>> for example:
>>
>> (...)
>> Resetting offset for partition com.arquivei.dataeng.andre-0 to offset 10.
>> Resetting offset for partition com.arquivei.dataeng.andre-0 to offset 15.
>> ERROR
>> Resetting offset for partition com.arquivei.dataeng.andre-0 to offset 22.
>> Resetting offset for partition com.arquivei.dataeng.andre-0 to offset 30.
>> (...)
>>
>> But when checking the Kafka Consumer Group, the current offset stays at
>> 15, the commited offset from the last processed message, before the error.
>>
>> We'll file a bug, but we could now reproduce the issue in a Dev scenario.
>> We started the same pipeline using the direct runner, without Google
>> Dataflow. We blocked the Kafka Broker network and the same error was
>> thrown. Then we unblocked the network and the pipeline was able to
>> successfully process the subsequent messages.
>> When we started the same pipeline in the Dataflow runner and did the same
>> test, the same problem from our production scenario happened, Dataflow
>> couldn't process the new messages. Unfortunately, we've stopped the
>> dataflow job in production, but the problematic dev job is still running
>> and the log file of the VM is attached. Thank you very much.
>> Best regards
>>
>> Em ter, 11 de set de 2018 às 18:28, Raghu Angadi <ra...@google.com>
>> escreveu:
>>
>>> Specifically, I am interested if you have any thread running
>>> 'consumerPollLoop()' [1]. There should always be one (if a worker is
>>> assigned one of the partitions). It is possible that KafkaClient itself is
>>> hasn't recovered from the group coordinator error (though unlikely).
>>>
>>>
>>> https://github.com/apache/beam/blob/master/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaUnboundedReader.java#L570
>>>
>>> On Tue, Sep 11, 2018 at 12:31 PM Raghu Angadi <ra...@google.com>
>>> wrote:
>>>
>>>> Hi Eduardo,
>>>>
>>>> In case of any error, the pipeline should keep on trying to fetch. I
>>>> don't know about this particular error. Do you see any others afterwards in
>>>> the log?
>>>> Couple of things you could try if the logs are not useful :
>>>>  - login to one of the VMs and get stacktrace of java worker (look for
>>>> a container called java-streaming)
>>>>  - file a support bug or stackoverflow question with jobid so that
>>>> Dataflow oncall can take a look.
>>>>
>>>> Raghu.
>>>>
>>>>
>>>> On Tue, Sep 11, 2018 at 12:10 PM Eduardo Soldera <
>>>> eduardo.soldera@arquivei.com.br> wrote:
>>>>
>>>>> Hi,
>>>>> We have a Apache Beam pipeline running in Google Dataflow using
>>>>> KafkaIO. Suddenly the pipeline stop fetching Kafka messages at all, as our
>>>>> other workers from other pipelines continued to get Kafka messages.
>>>>>
>>>>> At the moment it stopped we got these messages:
>>>>>
>>>>> I  [Consumer clientId=consumer-1, groupId=genericPipe] Error sending fetch request (sessionId=1396189203, epoch=2431598) to node 3: org.apache.kafka.common.errors.DisconnectException.
>>>>> I  [Consumer clientId=consumer-1, groupId=genericPipe] Group coordinator 10.0.52.70:9093 (id: 2147483646 rack: null) is unavailable or invalid, will attempt rediscovery
>>>>> I  [Consumer clientId=consumer-1, groupId=genericPipe] Discovered group coordinator 10.0.52.70:9093 (id: 2147483646 rack: null)
>>>>>
>>>>> And then the pipeline stopped reading the messages.
>>>>>
>>>>> This is the KafkaIO setup  we have:
>>>>>
>>>>> KafkaIO.read[String,String]()
>>>>>   .withBootstrapServers(server)
>>>>>   .withTopic(topic)
>>>>>   .withKeyDeserializer(classOf[StringDeserializer])
>>>>>   .withValueDeserializer(classOf[StringDeserializer])
>>>>>   .updateConsumerProperties(properties)
>>>>>   .commitOffsetsInFinalize()
>>>>>   .withoutMetadata()
>>>>>
>>>>>  Any help will be much appreciated.
>>>>>
>>>>> Best regards,
>>>>> --
>>>>> Eduardo Soldera Garcia
>>>>> Data Engineer
>>>>> (16) 3509-5555 | www.arquivei.com.br
>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>> [image: Arquivei.com.br – Inteligência em Notas Fiscais]
>>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>>> [image: Google seleciona Arquivei para imersão e mentoria no Vale do
>>>>> Silício]
>>>>> <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad>
>>>>> <https://www.facebook.com/arquivei>
>>>>> <https://www.linkedin.com/company/arquivei>
>>>>> <https://www.youtube.com/watch?v=sSUUKxbXnxk>
>>>>>
>>>>
>>
>> --
>> Eduardo Soldera Garcia
>> Data Engineer
>> (16) 3509-5555 | www.arquivei.com.br
>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>> [image: Arquivei.com.br – Inteligência em Notas Fiscais]
>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>> [image: Google seleciona Arquivei para imersão e mentoria no Vale do
>> Silício]
>> <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad>
>> <https://www.facebook.com/arquivei>
>> <https://www.linkedin.com/company/arquivei>
>> <https://www.youtube.com/watch?v=sSUUKxbXnxk>
>>
>

-- 
Eduardo Soldera Garcia
Data Engineer
(16) 3509-5555 | www.arquivei.com.br
<https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
[image: Arquivei.com.br – Inteligência em Notas Fiscais]
<https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
[image: Google seleciona Arquivei para imersão e mentoria no Vale do
Silício]
<https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad>
<https://www.facebook.com/arquivei>
<https://www.linkedin.com/company/arquivei>
<https://www.youtube.com/watch?v=sSUUKxbXnxk>

Re: Problem with KafkaIO

Posted by Raghu Angadi <ra...@google.com>.
Thanks for debugging.
Can you provide the job_id of your dev job? The stacktrace shows that there
is no thread running 'consumerPollLoop()' which can explain stuck reader.
You will likely find a logs at line 594 & 587 [1].  Dataflow caches its
readers and DirectRunner may not. That can explain DirectRunner resume
reads. The expectation in KafkaIO is that Kafka client library takes care
of retrying in case of connection problems (as documented). It is possible
that in some cases poll() throws and we need to restart the client in
KafkaIO.

[1]:
https://github.com/apache/beam/blob/master/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaUnboundedReader.java#L594

On Wed, Sep 12, 2018 at 9:59 AM Eduardo Soldera <
eduardo.soldera@arquivei.com.br> wrote:

> Hi Raghu, thanks for your help.
> Just answering your previous question, the following logs were the same as
> before the error, as if the pipeline were still getting the messages, for
> example:
>
> (...)
> Resetting offset for partition com.arquivei.dataeng.andre-0 to offset 10.
> Resetting offset for partition com.arquivei.dataeng.andre-0 to offset 15.
> ERROR
> Resetting offset for partition com.arquivei.dataeng.andre-0 to offset 22.
> Resetting offset for partition com.arquivei.dataeng.andre-0 to offset 30.
> (...)
>
> But when checking the Kafka Consumer Group, the current offset stays at
> 15, the commited offset from the last processed message, before the error.
>
> We'll file a bug, but we could now reproduce the issue in a Dev scenario.
> We started the same pipeline using the direct runner, without Google
> Dataflow. We blocked the Kafka Broker network and the same error was
> thrown. Then we unblocked the network and the pipeline was able to
> successfully process the subsequent messages.
> When we started the same pipeline in the Dataflow runner and did the same
> test, the same problem from our production scenario happened, Dataflow
> couldn't process the new messages. Unfortunately, we've stopped the
> dataflow job in production, but the problematic dev job is still running
> and the log file of the VM is attached. Thank you very much.
> Best regards
>
> Em ter, 11 de set de 2018 às 18:28, Raghu Angadi <ra...@google.com>
> escreveu:
>
>> Specifically, I am interested if you have any thread running
>> 'consumerPollLoop()' [1]. There should always be one (if a worker is
>> assigned one of the partitions). It is possible that KafkaClient itself is
>> hasn't recovered from the group coordinator error (though unlikely).
>>
>>
>> https://github.com/apache/beam/blob/master/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaUnboundedReader.java#L570
>>
>> On Tue, Sep 11, 2018 at 12:31 PM Raghu Angadi <ra...@google.com> wrote:
>>
>>> Hi Eduardo,
>>>
>>> In case of any error, the pipeline should keep on trying to fetch. I
>>> don't know about this particular error. Do you see any others afterwards in
>>> the log?
>>> Couple of things you could try if the logs are not useful :
>>>  - login to one of the VMs and get stacktrace of java worker (look for a
>>> container called java-streaming)
>>>  - file a support bug or stackoverflow question with jobid so that
>>> Dataflow oncall can take a look.
>>>
>>> Raghu.
>>>
>>>
>>> On Tue, Sep 11, 2018 at 12:10 PM Eduardo Soldera <
>>> eduardo.soldera@arquivei.com.br> wrote:
>>>
>>>> Hi,
>>>> We have a Apache Beam pipeline running in Google Dataflow using
>>>> KafkaIO. Suddenly the pipeline stop fetching Kafka messages at all, as our
>>>> other workers from other pipelines continued to get Kafka messages.
>>>>
>>>> At the moment it stopped we got these messages:
>>>>
>>>> I  [Consumer clientId=consumer-1, groupId=genericPipe] Error sending fetch request (sessionId=1396189203, epoch=2431598) to node 3: org.apache.kafka.common.errors.DisconnectException.
>>>> I  [Consumer clientId=consumer-1, groupId=genericPipe] Group coordinator 10.0.52.70:9093 (id: 2147483646 rack: null) is unavailable or invalid, will attempt rediscovery
>>>> I  [Consumer clientId=consumer-1, groupId=genericPipe] Discovered group coordinator 10.0.52.70:9093 (id: 2147483646 rack: null)
>>>>
>>>> And then the pipeline stopped reading the messages.
>>>>
>>>> This is the KafkaIO setup  we have:
>>>>
>>>> KafkaIO.read[String,String]()
>>>>   .withBootstrapServers(server)
>>>>   .withTopic(topic)
>>>>   .withKeyDeserializer(classOf[StringDeserializer])
>>>>   .withValueDeserializer(classOf[StringDeserializer])
>>>>   .updateConsumerProperties(properties)
>>>>   .commitOffsetsInFinalize()
>>>>   .withoutMetadata()
>>>>
>>>>  Any help will be much appreciated.
>>>>
>>>> Best regards,
>>>> --
>>>> Eduardo Soldera Garcia
>>>> Data Engineer
>>>> (16) 3509-5555 | www.arquivei.com.br
>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>> [image: Arquivei.com.br – Inteligência em Notas Fiscais]
>>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>>> [image: Google seleciona Arquivei para imersão e mentoria no Vale do
>>>> Silício]
>>>> <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad>
>>>> <https://www.facebook.com/arquivei>
>>>> <https://www.linkedin.com/company/arquivei>
>>>> <https://www.youtube.com/watch?v=sSUUKxbXnxk>
>>>>
>>>
>
> --
> Eduardo Soldera Garcia
> Data Engineer
> (16) 3509-5555 | www.arquivei.com.br
> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
> [image: Arquivei.com.br – Inteligência em Notas Fiscais]
> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
> [image: Google seleciona Arquivei para imersão e mentoria no Vale do
> Silício]
> <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad>
> <https://www.facebook.com/arquivei>
> <https://www.linkedin.com/company/arquivei>
> <https://www.youtube.com/watch?v=sSUUKxbXnxk>
>

Re: Problem with KafkaIO

Posted by Eduardo Soldera <ed...@arquivei.com.br>.
Hi Raghu, thanks for your help.
Just answering your previous question, the following logs were the same as
before the error, as if the pipeline were still getting the messages, for
example:

(...)
Resetting offset for partition com.arquivei.dataeng.andre-0 to offset 10.
Resetting offset for partition com.arquivei.dataeng.andre-0 to offset 15.
ERROR
Resetting offset for partition com.arquivei.dataeng.andre-0 to offset 22.
Resetting offset for partition com.arquivei.dataeng.andre-0 to offset 30.
(...)

But when checking the Kafka Consumer Group, the current offset stays at 15,
the commited offset from the last processed message, before the error.

We'll file a bug, but we could now reproduce the issue in a Dev scenario.
We started the same pipeline using the direct runner, without Google
Dataflow. We blocked the Kafka Broker network and the same error was
thrown. Then we unblocked the network and the pipeline was able to
successfully process the subsequent messages.
When we started the same pipeline in the Dataflow runner and did the same
test, the same problem from our production scenario happened, Dataflow
couldn't process the new messages. Unfortunately, we've stopped the
dataflow job in production, but the problematic dev job is still running
and the log file of the VM is attached. Thank you very much.
Best regards

Em ter, 11 de set de 2018 às 18:28, Raghu Angadi <ra...@google.com>
escreveu:

> Specifically, I am interested if you have any thread running
> 'consumerPollLoop()' [1]. There should always be one (if a worker is
> assigned one of the partitions). It is possible that KafkaClient itself is
> hasn't recovered from the group coordinator error (though unlikely).
>
>
> https://github.com/apache/beam/blob/master/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaUnboundedReader.java#L570
>
> On Tue, Sep 11, 2018 at 12:31 PM Raghu Angadi <ra...@google.com> wrote:
>
>> Hi Eduardo,
>>
>> In case of any error, the pipeline should keep on trying to fetch. I
>> don't know about this particular error. Do you see any others afterwards in
>> the log?
>> Couple of things you could try if the logs are not useful :
>>  - login to one of the VMs and get stacktrace of java worker (look for a
>> container called java-streaming)
>>  - file a support bug or stackoverflow question with jobid so that
>> Dataflow oncall can take a look.
>>
>> Raghu.
>>
>>
>> On Tue, Sep 11, 2018 at 12:10 PM Eduardo Soldera <
>> eduardo.soldera@arquivei.com.br> wrote:
>>
>>> Hi,
>>> We have a Apache Beam pipeline running in Google Dataflow using KafkaIO.
>>> Suddenly the pipeline stop fetching Kafka messages at all, as our other
>>> workers from other pipelines continued to get Kafka messages.
>>>
>>> At the moment it stopped we got these messages:
>>>
>>> I  [Consumer clientId=consumer-1, groupId=genericPipe] Error sending fetch request (sessionId=1396189203, epoch=2431598) to node 3: org.apache.kafka.common.errors.DisconnectException.
>>> I  [Consumer clientId=consumer-1, groupId=genericPipe] Group coordinator 10.0.52.70:9093 (id: 2147483646 rack: null) is unavailable or invalid, will attempt rediscovery
>>> I  [Consumer clientId=consumer-1, groupId=genericPipe] Discovered group coordinator 10.0.52.70:9093 (id: 2147483646 rack: null)
>>>
>>> And then the pipeline stopped reading the messages.
>>>
>>> This is the KafkaIO setup  we have:
>>>
>>> KafkaIO.read[String,String]()
>>>   .withBootstrapServers(server)
>>>   .withTopic(topic)
>>>   .withKeyDeserializer(classOf[StringDeserializer])
>>>   .withValueDeserializer(classOf[StringDeserializer])
>>>   .updateConsumerProperties(properties)
>>>   .commitOffsetsInFinalize()
>>>   .withoutMetadata()
>>>
>>>  Any help will be much appreciated.
>>>
>>> Best regards,
>>> --
>>> Eduardo Soldera Garcia
>>> Data Engineer
>>> (16) 3509-5555 | www.arquivei.com.br
>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>> [image: Arquivei.com.br – Inteligência em Notas Fiscais]
>>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>>> [image: Google seleciona Arquivei para imersão e mentoria no Vale do
>>> Silício]
>>> <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad>
>>> <https://www.facebook.com/arquivei>
>>> <https://www.linkedin.com/company/arquivei>
>>> <https://www.youtube.com/watch?v=sSUUKxbXnxk>
>>>
>>

-- 
Eduardo Soldera Garcia
Data Engineer
(16) 3509-5555 | www.arquivei.com.br
<https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
[image: Arquivei.com.br – Inteligência em Notas Fiscais]
<https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
[image: Google seleciona Arquivei para imersão e mentoria no Vale do
Silício]
<https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad>
<https://www.facebook.com/arquivei>
<https://www.linkedin.com/company/arquivei>
<https://www.youtube.com/watch?v=sSUUKxbXnxk>

Re: Problem with KafkaIO

Posted by Raghu Angadi <ra...@google.com>.
Specifically, I am interested if you have any thread running
'consumerPollLoop()' [1]. There should always be one (if a worker is
assigned one of the partitions). It is possible that KafkaClient itself is
hasn't recovered from the group coordinator error (though unlikely).

https://github.com/apache/beam/blob/master/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaUnboundedReader.java#L570

On Tue, Sep 11, 2018 at 12:31 PM Raghu Angadi <ra...@google.com> wrote:

> Hi Eduardo,
>
> In case of any error, the pipeline should keep on trying to fetch. I don't
> know about this particular error. Do you see any others afterwards in the
> log?
> Couple of things you could try if the logs are not useful :
>  - login to one of the VMs and get stacktrace of java worker (look for a
> container called java-streaming)
>  - file a support bug or stackoverflow question with jobid so that
> Dataflow oncall can take a look.
>
> Raghu.
>
>
> On Tue, Sep 11, 2018 at 12:10 PM Eduardo Soldera <
> eduardo.soldera@arquivei.com.br> wrote:
>
>> Hi,
>> We have a Apache Beam pipeline running in Google Dataflow using KafkaIO.
>> Suddenly the pipeline stop fetching Kafka messages at all, as our other
>> workers from other pipelines continued to get Kafka messages.
>>
>> At the moment it stopped we got these messages:
>>
>> I  [Consumer clientId=consumer-1, groupId=genericPipe] Error sending fetch request (sessionId=1396189203, epoch=2431598) to node 3: org.apache.kafka.common.errors.DisconnectException.
>> I  [Consumer clientId=consumer-1, groupId=genericPipe] Group coordinator 10.0.52.70:9093 (id: 2147483646 rack: null) is unavailable or invalid, will attempt rediscovery
>> I  [Consumer clientId=consumer-1, groupId=genericPipe] Discovered group coordinator 10.0.52.70:9093 (id: 2147483646 rack: null)
>>
>> And then the pipeline stopped reading the messages.
>>
>> This is the KafkaIO setup  we have:
>>
>> KafkaIO.read[String,String]()
>>   .withBootstrapServers(server)
>>   .withTopic(topic)
>>   .withKeyDeserializer(classOf[StringDeserializer])
>>   .withValueDeserializer(classOf[StringDeserializer])
>>   .updateConsumerProperties(properties)
>>   .commitOffsetsInFinalize()
>>   .withoutMetadata()
>>
>>  Any help will be much appreciated.
>>
>> Best regards,
>> --
>> Eduardo Soldera Garcia
>> Data Engineer
>> (16) 3509-5555 | www.arquivei.com.br
>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>> [image: Arquivei.com.br – Inteligência em Notas Fiscais]
>> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
>> [image: Google seleciona Arquivei para imersão e mentoria no Vale do
>> Silício]
>> <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad>
>> <https://www.facebook.com/arquivei>
>> <https://www.linkedin.com/company/arquivei>
>> <https://www.youtube.com/watch?v=sSUUKxbXnxk>
>>
>

Re: Problem with KafkaIO

Posted by Raghu Angadi <ra...@google.com>.
Hi Eduardo,

In case of any error, the pipeline should keep on trying to fetch. I don't
know about this particular error. Do you see any others afterwards in the
log?
Couple of things you could try if the logs are not useful :
 - login to one of the VMs and get stacktrace of java worker (look for a
container called java-streaming)
 - file a support bug or stackoverflow question with jobid so that Dataflow
oncall can take a look.

Raghu.


On Tue, Sep 11, 2018 at 12:10 PM Eduardo Soldera <
eduardo.soldera@arquivei.com.br> wrote:

> Hi,
> We have a Apache Beam pipeline running in Google Dataflow using KafkaIO.
> Suddenly the pipeline stop fetching Kafka messages at all, as our other
> workers from other pipelines continued to get Kafka messages.
>
> At the moment it stopped we got these messages:
>
> I  [Consumer clientId=consumer-1, groupId=genericPipe] Error sending fetch request (sessionId=1396189203, epoch=2431598) to node 3: org.apache.kafka.common.errors.DisconnectException.
> I  [Consumer clientId=consumer-1, groupId=genericPipe] Group coordinator 10.0.52.70:9093 (id: 2147483646 rack: null) is unavailable or invalid, will attempt rediscovery
> I  [Consumer clientId=consumer-1, groupId=genericPipe] Discovered group coordinator 10.0.52.70:9093 (id: 2147483646 rack: null)
>
> And then the pipeline stopped reading the messages.
>
> This is the KafkaIO setup  we have:
>
> KafkaIO.read[String,String]()
>   .withBootstrapServers(server)
>   .withTopic(topic)
>   .withKeyDeserializer(classOf[StringDeserializer])
>   .withValueDeserializer(classOf[StringDeserializer])
>   .updateConsumerProperties(properties)
>   .commitOffsetsInFinalize()
>   .withoutMetadata()
>
>  Any help will be much appreciated.
>
> Best regards,
> --
> Eduardo Soldera Garcia
> Data Engineer
> (16) 3509-5555 | www.arquivei.com.br
> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
> [image: Arquivei.com.br – Inteligência em Notas Fiscais]
> <https://arquivei.com.br/?utm_campaign=assinatura-email&utm_content=assinatura>
> [image: Google seleciona Arquivei para imersão e mentoria no Vale do
> Silício]
> <https://arquivei.com.br/blog/google-seleciona-arquivei/?utm_campaign=assinatura-email-launchpad&utm_content=assinatura-launchpad>
> <https://www.facebook.com/arquivei>
> <https://www.linkedin.com/company/arquivei>
> <https://www.youtube.com/watch?v=sSUUKxbXnxk>
>