[jira] [Created] (KAFKA-8759) Message Order is reversed when client run behind a VPN

["M. Manna (JIRA)" <ji...@apache.org>]

M. Manna created KAFKA-8759:
-------------------------------

             Summary: Message Order is reversed when client run behind a VPN
                 Key: KAFKA-8759
                 URL: https://issues.apache.org/jira/browse/KAFKA-8759
             Project: Kafka
          Issue Type: Bug
          Components: clients
    Affects Versions: 2.2.0
            Reporter: M. Manna


We have noticed this behaviour whilst testing console producer against a kafka service installed on GCP. We have been using a fork from confluent Helm Chart.

[https://github.com/helm/charts/tree/master/incubator/kafka]

FYI - we've used cp 5.3.0 with Apache Kafka 2.3.0

Our VPN connection throughput was 1 mbps. Upon connecting to VPN, we opened a console producer client (2.2.0) with the following command:


{code:java}
kafka-console-producer.bat --topic some_topic --broker-list gcp_broker1:19092,gcp_broker2:19092,gcp_broker3:19092{code}
 

Similarly, we ran a consumer with the following command before publishing messages
{code:java}
kafka-console-consumer.bat --topic some_topic --bootstrap-server gcp_broker1:19092,gcp_broker2:19092,gcp_broker3:19092{code}

For producer console, we did receive a carat (>) prompt for publishing, so we entered messages:
{code:java}
>one
>two
>three
>{code}
After a while, it responded with NETWORK_EXCEPTION


{code:java}
[2019-08-02 11:17:19,690] WARN [Producer clientId=console-producer] Got error produce response with correlation id 8 on topic-partition some_topic-0, retrying (2 attempts left). Error: NETWORK_EXCEPTION (org.apache.kafka.clients.producer.internals.Sender){code}

We then hit "Enter" and received a carat (>) back


{code:java}
[2019-08-02 11:17:19,690] WARN [Producer clientId=console-producer] Got error produce response with correlation id 8 on topic-partition some_topic-0, retrying (2 attempts left). Error: NETWORK_EXCEPTION (org.apache.kafka.clients.producer.internals.Sender)

>{code}
 

Immediately after that, on consumer window, we received the following:
{code:java}
three
two
one{code}
 

We ran the same exercise from a regular network (wifi/lan) and didn't see this issue (i.e. works as described on Quickstart). 

This is slightly concerning for us since tunneling into a VPN shouldn't have any impact (or, should it) how kafka message protocol works over tcp. It seems that Kafka couldn't guarantee order of messages when network latency is involved. 

FYI 
1) We tried on VPN with --request_timeout_ms 120000 and still same results.
2) Our setup was 3 node (3 br, 3 zk) with every topic having 1 partition only (RF - 3).



 



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)