You are viewing a plain text version of this content. The canonical link for it is here.

Posted to users@kafka.apache.org by Paul Jensen <pa...@spottradingllc.com> on 2016/02/18 15:34:21 UTC

Periodic Disconnects

Running on Centos with Kafka 0.8.2.1

I’m experiencing periodic disconnects.  I’ve enabled keepalives, though I don’t think that’s related to the issue.

The fact that this happens at granularity of 5 minutes and always at 45 seconds after is interesting.  This may be some internal network issue, but wanted to see if anyone else had encountered this.


[paul.jensen@SLX029 ceres (feature-CERES-1873-kafka-vols *%)]$ grep Disconnected logs/20160217.log | grep -v bootstrap

17:05:45.679 INFO  [Applications::Risk::Ceres::KafkaProducer::event_cb(176)] Kafka Log ERROR fac:FAIL): CHIVLXSTG073.spottrading.com:9092/1: Receive failed: Disconnected

18:05:45.674 INFO  [Applications::Risk::Ceres::KafkaProducer::event_cb(176)] Kafka Log ERROR fac:FAIL): CHIVLXSTG073.spottrading.com:9092/1: Receive failed: Disconnected

18:40:46.281 INFO  [Applications::Risk::Ceres::KafkaProducer::event_cb(176)] Kafka Log ERROR fac:FAIL): CHIVLXSTG073.spottrading.com:9092/1: Receive failed: Disconnected

18:55:45.672 INFO  [Applications::Risk::Ceres::KafkaProducer::event_cb(176)] Kafka Log ERROR fac:FAIL): CHIVLXSTG073.spottrading.com:9092/1: Receive failed: Disconnected

19:35:45.673 INFO  [Applications::Risk::Ceres::KafkaProducer::event_cb(176)] Kafka Log ERROR fac:FAIL): CHIVLXSTG073.spottrading.com:9092/1: Receive failed: Disconnected

19:50:45.672 INFO  [Applications::Risk::Ceres::KafkaProducer::event_cb(176)] Kafka Log ERROR fac:FAIL): CHIVLXSTG073.spottrading.com:9092/1: Receive failed: Disconnected

20:10:46.286 INFO  [Applications::Risk::Ceres::KafkaProducer::event_cb(176)] Kafka Log ERROR fac:FAIL): CHIVLXSTG073.spottrading.com:9092/1: Receive failed: Disconnected

20:50:45.672 INFO  [Applications::Risk::Ceres::KafkaProducer::event_cb(176)] Kafka Log ERROR fac:FAIL): CHIVLXSTG073.spottrading.com:9092/1: Receive failed: Disconnected

21:00:45.980 INFO  [Applications::Risk::Ceres::KafkaProducer::event_cb(176)] Kafka Log ERROR fac:FAIL): CHIVLXSTG073.spottrading.com:9092/1: Receive failed: Disconnected

22:15:45.678 INFO  [Applications::Risk::Ceres::KafkaProducer::event_cb(176)] Kafka Log ERROR fac:FAIL): CHIVLXSTG073.spottrading.com:9092/1: Receive failed: Disconnected

22:35:45.673 INFO  [Applications::Risk::Ceres::KafkaProducer::event_cb(176)] Kafka Log ERROR fac:FAIL): CHIVLXSTG073.spottrading.com:9092/1: Receive failed: Disconnected

23:35:45.676 INFO  [Applications::Risk::Ceres::KafkaProducer::event_cb(176)] Kafka Log ERROR fac:FAIL): CHIVLXSTG073.spottrading.com:9092/1: Receive failed: Disconnected

[paul.jensen@SLX029 ceres (feature-CERES-1873-kafka-vols *%)]$ grep Disconnected logs/20160217.log | grep bootstrap

17:15:45.984 INFO  [Applications::Risk::Ceres::KafkaProducer::event_cb(176)] Kafka Log ERROR fac:FAIL): chivlxstg073:9092/bootstrap: Receive failed: Disconnected

18:20:45.674 INFO  [Applications::Risk::Ceres::KafkaProducer::event_cb(176)] Kafka Log ERROR fac:FAIL): chivlxstg073:9092/bootstrap: Receive failed: Disconnected

18:30:45.983 INFO  [Applications::Risk::Ceres::KafkaProducer::event_cb(176)] Kafka Log ERROR fac:FAIL): chivlxstg073:9092/bootstrap: Receive failed: Disconnected

19:15:45.673 INFO  [Applications::Risk::Ceres::KafkaProducer::event_cb(176)] Kafka Log ERROR fac:FAIL): chivlxstg073:9092/bootstrap: Receive failed: Disconnected

20:00:45.987 INFO  [Applications::Risk::Ceres::KafkaProducer::event_cb(176)] Kafka Log ERROR fac:FAIL): chivlxstg073:9092/bootstrap: Receive failed: Disconnected

20:25:45.676 INFO  [Applications::Risk::Ceres::KafkaProducer::event_cb(176)] Kafka Log ERROR fac:FAIL): chivlxstg073:9092/bootstrap: Receive failed: Disconnected

20:35:45.982 INFO  [Applications::Risk::Ceres::KafkaProducer::event_cb(176)] Kafka Log ERROR fac:FAIL): chivlxstg073:9092/bootstrap: Receive failed: Disconnected

21:15:45.678 INFO  [Applications::Risk::Ceres::KafkaProducer::event_cb(176)] Kafka Log ERROR fac:FAIL): chivlxstg073:9092/bootstrap: Receive failed: Disconnected

21:35:45.672 INFO  [Applications::Risk::Ceres::KafkaProducer::event_cb(176)] Kafka Log ERROR fac:FAIL): chivlxstg073:9092/bootstrap: Receive failed: Disconnected

21:50:45.672 INFO  [Applications::Risk::Ceres::KafkaProducer::event_cb(176)] Kafka Log ERROR fac:FAIL): chivlxstg073:9092/bootstrap: Receive failed: Disconnected

22:00:45.978 INFO  [Applications::Risk::Ceres::KafkaProducer::event_cb(176)] Kafka Log ERROR fac:FAIL): chivlxstg073:9092/bootstrap: Receive failed: Disconnected

22:50:45.672 INFO  [Applications::Risk::Ceres::KafkaProducer::event_cb(176)] Kafka Log ERROR fac:FAIL): chivlxstg073:9092/bootstrap: Receive failed: Disconnected

23:50:45.674 INFO  [Applications::Risk::Ceres::KafkaProducer::event_cb(176)] Kafka Log ERROR fac:FAIL): chivlxstg073:9092/bootstrap: Receive failed: Disconnectedd


Thanks,

Paul




________________________________

    Paul Jensen
     Software Developer IV

     Spot Trading L.L.C
     440 South LaSalle St., Suite 2800
     Chicago, IL 60605
     Office: 312.362.4550
     Direct: 312-362-4711
     Fax: 312.362.4551
     paul.jensen@spottradingllc.com
     www.spottradingllc.com<http://www.spottradingllc.com/>

________________________________

The information contained in this message may be privileged and confidential and protected from disclosure. If the reader of this message is not the intended recipient, or an employee or agent responsible for delivering this message to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please notify us immediately by replying to the message and deleting it from your computer. Thank you. Spot Trading, LLC

Re: Periodic Disconnects

Posted by John Bickerstaff <jo...@johnbickerstaff.com>.

This may not be helpful, but the first thing I've learned to check in
similar situations is whether there is significant time-drift between VMs
and actual hardware.  Some combination of time-drift and a time-sensitive
security check could be causing this.  IIRC, CentOS has a funky issue with
getting time, even when properly set up...  At least I recall that issue on
my last gig with CentOS VMs.

HTH...

--JohnB

On Thu, Feb 18, 2016 at 7:34 AM, Paul Jensen <paul.jensen@spottradingllc.com
> wrote:

> Running on Centos with Kafka 0.8.2.1
>
> I’m experiencing periodic disconnects.  I’ve enabled keepalives, though I
> don’t think that’s related to the issue.
>
> The fact that this happens at granularity of 5 minutes and always at 45
> seconds after is interesting.  This may be some internal network issue, but
> wanted to see if anyone else had encountered this.
>
>
> [paul.jensen@SLX029 ceres (feature-CERES-1873-kafka-vols *%)]$ grep
> Disconnected logs/20160217.log | grep -v bootstrap
>
> 17:05:45.679 INFO
> [Applications::Risk::Ceres::KafkaProducer::event_cb(176)] Kafka Log ERROR
> fac:FAIL): CHIVLXSTG073.spottrading.com:9092/1: Receive failed:
> Disconnected
>
> 18:05:45.674 INFO
> [Applications::Risk::Ceres::KafkaProducer::event_cb(176)] Kafka Log ERROR
> fac:FAIL): CHIVLXSTG073.spottrading.com:9092/1: Receive failed:
> Disconnected
>
> 18:40:46.281 INFO
> [Applications::Risk::Ceres::KafkaProducer::event_cb(176)] Kafka Log ERROR
> fac:FAIL): CHIVLXSTG073.spottrading.com:9092/1: Receive failed:
> Disconnected
>
> 18:55:45.672 INFO
> [Applications::Risk::Ceres::KafkaProducer::event_cb(176)] Kafka Log ERROR
> fac:FAIL): CHIVLXSTG073.spottrading.com:9092/1: Receive failed:
> Disconnected
>
> 19:35:45.673 INFO
> [Applications::Risk::Ceres::KafkaProducer::event_cb(176)] Kafka Log ERROR
> fac:FAIL): CHIVLXSTG073.spottrading.com:9092/1: Receive failed:
> Disconnected
>
> 19:50:45.672 INFO
> [Applications::Risk::Ceres::KafkaProducer::event_cb(176)] Kafka Log ERROR
> fac:FAIL): CHIVLXSTG073.spottrading.com:9092/1: Receive failed:
> Disconnected
>
> 20:10:46.286 INFO
> [Applications::Risk::Ceres::KafkaProducer::event_cb(176)] Kafka Log ERROR
> fac:FAIL): CHIVLXSTG073.spottrading.com:9092/1: Receive failed:
> Disconnected
>
> 20:50:45.672 INFO
> [Applications::Risk::Ceres::KafkaProducer::event_cb(176)] Kafka Log ERROR
> fac:FAIL): CHIVLXSTG073.spottrading.com:9092/1: Receive failed:
> Disconnected
>
> 21:00:45.980 INFO
> [Applications::Risk::Ceres::KafkaProducer::event_cb(176)] Kafka Log ERROR
> fac:FAIL): CHIVLXSTG073.spottrading.com:9092/1: Receive failed:
> Disconnected
>
> 22:15:45.678 INFO
> [Applications::Risk::Ceres::KafkaProducer::event_cb(176)] Kafka Log ERROR
> fac:FAIL): CHIVLXSTG073.spottrading.com:9092/1: Receive failed:
> Disconnected
>
> 22:35:45.673 INFO
> [Applications::Risk::Ceres::KafkaProducer::event_cb(176)] Kafka Log ERROR
> fac:FAIL): CHIVLXSTG073.spottrading.com:9092/1: Receive failed:
> Disconnected
>
> 23:35:45.676 INFO
> [Applications::Risk::Ceres::KafkaProducer::event_cb(176)] Kafka Log ERROR
> fac:FAIL): CHIVLXSTG073.spottrading.com:9092/1: Receive failed:
> Disconnected
>
> [paul.jensen@SLX029 ceres (feature-CERES-1873-kafka-vols *%)]$ grep
> Disconnected logs/20160217.log | grep bootstrap
>
> 17:15:45.984 INFO
> [Applications::Risk::Ceres::KafkaProducer::event_cb(176)] Kafka Log ERROR
> fac:FAIL): chivlxstg073:9092/bootstrap: Receive failed: Disconnected
>
> 18:20:45.674 INFO
> [Applications::Risk::Ceres::KafkaProducer::event_cb(176)] Kafka Log ERROR
> fac:FAIL): chivlxstg073:9092/bootstrap: Receive failed: Disconnected
>
> 18:30:45.983 INFO
> [Applications::Risk::Ceres::KafkaProducer::event_cb(176)] Kafka Log ERROR
> fac:FAIL): chivlxstg073:9092/bootstrap: Receive failed: Disconnected
>
> 19:15:45.673 INFO
> [Applications::Risk::Ceres::KafkaProducer::event_cb(176)] Kafka Log ERROR
> fac:FAIL): chivlxstg073:9092/bootstrap: Receive failed: Disconnected
>
> 20:00:45.987 INFO
> [Applications::Risk::Ceres::KafkaProducer::event_cb(176)] Kafka Log ERROR
> fac:FAIL): chivlxstg073:9092/bootstrap: Receive failed: Disconnected
>
> 20:25:45.676 INFO
> [Applications::Risk::Ceres::KafkaProducer::event_cb(176)] Kafka Log ERROR
> fac:FAIL): chivlxstg073:9092/bootstrap: Receive failed: Disconnected
>
> 20:35:45.982 INFO
> [Applications::Risk::Ceres::KafkaProducer::event_cb(176)] Kafka Log ERROR
> fac:FAIL): chivlxstg073:9092/bootstrap: Receive failed: Disconnected
>
> 21:15:45.678 INFO
> [Applications::Risk::Ceres::KafkaProducer::event_cb(176)] Kafka Log ERROR
> fac:FAIL): chivlxstg073:9092/bootstrap: Receive failed: Disconnected
>
> 21:35:45.672 INFO
> [Applications::Risk::Ceres::KafkaProducer::event_cb(176)] Kafka Log ERROR
> fac:FAIL): chivlxstg073:9092/bootstrap: Receive failed: Disconnected
>
> 21:50:45.672 INFO
> [Applications::Risk::Ceres::KafkaProducer::event_cb(176)] Kafka Log ERROR
> fac:FAIL): chivlxstg073:9092/bootstrap: Receive failed: Disconnected
>
> 22:00:45.978 INFO
> [Applications::Risk::Ceres::KafkaProducer::event_cb(176)] Kafka Log ERROR
> fac:FAIL): chivlxstg073:9092/bootstrap: Receive failed: Disconnected
>
> 22:50:45.672 INFO
> [Applications::Risk::Ceres::KafkaProducer::event_cb(176)] Kafka Log ERROR
> fac:FAIL): chivlxstg073:9092/bootstrap: Receive failed: Disconnected
>
> 23:50:45.674 INFO
> [Applications::Risk::Ceres::KafkaProducer::event_cb(176)] Kafka Log ERROR
> fac:FAIL): chivlxstg073:9092/bootstrap: Receive failed: Disconnectedd
>
>
> Thanks,
>
> Paul
>
>
>
>
> ________________________________
>
>     Paul Jensen
>      Software Developer IV
>
>      Spot Trading L.L.C
>      440 South LaSalle St., Suite 2800
>      Chicago, IL 60605
>      Office: 312.362.4550
>      Direct: 312-362-4711
>      Fax: 312.362.4551
>      paul.jensen@spottradingllc.com
>      www.spottradingllc.com<http://www.spottradingllc.com/>
>
> ________________________________
>
> The information contained in this message may be privileged and
> confidential and protected from disclosure. If the reader of this message
> is not the intended recipient, or an employee or agent responsible for
> delivering this message to the intended recipient, you are hereby notified
> that any dissemination, distribution or copying of this communication is
> strictly prohibited. If you have received this communication in error,
> please notify us immediately by replying to the message and deleting it
> from your computer. Thank you. Spot Trading, LLC
>
>
>