答复: The pulsar go client missing data when transfer

["孙刘昌 (sunliuchang)" <su...@wifi.com>]

Send faild ,re-send

发件人: 孙刘昌(sunliuchang)
发送时间: 2019年12月9日 16:00
收件人: 'dev@pulsar.apache.org' <de...@pulsar.apache.org>; 'users@pulsar.apache.org' <us...@pulsar.apache.org>; 'commits@pulsar.apache.org' <co...@pulsar.apache.org>
主题: The pulsar go client missing data when transfer

Hi  Guys

Now I found one problem relate to pulsar go client

The whole process of mine is   csv file --> filebeat-->pulsar--> flume--> hive

At past time I verified the data count ,found that the data by this path missing about 0.46% compare with the path of logstash-->kafka .

No idea the reason why ,so I tried below steps

1 Reduce the amount of data from 100% to 50%, excluding data loss caused by too much pressure in the pulsar cluster, the data loss is still 0.46%
2 Reduce the amount of data from 50% to 10%, make the pulsar cluster stress-free, and the data loss is still 0.46%
3 Use bin / pulsar-perf to quantitatively send test test data to the pulsar cluster, which is 100% acceptable, eliminating the problems of the pulsar cluster
4 Quantitatively send test data to the pulsar cluster with filebeat. 99.6% can receive it (send 36077, receive 35091)
[cid:image001.png@01D5AEAC.3DFB3960]

I put the data file in the attachment. Please take a look at this problem. Now the loss rate is about five thousandths, which is a little too large.

Another thing to say, I use the asynchronous sending of the go client, which has been confirmed with @tuteng

Hopefully your response
本邮件及其附件含有保密信息并受法律保护。如果您错收了本邮件，请勿出于任何目的使用、阅读、复制、打印、存储或向任何人披露本邮件及其附件所含信息，并请您即刻将此误发情形通知发件人，并将本邮件从系统中永久删除，谢谢。