Kafka lost data after rolling restart.

[samuel chan <c_...@apple.com>]

The following is my kafka cluster setting:

Kafka version: 2.8.0-0.8.0
5 Zookeepers
16 kafka nodes
90 partitions for each of topic
RC: 3

We got a wired issue for Kafka restarting.

1) modify retention hours in server properties (from 72 hours to 168 hours)
2) Rolling restart Kafka nodes one by one.  Graceful shutdown each of node and wait 5 mins and then restart another node. 
3) Part of data before restarting has been deleted. For example, the retention days is 3 days data in Kafka , after restarted , past two days data has been deleted . So there was only remained three days ago data ( only one day data) . 
4) Data has been removed for every partition and replication. It looks kafka deleted instead of file carless deleted or file / folder coruption . 
4) Check offset by topic and partition , the offset has been changed after restart.


We checked server.log , kafka-request.log and state-change.log, there is no any error showing. 

The log size is huge that please feel free to let me know if I need to upload those log files. 

Thanks
Sam