Message offset backtracked with rolling downgrade

[Helen Weng <he...@gmail.com>]

Hello,

We recently tried doing a rolling upgrade from 0.9.0 to 1.0.0. We had
upgraded 2 brokers when we began seeing decode errors with our python
client in boto3 trying to decode ProduceRequest_v2. But other than a large
spew of error messages, we were still consuming and producing normally.
When we tried to downgrade, however, the messageoffset jumped back to when
we initially upgraded. Has anyone seen this kind of behavior before, and
what did you do to remedy it?

Thanks for the help.

Helen Weng