You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Soman Ullah <so...@gmail.com> on 2019/12/07 14:37:59 UTC
Kafka transactions not working in 2.3.1
Hello,
I recently upgraded our kafka cluster from Kafka version 0.10.1 to 2.3.1
I've confirmed the version has updated using the following command:
/home/wrte/install/kafka/bin/kafka-topics.sh --version
2.3.1.1_212.11 (Commit:8c923fb4c62a38ae)
However I'm unable to use the transactional features as the producer fails
with
Caused by: org.apache.kafka.common.errors.UnsupportedVersionException:
The broker does not support ADD_PARTITIONS_TO_TXN
When I double checked the broker api version, it says transactions aren't
supported:
/home/wrte/install/kafka/bin/kafka-broker-api-versions.sh
--bootstrap-server localhost:4443 | grep -i txn
EndTxn(26): UNSUPPORTED,
WriteTxnMarkers(27): UNSUPPORTED,
TxnOffsetCommit(28): UNSUPPORTED,
AddPartitionsToTxn(24): UNSUPPORTED,
AddOffsetsToTxn(25): UNSUPPORTED,
I can also see that a new topic: __transaction_state was created on the
broker. How can I fix this and start using transactions?
Thanks,
Soman
Re: Kafka transactions not working in 2.3.1
Posted by Jonathan Santilli <jo...@gmail.com>.
Hello Soman,
again, hard to tell, this is what the docs say:
"...if you are upgrading from a version prior to 0.11.0.x, then
CURRENT_MESSAGE_FORMAT_VERSION should be set to match CURRENT_KAFKA_VERSION.
"
also:
"...Once the brokers begin using the latest protocol version, it will no
longer be possible to downgrade the cluster to an older version."
So, I guess you have tested or are testing in a DEV environment before
going to prod, if so, and you have not restarted the brokers with the
newest protocol version, I think it could work.
Again, I have not more info than the one you are providing, this is my leap
of faith.
Cheers!
--
Jonathan
On Mon, Dec 9, 2019 at 12:30 AM Soman Ullah <so...@gmail.com> wrote:
> Hello Jonathan,
> I believe I did not set *log.message.format.version* to the current version
> before I upgraded to 2.3. However, I did set inter.broker.protocol.version
> though. Would reverting back to 0.10.1 and upgrading again with
> *log.message.format.version
> *work?
>
> Thanks,
> Soman
>
> On Sun, Dec 8, 2019 at 1:40 PM Jonathan Santilli <
> jonathansantilli@gmail.com>
> wrote:
>
> > Hard to tell without logs and more context, I mean, Is a big jump from
> > 0.10.1 to 2.3.1.
> > Did you follow all the instructions/comments? as in the upgrade
> > documentation https://kafka.apache.org/documentation/#upgrade_2_3_0?
> >
> > Cheers!
> > --
> > Jonathan
> >
> >
> >
> >
> > On Sun, Dec 8, 2019 at 3:18 PM Soman Ullah <so...@gmail.com> wrote:
> >
> > > Hello Jonathan,
> > > I've installed the same version on all the brokers. The brokers were
> > > restarted as well. Any suggestions on how I can fix this?
> > >
> > > Thanks,
> > > Soman
> > >
> > > On Sat, Dec 7, 2019 at 9:17 AM Jonathan Santilli <
> > > jonathansantilli@gmail.com>
> > > wrote:
> > >
> > > > Hello,
> > > >
> > > > have you ensured you have installed the same version in all brokers?
> > Did
> > > > you restart all brokers after the update as indicated in the rolling
> > > > upgrade instructions?
> > > >
> > > > Cheers!
> > > >
> > > > On Sat, Dec 7, 2019, 2:38 PM Soman Ullah <so...@gmail.com>
> wrote:
> > > >
> > > > > Hello,
> > > > >
> > > > > I recently upgraded our kafka cluster from Kafka version 0.10.1 to
> > > 2.3.1
> > > > > I've confirmed the version has updated using the following command:
> > > > >
> > > > > /home/wrte/install/kafka/bin/kafka-topics.sh --version
> > > > > 2.3.1.1_212.11 (Commit:8c923fb4c62a38ae)
> > > > >
> > > > > However I'm unable to use the transactional features as the
> producer
> > > > fails
> > > > > with
> > > > >
> > > > > Caused by:
> > org.apache.kafka.common.errors.UnsupportedVersionException:
> > > > > The broker does not support ADD_PARTITIONS_TO_TXN
> > > > >
> > > > > When I double checked the broker api version, it says transactions
> > > aren't
> > > > > supported:
> > > > >
> > > > > /home/wrte/install/kafka/bin/kafka-broker-api-versions.sh
> > > > > --bootstrap-server localhost:4443 | grep -i txn
> > > > > EndTxn(26): UNSUPPORTED,
> > > > > WriteTxnMarkers(27): UNSUPPORTED,
> > > > > TxnOffsetCommit(28): UNSUPPORTED,
> > > > > AddPartitionsToTxn(24): UNSUPPORTED,
> > > > > AddOffsetsToTxn(25): UNSUPPORTED,
> > > > >
> > > > > I can also see that a new topic: __transaction_state was created on
> > the
> > > > > broker. How can I fix this and start using transactions?
> > > > >
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Soman
> > > > >
> > > >
> > >
> >
> >
> > --
> > Santilli Jonathan
> >
>
--
Santilli Jonathan
Re: Kafka transactions not working in 2.3.1
Posted by Soman Ullah <so...@gmail.com>.
Hello Jonathan,
I believe I did not set *log.message.format.version* to the current version
before I upgraded to 2.3. However, I did set inter.broker.protocol.version
though. Would reverting back to 0.10.1 and upgrading again with
*log.message.format.version
*work?
Thanks,
Soman
On Sun, Dec 8, 2019 at 1:40 PM Jonathan Santilli <jo...@gmail.com>
wrote:
> Hard to tell without logs and more context, I mean, Is a big jump from
> 0.10.1 to 2.3.1.
> Did you follow all the instructions/comments? as in the upgrade
> documentation https://kafka.apache.org/documentation/#upgrade_2_3_0?
>
> Cheers!
> --
> Jonathan
>
>
>
>
> On Sun, Dec 8, 2019 at 3:18 PM Soman Ullah <so...@gmail.com> wrote:
>
> > Hello Jonathan,
> > I've installed the same version on all the brokers. The brokers were
> > restarted as well. Any suggestions on how I can fix this?
> >
> > Thanks,
> > Soman
> >
> > On Sat, Dec 7, 2019 at 9:17 AM Jonathan Santilli <
> > jonathansantilli@gmail.com>
> > wrote:
> >
> > > Hello,
> > >
> > > have you ensured you have installed the same version in all brokers?
> Did
> > > you restart all brokers after the update as indicated in the rolling
> > > upgrade instructions?
> > >
> > > Cheers!
> > >
> > > On Sat, Dec 7, 2019, 2:38 PM Soman Ullah <so...@gmail.com> wrote:
> > >
> > > > Hello,
> > > >
> > > > I recently upgraded our kafka cluster from Kafka version 0.10.1 to
> > 2.3.1
> > > > I've confirmed the version has updated using the following command:
> > > >
> > > > /home/wrte/install/kafka/bin/kafka-topics.sh --version
> > > > 2.3.1.1_212.11 (Commit:8c923fb4c62a38ae)
> > > >
> > > > However I'm unable to use the transactional features as the producer
> > > fails
> > > > with
> > > >
> > > > Caused by:
> org.apache.kafka.common.errors.UnsupportedVersionException:
> > > > The broker does not support ADD_PARTITIONS_TO_TXN
> > > >
> > > > When I double checked the broker api version, it says transactions
> > aren't
> > > > supported:
> > > >
> > > > /home/wrte/install/kafka/bin/kafka-broker-api-versions.sh
> > > > --bootstrap-server localhost:4443 | grep -i txn
> > > > EndTxn(26): UNSUPPORTED,
> > > > WriteTxnMarkers(27): UNSUPPORTED,
> > > > TxnOffsetCommit(28): UNSUPPORTED,
> > > > AddPartitionsToTxn(24): UNSUPPORTED,
> > > > AddOffsetsToTxn(25): UNSUPPORTED,
> > > >
> > > > I can also see that a new topic: __transaction_state was created on
> the
> > > > broker. How can I fix this and start using transactions?
> > > >
> > > >
> > > > Thanks,
> > > >
> > > > Soman
> > > >
> > >
> >
>
>
> --
> Santilli Jonathan
>
Re: Kafka transactions not working in 2.3.1
Posted by Jonathan Santilli <jo...@gmail.com>.
Hard to tell without logs and more context, I mean, Is a big jump from
0.10.1 to 2.3.1.
Did you follow all the instructions/comments? as in the upgrade
documentation https://kafka.apache.org/documentation/#upgrade_2_3_0?
Cheers!
--
Jonathan
On Sun, Dec 8, 2019 at 3:18 PM Soman Ullah <so...@gmail.com> wrote:
> Hello Jonathan,
> I've installed the same version on all the brokers. The brokers were
> restarted as well. Any suggestions on how I can fix this?
>
> Thanks,
> Soman
>
> On Sat, Dec 7, 2019 at 9:17 AM Jonathan Santilli <
> jonathansantilli@gmail.com>
> wrote:
>
> > Hello,
> >
> > have you ensured you have installed the same version in all brokers? Did
> > you restart all brokers after the update as indicated in the rolling
> > upgrade instructions?
> >
> > Cheers!
> >
> > On Sat, Dec 7, 2019, 2:38 PM Soman Ullah <so...@gmail.com> wrote:
> >
> > > Hello,
> > >
> > > I recently upgraded our kafka cluster from Kafka version 0.10.1 to
> 2.3.1
> > > I've confirmed the version has updated using the following command:
> > >
> > > /home/wrte/install/kafka/bin/kafka-topics.sh --version
> > > 2.3.1.1_212.11 (Commit:8c923fb4c62a38ae)
> > >
> > > However I'm unable to use the transactional features as the producer
> > fails
> > > with
> > >
> > > Caused by: org.apache.kafka.common.errors.UnsupportedVersionException:
> > > The broker does not support ADD_PARTITIONS_TO_TXN
> > >
> > > When I double checked the broker api version, it says transactions
> aren't
> > > supported:
> > >
> > > /home/wrte/install/kafka/bin/kafka-broker-api-versions.sh
> > > --bootstrap-server localhost:4443 | grep -i txn
> > > EndTxn(26): UNSUPPORTED,
> > > WriteTxnMarkers(27): UNSUPPORTED,
> > > TxnOffsetCommit(28): UNSUPPORTED,
> > > AddPartitionsToTxn(24): UNSUPPORTED,
> > > AddOffsetsToTxn(25): UNSUPPORTED,
> > >
> > > I can also see that a new topic: __transaction_state was created on the
> > > broker. How can I fix this and start using transactions?
> > >
> > >
> > > Thanks,
> > >
> > > Soman
> > >
> >
>
--
Santilli Jonathan
Re: Kafka transactions not working in 2.3.1
Posted by Soman Ullah <so...@gmail.com>.
Hello Jonathan,
I've installed the same version on all the brokers. The brokers were
restarted as well. Any suggestions on how I can fix this?
Thanks,
Soman
On Sat, Dec 7, 2019 at 9:17 AM Jonathan Santilli <jo...@gmail.com>
wrote:
> Hello,
>
> have you ensured you have installed the same version in all brokers? Did
> you restart all brokers after the update as indicated in the rolling
> upgrade instructions?
>
> Cheers!
>
> On Sat, Dec 7, 2019, 2:38 PM Soman Ullah <so...@gmail.com> wrote:
>
> > Hello,
> >
> > I recently upgraded our kafka cluster from Kafka version 0.10.1 to 2.3.1
> > I've confirmed the version has updated using the following command:
> >
> > /home/wrte/install/kafka/bin/kafka-topics.sh --version
> > 2.3.1.1_212.11 (Commit:8c923fb4c62a38ae)
> >
> > However I'm unable to use the transactional features as the producer
> fails
> > with
> >
> > Caused by: org.apache.kafka.common.errors.UnsupportedVersionException:
> > The broker does not support ADD_PARTITIONS_TO_TXN
> >
> > When I double checked the broker api version, it says transactions aren't
> > supported:
> >
> > /home/wrte/install/kafka/bin/kafka-broker-api-versions.sh
> > --bootstrap-server localhost:4443 | grep -i txn
> > EndTxn(26): UNSUPPORTED,
> > WriteTxnMarkers(27): UNSUPPORTED,
> > TxnOffsetCommit(28): UNSUPPORTED,
> > AddPartitionsToTxn(24): UNSUPPORTED,
> > AddOffsetsToTxn(25): UNSUPPORTED,
> >
> > I can also see that a new topic: __transaction_state was created on the
> > broker. How can I fix this and start using transactions?
> >
> >
> > Thanks,
> >
> > Soman
> >
>
Re: Kafka transactions not working in 2.3.1
Posted by Jonathan Santilli <jo...@gmail.com>.
Hello,
have you ensured you have installed the same version in all brokers? Did
you restart all brokers after the update as indicated in the rolling
upgrade instructions?
Cheers!
On Sat, Dec 7, 2019, 2:38 PM Soman Ullah <so...@gmail.com> wrote:
> Hello,
>
> I recently upgraded our kafka cluster from Kafka version 0.10.1 to 2.3.1
> I've confirmed the version has updated using the following command:
>
> /home/wrte/install/kafka/bin/kafka-topics.sh --version
> 2.3.1.1_212.11 (Commit:8c923fb4c62a38ae)
>
> However I'm unable to use the transactional features as the producer fails
> with
>
> Caused by: org.apache.kafka.common.errors.UnsupportedVersionException:
> The broker does not support ADD_PARTITIONS_TO_TXN
>
> When I double checked the broker api version, it says transactions aren't
> supported:
>
> /home/wrte/install/kafka/bin/kafka-broker-api-versions.sh
> --bootstrap-server localhost:4443 | grep -i txn
> EndTxn(26): UNSUPPORTED,
> WriteTxnMarkers(27): UNSUPPORTED,
> TxnOffsetCommit(28): UNSUPPORTED,
> AddPartitionsToTxn(24): UNSUPPORTED,
> AddOffsetsToTxn(25): UNSUPPORTED,
>
> I can also see that a new topic: __transaction_state was created on the
> broker. How can I fix this and start using transactions?
>
>
> Thanks,
>
> Soman
>