mixing assign and subscribe for a group

["Ranjan, Shailesh (Contractor - CB - Front Office L, Group IT)" <Sh...@lloydsbanking.com.INVALID>]

Classification: Public

Hi All,

We have a kafka cluster where we are using consumer.subscribe() for a group id. We also are trying to write a simple monitoring tool running as a separate instance on a different host than the consumers for checking the consumer offsets and for that we have used below code which as I  understand is mixing assign and subscribe here.

I wanted to understand what problems/issues will happen if I have the below code where I have assigned a partition in to a new instance (which would have been assigned to a consumer, what happens to that consumers ownership of the partition?) belonging to the same group but not calling poll() but just getting the current position.

Code snippet:

try (KafkaConsumer<String, String> consumer = createConsumer(consumerGroup)) {
   TopicPartition topicPartition = new TopicPartition(topic, Integer.parseInt(partition));
   consumer.assign(Collections.singletonList(topicPartition));
   return consumer.position(topicPartition);
} catch (Exception ex) {
   log.error("Error occurres when try to get consumer group position");
   return -1;
}


Thanks,
Shailesh
Lloyds Banking Group plc. Registered Office: The Mound, Edinburgh EH1 1YZ. Registered in Scotland no. SC95000. Telephone: 0131 225 4555.

Lloyds Bank plc. Registered Office: 25 Gresham Street, London EC2V 7HN. Registered in England and Wales no. 2065. Telephone 0207626 1500.

Bank of Scotland plc. Registered Office: The Mound, Edinburgh EH1 1YZ. Registered in Scotland no. SC327000. Telephone: 03457 801 801.

Lloyds Bank Corporate Markets plc. Registered office: 25 Gresham Street, London EC2V 7HN. Registered in England and Wales no. 10399850.

Scottish Widows Schroder Personal Wealth Limited. Registered Office: 25 Gresham Street, London EC2V 7HN. Registered in England and Wales no. 11722983.

Lloyds Bank plc, Bank of Scotland plc and Lloyds Bank Corporate Markets plc are authorised by the Prudential Regulation Authority and regulated by the Financial Conduct Authority and Prudential Regulation Authority.

Scottish Widows Schroder Personal Wealth Limited is authorised and regulated by the Financial Conduct Authority.

Lloyds Bank Corporate Markets Wertpapierhandelsbank GmbH is a wholly-owned subsidiary of Lloyds Bank Corporate Markets plc. Lloyds Bank Corporate Markets Wertpapierhandelsbank GmbH has its registered office at Thurn-und-Taxis Platz 6, 60313 Frankfurt, Germany. The company is registered with the Amtsgericht Frankfurt am Main, HRB 111650. Lloyds Bank Corporate Markets Wertpapierhandelsbank GmbH is supervised by the Bundesanstalt für Finanzdienstleistungsaufsicht.

Halifax is a division of Bank of Scotland plc.

HBOS plc. Registered Office: The Mound, Edinburgh EH1 1YZ. Registered in Scotland no. SC218813.



This e-mail (including any attachments) is private and confidential and may contain privileged material. If you have received this e-mail in error, please notify the sender and delete it (including any attachments) immediately. You must not copy, distribute, disclose or use any of the information in it or any attachments. Telephone calls may be monitored or recorded.

Re: mixing assign and subscribe for a group

[Ryanne Dolan <ry...@gmail.com>]

Shailesh, the users list may be a better place for this question.

Assign() and subscribe() cannot be used on the same Consumer. Moreover, an
assign()'d Consumer is not really a member of a group.

You should use AdminClient.listConsumerGroupOffsets() for this purpose.

Ryanne

On Thu, Aug 22, 2019 at 9:35 AM Ranjan, Shailesh (Contractor - CB - Front
Office L, Group IT) <Sh...@lloydsbanking.com.invalid> wrote:

> Classification: Public
>
> Hi All,
>
> We have a kafka cluster where we are using consumer.subscribe() for a
> group id. We also are trying to write a simple monitoring tool running as a
> separate instance on a different host than the consumers for checking the
> consumer offsets and for that we have used below code which as I
> understand is mixing assign and subscribe here.
>
> I wanted to understand what problems/issues will happen if I have the
> below code where I have assigned a partition in to a new instance (which
> would have been assigned to a consumer, what happens to that consumers
> ownership of the partition?) belonging to the same group but not calling
> poll() but just getting the current position.
>
> Code snippet:
>
> try (KafkaConsumer<String, String> consumer =
> createConsumer(consumerGroup)) {
>    TopicPartition topicPartition = new TopicPartition(topic,
> Integer.parseInt(partition));
>    consumer.assign(Collections.singletonList(topicPartition));
>    return consumer.position(topicPartition);
> } catch (Exception ex) {
>    log.error("Error occurres when try to get consumer group position");
>    return -1;
> }
>
>
> Thanks,
> Shailesh
> Lloyds Banking Group plc. Registered Office: The Mound, Edinburgh EH1 1YZ.
> Registered in Scotland no. SC95000. Telephone: 0131 225 4555.
>
> Lloyds Bank plc. Registered Office: 25 Gresham Street, London EC2V 7HN.
> Registered in England and Wales no. 2065. Telephone 0207626 1500.
>
> Bank of Scotland plc. Registered Office: The Mound, Edinburgh EH1 1YZ.
> Registered in Scotland no. SC327000. Telephone: 03457 801 801.
>
> Lloyds Bank Corporate Markets plc. Registered office: 25 Gresham Street,
> London EC2V 7HN. Registered in England and Wales no. 10399850.
>
> Scottish Widows Schroder Personal Wealth Limited. Registered Office: 25
> Gresham Street, London EC2V 7HN. Registered in England and Wales no.
> 11722983.
>
> Lloyds Bank plc, Bank of Scotland plc and Lloyds Bank Corporate Markets
> plc are authorised by the Prudential Regulation Authority and regulated by
> the Financial Conduct Authority and Prudential Regulation Authority.
>
> Scottish Widows Schroder Personal Wealth Limited is authorised and
> regulated by the Financial Conduct Authority.
>
> Lloyds Bank Corporate Markets Wertpapierhandelsbank GmbH is a wholly-owned
> subsidiary of Lloyds Bank Corporate Markets plc. Lloyds Bank Corporate
> Markets Wertpapierhandelsbank GmbH has its registered office at
> Thurn-und-Taxis Platz 6, 60313 Frankfurt, Germany. The company is
> registered with the Amtsgericht Frankfurt am Main, HRB 111650. Lloyds Bank
> Corporate Markets Wertpapierhandelsbank GmbH is supervised by the
> Bundesanstalt für Finanzdienstleistungsaufsicht.
>
> Halifax is a division of Bank of Scotland plc.
>
> HBOS plc. Registered Office: The Mound, Edinburgh EH1 1YZ. Registered in
> Scotland no. SC218813.
>
>
>
> This e-mail (including any attachments) is private and confidential and
> may contain privileged material. If you have received this e-mail in error,
> please notify the sender and delete it (including any attachments)
> immediately. You must not copy, distribute, disclose or use any of the
> information in it or any attachments. Telephone calls may be monitored or
> recorded.
>