You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by John Menke <jm...@numerix.com> on 2015/12/09 18:30:22 UTC

Match Producer and RecordMetadata with Consumer and ConsumerRecord

Can a correlationID be created from a ConsumerRecord that will allow for identification of the corresponding RecordMetaData instance that was returned from the Producer.send() method?

I am Looking at the JavaDocs and the Producer returns RecordMetadata which has the following signature:

RecordMetadata(TopicPartition topicPartition, long baseOffset, long relativeOffset) 
	
I am not sure if this can match to 
	
ConsumerRecord(java.lang.String topic, int partition, long offset, K key, V value)

Can we match producer requests to Consumer Records by matching these values?

RecordMetaData.TopicPartion.topic = ConsumerRecord.topic
RecordMetaData.TopicPartion.partition = ConsumerRecord.partition
RecordMetaData(baseOffset + relativeOffset) = ConsumerRecord.offset

In other words, can a CorrelationID be created from these values that will allow Consumers to link back to the Producer send() event.  (In the client calling code)




	


Re: Match Producer and RecordMetadata with Consumer and ConsumerRecord

Posted by John Menke <jm...@numerix.com>.
Thanks Ewen! This will help us.

________________________________________
From: Ewen Cheslack-Postava <ew...@confluent.io>
Sent: Thursday, December 10, 2015 1:50 AM
To: users@kafka.apache.org
Cc: Michael Pasacrita; Satish Sankaranarayanan
Subject: Re: Match Producer and RecordMetadata with Consumer and ConsumerRecord

John,

Your question was a bit confusing because CorrelationID has a specific
meaning in the Kafka protocols, but those are an implementation detail that
you, as a user of the API, should not need to worry about. CorrelationIDs
as defined by the protocol are not exposed to the user (and do not
correspond to individual Kafka messages -- they correspond to
request/response pairs in the protocol).

In Kafka you can uniquely identify a message by the (topic, partition,
offset) tuple. To match up the record passed to a producer and the record
received by a consumer, you would use, as you suggested in your first email:

Producer: RecordMetadata.topic(), RecordMetadata.partition(),
RecordMetadata.offset()
Consumer: ConsumerRecord.topic(), ConsumerRecord.partition(),
ConsumerRecord.offset()

-Ewen

On Wed, Dec 9, 2015 at 11:29 AM, John Menke <jm...@numerix.com> wrote:

> Gwen, thanks for the reply:
>
> Just to confirm - are we talking about the combination of the IDs that I
> layed out as being the "CorrelationID" ?
>
> I guess the alternative is to pass our own CorrelationID as part of the
> data.   Would you agree with this?
>
> I do see something at this address that talks about a "user defined"
> CorrelationID
>
>
> https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol#AGuideToTheKafkaProtocol-ProduceResponse
>
> CorrelationId:
>
> This is a user-supplied integer. It will be passed back in the response by
> the server, unmodified. It is useful for matching request and response
> between the client and server.
>
>
>
> -----Original Message-----
> From: Gwen Shapira [mailto:gwen@confluent.io]
> Sent: Wednesday, December 09, 2015 12:50 PM
> To: users@kafka.apache.org
> Subject: Re: Match Producer and RecordMetadata with Consumer and
> ConsumerRecord
>
> Correlation ID is for a request (i.e. separate ID for produce request and
> a fetch request), not a record. So it can't be used in the way you are
> trying to.
>
> On Wed, Dec 9, 2015 at 9:30 AM, John Menke <jm...@numerix.com> wrote:
>
> > Can a correlationID be created from a ConsumerRecord that will allow
> > for identification of the corresponding RecordMetaData instance that
> > was returned from the Producer.send() method?
> >
> > I am Looking at the JavaDocs and the Producer returns RecordMetadata
> > which has the following signature:
> >
> > RecordMetadata(TopicPartition topicPartition, long baseOffset, long
> > relativeOffset)
> >
> > I am not sure if this can match to
> >
> > ConsumerRecord(java.lang.String topic, int partition, long offset, K
> > key, V value)
> >
> > Can we match producer requests to Consumer Records by matching these
> > values?
> >
> > RecordMetaData.TopicPartion.topic = ConsumerRecord.topic
> > RecordMetaData.TopicPartion.partition = ConsumerRecord.partition
> > RecordMetaData(baseOffset + relativeOffset) = ConsumerRecord.offset
> >
> > In other words, can a CorrelationID be created from these values that
> > will allow Consumers to link back to the Producer send() event.  (In
> > the client calling code)
> >
> >
> >
> >
> >
> >
> >
>



--
Thanks,
Ewen

Re: Match Producer and RecordMetadata with Consumer and ConsumerRecord

Posted by Ewen Cheslack-Postava <ew...@confluent.io>.
John,

Your question was a bit confusing because CorrelationID has a specific
meaning in the Kafka protocols, but those are an implementation detail that
you, as a user of the API, should not need to worry about. CorrelationIDs
as defined by the protocol are not exposed to the user (and do not
correspond to individual Kafka messages -- they correspond to
request/response pairs in the protocol).

In Kafka you can uniquely identify a message by the (topic, partition,
offset) tuple. To match up the record passed to a producer and the record
received by a consumer, you would use, as you suggested in your first email:

Producer: RecordMetadata.topic(), RecordMetadata.partition(),
RecordMetadata.offset()
Consumer: ConsumerRecord.topic(), ConsumerRecord.partition(),
ConsumerRecord.offset()

-Ewen

On Wed, Dec 9, 2015 at 11:29 AM, John Menke <jm...@numerix.com> wrote:

> Gwen, thanks for the reply:
>
> Just to confirm - are we talking about the combination of the IDs that I
> layed out as being the "CorrelationID" ?
>
> I guess the alternative is to pass our own CorrelationID as part of the
> data.   Would you agree with this?
>
> I do see something at this address that talks about a "user defined"
> CorrelationID
>
>
> https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol#AGuideToTheKafkaProtocol-ProduceResponse
>
> CorrelationId:
>
> This is a user-supplied integer. It will be passed back in the response by
> the server, unmodified. It is useful for matching request and response
> between the client and server.
>
>
>
> -----Original Message-----
> From: Gwen Shapira [mailto:gwen@confluent.io]
> Sent: Wednesday, December 09, 2015 12:50 PM
> To: users@kafka.apache.org
> Subject: Re: Match Producer and RecordMetadata with Consumer and
> ConsumerRecord
>
> Correlation ID is for a request (i.e. separate ID for produce request and
> a fetch request), not a record. So it can't be used in the way you are
> trying to.
>
> On Wed, Dec 9, 2015 at 9:30 AM, John Menke <jm...@numerix.com> wrote:
>
> > Can a correlationID be created from a ConsumerRecord that will allow
> > for identification of the corresponding RecordMetaData instance that
> > was returned from the Producer.send() method?
> >
> > I am Looking at the JavaDocs and the Producer returns RecordMetadata
> > which has the following signature:
> >
> > RecordMetadata(TopicPartition topicPartition, long baseOffset, long
> > relativeOffset)
> >
> > I am not sure if this can match to
> >
> > ConsumerRecord(java.lang.String topic, int partition, long offset, K
> > key, V value)
> >
> > Can we match producer requests to Consumer Records by matching these
> > values?
> >
> > RecordMetaData.TopicPartion.topic = ConsumerRecord.topic
> > RecordMetaData.TopicPartion.partition = ConsumerRecord.partition
> > RecordMetaData(baseOffset + relativeOffset) = ConsumerRecord.offset
> >
> > In other words, can a CorrelationID be created from these values that
> > will allow Consumers to link back to the Producer send() event.  (In
> > the client calling code)
> >
> >
> >
> >
> >
> >
> >
>



-- 
Thanks,
Ewen

RE: Match Producer and RecordMetadata with Consumer and ConsumerRecord

Posted by John Menke <jm...@numerix.com>.
Gwen, thanks for the reply:

Just to confirm - are we talking about the combination of the IDs that I layed out as being the "CorrelationID" ?  

I guess the alternative is to pass our own CorrelationID as part of the data.   Would you agree with this?

I do see something at this address that talks about a "user defined" CorrelationID

https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol#AGuideToTheKafkaProtocol-ProduceResponse

CorrelationId:	

This is a user-supplied integer. It will be passed back in the response by the server, unmodified. It is useful for matching request and response between the client and server.



-----Original Message-----
From: Gwen Shapira [mailto:gwen@confluent.io] 
Sent: Wednesday, December 09, 2015 12:50 PM
To: users@kafka.apache.org
Subject: Re: Match Producer and RecordMetadata with Consumer and ConsumerRecord

Correlation ID is for a request (i.e. separate ID for produce request and a fetch request), not a record. So it can't be used in the way you are trying to.

On Wed, Dec 9, 2015 at 9:30 AM, John Menke <jm...@numerix.com> wrote:

> Can a correlationID be created from a ConsumerRecord that will allow 
> for identification of the corresponding RecordMetaData instance that 
> was returned from the Producer.send() method?
>
> I am Looking at the JavaDocs and the Producer returns RecordMetadata 
> which has the following signature:
>
> RecordMetadata(TopicPartition topicPartition, long baseOffset, long
> relativeOffset)
>
> I am not sure if this can match to
>
> ConsumerRecord(java.lang.String topic, int partition, long offset, K 
> key, V value)
>
> Can we match producer requests to Consumer Records by matching these 
> values?
>
> RecordMetaData.TopicPartion.topic = ConsumerRecord.topic 
> RecordMetaData.TopicPartion.partition = ConsumerRecord.partition 
> RecordMetaData(baseOffset + relativeOffset) = ConsumerRecord.offset
>
> In other words, can a CorrelationID be created from these values that 
> will allow Consumers to link back to the Producer send() event.  (In 
> the client calling code)
>
>
>
>
>
>
>

Re: Match Producer and RecordMetadata with Consumer and ConsumerRecord

Posted by Gwen Shapira <gw...@confluent.io>.
Correlation ID is for a request (i.e. separate ID for produce request and a
fetch request), not a record. So it can't be used in the way you are trying
to.

On Wed, Dec 9, 2015 at 9:30 AM, John Menke <jm...@numerix.com> wrote:

> Can a correlationID be created from a ConsumerRecord that will allow for
> identification of the corresponding RecordMetaData instance that was
> returned from the Producer.send() method?
>
> I am Looking at the JavaDocs and the Producer returns RecordMetadata which
> has the following signature:
>
> RecordMetadata(TopicPartition topicPartition, long baseOffset, long
> relativeOffset)
>
> I am not sure if this can match to
>
> ConsumerRecord(java.lang.String topic, int partition, long offset, K key,
> V value)
>
> Can we match producer requests to Consumer Records by matching these
> values?
>
> RecordMetaData.TopicPartion.topic = ConsumerRecord.topic
> RecordMetaData.TopicPartion.partition = ConsumerRecord.partition
> RecordMetaData(baseOffset + relativeOffset) = ConsumerRecord.offset
>
> In other words, can a CorrelationID be created from these values that will
> allow Consumers to link back to the Producer send() event.  (In the client
> calling code)
>
>
>
>
>
>
>