You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by John Menke <jm...@numerix.com> on 2015/12/09 18:30:22 UTC
Match Producer and RecordMetadata with Consumer and ConsumerRecord
Can a correlationID be created from a ConsumerRecord that will allow for identification of the corresponding RecordMetaData instance that was returned from the Producer.send() method?
I am Looking at the JavaDocs and the Producer returns RecordMetadata which has the following signature:
RecordMetadata(TopicPartition topicPartition, long baseOffset, long relativeOffset)
I am not sure if this can match to
ConsumerRecord(java.lang.String topic, int partition, long offset, K key, V value)
Can we match producer requests to Consumer Records by matching these values?
RecordMetaData.TopicPartion.topic = ConsumerRecord.topic
RecordMetaData.TopicPartion.partition = ConsumerRecord.partition
RecordMetaData(baseOffset + relativeOffset) = ConsumerRecord.offset
In other words, can a CorrelationID be created from these values that will allow Consumers to link back to the Producer send() event. (In the client calling code)
Re: Match Producer and RecordMetadata with Consumer and
ConsumerRecord
Posted by John Menke <jm...@numerix.com>.
Thanks Ewen! This will help us.
________________________________________
From: Ewen Cheslack-Postava <ew...@confluent.io>
Sent: Thursday, December 10, 2015 1:50 AM
To: users@kafka.apache.org
Cc: Michael Pasacrita; Satish Sankaranarayanan
Subject: Re: Match Producer and RecordMetadata with Consumer and ConsumerRecord
John,
Your question was a bit confusing because CorrelationID has a specific
meaning in the Kafka protocols, but those are an implementation detail that
you, as a user of the API, should not need to worry about. CorrelationIDs
as defined by the protocol are not exposed to the user (and do not
correspond to individual Kafka messages -- they correspond to
request/response pairs in the protocol).
In Kafka you can uniquely identify a message by the (topic, partition,
offset) tuple. To match up the record passed to a producer and the record
received by a consumer, you would use, as you suggested in your first email:
Producer: RecordMetadata.topic(), RecordMetadata.partition(),
RecordMetadata.offset()
Consumer: ConsumerRecord.topic(), ConsumerRecord.partition(),
ConsumerRecord.offset()
-Ewen
On Wed, Dec 9, 2015 at 11:29 AM, John Menke <jm...@numerix.com> wrote:
> Gwen, thanks for the reply:
>
> Just to confirm - are we talking about the combination of the IDs that I
> layed out as being the "CorrelationID" ?
>
> I guess the alternative is to pass our own CorrelationID as part of the
> data. Would you agree with this?
>
> I do see something at this address that talks about a "user defined"
> CorrelationID
>
>
> https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol#AGuideToTheKafkaProtocol-ProduceResponse
>
> CorrelationId:
>
> This is a user-supplied integer. It will be passed back in the response by
> the server, unmodified. It is useful for matching request and response
> between the client and server.
>
>
>
> -----Original Message-----
> From: Gwen Shapira [mailto:gwen@confluent.io]
> Sent: Wednesday, December 09, 2015 12:50 PM
> To: users@kafka.apache.org
> Subject: Re: Match Producer and RecordMetadata with Consumer and
> ConsumerRecord
>
> Correlation ID is for a request (i.e. separate ID for produce request and
> a fetch request), not a record. So it can't be used in the way you are
> trying to.
>
> On Wed, Dec 9, 2015 at 9:30 AM, John Menke <jm...@numerix.com> wrote:
>
> > Can a correlationID be created from a ConsumerRecord that will allow
> > for identification of the corresponding RecordMetaData instance that
> > was returned from the Producer.send() method?
> >
> > I am Looking at the JavaDocs and the Producer returns RecordMetadata
> > which has the following signature:
> >
> > RecordMetadata(TopicPartition topicPartition, long baseOffset, long
> > relativeOffset)
> >
> > I am not sure if this can match to
> >
> > ConsumerRecord(java.lang.String topic, int partition, long offset, K
> > key, V value)
> >
> > Can we match producer requests to Consumer Records by matching these
> > values?
> >
> > RecordMetaData.TopicPartion.topic = ConsumerRecord.topic
> > RecordMetaData.TopicPartion.partition = ConsumerRecord.partition
> > RecordMetaData(baseOffset + relativeOffset) = ConsumerRecord.offset
> >
> > In other words, can a CorrelationID be created from these values that
> > will allow Consumers to link back to the Producer send() event. (In
> > the client calling code)
> >
> >
> >
> >
> >
> >
> >
>
--
Thanks,
Ewen
Re: Match Producer and RecordMetadata with Consumer and ConsumerRecord
Posted by Ewen Cheslack-Postava <ew...@confluent.io>.
John,
Your question was a bit confusing because CorrelationID has a specific
meaning in the Kafka protocols, but those are an implementation detail that
you, as a user of the API, should not need to worry about. CorrelationIDs
as defined by the protocol are not exposed to the user (and do not
correspond to individual Kafka messages -- they correspond to
request/response pairs in the protocol).
In Kafka you can uniquely identify a message by the (topic, partition,
offset) tuple. To match up the record passed to a producer and the record
received by a consumer, you would use, as you suggested in your first email:
Producer: RecordMetadata.topic(), RecordMetadata.partition(),
RecordMetadata.offset()
Consumer: ConsumerRecord.topic(), ConsumerRecord.partition(),
ConsumerRecord.offset()
-Ewen
On Wed, Dec 9, 2015 at 11:29 AM, John Menke <jm...@numerix.com> wrote:
> Gwen, thanks for the reply:
>
> Just to confirm - are we talking about the combination of the IDs that I
> layed out as being the "CorrelationID" ?
>
> I guess the alternative is to pass our own CorrelationID as part of the
> data. Would you agree with this?
>
> I do see something at this address that talks about a "user defined"
> CorrelationID
>
>
> https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol#AGuideToTheKafkaProtocol-ProduceResponse
>
> CorrelationId:
>
> This is a user-supplied integer. It will be passed back in the response by
> the server, unmodified. It is useful for matching request and response
> between the client and server.
>
>
>
> -----Original Message-----
> From: Gwen Shapira [mailto:gwen@confluent.io]
> Sent: Wednesday, December 09, 2015 12:50 PM
> To: users@kafka.apache.org
> Subject: Re: Match Producer and RecordMetadata with Consumer and
> ConsumerRecord
>
> Correlation ID is for a request (i.e. separate ID for produce request and
> a fetch request), not a record. So it can't be used in the way you are
> trying to.
>
> On Wed, Dec 9, 2015 at 9:30 AM, John Menke <jm...@numerix.com> wrote:
>
> > Can a correlationID be created from a ConsumerRecord that will allow
> > for identification of the corresponding RecordMetaData instance that
> > was returned from the Producer.send() method?
> >
> > I am Looking at the JavaDocs and the Producer returns RecordMetadata
> > which has the following signature:
> >
> > RecordMetadata(TopicPartition topicPartition, long baseOffset, long
> > relativeOffset)
> >
> > I am not sure if this can match to
> >
> > ConsumerRecord(java.lang.String topic, int partition, long offset, K
> > key, V value)
> >
> > Can we match producer requests to Consumer Records by matching these
> > values?
> >
> > RecordMetaData.TopicPartion.topic = ConsumerRecord.topic
> > RecordMetaData.TopicPartion.partition = ConsumerRecord.partition
> > RecordMetaData(baseOffset + relativeOffset) = ConsumerRecord.offset
> >
> > In other words, can a CorrelationID be created from these values that
> > will allow Consumers to link back to the Producer send() event. (In
> > the client calling code)
> >
> >
> >
> >
> >
> >
> >
>
--
Thanks,
Ewen
RE: Match Producer and RecordMetadata with Consumer and
ConsumerRecord
Posted by John Menke <jm...@numerix.com>.
Gwen, thanks for the reply:
Just to confirm - are we talking about the combination of the IDs that I layed out as being the "CorrelationID" ?
I guess the alternative is to pass our own CorrelationID as part of the data. Would you agree with this?
I do see something at this address that talks about a "user defined" CorrelationID
https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol#AGuideToTheKafkaProtocol-ProduceResponse
CorrelationId:
This is a user-supplied integer. It will be passed back in the response by the server, unmodified. It is useful for matching request and response between the client and server.
-----Original Message-----
From: Gwen Shapira [mailto:gwen@confluent.io]
Sent: Wednesday, December 09, 2015 12:50 PM
To: users@kafka.apache.org
Subject: Re: Match Producer and RecordMetadata with Consumer and ConsumerRecord
Correlation ID is for a request (i.e. separate ID for produce request and a fetch request), not a record. So it can't be used in the way you are trying to.
On Wed, Dec 9, 2015 at 9:30 AM, John Menke <jm...@numerix.com> wrote:
> Can a correlationID be created from a ConsumerRecord that will allow
> for identification of the corresponding RecordMetaData instance that
> was returned from the Producer.send() method?
>
> I am Looking at the JavaDocs and the Producer returns RecordMetadata
> which has the following signature:
>
> RecordMetadata(TopicPartition topicPartition, long baseOffset, long
> relativeOffset)
>
> I am not sure if this can match to
>
> ConsumerRecord(java.lang.String topic, int partition, long offset, K
> key, V value)
>
> Can we match producer requests to Consumer Records by matching these
> values?
>
> RecordMetaData.TopicPartion.topic = ConsumerRecord.topic
> RecordMetaData.TopicPartion.partition = ConsumerRecord.partition
> RecordMetaData(baseOffset + relativeOffset) = ConsumerRecord.offset
>
> In other words, can a CorrelationID be created from these values that
> will allow Consumers to link back to the Producer send() event. (In
> the client calling code)
>
>
>
>
>
>
>
Re: Match Producer and RecordMetadata with Consumer and ConsumerRecord
Posted by Gwen Shapira <gw...@confluent.io>.
Correlation ID is for a request (i.e. separate ID for produce request and a
fetch request), not a record. So it can't be used in the way you are trying
to.
On Wed, Dec 9, 2015 at 9:30 AM, John Menke <jm...@numerix.com> wrote:
> Can a correlationID be created from a ConsumerRecord that will allow for
> identification of the corresponding RecordMetaData instance that was
> returned from the Producer.send() method?
>
> I am Looking at the JavaDocs and the Producer returns RecordMetadata which
> has the following signature:
>
> RecordMetadata(TopicPartition topicPartition, long baseOffset, long
> relativeOffset)
>
> I am not sure if this can match to
>
> ConsumerRecord(java.lang.String topic, int partition, long offset, K key,
> V value)
>
> Can we match producer requests to Consumer Records by matching these
> values?
>
> RecordMetaData.TopicPartion.topic = ConsumerRecord.topic
> RecordMetaData.TopicPartion.partition = ConsumerRecord.partition
> RecordMetaData(baseOffset + relativeOffset) = ConsumerRecord.offset
>
> In other words, can a CorrelationID be created from these values that will
> allow Consumers to link back to the Producer send() event. (In the client
> calling code)
>
>
>
>
>
>
>