You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Matt Magoffin <ap...@msqr.us> on 2017/03/20 22:54:50 UTC
validate identity of producer in each record
Hello,
I am new to Kafka and am looking for a way for consumers to be able to identify the producer of each message in a topic. There are a large number of producers (lets say on the order of millions), and each producer would be connecting via SSL and using a unique client certificate. Essentially I'd like consumers to know the certificate of the producer of any given message.
The producer identify of the message must not be forgeable, i.e. producer A must not be allowed to generate records that appear to consumers to be from producer B.
Is it possible for Kafka to be configured to perform this type of identity validation? For example:
* reject records that contain a certificate identity that differs from the producer connection’s client certificate
* inject the producer connection’s certificate identity into each record
Or would a proxy application need to sit in front of Kafka to perform one of these functions?
Thank you in advance for offering any advice,
Matt
Re: validate identity of producer in each record
Posted by Matt Magoffin <ap...@msqr.us>.
OK, thank you for that. I looked at org.apache.kafka.connect.transforms.Transformation and org.apache.kafka.connect.source.SourceRecord, but am not discovering where the authenticated username (or Principal) might be available for the call to Transformation.apply()… or does Connect even support SSL and client certificate based authentication in the first place? If not, I think Connect may not be quite right for my use case.
— m@
> On 21/03/2017, at 4:59 PM, Hans Jespersen <ha...@confluent.io> wrote:
>
> Nothing on the broker today but if you use Kafka Connect API in 0.10.2 and above there is a pluggable interface called Transformations.
>
> See org.apache.kafka.connect.transforms in
> https://kafka.apache.org/0102/javadoc/index.html?org/apache/kafka/connect
>
> Source Connector transformations happen before storage in the Kafka log and Sink Connector transformations happen afterwards for consumers.
>
> -hans
Re: validate identity of producer in each record
Posted by Hans Jespersen <ha...@confluent.io>.
Nothing on the broker today but if you use Kafka Connect API in 0.10.2 and above there is a pluggable interface called Transformations.
See org.apache.kafka.connect.transforms in
https://kafka.apache.org/0102/javadoc/index.html?org/apache/kafka/connect
Source Connector transformations happen before storage in the Kafka log and Sink Connector transformations happen afterwards for consumers.
-hans
> On Mar 20, 2017, at 6:52 PM, Matt Magoffin <ap...@msqr.us> wrote:
>
> Thanks, Hans.
>
> Signing messages is a good idea. Other than that, is there possibly an extension point in Kafka itself on the receiving of records, before they are stored/distributed? I was thinking along the lines of
>
> org.apache.kafka.clients.producer.ProducerInterceptor
>
> but on the server side?
>
> — m@
>
>> On 21/03/2017, at 12:22 PM, Hans Jespersen <ha...@confluent.io> wrote:
>>
>> You can configure Kafka with ACLs that only allow certain users to
>> produce/consume to certain topics but if multiple producers are allowed to
>> produce to a shared topic then you cannot identify them without adding
>> something to the messages.
>>
>> For example, you can have each producer digitally sign (or encrypt) each
>> message and include the signature as a separate field (ie. separate from
>> the original message body). Then the consumers can independently check that
>> the signature is valid and that he message comes from a known/valid
>> publisher. This pattern is similar to how signed email messages work.
>>
>> -hans
>
Re: validate identity of producer in each record
Posted by Matt Magoffin <ap...@msqr.us>.
Thanks, Hans.
Signing messages is a good idea. Other than that, is there possibly an extension point in Kafka itself on the receiving of records, before they are stored/distributed? I was thinking along the lines of
org.apache.kafka.clients.producer.ProducerInterceptor
but on the server side?
— m@
> On 21/03/2017, at 12:22 PM, Hans Jespersen <ha...@confluent.io> wrote:
>
> You can configure Kafka with ACLs that only allow certain users to
> produce/consume to certain topics but if multiple producers are allowed to
> produce to a shared topic then you cannot identify them without adding
> something to the messages.
>
> For example, you can have each producer digitally sign (or encrypt) each
> message and include the signature as a separate field (ie. separate from
> the original message body). Then the consumers can independently check that
> the signature is valid and that he message comes from a known/valid
> publisher. This pattern is similar to how signed email messages work.
>
> -hans
Re: validate identity of producer in each record
Posted by Matt Magoffin <ma...@msqr.us>.
Thanks, Hans.
Signing messages is a good idea. Other than that, is there possibly an extension point in Kafka itself on the receiving of records, before they are stored/distributed? I was thinking along the lines of
org.apache.kafka.clients.producer.ProducerInterceptor
but on the server side?
— m@
> On 21/03/2017, at 12:22 PM, Hans Jespersen <ha...@confluent.io> wrote:
>
> You can configure Kafka with ACLs that only allow certain users to
> produce/consume to certain topics but if multiple producers are allowed to
> produce to a shared topic then you cannot identify them without adding
> something to the messages.
>
> For example, you can have each producer digitally sign (or encrypt) each
> message and include the signature as a separate field (ie. separate from
> the original message body). Then the consumers can independently check that
> the signature is valid and that he message comes from a known/valid
> publisher. This pattern is similar to how signed email messages work.
>
> -hans
Re: validate identity of producer in each record
Posted by Hans Jespersen <ha...@confluent.io>.
You can configure Kafka with ACLs that only allow certain users to
produce/consume to certain topics but if multiple producers are allowed to
produce to a shared topic then you cannot identify them without adding
something to the messages.
For example, you can have each producer digitally sign (or encrypt) each
message and include the signature as a separate field (ie. separate from
the original message body). Then the consumers can independently check that
the signature is valid and that he message comes from a known/valid
publisher. This pattern is similar to how signed email messages work.
-hans
/**
* Hans Jespersen, Principal Systems Engineer, Confluent Inc.
* hans@confluent.io (650)924-2670
*/
On Mon, Mar 20, 2017 at 3:54 PM, Matt Magoffin <ap...@msqr.us> wrote:
> Hello,
>
> I am new to Kafka and am looking for a way for consumers to be able to
> identify the producer of each message in a topic. There are a large number
> of producers (lets say on the order of millions), and each producer would
> be connecting via SSL and using a unique client certificate. Essentially
> I'd like consumers to know the certificate of the producer of any given
> message.
>
> The producer identify of the message must not be forgeable, i.e. producer
> A must not be allowed to generate records that appear to consumers to be
> from producer B.
>
> Is it possible for Kafka to be configured to perform this type of identity
> validation? For example:
>
> * reject records that contain a certificate identity that differs from the
> producer connection’s client certificate
> * inject the producer connection’s certificate identity into each record
>
> Or would a proxy application need to sit in front of Kafka to perform one
> of these functions?
>
> Thank you in advance for offering any advice,
> Matt
>
>