Add Keys to Kafka Notifications - ATLAS-4323

[Adam Bellemare <ad...@gmail.com>]

HI All

The JIRA I filed: https://issues.apache.org/jira/browse/ATLAS-4323

I want to add Keys to the Kafka notifications. My current plan is to use
the GUID as the key for the kafka event. I noticed, however, that the V1
notifications appear to use LONG Ids.

I am evaluating a few options here:

1) Make it so that all V2 notifications have keys by default. This won't
affect any existing behaviour since null keys give no guarantees of
partition data locality. The downside here is that there would be a minor
increase in the amount of data sent over the wire (the GUID string).
2) Leave V1 notifications entirely alone, and only enable GUID String keys
on V2 notifications.
3) Enable notifications with LONG key V1, String GUID key V2.

The main issue is around serializing the arbitrary key type to Bytes. If we
decide to only go with String GUIDs then this is much easier (just for V2),
but I am curious as to if V1 users would also want this data locality?

The most robust solution is to just support keys of arbitrary types. If
this is the desired outcome I can do this as well, I just wanted to get a
feeling for the limits of scope of responsibility for V1 and V2
notifications.

Thanks
Adam

Re: Add Keys to Kafka Notifications - ATLAS-4323

[Adam Bellemare <ad...@gmail.com>]

Hi All

I have an initial PR for this issue.
https://github.com/apache/atlas/pull/139

The main points:
1) Only String key format is supported. It is not clear to me that we need
support for more than this, particularly if we're keying on GUID (which
seems to be the best candidate).
2) Previous compatibility remains unchanged, since null keys are randomly
distributed amongst partitions.
3) V1 notifications just use null.
4) V2 notifications use the GUID.

Please let me know if there are areas I need to address, make clearer,
change, etc. I am hesitant to support arbitrary key types that can't be
represented as a String without a strong use-case.





On Wed, Jun 2, 2021 at 3:45 PM Ashutosh Mestry <am...@cloudera.com.invalid>
wrote:

> Hi Adam
>
> It is OK to ignore V1 types. Majority of customer have moved to newer
> versions that use V2 messages.
>
> Best regards,
>
> ~ ashutosh
> Ashutosh Mestry<ma...@cloudera.com>
> .......
> No hurry, no pause. – Tim Ferriss, Life Hacker, Author
>
>
> From: Adam Bellemare <ad...@gmail.com>
> Date: Wednesday, June 2, 2021 at 11:04 AM
> To: dev@atlas.apache.org <de...@atlas.apache.org>
> Subject: Add Keys to Kafka Notifications - ATLAS-4323
> HI All
>
> The JIRA I filed: https://issues.apache.org/jira/browse/ATLAS-4323
>
> I want to add Keys to the Kafka notifications. My current plan is to use
> the GUID as the key for the kafka event. I noticed, however, that the V1
> notifications appear to use LONG Ids.
>
> I am evaluating a few options here:
>
> 1) Make it so that all V2 notifications have keys by default. This won't
> affect any existing behaviour since null keys give no guarantees of
> partition data locality. The downside here is that there would be a minor
> increase in the amount of data sent over the wire (the GUID string).
> 2) Leave V1 notifications entirely alone, and only enable GUID String keys
> on V2 notifications.
> 3) Enable notifications with LONG key V1, String GUID key V2.
>
> The main issue is around serializing the arbitrary key type to Bytes. If we
> decide to only go with String GUIDs then this is much easier (just for V2),
> but I am curious as to if V1 users would also want this data locality?
>
> The most robust solution is to just support keys of arbitrary types. If
> this is the desired outcome I can do this as well, I just wanted to get a
> feeling for the limits of scope of responsibility for V1 and V2
> notifications.
>
> Thanks
> Adam
>

Re: Add Keys to Kafka Notifications - ATLAS-4323

[Ashutosh Mestry <am...@cloudera.com.INVALID>]

Hi Adam

It is OK to ignore V1 types. Majority of customer have moved to newer versions that use V2 messages.

Best regards,

~ ashutosh
Ashutosh Mestry<ma...@cloudera.com>
.......
No hurry, no pause. – Tim Ferriss, Life Hacker, Author


From: Adam Bellemare <ad...@gmail.com>
Date: Wednesday, June 2, 2021 at 11:04 AM
To: dev@atlas.apache.org <de...@atlas.apache.org>
Subject: Add Keys to Kafka Notifications - ATLAS-4323
HI All

The JIRA I filed: https://issues.apache.org/jira/browse/ATLAS-4323

I want to add Keys to the Kafka notifications. My current plan is to use
the GUID as the key for the kafka event. I noticed, however, that the V1
notifications appear to use LONG Ids.

I am evaluating a few options here:

1) Make it so that all V2 notifications have keys by default. This won't
affect any existing behaviour since null keys give no guarantees of
partition data locality. The downside here is that there would be a minor
increase in the amount of data sent over the wire (the GUID string).
2) Leave V1 notifications entirely alone, and only enable GUID String keys
on V2 notifications.
3) Enable notifications with LONG key V1, String GUID key V2.

The main issue is around serializing the arbitrary key type to Bytes. If we
decide to only go with String GUIDs then this is much easier (just for V2),
but I am curious as to if V1 users would also want this data locality?

The most robust solution is to just support keys of arbitrary types. If
this is the desired outcome I can do this as well, I just wanted to get a
feeling for the limits of scope of responsibility for V1 and V2
notifications.

Thanks
Adam