You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@atlas.apache.org by Ashutosh Mestry <am...@hortonworks.com> on 2017/09/20 22:44:00 UTC
Re: Review Request 61807: ATLAS-2075: Kafka Messages: Split & Combine
Implementation
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61807/
-----------------------------------------------------------
(Updated Sept. 20, 2017, 10:43 p.m.)
Review request for atlas, Madhan Neethiraj and Nixon Rodrigues.
Changes
-------
Updates include:
- Implemented _VersionedMessageHeader_ for lighter deserialization.
- Removed dependence on version number to identify message types.
- Added/updated unit tests.
- Simplified serialization process.
Bugs: ATLAS-2075
https://issues.apache.org/jira/browse/ATLAS-2075
Repository: atlas
Description
-------
**Approach**
- Message size is checked. It is split into parts if it exceeds the size threshold.
- New message type _Split_ derived from _VersionedMessage_ holds information of the individual parts of the message.
- _MessageSplitter_ splits message into smaller packets and encloses them in a message of type _Split_.
- _MessageCombiner_ combines array of split messages.
- _VersionedMessageDeserializer_ is modified to support the new message type.
**Additional configuration item**
- _atlas.notification.message.size.threshold_ Needs to be added to Hive's atlas configuration as well as _Atlas_'s own _atlas-application.properties_.
- Default is set to a number closer to 1MB.
Diffs (updated)
-----
intg/src/main/java/org/apache/atlas/type/AtlasType.java ae7044d3
intg/src/test/java/org/apache/atlas/TestAtlasTypeJSONSerialize.java PRE-CREATION
notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaConsumer.java d3b4e49e
notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java 38889ef8
notification/src/main/java/org/apache/atlas/notification/AbstractNotification.java cb44fc68
notification/src/main/java/org/apache/atlas/notification/MessageCombiner.java PRE-CREATION
notification/src/main/java/org/apache/atlas/notification/MessageTransformer.java PRE-CREATION
notification/src/main/java/org/apache/atlas/notification/MessageVersion.java 6ef407ac
notification/src/main/java/org/apache/atlas/notification/VersionedMessage.java 1929eb46
notification/src/main/java/org/apache/atlas/notification/VersionedMessageDeserializer.java cc2099e7
notification/src/main/java/org/apache/atlas/notification/VersionedMessageHeader.java PRE-CREATION
notification/src/main/java/org/apache/atlas/notification/messages/Compressed.java PRE-CREATION
notification/src/main/java/org/apache/atlas/notification/messages/Split.java PRE-CREATION
notification/src/main/java/org/apache/atlas/utils/StringSplitterCombiner.java PRE-CREATION
notification/src/test/java/org/apache/atlas/hook/StringSplitCombineTest.java PRE-CREATION
notification/src/test/java/org/apache/atlas/notification/AbstractNotificationConsumerTest.java 3b2a093b
notification/src/test/java/org/apache/atlas/notification/AbstractNotificationTest.java 61107a9f
notification/src/test/java/org/apache/atlas/notification/MessageCombinerTest.java PRE-CREATION
notification/src/test/java/org/apache/atlas/notification/MessageSplitterTest.java PRE-CREATION
notification/src/test/java/org/apache/atlas/notification/SplitCombinerUtils.java PRE-CREATION
notification/src/test/java/org/apache/atlas/notification/entity/EntityMessageDeserializerTest.java be324277
notification/src/test/java/org/apache/atlas/notification/hook/HookMessageDeserializerTest.java 3724fd5f
notification/src/test/java/org/apache/atlas/notification/hook/HookNotificationTest.java dd3257e0
webapp/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java 2669c739
Diff: https://reviews.apache.org/r/61807/diff/2/
Changes: https://reviews.apache.org/r/61807/diff/1-2/
Testing
-------
**Unit tests**
- Added unit tests for new functionality.
- Refactored few of existing unit tests.
- Added tests to verify backward compatibility.
**Functional tests**
- Verified regular flow.
- Verified cases:
- Case where server is shutdown before all the parts of a message are committed.
Thanks,
Ashutosh Mestry
Re: Review Request 61807: ATLAS-2075: Kafka Messages: Split & Combine
Implementation
Posted by Ashutosh Mestry <am...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61807/
-----------------------------------------------------------
(Updated Sept. 25, 2017, 6:10 p.m.)
Review request for atlas, Madhan Neethiraj and Nixon Rodrigues.
Changes
-------
Updates include:
- Moved changes related to JSON compression to _JsonUtils_.
- Additional refactoring for simplicity.
Bugs: ATLAS-2075
https://issues.apache.org/jira/browse/ATLAS-2075
Repository: atlas
Description (updated)
-------
**Approach**
- Message size is checked. It is split into parts if it exceeds the size threshold.
- New message type _Split_ derived from _VersionedMessage_ holds information of the individual parts of the message.
- _MessageSplitter_ splits message into smaller packets and encloses them in a message of type _Split_.
- _MessageCombiner_ combines array of split messages.
- _VersionedMessageDeserializer_ is modified to support the new message type.
**Additional configuration item**
- _atlas.notification.message.size.threshold_ Needs to be added to Hive's atlas configuration as well as _Atlas_'s own _atlas-application.properties_.
- Default is set to a number closer to 1MB.
**For testing**
- Set threshold to a low number.
- Update _atlas-application.properties_:
- Add _atlas.notification.message.size.threshold_ to 1024.
Diffs (updated)
-----
notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaConsumer.java d3b4e49e
notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java 38889ef8
notification/src/main/java/org/apache/atlas/notification/AbstractNotification.java cb44fc68
notification/src/main/java/org/apache/atlas/notification/MessageCombiner.java PRE-CREATION
notification/src/main/java/org/apache/atlas/notification/MessageTransformer.java PRE-CREATION
notification/src/main/java/org/apache/atlas/notification/MessageVersion.java 6ef407ac
notification/src/main/java/org/apache/atlas/notification/VersionedMessage.java 1929eb46
notification/src/main/java/org/apache/atlas/notification/VersionedMessageDeserializer.java cc2099e7
notification/src/main/java/org/apache/atlas/notification/VersionedMessageHeader.java PRE-CREATION
notification/src/main/java/org/apache/atlas/notification/messages/Compressed.java PRE-CREATION
notification/src/main/java/org/apache/atlas/notification/messages/Split.java PRE-CREATION
notification/src/main/java/org/apache/atlas/utils/JsonUtils.java PRE-CREATION
notification/src/main/java/org/apache/atlas/utils/StringSplitterCombiner.java PRE-CREATION
notification/src/test/java/org/apache/atlas/hook/StringSplitCombineTest.java PRE-CREATION
notification/src/test/java/org/apache/atlas/notification/AbstractNotificationConsumerTest.java 3b2a093b
notification/src/test/java/org/apache/atlas/notification/AbstractNotificationTest.java 61107a9f
notification/src/test/java/org/apache/atlas/notification/MessageCombinerTest.java PRE-CREATION
notification/src/test/java/org/apache/atlas/notification/MessageSplitterTest.java PRE-CREATION
notification/src/test/java/org/apache/atlas/notification/SplitCombinerUtils.java PRE-CREATION
notification/src/test/java/org/apache/atlas/notification/entity/EntityMessageDeserializerTest.java be324277
notification/src/test/java/org/apache/atlas/notification/hook/HookMessageDeserializerTest.java 3724fd5f
notification/src/test/java/org/apache/atlas/notification/hook/HookNotificationTest.java dd3257e0
notification/src/test/java/org/apache/atlas/utils/TestJsonUtils.java PRE-CREATION
webapp/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java 2669c739
webapp/src/test/java/org/apache/atlas/util/JsonBenchmarkTest.java PRE-CREATION
Diff: https://reviews.apache.org/r/61807/diff/3/
Changes: https://reviews.apache.org/r/61807/diff/2-3/
Testing
-------
**Unit tests**
- Added unit tests for new functionality.
- Refactored few of existing unit tests.
- Added tests to verify backward compatibility.
**Functional tests**
- Verified regular flow.
- Verified cases:
- Case where server is shutdown before all the parts of a message are committed.
Thanks,
Ashutosh Mestry