You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@atlas.apache.org by Ashutosh Mestry <am...@hortonworks.com> on 2018/05/01 23:51:50 UTC
Review Request 66899: Large Notification Messages: Avoid Duplicate
Processing
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66899/
-----------------------------------------------------------
Review request for atlas and Madhan Neethiraj.
Bugs: ATLAS-2634
https://issues.apache.org/jira/browse/ATLAS-2634
Repository: atlas
Description
-------
**Approach**
Modified _AtlasNotificationMessageDeserializer_:
- Large message is defined as message that has been subjected to split or compression. The deserialization follows a separate path to accumulate splits or decompress compressed messages.
- Algorithm:
- Added internal class that keep a map of message ids and number of times the message was fetched.
- Large messages are added to this map.
- Everytime the message is fetch, the fetch count is incremented.
- For message that is not large, fetch count of 0 is returned.
- If a fetch count is greater than 1, which is that the message is retrieved more than once, commit on Kafka is called immediately. This message is not handed to _NotificationHookConsumer_ for further processing.
Modified _AtlasKafkaConsumer_: Handle the new fetch logic.
Modified _NotificationHookConsumer_: Extracted _ATLAS_HOOK_ topic name.
**Additional Refactoring**
None.
Diffs
-----
notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaConsumer.java fd0c4e48a
notification/src/main/java/org/apache/atlas/notification/AbstractNotificationConsumer.java c3940ceb4
notification/src/main/java/org/apache/atlas/notification/AtlasNotificationMessageDeserializer.java 3264e264c
notification/src/test/java/org/apache/atlas/notification/AbstractNotificationConsumerTest.java c9638309d
notification/src/test/java/org/apache/atlas/notification/hook/HookNotificationDeserializerTest.java d04817024
webapp/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java 7a4596a7e
Diff: https://reviews.apache.org/r/66899/diff/1/
Testing
-------
**Unit tests**
Additional unit tests to test specific scenarios.
Thanks,
Ashutosh Mestry
Re: Review Request 66899: Large Notification Messages: Avoid Duplicate
Processing
Posted by Madhan Neethiraj <ma...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66899/#review202459
-----------------------------------------------------------
Ship it!
Ship It!
- Madhan Neethiraj
On May 4, 2018, 4:33 p.m., Ashutosh Mestry wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66899/
> -----------------------------------------------------------
>
> (Updated May 4, 2018, 4:33 p.m.)
>
>
> Review request for atlas and Madhan Neethiraj.
>
>
> Bugs: ATLAS-2634
> https://issues.apache.org/jira/browse/ATLAS-2634
>
>
> Repository: atlas
>
>
> Description
> -------
>
> **Approach**
> Simplified approach.
>
> **Algorithm**
> - Modified: _NotificationHookConsumer_:
> - when commit() fails, record current offset of the message.
> - on receiving a message
> - if _offset_ is same as that of the received message, skip processing the message, go ahead and commit.
> - reset recorded offset.
> Added appropriate log messages.
>
> **Additional Refactoring**
> None.
>
>
> Diffs
> -----
>
> notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java 80dc51423
> webapp/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java 7a4596a7e
> webapp/src/test/java/org/apache/atlas/notification/NotificationHookConsumerKafkaTest.java d2b3dfdab
>
>
> Diff: https://reviews.apache.org/r/66899/diff/5/
>
>
> Testing
> -------
>
> **Unit tests**
> Additional unit tests to test specific scenarios.
>
>
> Thanks,
>
> Ashutosh Mestry
>
>
Re: Review Request 66899: Large Notification Messages: Avoid Duplicate
Processing
Posted by Ashutosh Mestry <am...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66899/
-----------------------------------------------------------
(Updated May 4, 2018, 4:33 p.m.)
Review request for atlas and Madhan Neethiraj.
Changes
-------
Updates include:
- Addressed review comments.
- Updated unit tests.
Bugs: ATLAS-2634
https://issues.apache.org/jira/browse/ATLAS-2634
Repository: atlas
Description
-------
**Approach**
Simplified approach.
**Algorithm**
- Modified: _NotificationHookConsumer_:
- when commit() fails, record current offset of the message.
- on receiving a message
- if _offset_ is same as that of the received message, skip processing the message, go ahead and commit.
- reset recorded offset.
Added appropriate log messages.
**Additional Refactoring**
None.
Diffs (updated)
-----
notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java 80dc51423
webapp/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java 7a4596a7e
webapp/src/test/java/org/apache/atlas/notification/NotificationHookConsumerKafkaTest.java d2b3dfdab
Diff: https://reviews.apache.org/r/66899/diff/4/
Changes: https://reviews.apache.org/r/66899/diff/3-4/
Testing
-------
**Unit tests**
Additional unit tests to test specific scenarios.
Thanks,
Ashutosh Mestry
Re: Review Request 66899: Large Notification Messages: Avoid Duplicate
Processing
Posted by Madhan Neethiraj <ma...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66899/#review202396
-----------------------------------------------------------
webapp/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java
Lines 577 (patched)
<https://reviews.apache.org/r/66899/#comment284205>
this catch block eats the exception. Instead, consider using finally.
long offsetFailedCommit = -1;
void commit(AtlasKafkaMessage<HookNotification> kafkaMessage) {
boolean isCommitSuccess = false;
try {
...
isCommitSuccess = true;
} finally {
offsetFailedCommit = isCommitSuccess ? -1 : kafkaMessage.getOffset();
}
}
void handleMessage(AtlasKafkaMessage<HookNotification> kafkaMsg) throws AtlasServiceException, AtlasException {
if(offsetFailedCommit != -1 && offsetFailedCommit == kafkaMsg.getOffset()) \\\{
LOG.info("message was already processed: offset={}", kafkaMsg.getOffset());
offsetFailedCommit = -1;
consumer.commit(new TopicPartition("ATLAS_HOOK", kafkaMsg.getPartition()), kafkaMessage.getOffset() + 1);
return;
}
...
}
- Madhan Neethiraj
On May 3, 2018, 11:29 p.m., Ashutosh Mestry wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66899/
> -----------------------------------------------------------
>
> (Updated May 3, 2018, 11:29 p.m.)
>
>
> Review request for atlas and Madhan Neethiraj.
>
>
> Bugs: ATLAS-2634
> https://issues.apache.org/jira/browse/ATLAS-2634
>
>
> Repository: atlas
>
>
> Description
> -------
>
> **Approach**
> Simplified approach.
>
> **Algorithm**
> - Modified: _NotificationHookConsumer_:
> - when commit() fails, record current offset of the message.
> - on receiving a message
> - if _offset_ is same as that of the received message, skip processing the message, go ahead and commit.
> - reset recorded offset.
> Added appropriate log messages.
>
> **Additional Refactoring**
> None.
>
>
> Diffs
> -----
>
> notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java 80dc51423
> webapp/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java 7a4596a7e
> webapp/src/test/java/org/apache/atlas/notification/NotificationHookConsumerKafkaTest.java d2b3dfdab
>
>
> Diff: https://reviews.apache.org/r/66899/diff/3/
>
>
> Testing
> -------
>
> **Unit tests**
> Additional unit tests to test specific scenarios.
>
>
> Thanks,
>
> Ashutosh Mestry
>
>
Re: Review Request 66899: Large Notification Messages: Avoid Duplicate
Processing
Posted by Ashutosh Mestry <am...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66899/
-----------------------------------------------------------
(Updated May 3, 2018, 11:29 p.m.)
Review request for atlas and Madhan Neethiraj.
Changes
-------
Updates include: Implemenation change.
Bugs: ATLAS-2634
https://issues.apache.org/jira/browse/ATLAS-2634
Repository: atlas
Description (updated)
-------
**Approach**
Simplified approach.
**Algorithm**
- Modified: _NotificationHookConsumer_:
- when commit() fails, record current offset of the message.
- on receiving a message
- if _offset_ is same as that of the received message, skip processing the message, go ahead and commit.
- reset recorded offset.
Added appropriate log messages.
**Additional Refactoring**
None.
Diffs (updated)
-----
notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java 80dc51423
webapp/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java 7a4596a7e
webapp/src/test/java/org/apache/atlas/notification/NotificationHookConsumerKafkaTest.java d2b3dfdab
Diff: https://reviews.apache.org/r/66899/diff/3/
Changes: https://reviews.apache.org/r/66899/diff/2-3/
Testing
-------
**Unit tests**
Additional unit tests to test specific scenarios.
Thanks,
Ashutosh Mestry