You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@atlas.apache.org by Ashutosh Mestry <am...@hortonworks.com> on 2018/05/01 23:51:50 UTC

Review Request 66899: Large Notification Messages: Avoid Duplicate Processing

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66899/
-----------------------------------------------------------

Review request for atlas and Madhan Neethiraj.


Bugs: ATLAS-2634
    https://issues.apache.org/jira/browse/ATLAS-2634


Repository: atlas


Description
-------

**Approach**
Modified _AtlasNotificationMessageDeserializer_:
- Large message is defined as message that has been subjected to split or compression. The deserialization follows a separate path to accumulate splits or decompress compressed messages.
- Algorithm:
  - Added internal class that keep a map of message ids and number of times the message was fetched.
  - Large messages are added to this map.
  - Everytime the message is fetch, the fetch count is incremented.
  - For message that is not large, fetch count of 0 is returned.
  - If a fetch count is greater than 1, which is that the message is retrieved more than once, commit on Kafka is called immediately. This message is not handed to _NotificationHookConsumer_ for further processing.

Modified _AtlasKafkaConsumer_: Handle the new fetch logic.
Modified _NotificationHookConsumer_: Extracted _ATLAS_HOOK_ topic name.

**Additional Refactoring**
None.


Diffs
-----

  notification/src/main/java/org/apache/atlas/kafka/AtlasKafkaConsumer.java fd0c4e48a 
  notification/src/main/java/org/apache/atlas/notification/AbstractNotificationConsumer.java c3940ceb4 
  notification/src/main/java/org/apache/atlas/notification/AtlasNotificationMessageDeserializer.java 3264e264c 
  notification/src/test/java/org/apache/atlas/notification/AbstractNotificationConsumerTest.java c9638309d 
  notification/src/test/java/org/apache/atlas/notification/hook/HookNotificationDeserializerTest.java d04817024 
  webapp/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java 7a4596a7e 


Diff: https://reviews.apache.org/r/66899/diff/1/


Testing
-------

**Unit tests**
Additional unit tests to test specific scenarios.


Thanks,

Ashutosh Mestry


Re: Review Request 66899: Large Notification Messages: Avoid Duplicate Processing

Posted by Madhan Neethiraj <ma...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66899/#review202459
-----------------------------------------------------------


Ship it!




Ship It!

- Madhan Neethiraj


On May 4, 2018, 4:33 p.m., Ashutosh Mestry wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66899/
> -----------------------------------------------------------
> 
> (Updated May 4, 2018, 4:33 p.m.)
> 
> 
> Review request for atlas and Madhan Neethiraj.
> 
> 
> Bugs: ATLAS-2634
>     https://issues.apache.org/jira/browse/ATLAS-2634
> 
> 
> Repository: atlas
> 
> 
> Description
> -------
> 
> **Approach**
> Simplified approach.
> 
> **Algorithm**
>  - Modified: _NotificationHookConsumer_:
>    - when commit() fails, record current offset of the message.
>    - on receiving a message
>       - if _offset_ is same as that of the received message, skip processing the message, go ahead and commit.
>       - reset recorded offset.
> Added appropriate log messages.
> 
> **Additional Refactoring**
> None.
> 
> 
> Diffs
> -----
> 
>   notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java 80dc51423 
>   webapp/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java 7a4596a7e 
>   webapp/src/test/java/org/apache/atlas/notification/NotificationHookConsumerKafkaTest.java d2b3dfdab 
> 
> 
> Diff: https://reviews.apache.org/r/66899/diff/5/
> 
> 
> Testing
> -------
> 
> **Unit tests**
> Additional unit tests to test specific scenarios.
> 
> 
> Thanks,
> 
> Ashutosh Mestry
> 
>


Re: Review Request 66899: Large Notification Messages: Avoid Duplicate Processing

Posted by Ashutosh Mestry <am...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66899/
-----------------------------------------------------------

(Updated May 4, 2018, 4:33 p.m.)


Review request for atlas and Madhan Neethiraj.


Changes
-------

Updates include:
- Addressed review comments.
- Updated unit tests.


Bugs: ATLAS-2634
    https://issues.apache.org/jira/browse/ATLAS-2634


Repository: atlas


Description
-------

**Approach**
Simplified approach.

**Algorithm**
 - Modified: _NotificationHookConsumer_:
   - when commit() fails, record current offset of the message.
   - on receiving a message
      - if _offset_ is same as that of the received message, skip processing the message, go ahead and commit.
      - reset recorded offset.
Added appropriate log messages.

**Additional Refactoring**
None.


Diffs (updated)
-----

  notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java 80dc51423 
  webapp/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java 7a4596a7e 
  webapp/src/test/java/org/apache/atlas/notification/NotificationHookConsumerKafkaTest.java d2b3dfdab 


Diff: https://reviews.apache.org/r/66899/diff/4/

Changes: https://reviews.apache.org/r/66899/diff/3-4/


Testing
-------

**Unit tests**
Additional unit tests to test specific scenarios.


Thanks,

Ashutosh Mestry


Re: Review Request 66899: Large Notification Messages: Avoid Duplicate Processing

Posted by Madhan Neethiraj <ma...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66899/#review202396
-----------------------------------------------------------




webapp/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java
Lines 577 (patched)
<https://reviews.apache.org/r/66899/#comment284205>

    this catch block eats the exception. Instead, consider using finally.
    
    long offsetFailedCommit = -1;
    
    void commit(AtlasKafkaMessage<HookNotification> kafkaMessage) {
      boolean isCommitSuccess = false;
    
      try {
        ...
    
        isCommitSuccess = true;
      } finally {
        offsetFailedCommit = isCommitSuccess ? -1 : kafkaMessage.getOffset();
      }
    }
    
    void handleMessage(AtlasKafkaMessage<HookNotification> kafkaMsg) throws AtlasServiceException, AtlasException {
    
      if(offsetFailedCommit != -1 && offsetFailedCommit == kafkaMsg.getOffset()) \\\{
        LOG.info("message was already processed: offset={}", kafkaMsg.getOffset());
    
        offsetFailedCommit = -1;
    
        consumer.commit(new TopicPartition("ATLAS_HOOK", kafkaMsg.getPartition()), kafkaMessage.getOffset() + 1);
    
        return;
      }
    
      ...
    
    }


- Madhan Neethiraj


On May 3, 2018, 11:29 p.m., Ashutosh Mestry wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66899/
> -----------------------------------------------------------
> 
> (Updated May 3, 2018, 11:29 p.m.)
> 
> 
> Review request for atlas and Madhan Neethiraj.
> 
> 
> Bugs: ATLAS-2634
>     https://issues.apache.org/jira/browse/ATLAS-2634
> 
> 
> Repository: atlas
> 
> 
> Description
> -------
> 
> **Approach**
> Simplified approach.
> 
> **Algorithm**
>  - Modified: _NotificationHookConsumer_:
>    - when commit() fails, record current offset of the message.
>    - on receiving a message
>       - if _offset_ is same as that of the received message, skip processing the message, go ahead and commit.
>       - reset recorded offset.
> Added appropriate log messages.
> 
> **Additional Refactoring**
> None.
> 
> 
> Diffs
> -----
> 
>   notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java 80dc51423 
>   webapp/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java 7a4596a7e 
>   webapp/src/test/java/org/apache/atlas/notification/NotificationHookConsumerKafkaTest.java d2b3dfdab 
> 
> 
> Diff: https://reviews.apache.org/r/66899/diff/3/
> 
> 
> Testing
> -------
> 
> **Unit tests**
> Additional unit tests to test specific scenarios.
> 
> 
> Thanks,
> 
> Ashutosh Mestry
> 
>


Re: Review Request 66899: Large Notification Messages: Avoid Duplicate Processing

Posted by Ashutosh Mestry <am...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66899/
-----------------------------------------------------------

(Updated May 3, 2018, 11:29 p.m.)


Review request for atlas and Madhan Neethiraj.


Changes
-------

Updates include: Implemenation change.


Bugs: ATLAS-2634
    https://issues.apache.org/jira/browse/ATLAS-2634


Repository: atlas


Description (updated)
-------

**Approach**
Simplified approach.

**Algorithm**
 - Modified: _NotificationHookConsumer_:
   - when commit() fails, record current offset of the message.
   - on receiving a message
      - if _offset_ is same as that of the received message, skip processing the message, go ahead and commit.
      - reset recorded offset.
Added appropriate log messages.

**Additional Refactoring**
None.


Diffs (updated)
-----

  notification/src/main/java/org/apache/atlas/kafka/KafkaNotification.java 80dc51423 
  webapp/src/main/java/org/apache/atlas/notification/NotificationHookConsumer.java 7a4596a7e 
  webapp/src/test/java/org/apache/atlas/notification/NotificationHookConsumerKafkaTest.java d2b3dfdab 


Diff: https://reviews.apache.org/r/66899/diff/3/

Changes: https://reviews.apache.org/r/66899/diff/2-3/


Testing
-------

**Unit tests**
Additional unit tests to test specific scenarios.


Thanks,

Ashutosh Mestry