You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2021/03/18 22:26:53 UTC

[GitHub] [pulsar] david-streamlio opened a new issue #9966: Fail immediately on ErrorNotifierRecord inside BatchPushSource

david-streamlio opened a new issue #9966:
URL: https://github.com/apache/pulsar/issues/9966


   **Is your enhancement request related to a problem? Please describe.**
   When an IO connector that implements the BatchPushSource interface wants to fail a task, it publishes a `ErrorNotifierRecord` to the internal queue. However, the exception isn't thrown until the readNext method reads the message from the internal queue.
   
   This results in an unnecessary delay between the time the error condition was encountered and the time when the exception is thrown. In addition, a lot of records are processed that shouldn't be if the task is going to get re-processed (duplicate data), etc.
   
   **Describe the solution you'd like**
   Change the logic in the consume method to check for the presence of a `ErrorNotifierRecord` and throw the exception then rather than inside the `readNext()` method
   
   **Describe alternatives you've considered**
   Leaving the code as-is
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [pulsar] jerrypeng commented on issue #9966: Fail immediately on ErrorNotifierRecord inside BatchPushSource

Posted by GitBox <gi...@apache.org>.
jerrypeng commented on issue #9966:
URL: https://github.com/apache/pulsar/issues/9966#issuecomment-802579670


   Perhaps there should be two modes of notifying a fatal error.  One is to restart without flushing the internal queue of records.  The other one is to restart after flushing the internal queue of records.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [pulsar] david-streamlio commented on issue #9966: Fail immediately on ErrorNotifierRecord inside BatchPushSource

Posted by GitBox <gi...@apache.org>.
david-streamlio commented on issue #9966:
URL: https://github.com/apache/pulsar/issues/9966#issuecomment-803035421


   > Perhaps there should be two modes of notifying a fatal error. One is to restart without flushing the internal queue of records. The other one is to restart after flushing the internal queue of records.
   
   Agreed, there could be more than one way to handle a single bad record depending upon the type of error, and the needs of the class implementing the BatchPushSource, etc


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [pulsar] codelipenghui commented on issue #9966: Fail immediately on ErrorNotifierRecord inside BatchPushSource

Posted by GitBox <gi...@apache.org>.
codelipenghui commented on issue #9966:
URL: https://github.com/apache/pulsar/issues/9966#issuecomment-1058892231


   The issue had no activity for 30 days, mark with Stale label.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [pulsar] david-streamlio commented on issue #9966: Fail immediately on ErrorNotifierRecord inside BatchPushSource

Posted by GitBox <gi...@apache.org>.
david-streamlio commented on issue #9966:
URL: https://github.com/apache/pulsar/issues/9966#issuecomment-803033947


   I guess my main concern is the duplication of data if a task is failed and re-processed. Consider a scenario where the task is a list of 1,000 records to be put into a database, and we fail on record 500 on a non-connectivity errors such as "invalid foreign key ".  Then we proceed with inserting the remaining 499 records before failing the entire task. 
   
   The task message will get redelivered and the same 999 records will be inserted a second time. Even if these second inserts were just duplicates, they would all be unnecessary calls to the DB. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [pulsar] jerrypeng commented on issue #9966: Fail immediately on ErrorNotifierRecord inside BatchPushSource

Posted by GitBox <gi...@apache.org>.
jerrypeng commented on issue #9966:
URL: https://github.com/apache/pulsar/issues/9966#issuecomment-802578589


   > When an IO connector that implements the BatchPushSource interface wants to fail a task, it publishes a ErrorNotifierRecord to the internal queue. However, the exception isn't thrown until the readNext method reads the message from the internal queue.
   
   This is done intentionally so messages that were already consumed will still get a chance to be published.
   
   > In addition, a lot of records are processed that shouldn't be
   
   In what scenario will records get processed that shouldn't be processed?  The ErrorNotifierRecord is sent in stream, thus messages that came before it should be valid messages that can be sent out.
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org