You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by "DavidRayner (via GitHub)" <gi...@apache.org> on 2024/04/16 12:45:37 UTC

[PR] [improve][fn] Flush logs to log-topic periodically for Go Pulsar functions [pulsar]

DavidRayner opened a new pull request, #22516:
URL: https://github.com/apache/pulsar/pull/22516

   ### Motivation
   
   <!-- Explain here the context, and why you're making that change. What is the problem you're trying to solve. -->
   
   Currently log messages for Go pulsar functions are only sent out to the log topic when `addLogTopicHandler` is called, which only happens when a Pulsar message is received (just before it is processed).
   
   Therefore, if a function is not processing any data, any log statements from healthchecks etc will not be sent to the log topic. Also, any log statements that occur when processing a message will only appear when another message is processed.
   
   ### Modifications
   
   After `logAppender.Start` is called, a new method `setupLogHandlerTicker` is called to start a ticker in a goroutine.  
   The ticker calls `flushLogsToTopicHandler` (previously named `addLogTopicHandler`) every 100ms.
   
   The `logAppender.Stop` method has been updated to stop the ticker after one final flush of logs to the log-topic.
   
   A mutex has been added to `flushLogsToTopicHandler` to ensure that the ticker does not attempt to flush log statements to the log-topic whilst they are being flushed when a Pulsar message is received.
   
   ### Verifying this change
   
   - [ ] Make sure that the change passes the CI checks.
   
   This change added tests and can be verified as follows:
   
    - Unit tests have been added to test that the new ticker will flush logs out to the log-topic.
   
   ### Does this pull request potentially affect one of the following parts:
   
   <!-- DO NOT REMOVE THIS SECTION. CHECK THE PROPER BOX ONLY. -->
   
   *If the box was checked, please highlight the changes*
   
   - [ ] Dependencies (add or upgrade a dependency)
   - [ ] The public API
   - [ ] The schema
   - [ ] The default values of configurations
   - [ ] The threading model
   - [ ] The binary protocol
   - [ ] The REST endpoints
   - [ ] The admin CLI options
   - [ ] The metrics
   - [ ] Anything that affects deployment
   
   ### Documentation
   
   <!-- DO NOT REMOVE THIS SECTION. CHECK THE PROPER BOX ONLY. -->
   
   - [ ] `doc` <!-- Your PR contains doc changes. -->
   - [ ] `doc-required` <!-- Your PR changes impact docs and you will update later -->
   - [x] `doc-not-needed` <!-- Your PR changes do not impact docs -->
   - [ ] `doc-complete` <!-- Docs have been already added -->
   
   ### Matching PR in forked repository
   
   PR in forked repository: https://github.com/DavidRayner/pulsar/pull/2
   
   <!--
   After opening this PR, the build in apache/pulsar will fail and instructions will
   be provided for opening a PR in the PR author's forked repository.
   
   apache/pulsar pull requests should be first tested in your own fork since the 
   apache/pulsar CI based on GitHub Actions has constrained resources and quota.
   GitHub Actions provides separate quota for pull requests that are executed in 
   a forked repository.
   
   The tests will be run in the forked repository until all PR review comments have
   been handled, the tests pass and the PR is approved by a reviewer.
   -->
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [fix][fn] Send Go function logs to the log topic periodically and on shutdown [pulsar]

Posted by "merlimat (via GitHub)" <gi...@apache.org>.
merlimat merged PR #22516:
URL: https://github.com/apache/pulsar/pull/22516


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [fix][fn] Send Go function logs to the log topic periodically and on shutdown [pulsar]

Posted by "merlimat (via GitHub)" <gi...@apache.org>.
merlimat commented on PR #22516:
URL: https://github.com/apache/pulsar/pull/22516#issuecomment-6353286922

   Thanks @DavidRayner, this fixes a real problem: Go function logs were only sent when the next message arrived. I pushed a rework to your branch. The log buffer is now locked, the appender owns the flush goroutine and waits for it on stop, the remaining logs are sent before the producer closes, and the nil-dereference when no log topic is configured is gone. @lhotari @freeznet could you take another look, since the code changed after your approvals?
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org