You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@nifi.apache.org by GitBox <gi...@apache.org> on 2022/11/08 20:44:04 UTC
[GitHub] [nifi] exceptionfactory opened a new pull request, #6634: NIFI-10780 Improve ListenSyslog and ListenTCP Event Queuing
exceptionfactory opened a new pull request, #6634:
URL: https://github.com/apache/nifi/pull/6634
# Summary
[NIFI-10780](https://issues.apache.org/jira/browse/NIFI-10780) Improves `ListenSyslog` and `ListenTCP` handling of queued messages when reaching the maximum configured queue size. Changes to the supporting `ByteArrayMessageChannelHandler` allow the handler to retry queuing messages as long as the Processor is running.
The changes update `ByteArrayMessageChannelHandler` to call `BlockingQueue.offer()` with a 500 ms timeout, instead of `BlockingQueue.add()`, which throws an `IllegalStateException` when attempt to add to a full queue. The handler loop checks the shutting down status of the managing `EventExecutor` to stop trying and throw an exception when the framework instructs the Processor to shutdown the listening event server.
Additional changes include adjusting the `NettyEventServer` to call `ChannelFuture.await()` instead of `ChannelFuture.syncUninterruptibly()` for the `Channel.close()` operation. This is necessary so that the `shutdown()` method proceeds to call `shutdownGracefully()`, which informs handlers that the executor is shutting down.
Queue failures can occur when the Processor is being stopped, the queue is full, and the listening event server is attempting to add a message to the queue. This scenario will result in throwing a new `EventDroppedException`, logged at the warning level.
# Tracking
Please complete the following tracking steps prior to pull request creation.
### Issue Tracking
- [X] [Apache NiFi Jira](https://issues.apache.org/jira/browse/NIFI) issue created
### Pull Request Tracking
- [X] Pull Request title starts with Apache NiFi Jira issue number, such as `NIFI-00000`
- [X] Pull Request commit message starts with Apache NiFi Jira issue number, as such `NIFI-00000`
### Pull Request Formatting
- [X] Pull Request based on current revision of the `main` branch
- [X] Pull Request refers to a feature branch with one commit containing changes
# Verification
Please indicate the verification steps performed prior to pull request creation.
### Build
- [X] Build completed using `mvn clean install -P contrib-check`
- [X] JDK 8
- [X] JDK 11
- [X] JDK 17
### Licensing
- [ ] New dependencies are compatible with the [Apache License 2.0](https://apache.org/licenses/LICENSE-2.0) according to the [License Policy](https://www.apache.org/legal/resolved.html)
- [ ] New dependencies are documented in applicable `LICENSE` and `NOTICE` files
### Documentation
- [ ] Documentation formatting appears as expected in rendered files
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] thenatog closed pull request #6634: NIFI-10780 Improve ListenSyslog and ListenTCP Event Queuing
Posted by GitBox <gi...@apache.org>.
thenatog closed pull request #6634: NIFI-10780 Improve ListenSyslog and ListenTCP Event Queuing
URL: https://github.com/apache/nifi/pull/6634
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] thenatog commented on pull request #6634: NIFI-10780 Improve ListenSyslog and ListenTCP Event Queuing
Posted by GitBox <gi...@apache.org>.
thenatog commented on PR #6634:
URL: https://github.com/apache/nifi/pull/6634#issuecomment-1310844667
Tested ListenTCP and ListenSyslog and found them to be in good working order. I did experience an 'EventDroppedException' when stopping the processor at some point. I also did a simple performance test and found that on my machine with a single thread, ListenSyslog could read about 2M short messages per 5m, and ListenTCP could read about 1.2m short messages per 5m.
For ListenTCP, single threaded, when sending 1MB messages I'm getting around 290MB throughput.
Functionally it looks good, I don't know if any of the changes would have affected performance but thought I'd check it out.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] exceptionfactory commented on pull request #6634: NIFI-10780 Improve ListenSyslog and ListenTCP Event Queuing
Posted by GitBox <gi...@apache.org>.
exceptionfactory commented on PR #6634:
URL: https://github.com/apache/nifi/pull/6634#issuecomment-1310850349
Thanks for the thorough review and performance testing @thenatog!
The `EventDroppedException` is expected when stopping the Processor prior to reading all of the events on to the FlowFile queue.
Performance will vary based on a variety of factors, such as concurrent tasks and batch size, but those are helpful references.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] thenatog commented on pull request #6634: NIFI-10780 Improve ListenSyslog and ListenTCP Event Queuing
Posted by GitBox <gi...@apache.org>.
thenatog commented on PR #6634:
URL: https://github.com/apache/nifi/pull/6634#issuecomment-1310861341
Sounds good, +1 will merge.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] thenatog commented on pull request #6634: NIFI-10780 Improve ListenSyslog and ListenTCP Event Queuing
Posted by GitBox <gi...@apache.org>.
thenatog commented on PR #6634:
URL: https://github.com/apache/nifi/pull/6634#issuecomment-1308183576
Will review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org