You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@nifi.apache.org by GitBox <gi...@apache.org> on 2022/11/09 11:44:33 UTC
[GitHub] [nifi-minifi-cpp] adamdebreceni opened a new pull request, #1449: MINIFICPP-1978 - Flush MergeContent bundles when its size would grow beyond max group size
adamdebreceni opened a new pull request, #1449:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1449
Thank you for submitting a contribution to Apache NiFi - MiNiFi C++.
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?
- [ ] Does your PR title start with MINIFICPP-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.
- [ ] Has your PR been rebased against the latest commit within the target branch (typically main)?
- [ ] Is your initial contribution a single, squashed commit?
### For code changes:
- [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the LICENSE file?
- [ ] If applicable, have you updated the NOTICE file?
### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in which it is rendered?
### Note:
Please ensure that once the PR is submitted, you check GitHub Actions CI results for build issues and submit an update to your PR as soon as possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] fgerlits closed pull request #1449: MINIFICPP-1978 - Flush MergeContent bundles when its size would grow beyond max group size
Posted by GitBox <gi...@apache.org>.
fgerlits closed pull request #1449: MINIFICPP-1978 - Flush MergeContent bundles when its size would grow beyond max group size
URL: https://github.com/apache/nifi-minifi-cpp/pull/1449
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] adamdebreceni commented on a diff in pull request #1449: MINIFICPP-1978 - Flush MergeContent bundles when its size would grow beyond max group size
Posted by GitBox <gi...@apache.org>.
adamdebreceni commented on code in PR #1449:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1449#discussion_r1017906452
##########
extensions/libarchive/BinFiles.h:
##########
@@ -87,8 +87,10 @@ class Bin {
}
}
- if ((queued_data_size_ + flow->getSize()) > maxSize_ || (queue_.size() + 1) > maxEntries_)
+ if ((queued_data_size_ + flow->getSize()) > maxSize_ || (queue_.size() + 1) > maxEntries_) {
+ closed_ = true;
Review Comment:
incoming flow files that are larger than max group size are immediately assigned their own bin, and flushed by themselves, so they won't be a problem, flow files that are large but not max size large, could cause the flush of a single bin, as we only try to insert into a single bin (the last in the group's queue) before creating a new bin for it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] adam-markovics commented on pull request #1449: MINIFICPP-1978 - Flush MergeContent bundles when its size would grow beyond max group size
Posted by GitBox <gi...@apache.org>.
adam-markovics commented on PR #1449:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1449#issuecomment-1308818056
Could you please update BinFiles documentation in `PROCESSORS.md`? To me it is not clear what this Processor is doing, and it could also be confusing for end users. Currently description is: "Bins flow files into buckets based on the number of entries or size of entries". I don't understand the following verbs and nouns in this context : "bins", "buckets", "entries". Please add some more explanation. Property description should also be updated, for example format is missing for Max Bin Age.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] adamdebreceni commented on pull request #1449: MINIFICPP-1978 - Flush MergeContent bundles when its size would grow beyond max group size
Posted by GitBox <gi...@apache.org>.
adamdebreceni commented on PR #1449:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1449#issuecomment-1309099362
> Could you please update BinFiles documentation in `PROCESSORS.md`? To me it is not clear what this Processor is doing, and it could also be confusing for end users. Currently description is: "Bins flow files into buckets based on the number of entries or size of entries". I don't understand the following verbs and nouns in this context : "bins", "buckets", "entries". Please add some more explanation. Property description should also be updated, for example format is missing for Max Bin Age.
definitely something we should address, created a ticket for it: https://issues.apache.org/jira/browse/MINIFICPP-1982
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] szaszm commented on a diff in pull request #1449: MINIFICPP-1978 - Flush MergeContent bundles when its size would grow beyond max group size
Posted by GitBox <gi...@apache.org>.
szaszm commented on code in PR #1449:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1449#discussion_r1017846663
##########
extensions/libarchive/BinFiles.h:
##########
@@ -87,8 +87,10 @@ class Bin {
}
}
- if ((queued_data_size_ + flow->getSize()) > maxSize_ || (queue_.size() + 1) > maxEntries_)
+ if ((queued_data_size_ + flow->getSize()) > maxSize_ || (queue_.size() + 1) > maxEntries_) {
+ closed_ = true;
Review Comment:
Do you think it would make sense to penalize flow files that are larger than the max group size? Currently if there is such a flow file, it will keep flushing out all bins regardless of how "full" they are.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org