You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by Boris Tyukin <bo...@boristyukin.com> on 2020/04/03 13:27:59 UTC
MergeRecord issue - Could not merge bin .. encountered before this
bin was evicted
Hey guys,
weird issue, NiFi 1.9.2 both clustered mode and a single NiFi node.
I have this process below that runs every 15 minutes. 4 or 5 times a day a
get an error below now in the past few days while it worked fine for weeks
with no errors at all. Help me understand why please.
from help file:
*If the <Merge Strategy> property is set to "Defragment" then a bin is full
only when the number of FlowFiles in the bin is equal to the number
specified by the "fragment.count" attribute of one of the FlowFiles in the
bin*. All FlowFiles that have this attribute must have the same value for
this attribute, or else they will be routed to the "failure" relationship.
It is not necessary that all FlowFiles have this value, but at least one
FlowFile in the bin must have this value or the bin will never be complete.
If all of the necessary FlowFiles are not binned together by the point at
which the bin times amount (*as specified by the <Max Bin Age> property),*
then the FlowFiles will all be routed to the 'failure' relationship instead
of being merged together.
I never set MAx Bin Age or Max size and till now it worked fine. Was this
behaviour changed possibly and now either max age or max size needs to be
set for defragement strategy?
[image: image.png]
bulletin error
[{"objectId":"b297e4a7-cc7d-4e66-87bc-3cc46a72d0a4","platform":"nifi","bulletinId":518,"bulletinCategory":"Log
Message","bulletinGroupId":"ecd049fd-c23e-3010-923e-4295c38b9606","bulletinGroupName":"Get
COVID19
encounters","bulletinLevel":"ERROR","bulletinMessage":"MergeRecord[id=09d508eb-7e53-37cc-b7fb-66fb3fbe1432]
Could not merge bin with 2 FlowFiles because the 'fragment.count' attribute
had a value of '3' but only 2 of 3 FlowFiles were encountered before this
bin was evicted (due to to Max Bin Age being reached or due to the Maximum
Number of Bins being
exceeded).","bulletinNodeId":"2bff8aac-ace6-4cdd-8b19-a6e093a48ca2","bulletinSourceId":"09d508eb-7e53-37cc-b7fb-66fb3fbe1432","bulletinSourceName":"MergeRecord","bulletinSourceType":"PROCESSOR","bulletinTimestamp":"2020-04-02T23:43:00.252Z"},{"objectId":"c21832e8-27bf-4f5b-a373-06c94256a08a","platform":"nifi","bulletinId":517,"bulletinCategory":"Log
Message","bulletinGroupId":"ecd049fd-c23e-3010-923e-4295c38b9606","bulletinGroupName":"Get
COVID19
encounters","bulletinLevel":"ERROR","bulletinMessage":"MergeRecord[id=09d508eb-7e53-37cc-b7fb-66fb3fbe1432]
Could not merge bin with 2 FlowFiles because the 'fragment.count' attribute
had a value of '3' but only 2 of 3 FlowFiles were encountered before this
bin was evicted (due to to Max Bin Age being reached or due to the Maximum
Number of Bins being
exceeded).","bulletinNodeId":"2bff8aac-ace6-4cdd-8b19-a6e093a48ca2","bulletinSourceId":"09d508eb-7e53-37cc-b7fb-66fb3fbe1432","bulletinSourceName":"MergeRecord","bulletinSourceType":"PROCESSOR","bulletinTimestamp":"2020-04-02T23:43:00.244Z"},{"objectId":"4cbc4dd5-3326-49db-ad62-2898282f285e","platform":"nifi","bulletinId":516,"bulletinCategory":"Log
Message","bulletinGroupId":"ecd049fd-c23e-3010-923e-4295c38b9606","bulletinGroupName":"Get
COVID19
encounters","bulletinLevel":"ERROR","bulletinMessage":"MergeRecord[id=09d508eb-7e53-37cc-b7fb-66fb3fbe1432]
Could not merge bin with 2 FlowFiles because the 'fragment.count' attribute
had a value of '3' but only 2 of 3 FlowFiles were encountered before this
bin was evicted (due to to Max Bin Age being reached or due to the Maximum
Number of Bins being
exceeded).","bulletinNodeId":"2bff8aac-ace6-4cdd-8b19-a6e093a48ca2","bulletinSourceId":"09d508eb-7e53-37cc-b7fb-66fb3fbe1432","bulletinSourceName":"MergeRecord","bulletinSourceType":"PROCESSOR","bulletinTimestamp":"2020-04-02T23:43:00.243Z"},{"objectId":"6c039c2c-3640-4890-8afd-dfdb74aafcf3","platform":"nifi","bulletinId":515,"bulletinCategory":"Log
Message","bulletinGroupId":"ecd049fd-c23e-3010-923e-4295c38b9606","bulletinGroupName":"Get
COVID19
encounters","bulletinLevel":"ERROR","bulletinMessage":"MergeRecord[id=09d508eb-7e53-37cc-b7fb-66fb3fbe1432]
Could not merge bin with 2 FlowFiles because the 'fragment.count' attribute
had a value of '3' but only 2 of 3 FlowFiles were encountered before this
bin was evicted (due to to Max Bin Age being reached or due to the Maximum
Number of Bins being
exceeded).","bulletinNodeId":"2bff8aac-ace6-4cdd-8b19-a6e093a48ca2","bulletinSourceId":"09d508eb-7e53-37cc-b7fb-66fb3fbe1432","bulletinSourceName":"MergeRecord","bulletinSourceType":"PROCESSOR","bulletinTimestamp":"2020-04-02T23:43:00.241Z"},{"objectId":"1fd7f1f4-c45e-4b31-a5b7-71ce5014128d","platform":"nifi","bulletinId":514,"bulletinCategory":"Log
Message","bulletinGroupId":"ecd049fd-c23e-3010-923e-4295c38b9606","bulletinGroupName":"Get
COVID19
encounters","bulletinLevel":"ERROR","bulletinMessage":"MergeRecord[id=09d508eb-7e53-37cc-b7fb-66fb3fbe1432]
Could not merge bin with 2 FlowFiles because the 'fragment.count' attribute
had a value of '3' but only 2 of 3 FlowFiles were encountered before this
bin was evicted (due to to Max Bin Age being reached or due to the Maximum
Number of Bins being
exceeded).","bulletinNodeId":"2bff8aac-ace6-4cdd-8b19-a6e093a48ca2","bulletinSourceId":"09d508eb-7e53-37cc-b7fb-66fb3fbe1432","bulletinSourceName":"MergeRecord","bulletinSourceType":"PROCESSOR","bulletinTimestamp":"2020-04-02T23:43:00.240Z"}]
Re: MergeRecord issue - Could not merge bin .. encountered before
this bin was evicted
Posted by Boris Tyukin <bo...@boristyukin.com>.
if anyone interested, I switched to MergeContent leaving defragment
strategy and all the same and it works like a charm.
I guess I either do not understand how MergeRecord processor supposed to
work in defragment mode or documentation is not accurate.
On Fri, Apr 3, 2020 at 9:27 AM Boris Tyukin <bo...@boristyukin.com> wrote:
> Hey guys,
>
> weird issue, NiFi 1.9.2 both clustered mode and a single NiFi node.
>
> I have this process below that runs every 15 minutes. 4 or 5 times a day a
> get an error below now in the past few days while it worked fine for weeks
> with no errors at all. Help me understand why please.
>
> from help file:
>
> *If the <Merge Strategy> property is set to "Defragment" then a bin is
> full only when the number of FlowFiles in the bin is equal to the number
> specified by the "fragment.count" attribute of one of the FlowFiles in the
> bin*. All FlowFiles that have this attribute must have the same value for
> this attribute, or else they will be routed to the "failure" relationship.
> It is not necessary that all FlowFiles have this value, but at least one
> FlowFile in the bin must have this value or the bin will never be complete.
> If all of the necessary FlowFiles are not binned together by the point at
> which the bin times amount (*as specified by the <Max Bin Age> property),*
> then the FlowFiles will all be routed to the 'failure' relationship instead
> of being merged together.
>
>
> I never set MAx Bin Age or Max size and till now it worked fine. Was this
> behaviour changed possibly and now either max age or max size needs to be
> set for defragement strategy?
>
> [image: image.png]
> bulletin error
>
> [{"objectId":"b297e4a7-cc7d-4e66-87bc-3cc46a72d0a4","platform":"nifi","bulletinId":518,"bulletinCategory":"Log
> Message","bulletinGroupId":"ecd049fd-c23e-3010-923e-4295c38b9606","bulletinGroupName":"Get
> COVID19
> encounters","bulletinLevel":"ERROR","bulletinMessage":"MergeRecord[id=09d508eb-7e53-37cc-b7fb-66fb3fbe1432]
> Could not merge bin with 2 FlowFiles because the 'fragment.count' attribute
> had a value of '3' but only 2 of 3 FlowFiles were encountered before this
> bin was evicted (due to to Max Bin Age being reached or due to the Maximum
> Number of Bins being
> exceeded).","bulletinNodeId":"2bff8aac-ace6-4cdd-8b19-a6e093a48ca2","bulletinSourceId":"09d508eb-7e53-37cc-b7fb-66fb3fbe1432","bulletinSourceName":"MergeRecord","bulletinSourceType":"PROCESSOR","bulletinTimestamp":"2020-04-02T23:43:00.252Z"},{"objectId":"c21832e8-27bf-4f5b-a373-06c94256a08a","platform":"nifi","bulletinId":517,"bulletinCategory":"Log
> Message","bulletinGroupId":"ecd049fd-c23e-3010-923e-4295c38b9606","bulletinGroupName":"Get
> COVID19
> encounters","bulletinLevel":"ERROR","bulletinMessage":"MergeRecord[id=09d508eb-7e53-37cc-b7fb-66fb3fbe1432]
> Could not merge bin with 2 FlowFiles because the 'fragment.count' attribute
> had a value of '3' but only 2 of 3 FlowFiles were encountered before this
> bin was evicted (due to to Max Bin Age being reached or due to the Maximum
> Number of Bins being
> exceeded).","bulletinNodeId":"2bff8aac-ace6-4cdd-8b19-a6e093a48ca2","bulletinSourceId":"09d508eb-7e53-37cc-b7fb-66fb3fbe1432","bulletinSourceName":"MergeRecord","bulletinSourceType":"PROCESSOR","bulletinTimestamp":"2020-04-02T23:43:00.244Z"},{"objectId":"4cbc4dd5-3326-49db-ad62-2898282f285e","platform":"nifi","bulletinId":516,"bulletinCategory":"Log
> Message","bulletinGroupId":"ecd049fd-c23e-3010-923e-4295c38b9606","bulletinGroupName":"Get
> COVID19
> encounters","bulletinLevel":"ERROR","bulletinMessage":"MergeRecord[id=09d508eb-7e53-37cc-b7fb-66fb3fbe1432]
> Could not merge bin with 2 FlowFiles because the 'fragment.count' attribute
> had a value of '3' but only 2 of 3 FlowFiles were encountered before this
> bin was evicted (due to to Max Bin Age being reached or due to the Maximum
> Number of Bins being
> exceeded).","bulletinNodeId":"2bff8aac-ace6-4cdd-8b19-a6e093a48ca2","bulletinSourceId":"09d508eb-7e53-37cc-b7fb-66fb3fbe1432","bulletinSourceName":"MergeRecord","bulletinSourceType":"PROCESSOR","bulletinTimestamp":"2020-04-02T23:43:00.243Z"},{"objectId":"6c039c2c-3640-4890-8afd-dfdb74aafcf3","platform":"nifi","bulletinId":515,"bulletinCategory":"Log
> Message","bulletinGroupId":"ecd049fd-c23e-3010-923e-4295c38b9606","bulletinGroupName":"Get
> COVID19
> encounters","bulletinLevel":"ERROR","bulletinMessage":"MergeRecord[id=09d508eb-7e53-37cc-b7fb-66fb3fbe1432]
> Could not merge bin with 2 FlowFiles because the 'fragment.count' attribute
> had a value of '3' but only 2 of 3 FlowFiles were encountered before this
> bin was evicted (due to to Max Bin Age being reached or due to the Maximum
> Number of Bins being
> exceeded).","bulletinNodeId":"2bff8aac-ace6-4cdd-8b19-a6e093a48ca2","bulletinSourceId":"09d508eb-7e53-37cc-b7fb-66fb3fbe1432","bulletinSourceName":"MergeRecord","bulletinSourceType":"PROCESSOR","bulletinTimestamp":"2020-04-02T23:43:00.241Z"},{"objectId":"1fd7f1f4-c45e-4b31-a5b7-71ce5014128d","platform":"nifi","bulletinId":514,"bulletinCategory":"Log
> Message","bulletinGroupId":"ecd049fd-c23e-3010-923e-4295c38b9606","bulletinGroupName":"Get
> COVID19
> encounters","bulletinLevel":"ERROR","bulletinMessage":"MergeRecord[id=09d508eb-7e53-37cc-b7fb-66fb3fbe1432]
> Could not merge bin with 2 FlowFiles because the 'fragment.count' attribute
> had a value of '3' but only 2 of 3 FlowFiles were encountered before this
> bin was evicted (due to to Max Bin Age being reached or due to the Maximum
> Number of Bins being
> exceeded).","bulletinNodeId":"2bff8aac-ace6-4cdd-8b19-a6e093a48ca2","bulletinSourceId":"09d508eb-7e53-37cc-b7fb-66fb3fbe1432","bulletinSourceName":"MergeRecord","bulletinSourceType":"PROCESSOR","bulletinTimestamp":"2020-04-02T23:43:00.240Z"}]
>