You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by Kiran <ki...@protonmail.com> on 2018/07/10 20:35:49 UTC

Adding a file to a zip file

Hello,

I've got a requirement to add a JSON file to an existing zip file.

I'm doing this by:

- Unpacking the ZIP file
- Increment the fragment.index and fragment.count of the original files
- Create the JSON file and set the fragment.index to 1 and set the fragment.count
- Merge the contents of the files to create the resulting ZIP file

I've attached an image of the data flow and the settings for the MergeContent processor.

When I process the ZIP files one by one this works fine but when I process the ZIP files in bulk some work and others fail the MergeContent processor. I'm guessing that it's to do with the settings of the MergeContent processor. Can anyone provide me with insight on what I'm doing wrong here?

Thanks

Kiran

Re: [EXT] Adding a file to a zip file

Posted by Kiran <ki...@protonmail.com>.
Mark,

Thank you that was the issue and it's all working fine now.

Thanks

‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
On 11 July 2018 3:32 PM, Mark Payne <ma...@hotmail.com> wrote:

> Kiran,
>
> What do you have set for the "Maximum number of Bins" property of MergeContent?
> Each 'zip bundle' will have all of the FlowFiles added to the same bucket.
> So if you have more 'zip bundles' coming in than you have available buckets,
> it will evict one of the bins before all of its FlowFiles have arrived. I suspect this
> is your issue. If so, you can probably increase the number of available bins to
> take care of this.
>
> Thanks
> -Mark
>
>> From: Kiran [mailto:kiran.bdi@protonmail.com]
>> Sent: Tuesday, July 10, 2018 2:36 PM
>> To: users <us...@nifi.apache.org>
>> Subject: [EXT] Adding a file to a zip file
>>
>> Hello,
>>
>> I've got a requirement to add a JSON file to an existing zip file.
>>
>> I'm doing this by:
>>
>> - Unpacking the ZIP file
>> - Increment the fragment.index and fragment.count of the original files
>> - Create the JSON file and set the fragment.index to 1 and set the fragment.count
>> - Merge the contents of the files to create the resulting ZIP file
>>
>> I've attached an image of the data flow and the settings for the MergeContent processor.
>>
>> When I process the ZIP files one by one this works fine but when I process the ZIP files in bulk some work and others fail the MergeContent processor. I'm guessing that it's to do with the settings of the MergeContent processor. Can anyone provide me with insight on what I'm doing wrong here?
>>
>> Thanks
>>
>> Kiran

Re: [EXT] Adding a file to a zip file

Posted by Mark Payne <ma...@hotmail.com>.
Kiran,

What do you have set for the "Maximum number of Bins" property of MergeContent?
Each 'zip bundle' will have all of the FlowFiles added to the same bucket.
So if you have more 'zip bundles' coming in than you have available buckets,
it will evict one of the bins before all of its FlowFiles have arrived. I suspect this
is your issue. If so, you can probably increase the number of available bins to
take care of this.

Thanks
-Mark



From: Kiran [mailto:kiran.bdi@protonmail.com]
Sent: Tuesday, July 10, 2018 2:36 PM
To: users <us...@nifi.apache.org>>
Subject: [EXT] Adding a file to a zip file

Hello,

I've got a requirement to add a JSON file to an existing zip file.

I'm doing this by:

  1.  Unpacking the ZIP file
  2.  Increment the fragment.index and fragment.count of the original files
  3.  Create the JSON file and set the fragment.index to 1 and set the fragment.count
  4.  Merge the contents of the files to create the resulting ZIP file

I've attached an image of the data flow and the settings for the MergeContent processor.

When I process the ZIP files one by one this works fine but when I process the ZIP files in bulk some work and others fail the MergeContent processor. I'm guessing that it's to do with the settings of the MergeContent processor. Can anyone provide me with insight on what I'm doing wrong here?

Thanks

Kiran

RE: [EXT] Adding a file to a zip file

Posted by "Peter Wicks (pwicks)" <pw...@micron.com>.
Hi Kiran,

In your flow, how do you avoid duplicate files going into MergeContent?

For example:

  1.  file1.zip goes into Unpack zip file, it contains 5 files.
  2.  These 5 files are sent down both success paths (AttributeToJSON and increment fragment index and count)
  3.  5 files show up at Merge Content, and are waiting for that 1 file.
  4.  Meanwhile 5 files show up to AttrigutesToJSON, and then have their fragment.index set to 1…
  5.  10 files are now available to MergeContent with the same Fragment Identifier and covering all required indexes…

Is this not happening?

Thanks,
  Peter

From: Kiran [mailto:kiran.bdi@protonmail.com]
Sent: Tuesday, July 10, 2018 2:36 PM
To: users <us...@nifi.apache.org>
Subject: [EXT] Adding a file to a zip file

Hello,

I've got a requirement to add a JSON file to an existing zip file.

I'm doing this by:

  1.  Unpacking the ZIP file
  2.  Increment the fragment.index and fragment.count of the original files
  3.  Create the JSON file and set the fragment.index to 1 and set the fragment.count
  4.  Merge the contents of the files to create the resulting ZIP file
I've attached an image of the data flow and the settings for the MergeContent processor.

When I process the ZIP files one by one this works fine but when I process the ZIP files in bulk some work and others fail the MergeContent processor. I'm guessing that it's to do with the settings of the MergeContent processor. Can anyone provide me with insight on what I'm doing wrong here?

Thanks

Kiran