You are viewing a plain text version of this content. The canonical link for it is here.

Posted to users@nifi.apache.org by Samarendra Sahoo <sa...@gmail.com> on 2020/07/14 13:09:41 UTC

Loading set-of files to tables

Hi all,
We have a need where we get a list containing a set of source files and
target table names. Would like to load the files in parallel (as much as
possible). The list need to be picked up from the database and it can
change for each of the runs.

Could you pls suggest an approach for this.

FYI, we are able to get this done sequentially by updating the parameter
context with the source file and target tables.

Thanks a lot

Re: Loading set-of files to tables

Posted by Samarendra Sahoo <sa...@gmail.com>.

Thanks a lot Bryan. Below is what I ve done so far.


1. Set the source files and target tables mapping.





2.Create separate flow files. Extract data and load it to target table.







On Tue, Jul 14, 2020 at 7:23 PM Bryan Bende <bb...@gmail.com> wrote:

> Hello,
>
> Can you describe the flow and processors that you used to make it work
> with parameter context?
>
> It will help people better understand what you are trying to do and how to
> make it more dynamic.
>
> Thanks,
>
> Bryan
>
> On Tue, Jul 14, 2020 at 9:09 AM Samarendra Sahoo <
> sahoo.samarendra@gmail.com> wrote:
>
>> Hi all,
>> We have a need where we get a list containing a set of source files and
>> target table names. Would like to load the files in parallel (as much as
>> possible). The list need to be picked up from the database and it can
>> change for each of the runs.
>>
>> Could you pls suggest an approach for this.
>>
>> FYI, we are able to get this done sequentially by updating the parameter
>> context with the source file and target tables.
>>
>> Thanks a lot
>>
>

Re: Loading set-of files to tables

Posted by Bryan Bende <bb...@gmail.com>.

Hello,

Can you describe the flow and processors that you used to make it work with
parameter context?

It will help people better understand what you are trying to do and how to
make it more dynamic.

Thanks,

Bryan

On Tue, Jul 14, 2020 at 9:09 AM Samarendra Sahoo <sa...@gmail.com>
wrote:

> Hi all,
> We have a need where we get a list containing a set of source files and
> target table names. Would like to load the files in parallel (as much as
> possible). The list need to be picked up from the database and it can
> change for each of the runs.
>
> Could you pls suggest an approach for this.
>
> FYI, we are able to get this done sequentially by updating the parameter
> context with the source file and target tables.
>
> Thanks a lot
>