Re: Need Help! Execute stream process not able to pick up files from queue.

[Raghu G <ra...@gmail.com>]

Hi all,

Case1:
While working with a single file or couple of files flow seems to be
working good.

Case2:
While working with large number of files with 17mb/queue and total of 5
queues (around 75mb)

file gets into queue but execute processor doesn't pick up the file and
hangs.
and unable to restart the processor once it is stopped.

empty queue will not empty all the queue but a single file is left off.
which can be cleared.
Need to restart NIFI and clear the queue.

I quite unsure if the problem is with configuring NIFI like flowfile
repository.
below attached is my xml template.

Thanks,
Raghu

Re: Need Help! Execute stream process not able to pick up files from queue.

[Raghu G <ra...@gmail.com>]

Python file does execute right.
Followed all the best practices mentioned in hortonworks.
(
https://community.hortonworks.com/articles/7882/hdfnifi-best-practices-for-setting-up-a-high-perfo.html
)

Currently I am using a dedicated linux server on amazon ec2. with config as
follows
no.of cores :8
ram:64gb
hardisk:160gb.

Thanks
Raghu

On Tue, Apr 26, 2016 at 12:17 AM, karthi keyan <ka...@gmail.com>
wrote:

> Ragu,
>
> Is NIFI is tuned for HEAP ? i have walk through your template you have get
> the file and distribute the file to multiple python files to process. In
> what case it get hangs does the Python file is executed rightly ?
>
> Best,
> Karthik
>
> On Tue, Apr 26, 2016 at 12:26 AM, Raghu G <ra...@gmail.com> wrote:
>
>> Hi all,
>>
>> Case1:
>> While working with a single file or couple of files flow seems to be
>> working good.
>>
>> Case2:
>> While working with large number of files with 17mb/queue and total of 5
>> queues (around 75mb)
>>
>> file gets into queue but execute processor doesn't pick up the file and
>> hangs.
>> and unable to restart the processor once it is stopped.
>>
>> empty queue will not empty all the queue but a single file is left off.
>> which can be cleared.
>> Need to restart NIFI and clear the queue.
>>
>> I quite unsure if the problem is with configuring NIFI like flowfile
>> repository.
>> below attached is my xml template.
>>
>> Thanks,
>> Raghu
>>
>
>

Re: Need Help! Execute stream process not able to pick up files from queue.

[karthi keyan <ka...@gmail.com>]

Ragu,

Is NIFI is tuned for HEAP ? i have walk through your template you have get
the file and distribute the file to multiple python files to process. In
what case it get hangs does the Python file is executed rightly ?

Best,
Karthik

On Tue, Apr 26, 2016 at 12:26 AM, Raghu G <ra...@gmail.com> wrote:

> Hi all,
>
> Case1:
> While working with a single file or couple of files flow seems to be
> working good.
>
> Case2:
> While working with large number of files with 17mb/queue and total of 5
> queues (around 75mb)
>
> file gets into queue but execute processor doesn't pick up the file and
> hangs.
> and unable to restart the processor once it is stopped.
>
> empty queue will not empty all the queue but a single file is left off.
> which can be cleared.
> Need to restart NIFI and clear the queue.
>
> I quite unsure if the problem is with configuring NIFI like flowfile
> repository.
> below attached is my xml template.
>
> Thanks,
> Raghu
>