You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nifi.apache.org by Edgardo Vega <ed...@gmail.com> on 2015/05/06 21:22:51 UTC
Threads
The admin guide says the Maximum Forked Processes section says "NiFi may
be configured to generate a significant number of threads." How is this
done? I read the rest of the documentation and didn't see a way to define
the number of threads to use. Also any recommendation on number of threads
per code?
I have been seeing the number of threads in the gui pegged at 10 that is
why I went looking for how to change that number.
Thanks in advance,
Edgardo
Re: Threads
Posted by Joe Witt <jo...@gmail.com>.
Edgardo
It's not really a clear winner in either place. Your intuition here
is totally fair and indeed it used to be a nifi properties config
item. Then we had stronger cases for it being a flow specific
configuration item. I think the reality is that you need a bit of
both. The total number of available threads can impact the way
someone designing a flow thinks. This is seen in how they allocate a
number of 'concurrent tasks' to a given processor. This means it is
in a sense more of a 'flow configuration' item. But you're absolutely
right that the other way of looking at this is that the total number
of threads will often be a function of the capabilities of a given
system.
So, I'd propose us supporting both if this turns out to be worth
tackling. The current method works well. But we could add back a
configuration property item that is more tied to the 'system' than the
flow and that property would be 'max threads' or something. Then at
runtime we'd set the actual 'max threads' to the lesser of the flow
configured value or the nifi properties value.
Thanks
Joe
On Wed, May 6, 2015 at 7:07 PM, Matt Gilman <ma...@gmail.com> wrote:
> Ah, yes Corey you're totally right. Sorry about that. Also wanted to point out that starting with this upcoming release (0.1.0-incubating) the controller services and reporting task will configurable in the UI and become part of the flow configuration.
>
> Sent from my iPhone
>
>> On May 6, 2015, at 6:23 PM, Corey Flowers <cf...@onyxpoint.com> wrote:
>>
>> Hey guys,
>>
>> The flow.xml.gz should not be updated by puppet. Puppet is great for
>> config files, controller services, reporting tasks, really anything
>> "variable/value" or "tag/value" like that but not the flow.xml.gz. It is
>> constantly changed and dynamic in its nature, which doesn't lend well to
>> the overall concept of puppet manifest control. Just a heads up.
>>
>> I will let you guys decide about the idea to make these values config
>> variables.
>>
>> Later!
>>
>>
>>> On Wednesday, May 6, 2015, Edgardo Vega <ed...@gmail.com> wrote:
>>>
>>> That's an interesting place to store that information wouldn't it make more
>>> sense to be in nifi.properties? If your sharing the flow.xml.gz on multiple
>>> machines with different specs you would have to manually tweak the flow on
>>> each machine.
>>>
>>> I would gladly make a ticket if it seems to make more sense for these
>>> properties to be moved. What's the consensus?
>>>
>>> On Wednesday, May 6, 2015, Matt Gilman <matt.c.gilman@gmail.com
>>> <javascript:;>> wrote:
>>>
>>>> It's actually saved as part of your flow. So assuming your flow is part
>>> of
>>>> your puppet configuration, you should be good. In a standalone instance
>>> it
>>>> would be in <NIFI_HOME>/conf/flow.xml.gz.
>>>>
>>>> Matt
>>>>
>>>> On Wed, May 6, 2015 at 3:47 PM, Edgardo Vega <edgardo.vega@gmail.com
>>> <javascript:;>
>>>> <javascript:;>> wrote:
>>>>
>>>>> Thanks Matt. Is this configurable in a file or through a java argument?
>>>>> That way we can set that via puppet.
>>>>>
>>>>> Cheers,
>>>>>
>>>>> Edgardo
>>>>>
>>>>> On Wed, May 6, 2015 at 3:31 PM, Matt Gilman <matt.c.gilman@gmail.com
>>> <javascript:;>
>>>> <javascript:;>>
>>>>> wrote:
>>>>>
>>>>>> The existing documentation is a little short but it is mentioned in
>>> the
>>>>>> Other Management Features section [1]. In the UI can you click the
>>>> wrench
>>>>>> and screwdriver icon in the upper right. There you can set the name
>>> of
>>>>> your
>>>>>> dataflow, it's description, and the size of the timer/event driven
>>>> thread
>>>>>> pools.
>>>>>>
>>>>>> Matt
>>>>>>
>>>>>> [1]
>>> https://nifi.incubator.apache.org/docs/nifi-docs/user-guide.html#other_management_features
>>>>>>
>>>>>> On Wed, May 6, 2015 at 3:22 PM, Edgardo Vega <edgardo.vega@gmail.com
>>> <javascript:;>
>>>> <javascript:;>>
>>>>>> wrote:
>>>>>>
>>>>>>> The admin guide says the Maximum Forked Processes section says
>>> "NiFi
>>>>> may
>>>>>>> be configured to generate a significant number of threads." How is
>>>> this
>>>>>>> done? I read the rest of the documentation and didn't see a way to
>>>>> define
>>>>>>> the number of threads to use. Also any recommendation on number of
>>>>>> threads
>>>>>>> per code?
>>>>>>>
>>>>>>> I have been seeing the number of threads in the gui pegged at 10
>>> that
>>>>> is
>>>>>>> why I went looking for how to change that number.
>>>>>>>
>>>>>>> Thanks in advance,
>>>>>>>
>>>>>>> Edgardo
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Cheers,
>>>>>
>>>>> Edgardo
>>>
>>>
>>> --
>>> Cheers,
>>>
>>> Edgardo
>>>
>>> Sent from Gmail Mobile
>>
>>
>> --
>> Corey Flowers
>> Vice President, Onyx Point, Inc
>> (410) 541-6699
>> cflowers@onyxpoint.com
>>
>> -- This account not approved for unencrypted proprietary information --
Re: Threads
Posted by Matt Gilman <ma...@gmail.com>.
Ah, yes Corey you're totally right. Sorry about that. Also wanted to point out that starting with this upcoming release (0.1.0-incubating) the controller services and reporting task will configurable in the UI and become part of the flow configuration.
Sent from my iPhone
> On May 6, 2015, at 6:23 PM, Corey Flowers <cf...@onyxpoint.com> wrote:
>
> Hey guys,
>
> The flow.xml.gz should not be updated by puppet. Puppet is great for
> config files, controller services, reporting tasks, really anything
> "variable/value" or "tag/value" like that but not the flow.xml.gz. It is
> constantly changed and dynamic in its nature, which doesn't lend well to
> the overall concept of puppet manifest control. Just a heads up.
>
> I will let you guys decide about the idea to make these values config
> variables.
>
> Later!
>
>
>> On Wednesday, May 6, 2015, Edgardo Vega <ed...@gmail.com> wrote:
>>
>> That's an interesting place to store that information wouldn't it make more
>> sense to be in nifi.properties? If your sharing the flow.xml.gz on multiple
>> machines with different specs you would have to manually tweak the flow on
>> each machine.
>>
>> I would gladly make a ticket if it seems to make more sense for these
>> properties to be moved. What's the consensus?
>>
>> On Wednesday, May 6, 2015, Matt Gilman <matt.c.gilman@gmail.com
>> <javascript:;>> wrote:
>>
>>> It's actually saved as part of your flow. So assuming your flow is part
>> of
>>> your puppet configuration, you should be good. In a standalone instance
>> it
>>> would be in <NIFI_HOME>/conf/flow.xml.gz.
>>>
>>> Matt
>>>
>>> On Wed, May 6, 2015 at 3:47 PM, Edgardo Vega <edgardo.vega@gmail.com
>> <javascript:;>
>>> <javascript:;>> wrote:
>>>
>>>> Thanks Matt. Is this configurable in a file or through a java argument?
>>>> That way we can set that via puppet.
>>>>
>>>> Cheers,
>>>>
>>>> Edgardo
>>>>
>>>> On Wed, May 6, 2015 at 3:31 PM, Matt Gilman <matt.c.gilman@gmail.com
>> <javascript:;>
>>> <javascript:;>>
>>>> wrote:
>>>>
>>>>> The existing documentation is a little short but it is mentioned in
>> the
>>>>> Other Management Features section [1]. In the UI can you click the
>>> wrench
>>>>> and screwdriver icon in the upper right. There you can set the name
>> of
>>>> your
>>>>> dataflow, it's description, and the size of the timer/event driven
>>> thread
>>>>> pools.
>>>>>
>>>>> Matt
>>>>>
>>>>> [1]
>> https://nifi.incubator.apache.org/docs/nifi-docs/user-guide.html#other_management_features
>>>>>
>>>>> On Wed, May 6, 2015 at 3:22 PM, Edgardo Vega <edgardo.vega@gmail.com
>> <javascript:;>
>>> <javascript:;>>
>>>>> wrote:
>>>>>
>>>>>> The admin guide says the Maximum Forked Processes section says
>> "NiFi
>>>> may
>>>>>> be configured to generate a significant number of threads." How is
>>> this
>>>>>> done? I read the rest of the documentation and didn't see a way to
>>>> define
>>>>>> the number of threads to use. Also any recommendation on number of
>>>>> threads
>>>>>> per code?
>>>>>>
>>>>>> I have been seeing the number of threads in the gui pegged at 10
>> that
>>>> is
>>>>>> why I went looking for how to change that number.
>>>>>>
>>>>>> Thanks in advance,
>>>>>>
>>>>>> Edgardo
>>>>
>>>>
>>>>
>>>> --
>>>> Cheers,
>>>>
>>>> Edgardo
>>
>>
>> --
>> Cheers,
>>
>> Edgardo
>>
>> Sent from Gmail Mobile
>
>
> --
> Corey Flowers
> Vice President, Onyx Point, Inc
> (410) 541-6699
> cflowers@onyxpoint.com
>
> -- This account not approved for unencrypted proprietary information --
Re: Threads
Posted by Corey Flowers <cf...@onyxpoint.com>.
Hey guys,
The flow.xml.gz should not be updated by puppet. Puppet is great for
config files, controller services, reporting tasks, really anything
"variable/value" or "tag/value" like that but not the flow.xml.gz. It is
constantly changed and dynamic in its nature, which doesn't lend well to
the overall concept of puppet manifest control. Just a heads up.
I will let you guys decide about the idea to make these values config
variables.
Later!
On Wednesday, May 6, 2015, Edgardo Vega <ed...@gmail.com> wrote:
> That's an interesting place to store that information wouldn't it make more
> sense to be in nifi.properties? If your sharing the flow.xml.gz on multiple
> machines with different specs you would have to manually tweak the flow on
> each machine.
>
> I would gladly make a ticket if it seems to make more sense for these
> properties to be moved. What's the consensus?
>
> On Wednesday, May 6, 2015, Matt Gilman <matt.c.gilman@gmail.com
> <javascript:;>> wrote:
>
> > It's actually saved as part of your flow. So assuming your flow is part
> of
> > your puppet configuration, you should be good. In a standalone instance
> it
> > would be in <NIFI_HOME>/conf/flow.xml.gz.
> >
> > Matt
> >
> > On Wed, May 6, 2015 at 3:47 PM, Edgardo Vega <edgardo.vega@gmail.com
> <javascript:;>
> > <javascript:;>> wrote:
> >
> > > Thanks Matt. Is this configurable in a file or through a java argument?
> > > That way we can set that via puppet.
> > >
> > > Cheers,
> > >
> > > Edgardo
> > >
> > > On Wed, May 6, 2015 at 3:31 PM, Matt Gilman <matt.c.gilman@gmail.com
> <javascript:;>
> > <javascript:;>>
> > > wrote:
> > >
> > > > The existing documentation is a little short but it is mentioned in
> the
> > > > Other Management Features section [1]. In the UI can you click the
> > wrench
> > > > and screwdriver icon in the upper right. There you can set the name
> of
> > > your
> > > > dataflow, it's description, and the size of the timer/event driven
> > thread
> > > > pools.
> > > >
> > > > Matt
> > > >
> > > > [1]
> > > >
> > > >
> > >
> >
> https://nifi.incubator.apache.org/docs/nifi-docs/user-guide.html#other_management_features
> > > >
> > > > On Wed, May 6, 2015 at 3:22 PM, Edgardo Vega <edgardo.vega@gmail.com
> <javascript:;>
> > <javascript:;>>
> > > > wrote:
> > > >
> > > > > The admin guide says the Maximum Forked Processes section says
> "NiFi
> > > may
> > > > > be configured to generate a significant number of threads." How is
> > this
> > > > > done? I read the rest of the documentation and didn't see a way to
> > > define
> > > > > the number of threads to use. Also any recommendation on number of
> > > > threads
> > > > > per code?
> > > > >
> > > > > I have been seeing the number of threads in the gui pegged at 10
> that
> > > is
> > > > > why I went looking for how to change that number.
> > > > >
> > > > > Thanks in advance,
> > > > >
> > > > > Edgardo
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > Cheers,
> > >
> > > Edgardo
> > >
> >
>
>
> --
> Cheers,
>
> Edgardo
>
> Sent from Gmail Mobile
>
--
Corey Flowers
Vice President, Onyx Point, Inc
(410) 541-6699
cflowers@onyxpoint.com
-- This account not approved for unencrypted proprietary information --
Re: Threads
Posted by Edgardo Vega <ed...@gmail.com>.
That's an interesting place to store that information wouldn't it make more
sense to be in nifi.properties? If your sharing the flow.xml.gz on multiple
machines with different specs you would have to manually tweak the flow on
each machine.
I would gladly make a ticket if it seems to make more sense for these
properties to be moved. What's the consensus?
On Wednesday, May 6, 2015, Matt Gilman <ma...@gmail.com> wrote:
> It's actually saved as part of your flow. So assuming your flow is part of
> your puppet configuration, you should be good. In a standalone instance it
> would be in <NIFI_HOME>/conf/flow.xml.gz.
>
> Matt
>
> On Wed, May 6, 2015 at 3:47 PM, Edgardo Vega <edgardo.vega@gmail.com
> <javascript:;>> wrote:
>
> > Thanks Matt. Is this configurable in a file or through a java argument?
> > That way we can set that via puppet.
> >
> > Cheers,
> >
> > Edgardo
> >
> > On Wed, May 6, 2015 at 3:31 PM, Matt Gilman <matt.c.gilman@gmail.com
> <javascript:;>>
> > wrote:
> >
> > > The existing documentation is a little short but it is mentioned in the
> > > Other Management Features section [1]. In the UI can you click the
> wrench
> > > and screwdriver icon in the upper right. There you can set the name of
> > your
> > > dataflow, it's description, and the size of the timer/event driven
> thread
> > > pools.
> > >
> > > Matt
> > >
> > > [1]
> > >
> > >
> >
> https://nifi.incubator.apache.org/docs/nifi-docs/user-guide.html#other_management_features
> > >
> > > On Wed, May 6, 2015 at 3:22 PM, Edgardo Vega <edgardo.vega@gmail.com
> <javascript:;>>
> > > wrote:
> > >
> > > > The admin guide says the Maximum Forked Processes section says "NiFi
> > may
> > > > be configured to generate a significant number of threads." How is
> this
> > > > done? I read the rest of the documentation and didn't see a way to
> > define
> > > > the number of threads to use. Also any recommendation on number of
> > > threads
> > > > per code?
> > > >
> > > > I have been seeing the number of threads in the gui pegged at 10 that
> > is
> > > > why I went looking for how to change that number.
> > > >
> > > > Thanks in advance,
> > > >
> > > > Edgardo
> > > >
> > >
> >
> >
> >
> > --
> > Cheers,
> >
> > Edgardo
> >
>
--
Cheers,
Edgardo
Sent from Gmail Mobile
Re: Threads
Posted by Matt Gilman <ma...@gmail.com>.
It's actually saved as part of your flow. So assuming your flow is part of
your puppet configuration, you should be good. In a standalone instance it
would be in <NIFI_HOME>/conf/flow.xml.gz.
Matt
On Wed, May 6, 2015 at 3:47 PM, Edgardo Vega <ed...@gmail.com> wrote:
> Thanks Matt. Is this configurable in a file or through a java argument?
> That way we can set that via puppet.
>
> Cheers,
>
> Edgardo
>
> On Wed, May 6, 2015 at 3:31 PM, Matt Gilman <ma...@gmail.com>
> wrote:
>
> > The existing documentation is a little short but it is mentioned in the
> > Other Management Features section [1]. In the UI can you click the wrench
> > and screwdriver icon in the upper right. There you can set the name of
> your
> > dataflow, it's description, and the size of the timer/event driven thread
> > pools.
> >
> > Matt
> >
> > [1]
> >
> >
> https://nifi.incubator.apache.org/docs/nifi-docs/user-guide.html#other_management_features
> >
> > On Wed, May 6, 2015 at 3:22 PM, Edgardo Vega <ed...@gmail.com>
> > wrote:
> >
> > > The admin guide says the Maximum Forked Processes section says "NiFi
> may
> > > be configured to generate a significant number of threads." How is this
> > > done? I read the rest of the documentation and didn't see a way to
> define
> > > the number of threads to use. Also any recommendation on number of
> > threads
> > > per code?
> > >
> > > I have been seeing the number of threads in the gui pegged at 10 that
> is
> > > why I went looking for how to change that number.
> > >
> > > Thanks in advance,
> > >
> > > Edgardo
> > >
> >
>
>
>
> --
> Cheers,
>
> Edgardo
>
Re: Threads
Posted by Edgardo Vega <ed...@gmail.com>.
Thanks Matt. Is this configurable in a file or through a java argument?
That way we can set that via puppet.
Cheers,
Edgardo
On Wed, May 6, 2015 at 3:31 PM, Matt Gilman <ma...@gmail.com> wrote:
> The existing documentation is a little short but it is mentioned in the
> Other Management Features section [1]. In the UI can you click the wrench
> and screwdriver icon in the upper right. There you can set the name of your
> dataflow, it's description, and the size of the timer/event driven thread
> pools.
>
> Matt
>
> [1]
>
> https://nifi.incubator.apache.org/docs/nifi-docs/user-guide.html#other_management_features
>
> On Wed, May 6, 2015 at 3:22 PM, Edgardo Vega <ed...@gmail.com>
> wrote:
>
> > The admin guide says the Maximum Forked Processes section says "NiFi may
> > be configured to generate a significant number of threads." How is this
> > done? I read the rest of the documentation and didn't see a way to define
> > the number of threads to use. Also any recommendation on number of
> threads
> > per code?
> >
> > I have been seeing the number of threads in the gui pegged at 10 that is
> > why I went looking for how to change that number.
> >
> > Thanks in advance,
> >
> > Edgardo
> >
>
--
Cheers,
Edgardo
Re: Threads
Posted by Matt Gilman <ma...@gmail.com>.
The existing documentation is a little short but it is mentioned in the
Other Management Features section [1]. In the UI can you click the wrench
and screwdriver icon in the upper right. There you can set the name of your
dataflow, it's description, and the size of the timer/event driven thread
pools.
Matt
[1]
https://nifi.incubator.apache.org/docs/nifi-docs/user-guide.html#other_management_features
On Wed, May 6, 2015 at 3:22 PM, Edgardo Vega <ed...@gmail.com> wrote:
> The admin guide says the Maximum Forked Processes section says "NiFi may
> be configured to generate a significant number of threads." How is this
> done? I read the rest of the documentation and didn't see a way to define
> the number of threads to use. Also any recommendation on number of threads
> per code?
>
> I have been seeing the number of threads in the gui pegged at 10 that is
> why I went looking for how to change that number.
>
> Thanks in advance,
>
> Edgardo
>