You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Yuta Morisawa <yu...@kddi-research.jp> on 2017/12/05 04:13:26 UTC
CPU Cores of JobManager
Hi
Now I am looking for the way to increase the number of allocated CPU
cores because my JobManagaer WEBUI is very heavy and sometimes freeze.
I think this is caused by the resource shortage of JobManager.
How can I increase the number of CPU for JobManager in YARN mode?
Thanks
Yuta
--
Re: CPU Cores of JobManager
Posted by Yuta Morisawa <yu...@kddi-research.jp>.
Thank you for information.
I will investigate my cluster more carefully.
Anyway, at this moment, the WEBUI works well(I don't know why).
If I get some updates, I will inform you.
By the way, I have another problem...
When I run multiple Flink jobs in YARN, the job-logs are mixed.
cf) yarn logs -applicationId <id>
How can I avoid it?
Regards,
Yuta
On 2017/12/06 18:52, Chesnay Schepler wrote:
> That said, metrics could still be the issue, with FLINK-7368
> <https://issues.apache.org/jira/browse/FLINK-7368> being the prime
> suspect. This was fixed for 1.3.3 and 1.4 (both unreleased though).
>
> On 06.12.2017 10:36, Chesnay Schepler wrote:
>> This can only have an effect if "*taskmanager.net.detailed-metrics*"
>> is enabled (by default it is disabled).
>> Otherwise an increase in sub-channels will neither increase the number
>> of metrics nor the number of polls by the UI.
>>
>> On 06.12.2017 10:22, Stephan Ewen wrote:
>>> A 'rebalance()' should no impose load on the JM itself. There are
>>> more sub-channels in that case, but the JobManager is not
>>> coordinating anything for them unless there is a failure/recovery.
>>>
>>> Each subchannel generates additional metrics, though - might that be
>>> an issue?
>>>
>>> On Wed, Dec 6, 2017 at 9:50 AM, Yuta Morisawa
>>> <yu-morisawa@kddi-research.jp <ma...@kddi-research.jp>>
>>> wrote:
>>>
>>> > Which version of Fllink are you using?
>>> I'm using Flink 1.3.0.
>>>
>>> > Is the high CPU usage caused by the browser or the JVM?
>>> The CPU usage of browser is not high, the JVM is not also.
>>>
>>> I found that if I remove the rebalance function in my code, the
>>> WEBUI works well.
>>> I don't know why but I think rebalance function imposes a load on
>>> JobManager.
>>>
>>>
>>> Regards,
>>> Yuta
>>>
>>>
>>> On 2017/12/06 17:08, Chesnay Schepler wrote:
>>>
>>> Which version of Fllink are you using?
>>>
>>> Is the high CPU usage caused by the browser or the JVM?
>>>
>>> One thing we should investigate is how many requests the UI
>>> sends to the JobManager,
>>> but I don't know of a good way to do that without modifying
>>> the source.
>>>
>>> On 06.12.2017 01 <tel:06.12.2017%2001>:20, Yuta Morisawa wrote:
>>>
>>> Hi Timo
>>>
>>> I execute streaming job without checkpointing and I don't
>>> configure any state backend, so it may be
>>> "MemoryStateBackend".
>>>
>>> Actually, my streaming app just reads data from kafka and
>>> writes it to an external DB. Its not so complicated.
>>>
>>> Regards,
>>> Yuta
>>>
>>> On 2017/12/05 19:55, Timo Walther wrote:
>>>
>>> I had some profiling tool like jvisualvm in mind. Are
>>> you executing streaming or batch jobs? If streaming,
>>> is checkpointing enabled and which type of statebackend?
>>>
>>> @Chesnay do you have experience with slow behavior of
>>> the Web UI?
>>>
>>> Regards,
>>> Timo
>>>
>>>
>>> Am 12/5/17 um 10:37 AM schrieb Yuta Morisawa:
>>>
>>> Hi Timo
>>>
>>> Thank you for your early reply.
>>>
>>> These are commands which I run my apps.
>>> ./bin/yarn-session.sh -n 20 -jm 6000 -tm 24000 -s 10
>>> ./bin/flink run -p 100 <jar1> <jar1 parameters>
>>> ./bin/flink run -p 100 <jar2> <jar2 parameters>
>>>
>>> So, JobManager Heap Memory = 6000 MB and it
>>> manages 2 jobs.
>>>
>>> > Maybe you can use a profiler and find out which
>>> component consumes so
>>> > much CPU resources?
>>> You mean Java Flight Recorder or JITWatch?
>>> Or, Flink has original profiler?
>>> https://ci.apache.org/projects/flink/flink-docs-release-1.3/monitoring/application_profiling.html
>>> <https://ci.apache.org/projects/flink/flink-docs-release-1.3/monitoring/application_profiling.html>
>>>
>>>
>>> Regards,
>>> Yuta
>>>
>>> On 2017/12/05 18:02, Timo Walther wrote:
>>>
>>> Hi Yuta,
>>>
>>> as far as I know you cannot assign more cores
>>> to a JobManager.
>>>
>>> Can you tell us a bit more about your
>>> environment? How many jobs does the
>>> JobManager has to manage? How much heap
>>> memory is assigned to the JobManager?
>>>
>>> Maybe you can use a profiler and find out
>>> which component consumes so much CPU resources?
>>>
>>> Regards,
>>> Timo
>>>
>>>
>>> Am 12/5/17 um 5:13 AM schrieb Yuta Morisawa:
>>>
>>> Hi
>>>
>>> Now I am looking for the way to increase
>>> the number of allocated CPU cores because
>>> my JobManagaer WEBUI is very heavy and
>>> sometimes freeze.
>>>
>>> I think this is caused by the resource
>>> shortage of JobManager.
>>> How can I increase the number of CPU for
>>> JobManager in YARN mode?
>>>
>>>
>>>
>>> Thanks
>>> Yuta
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>
>
Re: CPU Cores of JobManager
Posted by Chesnay Schepler <ch...@apache.org>.
That said, metrics could still be the issue, with FLINK-7368
<https://issues.apache.org/jira/browse/FLINK-7368> being the prime
suspect. This was fixed for 1.3.3 and 1.4 (both unreleased though).
On 06.12.2017 10:36, Chesnay Schepler wrote:
> This can only have an effect if "*taskmanager.net.detailed-metrics*"
> is enabled (by default it is disabled).
> Otherwise an increase in sub-channels will neither increase the number
> of metrics nor the number of polls by the UI.
>
> On 06.12.2017 10:22, Stephan Ewen wrote:
>> A 'rebalance()' should no impose load on the JM itself. There are
>> more sub-channels in that case, but the JobManager is not
>> coordinating anything for them unless there is a failure/recovery.
>>
>> Each subchannel generates additional metrics, though - might that be
>> an issue?
>>
>> On Wed, Dec 6, 2017 at 9:50 AM, Yuta Morisawa
>> <yu-morisawa@kddi-research.jp <ma...@kddi-research.jp>>
>> wrote:
>>
>> > Which version of Fllink are you using?
>> I'm using Flink 1.3.0.
>>
>> > Is the high CPU usage caused by the browser or the JVM?
>> The CPU usage of browser is not high, the JVM is not also.
>>
>> I found that if I remove the rebalance function in my code, the
>> WEBUI works well.
>> I don't know why but I think rebalance function imposes a load on
>> JobManager.
>>
>>
>> Regards,
>> Yuta
>>
>>
>> On 2017/12/06 17:08, Chesnay Schepler wrote:
>>
>> Which version of Fllink are you using?
>>
>> Is the high CPU usage caused by the browser or the JVM?
>>
>> One thing we should investigate is how many requests the UI
>> sends to the JobManager,
>> but I don't know of a good way to do that without modifying
>> the source.
>>
>> On 06.12.2017 01 <tel:06.12.2017%2001>:20, Yuta Morisawa wrote:
>>
>> Hi Timo
>>
>> I execute streaming job without checkpointing and I don't
>> configure any state backend, so it may be
>> "MemoryStateBackend".
>>
>> Actually, my streaming app just reads data from kafka and
>> writes it to an external DB. Its not so complicated.
>>
>> Regards,
>> Yuta
>>
>> On 2017/12/05 19:55, Timo Walther wrote:
>>
>> I had some profiling tool like jvisualvm in mind. Are
>> you executing streaming or batch jobs? If streaming,
>> is checkpointing enabled and which type of statebackend?
>>
>> @Chesnay do you have experience with slow behavior of
>> the Web UI?
>>
>> Regards,
>> Timo
>>
>>
>> Am 12/5/17 um 10:37 AM schrieb Yuta Morisawa:
>>
>> Hi Timo
>>
>> Thank you for your early reply.
>>
>> These are commands which I run my apps.
>> ./bin/yarn-session.sh -n 20 -jm 6000 -tm 24000 -s 10
>> ./bin/flink run -p 100 <jar1> <jar1 parameters>
>> ./bin/flink run -p 100 <jar2> <jar2 parameters>
>>
>> So, JobManager Heap Memory = 6000 MB and it
>> manages 2 jobs.
>>
>> > Maybe you can use a profiler and find out which
>> component consumes so
>> > much CPU resources?
>> You mean Java Flight Recorder or JITWatch?
>> Or, Flink has original profiler?
>> https://ci.apache.org/projects/flink/flink-docs-release-1.3/monitoring/application_profiling.html
>> <https://ci.apache.org/projects/flink/flink-docs-release-1.3/monitoring/application_profiling.html>
>>
>>
>> Regards,
>> Yuta
>>
>> On 2017/12/05 18:02, Timo Walther wrote:
>>
>> Hi Yuta,
>>
>> as far as I know you cannot assign more cores
>> to a JobManager.
>>
>> Can you tell us a bit more about your
>> environment? How many jobs does the
>> JobManager has to manage? How much heap
>> memory is assigned to the JobManager?
>>
>> Maybe you can use a profiler and find out
>> which component consumes so much CPU resources?
>>
>> Regards,
>> Timo
>>
>>
>> Am 12/5/17 um 5:13 AM schrieb Yuta Morisawa:
>>
>> Hi
>>
>> Now I am looking for the way to increase
>> the number of allocated CPU cores because
>> my JobManagaer WEBUI is very heavy and
>> sometimes freeze.
>>
>> I think this is caused by the resource
>> shortage of JobManager.
>> How can I increase the number of CPU for
>> JobManager in YARN mode?
>>
>>
>>
>> Thanks
>> Yuta
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>
Re: CPU Cores of JobManager
Posted by Chesnay Schepler <ch...@apache.org>.
This can only have an effect if "*taskmanager.net.detailed-metrics*" is
enabled (by default it is disabled).
Otherwise an increase in sub-channels will neither increase the number
of metrics nor the number of polls by the UI.
On 06.12.2017 10:22, Stephan Ewen wrote:
> A 'rebalance()' should no impose load on the JM itself. There are more
> sub-channels in that case, but the JobManager is not coordinating
> anything for them unless there is a failure/recovery.
>
> Each subchannel generates additional metrics, though - might that be
> an issue?
>
> On Wed, Dec 6, 2017 at 9:50 AM, Yuta Morisawa
> <yu-morisawa@kddi-research.jp <ma...@kddi-research.jp>>
> wrote:
>
> > Which version of Fllink are you using?
> I'm using Flink 1.3.0.
>
> > Is the high CPU usage caused by the browser or the JVM?
> The CPU usage of browser is not high, the JVM is not also.
>
> I found that if I remove the rebalance function in my code, the
> WEBUI works well.
> I don't know why but I think rebalance function imposes a load on
> JobManager.
>
>
> Regards,
> Yuta
>
>
> On 2017/12/06 17:08, Chesnay Schepler wrote:
>
> Which version of Fllink are you using?
>
> Is the high CPU usage caused by the browser or the JVM?
>
> One thing we should investigate is how many requests the UI
> sends to the JobManager,
> but I don't know of a good way to do that without modifying
> the source.
>
> On 06.12.2017 01 <tel:06.12.2017%2001>:20, Yuta Morisawa wrote:
>
> Hi Timo
>
> I execute streaming job without checkpointing and I don't
> configure any state backend, so it may be
> "MemoryStateBackend".
>
> Actually, my streaming app just reads data from kafka and
> writes it to an external DB. Its not so complicated.
>
> Regards,
> Yuta
>
> On 2017/12/05 19:55, Timo Walther wrote:
>
> I had some profiling tool like jvisualvm in mind. Are
> you executing streaming or batch jobs? If streaming,
> is checkpointing enabled and which type of statebackend?
>
> @Chesnay do you have experience with slow behavior of
> the Web UI?
>
> Regards,
> Timo
>
>
> Am 12/5/17 um 10:37 AM schrieb Yuta Morisawa:
>
> Hi Timo
>
> Thank you for your early reply.
>
> These are commands which I run my apps.
> ./bin/yarn-session.sh -n 20 -jm 6000 -tm 24000 -s 10
> ./bin/flink run -p 100 <jar1> <jar1 parameters>
> ./bin/flink run -p 100 <jar2> <jar2 parameters>
>
> So, JobManager Heap Memory = 6000 MB and it
> manages 2 jobs.
>
> > Maybe you can use a profiler and find out which
> component consumes so
> > much CPU resources?
> You mean Java Flight Recorder or JITWatch?
> Or, Flink has original profiler?
> https://ci.apache.org/projects/flink/flink-docs-release-1.3/monitoring/application_profiling.html
> <https://ci.apache.org/projects/flink/flink-docs-release-1.3/monitoring/application_profiling.html>
>
>
> Regards,
> Yuta
>
> On 2017/12/05 18:02, Timo Walther wrote:
>
> Hi Yuta,
>
> as far as I know you cannot assign more cores
> to a JobManager.
>
> Can you tell us a bit more about your
> environment? How many jobs does the JobManager
> has to manage? How much heap memory is
> assigned to the JobManager?
>
> Maybe you can use a profiler and find out
> which component consumes so much CPU resources?
>
> Regards,
> Timo
>
>
> Am 12/5/17 um 5:13 AM schrieb Yuta Morisawa:
>
> Hi
>
> Now I am looking for the way to increase
> the number of allocated CPU cores because
> my JobManagaer WEBUI is very heavy and
> sometimes freeze.
>
> I think this is caused by the resource
> shortage of JobManager.
> How can I increase the number of CPU for
> JobManager in YARN mode?
>
>
>
> Thanks
> Yuta
>
>
>
>
>
>
>
>
>
>
>
>
Re: CPU Cores of JobManager
Posted by Stephan Ewen <se...@apache.org>.
A 'rebalance()' should no impose load on the JM itself. There are more
sub-channels in that case, but the JobManager is not coordinating anything
for them unless there is a failure/recovery.
Each subchannel generates additional metrics, though - might that be an
issue?
On Wed, Dec 6, 2017 at 9:50 AM, Yuta Morisawa <yu...@kddi-research.jp>
wrote:
> > Which version of Fllink are you using?
> I'm using Flink 1.3.0.
>
> > Is the high CPU usage caused by the browser or the JVM?
> The CPU usage of browser is not high, the JVM is not also.
>
> I found that if I remove the rebalance function in my code, the WEBUI
> works well.
> I don't know why but I think rebalance function imposes a load on
> JobManager.
>
>
> Regards,
> Yuta
>
>
> On 2017/12/06 17:08, Chesnay Schepler wrote:
>
>> Which version of Fllink are you using?
>>
>> Is the high CPU usage caused by the browser or the JVM?
>>
>> One thing we should investigate is how many requests the UI sends to the
>> JobManager,
>> but I don't know of a good way to do that without modifying the source.
>>
>> On 06.12.2017 01:20, Yuta Morisawa wrote:
>>
>>> Hi Timo
>>>
>>> I execute streaming job without checkpointing and I don't configure any
>>> state backend, so it may be "MemoryStateBackend".
>>>
>>> Actually, my streaming app just reads data from kafka and writes it to
>>> an external DB. Its not so complicated.
>>>
>>> Regards,
>>> Yuta
>>>
>>> On 2017/12/05 19:55, Timo Walther wrote:
>>>
>>>> I had some profiling tool like jvisualvm in mind. Are you executing
>>>> streaming or batch jobs? If streaming, is checkpointing enabled and which
>>>> type of statebackend?
>>>>
>>>> @Chesnay do you have experience with slow behavior of the Web UI?
>>>>
>>>> Regards,
>>>> Timo
>>>>
>>>>
>>>> Am 12/5/17 um 10:37 AM schrieb Yuta Morisawa:
>>>>
>>>>> Hi Timo
>>>>>
>>>>> Thank you for your early reply.
>>>>>
>>>>> These are commands which I run my apps.
>>>>> ./bin/yarn-session.sh -n 20 -jm 6000 -tm 24000 -s 10
>>>>> ./bin/flink run -p 100 <jar1> <jar1 parameters>
>>>>> ./bin/flink run -p 100 <jar2> <jar2 parameters>
>>>>>
>>>>> So, JobManager Heap Memory = 6000 MB and it manages 2 jobs.
>>>>>
>>>>> > Maybe you can use a profiler and find out which component consumes so
>>>>> > much CPU resources?
>>>>> You mean Java Flight Recorder or JITWatch?
>>>>> Or, Flink has original profiler?
>>>>> https://ci.apache.org/projects/flink/flink-docs-release-1.3/
>>>>> monitoring/application_profiling.html
>>>>>
>>>>> Regards,
>>>>> Yuta
>>>>>
>>>>> On 2017/12/05 18:02, Timo Walther wrote:
>>>>>
>>>>>> Hi Yuta,
>>>>>>
>>>>>> as far as I know you cannot assign more cores to a JobManager.
>>>>>>
>>>>>> Can you tell us a bit more about your environment? How many jobs does
>>>>>> the JobManager has to manage? How much heap memory is assigned to the
>>>>>> JobManager?
>>>>>>
>>>>>> Maybe you can use a profiler and find out which component consumes so
>>>>>> much CPU resources?
>>>>>>
>>>>>> Regards,
>>>>>> Timo
>>>>>>
>>>>>>
>>>>>> Am 12/5/17 um 5:13 AM schrieb Yuta Morisawa:
>>>>>>
>>>>>>> Hi
>>>>>>>
>>>>>>> Now I am looking for the way to increase the number of allocated CPU
>>>>>>> cores because my JobManagaer WEBUI is very heavy and sometimes freeze.
>>>>>>>
>>>>>>> I think this is caused by the resource shortage of JobManager.
>>>>>>> How can I increase the number of CPU for JobManager in YARN mode?
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Thanks
>>>>>>> Yuta
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>
>>>>
>>>
>>>
>>
>>
>
Re: CPU Cores of JobManager
Posted by Yuta Morisawa <yu...@kddi-research.jp>.
> Which version of Fllink are you using?
I'm using Flink 1.3.0.
> Is the high CPU usage caused by the browser or the JVM?
The CPU usage of browser is not high, the JVM is not also.
I found that if I remove the rebalance function in my code, the WEBUI
works well.
I don't know why but I think rebalance function imposes a load on
JobManager.
Regards,
Yuta
On 2017/12/06 17:08, Chesnay Schepler wrote:
> Which version of Fllink are you using?
>
> Is the high CPU usage caused by the browser or the JVM?
>
> One thing we should investigate is how many requests the UI sends to the
> JobManager,
> but I don't know of a good way to do that without modifying the source.
>
> On 06.12.2017 01:20, Yuta Morisawa wrote:
>> Hi Timo
>>
>> I execute streaming job without checkpointing and I don't configure
>> any state backend, so it may be "MemoryStateBackend".
>>
>> Actually, my streaming app just reads data from kafka and writes it to
>> an external DB. Its not so complicated.
>>
>> Regards,
>> Yuta
>>
>> On 2017/12/05 19:55, Timo Walther wrote:
>>> I had some profiling tool like jvisualvm in mind. Are you executing
>>> streaming or batch jobs? If streaming, is checkpointing enabled and
>>> which type of statebackend?
>>>
>>> @Chesnay do you have experience with slow behavior of the Web UI?
>>>
>>> Regards,
>>> Timo
>>>
>>>
>>> Am 12/5/17 um 10:37 AM schrieb Yuta Morisawa:
>>>> Hi Timo
>>>>
>>>> Thank you for your early reply.
>>>>
>>>> These are commands which I run my apps.
>>>> ./bin/yarn-session.sh -n 20 -jm 6000 -tm 24000 -s 10
>>>> ./bin/flink run -p 100 <jar1> <jar1 parameters>
>>>> ./bin/flink run -p 100 <jar2> <jar2 parameters>
>>>>
>>>> So, JobManager Heap Memory = 6000 MB and it manages 2 jobs.
>>>>
>>>> > Maybe you can use a profiler and find out which component consumes so
>>>> > much CPU resources?
>>>> You mean Java Flight Recorder or JITWatch?
>>>> Or, Flink has original profiler?
>>>> https://ci.apache.org/projects/flink/flink-docs-release-1.3/monitoring/application_profiling.html
>>>>
>>>>
>>>> Regards,
>>>> Yuta
>>>>
>>>> On 2017/12/05 18:02, Timo Walther wrote:
>>>>> Hi Yuta,
>>>>>
>>>>> as far as I know you cannot assign more cores to a JobManager.
>>>>>
>>>>> Can you tell us a bit more about your environment? How many jobs
>>>>> does the JobManager has to manage? How much heap memory is assigned
>>>>> to the JobManager?
>>>>>
>>>>> Maybe you can use a profiler and find out which component consumes
>>>>> so much CPU resources?
>>>>>
>>>>> Regards,
>>>>> Timo
>>>>>
>>>>>
>>>>> Am 12/5/17 um 5:13 AM schrieb Yuta Morisawa:
>>>>>> Hi
>>>>>>
>>>>>> Now I am looking for the way to increase the number of allocated
>>>>>> CPU cores because my JobManagaer WEBUI is very heavy and sometimes
>>>>>> freeze.
>>>>>>
>>>>>> I think this is caused by the resource shortage of JobManager.
>>>>>> How can I increase the number of CPU for JobManager in YARN mode?
>>>>>>
>>>>>>
>>>>>>
>>>>>> Thanks
>>>>>> Yuta
>>>>>
>>>>>
>>>>>
>>>
>>>
>>
>>
>
>
Re: CPU Cores of JobManager
Posted by Chesnay Schepler <ch...@apache.org>.
Which version of Fllink are you using?
Is the high CPU usage caused by the browser or the JVM?
One thing we should investigate is how many requests the UI sends to the
JobManager,
but I don't know of a good way to do that without modifying the source.
On 06.12.2017 01:20, Yuta Morisawa wrote:
> Hi Timo
>
> I execute streaming job without checkpointing and I don't configure
> any state backend, so it may be "MemoryStateBackend".
>
> Actually, my streaming app just reads data from kafka and writes it to
> an external DB. Its not so complicated.
>
> Regards,
> Yuta
>
> On 2017/12/05 19:55, Timo Walther wrote:
>> I had some profiling tool like jvisualvm in mind. Are you executing
>> streaming or batch jobs? If streaming, is checkpointing enabled and
>> which type of statebackend?
>>
>> @Chesnay do you have experience with slow behavior of the Web UI?
>>
>> Regards,
>> Timo
>>
>>
>> Am 12/5/17 um 10:37 AM schrieb Yuta Morisawa:
>>> Hi Timo
>>>
>>> Thank you for your early reply.
>>>
>>> These are commands which I run my apps.
>>> ./bin/yarn-session.sh -n 20 -jm 6000 -tm 24000 -s 10
>>> ./bin/flink run -p 100 <jar1> <jar1 parameters>
>>> ./bin/flink run -p 100 <jar2> <jar2 parameters>
>>>
>>> So, JobManager Heap Memory = 6000 MB and it manages 2 jobs.
>>>
>>> > Maybe you can use a profiler and find out which component consumes so
>>> > much CPU resources?
>>> You mean Java Flight Recorder or JITWatch?
>>> Or, Flink has original profiler?
>>> https://ci.apache.org/projects/flink/flink-docs-release-1.3/monitoring/application_profiling.html
>>>
>>>
>>> Regards,
>>> Yuta
>>>
>>> On 2017/12/05 18:02, Timo Walther wrote:
>>>> Hi Yuta,
>>>>
>>>> as far as I know you cannot assign more cores to a JobManager.
>>>>
>>>> Can you tell us a bit more about your environment? How many jobs
>>>> does the JobManager has to manage? How much heap memory is assigned
>>>> to the JobManager?
>>>>
>>>> Maybe you can use a profiler and find out which component consumes
>>>> so much CPU resources?
>>>>
>>>> Regards,
>>>> Timo
>>>>
>>>>
>>>> Am 12/5/17 um 5:13 AM schrieb Yuta Morisawa:
>>>>> Hi
>>>>>
>>>>> Now I am looking for the way to increase the number of allocated
>>>>> CPU cores because my JobManagaer WEBUI is very heavy and sometimes
>>>>> freeze.
>>>>>
>>>>> I think this is caused by the resource shortage of JobManager.
>>>>> How can I increase the number of CPU for JobManager in YARN mode?
>>>>>
>>>>>
>>>>>
>>>>> Thanks
>>>>> Yuta
>>>>
>>>>
>>>>
>>
>>
>
>
Re: CPU Cores of JobManager
Posted by Yuta Morisawa <yu...@kddi-research.jp>.
Hi Timo
I execute streaming job without checkpointing and I don't configure any
state backend, so it may be "MemoryStateBackend".
Actually, my streaming app just reads data from kafka and writes it to
an external DB. Its not so complicated.
Regards,
Yuta
On 2017/12/05 19:55, Timo Walther wrote:
> I had some profiling tool like jvisualvm in mind. Are you executing
> streaming or batch jobs? If streaming, is checkpointing enabled and
> which type of statebackend?
>
> @Chesnay do you have experience with slow behavior of the Web UI?
>
> Regards,
> Timo
>
>
> Am 12/5/17 um 10:37 AM schrieb Yuta Morisawa:
>> Hi Timo
>>
>> Thank you for your early reply.
>>
>> These are commands which I run my apps.
>> ./bin/yarn-session.sh -n 20 -jm 6000 -tm 24000 -s 10
>> ./bin/flink run -p 100 <jar1> <jar1 parameters>
>> ./bin/flink run -p 100 <jar2> <jar2 parameters>
>>
>> So, JobManager Heap Memory = 6000 MB and it manages 2 jobs.
>>
>> > Maybe you can use a profiler and find out which component consumes so
>> > much CPU resources?
>> You mean Java Flight Recorder or JITWatch?
>> Or, Flink has original profiler?
>> https://ci.apache.org/projects/flink/flink-docs-release-1.3/monitoring/application_profiling.html
>>
>>
>> Regards,
>> Yuta
>>
>> On 2017/12/05 18:02, Timo Walther wrote:
>>> Hi Yuta,
>>>
>>> as far as I know you cannot assign more cores to a JobManager.
>>>
>>> Can you tell us a bit more about your environment? How many jobs does
>>> the JobManager has to manage? How much heap memory is assigned to the
>>> JobManager?
>>>
>>> Maybe you can use a profiler and find out which component consumes so
>>> much CPU resources?
>>>
>>> Regards,
>>> Timo
>>>
>>>
>>> Am 12/5/17 um 5:13 AM schrieb Yuta Morisawa:
>>>> Hi
>>>>
>>>> Now I am looking for the way to increase the number of allocated CPU
>>>> cores because my JobManagaer WEBUI is very heavy and sometimes freeze.
>>>>
>>>> I think this is caused by the resource shortage of JobManager.
>>>> How can I increase the number of CPU for JobManager in YARN mode?
>>>>
>>>>
>>>>
>>>> Thanks
>>>> Yuta
>>>
>>>
>>>
>
>
Re: CPU Cores of JobManager
Posted by Timo Walther <tw...@apache.org>.
I had some profiling tool like jvisualvm in mind. Are you executing
streaming or batch jobs? If streaming, is checkpointing enabled and
which type of statebackend?
@Chesnay do you have experience with slow behavior of the Web UI?
Regards,
Timo
Am 12/5/17 um 10:37 AM schrieb Yuta Morisawa:
> Hi Timo
>
> Thank you for your early reply.
>
> These are commands which I run my apps.
> ./bin/yarn-session.sh -n 20 -jm 6000 -tm 24000 -s 10
> ./bin/flink run -p 100 <jar1> <jar1 parameters>
> ./bin/flink run -p 100 <jar2> <jar2 parameters>
>
> So, JobManager Heap Memory = 6000 MB and it manages 2 jobs.
>
> > Maybe you can use a profiler and find out which component consumes so
> > much CPU resources?
> You mean Java Flight Recorder or JITWatch?
> Or, Flink has original profiler?
> https://ci.apache.org/projects/flink/flink-docs-release-1.3/monitoring/application_profiling.html
>
>
> Regards,
> Yuta
>
> On 2017/12/05 18:02, Timo Walther wrote:
>> Hi Yuta,
>>
>> as far as I know you cannot assign more cores to a JobManager.
>>
>> Can you tell us a bit more about your environment? How many jobs does
>> the JobManager has to manage? How much heap memory is assigned to the
>> JobManager?
>>
>> Maybe you can use a profiler and find out which component consumes so
>> much CPU resources?
>>
>> Regards,
>> Timo
>>
>>
>> Am 12/5/17 um 5:13 AM schrieb Yuta Morisawa:
>>> Hi
>>>
>>> Now I am looking for the way to increase the number of allocated CPU
>>> cores because my JobManagaer WEBUI is very heavy and sometimes freeze.
>>>
>>> I think this is caused by the resource shortage of JobManager.
>>> How can I increase the number of CPU for JobManager in YARN mode?
>>>
>>>
>>>
>>> Thanks
>>> Yuta
>>
>>
>>
Re: CPU Cores of JobManager
Posted by Yuta Morisawa <yu...@kddi-research.jp>.
Hi Timo
Thank you for your early reply.
These are commands which I run my apps.
./bin/yarn-session.sh -n 20 -jm 6000 -tm 24000 -s 10
./bin/flink run -p 100 <jar1> <jar1 parameters>
./bin/flink run -p 100 <jar2> <jar2 parameters>
So, JobManager Heap Memory = 6000 MB and it manages 2 jobs.
> Maybe you can use a profiler and find out which component consumes so
> much CPU resources?
You mean Java Flight Recorder or JITWatch?
Or, Flink has original profiler?
https://ci.apache.org/projects/flink/flink-docs-release-1.3/monitoring/application_profiling.html
Regards,
Yuta
On 2017/12/05 18:02, Timo Walther wrote:
> Hi Yuta,
>
> as far as I know you cannot assign more cores to a JobManager.
>
> Can you tell us a bit more about your environment? How many jobs does
> the JobManager has to manage? How much heap memory is assigned to the
> JobManager?
>
> Maybe you can use a profiler and find out which component consumes so
> much CPU resources?
>
> Regards,
> Timo
>
>
> Am 12/5/17 um 5:13 AM schrieb Yuta Morisawa:
>> Hi
>>
>> Now I am looking for the way to increase the number of allocated CPU
>> cores because my JobManagaer WEBUI is very heavy and sometimes freeze.
>>
>> I think this is caused by the resource shortage of JobManager.
>> How can I increase the number of CPU for JobManager in YARN mode?
>>
>>
>>
>> Thanks
>> Yuta
>
>
>
Re: CPU Cores of JobManager
Posted by Timo Walther <tw...@apache.org>.
Hi Yuta,
as far as I know you cannot assign more cores to a JobManager.
Can you tell us a bit more about your environment? How many jobs does
the JobManager has to manage? How much heap memory is assigned to the
JobManager?
Maybe you can use a profiler and find out which component consumes so
much CPU resources?
Regards,
Timo
Am 12/5/17 um 5:13 AM schrieb Yuta Morisawa:
> Hi
>
> Now I am looking for the way to increase the number of allocated CPU
> cores because my JobManagaer WEBUI is very heavy and sometimes freeze.
>
> I think this is caused by the resource shortage of JobManager.
> How can I increase the number of CPU for JobManager in YARN mode?
>
>
>
> Thanks
> Yuta