You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flink.apache.org by Hilmi Yildirim <Hi...@dfki.de> on 2016/01/22 11:44:51 UTC
Flink Yarn Stack Size
Hi,
does anybody know how I can increase the stack size of a yarn session?
Best Regards,
Hilmi
--
==================================================================
Hilmi Yildirim, M.Sc.
Researcher
DFKI GmbH
Intelligente Analytik für Massendaten
DFKI Projektbüro Berlin
Alt-Moabit 91c
D-10559 Berlin
Phone: +49 30 23895 1814
E-Mail: Hilmi.Yildirim@dfki.de
-------------------------------------------------------------
Deutsches Forschungszentrum fuer Kuenstliche Intelligenz GmbH
Firmensitz: Trippstadter Strasse 122, D-67663 Kaiserslautern
Geschaeftsfuehrung:
Prof. Dr. Dr. h.c. mult. Wolfgang Wahlster (Vorsitzender)
Dr. Walter Olthoff
Vorsitzender des Aufsichtsrats:
Prof. Dr. h.c. Hans A. Aukes
Amtsgericht Kaiserslautern, HRB 2313
-------------------------------------------------------------
Re: Flink Yarn Stack Size
Posted by Hilmi Yildirim <Hi...@dfki.de>.
Hi,
I have solved the problem. I used an object which contained a huge
generic MutableList. When kryo serializes the MutableList it does many
recursive calls which lead to a huge stack and to the
StackOverFlowError. I changed the MutableList to a Set and now kryo does
not do so many recursive calls and the stack keeps small :)
Best Regards,
Hilmi
Am 25.01.2016 um 16:34 schrieb Stephan Ewen:
> @Hilmi:
>
> A stack size of 1GB is too much, maybe the JVM ignores it? Can you try
> "-Xss2m" or so?
>
> Is this to prevent stack overflow errors in some code?
>
> Stephan
>
>
> On Fri, Jan 22, 2016 at 2:21 PM, Robert Metzger <rm...@apache.org> wrote:
>
>> Hi,
>>
>> yes, looks like the argument is passed correctly.
>>
>> By the way: Why are you sending your answers always twice to the mailing
>> list?
>>
>> On Fri, Jan 22, 2016 at 2:17 PM, Hilmi Yildirim <
>> hilmi.yildirim.dfki@gmail.com> wrote:
>>
>>> Hi Robert,
>>> in the logs I found this
>>>
>>> 13:54:41,291 INFO org.apache.flink.yarn.YarnTaskManagerRunner
>>> - JVM Options:
>>> 13:54:41,292 INFO org.apache.flink.yarn.YarnTaskManagerRunner
>>> - -Xms3072m
>>> 13:54:41,292 INFO org.apache.flink.yarn.YarnTaskManagerRunner
>>> - -Xmx3072m
>>> 13:54:41,292 INFO org.apache.flink.yarn.YarnTaskManagerRunner
>>> - -XX:MaxDirectMemorySize=3072m
>>> 13:54:41,292 INFO org.apache.flink.yarn.YarnTaskManagerRunner
>>> - -Xss1024m
>>> 13:54:41,292 INFO org.apache.flink.yarn.YarnTaskManagerRunner
>>> -
>>>
>> -Dlog.file=/var/log/hadoop-yarn/container/application_1453384265041_0030/container_1453384265041_0030_01_000002/taskmanager.log
>>> 13:54:41,292 INFO org.apache.flink.yarn.YarnTaskManagerRunner
>>> - -Dlogback.configurationFile=file:logback.xml
>>> 13:54:41,292 INFO org.apache.flink.yarn.YarnTaskManagerRunner
>>> - -Dlog4j.configuration=file:log4j.properties
>>>
>>>
>>> It seems that the argument is passed to the Taskmanager. Or am I wrong?
>>>
>>> Best Regards,
>>> Hilmi
>>>
>>>
>>>
>>> Am 22.01.2016 um 12:25 schrieb Robert Metzger:
>>>
>>>> Hi Hilmi,
>>>>
>>>> it means that each JVM the YARN session is starting (JobManager and
>>>> TaskManagers) is initialized with that parameter.
>>>>
>>>> Can you check the logs of the TaskManager to see if the option has been
>>>> passed properly?
>>>>
>>>> On Fri, Jan 22, 2016 at 12:22 PM, Hilmi Yildirim <
>> Hilmi.Yildirim@dfki.de>
>>>> wrote:
>>>>
>>>> Hi,
>>>>> I added the following line to my flink-conf.yaml:
>>>>>
>>>>> env.java.opts: "-Xss1024m"
>>>>>
>>>>> But it does not work. Does this line mean that every single slot of
>> each
>>>>> task manager uses a stack size of 1 GB?
>>>>>
>>>>> Best Regards,
>>>>> Hilmi
>>>>>
>>>>> Am 22.01.2016 um 11:46 schrieb Robert Metzger:
>>>>>
>>>>> Hi Hilmi,
>>>>>> Flink on YARN is respecting the env.java.opts configuration parameter
>>>>>> from
>>>>>> the flink-conf.yaml.
>>>>>>
>>>>>> 2016-01-22 11:44 GMT+01:00 Hilmi Yildirim <Hi...@dfki.de>:
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>>> does anybody know how I can increase the stack size of a yarn
>> session?
>>>>>>> Best Regards,
>>>>>>> Hilmi
>>>>>>>
>>>>>>> --
>>>>>>> ==================================================================
>>>>>>> Hilmi Yildirim, M.Sc.
>>>>>>> Researcher
>>>>>>>
>>>>>>> DFKI GmbH
>>>>>>> Intelligente Analytik für Massendaten
>>>>>>> DFKI Projektbüro Berlin
>>>>>>> Alt-Moabit 91c
>>>>>>> D-10559 Berlin
>>>>>>> Phone: +49 30 23895 1814
>>>>>>>
>>>>>>> E-Mail: Hilmi.Yildirim@dfki.de
>>>>>>>
>>>>>>> -------------------------------------------------------------
>>>>>>> Deutsches Forschungszentrum fuer Kuenstliche Intelligenz GmbH
>>>>>>> Firmensitz: Trippstadter Strasse 122, D-67663 Kaiserslautern
>>>>>>>
>>>>>>> Geschaeftsfuehrung:
>>>>>>> Prof. Dr. Dr. h.c. mult. Wolfgang Wahlster (Vorsitzender)
>>>>>>> Dr. Walter Olthoff
>>>>>>>
>>>>>>> Vorsitzender des Aufsichtsrats:
>>>>>>> Prof. Dr. h.c. Hans A. Aukes
>>>>>>>
>>>>>>> Amtsgericht Kaiserslautern, HRB 2313
>>>>>>> -------------------------------------------------------------
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>> ==================================================================
>>>>> Hilmi Yildirim, M.Sc.
>>>>> Researcher
>>>>>
>>>>> DFKI GmbH
>>>>> Intelligente Analytik für Massendaten
>>>>> DFKI Projektbüro Berlin
>>>>> Alt-Moabit 91c
>>>>> D-10559 Berlin
>>>>> Phone: +49 30 23895 1814
>>>>>
>>>>> E-Mail: Hilmi.Yildirim@dfki.de
>>>>>
>>>>> -------------------------------------------------------------
>>>>> Deutsches Forschungszentrum fuer Kuenstliche Intelligenz GmbH
>>>>> Firmensitz: Trippstadter Strasse 122, D-67663 Kaiserslautern
>>>>>
>>>>> Geschaeftsfuehrung:
>>>>> Prof. Dr. Dr. h.c. mult. Wolfgang Wahlster (Vorsitzender)
>>>>> Dr. Walter Olthoff
>>>>>
>>>>> Vorsitzender des Aufsichtsrats:
>>>>> Prof. Dr. h.c. Hans A. Aukes
>>>>>
>>>>> Amtsgericht Kaiserslautern, HRB 2313
>>>>> -------------------------------------------------------------
>>>>>
>>>>>
>>>>>
--
==================================================================
Hilmi Yildirim, M.Sc.
Researcher
DFKI GmbH
Intelligente Analytik für Massendaten
DFKI Projektbüro Berlin
Alt-Moabit 91c
D-10559 Berlin
Phone: +49 30 23895 1814
E-Mail: Hilmi.Yildirim@dfki.de
-------------------------------------------------------------
Deutsches Forschungszentrum fuer Kuenstliche Intelligenz GmbH
Firmensitz: Trippstadter Strasse 122, D-67663 Kaiserslautern
Geschaeftsfuehrung:
Prof. Dr. Dr. h.c. mult. Wolfgang Wahlster (Vorsitzender)
Dr. Walter Olthoff
Vorsitzender des Aufsichtsrats:
Prof. Dr. h.c. Hans A. Aukes
Amtsgericht Kaiserslautern, HRB 2313
-------------------------------------------------------------
Re: Flink Yarn Stack Size
Posted by Stephan Ewen <se...@apache.org>.
@Hilmi:
A stack size of 1GB is too much, maybe the JVM ignores it? Can you try
"-Xss2m" or so?
Is this to prevent stack overflow errors in some code?
Stephan
On Fri, Jan 22, 2016 at 2:21 PM, Robert Metzger <rm...@apache.org> wrote:
> Hi,
>
> yes, looks like the argument is passed correctly.
>
> By the way: Why are you sending your answers always twice to the mailing
> list?
>
> On Fri, Jan 22, 2016 at 2:17 PM, Hilmi Yildirim <
> hilmi.yildirim.dfki@gmail.com> wrote:
>
> > Hi Robert,
> > in the logs I found this
> >
> > 13:54:41,291 INFO org.apache.flink.yarn.YarnTaskManagerRunner
> > - JVM Options:
> > 13:54:41,292 INFO org.apache.flink.yarn.YarnTaskManagerRunner
> > - -Xms3072m
> > 13:54:41,292 INFO org.apache.flink.yarn.YarnTaskManagerRunner
> > - -Xmx3072m
> > 13:54:41,292 INFO org.apache.flink.yarn.YarnTaskManagerRunner
> > - -XX:MaxDirectMemorySize=3072m
> > 13:54:41,292 INFO org.apache.flink.yarn.YarnTaskManagerRunner
> > - -Xss1024m
> > 13:54:41,292 INFO org.apache.flink.yarn.YarnTaskManagerRunner
> > -
> >
> -Dlog.file=/var/log/hadoop-yarn/container/application_1453384265041_0030/container_1453384265041_0030_01_000002/taskmanager.log
> > 13:54:41,292 INFO org.apache.flink.yarn.YarnTaskManagerRunner
> > - -Dlogback.configurationFile=file:logback.xml
> > 13:54:41,292 INFO org.apache.flink.yarn.YarnTaskManagerRunner
> > - -Dlog4j.configuration=file:log4j.properties
> >
> >
> > It seems that the argument is passed to the Taskmanager. Or am I wrong?
> >
> > Best Regards,
> > Hilmi
> >
> >
> >
> > Am 22.01.2016 um 12:25 schrieb Robert Metzger:
> >
> >> Hi Hilmi,
> >>
> >> it means that each JVM the YARN session is starting (JobManager and
> >> TaskManagers) is initialized with that parameter.
> >>
> >> Can you check the logs of the TaskManager to see if the option has been
> >> passed properly?
> >>
> >> On Fri, Jan 22, 2016 at 12:22 PM, Hilmi Yildirim <
> Hilmi.Yildirim@dfki.de>
> >> wrote:
> >>
> >> Hi,
> >>> I added the following line to my flink-conf.yaml:
> >>>
> >>> env.java.opts: "-Xss1024m"
> >>>
> >>> But it does not work. Does this line mean that every single slot of
> each
> >>> task manager uses a stack size of 1 GB?
> >>>
> >>> Best Regards,
> >>> Hilmi
> >>>
> >>> Am 22.01.2016 um 11:46 schrieb Robert Metzger:
> >>>
> >>> Hi Hilmi,
> >>>>
> >>>> Flink on YARN is respecting the env.java.opts configuration parameter
> >>>> from
> >>>> the flink-conf.yaml.
> >>>>
> >>>> 2016-01-22 11:44 GMT+01:00 Hilmi Yildirim <Hi...@dfki.de>:
> >>>>
> >>>> Hi,
> >>>>
> >>>>> does anybody know how I can increase the stack size of a yarn
> session?
> >>>>>
> >>>>> Best Regards,
> >>>>> Hilmi
> >>>>>
> >>>>> --
> >>>>> ==================================================================
> >>>>> Hilmi Yildirim, M.Sc.
> >>>>> Researcher
> >>>>>
> >>>>> DFKI GmbH
> >>>>> Intelligente Analytik für Massendaten
> >>>>> DFKI Projektbüro Berlin
> >>>>> Alt-Moabit 91c
> >>>>> D-10559 Berlin
> >>>>> Phone: +49 30 23895 1814
> >>>>>
> >>>>> E-Mail: Hilmi.Yildirim@dfki.de
> >>>>>
> >>>>> -------------------------------------------------------------
> >>>>> Deutsches Forschungszentrum fuer Kuenstliche Intelligenz GmbH
> >>>>> Firmensitz: Trippstadter Strasse 122, D-67663 Kaiserslautern
> >>>>>
> >>>>> Geschaeftsfuehrung:
> >>>>> Prof. Dr. Dr. h.c. mult. Wolfgang Wahlster (Vorsitzender)
> >>>>> Dr. Walter Olthoff
> >>>>>
> >>>>> Vorsitzender des Aufsichtsrats:
> >>>>> Prof. Dr. h.c. Hans A. Aukes
> >>>>>
> >>>>> Amtsgericht Kaiserslautern, HRB 2313
> >>>>> -------------------------------------------------------------
> >>>>>
> >>>>>
> >>>>>
> >>>>> --
> >>> ==================================================================
> >>> Hilmi Yildirim, M.Sc.
> >>> Researcher
> >>>
> >>> DFKI GmbH
> >>> Intelligente Analytik für Massendaten
> >>> DFKI Projektbüro Berlin
> >>> Alt-Moabit 91c
> >>> D-10559 Berlin
> >>> Phone: +49 30 23895 1814
> >>>
> >>> E-Mail: Hilmi.Yildirim@dfki.de
> >>>
> >>> -------------------------------------------------------------
> >>> Deutsches Forschungszentrum fuer Kuenstliche Intelligenz GmbH
> >>> Firmensitz: Trippstadter Strasse 122, D-67663 Kaiserslautern
> >>>
> >>> Geschaeftsfuehrung:
> >>> Prof. Dr. Dr. h.c. mult. Wolfgang Wahlster (Vorsitzender)
> >>> Dr. Walter Olthoff
> >>>
> >>> Vorsitzender des Aufsichtsrats:
> >>> Prof. Dr. h.c. Hans A. Aukes
> >>>
> >>> Amtsgericht Kaiserslautern, HRB 2313
> >>> -------------------------------------------------------------
> >>>
> >>>
> >>>
> >
>
Re: Flink Yarn Stack Size
Posted by Robert Metzger <rm...@apache.org>.
Hi,
yes, looks like the argument is passed correctly.
By the way: Why are you sending your answers always twice to the mailing
list?
On Fri, Jan 22, 2016 at 2:17 PM, Hilmi Yildirim <
hilmi.yildirim.dfki@gmail.com> wrote:
> Hi Robert,
> in the logs I found this
>
> 13:54:41,291 INFO org.apache.flink.yarn.YarnTaskManagerRunner
> - JVM Options:
> 13:54:41,292 INFO org.apache.flink.yarn.YarnTaskManagerRunner
> - -Xms3072m
> 13:54:41,292 INFO org.apache.flink.yarn.YarnTaskManagerRunner
> - -Xmx3072m
> 13:54:41,292 INFO org.apache.flink.yarn.YarnTaskManagerRunner
> - -XX:MaxDirectMemorySize=3072m
> 13:54:41,292 INFO org.apache.flink.yarn.YarnTaskManagerRunner
> - -Xss1024m
> 13:54:41,292 INFO org.apache.flink.yarn.YarnTaskManagerRunner
> -
> -Dlog.file=/var/log/hadoop-yarn/container/application_1453384265041_0030/container_1453384265041_0030_01_000002/taskmanager.log
> 13:54:41,292 INFO org.apache.flink.yarn.YarnTaskManagerRunner
> - -Dlogback.configurationFile=file:logback.xml
> 13:54:41,292 INFO org.apache.flink.yarn.YarnTaskManagerRunner
> - -Dlog4j.configuration=file:log4j.properties
>
>
> It seems that the argument is passed to the Taskmanager. Or am I wrong?
>
> Best Regards,
> Hilmi
>
>
>
> Am 22.01.2016 um 12:25 schrieb Robert Metzger:
>
>> Hi Hilmi,
>>
>> it means that each JVM the YARN session is starting (JobManager and
>> TaskManagers) is initialized with that parameter.
>>
>> Can you check the logs of the TaskManager to see if the option has been
>> passed properly?
>>
>> On Fri, Jan 22, 2016 at 12:22 PM, Hilmi Yildirim <Hi...@dfki.de>
>> wrote:
>>
>> Hi,
>>> I added the following line to my flink-conf.yaml:
>>>
>>> env.java.opts: "-Xss1024m"
>>>
>>> But it does not work. Does this line mean that every single slot of each
>>> task manager uses a stack size of 1 GB?
>>>
>>> Best Regards,
>>> Hilmi
>>>
>>> Am 22.01.2016 um 11:46 schrieb Robert Metzger:
>>>
>>> Hi Hilmi,
>>>>
>>>> Flink on YARN is respecting the env.java.opts configuration parameter
>>>> from
>>>> the flink-conf.yaml.
>>>>
>>>> 2016-01-22 11:44 GMT+01:00 Hilmi Yildirim <Hi...@dfki.de>:
>>>>
>>>> Hi,
>>>>
>>>>> does anybody know how I can increase the stack size of a yarn session?
>>>>>
>>>>> Best Regards,
>>>>> Hilmi
>>>>>
>>>>> --
>>>>> ==================================================================
>>>>> Hilmi Yildirim, M.Sc.
>>>>> Researcher
>>>>>
>>>>> DFKI GmbH
>>>>> Intelligente Analytik für Massendaten
>>>>> DFKI Projektbüro Berlin
>>>>> Alt-Moabit 91c
>>>>> D-10559 Berlin
>>>>> Phone: +49 30 23895 1814
>>>>>
>>>>> E-Mail: Hilmi.Yildirim@dfki.de
>>>>>
>>>>> -------------------------------------------------------------
>>>>> Deutsches Forschungszentrum fuer Kuenstliche Intelligenz GmbH
>>>>> Firmensitz: Trippstadter Strasse 122, D-67663 Kaiserslautern
>>>>>
>>>>> Geschaeftsfuehrung:
>>>>> Prof. Dr. Dr. h.c. mult. Wolfgang Wahlster (Vorsitzender)
>>>>> Dr. Walter Olthoff
>>>>>
>>>>> Vorsitzender des Aufsichtsrats:
>>>>> Prof. Dr. h.c. Hans A. Aukes
>>>>>
>>>>> Amtsgericht Kaiserslautern, HRB 2313
>>>>> -------------------------------------------------------------
>>>>>
>>>>>
>>>>>
>>>>> --
>>> ==================================================================
>>> Hilmi Yildirim, M.Sc.
>>> Researcher
>>>
>>> DFKI GmbH
>>> Intelligente Analytik für Massendaten
>>> DFKI Projektbüro Berlin
>>> Alt-Moabit 91c
>>> D-10559 Berlin
>>> Phone: +49 30 23895 1814
>>>
>>> E-Mail: Hilmi.Yildirim@dfki.de
>>>
>>> -------------------------------------------------------------
>>> Deutsches Forschungszentrum fuer Kuenstliche Intelligenz GmbH
>>> Firmensitz: Trippstadter Strasse 122, D-67663 Kaiserslautern
>>>
>>> Geschaeftsfuehrung:
>>> Prof. Dr. Dr. h.c. mult. Wolfgang Wahlster (Vorsitzender)
>>> Dr. Walter Olthoff
>>>
>>> Vorsitzender des Aufsichtsrats:
>>> Prof. Dr. h.c. Hans A. Aukes
>>>
>>> Amtsgericht Kaiserslautern, HRB 2313
>>> -------------------------------------------------------------
>>>
>>>
>>>
>
Re: Flink Yarn Stack Size
Posted by Hilmi Yildirim <hi...@gmail.com>.
Hi Robert,
in the logs I found this
13:54:41,291 INFO
org.apache.flink.yarn.YarnTaskManagerRunner - JVM
Options:
13:54:41,292 INFO
org.apache.flink.yarn.YarnTaskManagerRunner - -Xms3072m
13:54:41,292 INFO
org.apache.flink.yarn.YarnTaskManagerRunner - -Xmx3072m
13:54:41,292 INFO
org.apache.flink.yarn.YarnTaskManagerRunner -
-XX:MaxDirectMemorySize=3072m
13:54:41,292 INFO
org.apache.flink.yarn.YarnTaskManagerRunner - -Xss1024m
13:54:41,292 INFO
org.apache.flink.yarn.YarnTaskManagerRunner -
-Dlog.file=/var/log/hadoop-yarn/container/application_1453384265041_0030/container_1453384265041_0030_01_000002/taskmanager.log
13:54:41,292 INFO
org.apache.flink.yarn.YarnTaskManagerRunner -
-Dlogback.configurationFile=file:logback.xml
13:54:41,292 INFO
org.apache.flink.yarn.YarnTaskManagerRunner -
-Dlog4j.configuration=file:log4j.properties
It seems that the argument is passed to the Taskmanager. Or am I wrong?
Best Regards,
Hilmi
Am 22.01.2016 um 12:25 schrieb Robert Metzger:
> Hi Hilmi,
>
> it means that each JVM the YARN session is starting (JobManager and
> TaskManagers) is initialized with that parameter.
>
> Can you check the logs of the TaskManager to see if the option has been
> passed properly?
>
> On Fri, Jan 22, 2016 at 12:22 PM, Hilmi Yildirim <Hi...@dfki.de>
> wrote:
>
>> Hi,
>> I added the following line to my flink-conf.yaml:
>>
>> env.java.opts: "-Xss1024m"
>>
>> But it does not work. Does this line mean that every single slot of each
>> task manager uses a stack size of 1 GB?
>>
>> Best Regards,
>> Hilmi
>>
>> Am 22.01.2016 um 11:46 schrieb Robert Metzger:
>>
>>> Hi Hilmi,
>>>
>>> Flink on YARN is respecting the env.java.opts configuration parameter from
>>> the flink-conf.yaml.
>>>
>>> 2016-01-22 11:44 GMT+01:00 Hilmi Yildirim <Hi...@dfki.de>:
>>>
>>> Hi,
>>>> does anybody know how I can increase the stack size of a yarn session?
>>>>
>>>> Best Regards,
>>>> Hilmi
>>>>
>>>> --
>>>> ==================================================================
>>>> Hilmi Yildirim, M.Sc.
>>>> Researcher
>>>>
>>>> DFKI GmbH
>>>> Intelligente Analytik für Massendaten
>>>> DFKI Projektbüro Berlin
>>>> Alt-Moabit 91c
>>>> D-10559 Berlin
>>>> Phone: +49 30 23895 1814
>>>>
>>>> E-Mail: Hilmi.Yildirim@dfki.de
>>>>
>>>> -------------------------------------------------------------
>>>> Deutsches Forschungszentrum fuer Kuenstliche Intelligenz GmbH
>>>> Firmensitz: Trippstadter Strasse 122, D-67663 Kaiserslautern
>>>>
>>>> Geschaeftsfuehrung:
>>>> Prof. Dr. Dr. h.c. mult. Wolfgang Wahlster (Vorsitzender)
>>>> Dr. Walter Olthoff
>>>>
>>>> Vorsitzender des Aufsichtsrats:
>>>> Prof. Dr. h.c. Hans A. Aukes
>>>>
>>>> Amtsgericht Kaiserslautern, HRB 2313
>>>> -------------------------------------------------------------
>>>>
>>>>
>>>>
>> --
>> ==================================================================
>> Hilmi Yildirim, M.Sc.
>> Researcher
>>
>> DFKI GmbH
>> Intelligente Analytik für Massendaten
>> DFKI Projektbüro Berlin
>> Alt-Moabit 91c
>> D-10559 Berlin
>> Phone: +49 30 23895 1814
>>
>> E-Mail: Hilmi.Yildirim@dfki.de
>>
>> -------------------------------------------------------------
>> Deutsches Forschungszentrum fuer Kuenstliche Intelligenz GmbH
>> Firmensitz: Trippstadter Strasse 122, D-67663 Kaiserslautern
>>
>> Geschaeftsfuehrung:
>> Prof. Dr. Dr. h.c. mult. Wolfgang Wahlster (Vorsitzender)
>> Dr. Walter Olthoff
>>
>> Vorsitzender des Aufsichtsrats:
>> Prof. Dr. h.c. Hans A. Aukes
>>
>> Amtsgericht Kaiserslautern, HRB 2313
>> -------------------------------------------------------------
>>
>>
Re: Flink Yarn Stack Size
Posted by Hilmi Yildirim <Hi...@dfki.de>.
Hi Robert,
in the logs I found this
13:54:41,291 INFO
org.apache.flink.yarn.YarnTaskManagerRunner - JVM
Options:
13:54:41,292 INFO
org.apache.flink.yarn.YarnTaskManagerRunner - -Xms3072m
13:54:41,292 INFO
org.apache.flink.yarn.YarnTaskManagerRunner - -Xmx3072m
13:54:41,292 INFO
org.apache.flink.yarn.YarnTaskManagerRunner -
-XX:MaxDirectMemorySize=3072m
13:54:41,292 INFO
org.apache.flink.yarn.YarnTaskManagerRunner - -Xss1024m
13:54:41,292 INFO
org.apache.flink.yarn.YarnTaskManagerRunner -
-Dlog.file=/var/log/hadoop-yarn/container/application_1453384265041_0030/container_1453384265041_0030_01_000002/taskmanager.log
13:54:41,292 INFO
org.apache.flink.yarn.YarnTaskManagerRunner -
-Dlogback.configurationFile=file:logback.xml
13:54:41,292 INFO
org.apache.flink.yarn.YarnTaskManagerRunner -
-Dlog4j.configuration=file:log4j.properties
It seems that the argument is passed to the Taskmanager. Or am I wrong?
Best Regards,
Hilmi
Am 22.01.2016 um 12:25 schrieb Robert Metzger:
> Hi Hilmi,
>
> it means that each JVM the YARN session is starting (JobManager and
> TaskManagers) is initialized with that parameter.
>
> Can you check the logs of the TaskManager to see if the option has been
> passed properly?
>
> On Fri, Jan 22, 2016 at 12:22 PM, Hilmi Yildirim <Hi...@dfki.de>
> wrote:
>
>> Hi,
>> I added the following line to my flink-conf.yaml:
>>
>> env.java.opts: "-Xss1024m"
>>
>> But it does not work. Does this line mean that every single slot of each
>> task manager uses a stack size of 1 GB?
>>
>> Best Regards,
>> Hilmi
>>
>> Am 22.01.2016 um 11:46 schrieb Robert Metzger:
>>
>>> Hi Hilmi,
>>>
>>> Flink on YARN is respecting the env.java.opts configuration parameter from
>>> the flink-conf.yaml.
>>>
>>> 2016-01-22 11:44 GMT+01:00 Hilmi Yildirim <Hi...@dfki.de>:
>>>
>>> Hi,
>>>> does anybody know how I can increase the stack size of a yarn session?
>>>>
>>>> Best Regards,
>>>> Hilmi
>>>>
>>>> --
>>>> ==================================================================
>>>> Hilmi Yildirim, M.Sc.
>>>> Researcher
>>>>
>>>> DFKI GmbH
>>>> Intelligente Analytik für Massendaten
>>>> DFKI Projektbüro Berlin
>>>> Alt-Moabit 91c
>>>> D-10559 Berlin
>>>> Phone: +49 30 23895 1814
>>>>
>>>> E-Mail: Hilmi.Yildirim@dfki.de
>>>>
>>>> -------------------------------------------------------------
>>>> Deutsches Forschungszentrum fuer Kuenstliche Intelligenz GmbH
>>>> Firmensitz: Trippstadter Strasse 122, D-67663 Kaiserslautern
>>>>
>>>> Geschaeftsfuehrung:
>>>> Prof. Dr. Dr. h.c. mult. Wolfgang Wahlster (Vorsitzender)
>>>> Dr. Walter Olthoff
>>>>
>>>> Vorsitzender des Aufsichtsrats:
>>>> Prof. Dr. h.c. Hans A. Aukes
>>>>
>>>> Amtsgericht Kaiserslautern, HRB 2313
>>>> -------------------------------------------------------------
>>>>
>>>>
>>>>
>> --
>> ==================================================================
>> Hilmi Yildirim, M.Sc.
>> Researcher
>>
>> DFKI GmbH
>> Intelligente Analytik für Massendaten
>> DFKI Projektbüro Berlin
>> Alt-Moabit 91c
>> D-10559 Berlin
>> Phone: +49 30 23895 1814
>>
>> E-Mail: Hilmi.Yildirim@dfki.de
>>
>> -------------------------------------------------------------
>> Deutsches Forschungszentrum fuer Kuenstliche Intelligenz GmbH
>> Firmensitz: Trippstadter Strasse 122, D-67663 Kaiserslautern
>>
>> Geschaeftsfuehrung:
>> Prof. Dr. Dr. h.c. mult. Wolfgang Wahlster (Vorsitzender)
>> Dr. Walter Olthoff
>>
>> Vorsitzender des Aufsichtsrats:
>> Prof. Dr. h.c. Hans A. Aukes
>>
>> Amtsgericht Kaiserslautern, HRB 2313
>> -------------------------------------------------------------
>>
>>
--
==================================================================
Hilmi Yildirim, M.Sc.
Researcher
DFKI GmbH
Intelligente Analytik für Massendaten
DFKI Projektbüro Berlin
Alt-Moabit 91c
D-10559 Berlin
Phone: +49 30 23895 1814
E-Mail: Hilmi.Yildirim@dfki.de
-------------------------------------------------------------
Deutsches Forschungszentrum fuer Kuenstliche Intelligenz GmbH
Firmensitz: Trippstadter Strasse 122, D-67663 Kaiserslautern
Geschaeftsfuehrung:
Prof. Dr. Dr. h.c. mult. Wolfgang Wahlster (Vorsitzender)
Dr. Walter Olthoff
Vorsitzender des Aufsichtsrats:
Prof. Dr. h.c. Hans A. Aukes
Amtsgericht Kaiserslautern, HRB 2313
-------------------------------------------------------------
Re: Flink Yarn Stack Size
Posted by Robert Metzger <rm...@apache.org>.
Hi Hilmi,
it means that each JVM the YARN session is starting (JobManager and
TaskManagers) is initialized with that parameter.
Can you check the logs of the TaskManager to see if the option has been
passed properly?
On Fri, Jan 22, 2016 at 12:22 PM, Hilmi Yildirim <Hi...@dfki.de>
wrote:
> Hi,
> I added the following line to my flink-conf.yaml:
>
> env.java.opts: "-Xss1024m"
>
> But it does not work. Does this line mean that every single slot of each
> task manager uses a stack size of 1 GB?
>
> Best Regards,
> Hilmi
>
> Am 22.01.2016 um 11:46 schrieb Robert Metzger:
>
>> Hi Hilmi,
>>
>> Flink on YARN is respecting the env.java.opts configuration parameter from
>> the flink-conf.yaml.
>>
>> 2016-01-22 11:44 GMT+01:00 Hilmi Yildirim <Hi...@dfki.de>:
>>
>> Hi,
>>> does anybody know how I can increase the stack size of a yarn session?
>>>
>>> Best Regards,
>>> Hilmi
>>>
>>> --
>>> ==================================================================
>>> Hilmi Yildirim, M.Sc.
>>> Researcher
>>>
>>> DFKI GmbH
>>> Intelligente Analytik für Massendaten
>>> DFKI Projektbüro Berlin
>>> Alt-Moabit 91c
>>> D-10559 Berlin
>>> Phone: +49 30 23895 1814
>>>
>>> E-Mail: Hilmi.Yildirim@dfki.de
>>>
>>> -------------------------------------------------------------
>>> Deutsches Forschungszentrum fuer Kuenstliche Intelligenz GmbH
>>> Firmensitz: Trippstadter Strasse 122, D-67663 Kaiserslautern
>>>
>>> Geschaeftsfuehrung:
>>> Prof. Dr. Dr. h.c. mult. Wolfgang Wahlster (Vorsitzender)
>>> Dr. Walter Olthoff
>>>
>>> Vorsitzender des Aufsichtsrats:
>>> Prof. Dr. h.c. Hans A. Aukes
>>>
>>> Amtsgericht Kaiserslautern, HRB 2313
>>> -------------------------------------------------------------
>>>
>>>
>>>
>
> --
> ==================================================================
> Hilmi Yildirim, M.Sc.
> Researcher
>
> DFKI GmbH
> Intelligente Analytik für Massendaten
> DFKI Projektbüro Berlin
> Alt-Moabit 91c
> D-10559 Berlin
> Phone: +49 30 23895 1814
>
> E-Mail: Hilmi.Yildirim@dfki.de
>
> -------------------------------------------------------------
> Deutsches Forschungszentrum fuer Kuenstliche Intelligenz GmbH
> Firmensitz: Trippstadter Strasse 122, D-67663 Kaiserslautern
>
> Geschaeftsfuehrung:
> Prof. Dr. Dr. h.c. mult. Wolfgang Wahlster (Vorsitzender)
> Dr. Walter Olthoff
>
> Vorsitzender des Aufsichtsrats:
> Prof. Dr. h.c. Hans A. Aukes
>
> Amtsgericht Kaiserslautern, HRB 2313
> -------------------------------------------------------------
>
>
Re: Flink Yarn Stack Size
Posted by Hilmi Yildirim <Hi...@dfki.de>.
Hi,
I added the following line to my flink-conf.yaml:
env.java.opts: "-Xss1024m"
But it does not work. Does this line mean that every single slot of each
task manager uses a stack size of 1 GB?
Best Regards,
Hilmi
Am 22.01.2016 um 11:46 schrieb Robert Metzger:
> Hi Hilmi,
>
> Flink on YARN is respecting the env.java.opts configuration parameter from
> the flink-conf.yaml.
>
> 2016-01-22 11:44 GMT+01:00 Hilmi Yildirim <Hi...@dfki.de>:
>
>> Hi,
>> does anybody know how I can increase the stack size of a yarn session?
>>
>> Best Regards,
>> Hilmi
>>
>> --
>> ==================================================================
>> Hilmi Yildirim, M.Sc.
>> Researcher
>>
>> DFKI GmbH
>> Intelligente Analytik für Massendaten
>> DFKI Projektbüro Berlin
>> Alt-Moabit 91c
>> D-10559 Berlin
>> Phone: +49 30 23895 1814
>>
>> E-Mail: Hilmi.Yildirim@dfki.de
>>
>> -------------------------------------------------------------
>> Deutsches Forschungszentrum fuer Kuenstliche Intelligenz GmbH
>> Firmensitz: Trippstadter Strasse 122, D-67663 Kaiserslautern
>>
>> Geschaeftsfuehrung:
>> Prof. Dr. Dr. h.c. mult. Wolfgang Wahlster (Vorsitzender)
>> Dr. Walter Olthoff
>>
>> Vorsitzender des Aufsichtsrats:
>> Prof. Dr. h.c. Hans A. Aukes
>>
>> Amtsgericht Kaiserslautern, HRB 2313
>> -------------------------------------------------------------
>>
>>
--
==================================================================
Hilmi Yildirim, M.Sc.
Researcher
DFKI GmbH
Intelligente Analytik für Massendaten
DFKI Projektbüro Berlin
Alt-Moabit 91c
D-10559 Berlin
Phone: +49 30 23895 1814
E-Mail: Hilmi.Yildirim@dfki.de
-------------------------------------------------------------
Deutsches Forschungszentrum fuer Kuenstliche Intelligenz GmbH
Firmensitz: Trippstadter Strasse 122, D-67663 Kaiserslautern
Geschaeftsfuehrung:
Prof. Dr. Dr. h.c. mult. Wolfgang Wahlster (Vorsitzender)
Dr. Walter Olthoff
Vorsitzender des Aufsichtsrats:
Prof. Dr. h.c. Hans A. Aukes
Amtsgericht Kaiserslautern, HRB 2313
-------------------------------------------------------------
Re: Flink Yarn Stack Size
Posted by Hilmi Yildirim <hi...@gmail.com>.
Hi,
I added the following line to my flink-conf.yaml:
env.java.opts: "-Xss1024m"
But it does not work. Does this line mean that every single slot of each
task manager uses a stack size of 1 GB?
Best Regards,
Hilmi
Am 22.01.2016 um 11:46 schrieb Robert Metzger:
> Hi Hilmi,
>
> Flink on YARN is respecting the env.java.opts configuration parameter from
> the flink-conf.yaml.
>
> 2016-01-22 11:44 GMT+01:00 Hilmi Yildirim <Hi...@dfki.de>:
>
>> Hi,
>> does anybody know how I can increase the stack size of a yarn session?
>>
>> Best Regards,
>> Hilmi
>>
>> --
>> ==================================================================
>> Hilmi Yildirim, M.Sc.
>> Researcher
>>
>> DFKI GmbH
>> Intelligente Analytik für Massendaten
>> DFKI Projektbüro Berlin
>> Alt-Moabit 91c
>> D-10559 Berlin
>> Phone: +49 30 23895 1814
>>
>> E-Mail: Hilmi.Yildirim@dfki.de
>>
>> -------------------------------------------------------------
>> Deutsches Forschungszentrum fuer Kuenstliche Intelligenz GmbH
>> Firmensitz: Trippstadter Strasse 122, D-67663 Kaiserslautern
>>
>> Geschaeftsfuehrung:
>> Prof. Dr. Dr. h.c. mult. Wolfgang Wahlster (Vorsitzender)
>> Dr. Walter Olthoff
>>
>> Vorsitzender des Aufsichtsrats:
>> Prof. Dr. h.c. Hans A. Aukes
>>
>> Amtsgericht Kaiserslautern, HRB 2313
>> -------------------------------------------------------------
>>
>>
Re: Flink Yarn Stack Size
Posted by Robert Metzger <rm...@apache.org>.
Hi Hilmi,
Flink on YARN is respecting the env.java.opts configuration parameter from
the flink-conf.yaml.
2016-01-22 11:44 GMT+01:00 Hilmi Yildirim <Hi...@dfki.de>:
> Hi,
> does anybody know how I can increase the stack size of a yarn session?
>
> Best Regards,
> Hilmi
>
> --
> ==================================================================
> Hilmi Yildirim, M.Sc.
> Researcher
>
> DFKI GmbH
> Intelligente Analytik für Massendaten
> DFKI Projektbüro Berlin
> Alt-Moabit 91c
> D-10559 Berlin
> Phone: +49 30 23895 1814
>
> E-Mail: Hilmi.Yildirim@dfki.de
>
> -------------------------------------------------------------
> Deutsches Forschungszentrum fuer Kuenstliche Intelligenz GmbH
> Firmensitz: Trippstadter Strasse 122, D-67663 Kaiserslautern
>
> Geschaeftsfuehrung:
> Prof. Dr. Dr. h.c. mult. Wolfgang Wahlster (Vorsitzender)
> Dr. Walter Olthoff
>
> Vorsitzender des Aufsichtsrats:
> Prof. Dr. h.c. Hans A. Aukes
>
> Amtsgericht Kaiserslautern, HRB 2313
> -------------------------------------------------------------
>
>