You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Brian Chhun <br...@getbraintree.com> on 2015/12/01 21:40:24 UTC
Including option for starting job and task managers in the foreground
Hi All,
Is it possible to include a command line flag for starting job and task
managers in the foreground? Currently, `bin/jobmanager.sh` and
`bin/taskmanager.sh` rely on `bin/flink-daemon.sh`, which starts these
things in the background. I'd like to execute these commands inside a
docker container, but it's expected that the process is running in the
foreground. I think it might be useful to have it run in the foreground so
that it can be hooked into some process supervisors. Any suggestions are
appreciated.
Thanks,
Brian
Re: Including option for starting job and task managers in the foreground
Posted by Brian Chhun <br...@getbraintree.com>.
Thanks Max, I took a look at making this change directly to the scripts. I
was initially thinking about making a separate script whose only
responsibility is to run the command in the foreground, so that the
flink-daemon.sh could delegate to this script. I didn't get very far into
though, mostly trying to find a nice way to share the logging configuration
and passing of parameters.
While doing this, it did occur to me that if the process is run in the
foreground, the logging should be written to standard out. This ended up
being slightly hairy, at least in the job manager case, because the web UI
client expects the standard out of the job manager to be written to a file,
but I seem to have gotten things to work. I just wanted to mention this
detail in case it's the convention for things running in the foreground, in
which case it may need to be implemented alongside foregrounding the
process.
Thanks,
Brian
On Thu, Dec 3, 2015 at 3:55 AM, Maximilian Michels <mx...@apache.org> wrote:
> I think the way supervisor is used in the Docker scripts is a bit hacky.
> It is simply started in the foreground and does nothing. Supervisor is
> actually a really nice utility to start processes in Docker containers and
> monitor them.
>
> Nevertheless, supervisor also expects commands to stay in the foreground.
> A common way to work around this, is to create a script which monitors the
> daemon process' pid. Thinking about this, I think we could actually add the
> foreground functionality directly in the jobmanager / taskmanager shell
> script like you suggested.
>
> In the meantime, you could also use a simple script like this:
>
> #!/usr/bin/env bash
> # daemonize job manager
> ./bin/jobmanager start cluster
> # wait until process goes down
> wait $!
>
> Cheers,
> Max
>
> On Wed, Dec 2, 2015 at 7:16 PM, Brian Chhun <br...@getbraintree.com>
> wrote:
>
>> Thanks, I'm basing the things I'm doing based on what I see there. One
>> thing that's not clear to me in that example is why supervisor is used to
>> keep the container alive, rather than using some simpler means. It doesn't
>> look like it's been configured to supervise anything.
>>
>> On Wed, Dec 2, 2015 at 11:44 AM, Maximilian Michels <mx...@apache.org>
>> wrote:
>>
>>> Have you looked at
>>> https://github.com/apache/flink/tree/master/flink-contrib/docker-flink
>>> ? This demonstrates how to use Flink with Docker. In particular it
>>> states: "Images [..] run Supervisor to stay alive when running
>>> containers."
>>>
>>> Have a look at flink/config-flink.sh.
>>>
>>> Cheers,
>>> Max
>>>
>>> On Wed, Dec 2, 2015 at 6:29 PM, Brian Chhun
>>> <br...@getbraintree.com> wrote:
>>> > Yep, I think this makes sense. I'm currently patching the
>>> flink-daemon.sh
>>> > script to remove the `&`, but I don't think it's a very robust
>>> solution,
>>> > particularly when this script changes across versions of Flink. I'm
>>> very new
>>> > to Docker, but the resources I've found indicates that the process
>>> must run
>>> > in the foreground, though people seem to get around it with some hacks.
>>> >
>>> > When I have some time, I can look into refactoring some parts of the
>>> scripts
>>> > so that it can be started in the foreground.
>>> >
>>> > Thanks,
>>> > Brian
>>> >
>>> > On Wed, Dec 2, 2015 at 3:22 AM, Maximilian Michels <mx...@apache.org>
>>> wrote:
>>> >>
>>> >> Hi Brian,
>>> >>
>>> >> I don't recall Docker requires commands to run in the foreground.
>>> Still,
>>> >> if that is your requirement, simply remove the "&" at the end of this
>>> line
>>> >> in flink-daemon.sh:
>>> >>
>>> >> $JAVA_RUN $JVM_ARGS ${FLINK_ENV_JAVA_OPTS} "${log_setting[@]}"
>>> -classpath
>>> >> "`manglePathList "$FLINK_TM_CLASSPATH:$INTERNAL_HADOOP_CLASSPATHS"`"
>>> >> ${CLASS_TO_RUN} "${ARGS[@]}" > "$out" 2>&1 < /dev/null &
>>> >>
>>> >> Cheers,
>>> >> Max
>>> >>
>>> >> On Wed, Dec 2, 2015 at 9:26 AM, Till Rohrmann <tr...@apache.org>
>>> >> wrote:
>>> >>>
>>> >>> Hi Brian,
>>> >>>
>>> >>> as far as I know this is at the moment not possible with our scripts.
>>> >>> However it should be relatively easy to add by simply executing the
>>> Java
>>> >>> command in flink-daemon.sh in the foreground. Do you want to add
>>> this?
>>> >>>
>>> >>> Cheers,
>>> >>> Till
>>> >>>
>>> >>> On Dec 1, 2015 9:40 PM, "Brian Chhun" <br...@getbraintree.com>
>>> >>> wrote:
>>> >>>>
>>> >>>> Hi All,
>>> >>>>
>>> >>>> Is it possible to include a command line flag for starting job and
>>> task
>>> >>>> managers in the foreground? Currently, `bin/jobmanager.sh` and
>>> >>>> `bin/taskmanager.sh` rely on `bin/flink-daemon.sh`, which starts
>>> these
>>> >>>> things in the background. I'd like to execute these commands inside
>>> a docker
>>> >>>> container, but it's expected that the process is running in the
>>> foreground.
>>> >>>> I think it might be useful to have it run in the foreground so that
>>> it can
>>> >>>> be hooked into some process supervisors. Any suggestions are
>>> appreciated.
>>> >>>>
>>> >>>>
>>> >>>> Thanks,
>>> >>>> Brian
>>> >>
>>> >>
>>> >
>>>
>>
>>
>
Re: Including option for starting job and task managers in the foreground
Posted by Maximilian Michels <mx...@apache.org>.
I think the way supervisor is used in the Docker scripts is a bit hacky. It
is simply started in the foreground and does nothing. Supervisor is
actually a really nice utility to start processes in Docker containers and
monitor them.
Nevertheless, supervisor also expects commands to stay in the foreground. A
common way to work around this, is to create a script which monitors the
daemon process' pid. Thinking about this, I think we could actually add the
foreground functionality directly in the jobmanager / taskmanager shell
script like you suggested.
In the meantime, you could also use a simple script like this:
#!/usr/bin/env bash
# daemonize job manager
./bin/jobmanager start cluster
# wait until process goes down
wait $!
Cheers,
Max
On Wed, Dec 2, 2015 at 7:16 PM, Brian Chhun <br...@getbraintree.com>
wrote:
> Thanks, I'm basing the things I'm doing based on what I see there. One
> thing that's not clear to me in that example is why supervisor is used to
> keep the container alive, rather than using some simpler means. It doesn't
> look like it's been configured to supervise anything.
>
> On Wed, Dec 2, 2015 at 11:44 AM, Maximilian Michels <mx...@apache.org>
> wrote:
>
>> Have you looked at
>> https://github.com/apache/flink/tree/master/flink-contrib/docker-flink
>> ? This demonstrates how to use Flink with Docker. In particular it
>> states: "Images [..] run Supervisor to stay alive when running
>> containers."
>>
>> Have a look at flink/config-flink.sh.
>>
>> Cheers,
>> Max
>>
>> On Wed, Dec 2, 2015 at 6:29 PM, Brian Chhun
>> <br...@getbraintree.com> wrote:
>> > Yep, I think this makes sense. I'm currently patching the
>> flink-daemon.sh
>> > script to remove the `&`, but I don't think it's a very robust solution,
>> > particularly when this script changes across versions of Flink. I'm
>> very new
>> > to Docker, but the resources I've found indicates that the process must
>> run
>> > in the foreground, though people seem to get around it with some hacks.
>> >
>> > When I have some time, I can look into refactoring some parts of the
>> scripts
>> > so that it can be started in the foreground.
>> >
>> > Thanks,
>> > Brian
>> >
>> > On Wed, Dec 2, 2015 at 3:22 AM, Maximilian Michels <mx...@apache.org>
>> wrote:
>> >>
>> >> Hi Brian,
>> >>
>> >> I don't recall Docker requires commands to run in the foreground.
>> Still,
>> >> if that is your requirement, simply remove the "&" at the end of this
>> line
>> >> in flink-daemon.sh:
>> >>
>> >> $JAVA_RUN $JVM_ARGS ${FLINK_ENV_JAVA_OPTS} "${log_setting[@]}"
>> -classpath
>> >> "`manglePathList "$FLINK_TM_CLASSPATH:$INTERNAL_HADOOP_CLASSPATHS"`"
>> >> ${CLASS_TO_RUN} "${ARGS[@]}" > "$out" 2>&1 < /dev/null &
>> >>
>> >> Cheers,
>> >> Max
>> >>
>> >> On Wed, Dec 2, 2015 at 9:26 AM, Till Rohrmann <tr...@apache.org>
>> >> wrote:
>> >>>
>> >>> Hi Brian,
>> >>>
>> >>> as far as I know this is at the moment not possible with our scripts.
>> >>> However it should be relatively easy to add by simply executing the
>> Java
>> >>> command in flink-daemon.sh in the foreground. Do you want to add this?
>> >>>
>> >>> Cheers,
>> >>> Till
>> >>>
>> >>> On Dec 1, 2015 9:40 PM, "Brian Chhun" <br...@getbraintree.com>
>> >>> wrote:
>> >>>>
>> >>>> Hi All,
>> >>>>
>> >>>> Is it possible to include a command line flag for starting job and
>> task
>> >>>> managers in the foreground? Currently, `bin/jobmanager.sh` and
>> >>>> `bin/taskmanager.sh` rely on `bin/flink-daemon.sh`, which starts
>> these
>> >>>> things in the background. I'd like to execute these commands inside
>> a docker
>> >>>> container, but it's expected that the process is running in the
>> foreground.
>> >>>> I think it might be useful to have it run in the foreground so that
>> it can
>> >>>> be hooked into some process supervisors. Any suggestions are
>> appreciated.
>> >>>>
>> >>>>
>> >>>> Thanks,
>> >>>> Brian
>> >>
>> >>
>> >
>>
>
>
Re: Including option for starting job and task managers in the foreground
Posted by Brian Chhun <br...@getbraintree.com>.
Thanks, I'm basing the things I'm doing based on what I see there. One
thing that's not clear to me in that example is why supervisor is used to
keep the container alive, rather than using some simpler means. It doesn't
look like it's been configured to supervise anything.
On Wed, Dec 2, 2015 at 11:44 AM, Maximilian Michels <mx...@apache.org> wrote:
> Have you looked at
> https://github.com/apache/flink/tree/master/flink-contrib/docker-flink
> ? This demonstrates how to use Flink with Docker. In particular it
> states: "Images [..] run Supervisor to stay alive when running
> containers."
>
> Have a look at flink/config-flink.sh.
>
> Cheers,
> Max
>
> On Wed, Dec 2, 2015 at 6:29 PM, Brian Chhun
> <br...@getbraintree.com> wrote:
> > Yep, I think this makes sense. I'm currently patching the flink-daemon.sh
> > script to remove the `&`, but I don't think it's a very robust solution,
> > particularly when this script changes across versions of Flink. I'm very
> new
> > to Docker, but the resources I've found indicates that the process must
> run
> > in the foreground, though people seem to get around it with some hacks.
> >
> > When I have some time, I can look into refactoring some parts of the
> scripts
> > so that it can be started in the foreground.
> >
> > Thanks,
> > Brian
> >
> > On Wed, Dec 2, 2015 at 3:22 AM, Maximilian Michels <mx...@apache.org>
> wrote:
> >>
> >> Hi Brian,
> >>
> >> I don't recall Docker requires commands to run in the foreground. Still,
> >> if that is your requirement, simply remove the "&" at the end of this
> line
> >> in flink-daemon.sh:
> >>
> >> $JAVA_RUN $JVM_ARGS ${FLINK_ENV_JAVA_OPTS} "${log_setting[@]}"
> -classpath
> >> "`manglePathList "$FLINK_TM_CLASSPATH:$INTERNAL_HADOOP_CLASSPATHS"`"
> >> ${CLASS_TO_RUN} "${ARGS[@]}" > "$out" 2>&1 < /dev/null &
> >>
> >> Cheers,
> >> Max
> >>
> >> On Wed, Dec 2, 2015 at 9:26 AM, Till Rohrmann <tr...@apache.org>
> >> wrote:
> >>>
> >>> Hi Brian,
> >>>
> >>> as far as I know this is at the moment not possible with our scripts.
> >>> However it should be relatively easy to add by simply executing the
> Java
> >>> command in flink-daemon.sh in the foreground. Do you want to add this?
> >>>
> >>> Cheers,
> >>> Till
> >>>
> >>> On Dec 1, 2015 9:40 PM, "Brian Chhun" <br...@getbraintree.com>
> >>> wrote:
> >>>>
> >>>> Hi All,
> >>>>
> >>>> Is it possible to include a command line flag for starting job and
> task
> >>>> managers in the foreground? Currently, `bin/jobmanager.sh` and
> >>>> `bin/taskmanager.sh` rely on `bin/flink-daemon.sh`, which starts these
> >>>> things in the background. I'd like to execute these commands inside a
> docker
> >>>> container, but it's expected that the process is running in the
> foreground.
> >>>> I think it might be useful to have it run in the foreground so that
> it can
> >>>> be hooked into some process supervisors. Any suggestions are
> appreciated.
> >>>>
> >>>>
> >>>> Thanks,
> >>>> Brian
> >>
> >>
> >
>
Re: Including option for starting job and task managers in the foreground
Posted by Maximilian Michels <mx...@apache.org>.
Have you looked at
https://github.com/apache/flink/tree/master/flink-contrib/docker-flink
? This demonstrates how to use Flink with Docker. In particular it
states: "Images [..] run Supervisor to stay alive when running
containers."
Have a look at flink/config-flink.sh.
Cheers,
Max
On Wed, Dec 2, 2015 at 6:29 PM, Brian Chhun
<br...@getbraintree.com> wrote:
> Yep, I think this makes sense. I'm currently patching the flink-daemon.sh
> script to remove the `&`, but I don't think it's a very robust solution,
> particularly when this script changes across versions of Flink. I'm very new
> to Docker, but the resources I've found indicates that the process must run
> in the foreground, though people seem to get around it with some hacks.
>
> When I have some time, I can look into refactoring some parts of the scripts
> so that it can be started in the foreground.
>
> Thanks,
> Brian
>
> On Wed, Dec 2, 2015 at 3:22 AM, Maximilian Michels <mx...@apache.org> wrote:
>>
>> Hi Brian,
>>
>> I don't recall Docker requires commands to run in the foreground. Still,
>> if that is your requirement, simply remove the "&" at the end of this line
>> in flink-daemon.sh:
>>
>> $JAVA_RUN $JVM_ARGS ${FLINK_ENV_JAVA_OPTS} "${log_setting[@]}" -classpath
>> "`manglePathList "$FLINK_TM_CLASSPATH:$INTERNAL_HADOOP_CLASSPATHS"`"
>> ${CLASS_TO_RUN} "${ARGS[@]}" > "$out" 2>&1 < /dev/null &
>>
>> Cheers,
>> Max
>>
>> On Wed, Dec 2, 2015 at 9:26 AM, Till Rohrmann <tr...@apache.org>
>> wrote:
>>>
>>> Hi Brian,
>>>
>>> as far as I know this is at the moment not possible with our scripts.
>>> However it should be relatively easy to add by simply executing the Java
>>> command in flink-daemon.sh in the foreground. Do you want to add this?
>>>
>>> Cheers,
>>> Till
>>>
>>> On Dec 1, 2015 9:40 PM, "Brian Chhun" <br...@getbraintree.com>
>>> wrote:
>>>>
>>>> Hi All,
>>>>
>>>> Is it possible to include a command line flag for starting job and task
>>>> managers in the foreground? Currently, `bin/jobmanager.sh` and
>>>> `bin/taskmanager.sh` rely on `bin/flink-daemon.sh`, which starts these
>>>> things in the background. I'd like to execute these commands inside a docker
>>>> container, but it's expected that the process is running in the foreground.
>>>> I think it might be useful to have it run in the foreground so that it can
>>>> be hooked into some process supervisors. Any suggestions are appreciated.
>>>>
>>>>
>>>> Thanks,
>>>> Brian
>>
>>
>
Re: Including option for starting job and task managers in the foreground
Posted by Brian Chhun <br...@getbraintree.com>.
Yep, I think this makes sense. I'm currently patching the flink-daemon.sh
script to remove the `&`, but I don't think it's a very robust solution,
particularly when this script changes across versions of Flink. I'm very
new to Docker, but the resources I've found indicates that the process must
run in the foreground, though people seem to get around it with some hacks.
When I have some time, I can look into refactoring some parts of the
scripts so that it can be started in the foreground.
Thanks,
Brian
On Wed, Dec 2, 2015 at 3:22 AM, Maximilian Michels <mx...@apache.org> wrote:
> Hi Brian,
>
> I don't recall Docker requires commands to run in the foreground. Still,
> if that is your requirement, simply remove the "&" at the end of this line
> in flink-daemon.sh:
>
> $JAVA_RUN $JVM_ARGS ${FLINK_ENV_JAVA_OPTS} "${log_setting[@]}" -classpath
> "`manglePathList "$FLINK_TM_CLASSPATH:$INTERNAL_HADOOP_CLASSPATHS"`"
> ${CLASS_TO_RUN} "${ARGS[@]}" > "$out" 2>&1 < /dev/null &
>
> Cheers,
> Max
>
> On Wed, Dec 2, 2015 at 9:26 AM, Till Rohrmann <tr...@apache.org>
> wrote:
>
>> Hi Brian,
>>
>> as far as I know this is at the moment not possible with our scripts.
>> However it should be relatively easy to add by simply executing the Java
>> command in flink-daemon.sh in the foreground. Do you want to add this?
>>
>> Cheers,
>> Till
>> On Dec 1, 2015 9:40 PM, "Brian Chhun" <br...@getbraintree.com>
>> wrote:
>>
>>> Hi All,
>>>
>>> Is it possible to include a command line flag for starting job and task
>>> managers in the foreground? Currently, `bin/jobmanager.sh` and
>>> `bin/taskmanager.sh` rely on `bin/flink-daemon.sh`, which starts these
>>> things in the background. I'd like to execute these commands inside a
>>> docker container, but it's expected that the process is running in the
>>> foreground. I think it might be useful to have it run in the foreground so
>>> that it can be hooked into some process supervisors. Any suggestions are
>>> appreciated.
>>>
>>>
>>> Thanks,
>>> Brian
>>>
>>
>
Re: Including option for starting job and task managers in the foreground
Posted by Maximilian Michels <mx...@apache.org>.
Hi Brian,
I don't recall Docker requires commands to run in the foreground. Still, if
that is your requirement, simply remove the "&" at the end of this line in
flink-daemon.sh:
$JAVA_RUN $JVM_ARGS ${FLINK_ENV_JAVA_OPTS} "${log_setting[@]}" -classpath
"`manglePathList "$FLINK_TM_CLASSPATH:$INTERNAL_HADOOP_CLASSPATHS"`"
${CLASS_TO_RUN} "${ARGS[@]}" > "$out" 2>&1 < /dev/null &
Cheers,
Max
On Wed, Dec 2, 2015 at 9:26 AM, Till Rohrmann <tr...@apache.org> wrote:
> Hi Brian,
>
> as far as I know this is at the moment not possible with our scripts.
> However it should be relatively easy to add by simply executing the Java
> command in flink-daemon.sh in the foreground. Do you want to add this?
>
> Cheers,
> Till
> On Dec 1, 2015 9:40 PM, "Brian Chhun" <br...@getbraintree.com>
> wrote:
>
>> Hi All,
>>
>> Is it possible to include a command line flag for starting job and task
>> managers in the foreground? Currently, `bin/jobmanager.sh` and
>> `bin/taskmanager.sh` rely on `bin/flink-daemon.sh`, which starts these
>> things in the background. I'd like to execute these commands inside a
>> docker container, but it's expected that the process is running in the
>> foreground. I think it might be useful to have it run in the foreground so
>> that it can be hooked into some process supervisors. Any suggestions are
>> appreciated.
>>
>>
>> Thanks,
>> Brian
>>
>
Re: Including option for starting job and task managers in the foreground
Posted by Till Rohrmann <tr...@apache.org>.
Hi Brian,
as far as I know this is at the moment not possible with our scripts.
However it should be relatively easy to add by simply executing the Java
command in flink-daemon.sh in the foreground. Do you want to add this?
Cheers,
Till
On Dec 1, 2015 9:40 PM, "Brian Chhun" <br...@getbraintree.com> wrote:
> Hi All,
>
> Is it possible to include a command line flag for starting job and task
> managers in the foreground? Currently, `bin/jobmanager.sh` and
> `bin/taskmanager.sh` rely on `bin/flink-daemon.sh`, which starts these
> things in the background. I'd like to execute these commands inside a
> docker container, but it's expected that the process is running in the
> foreground. I think it might be useful to have it run in the foreground so
> that it can be hooked into some process supervisors. Any suggestions are
> appreciated.
>
>
> Thanks,
> Brian
>