Re: Running Spark master/slave instances in non Daemon mode

[Jakob Odersky <ja...@odersky.com>]

Hi Mike,
I can imagine the trouble that daemonization is causing and I think
that having non-forking start script is a good idea. A simple,
non-intrusive, fix could be to change the "spark-daemon.sh" script to
conditionally omit the "nohup &".
Personally, I think the semantically correct approach would be to also
rename "spark-daemon" to something else (since it won't necessarily
start a background process anymore), however that may have the
potential to break things, in which case it is probably not worth
cosmetic rename.

best,
--Jakob


On Thu, Sep 29, 2016 at 6:47 PM, Mike Ihbe <mi...@mustwin.com> wrote:
> Our particular use case is for Nomad, using the "exec" configuration
> described here: https://www.nomadproject.io/docs/drivers/exec.html. It's not
> exactly a container, just a cgroup. It performs a simple fork/exec of a
> command and binds to the output fds from that process, so daemonizing is
> causing us minor hardship and seems like an easy thing to make optional.
> We'd be happy to make the PR as well.
>
> --Mike
>
> On Thu, Sep 29, 2016 at 5:25 PM, Jakob Odersky <ja...@odersky.com> wrote:
>>
>> I'm curious, what kind of container solutions require foreground
>> processes? Most init systems work fine with "starter" processes that
>> run other processes. IIRC systemd and start-stop-daemon have an option
>> called "fork", that will expect the main process to run another one in
>> the background and only consider the former complete when the latter
>> exits. I'm not against having a non-forking start script, I'm just
>> wondering where you'd run into issues.
>>
>> Regarding the logging, would it be an option to create a custom slf4j
>> logger that uses the standard mechanisms exposed by the system?
>>
>> best,
>> --Jakob
>>
>> On Thu, Sep 29, 2016 at 3:46 PM, jpuro <jp...@mustwin.com> wrote:
>> > Hi,
>> >
>> > I recently tried deploying Spark master and slave instances to container
>> > based environments such as Docker, Nomad etc. There are two issues that
>> > I've
>> > found with how the startup scripts work. The sbin/start-master.sh and
>> > sbin/start-slave.sh start a daemon by default, but this isn't as
>> > compatible
>> > with container deployments as one would think. The first issue is that
>> > the
>> > daemon runs in the background and some container solutions require the
>> > apps
>> > to run in the foreground or they consider the application to not be
>> > running
>> > and they may close down the task. The second issue is that logs don't
>> > seem
>> > to get integrated with the logging mechanism in the container solution.
>> > What
>> > is the possibility of adding additional flags or startup scripts for
>> > supporting Spark to run in the foreground? It would be great if a flag
>> > like
>> > SPARK_NO_DAEMONIZE could be added or another script for foreground
>> > execution.
>> >
>> > Regards,
>> >
>> > Jeff
>> >
>> >
>> >
>> > --
>> > View this message in context:
>> > http://apache-spark-developers-list.1001551.n3.nabble.com/Running-Spark-master-slave-instances-in-non-Daemon-mode-tp19172.html
>> > Sent from the Apache Spark Developers List mailing list archive at
>> > Nabble.com.
>> >
>> > ---------------------------------------------------------------------
>> > To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>> >
>>
>> ---------------------------------------------------------------------
>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>
>
>
>
> --
> Mike Ihbe
> MustWin - Principal
>
> mike@mustwin.com
> mikejihbe@gmail.com
> skype: mikeihbe
> Cell: 651.283.0815

---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org