Spark Bootstrapping on Mesos

[Stephen Carman <sc...@coldlight.com>]

Hi,

We have a small mesos cluster and we’d like to be able to initialize some of our classes, mostly we have a vfs we setup to be allow our code
to access S3, but there doesn’t seem to be any readily obvious way to bootstrap these kind of classes so that they have the properly initialized configuration they need
to operate.

Is there some accepted way to accomplish this?


thanks,
Steve
This e-mail is intended solely for the above-mentioned recipient and it may contain confidential or privileged information. If you have received it in error, please notify us immediately and delete the e-mail. You must not copy, distribute, disclose or take any action in reliance on it. In addition, the contents of an attachment to this e-mail may contain software viruses which could damage your own computer system. While ColdLight Solutions, LLC has taken every reasonable precaution to minimize this risk, we cannot accept liability for any damage which you sustain as a result of software viruses. You should perform your own virus checks before opening the attachment.

Re: Spark Bootstrapping on Mesos

[Stephen Carman <sc...@coldlight.com>]

Okay my apologies perhaps I’m just not explaining this well.

So we have an application, that creates a spark context and adds all the necessary jars, the job is submitted to the cluster
and as it runs the code it attempts to pull the data necessary for the job from s3. We use our own version of Presto’s s3 filesystem
to be able to read and write the data from s3, but that needs to be initialized with our keys and the s3 bucket before the code can understand
how to handle that file. All the Jars and everything get to the slave fine, so it has no problems with ClassNotFound, it just doesn’t seem to be initilizing
this s3 file system, so when it tries to download the data from s3 it fails saying it doesn’t know how to handle an s3 URI.

Hopefully that makes sense, any insight?

Thanks,
Steve


On May 13, 2015, at 2:03 PM, Tim Chen <ti...@mesosphere.io>> wrote:

Hi Stephen,

I'm not quite sure what you mean by bootstrapping classes, do you have some particular examples?

Usually to run any user jar you just need it to be reachable by your slaves so it can be either S3 or any accessible place, then you just provide your jar url when you run spark-submit.

Tim

On Wed, May 13, 2015 at 8:09 AM, Stephen Carman <sc...@coldlight.com>> wrote:
Hi,

We have a small mesos cluster and we’d like to be able to initialize some of our classes, mostly we have a vfs we setup to be allow our code
to access S3, but there doesn’t seem to be any readily obvious way to bootstrap these kind of classes so that they have the properly initialized configuration they need
to operate.

Is there some accepted way to accomplish this?


thanks,
Steve
This e-mail is intended solely for the above-mentioned recipient and it may contain confidential or privileged information. If you have received it in error, please notify us immediately and delete the e-mail. You must not copy, distribute, disclose or take any action in reliance on it. In addition, the contents of an attachment to this e-mail may contain software viruses which could damage your own computer system. While ColdLight Solutions, LLC has taken every reasonable precaution to minimize this risk, we cannot accept liability for any damage which you sustain as a result of software viruses. You should perform your own virus checks before opening the attachment.


This e-mail is intended solely for the above-mentioned recipient and it may contain confidential or privileged information. If you have received it in error, please notify us immediately and delete the e-mail. You must not copy, distribute, disclose or take any action in reliance on it. In addition, the contents of an attachment to this e-mail may contain software viruses which could damage your own computer system. While ColdLight Solutions, LLC has taken every reasonable precaution to minimize this risk, we cannot accept liability for any damage which you sustain as a result of software viruses. You should perform your own virus checks before opening the attachment.

Re: Spark Bootstrapping on Mesos

[Tim Chen <ti...@mesosphere.io>]

Hi Stephen,

I'm not quite sure what you mean by bootstrapping classes, do you have some
particular examples?

Usually to run any user jar you just need it to be reachable by your slaves
so it can be either S3 or any accessible place, then you just provide your
jar url when you run spark-submit.

Tim

On Wed, May 13, 2015 at 8:09 AM, Stephen Carman <sc...@coldlight.com>
wrote:

> Hi,
>
> We have a small mesos cluster and we’d like to be able to initialize some
> of our classes, mostly we have a vfs we setup to be allow our code
> to access S3, but there doesn’t seem to be any readily obvious way to
> bootstrap these kind of classes so that they have the properly initialized
> configuration they need
> to operate.
>
> Is there some accepted way to accomplish this?
>
>
> thanks,
> Steve
> This e-mail is intended solely for the above-mentioned recipient and it
> may contain confidential or privileged information. If you have received it
> in error, please notify us immediately and delete the e-mail. You must not
> copy, distribute, disclose or take any action in reliance on it. In
> addition, the contents of an attachment to this e-mail may contain software
> viruses which could damage your own computer system. While ColdLight
> Solutions, LLC has taken every reasonable precaution to minimize this risk,
> we cannot accept liability for any damage which you sustain as a result of
> software viruses. You should perform your own virus checks before opening
> the attachment.
>