You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@livy.apache.org by Damon Cortesi <da...@apache.org> on 2022/12/03 01:20:45 UTC
Re: [DISCUSS] Livy Dev and Test Environment
Coming back to this as I get my dev environment up and running, there's definitely an intermix of dependencies between Spark, Python, and R that I'm still working out.
For example, when I try to start sparkR I get an error message that "package ‘SparkR’ was built under R version 4.0.4", but locally I have R version 3.5.2 installed. Spark 3.3.1 says you need R 3.5+. That said, think my version of R works with Spark2 (at least the tests indicate that...)
It'd be great to have a minimum viable environment with specific versions and I hope to have that in a Docker environment by early next week. :)
Currently I'm just basing it off a debian image with Java8, although there are Spark images that could be useful...
Damon
On 2022/11/20 18:55:35 larry mccay wrote:
> Considering there is no download for anything older than 3.2.x on the
> referred download page, we likely need some change to the README.md to
> reflect a more modern version.
> We also need more explicit instructions for installing Spark than just the
> download. Whether we detail this or point to Spark docs that are sufficient
> is certainly a consideration.
>
> At the end of the day, we are missing any sort of quick start guide for
> devs to be able to successfully build and/or run tests.
>
> Thoughts?
>
> On Sat, Nov 19, 2022 at 6:23 PM larry mccay <la...@gmail.com> wrote:
>
> > Hey Folks -
> >
> > Our Livy README.md indicates the following:
> >
> > To run Livy, you will also need a Spark installation. You can get Spark
> > releases at https://spark.apache.org/downloads.html.
> >
> > Livy requires Spark 2.4+. You can switch to a different version of Spark
> > by setting the SPARK_HOME environment variable in the Livy server
> > process, without needing to rebuild Livy.
> >
> > Do we have any variation on this setup at this point in the real world?
> >
> > What do your dev environments actually look like and how are you
> > installing what versions of Spark as a dependency?
> >
> > Thanks!
> >
> > --larry
> >
>
Re: [DISCUSS] Livy Dev and Test Environment
Posted by Damon Cortesi <da...@apache.org>.
Thanks, Sumit!
I think I'm leaning towards sticking with ubuntu base images as well. There's not a huge concern here re: image size, so it'd be nice to have a more familiar environment especially if folks end up using this as a local dev environment.
On 2022/12/07 23:09:56 Sumit Kumar wrote:
> Thanks Damon for the PR. I'm excited about this development. I'm in favor of minimal dependencies and propose that we stick with ubuntu base images. I've had better success installing maven and other tools individually rather than using the blotted container images. With layering, those tools won't be installed again, so it's fine. I will share some more comments in the PR.
>
>
>
>
>
>
>
> ---- On Wed, 07 Dec 2022 14:49:43 -0800 Damon Cortesi <da...@apache.org> wrote ---
>
>
>
> Appreciate the feedback!
>
> Regarding the first two points, I think we'll learn more as we begin to modernize the project including deprecation of Python2 and moving to a newer version of Java.
>
> The maven/alpine images are nice because they've got two crucial build dependencies easily pinned, whereas with Ubuntu those versions are more static but that doesn't preclude us from downloading a specific version if need be.
>
> > Wouldn't publishing it as an image make it harder to make changes than just changing the dockerfile
> Yes, ineed. For now, I think the Dockerfile will just be used locally for build/test with a known-good environment. It could be nice to provide these images on github so folks don't need to build them themselves...but certainly not a requirement.
>
> Thanks!
>
> Damon
>
> On 2022/12/07 22:40:44 larry mccay wrote:
> > Hi Damon -
> >
> > I saw the PR - that's great!
> >
> > - Is it important to have the Dockerfile as close to travis as possible
> > I could see how this could be meaningful but I don't consider it a must.
> >
> > - Will alpine-based images be tough to maintain/support
> > I lack whatever familiarity I need to contribute to this question.
> >
> > - Will we publish some version of the resulting images, if so size could be
> > important
> > I don't imagine that we would but perhaps I am thinking about the images
> > differently.
> > Do you mean the local image that is created when spinning up the build
> > environment?
> > Wouldn't publishing it as an image make it harder to make changes than just
> > changing the dockerfile, etc and letting it be realized on local dev
> > machines?
> > Would this be more for releng orgs that need to build as well?
> >
> > thanks!
> >
> > --larry
> >
> > On Wed, Dec 7, 2022 at 5:32 PM Damon Cortesi <ma...@apache.org> wrote:
> >
> > > I've got an initial PR for this here:
> > > https://github.com/apache/incubator-livy/pull/367
> > >
> > > It uses Ubuntu Xenial (similar to our Travis environment) and installs the
> > > necessary Python packages and R. I have not added Spark to the package as
> > > it gets pulled down as a dependency, but am considering doing that since
> > > it's such a large download.
> > >
> > > I've got another version of the Dockerfile based off of
> > > "maven:3-jdk-8-alpine" that works as well. It's quite a bit smaller than
> > > the xenial version (521MB vs. 1.18GB), but still figuring out exactly how I
> > > want the Docker image to work. A couple open questions
> > >
> > > - Is it important to have the Dockerfile as close to travis as possible
> > > - Will alpine-based images be tough to maintain/support
> > > - Will we publish some version of the resulting images, if so size could
> > > be important
> > >
> > > Damon
> > >
> > > On 2022/12/03 01:20:45 Damon Cortesi wrote:
> > > > Coming back to this as I get my dev environment up and running, there's
> > > definitely an intermix of dependencies between Spark, Python, and R that
> > > I'm still working out.
> > > >
> > > > For example, when I try to start sparkR I get an error message that
> > > "package ‘SparkR’ was built under R version 4.0.4", but locally I have R
> > > version 3.5.2 installed. Spark 3.3.1 says you need R 3.5+. That said, think
> > > my version of R works with Spark2 (at least the tests indicate that...)
> > > >
> > > > It'd be great to have a minimum viable environment with specific
> > > versions and I hope to have that in a Docker environment by early next
> > > week. :)
> > > >
> > > > Currently I'm just basing it off a debian image with Java8, although
> > > there are Spark images that could be useful...
> > > >
> > > > Damon
> > > >
> > > > On 2022/11/20 18:55:35 larry mccay wrote:
> > > > > Considering there is no download for anything older than 3.2.x on the
> > > > > referred download page, we likely need some change to the README.md to
> > > > > reflect a more modern version.
> > > > > We also need more explicit instructions for installing Spark than just
> > > the
> > > > > download. Whether we detail this or point to Spark docs that are
> > > sufficient
> > > > > is certainly a consideration.
> > > > >
> > > > > At the end of the day, we are missing any sort of quick start guide for
> > > > > devs to be able to successfully build and/or run tests.
> > > > >
> > > > > Thoughts?
> > > > >
> > > > > On Sat, Nov 19, 2022 at 6:23 PM larry mccay <ma...@gmail.com>
> > > wrote:
> > > > >
> > > > > > Hey Folks -
> > > > > >
> > > > > > Our Livy README.md indicates the following:
> > > > > >
> > > > > > To run Livy, you will also need a Spark installation. You can get
> > > Spark
> > > > > > releases at https://spark.apache.org/downloads.html.
> > > > > >
> > > > > > Livy requires Spark 2.4+. You can switch to a different version of
> > > Spark
> > > > > > by setting the SPARK_HOME environment variable in the Livy server
> > > > > > process, without needing to rebuild Livy.
> > > > > >
> > > > > > Do we have any variation on this setup at this point in the real
> > > world?
> > > > > >
> > > > > > What do your dev environments actually look like and how are you
> > > > > > installing what versions of Spark as a dependency?
> > > > > >
> > > > > > Thanks!
> > > > > >
> > > > > > --larry
> > > > > >
> > > > >
> > > >
> > >
> >
Re: [DISCUSS] Livy Dev and Test Environment
Posted by Sumit Kumar <ks...@apache.org>.
Thanks Damon for the PR. I'm excited about this development. I'm in favor of minimal dependencies and propose that we stick with ubuntu base images. I've had better success installing maven and other tools individually rather than using the blotted container images. With layering, those tools won't be installed again, so it's fine. I will share some more comments in the PR.
---- On Wed, 07 Dec 2022 14:49:43 -0800 Damon Cortesi <da...@apache.org> wrote ---
Appreciate the feedback!
Regarding the first two points, I think we'll learn more as we begin to modernize the project including deprecation of Python2 and moving to a newer version of Java.
The maven/alpine images are nice because they've got two crucial build dependencies easily pinned, whereas with Ubuntu those versions are more static but that doesn't preclude us from downloading a specific version if need be.
> Wouldn't publishing it as an image make it harder to make changes than just changing the dockerfile
Yes, ineed. For now, I think the Dockerfile will just be used locally for build/test with a known-good environment. It could be nice to provide these images on github so folks don't need to build them themselves...but certainly not a requirement.
Thanks!
Damon
On 2022/12/07 22:40:44 larry mccay wrote:
> Hi Damon -
>
> I saw the PR - that's great!
>
> - Is it important to have the Dockerfile as close to travis as possible
> I could see how this could be meaningful but I don't consider it a must.
>
> - Will alpine-based images be tough to maintain/support
> I lack whatever familiarity I need to contribute to this question.
>
> - Will we publish some version of the resulting images, if so size could be
> important
> I don't imagine that we would but perhaps I am thinking about the images
> differently.
> Do you mean the local image that is created when spinning up the build
> environment?
> Wouldn't publishing it as an image make it harder to make changes than just
> changing the dockerfile, etc and letting it be realized on local dev
> machines?
> Would this be more for releng orgs that need to build as well?
>
> thanks!
>
> --larry
>
> On Wed, Dec 7, 2022 at 5:32 PM Damon Cortesi <ma...@apache.org> wrote:
>
> > I've got an initial PR for this here:
> > https://github.com/apache/incubator-livy/pull/367
> >
> > It uses Ubuntu Xenial (similar to our Travis environment) and installs the
> > necessary Python packages and R. I have not added Spark to the package as
> > it gets pulled down as a dependency, but am considering doing that since
> > it's such a large download.
> >
> > I've got another version of the Dockerfile based off of
> > "maven:3-jdk-8-alpine" that works as well. It's quite a bit smaller than
> > the xenial version (521MB vs. 1.18GB), but still figuring out exactly how I
> > want the Docker image to work. A couple open questions
> >
> > - Is it important to have the Dockerfile as close to travis as possible
> > - Will alpine-based images be tough to maintain/support
> > - Will we publish some version of the resulting images, if so size could
> > be important
> >
> > Damon
> >
> > On 2022/12/03 01:20:45 Damon Cortesi wrote:
> > > Coming back to this as I get my dev environment up and running, there's
> > definitely an intermix of dependencies between Spark, Python, and R that
> > I'm still working out.
> > >
> > > For example, when I try to start sparkR I get an error message that
> > "package ‘SparkR’ was built under R version 4.0.4", but locally I have R
> > version 3.5.2 installed. Spark 3.3.1 says you need R 3.5+. That said, think
> > my version of R works with Spark2 (at least the tests indicate that...)
> > >
> > > It'd be great to have a minimum viable environment with specific
> > versions and I hope to have that in a Docker environment by early next
> > week. :)
> > >
> > > Currently I'm just basing it off a debian image with Java8, although
> > there are Spark images that could be useful...
> > >
> > > Damon
> > >
> > > On 2022/11/20 18:55:35 larry mccay wrote:
> > > > Considering there is no download for anything older than 3.2.x on the
> > > > referred download page, we likely need some change to the README.md to
> > > > reflect a more modern version.
> > > > We also need more explicit instructions for installing Spark than just
> > the
> > > > download. Whether we detail this or point to Spark docs that are
> > sufficient
> > > > is certainly a consideration.
> > > >
> > > > At the end of the day, we are missing any sort of quick start guide for
> > > > devs to be able to successfully build and/or run tests.
> > > >
> > > > Thoughts?
> > > >
> > > > On Sat, Nov 19, 2022 at 6:23 PM larry mccay <ma...@gmail.com>
> > wrote:
> > > >
> > > > > Hey Folks -
> > > > >
> > > > > Our Livy README.md indicates the following:
> > > > >
> > > > > To run Livy, you will also need a Spark installation. You can get
> > Spark
> > > > > releases at https://spark.apache.org/downloads.html.
> > > > >
> > > > > Livy requires Spark 2.4+. You can switch to a different version of
> > Spark
> > > > > by setting the SPARK_HOME environment variable in the Livy server
> > > > > process, without needing to rebuild Livy.
> > > > >
> > > > > Do we have any variation on this setup at this point in the real
> > world?
> > > > >
> > > > > What do your dev environments actually look like and how are you
> > > > > installing what versions of Spark as a dependency?
> > > > >
> > > > > Thanks!
> > > > >
> > > > > --larry
> > > > >
> > > >
> > >
> >
>
Re: [DISCUSS] Livy Dev and Test Environment
Posted by Damon Cortesi <da...@apache.org>.
Appreciate the feedback!
Regarding the first two points, I think we'll learn more as we begin to modernize the project including deprecation of Python2 and moving to a newer version of Java.
The maven/alpine images are nice because they've got two crucial build dependencies easily pinned, whereas with Ubuntu those versions are more static but that doesn't preclude us from downloading a specific version if need be.
> Wouldn't publishing it as an image make it harder to make changes than just changing the dockerfile
Yes, ineed. For now, I think the Dockerfile will just be used locally for build/test with a known-good environment. It could be nice to provide these images on github so folks don't need to build them themselves...but certainly not a requirement.
Thanks!
Damon
On 2022/12/07 22:40:44 larry mccay wrote:
> Hi Damon -
>
> I saw the PR - that's great!
>
> - Is it important to have the Dockerfile as close to travis as possible
> I could see how this could be meaningful but I don't consider it a must.
>
> - Will alpine-based images be tough to maintain/support
> I lack whatever familiarity I need to contribute to this question.
>
> - Will we publish some version of the resulting images, if so size could be
> important
> I don't imagine that we would but perhaps I am thinking about the images
> differently.
> Do you mean the local image that is created when spinning up the build
> environment?
> Wouldn't publishing it as an image make it harder to make changes than just
> changing the dockerfile, etc and letting it be realized on local dev
> machines?
> Would this be more for releng orgs that need to build as well?
>
> thanks!
>
> --larry
>
> On Wed, Dec 7, 2022 at 5:32 PM Damon Cortesi <da...@apache.org> wrote:
>
> > I've got an initial PR for this here:
> > https://github.com/apache/incubator-livy/pull/367
> >
> > It uses Ubuntu Xenial (similar to our Travis environment) and installs the
> > necessary Python packages and R. I have not added Spark to the package as
> > it gets pulled down as a dependency, but am considering doing that since
> > it's such a large download.
> >
> > I've got another version of the Dockerfile based off of
> > "maven:3-jdk-8-alpine" that works as well. It's quite a bit smaller than
> > the xenial version (521MB vs. 1.18GB), but still figuring out exactly how I
> > want the Docker image to work. A couple open questions
> >
> > - Is it important to have the Dockerfile as close to travis as possible
> > - Will alpine-based images be tough to maintain/support
> > - Will we publish some version of the resulting images, if so size could
> > be important
> >
> > Damon
> >
> > On 2022/12/03 01:20:45 Damon Cortesi wrote:
> > > Coming back to this as I get my dev environment up and running, there's
> > definitely an intermix of dependencies between Spark, Python, and R that
> > I'm still working out.
> > >
> > > For example, when I try to start sparkR I get an error message that
> > "package ‘SparkR’ was built under R version 4.0.4", but locally I have R
> > version 3.5.2 installed. Spark 3.3.1 says you need R 3.5+. That said, think
> > my version of R works with Spark2 (at least the tests indicate that...)
> > >
> > > It'd be great to have a minimum viable environment with specific
> > versions and I hope to have that in a Docker environment by early next
> > week. :)
> > >
> > > Currently I'm just basing it off a debian image with Java8, although
> > there are Spark images that could be useful...
> > >
> > > Damon
> > >
> > > On 2022/11/20 18:55:35 larry mccay wrote:
> > > > Considering there is no download for anything older than 3.2.x on the
> > > > referred download page, we likely need some change to the README.md to
> > > > reflect a more modern version.
> > > > We also need more explicit instructions for installing Spark than just
> > the
> > > > download. Whether we detail this or point to Spark docs that are
> > sufficient
> > > > is certainly a consideration.
> > > >
> > > > At the end of the day, we are missing any sort of quick start guide for
> > > > devs to be able to successfully build and/or run tests.
> > > >
> > > > Thoughts?
> > > >
> > > > On Sat, Nov 19, 2022 at 6:23 PM larry mccay <la...@gmail.com>
> > wrote:
> > > >
> > > > > Hey Folks -
> > > > >
> > > > > Our Livy README.md indicates the following:
> > > > >
> > > > > To run Livy, you will also need a Spark installation. You can get
> > Spark
> > > > > releases at https://spark.apache.org/downloads.html.
> > > > >
> > > > > Livy requires Spark 2.4+. You can switch to a different version of
> > Spark
> > > > > by setting the SPARK_HOME environment variable in the Livy server
> > > > > process, without needing to rebuild Livy.
> > > > >
> > > > > Do we have any variation on this setup at this point in the real
> > world?
> > > > >
> > > > > What do your dev environments actually look like and how are you
> > > > > installing what versions of Spark as a dependency?
> > > > >
> > > > > Thanks!
> > > > >
> > > > > --larry
> > > > >
> > > >
> > >
> >
>
Re: [DISCUSS] Livy Dev and Test Environment
Posted by larry mccay <lm...@apache.org>.
Hi Damon -
I saw the PR - that's great!
- Is it important to have the Dockerfile as close to travis as possible
I could see how this could be meaningful but I don't consider it a must.
- Will alpine-based images be tough to maintain/support
I lack whatever familiarity I need to contribute to this question.
- Will we publish some version of the resulting images, if so size could be
important
I don't imagine that we would but perhaps I am thinking about the images
differently.
Do you mean the local image that is created when spinning up the build
environment?
Wouldn't publishing it as an image make it harder to make changes than just
changing the dockerfile, etc and letting it be realized on local dev
machines?
Would this be more for releng orgs that need to build as well?
thanks!
--larry
On Wed, Dec 7, 2022 at 5:32 PM Damon Cortesi <da...@apache.org> wrote:
> I've got an initial PR for this here:
> https://github.com/apache/incubator-livy/pull/367
>
> It uses Ubuntu Xenial (similar to our Travis environment) and installs the
> necessary Python packages and R. I have not added Spark to the package as
> it gets pulled down as a dependency, but am considering doing that since
> it's such a large download.
>
> I've got another version of the Dockerfile based off of
> "maven:3-jdk-8-alpine" that works as well. It's quite a bit smaller than
> the xenial version (521MB vs. 1.18GB), but still figuring out exactly how I
> want the Docker image to work. A couple open questions
>
> - Is it important to have the Dockerfile as close to travis as possible
> - Will alpine-based images be tough to maintain/support
> - Will we publish some version of the resulting images, if so size could
> be important
>
> Damon
>
> On 2022/12/03 01:20:45 Damon Cortesi wrote:
> > Coming back to this as I get my dev environment up and running, there's
> definitely an intermix of dependencies between Spark, Python, and R that
> I'm still working out.
> >
> > For example, when I try to start sparkR I get an error message that
> "package ‘SparkR’ was built under R version 4.0.4", but locally I have R
> version 3.5.2 installed. Spark 3.3.1 says you need R 3.5+. That said, think
> my version of R works with Spark2 (at least the tests indicate that...)
> >
> > It'd be great to have a minimum viable environment with specific
> versions and I hope to have that in a Docker environment by early next
> week. :)
> >
> > Currently I'm just basing it off a debian image with Java8, although
> there are Spark images that could be useful...
> >
> > Damon
> >
> > On 2022/11/20 18:55:35 larry mccay wrote:
> > > Considering there is no download for anything older than 3.2.x on the
> > > referred download page, we likely need some change to the README.md to
> > > reflect a more modern version.
> > > We also need more explicit instructions for installing Spark than just
> the
> > > download. Whether we detail this or point to Spark docs that are
> sufficient
> > > is certainly a consideration.
> > >
> > > At the end of the day, we are missing any sort of quick start guide for
> > > devs to be able to successfully build and/or run tests.
> > >
> > > Thoughts?
> > >
> > > On Sat, Nov 19, 2022 at 6:23 PM larry mccay <la...@gmail.com>
> wrote:
> > >
> > > > Hey Folks -
> > > >
> > > > Our Livy README.md indicates the following:
> > > >
> > > > To run Livy, you will also need a Spark installation. You can get
> Spark
> > > > releases at https://spark.apache.org/downloads.html.
> > > >
> > > > Livy requires Spark 2.4+. You can switch to a different version of
> Spark
> > > > by setting the SPARK_HOME environment variable in the Livy server
> > > > process, without needing to rebuild Livy.
> > > >
> > > > Do we have any variation on this setup at this point in the real
> world?
> > > >
> > > > What do your dev environments actually look like and how are you
> > > > installing what versions of Spark as a dependency?
> > > >
> > > > Thanks!
> > > >
> > > > --larry
> > > >
> > >
> >
>
Re: [DISCUSS] Livy Dev and Test Environment
Posted by Damon Cortesi <da...@apache.org>.
I've got an initial PR for this here: https://github.com/apache/incubator-livy/pull/367
It uses Ubuntu Xenial (similar to our Travis environment) and installs the necessary Python packages and R. I have not added Spark to the package as it gets pulled down as a dependency, but am considering doing that since it's such a large download.
I've got another version of the Dockerfile based off of "maven:3-jdk-8-alpine" that works as well. It's quite a bit smaller than the xenial version (521MB vs. 1.18GB), but still figuring out exactly how I want the Docker image to work. A couple open questions
- Is it important to have the Dockerfile as close to travis as possible
- Will alpine-based images be tough to maintain/support
- Will we publish some version of the resulting images, if so size could be important
Damon
On 2022/12/03 01:20:45 Damon Cortesi wrote:
> Coming back to this as I get my dev environment up and running, there's definitely an intermix of dependencies between Spark, Python, and R that I'm still working out.
>
> For example, when I try to start sparkR I get an error message that "package ‘SparkR’ was built under R version 4.0.4", but locally I have R version 3.5.2 installed. Spark 3.3.1 says you need R 3.5+. That said, think my version of R works with Spark2 (at least the tests indicate that...)
>
> It'd be great to have a minimum viable environment with specific versions and I hope to have that in a Docker environment by early next week. :)
>
> Currently I'm just basing it off a debian image with Java8, although there are Spark images that could be useful...
>
> Damon
>
> On 2022/11/20 18:55:35 larry mccay wrote:
> > Considering there is no download for anything older than 3.2.x on the
> > referred download page, we likely need some change to the README.md to
> > reflect a more modern version.
> > We also need more explicit instructions for installing Spark than just the
> > download. Whether we detail this or point to Spark docs that are sufficient
> > is certainly a consideration.
> >
> > At the end of the day, we are missing any sort of quick start guide for
> > devs to be able to successfully build and/or run tests.
> >
> > Thoughts?
> >
> > On Sat, Nov 19, 2022 at 6:23 PM larry mccay <la...@gmail.com> wrote:
> >
> > > Hey Folks -
> > >
> > > Our Livy README.md indicates the following:
> > >
> > > To run Livy, you will also need a Spark installation. You can get Spark
> > > releases at https://spark.apache.org/downloads.html.
> > >
> > > Livy requires Spark 2.4+. You can switch to a different version of Spark
> > > by setting the SPARK_HOME environment variable in the Livy server
> > > process, without needing to rebuild Livy.
> > >
> > > Do we have any variation on this setup at this point in the real world?
> > >
> > > What do your dev environments actually look like and how are you
> > > installing what versions of Spark as a dependency?
> > >
> > > Thanks!
> > >
> > > --larry
> > >
> >
>
Re: [DISCUSS] Livy Dev and Test Environment
Posted by Sumit Kumar <ks...@apache.org>.
+1 I am looking forward to it too. ---- On Fri, 02 Dec 2022 23:12:36 -0800 jb@nanthrax.net wrote ----Hi,
yes, I think docker (and eventually k8s yaml files or helm chart) is way better.
I think we could include it in the project and update the started
guide accordingly.
Regards
JB
On Sat, Dec 3, 2022 at 2:44 AM larry mccay <lm...@apache.org> wrote:
>
> Yes, the docker based approach is a great way.
> @Damon - this sounds terrific - look forward to it next week!
>
>
>
> On Fri, Dec 2, 2022 at 8:20 PM Damon Cortesi <da...@apache.org> wrote:
>
> > Coming back to this as I get my dev environment up and running, there's
> > definitely an intermix of dependencies between Spark, Python, and R that
> > I'm still working out.
> >
> > For example, when I try to start sparkR I get an error message that
> > "package ‘SparkR’ was built under R version 4.0.4", but locally I have R
> > version 3.5.2 installed. Spark 3.3.1 says you need R 3.5+. That said, think
> > my version of R works with Spark2 (at least the tests indicate that...)
> >
> > It'd be great to have a minimum viable environment with specific versions
> > and I hope to have that in a Docker environment by early next week. :)
> >
> > Currently I'm just basing it off a debian image with Java8, although there
> > are Spark images that could be useful...
> >
> > Damon
> >
> > On 2022/11/20 18:55:35 larry mccay wrote:
> > > Considering there is no download for anything older than 3.2.x on the
> > > referred download page, we likely need some change to the README.md to
> > > reflect a more modern version.
> > > We also need more explicit instructions for installing Spark than just
> > the
> > > download. Whether we detail this or point to Spark docs that are
> > sufficient
> > > is certainly a consideration.
> > >
> > > At the end of the day, we are missing any sort of quick start guide for
> > > devs to be able to successfully build and/or run tests.
> > >
> > > Thoughts?
> > >
> > > On Sat, Nov 19, 2022 at 6:23 PM larry mccay <la...@gmail.com>
> > wrote:
> > >
> > > > Hey Folks -
> > > >
> > > > Our Livy README.md indicates the following:
> > > >
> > > > To run Livy, you will also need a Spark installation. You can get Spark
> > > > releases at https://spark.apache.org/downloads.html.
> > > >
> > > > Livy requires Spark 2.4+. You can switch to a different version of
> > Spark
> > > > by setting the SPARK_HOME environment variable in the Livy server
> > > > process, without needing to rebuild Livy.
> > > >
> > > > Do we have any variation on this setup at this point in the real world?
> > > >
> > > > What do your dev environments actually look like and how are you
> > > > installing what versions of Spark as a dependency?
> > > >
> > > > Thanks!
> > > >
> > > > --larry
> > > >
> > >
> >
Re: [DISCUSS] Livy Dev and Test Environment
Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.
Hi,
yes, I think docker (and eventually k8s yaml files or helm chart) is way better.
I think we could include it in the project and update the started
guide accordingly.
Regards
JB
On Sat, Dec 3, 2022 at 2:44 AM larry mccay <lm...@apache.org> wrote:
>
> Yes, the docker based approach is a great way.
> @Damon - this sounds terrific - look forward to it next week!
>
>
>
> On Fri, Dec 2, 2022 at 8:20 PM Damon Cortesi <da...@apache.org> wrote:
>
> > Coming back to this as I get my dev environment up and running, there's
> > definitely an intermix of dependencies between Spark, Python, and R that
> > I'm still working out.
> >
> > For example, when I try to start sparkR I get an error message that
> > "package ‘SparkR’ was built under R version 4.0.4", but locally I have R
> > version 3.5.2 installed. Spark 3.3.1 says you need R 3.5+. That said, think
> > my version of R works with Spark2 (at least the tests indicate that...)
> >
> > It'd be great to have a minimum viable environment with specific versions
> > and I hope to have that in a Docker environment by early next week. :)
> >
> > Currently I'm just basing it off a debian image with Java8, although there
> > are Spark images that could be useful...
> >
> > Damon
> >
> > On 2022/11/20 18:55:35 larry mccay wrote:
> > > Considering there is no download for anything older than 3.2.x on the
> > > referred download page, we likely need some change to the README.md to
> > > reflect a more modern version.
> > > We also need more explicit instructions for installing Spark than just
> > the
> > > download. Whether we detail this or point to Spark docs that are
> > sufficient
> > > is certainly a consideration.
> > >
> > > At the end of the day, we are missing any sort of quick start guide for
> > > devs to be able to successfully build and/or run tests.
> > >
> > > Thoughts?
> > >
> > > On Sat, Nov 19, 2022 at 6:23 PM larry mccay <la...@gmail.com>
> > wrote:
> > >
> > > > Hey Folks -
> > > >
> > > > Our Livy README.md indicates the following:
> > > >
> > > > To run Livy, you will also need a Spark installation. You can get Spark
> > > > releases at https://spark.apache.org/downloads.html.
> > > >
> > > > Livy requires Spark 2.4+. You can switch to a different version of
> > Spark
> > > > by setting the SPARK_HOME environment variable in the Livy server
> > > > process, without needing to rebuild Livy.
> > > >
> > > > Do we have any variation on this setup at this point in the real world?
> > > >
> > > > What do your dev environments actually look like and how are you
> > > > installing what versions of Spark as a dependency?
> > > >
> > > > Thanks!
> > > >
> > > > --larry
> > > >
> > >
> >
Re: [DISCUSS] Livy Dev and Test Environment
Posted by larry mccay <lm...@apache.org>.
Yes, the docker based approach is a great way.
@Damon - this sounds terrific - look forward to it next week!
On Fri, Dec 2, 2022 at 8:20 PM Damon Cortesi <da...@apache.org> wrote:
> Coming back to this as I get my dev environment up and running, there's
> definitely an intermix of dependencies between Spark, Python, and R that
> I'm still working out.
>
> For example, when I try to start sparkR I get an error message that
> "package ‘SparkR’ was built under R version 4.0.4", but locally I have R
> version 3.5.2 installed. Spark 3.3.1 says you need R 3.5+. That said, think
> my version of R works with Spark2 (at least the tests indicate that...)
>
> It'd be great to have a minimum viable environment with specific versions
> and I hope to have that in a Docker environment by early next week. :)
>
> Currently I'm just basing it off a debian image with Java8, although there
> are Spark images that could be useful...
>
> Damon
>
> On 2022/11/20 18:55:35 larry mccay wrote:
> > Considering there is no download for anything older than 3.2.x on the
> > referred download page, we likely need some change to the README.md to
> > reflect a more modern version.
> > We also need more explicit instructions for installing Spark than just
> the
> > download. Whether we detail this or point to Spark docs that are
> sufficient
> > is certainly a consideration.
> >
> > At the end of the day, we are missing any sort of quick start guide for
> > devs to be able to successfully build and/or run tests.
> >
> > Thoughts?
> >
> > On Sat, Nov 19, 2022 at 6:23 PM larry mccay <la...@gmail.com>
> wrote:
> >
> > > Hey Folks -
> > >
> > > Our Livy README.md indicates the following:
> > >
> > > To run Livy, you will also need a Spark installation. You can get Spark
> > > releases at https://spark.apache.org/downloads.html.
> > >
> > > Livy requires Spark 2.4+. You can switch to a different version of
> Spark
> > > by setting the SPARK_HOME environment variable in the Livy server
> > > process, without needing to rebuild Livy.
> > >
> > > Do we have any variation on this setup at this point in the real world?
> > >
> > > What do your dev environments actually look like and how are you
> > > installing what versions of Spark as a dependency?
> > >
> > > Thanks!
> > >
> > > --larry
> > >
> >
>