You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@toree.apache.org by Jakob Odersky <ja...@odersky.com> on 2017/01/31 01:47:46 UTC
Development model
Hi everyone,
I was wondering how everyone usually develops toree, specifically how
changes to toree are tested with a jupyter notebook? I couldn't find
any documentation on the website so I thought I'd ask here.
I tried running the various makefile targets, including `make dev` and
`make jupyter`, however those targets use a docker image that already
contains a published upstream version of toree, which thereby
overrides any changes made to the local version. Running "make
release", then pip-installing the archive and finally running "jupyter
install toree" somehow also uses an upstream version of toree.
Is there an easy way to build and install a local source tree?
thanks,
--Jakob
Re: Development model
Posted by Corey Stubbs <ca...@gmail.com>.
Sounds good.
Kind Regards,
Corey Stubbs
On Thu, Feb 2, 2017 at 9:50 AM Luciano Resende <lu...@gmail.com> wrote:
> I am not suggesting doing "everything" via SBT, just the basic compile,
> test, build and package so that contributors used to other sbt based
> projects fill comfortable getting started with the project. For all the
> other more optional/complex tasks, I am all for using a set of fully
> documented tasks.
>
> On Thu, Feb 2, 2017 at 7:23 AM, Corey Stubbs <ca...@gmail.com> wrote:
>
> > My typical development flow is to write code, run pip-release , and then
> > install the pip release locally on my machine (pip install
> > dist/toree-pip/pip-release; jupyter toree install), and test changes with
> > the install.
> >
> > +1 on working on documenting the make targets.
> >
> > In terms of getting everything done via SBT, I don't know if that will
> > ultimately be possible. Targets like, compile, test, and build can (and a
> > majority should be) done with SBT. However, since we are doing some
> python
> > things to ease the pain of installing toree as a kernel into the Jupyter
> > environment (via pip install).
> >
> > Regardless, I agree the Makefile has become out of hand and can use some
> > cleanup and loving.
> >
> > I can start the work on documenting the targets that are there and clean
> up
> > old ones. Once I get a PR open I will post it here and in the related
> > issue: https://issues.apache.org/jira/browse/TOREE-367
> >
> > On Tue, Jan 31, 2017 at 1:47 PM Luciano Resende <lu...@gmail.com>
> > wrote:
> >
> > > On Tue, Jan 31, 2017 at 9:56 AM, Marius van Niekerk <
> > > marius.v.niekerk@gmail.com> wrote:
> > >
> > > > I think just documenting development workflow and process properly
> will
> > > > help a great deal.
> > > > Building Toree is a non-trivial exercise sadly.
> > > >
> > > >
> > > Completely agree that is a non-trivial exercise, but should not be.
> > >
> > >
> > > --
> > > Luciano Resende
> > > http://twitter.com/lresende1975
> > > http://lresende.blogspot.com/
> > >
> >
>
>
>
> --
> Luciano Resende
> http://twitter.com/lresende1975
> http://lresende.blogspot.com/
>
Re: Development model
Posted by Luciano Resende <lu...@gmail.com>.
I am not suggesting doing "everything" via SBT, just the basic compile,
test, build and package so that contributors used to other sbt based
projects fill comfortable getting started with the project. For all the
other more optional/complex tasks, I am all for using a set of fully
documented tasks.
On Thu, Feb 2, 2017 at 7:23 AM, Corey Stubbs <ca...@gmail.com> wrote:
> My typical development flow is to write code, run pip-release , and then
> install the pip release locally on my machine (pip install
> dist/toree-pip/pip-release; jupyter toree install), and test changes with
> the install.
>
> +1 on working on documenting the make targets.
>
> In terms of getting everything done via SBT, I don't know if that will
> ultimately be possible. Targets like, compile, test, and build can (and a
> majority should be) done with SBT. However, since we are doing some python
> things to ease the pain of installing toree as a kernel into the Jupyter
> environment (via pip install).
>
> Regardless, I agree the Makefile has become out of hand and can use some
> cleanup and loving.
>
> I can start the work on documenting the targets that are there and clean up
> old ones. Once I get a PR open I will post it here and in the related
> issue: https://issues.apache.org/jira/browse/TOREE-367
>
> On Tue, Jan 31, 2017 at 1:47 PM Luciano Resende <lu...@gmail.com>
> wrote:
>
> > On Tue, Jan 31, 2017 at 9:56 AM, Marius van Niekerk <
> > marius.v.niekerk@gmail.com> wrote:
> >
> > > I think just documenting development workflow and process properly will
> > > help a great deal.
> > > Building Toree is a non-trivial exercise sadly.
> > >
> > >
> > Completely agree that is a non-trivial exercise, but should not be.
> >
> >
> > --
> > Luciano Resende
> > http://twitter.com/lresende1975
> > http://lresende.blogspot.com/
> >
>
--
Luciano Resende
http://twitter.com/lresende1975
http://lresende.blogspot.com/
Re: Development model
Posted by Corey Stubbs <ca...@gmail.com>.
My typical development flow is to write code, run pip-release , and then
install the pip release locally on my machine (pip install
dist/toree-pip/pip-release; jupyter toree install), and test changes with
the install.
+1 on working on documenting the make targets.
In terms of getting everything done via SBT, I don't know if that will
ultimately be possible. Targets like, compile, test, and build can (and a
majority should be) done with SBT. However, since we are doing some python
things to ease the pain of installing toree as a kernel into the Jupyter
environment (via pip install).
Regardless, I agree the Makefile has become out of hand and can use some
cleanup and loving.
I can start the work on documenting the targets that are there and clean up
old ones. Once I get a PR open I will post it here and in the related
issue: https://issues.apache.org/jira/browse/TOREE-367
On Tue, Jan 31, 2017 at 1:47 PM Luciano Resende <lu...@gmail.com>
wrote:
> On Tue, Jan 31, 2017 at 9:56 AM, Marius van Niekerk <
> marius.v.niekerk@gmail.com> wrote:
>
> > I think just documenting development workflow and process properly will
> > help a great deal.
> > Building Toree is a non-trivial exercise sadly.
> >
> >
> Completely agree that is a non-trivial exercise, but should not be.
>
>
> --
> Luciano Resende
> http://twitter.com/lresende1975
> http://lresende.blogspot.com/
>
Re: Development model
Posted by Luciano Resende <lu...@gmail.com>.
On Tue, Jan 31, 2017 at 9:56 AM, Marius van Niekerk <
marius.v.niekerk@gmail.com> wrote:
> I think just documenting development workflow and process properly will
> help a great deal.
> Building Toree is a non-trivial exercise sadly.
>
>
Completely agree that is a non-trivial exercise, but should not be.
--
Luciano Resende
http://twitter.com/lresende1975
http://lresende.blogspot.com/
Re: Development model
Posted by Marius van Niekerk <ma...@gmail.com>.
I think just documenting development workflow and process properly will
help a great deal.
Building Toree is a non-trivial exercise sadly.
On Tue, 31 Jan 2017 at 11:14 Luciano Resende <lu...@gmail.com> wrote:
> On Mon, Jan 30, 2017 at 5:47 PM, Jakob Odersky <ja...@odersky.com> wrote:
>
> > Hi everyone,
> > I was wondering how everyone usually develops toree, specifically how
> > changes to toree are tested with a jupyter notebook? I couldn't find
> > any documentation on the website so I thought I'd ask here.
> >
> > I tried running the various makefile targets, including `make dev` and
> > `make jupyter`, however those targets use a docker image that already
> > contains a published upstream version of toree, which thereby
> > overrides any changes made to the local version. Running "make
> > release", then pip-installing the archive and finally running "jupyter
> > install toree" somehow also uses an upstream version of toree.
> >
> > Is there an easy way to build and install a local source tree?
> >
> > thanks,
> > --Jakob
> >
>
>
> It seems that the project has various, and mostly undocumented, utilities
> done via make. To me this is good, but don't usually follow the patterns of
> projects using a specific build system (e.g. maven, sbt, etc).
>
> IMHO I believe we should be able to conform with the standards of a basic
> sbt project, and enable compile, unit test, package jars and distribution
> and publish locally using vanilla sbt commands. And then use and document
> the make facilities for more complex tasks such as tests that require
> docker, environment setup, build a jupyter env, etc Which will then make
> it easy for newcomers that are used to sbt to get started while they learn
> about the make magics.
>
> Thoughts ?
>
>
> --
> Luciano Resende
> http://twitter.com/lresende1975
> http://lresende.blogspot.com/
>
--
regards
Marius van Niekerk
Re: Development model
Posted by Luciano Resende <lu...@gmail.com>.
On Mon, Jan 30, 2017 at 5:47 PM, Jakob Odersky <ja...@odersky.com> wrote:
> Hi everyone,
> I was wondering how everyone usually develops toree, specifically how
> changes to toree are tested with a jupyter notebook? I couldn't find
> any documentation on the website so I thought I'd ask here.
>
> I tried running the various makefile targets, including `make dev` and
> `make jupyter`, however those targets use a docker image that already
> contains a published upstream version of toree, which thereby
> overrides any changes made to the local version. Running "make
> release", then pip-installing the archive and finally running "jupyter
> install toree" somehow also uses an upstream version of toree.
>
> Is there an easy way to build and install a local source tree?
>
> thanks,
> --Jakob
>
It seems that the project has various, and mostly undocumented, utilities
done via make. To me this is good, but don't usually follow the patterns of
projects using a specific build system (e.g. maven, sbt, etc).
IMHO I believe we should be able to conform with the standards of a basic
sbt project, and enable compile, unit test, package jars and distribution
and publish locally using vanilla sbt commands. And then use and document
the make facilities for more complex tasks such as tests that require
docker, environment setup, build a jupyter env, etc Which will then make
it easy for newcomers that are used to sbt to get started while they learn
about the make magics.
Thoughts ?
--
Luciano Resende
http://twitter.com/lresende1975
http://lresende.blogspot.com/
Re: Development model
Posted by Marius van Niekerk <ma...@gmail.com>.
Typically my flow has been the following.
Run the scalatests inside intellij fo what I'm working on.
follow by
make system-test
On Mon, 30 Jan 2017 at 20:48 Jakob Odersky <ja...@odersky.com> wrote:
> Hi everyone,
> I was wondering how everyone usually develops toree, specifically how
> changes to toree are tested with a jupyter notebook? I couldn't find
> any documentation on the website so I thought I'd ask here.
>
> I tried running the various makefile targets, including `make dev` and
> `make jupyter`, however those targets use a docker image that already
> contains a published upstream version of toree, which thereby
> overrides any changes made to the local version. Running "make
> release", then pip-installing the archive and finally running "jupyter
> install toree" somehow also uses an upstream version of toree.
>
> Is there an easy way to build and install a local source tree?
>
> thanks,
> --Jakob
>
--
regards
Marius van Niekerk