Avoiding "pio build" stage

[George Yarish <gy...@griddynamics.com>]

Hi!

In my company we use predictionio with universal-recommender and also
develop some custom engines for our needs.
Recently I've noticed a few problems in our development process:

1. We distribute source code directly on production servers and then build
it with "pio build". But can we integrate more common CI approach? For
example build binaries with Jenkins, store it in Artifactory and then just
pull jars on production servers for training/serving parts.

2. Some of our engines share common code and that would be great to migrate
on scala multi-modules project structure instead of copying the same
snippet every time, e.g.:

common/
engine1/
engine2/
build.sbt

But as far as I understand, "pio build" phase required build.sbt file to be
located exactly under engine directory.

Just want to understand our options here. Can we organise things that way?
I think we can if somehow migrate from "pio build" to "sbt assembly". But
it is not clear to me what "pio build" is doing apart of building jar.

Thank you,
George

Re: Avoiding "pio build" stage

[Donald Szeto <do...@apache.org>]

Hi George,

Thank you for your questions. We are in fact planning a rearchitecture
which would help with modern CI/CD and avoiding “pio build”. We will
include your concerns.

Regards,
Donald

On Mon, Dec 3, 2018 at 5:25 AM Naoki Takezoe <ta...@gmail.com> wrote:

> Hi George,
>
> On Sun, Dec 2, 2018 at 9:48 PM George Yarish <gy...@griddynamics.com>
> wrote:
> > 1. We distribute source code directly on production servers and then
> build it with "pio build". But can we integrate more common CI approach?
> For example build binaries with Jenkins, store it in Artifactory and then
> just pull jars on production servers for training/serving parts.
>
> PredictionIO requires template.json and engine.json in addition to the
> assembly jar file for training/serving. So it may work if you locate these
> files and an assembly jar file as same structure as template built by "pio
> build".
>
> > 2. Some of our engines share common code and that would be great to
> migrate on scala multi-modules project structure instead of copying the
> same snippet every time, e.g.:
> >
> > common/
> > engine1/
> > engine2/
> > build.sbt
> >
> > But as far as I understand, "pio build" phase required build.sbt file to
> be located exactly under engine directory.
> > Just want to understand our options here. Can we organise things that
> way?
>
> As you said, multi-modules project isn't supported. You have to create an
> another project for sharing your code and publish it as a library, or git
> submodule might be a good solution if you use git to manage your templates.
>
> > I think we can if somehow migrate from "pio build" to "sbt assembly".
> But it is not clear to me what "pio build" is doing apart of building jar.
>
> In fact, "pio build" is almost same as "sbt assembly" in default. The only
> difference is that it verifies template by looking template.json or
> engine.json before calling sbt.
>
> --
> Naoki Takezoe
>

Re: Avoiding "pio build" stage

[Naoki Takezoe <ta...@gmail.com>]

Hi George,

On Sun, Dec 2, 2018 at 9:48 PM George Yarish <gy...@griddynamics.com>
wrote:
> 1. We distribute source code directly on production servers and then
build it with "pio build". But can we integrate more common CI approach?
For example build binaries with Jenkins, store it in Artifactory and then
just pull jars on production servers for training/serving parts.

PredictionIO requires template.json and engine.json in addition to the
assembly jar file for training/serving. So it may work if you locate these
files and an assembly jar file as same structure as template built by "pio
build".

> 2. Some of our engines share common code and that would be great to
migrate on scala multi-modules project structure instead of copying the
same snippet every time, e.g.:
>
> common/
> engine1/
> engine2/
> build.sbt
>
> But as far as I understand, "pio build" phase required build.sbt file to
be located exactly under engine directory.
> Just want to understand our options here. Can we organise things that way?

As you said, multi-modules project isn't supported. You have to create an
another project for sharing your code and publish it as a library, or git
submodule might be a good solution if you use git to manage your templates.

> I think we can if somehow migrate from "pio build" to "sbt assembly". But
it is not clear to me what "pio build" is doing apart of building jar.

In fact, "pio build" is almost same as "sbt assembly" in default. The only
difference is that it verifies template by looking template.json or
engine.json before calling sbt.

--
Naoki Takezoe