You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@marvin.apache.org by Lucas Bonatto Miguel <lu...@apache.org> on 2018/10/12 01:21:12 UTC

Future support to R

Hi community, I already participated in several discussions with some of
you regarding adding support to R on Marvin. Allowing users to write
engines in R would be a great step for Marvin since that's the primary
language for many statisticians, physicists and math people.

One of the points that always arise during those conversations is the fact
that R is not fast nor scalable, and it would be hard to either make it
process large chunks of data for training or make it support high
throughput and have low latency for predictions. I want to start a thread
on whether we should invest more time trying to add R support to Marvin or
not. I invite you all to read about this related work:
https://medium.com/graalvm/faster-r-with-fastr-4b8db0e0dceb

I am looking forward to hearing opinions and ideas.

Thanks,
Lucas

Re: POCs, Prototypes, experiments ...

Posted by Lucas Bonatto Miguel <lu...@apache.org>.

Mentors help required:

We need the project created on Apache's Jira, the current link listed in
our incubator page points to a 404 Jira page. Could you help us to setup?

https://issues.apache.org/jira/projects/MARVIN

Thx,
Lucas

On Wed, Oct 31, 2018 at 11:37 AM Lucas Bonatto Miguel <lu...@apache.org>
wrote:

> Yes, since we haven't heard any concerns, I believe we're good to start
> coding.
>
> The next step is to create issues on Jira; I have all the issues ready to
> be written on Jira, however, I can't find Marvin on Apache's Jira.
>
> Could our mentors help to create our project on Jira?
>
> Thanks
>
> On Wed, Oct 31, 2018 at 10:42 AM Rafael Novello <
> rafa.reis.novello@gmail.com> wrote:
>
>> Guys,
>>
>> How can we organize the tasks to start working at this new version?
>>
>> I believe we had the core devs approval.
>>
>> Atenciosamente,
>> Rafael J. R. Novello
>>
>> Skype: rafael.novello
>> Blog: http://rafanovello.blogspot.com.br/
>>
>>
>> Em ter, 30 de out de 2018 às 14:30, Zhang Yifei <yi...@gmail.com>
>> escreveu:
>>
>> > +1
>> >
>> > Em seg, 29 de out de 2018 às 18:23, Rafael Novello <
>> > rafa.reis.novello@gmail.com> escreveu:
>> >
>> > > Lucas, yes! It make sense!
>> > >
>> > > In the end, CLI and REPL will share a big part of code and they will
>> be
>> > > just different interfaces to the same features.
>> > >
>> > > +1 for this new concept!
>> > >
>> > > Thanks for make it clear to me!
>> > > Atenciosamente,
>> > > Rafael J. R. Novello
>> > >
>> > > Skype: rafael.novello
>> > > Blog: http://rafanovello.blogspot.com.br/
>> > >
>> > >
>> > > Em seg, 29 de out de 2018 às 17:43, Lucas Bonatto Miguel <
>> > > lucasbm88@apache.org> escreveu:
>> > >
>> > > > That's a good point Rafael. I believe the main motivation for
>> adding a
>> > > REPL
>> > > > to Marvin's toolbox is to enhance user experience.
>> > > >
>> > > > Imo we can't just switch CLI per REPL. I don't think one excludes
>> the
>> > > > other, both have different use cases.
>> > > >
>> > > > We may be mixing too much stuff together, but in general, I think we
>> > are:
>> > > >  - Refactoring toolbox architecture to avoid unnecessarily
>> duplicated
>> > > logic
>> > > > when adding support to new languages.
>> > > >  - Adding a new interface to enable users to open an interactive
>> > session
>> > > > with the engine (i.e. REPL)
>> > > >  - Making it easier to add support for new languages.
>> > > >
>> > > > Does it make sense for you?
>> > > >
>> > > > On Mon, Oct 29, 2018 at 11:22 AM Rafael Novello <
>> > > > rafa.reis.novello@gmail.com>
>> > > > wrote:
>> > > >
>> > > > > I do not want to be the annoying conversation but...
>> > > > >
>> > > > > If we will have an unique CLI that will be able to handle any
>> kind of
>> > > > > engines (in any language) by using docker, Why we need a REPL?
>> > > > >
>> > > > > I think we can focus on creating this new architecture using
>> docker
>> > and
>> > > > > docker SDK and use only a CLI for that.
>> > > > >
>> > > > > Atenciosamente,
>> > > > > Rafael J. R. Novello
>> > > > >
>> > > > > Skype: rafael.novello
>> > > > > Blog: http://rafanovello.blogspot.com.br/
>> > > > >
>> > > > >
>> > > > > Em seg, 29 de out de 2018 às 14:58, Lucas Bonatto Miguel <
>> > > > > lucasbm88@apache.org> escreveu:
>> > > > >
>> > > > > > No, I think the idea is to do not create different versions of
>> the
>> > > CLI.
>> > > > > > Let's use one single tech stack for the CLI, ideally, it will be
>> > > shared
>> > > > > > with the REPL. The way I see the CLI would be very similar to
>> the
>> > > REPL,
>> > > > > but
>> > > > > > instead of opening an interactive session the user could send
>> > > > > > unattended commands. As an analogy, you can think on spark-shell
>> > (our
>> > > > > REPL)
>> > > > > > and spark-submit (our CLI).
>> > > > > >
>> > > > > > - Lucas
>> > > > > >
>> > > > > > On Mon, Oct 29, 2018 at 10:27 AM Rafael Novello <
>> > > > > > rafa.reis.novello@gmail.com>
>> > > > > > wrote:
>> > > > > >
>> > > > > > > Hi Guys!
>> > > > > > >
>> > > > > > > Let me ask about this new version that include CLI's and REPL.
>> > Will
>> > > > we
>> > > > > > > create different versions of CLI?
>> > > > > > >
>> > > > > > > One of the reasons to create the REPL is to avoid
>> re-implementing
>> > > the
>> > > > > > same
>> > > > > > > features many times,
>> > > > > > > each time for a different language.
>> > > > > > >
>> > > > > > > Regards!
>> > > > > > > Atenciosamente,
>> > > > > > > Rafael J. R. Novello
>> > > > > > >
>> > > > > > > Skype: rafael.novello
>> > > > > > > Blog: http://rafanovello.blogspot.com.br/
>> > > > > > >
>> > > > > > >
>> > > > > > > Em sex, 26 de out de 2018 às 20:54, Daniel Takabayashi <
>> > > > > > > daniel.takabayashi@gmail.com> escreveu:
>> > > > > > >
>> > > > > > > > +1
>> > > > > > > >
>> > > > > > > > Em sex, 26 de out de 2018 às 16:47, Lucas Bonatto Miguel <
>> > > > > > > > lucasbm88@apache.org> escreveu:
>> > > > > > > >
>> > > > > > > > > Makes perfect sense to me. I believe that if the majority
>> of
>> > > > people
>> > > > > > > > agrees
>> > > > > > > > > with this architecture we could create issues and start
>> > > > > implementing
>> > > > > > > it.
>> > > > > > > > >
>> > > > > > > > > +1
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > > On Fri, Oct 26, 2018 at 4:09 PM Daniel Takabayashi <
>> > > > > > > > > daniel.takabayashi@gmail.com> wrote:
>> > > > > > > > >
>> > > > > > > > > > I see... what about this new version?
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://docs.google.com/drawings/d/shajxIpLJHxxMbFgDXiPuhg/image?w=602&h=468&rev=1630&ac=1&parent=1ySERHGBXbHeyCMRookq5UfTuFkzzU0ugtjvR3rF3deY
>> > > > > > > > > >
>> > > > > > > > > > I changed the toolbox ...from 1 component to 3... docker
>> > > > client,
>> > > > > > cli
>> > > > > > > > and
>> > > > > > > > > > repl. Both (cli and repl) now are using the docker
>> client
>> > to
>> > > > > > control
>> > > > > > > > the
>> > > > > > > > > > engines.
>> > > > > > > > > >
>> > > > > > > > > > Em sex, 26 de out de 2018 às 14:49, Lucas Bonatto
>> Miguel <
>> > > > > > > > > > lucasbm88@apache.org> escreveu:
>> > > > > > > > > >
>> > > > > > > > > > > For sure, both suggestions are to make sure we're not
>> > > killing
>> > > > > the
>> > > > > > > CLI
>> > > > > > > > > and
>> > > > > > > > > > > that we have a component (the SDK client) that is
>> shared
>> > by
>> > > > CLI
>> > > > > > and
>> > > > > > > > > REPL.
>> > > > > > > > > > >
>> > > > > > > > > > > Thanks for clarifying where the system calls will
>> live.
>> > > > > > > > > > >
>> > > > > > > > > > > - Lucas
>> > > > > > > > > > >
>> > > > > > > > > > > On Fri, Oct 26, 2018 at 2:39 PM Daniel Takabayashi <
>> > > > > > > > > > > daniel.takabayashi@gmail.com> wrote:
>> > > > > > > > > > >
>> > > > > > > > > > > > Hi Lucas, could you please justified your two
>> > > suggestions?
>> > > > > > > > > > > >
>> > > > > > > > > > > > The general ideia is to use the Docker DSK to
>> execute
>> > > local
>> > > > > > > > commands,
>> > > > > > > > > > to
>> > > > > > > > > > > > consume outputs and also to execute Marvin "specific
>> > > > > language"
>> > > > > > > > > > commands.
>> > > > > > > > > > > > Today almost all CLIs commands in Marvin is a mixed
>> of
>> > > > > systems
>> > > > > > > > > commands
>> > > > > > > > > > > and
>> > > > > > > > > > > > library method calls. The plan is to put all
>> "specific
>> > > > > language
>> > > > > > > > > calls"
>> > > > > > > > > > > > inside the marvin-common-libs and group the systems
>> > call
>> > > on
>> > > > > the
>> > > > > > > > > toolbox
>> > > > > > > > > > > > (using the docker SDK in this task).
>> > > > > > > > > > > >
>> > > > > > > > > > > >
>> > > > > > > > > > > >
>> > > > > > > > > > > >
>> > > > > > > > > > > > Em qui, 25 de out de 2018 às 20:09, Lucas Bonatto
>> > Miguel
>> > > <
>> > > > > > > > > > > > lucasbm88@apache.org> escreveu:
>> > > > > > > > > > > >
>> > > > > > > > > > > > > Great, thanks.
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > So here some suggestions:
>> > > > > > > > > > > > >  1. We could create one new layer between Toolbox
>> and
>> > > the
>> > > > > > > > Engines,
>> > > > > > > > > > this
>> > > > > > > > > > > > > would be Docker client.
>> > > > > > > > > > > > >  2. Split the Toolbox layer in two. Toolbox CLI
>> and
>> > > > Toolbox
>> > > > > > > REPL
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > It's not clear to me if the commands listed on
>> > General
>> > > > > > Commands
>> > > > > > > > > will
>> > > > > > > > > > > need
>> > > > > > > > > > > > > to be written for every language, from the image I
>> > > > believe
>> > > > > > not,
>> > > > > > > > but
>> > > > > > > > > > how
>> > > > > > > > > > > > it
>> > > > > > > > > > > > > would work, since starting a notebook in Python is
>> > > > > different
>> > > > > > > than
>> > > > > > > > > > > > starting
>> > > > > > > > > > > > > in Scala.
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > - Lucas
>> > > > > > > > > > > > > On Thu, Oct 25, 2018 at 4:51 PM Daniel
>> Takabayashi <
>> > > > > > > > > > > > > daniel.takabayashi@gmail.com> wrote:
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > >
>> > > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://docs.google.com/drawings/d/shajxIpLJHxxMbFgDXiPuhg/image?w=602&h=461&rev=1423&ac=1&parent=1ySERHGBXbHeyCMRookq5UfTuFkzzU0ugtjvR3rF3deY
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > Em qui, 25 de out de 2018 às 07:38, Rafael
>> Novello
>> > <
>> > > > > > > > > > > > > > rafa.reis.novello@gmail.com> escreveu:
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > Hi Taka,
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > I can't see the image! Maybe the Apache email
>> > list
>> > > > > don't
>> > > > > > > > > support
>> > > > > > > > > > > > > > > attachments.
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > Best!
>> > > > > > > > > > > > > > > Rafael J. R. Novello
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > Skype: rafael.novello
>> > > > > > > > > > > > > > > Blog: http://rafanovello.blogspot.com.br/
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > Em qua, 24 de out de 2018 às 21:14, Daniel
>> > > > Takabayashi
>> > > > > <
>> > > > > > > > > > > > > > > daniel.takabayashi@gmail.com> escreveu:
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > Hi all,
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > To try to give a little more details about
>> this
>> > > > "new
>> > > > > > > > concept"
>> > > > > > > > > > > that
>> > > > > > > > > > > > we
>> > > > > > > > > > > > > > > want
>> > > > > > > > > > > > > > > > to bring to the marvin toolbox, I did this
>> > simple
>> > > > > > > > > architecture
>> > > > > > > > > > > > draw.
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > [image: marvin-architecture-views (5).png]
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > The general idea here is try to transform
>> the
>> > > > toolbox
>> > > > > > > > > something
>> > > > > > > > > > > > > > > > disconnected with the "language", something
>> > more
>> > > > > > > agnostic.
>> > > > > > > > > Also
>> > > > > > > > > > > in
>> > > > > > > > > > > > > this
>> > > > > > > > > > > > > > > new
>> > > > > > > > > > > > > > > > architecture we could use remote resource to
>> > > > process
>> > > > > > > > engines
>> > > > > > > > > > and
>> > > > > > > > > > > > make
>> > > > > > > > > > > > > > > easy
>> > > > > > > > > > > > > > > > the support for new languages.
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > This "new toolbox" will be the only thing
>> that
>> > a
>> > > > > Marvin
>> > > > > > > > user
>> > > > > > > > > > must
>> > > > > > > > > > > > to
>> > > > > > > > > > > > > > > > install and also we could start to support
>> > > > multiples
>> > > > > > O.S
>> > > > > > > > > (once
>> > > > > > > > > > > the
>> > > > > > > > > > > > > REPL
>> > > > > > > > > > > > > > > is
>> > > > > > > > > > > > > > > > a dummy application that only interprets
>> and by
>> > > > pass
>> > > > > > > > > commands).
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > Regards,
>> > > > > > > > > > > > > > > > Taka
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > > Em qua, 24 de out de 2018 às 09:52, Rafael
>> > > Novello
>> > > > <
>> > > > > > > > > > > > > > > > rafa.reis.novello@gmail.com> escreveu:
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > >> Alan,
>> > > > > > > > > > > > > > > >>
>> > > > > > > > > > > > > > > >> Yes! We are using the Docker SDK and it's
>> > > possible
>> > > > > to
>> > > > > > > use
>> > > > > > > > > the
>> > > > > > > > > > > API
>> > > > > > > > > > > > to
>> > > > > > > > > > > > > > > >> automate, but it's a little bit harder than
>> > > > automate
>> > > > > > CLI
>> > > > > > > > > > calls.
>> > > > > > > > > > > > > > > >>
>> > > > > > > > > > > > > > > >> Atenciosamente,
>> > > > > > > > > > > > > > > >> Rafael J. R. Novello
>> > > > > > > > > > > > > > > >>
>> > > > > > > > > > > > > > > >> Skype: rafael.novello
>> > > > > > > > > > > > > > > >> Blog: http://rafanovello.blogspot.com.br/
>> > > > > > > > > > > > > > > >>
>> > > > > > > > > > > > > > > >>
>> > > > > > > > > > > > > > > >> Em qua, 24 de out de 2018 às 12:02, Alan
>> > Silva <
>> > > > > > > > > > > jumpi@apache.org>
>> > > > > > > > > > > > > > > >> escreveu:
>> > > > > > > > > > > > > > > >>
>> > > > > > > > > > > > > > > >> > Hi,
>> > > > > > > > > > > > > > > >> >
>> > > > > > > > > > > > > > > >> > One question here, I understand that we
>> > start
>> > > to
>> > > > > use
>> > > > > > > > with
>> > > > > > > > > > this
>> > > > > > > > > > > > PoC
>> > > > > > > > > > > > > > the
>> > > > > > > > > > > > > > > >> > Docker SDK API, right?
>> > > > > > > > > > > > > > > >> >
>> > > > > > > > > > > > > > > >> > Why not use the API to expose some
>> endpoints
>> > > to
>> > > > > > permit
>> > > > > > > > > this
>> > > > > > > > > > > kind
>> > > > > > > > > > > > > of
>> > > > > > > > > > > > > > > >> > automation by devops?
>> > > > > > > > > > > > > > > >> >
>> > > > > > > > > > > > > > > >> > I think it is possible and it solves the
>> CLI
>> > > > > > problem,
>> > > > > > > > > right?
>> > > > > > > > > > > > > > > >> >
>> > > > > > > > > > > > > > > >> >
>> > > > > > > > > > > > > > > >> > On Tue, Oct 23, 2018 at 1:05 PM Rafael
>> > > Novello <
>> > > > > > > > > > > > > > > >> > rafa.reis.novello@gmail.com>
>> > > > > > > > > > > > > > > >> > wrote:
>> > > > > > > > > > > > > > > >> >
>> > > > > > > > > > > > > > > >> > > Lucas,
>> > > > > > > > > > > > > > > >> > >
>> > > > > > > > > > > > > > > >> > > Sorry, I didn't understood your
>> question
>> > > > bellow.
>> > > > > > > > > > > > > > > >> > >
>> > > > > > > > > > > > > > > >> > > "Would it make sense to use the same
>> > > solution
>> > > > > that
>> > > > > > > we
>> > > > > > > > > will
>> > > > > > > > > > > use
>> > > > > > > > > > > > > for
>> > > > > > > > > > > > > > > >> > having a
>> > > > > > > > > > > > > > > >> > > single-language REPL to have a
>> > > single-language
>> > > > > > CLI?"
>> > > > > > > > > > > > > > > >> > >
>> > > > > > > > > > > > > > > >> > > For DevOps purposes, maybe this new
>> > toolbox
>> > > > > > concept
>> > > > > > > is
>> > > > > > > > > not
>> > > > > > > > > > > > > ideal.
>> > > > > > > > > > > > > > I
>> > > > > > > > > > > > > > > >> think
>> > > > > > > > > > > > > > > >> > > we can keep the CLI inside the docker
>> > > > container
>> > > > > > but
>> > > > > > > it
>> > > > > > > > > > will
>> > > > > > > > > > > > not
>> > > > > > > > > > > > > > easy
>> > > > > > > > > > > > > > > >> to
>> > > > > > > > > > > > > > > >> > > automate
>> > > > > > > > > > > > > > > >> > > jobs by this way.
>> > > > > > > > > > > > > > > >> > >
>> > > > > > > > > > > > > > > >> > > How to deal with this issue? Voting?
>> > > > > > > > > > > > > > > >> > > Atenciosamente,
>> > > > > > > > > > > > > > > >> > > Rafael J. R. Novello
>> > > > > > > > > > > > > > > >> > >
>> > > > > > > > > > > > > > > >> > > Skype: rafael.novello
>> > > > > > > > > > > > > > > >> > > Blog:
>> http://rafanovello.blogspot.com.br/
>> > > > > > > > > > > > > > > >> > >
>> > > > > > > > > > > > > > > >> > >
>> > > > > > > > > > > > > > > >> > > Em sex, 19 de out de 2018 às 19:00,
>> Lucas
>> > > > > Bonatto
>> > > > > > > > > Miguel <
>> > > > > > > > > > > > > > > >> > > lucasbm88@apache.org> escreveu:
>> > > > > > > > > > > > > > > >> > >
>> > > > > > > > > > > > > > > >> > > > Got it! Thanks for clarifying.
>> > > > > > > > > > > > > > > >> > > >
>> > > > > > > > > > > > > > > >> > > > Would it make sense to use the same
>> > > solution
>> > > > > > that
>> > > > > > > we
>> > > > > > > > > > will
>> > > > > > > > > > > > use
>> > > > > > > > > > > > > > for
>> > > > > > > > > > > > > > > >> > having
>> > > > > > > > > > > > > > > >> > > a
>> > > > > > > > > > > > > > > >> > > > single-language REPL to have a
>> > > > single-language
>> > > > > > > CLI?
>> > > > > > > > My
>> > > > > > > > > > > only
>> > > > > > > > > > > > > > > concern
>> > > > > > > > > > > > > > > >> > with
>> > > > > > > > > > > > > > > >> > > > killing the CLI is that you remove an
>> > > > > essential
>> > > > > > > > > feature
>> > > > > > > > > > > for
>> > > > > > > > > > > > > > > DevOps.
>> > > > > > > > > > > > > > > >> > > >
>> > > > > > > > > > > > > > > >> > > > - Lucas
>> > > > > > > > > > > > > > > >> > > >
>> > > > > > > > > > > > > > > >> > > > On Fri, Oct 19, 2018 at 1:52 PM
>> Rafael
>> > > > > Novello <
>> > > > > > > > > > > > > > > >> > > > rafa.reis.novello@gmail.com>
>> > > > > > > > > > > > > > > >> > > > wrote:
>> > > > > > > > > > > > > > > >> > > >
>> > > > > > > > > > > > > > > >> > > > > Lucas,
>> > > > > > > > > > > > > > > >> > > > >
>> > > > > > > > > > > > > > > >> > > > > The idea is that REPL will
>> substitute
>> > > the
>> > > > > > actual
>> > > > > > > > > CLI.
>> > > > > > > > > > > It's
>> > > > > > > > > > > > > > > because
>> > > > > > > > > > > > > > > >> > with
>> > > > > > > > > > > > > > > >> > > > the
>> > > > > > > > > > > > > > > >> > > > > actual concept (using language
>> > specific
>> > > > CLI)
>> > > > > > we
>> > > > > > > > will
>> > > > > > > > > > > need
>> > > > > > > > > > > > to
>> > > > > > > > > > > > > > > >> > > re-implement
>> > > > > > > > > > > > > > > >> > > > > the same features many times and
>> > > probably
>> > > > > each
>> > > > > > > CLI
>> > > > > > > > > > will
>> > > > > > > > > > > > > have a
>> > > > > > > > > > > > > > > >> > > different
>> > > > > > > > > > > > > > > >> > > > > behavior because some language
>> > specific
>> > > > > > > > restrictions
>> > > > > > > > > > > > and/or
>> > > > > > > > > > > > > > > >> > > limitations.
>> > > > > > > > > > > > > > > >> > > > >
>> > > > > > > > > > > > > > > >> > > > > With this new concept, all users
>> will
>> > > have
>> > > > > the
>> > > > > > > > same
>> > > > > > > > > > > > > experience
>> > > > > > > > > > > > > > > >> > > > interacting
>> > > > > > > > > > > > > > > >> > > > > with Marvin REPL and they will use
>> the
>> > > > bests
>> > > > > > > tools
>> > > > > > > > > to
>> > > > > > > > > > > > > develop
>> > > > > > > > > > > > > > > >> their
>> > > > > > > > > > > > > > > >> > > > engines
>> > > > > > > > > > > > > > > >> > > > > (Jupyter Notebook and/or Lab with
>> > > > different
>> > > > > > > > > languages
>> > > > > > > > > > > > > support
>> > > > > > > > > > > > > > or
>> > > > > > > > > > > > > > > >> even
>> > > > > > > > > > > > > > > >> > > > > Apache Zeppelin). All the interact
>> > will
>> > > > > occur
>> > > > > > > > > through
>> > > > > > > > > > > REPL
>> > > > > > > > > > > > > and
>> > > > > > > > > > > > > > > >> docker
>> > > > > > > > > > > > > > > >> > > > > protocol.
>> > > > > > > > > > > > > > > >> > > > >
>> > > > > > > > > > > > > > > >> > > > > Alan, Yes! As Lucas said, the
>> concept
>> > is
>> > > > the
>> > > > > > > same
>> > > > > > > > > but
>> > > > > > > > > > we
>> > > > > > > > > > > > can
>> > > > > > > > > > > > > > use
>> > > > > > > > > > > > > > > >> > docker
>> > > > > > > > > > > > > > > >> > > > to
>> > > > > > > > > > > > > > > >> > > > > do the same job.
>> > > > > > > > > > > > > > > >> > > > >
>> > > > > > > > > > > > > > > >> > > > > Best!
>> > > > > > > > > > > > > > > >> > > > >
>> > > > > > > > > > > > > > > >> > > > > Em sex, 19 de out de 2018 às 00:39,
>> > > Lucas
>> > > > > > > Bonatto
>> > > > > > > > > > > Miguel <
>> > > > > > > > > > > > > > > >> > > > > lucasbm88@apache.org> escreveu:
>> > > > > > > > > > > > > > > >> > > > >
>> > > > > > > > > > > > > > > >> > > > > > Thanks for the clarifications
>> > Rafael,
>> > > so
>> > > > > > from
>> > > > > > > > > what I
>> > > > > > > > > > > > > > > understood,
>> > > > > > > > > > > > > > > >> > > > Marvin's
>> > > > > > > > > > > > > > > >> > > > > > developers would not use the
>> REPL to
>> > > > > explore
>> > > > > > > > data
>> > > > > > > > > > and
>> > > > > > > > > > > > test
>> > > > > > > > > > > > > > > >> models,
>> > > > > > > > > > > > > > > >> > > i.e.
>> > > > > > > > > > > > > > > >> > > > > > develop the engine. Is the idea
>> to
>> > > build
>> > > > > > > > something
>> > > > > > > > > > > more
>> > > > > > > > > > > > > like
>> > > > > > > > > > > > > > > an
>> > > > > > > > > > > > > > > >> > > > > interactive
>> > > > > > > > > > > > > > > >> > > > > > CLI? I think an interactive CLI
>> > would
>> > > be
>> > > > > > > useful
>> > > > > > > > > for
>> > > > > > > > > > > the
>> > > > > > > > > > > > > > > >> developer
>> > > > > > > > > > > > > > > >> > > > > > experience, however, it's
>> important
>> > to
>> > > > > keep
>> > > > > > > the
>> > > > > > > > > > > > unattended
>> > > > > > > > > > > > > > > >> > (current)
>> > > > > > > > > > > > > > > >> > > > CLI
>> > > > > > > > > > > > > > > >> > > > > in
>> > > > > > > > > > > > > > > >> > > > > > place for scenarios where users
>> want
>> > > to
>> > > > > > > automate
>> > > > > > > > > > > Marvin
>> > > > > > > > > > > > > > tasks.
>> > > > > > > > > > > > > > > >> > > > > >
>> > > > > > > > > > > > > > > >> > > > > > Alan, I believe the idea is still
>> > the
>> > > > same
>> > > > > > as
>> > > > > > > > you
>> > > > > > > > > > > > started,
>> > > > > > > > > > > > > > but
>> > > > > > > > > > > > > > > >> > using
>> > > > > > > > > > > > > > > >> > > > > > docker SDK now.
>> > > > > > > > > > > > > > > >> > > > > >
>> > > > > > > > > > > > > > > >> > > > > > - Lucas
>> > > > > > > > > > > > > > > >> > > > > >
>> > > > > > > > > > > > > > > >> > > > > > On Thu, Oct 18, 2018 at 1:29 PM
>> > Rafael
>> > > > > > > Novello <
>> > > > > > > > > > > > > > > >> > > > > > rafa.reis.novello@gmail.com>
>> > > > > > > > > > > > > > > >> > > > > > wrote:
>> > > > > > > > > > > > > > > >> > > > > >
>> > > > > > > > > > > > > > > >> > > > > > > Hi Lucas!
>> > > > > > > > > > > > > > > >> > > > > > >
>> > > > > > > > > > > > > > > >> > > > > > > First of all +1 for REPL POCs
>> to
>> > > > Apache!
>> > > > > > > > > > > > > > > >> > > > > > >
>> > > > > > > > > > > > > > > >> > > > > > > Let me help with some comments:
>> > > > > > > > > > > > > > > >> > > > > > >
>> > > > > > > > > > > > > > > >> > > > > > > 1 - We have tested NodeJS,
>> Scala
>> > and
>> > > > > > Python
>> > > > > > > > and
>> > > > > > > > > > the
>> > > > > > > > > > > > > > easiest
>> > > > > > > > > > > > > > > >> one
>> > > > > > > > > > > > > > > >> > was
>> > > > > > > > > > > > > > > >> > > > > > Python.
>> > > > > > > > > > > > > > > >> > > > > > > We have found a small project
>> [1]
>> > > that
>> > > > > > have
>> > > > > > > > all
>> > > > > > > > > > > > features
>> > > > > > > > > > > > > > we
>> > > > > > > > > > > > > > > >> > desired
>> > > > > > > > > > > > > > > >> > > > for
>> > > > > > > > > > > > > > > >> > > > > > > REPL:
>> > > > > > > > > > > > > > > >> > > > > > > - Autocomplete
>> > > > > > > > > > > > > > > >> > > > > > > - Python commands disabled (the
>> > user
>> > > > > have
>> > > > > > > only
>> > > > > > > > > the
>> > > > > > > > > > > > > > commands
>> > > > > > > > > > > > > > > we
>> > > > > > > > > > > > > > > >> > > > > provide).
>> > > > > > > > > > > > > > > >> > > > > > > The other languages REPL
>> options
>> > > don't
>> > > > > > have
>> > > > > > > > this
>> > > > > > > > > > > > > feature.
>> > > > > > > > > > > > > > > >> > > > > > > - Easy to show output
>> > > > > > > > > > > > > > > >> > > > > > > - Etc
>> > > > > > > > > > > > > > > >> > > > > > > So, I think the language chosen
>> > will
>> > > > not
>> > > > > > be
>> > > > > > > > > > > important
>> > > > > > > > > > > > > here
>> > > > > > > > > > > > > > > >> > because
>> > > > > > > > > > > > > > > >> > > > the
>> > > > > > > > > > > > > > > >> > > > > > user
>> > > > > > > > > > > > > > > >> > > > > > > will only interact with the
>> > commands
>> > > > > that
>> > > > > > we
>> > > > > > > > > > create.
>> > > > > > > > > > > > > > > >> > > > > > >
>> > > > > > > > > > > > > > > >> > > > > > > 2 - The "engine-generate"
>> command
>> > > will
>> > > > > > > > download
>> > > > > > > > > a
>> > > > > > > > > > > > docker
>> > > > > > > > > > > > > > > image
>> > > > > > > > > > > > > > > >> > that
>> > > > > > > > > > > > > > > >> > > > we
>> > > > > > > > > > > > > > > >> > > > > > > create for that language and
>> > start a
>> > > > > > > container
>> > > > > > > > > > with
>> > > > > > > > > > > > the
>> > > > > > > > > > > > > > > basic
>> > > > > > > > > > > > > > > >> > > project
>> > > > > > > > > > > > > > > >> > > > > > > structure for that language.
>> > > > > > > > > > > > > > > >> > > > > > >
>> > > > > > > > > > > > > > > >> > > > > > > 3 - The REPL client will use
>> the
>> > > > "docker
>> > > > > > > > > protocol"
>> > > > > > > > > > > to
>> > > > > > > > > > > > > run
>> > > > > > > > > > > > > > > >> > command,
>> > > > > > > > > > > > > > > >> > > > > > > start/stop services and etc
>> inside
>> > > the
>> > > > > > > > container
>> > > > > > > > > > and
>> > > > > > > > > > > > it
>> > > > > > > > > > > > > > will
>> > > > > > > > > > > > > > > >> > > receive
>> > > > > > > > > > > > > > > >> > > > > the
>> > > > > > > > > > > > > > > >> > > > > > > log stream to show. No, the
>> REPL
>> > > will
>> > > > no
>> > > > > > > pass
>> > > > > > > > > code
>> > > > > > > > > > > > > > snippets
>> > > > > > > > > > > > > > > >> for
>> > > > > > > > > > > > > > > >> > > > docker
>> > > > > > > > > > > > > > > >> > > > > > > container (I think it will not
>> be
>> > > > > > necessary)
>> > > > > > > > > > > > > > > >> > > > > > >
>> > > > > > > > > > > > > > > >> > > > > > > 4 - Yep! Like I said on the
>> first
>> > > > item.
>> > > > > > > > > > > > > > > >> > > > > > >
>> > > > > > > > > > > > > > > >> > > > > > > [1] -
>> > > > > > https://github.com/italorossi/ishell
>> > > > > > > > > > > > > > > >> > > > > > >
>> > > > > > > > > > > > > > > >> > > > > > > Let me know if there is any
>> other
>> > > > > > question!
>> > > > > > > > > > > > > > > >> > > > > > > Best!
>> > > > > > > > > > > > > > > >> > > > > > > Rafael J. R. Novello
>> > > > > > > > > > > > > > > >> > > > > > >
>> > > > > > > > > > > > > > > >> > > > > > > Skype: rafael.novello
>> > > > > > > > > > > > > > > >> > > > > > > Blog:
>> > > > > http://rafanovello.blogspot.com.br/
>> > > > > > > > > > > > > > > >> > > > > > >
>> > > > > > > > > > > > > > > >> > > > > > >
>> > > > > > > > > > > > > > > >> > > > > > > Em qui, 18 de out de 2018 às
>> > 17:00,
>> > > > > Lucas
>> > > > > > > > > Bonatto
>> > > > > > > > > > > > > Miguel <
>> > > > > > > > > > > > > > > >> > > > > > > lucasbm88@apache.org>
>> escreveu:
>> > > > > > > > > > > > > > > >> > > > > > >
>> > > > > > > > > > > > > > > >> > > > > > > > + 1 for migrating the REPL
>> repo
>> > to
>> > > > > > Apache
>> > > > > > > > > > > > > > > >> > > > > > > >
>> > > > > > > > > > > > > > > >> > > > > > > > I have a few questions about
>> the
>> > > > > > previous
>> > > > > > > > > > > > explanation:
>> > > > > > > > > > > > > > > >> > > > > > > >  1) The REPL itself, it
>> would be
>> > > an
>> > > > > > > > > application
>> > > > > > > > > > in
>> > > > > > > > > > > > > which
>> > > > > > > > > > > > > > > >> > > language?
>> > > > > > > > > > > > > > > >> > > > > > > Remember
>> > > > > > > > > > > > > > > >> > > > > > > > that the main idea is to
>> allow
>> > the
>> > > > > user
>> > > > > > to
>> > > > > > > > > > program
>> > > > > > > > > > > > on
>> > > > > > > > > > > > > > his
>> > > > > > > > > > > > > > > >> > > preferred
>> > > > > > > > > > > > > > > >> > > > > > > > language in the REPL.
>> > > > > > > > > > > > > > > >> > > > > > > >  2) Should the
>> engine-generate
>> > > > command
>> > > > > > > also
>> > > > > > > > > > > > generate a
>> > > > > > > > > > > > > > > >> docker
>> > > > > > > > > > > > > > > >> > > image
>> > > > > > > > > > > > > > > >> > > > > > with
>> > > > > > > > > > > > > > > >> > > > > > > > the user's application?
>> > > > > > > > > > > > > > > >> > > > > > > >  3) What type of
>> communication
>> > > would
>> > > > > > > happen
>> > > > > > > > > > > between
>> > > > > > > > > > > > > the
>> > > > > > > > > > > > > > > REPL
>> > > > > > > > > > > > > > > >> > and
>> > > > > > > > > > > > > > > >> > > > the
>> > > > > > > > > > > > > > > >> > > > > > > engine
>> > > > > > > > > > > > > > > >> > > > > > > > via Docker SDK? Would the
>> REPL
>> > > pass
>> > > > > > > snippets
>> > > > > > > > > of
>> > > > > > > > > > > code
>> > > > > > > > > > > > > to
>> > > > > > > > > > > > > > be
>> > > > > > > > > > > > > > > >> > > executed
>> > > > > > > > > > > > > > > >> > > > > by
>> > > > > > > > > > > > > > > >> > > > > > > the
>> > > > > > > > > > > > > > > >> > > > > > > > docker container?
>> > > > > > > > > > > > > > > >> > > > > > > >  4) Have you considered code
>> > > > > completion
>> > > > > > in
>> > > > > > > > the
>> > > > > > > > > > > REPL?
>> > > > > > > > > > > > > > > >> > > > > > > >
>> > > > > > > > > > > > > > > >> > > > > > > >
>> > > > > > > > > > > > > > > >> > > > > > > > On Thu, Oct 18, 2018 at 7:53
>> AM
>> > > > Zhang
>> > > > > > > Yifei
>> > > > > > > > <
>> > > > > > > > > > > > > > > >> > > > yifei.z.luiz@gmail.com>
>> > > > > > > > > > > > > > > >> > > > > > > > wrote:
>> > > > > > > > > > > > > > > >> > > > > > > >
>> > > > > > > > > > > > > > > >> > > > > > > > > Ok guys,
>> > > > > > > > > > > > > > > >> > > > > > > > >
>> > > > > > > > > > > > > > > >> > > > > > > > > The basic idea is to
>> provide
>> > > only
>> > > > > one
>> > > > > > > > > Toolbox
>> > > > > > > > > > > for
>> > > > > > > > > > > > > > > multiple
>> > > > > > > > > > > > > > > >> > > > > languages.
>> > > > > > > > > > > > > > > >> > > > > > > > > We are looking for
>> possibility
>> > > to
>> > > > > > build
>> > > > > > > a
>> > > > > > > > > > single
>> > > > > > > > > > > > > > Marvin
>> > > > > > > > > > > > > > > >> Repl,
>> > > > > > > > > > > > > > > >> > > > > > > > > instead of severals
>> toolboxes
>> > > with
>> > > > > > > > > differentes
>> > > > > > > > > > > > > > > interfaces
>> > > > > > > > > > > > > > > >> or
>> > > > > > > > > > > > > > > >> > > > > > commands.
>> > > > > > > > > > > > > > > >> > > > > > > > >
>> > > > > > > > > > > > > > > >> > > > > > > > > In this case, the
>> > > engine-generate
>> > > > > > > command
>> > > > > > > > > will
>> > > > > > > > > > > > > > download
>> > > > > > > > > > > > > > > >> and
>> > > > > > > > > > > > > > > >> > > > start a
>> > > > > > > > > > > > > > > >> > > > > > > > Docker
>> > > > > > > > > > > > > > > >> > > > > > > > > container with basic engine
>> > > > > structure
>> > > > > > > > > > > > corresponding
>> > > > > > > > > > > > > > the
>> > > > > > > > > > > > > > > >> > choosed
>> > > > > > > > > > > > > > > >> > > > > > > language.
>> > > > > > > > > > > > > > > >> > > > > > > > > this means we don't need to
>> > > build
>> > > > > > > > Toolboxes
>> > > > > > > > > of
>> > > > > > > > > > > > > > > differents
>> > > > > > > > > > > > > > > >> > > > > languages,
>> > > > > > > > > > > > > > > >> > > > > > we
>> > > > > > > > > > > > > > > >> > > > > > > > > just
>> > > > > > > > > > > > > > > >> > > > > > > > > build the engine template
>> of
>> > all
>> > > > > > > languages
>> > > > > > > > > > that
>> > > > > > > > > > > we
>> > > > > > > > > > > > > > want
>> > > > > > > > > > > > > > > to
>> > > > > > > > > > > > > > > >> > > > support
>> > > > > > > > > > > > > > > >> > > > > > and
>> > > > > > > > > > > > > > > >> > > > > > > > > provide it
>> > > > > > > > > > > > > > > >> > > > > > > > > as Docker containers
>> > > > > > > > > > > > > > > >> > > > > > > > >
>> > > > > > > > > > > > > > > >> > > > > > > > > We have started researches
>> > > around
>> > > > > our
>> > > > > > > > basic
>> > > > > > > > > > > > > > requirements
>> > > > > > > > > > > > > > > >> > like:
>> > > > > > > > > > > > > > > >> > > > > > > > > - Repl interface
>> > > > > > > > > > > > > > > >> > > > > > > > > - System communication
>> > > > > > > > > > > > > > > >> > > > > > > > > - Connection security
>> > > > > > > > > > > > > > > >> > > > > > > > > - Tool popularity
>> > > > > > > > > > > > > > > >> > > > > > > > > - Update complexity
>> > > > > > > > > > > > > > > >> > > > > > > > > - Languages support
>> > > > > > > > > > > > > > > >> > > > > > > > > - ......
>> > > > > > > > > > > > > > > >> > > > > > > > >
>> > > > > > > > > > > > > > > >> > > > > > > > > And we did some POC with
>> code
>> > > > here:
>> > > > > > > > > > > > > > > >> > > > > > > > >
>> > > > > > > https://github.com/marvin-ai/marvin-repl
>> > > > > > > > > > > > > > > >> > > > > > > > >
>> > > > > > > > > > > > > > > >> > > > > > > > > There is POC testing gRPC
>> > using
>> > > > > Scala
>> > > > > > > and
>> > > > > > > > > > > Python,
>> > > > > > > > > > > > > > > >> > > > > > > > > Repl inteface and Docker
>> SDK
>> > > with
>> > > > > > > NodeJS,
>> > > > > > > > > > > > > > > >> > > > > > > > > Repl interface and Docker
>> SDK
>> > > with
>> > > > > > > Python.
>> > > > > > > > > > > > > > > >> > > > > > > > >
>> > > > > > > > > > > > > > > >> > > > > > > > > At this moment we prefer
>> the
>> > > Repl
>> > > > > > > > interface
>> > > > > > > > > +
>> > > > > > > > > > > > Docker
>> > > > > > > > > > > > > > SDK
>> > > > > > > > > > > > > > > >> way,
>> > > > > > > > > > > > > > > >> > > > > because
>> > > > > > > > > > > > > > > >> > > > > > > > good
>> > > > > > > > > > > > > > > >> > > > > > > > > part of the requirements
>> > > > > > > > > > > > > > > >> > > > > > > > > will be guaranteed by
>> Docker.
>> > > > > > > > > > > > > > > >> > > > > > > > >
>> > > > > > > > > > > > > > > >> > > > > > > > > With this informations,
>> what
>> > do
>> > > > you
>> > > > > > > think?
>> > > > > > > > > > > Should
>> > > > > > > > > > > > we
>> > > > > > > > > > > > > > > >> submit
>> > > > > > > > > > > > > > > >> > all
>> > > > > > > > > > > > > > > >> > > > > this
>> > > > > > > > > > > > > > > >> > > > > > > POCs
>> > > > > > > > > > > > > > > >> > > > > > > > > to Apache Repo?
>> > > > > > > > > > > > > > > >> > > > > > > > > Please feel free to opine.
>> > > > > > > > > > > > > > > >> > > > > > > > >
>> > > > > > > > > > > > > > > >> > > > > > > > >
>> > > > > > > > > > > > > > > >> > > > > > > > > Thats all, thanks!!!
>> > > > > > > > > > > > > > > >> > > > > > > > >
>> > > > > > > > > > > > > > > >> > > > > > > > >
>> > > > > > > > > > > > > > > >> > > > > > > > > Em ter, 16 de out de 2018
>> às
>> > > > 18:55,
>> > > > > > > Daniel
>> > > > > > > > > > > > > > Takabayashi <
>> > > > > > > > > > > > > > > >> > > > > > > > >
>> daniel.takabayashi@gmail.com>
>> > > > > > escreveu:
>> > > > > > > > > > > > > > > >> > > > > > > > >
>> > > > > > > > > > > > > > > >> > > > > > > > > > Zhang,
>> > > > > > > > > > > > > > > >> > > > > > > > > >
>> > > > > > > > > > > > > > > >> > > > > > > > > > I think the best
>> approach is
>> > > > give
>> > > > > > us a
>> > > > > > > > > > better
>> > > > > > > > > > > > > > > >> explanation
>> > > > > > > > > > > > > > > >> > > about
>> > > > > > > > > > > > > > > >> > > > > > this
>> > > > > > > > > > > > > > > >> > > > > > > > new
>> > > > > > > > > > > > > > > >> > > > > > > > > > feature and how this can
>> > help
>> > > us
>> > > > > to
>> > > > > > > > > archive
>> > > > > > > > > > > the
>> > > > > > > > > > > > > > > desired
>> > > > > > > > > > > > > > > >> > > feature
>> > > > > > > > > > > > > > > >> > > > > > > > (support
>> > > > > > > > > > > > > > > >> > > > > > > > > > multiple languages).
>> Than if
>> > > the
>> > > > > > > > majority
>> > > > > > > > > > > agree
>> > > > > > > > > > > > > > than I
>> > > > > > > > > > > > > > > >> can
>> > > > > > > > > > > > > > > >> > do
>> > > > > > > > > > > > > > > >> > > > the
>> > > > > > > > > > > > > > > >> > > > > > > > merge,
>> > > > > > > > > > > > > > > >> > > > > > > > > > just like I did before,
>> but
>> > in
>> > > > > > another
>> > > > > > > > > > branch.
>> > > > > > > > > > > > > > > >> > > > > > > > > >
>> > > > > > > > > > > > > > > >> > > > > > > > > > Let me know if makes
>> sense,
>> > > ok?
>> > > > > > > > > > > > > > > >> > > > > > > > > >
>> > > > > > > > > > > > > > > >> > > > > > > > > > Taka
>> > > > > > > > > > > > > > > >> > > > > > > > > >
>> > > > > > > > > > > > > > > >> > > > > > > > > > Em ter, 16 de out de
>> 2018 às
>> > > > > 14:00,
>> > > > > > > > > Luciano
>> > > > > > > > > > > > > Resende
>> > > > > > > > > > > > > > <
>> > > > > > > > > > > > > > > >> > > > > > > > > luckbr1975@gmail.com>
>> > > > > > > > > > > > > > > >> > > > > > > > > > escreveu:
>> > > > > > > > > > > > > > > >> > > > > > > > > >
>> > > > > > > > > > > > > > > >> > > > > > > > > > > So, what is the POC, is
>> > it a
>> > > > > > > > refactoring
>> > > > > > > > > > of
>> > > > > > > > > > > > the
>> > > > > > > > > > > > > > > >> existing
>> > > > > > > > > > > > > > > >> > > > repo?
>> > > > > > > > > > > > > > > >> > > > > Or
>> > > > > > > > > > > > > > > >> > > > > > > is
>> > > > > > > > > > > > > > > >> > > > > > > > > > > it a new rewrite of the
>> > > repo?
>> > > > > > > > > > > > > > > >> > > > > > > > > > >
>> > > > > > > > > > > > > > > >> > > > > > > > > > > Just asking as it might
>> > make
>> > > > > sense
>> > > > > > > to
>> > > > > > > > > make
>> > > > > > > > > > > it
>> > > > > > > > > > > > a
>> > > > > > > > > > > > > > > branch
>> > > > > > > > > > > > > > > >> > then
>> > > > > > > > > > > > > > > >> > > > > > > actually
>> > > > > > > > > > > > > > > >> > > > > > > > a
>> > > > > > > > > > > > > > > >> > > > > > > > > > > thing in parallel, as
>> this
>> > > > will
>> > > > > > have
>> > > > > > > > an
>> > > > > > > > > > > effect
>> > > > > > > > > > > > > for
>> > > > > > > > > > > > > > > >> > > releases,
>> > > > > > > > > > > > > > > >> > > > > etc.
>> > > > > > > > > > > > > > > >> > > > > > > but
>> > > > > > > > > > > > > > > >> > > > > > > > > > > you guys know more here
>> > > than I
>> > > > > do.
>> > > > > > > > > > > > > > > >> > > > > > > > > > >
>> > > > > > > > > > > > > > > >> > > > > > > > > > > Also, it's probably
>> good
>> > to
>> > > > > have a
>> > > > > > > > write
>> > > > > > > > > > up
>> > > > > > > > > > > of
>> > > > > > > > > > > > > the
>> > > > > > > > > > > > > > > >> main
>> > > > > > > > > > > > > > > >> > > > > direction
>> > > > > > > > > > > > > > > >> > > > > > > of
>> > > > > > > > > > > > > > > >> > > > > > > > > > > the design that can
>> help
>> > > > people
>> > > > > > get
>> > > > > > > > > > familiar
>> > > > > > > > > > > > > with
>> > > > > > > > > > > > > > > the
>> > > > > > > > > > > > > > > >> new
>> > > > > > > > > > > > > > > >> > > > > > approach.
>> > > > > > > > > > > > > > > >> > > > > > > > > > > On Tue, Oct 16, 2018 at
>> > > 11:12
>> > > > AM
>> > > > > > > Zhang
>> > > > > > > > > > > Yifei <
>> > > > > > > > > > > > > > > >> > > > > > > yifei.z.luiz@gmail.com
>> > > > > > > > > > > > > > > >> > > > > > > > >
>> > > > > > > > > > > > > > > >> > > > > > > > > > > wrote:
>> > > > > > > > > > > > > > > >> > > > > > > > > > > >
>> > > > > > > > > > > > > > > >> > > > > > > > > > > > Hey guys,
>> > > > > > > > > > > > > > > >> > > > > > > > > > > >
>> > > > > > > > > > > > > > > >> > > > > > > > > > > > We have reorganized
>> the
>> > > Poc
>> > > > > > repo,
>> > > > > > > > and
>> > > > > > > > > > want
>> > > > > > > > > > > > to
>> > > > > > > > > > > > > > > merge
>> > > > > > > > > > > > > > > >> it
>> > > > > > > > > > > > > > > >> > to
>> > > > > > > > > > > > > > > >> > > > our
>> > > > > > > > > > > > > > > >> > > > > > > > Apache
>> > > > > > > > > > > > > > > >> > > > > > > > > > > repo.
>> > > > > > > > > > > > > > > >> > > > > > > > > > > > Just making sure here
>> > > before
>> > > > > we
>> > > > > > do
>> > > > > > > > the
>> > > > > > > > > > > > merge,
>> > > > > > > > > > > > > > > >> because
>> > > > > > > > > > > > > > > >> > im
>> > > > > > > > > > > > > > > >> > > > not
>> > > > > > > > > > > > > > > >> > > > > > sure
>> > > > > > > > > > > > > > > >> > > > > > > > if
>> > > > > > > > > > > > > > > >> > > > > > > > > > > doing
>> > > > > > > > > > > > > > > >> > > > > > > > > > > > this will perserve
>> the
>> > Git
>> > > > > > commit
>> > > > > > > > > > history.
>> > > > > > > > > > > > > > > >> > > > > > > > > > > > What we are planning
>> to
>> > do
>> > > > is:
>> > > > > > > > > > > > > > > >> > > > > > > > > > > >
>> > > > > > > > > > > > > > > >> > > > > > > > > > > > git filter-branch
>> > > > > > > > > --subdirectory-filter
>> > > > > > > > > > > <git
>> > > > > > > > > > > > > > > >> > > > > origin_repository
>> > > > > > > > > > > > > > > >> > > > > > > > > > directory>
>> > > > > > > > > > > > > > > >> > > > > > > > > > > > -- --all
>> > > > > > > > > > > > > > > >> > > > > > > > > > > > mkdir POCs/
>> > > > > > > > > > > > > > > >> > > > > > > > > > > > git mv * POCs/
>> > > > > > > > > > > > > > > >> > > > > > > > > > > > git commit -m
>> "colleted
>> > > the
>> > > > > data
>> > > > > > > to
>> > > > > > > > > > move"
>> > > > > > > > > > > > > > > >> > > > > > > > > > > >
>> > > > > > > > > > > > > > > >> > > > > > > > > > > > git clone <git
>> > > > > > > > > > destination_repository_url>
>> > > > > > > > > > > > cd
>> > > > > > > > > > > > > > <git
>> > > > > > > > > > > > > > > >> > > > > > > > > > > >
>> > > > > > destination_repository_directory>
>> > > > > > > > git
>> > > > > > > > > > > remote
>> > > > > > > > > > > > > add
>> > > > > > > > > > > > > > > >> > > > > > > > > > origin_repository_branch
>> > > > > > > > > > > > > > > >> > > > > > > > > > > > <git
>> origin_repository
>> > > > > > directory>
>> > > > > > > > git
>> > > > > > > > > > pull
>> > > > > > > > > > > > > > > >> > > > > > > origin_repository_branch
>> > > > > > > > > > > > > > > >> > > > > > > > > > > master
>> > > > > > > > > > > > > > > >> > > > > > > > > > > >
>> > > --allow-unrelated-histories
>> > > > > git
>> > > > > > > > remote
>> > > > > > > > > > rm
>> > > > > > > > > > > > > > > >> > > > > > > origin_repository_branch
>> > > > > > > > > > > > > > > >> > > > > > > > > > > >
>> > > > > > > > > > > > > > > >> > > > > > > > > > > >
>> > > > > > > > > > > > > > > >> > > > > > > > > > > > Thanks in advance !
>> > > > > > > > > > > > > > > >> > > > > > > > > > > >
>> > > > > > > > > > > > > > > >> > > > > > > > > > > >
>> > > > > > > > > > > > > > > >> > > > >
>> > > > > > > > > > > > >
>> > > > > > --------------------------------------------------------------
>> > > > > > > > > > > > > > > >> > > > > > > > > > > > Zhang Yifei
>> > > > > > > > > > > > > > > >> > > > > > > > > > >
>> > > > > > > > > > > > > > > >> > > > > > > > > > >
>> > > > > > > > > > > > > > > >> > > > > > > > > > >
>> > > > > > > > > > > > > > > >> > > > > > > > > > > --
>> > > > > > > > > > > > > > > >> > > > > > > > > > > Luciano Resende
>> > > > > > > > > > > > > > > >> > > > > > > > > > >
>> > > > http://twitter.com/lresende1975
>> > > > > > > > > > > > > > > >> > > > > > > > > > >
>> > > http://lresende.blogspot.com/
>> > > > > > > > > > > > > > > >> > > > > > > > > > >
>> > > > > > > > > > > > > > > >> > > > > > > > > >
>> > > > > > > > > > > > > > > >> > > > > > > > >
>> > > > > > > > > > > > > > > >> > > > > > > > >
>> > > > > > > > > > > > > > > >> > > > > > > > > --
>> > > > > > > > > > > > > > > >> > > > > > > > >
>> > > > > > > > > > > > > > > >> >
>> > > > > > > > > >
>> > > --------------------------------------------------------------
>> > > > > > > > > > > > > > > >> > > > > > > > > Zhang Yifei
>> > > > > > > > > > > > > > > >> > > > > > > > >
>> > > > > > > > > > > > > > > >> > > > > > > >
>> > > > > > > > > > > > > > > >> > > > > > >
>> > > > > > > > > > > > > > > >> > > > > >
>> > > > > > > > > > > > > > > >> > > > >
>> > > > > > > > > > > > > > > >> > > >
>> > > > > > > > > > > > > > > >> > >
>> > > > > > > > > > > > > > > >> >
>> > > > > > > > > > > > > > > >>
>> > > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > >
>> > > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> >
>> > --
>> > --------------------------------------------------------------
>> > Zhang Yifei
>> >
>>
>

Re: POCs, Prototypes, experiments ...

Posted by Lucas Bonatto Miguel <lu...@apache.org>.

Yes, since we haven't heard any concerns, I believe we're good to start
coding.

The next step is to create issues on Jira; I have all the issues ready to
be written on Jira, however, I can't find Marvin on Apache's Jira.

Could our mentors help to create our project on Jira?

Thanks

On Wed, Oct 31, 2018 at 10:42 AM Rafael Novello <ra...@gmail.com>
wrote:

> Guys,
>
> How can we organize the tasks to start working at this new version?
>
> I believe we had the core devs approval.
>
> Atenciosamente,
> Rafael J. R. Novello
>
> Skype: rafael.novello
> Blog: http://rafanovello.blogspot.com.br/
>
>
> Em ter, 30 de out de 2018 às 14:30, Zhang Yifei <yi...@gmail.com>
> escreveu:
>
> > +1
> >
> > Em seg, 29 de out de 2018 às 18:23, Rafael Novello <
> > rafa.reis.novello@gmail.com> escreveu:
> >
> > > Lucas, yes! It make sense!
> > >
> > > In the end, CLI and REPL will share a big part of code and they will be
> > > just different interfaces to the same features.
> > >
> > > +1 for this new concept!
> > >
> > > Thanks for make it clear to me!
> > > Atenciosamente,
> > > Rafael J. R. Novello
> > >
> > > Skype: rafael.novello
> > > Blog: http://rafanovello.blogspot.com.br/
> > >
> > >
> > > Em seg, 29 de out de 2018 às 17:43, Lucas Bonatto Miguel <
> > > lucasbm88@apache.org> escreveu:
> > >
> > > > That's a good point Rafael. I believe the main motivation for adding
> a
> > > REPL
> > > > to Marvin's toolbox is to enhance user experience.
> > > >
> > > > Imo we can't just switch CLI per REPL. I don't think one excludes the
> > > > other, both have different use cases.
> > > >
> > > > We may be mixing too much stuff together, but in general, I think we
> > are:
> > > >  - Refactoring toolbox architecture to avoid unnecessarily duplicated
> > > logic
> > > > when adding support to new languages.
> > > >  - Adding a new interface to enable users to open an interactive
> > session
> > > > with the engine (i.e. REPL)
> > > >  - Making it easier to add support for new languages.
> > > >
> > > > Does it make sense for you?
> > > >
> > > > On Mon, Oct 29, 2018 at 11:22 AM Rafael Novello <
> > > > rafa.reis.novello@gmail.com>
> > > > wrote:
> > > >
> > > > > I do not want to be the annoying conversation but...
> > > > >
> > > > > If we will have an unique CLI that will be able to handle any kind
> of
> > > > > engines (in any language) by using docker, Why we need a REPL?
> > > > >
> > > > > I think we can focus on creating this new architecture using docker
> > and
> > > > > docker SDK and use only a CLI for that.
> > > > >
> > > > > Atenciosamente,
> > > > > Rafael J. R. Novello
> > > > >
> > > > > Skype: rafael.novello
> > > > > Blog: http://rafanovello.blogspot.com.br/
> > > > >
> > > > >
> > > > > Em seg, 29 de out de 2018 às 14:58, Lucas Bonatto Miguel <
> > > > > lucasbm88@apache.org> escreveu:
> > > > >
> > > > > > No, I think the idea is to do not create different versions of
> the
> > > CLI.
> > > > > > Let's use one single tech stack for the CLI, ideally, it will be
> > > shared
> > > > > > with the REPL. The way I see the CLI would be very similar to the
> > > REPL,
> > > > > but
> > > > > > instead of opening an interactive session the user could send
> > > > > > unattended commands. As an analogy, you can think on spark-shell
> > (our
> > > > > REPL)
> > > > > > and spark-submit (our CLI).
> > > > > >
> > > > > > - Lucas
> > > > > >
> > > > > > On Mon, Oct 29, 2018 at 10:27 AM Rafael Novello <
> > > > > > rafa.reis.novello@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Hi Guys!
> > > > > > >
> > > > > > > Let me ask about this new version that include CLI's and REPL.
> > Will
> > > > we
> > > > > > > create different versions of CLI?
> > > > > > >
> > > > > > > One of the reasons to create the REPL is to avoid
> re-implementing
> > > the
> > > > > > same
> > > > > > > features many times,
> > > > > > > each time for a different language.
> > > > > > >
> > > > > > > Regards!
> > > > > > > Atenciosamente,
> > > > > > > Rafael J. R. Novello
> > > > > > >
> > > > > > > Skype: rafael.novello
> > > > > > > Blog: http://rafanovello.blogspot.com.br/
> > > > > > >
> > > > > > >
> > > > > > > Em sex, 26 de out de 2018 às 20:54, Daniel Takabayashi <
> > > > > > > daniel.takabayashi@gmail.com> escreveu:
> > > > > > >
> > > > > > > > +1
> > > > > > > >
> > > > > > > > Em sex, 26 de out de 2018 às 16:47, Lucas Bonatto Miguel <
> > > > > > > > lucasbm88@apache.org> escreveu:
> > > > > > > >
> > > > > > > > > Makes perfect sense to me. I believe that if the majority
> of
> > > > people
> > > > > > > > agrees
> > > > > > > > > with this architecture we could create issues and start
> > > > > implementing
> > > > > > > it.
> > > > > > > > >
> > > > > > > > > +1
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Fri, Oct 26, 2018 at 4:09 PM Daniel Takabayashi <
> > > > > > > > > daniel.takabayashi@gmail.com> wrote:
> > > > > > > > >
> > > > > > > > > > I see... what about this new version?
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://docs.google.com/drawings/d/shajxIpLJHxxMbFgDXiPuhg/image?w=602&h=468&rev=1630&ac=1&parent=1ySERHGBXbHeyCMRookq5UfTuFkzzU0ugtjvR3rF3deY
> > > > > > > > > >
> > > > > > > > > > I changed the toolbox ...from 1 component to 3... docker
> > > > client,
> > > > > > cli
> > > > > > > > and
> > > > > > > > > > repl. Both (cli and repl) now are using the docker client
> > to
> > > > > > control
> > > > > > > > the
> > > > > > > > > > engines.
> > > > > > > > > >
> > > > > > > > > > Em sex, 26 de out de 2018 às 14:49, Lucas Bonatto Miguel
> <
> > > > > > > > > > lucasbm88@apache.org> escreveu:
> > > > > > > > > >
> > > > > > > > > > > For sure, both suggestions are to make sure we're not
> > > killing
> > > > > the
> > > > > > > CLI
> > > > > > > > > and
> > > > > > > > > > > that we have a component (the SDK client) that is
> shared
> > by
> > > > CLI
> > > > > > and
> > > > > > > > > REPL.
> > > > > > > > > > >
> > > > > > > > > > > Thanks for clarifying where the system calls will live.
> > > > > > > > > > >
> > > > > > > > > > > - Lucas
> > > > > > > > > > >
> > > > > > > > > > > On Fri, Oct 26, 2018 at 2:39 PM Daniel Takabayashi <
> > > > > > > > > > > daniel.takabayashi@gmail.com> wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Hi Lucas, could you please justified your two
> > > suggestions?
> > > > > > > > > > > >
> > > > > > > > > > > > The general ideia is to use the Docker DSK to execute
> > > local
> > > > > > > > commands,
> > > > > > > > > > to
> > > > > > > > > > > > consume outputs and also to execute Marvin "specific
> > > > > language"
> > > > > > > > > > commands.
> > > > > > > > > > > > Today almost all CLIs commands in Marvin is a mixed
> of
> > > > > systems
> > > > > > > > > commands
> > > > > > > > > > > and
> > > > > > > > > > > > library method calls. The plan is to put all
> "specific
> > > > > language
> > > > > > > > > calls"
> > > > > > > > > > > > inside the marvin-common-libs and group the systems
> > call
> > > on
> > > > > the
> > > > > > > > > toolbox
> > > > > > > > > > > > (using the docker SDK in this task).
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > Em qui, 25 de out de 2018 às 20:09, Lucas Bonatto
> > Miguel
> > > <
> > > > > > > > > > > > lucasbm88@apache.org> escreveu:
> > > > > > > > > > > >
> > > > > > > > > > > > > Great, thanks.
> > > > > > > > > > > > >
> > > > > > > > > > > > > So here some suggestions:
> > > > > > > > > > > > >  1. We could create one new layer between Toolbox
> and
> > > the
> > > > > > > > Engines,
> > > > > > > > > > this
> > > > > > > > > > > > > would be Docker client.
> > > > > > > > > > > > >  2. Split the Toolbox layer in two. Toolbox CLI and
> > > > Toolbox
> > > > > > > REPL
> > > > > > > > > > > > >
> > > > > > > > > > > > > It's not clear to me if the commands listed on
> > General
> > > > > > Commands
> > > > > > > > > will
> > > > > > > > > > > need
> > > > > > > > > > > > > to be written for every language, from the image I
> > > > believe
> > > > > > not,
> > > > > > > > but
> > > > > > > > > > how
> > > > > > > > > > > > it
> > > > > > > > > > > > > would work, since starting a notebook in Python is
> > > > > different
> > > > > > > than
> > > > > > > > > > > > starting
> > > > > > > > > > > > > in Scala.
> > > > > > > > > > > > >
> > > > > > > > > > > > > - Lucas
> > > > > > > > > > > > > On Thu, Oct 25, 2018 at 4:51 PM Daniel Takabayashi
> <
> > > > > > > > > > > > > daniel.takabayashi@gmail.com> wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://docs.google.com/drawings/d/shajxIpLJHxxMbFgDXiPuhg/image?w=602&h=461&rev=1423&ac=1&parent=1ySERHGBXbHeyCMRookq5UfTuFkzzU0ugtjvR3rF3deY
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Em qui, 25 de out de 2018 às 07:38, Rafael
> Novello
> > <
> > > > > > > > > > > > > > rafa.reis.novello@gmail.com> escreveu:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Hi Taka,
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > I can't see the image! Maybe the Apache email
> > list
> > > > > don't
> > > > > > > > > support
> > > > > > > > > > > > > > > attachments.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Best!
> > > > > > > > > > > > > > > Rafael J. R. Novello
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Skype: rafael.novello
> > > > > > > > > > > > > > > Blog: http://rafanovello.blogspot.com.br/
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Em qua, 24 de out de 2018 às 21:14, Daniel
> > > > Takabayashi
> > > > > <
> > > > > > > > > > > > > > > daniel.takabayashi@gmail.com> escreveu:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Hi all,
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > To try to give a little more details about
> this
> > > > "new
> > > > > > > > concept"
> > > > > > > > > > > that
> > > > > > > > > > > > we
> > > > > > > > > > > > > > > want
> > > > > > > > > > > > > > > > to bring to the marvin toolbox, I did this
> > simple
> > > > > > > > > architecture
> > > > > > > > > > > > draw.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > [image: marvin-architecture-views (5).png]
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > The general idea here is try to transform the
> > > > toolbox
> > > > > > > > > something
> > > > > > > > > > > > > > > > disconnected with the "language", something
> > more
> > > > > > > agnostic.
> > > > > > > > > Also
> > > > > > > > > > > in
> > > > > > > > > > > > > this
> > > > > > > > > > > > > > > new
> > > > > > > > > > > > > > > > architecture we could use remote resource to
> > > > process
> > > > > > > > engines
> > > > > > > > > > and
> > > > > > > > > > > > make
> > > > > > > > > > > > > > > easy
> > > > > > > > > > > > > > > > the support for new languages.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > This "new toolbox" will be the only thing
> that
> > a
> > > > > Marvin
> > > > > > > > user
> > > > > > > > > > must
> > > > > > > > > > > > to
> > > > > > > > > > > > > > > > install and also we could start to support
> > > > multiples
> > > > > > O.S
> > > > > > > > > (once
> > > > > > > > > > > the
> > > > > > > > > > > > > REPL
> > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > a dummy application that only interprets and
> by
> > > > pass
> > > > > > > > > commands).
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Regards,
> > > > > > > > > > > > > > > > Taka
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Em qua, 24 de out de 2018 às 09:52, Rafael
> > > Novello
> > > > <
> > > > > > > > > > > > > > > > rafa.reis.novello@gmail.com> escreveu:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >> Alan,
> > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > >> Yes! We are using the Docker SDK and it's
> > > possible
> > > > > to
> > > > > > > use
> > > > > > > > > the
> > > > > > > > > > > API
> > > > > > > > > > > > to
> > > > > > > > > > > > > > > >> automate, but it's a little bit harder than
> > > > automate
> > > > > > CLI
> > > > > > > > > > calls.
> > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > >> Atenciosamente,
> > > > > > > > > > > > > > > >> Rafael J. R. Novello
> > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > >> Skype: rafael.novello
> > > > > > > > > > > > > > > >> Blog: http://rafanovello.blogspot.com.br/
> > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > >> Em qua, 24 de out de 2018 às 12:02, Alan
> > Silva <
> > > > > > > > > > > jumpi@apache.org>
> > > > > > > > > > > > > > > >> escreveu:
> > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > >> > Hi,
> > > > > > > > > > > > > > > >> >
> > > > > > > > > > > > > > > >> > One question here, I understand that we
> > start
> > > to
> > > > > use
> > > > > > > > with
> > > > > > > > > > this
> > > > > > > > > > > > PoC
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > >> > Docker SDK API, right?
> > > > > > > > > > > > > > > >> >
> > > > > > > > > > > > > > > >> > Why not use the API to expose some
> endpoints
> > > to
> > > > > > permit
> > > > > > > > > this
> > > > > > > > > > > kind
> > > > > > > > > > > > > of
> > > > > > > > > > > > > > > >> > automation by devops?
> > > > > > > > > > > > > > > >> >
> > > > > > > > > > > > > > > >> > I think it is possible and it solves the
> CLI
> > > > > > problem,
> > > > > > > > > right?
> > > > > > > > > > > > > > > >> >
> > > > > > > > > > > > > > > >> >
> > > > > > > > > > > > > > > >> > On Tue, Oct 23, 2018 at 1:05 PM Rafael
> > > Novello <
> > > > > > > > > > > > > > > >> > rafa.reis.novello@gmail.com>
> > > > > > > > > > > > > > > >> > wrote:
> > > > > > > > > > > > > > > >> >
> > > > > > > > > > > > > > > >> > > Lucas,
> > > > > > > > > > > > > > > >> > >
> > > > > > > > > > > > > > > >> > > Sorry, I didn't understood your question
> > > > bellow.
> > > > > > > > > > > > > > > >> > >
> > > > > > > > > > > > > > > >> > > "Would it make sense to use the same
> > > solution
> > > > > that
> > > > > > > we
> > > > > > > > > will
> > > > > > > > > > > use
> > > > > > > > > > > > > for
> > > > > > > > > > > > > > > >> > having a
> > > > > > > > > > > > > > > >> > > single-language REPL to have a
> > > single-language
> > > > > > CLI?"
> > > > > > > > > > > > > > > >> > >
> > > > > > > > > > > > > > > >> > > For DevOps purposes, maybe this new
> > toolbox
> > > > > > concept
> > > > > > > is
> > > > > > > > > not
> > > > > > > > > > > > > ideal.
> > > > > > > > > > > > > > I
> > > > > > > > > > > > > > > >> think
> > > > > > > > > > > > > > > >> > > we can keep the CLI inside the docker
> > > > container
> > > > > > but
> > > > > > > it
> > > > > > > > > > will
> > > > > > > > > > > > not
> > > > > > > > > > > > > > easy
> > > > > > > > > > > > > > > >> to
> > > > > > > > > > > > > > > >> > > automate
> > > > > > > > > > > > > > > >> > > jobs by this way.
> > > > > > > > > > > > > > > >> > >
> > > > > > > > > > > > > > > >> > > How to deal with this issue? Voting?
> > > > > > > > > > > > > > > >> > > Atenciosamente,
> > > > > > > > > > > > > > > >> > > Rafael J. R. Novello
> > > > > > > > > > > > > > > >> > >
> > > > > > > > > > > > > > > >> > > Skype: rafael.novello
> > > > > > > > > > > > > > > >> > > Blog:
> http://rafanovello.blogspot.com.br/
> > > > > > > > > > > > > > > >> > >
> > > > > > > > > > > > > > > >> > >
> > > > > > > > > > > > > > > >> > > Em sex, 19 de out de 2018 às 19:00,
> Lucas
> > > > > Bonatto
> > > > > > > > > Miguel <
> > > > > > > > > > > > > > > >> > > lucasbm88@apache.org> escreveu:
> > > > > > > > > > > > > > > >> > >
> > > > > > > > > > > > > > > >> > > > Got it! Thanks for clarifying.
> > > > > > > > > > > > > > > >> > > >
> > > > > > > > > > > > > > > >> > > > Would it make sense to use the same
> > > solution
> > > > > > that
> > > > > > > we
> > > > > > > > > > will
> > > > > > > > > > > > use
> > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > >> > having
> > > > > > > > > > > > > > > >> > > a
> > > > > > > > > > > > > > > >> > > > single-language REPL to have a
> > > > single-language
> > > > > > > CLI?
> > > > > > > > My
> > > > > > > > > > > only
> > > > > > > > > > > > > > > concern
> > > > > > > > > > > > > > > >> > with
> > > > > > > > > > > > > > > >> > > > killing the CLI is that you remove an
> > > > > essential
> > > > > > > > > feature
> > > > > > > > > > > for
> > > > > > > > > > > > > > > DevOps.
> > > > > > > > > > > > > > > >> > > >
> > > > > > > > > > > > > > > >> > > > - Lucas
> > > > > > > > > > > > > > > >> > > >
> > > > > > > > > > > > > > > >> > > > On Fri, Oct 19, 2018 at 1:52 PM Rafael
> > > > > Novello <
> > > > > > > > > > > > > > > >> > > > rafa.reis.novello@gmail.com>
> > > > > > > > > > > > > > > >> > > > wrote:
> > > > > > > > > > > > > > > >> > > >
> > > > > > > > > > > > > > > >> > > > > Lucas,
> > > > > > > > > > > > > > > >> > > > >
> > > > > > > > > > > > > > > >> > > > > The idea is that REPL will
> substitute
> > > the
> > > > > > actual
> > > > > > > > > CLI.
> > > > > > > > > > > It's
> > > > > > > > > > > > > > > because
> > > > > > > > > > > > > > > >> > with
> > > > > > > > > > > > > > > >> > > > the
> > > > > > > > > > > > > > > >> > > > > actual concept (using language
> > specific
> > > > CLI)
> > > > > > we
> > > > > > > > will
> > > > > > > > > > > need
> > > > > > > > > > > > to
> > > > > > > > > > > > > > > >> > > re-implement
> > > > > > > > > > > > > > > >> > > > > the same features many times and
> > > probably
> > > > > each
> > > > > > > CLI
> > > > > > > > > > will
> > > > > > > > > > > > > have a
> > > > > > > > > > > > > > > >> > > different
> > > > > > > > > > > > > > > >> > > > > behavior because some language
> > specific
> > > > > > > > restrictions
> > > > > > > > > > > > and/or
> > > > > > > > > > > > > > > >> > > limitations.
> > > > > > > > > > > > > > > >> > > > >
> > > > > > > > > > > > > > > >> > > > > With this new concept, all users
> will
> > > have
> > > > > the
> > > > > > > > same
> > > > > > > > > > > > > experience
> > > > > > > > > > > > > > > >> > > > interacting
> > > > > > > > > > > > > > > >> > > > > with Marvin REPL and they will use
> the
> > > > bests
> > > > > > > tools
> > > > > > > > > to
> > > > > > > > > > > > > develop
> > > > > > > > > > > > > > > >> their
> > > > > > > > > > > > > > > >> > > > engines
> > > > > > > > > > > > > > > >> > > > > (Jupyter Notebook and/or Lab with
> > > > different
> > > > > > > > > languages
> > > > > > > > > > > > > support
> > > > > > > > > > > > > > or
> > > > > > > > > > > > > > > >> even
> > > > > > > > > > > > > > > >> > > > > Apache Zeppelin). All the interact
> > will
> > > > > occur
> > > > > > > > > through
> > > > > > > > > > > REPL
> > > > > > > > > > > > > and
> > > > > > > > > > > > > > > >> docker
> > > > > > > > > > > > > > > >> > > > > protocol.
> > > > > > > > > > > > > > > >> > > > >
> > > > > > > > > > > > > > > >> > > > > Alan, Yes! As Lucas said, the
> concept
> > is
> > > > the
> > > > > > > same
> > > > > > > > > but
> > > > > > > > > > we
> > > > > > > > > > > > can
> > > > > > > > > > > > > > use
> > > > > > > > > > > > > > > >> > docker
> > > > > > > > > > > > > > > >> > > > to
> > > > > > > > > > > > > > > >> > > > > do the same job.
> > > > > > > > > > > > > > > >> > > > >
> > > > > > > > > > > > > > > >> > > > > Best!
> > > > > > > > > > > > > > > >> > > > >
> > > > > > > > > > > > > > > >> > > > > Em sex, 19 de out de 2018 às 00:39,
> > > Lucas
> > > > > > > Bonatto
> > > > > > > > > > > Miguel <
> > > > > > > > > > > > > > > >> > > > > lucasbm88@apache.org> escreveu:
> > > > > > > > > > > > > > > >> > > > >
> > > > > > > > > > > > > > > >> > > > > > Thanks for the clarifications
> > Rafael,
> > > so
> > > > > > from
> > > > > > > > > what I
> > > > > > > > > > > > > > > understood,
> > > > > > > > > > > > > > > >> > > > Marvin's
> > > > > > > > > > > > > > > >> > > > > > developers would not use the REPL
> to
> > > > > explore
> > > > > > > > data
> > > > > > > > > > and
> > > > > > > > > > > > test
> > > > > > > > > > > > > > > >> models,
> > > > > > > > > > > > > > > >> > > i.e.
> > > > > > > > > > > > > > > >> > > > > > develop the engine. Is the idea to
> > > build
> > > > > > > > something
> > > > > > > > > > > more
> > > > > > > > > > > > > like
> > > > > > > > > > > > > > > an
> > > > > > > > > > > > > > > >> > > > > interactive
> > > > > > > > > > > > > > > >> > > > > > CLI? I think an interactive CLI
> > would
> > > be
> > > > > > > useful
> > > > > > > > > for
> > > > > > > > > > > the
> > > > > > > > > > > > > > > >> developer
> > > > > > > > > > > > > > > >> > > > > > experience, however, it's
> important
> > to
> > > > > keep
> > > > > > > the
> > > > > > > > > > > > unattended
> > > > > > > > > > > > > > > >> > (current)
> > > > > > > > > > > > > > > >> > > > CLI
> > > > > > > > > > > > > > > >> > > > > in
> > > > > > > > > > > > > > > >> > > > > > place for scenarios where users
> want
> > > to
> > > > > > > automate
> > > > > > > > > > > Marvin
> > > > > > > > > > > > > > tasks.
> > > > > > > > > > > > > > > >> > > > > >
> > > > > > > > > > > > > > > >> > > > > > Alan, I believe the idea is still
> > the
> > > > same
> > > > > > as
> > > > > > > > you
> > > > > > > > > > > > started,
> > > > > > > > > > > > > > but
> > > > > > > > > > > > > > > >> > using
> > > > > > > > > > > > > > > >> > > > > > docker SDK now.
> > > > > > > > > > > > > > > >> > > > > >
> > > > > > > > > > > > > > > >> > > > > > - Lucas
> > > > > > > > > > > > > > > >> > > > > >
> > > > > > > > > > > > > > > >> > > > > > On Thu, Oct 18, 2018 at 1:29 PM
> > Rafael
> > > > > > > Novello <
> > > > > > > > > > > > > > > >> > > > > > rafa.reis.novello@gmail.com>
> > > > > > > > > > > > > > > >> > > > > > wrote:
> > > > > > > > > > > > > > > >> > > > > >
> > > > > > > > > > > > > > > >> > > > > > > Hi Lucas!
> > > > > > > > > > > > > > > >> > > > > > >
> > > > > > > > > > > > > > > >> > > > > > > First of all +1 for REPL POCs to
> > > > Apache!
> > > > > > > > > > > > > > > >> > > > > > >
> > > > > > > > > > > > > > > >> > > > > > > Let me help with some comments:
> > > > > > > > > > > > > > > >> > > > > > >
> > > > > > > > > > > > > > > >> > > > > > > 1 - We have tested NodeJS, Scala
> > and
> > > > > > Python
> > > > > > > > and
> > > > > > > > > > the
> > > > > > > > > > > > > > easiest
> > > > > > > > > > > > > > > >> one
> > > > > > > > > > > > > > > >> > was
> > > > > > > > > > > > > > > >> > > > > > Python.
> > > > > > > > > > > > > > > >> > > > > > > We have found a small project
> [1]
> > > that
> > > > > > have
> > > > > > > > all
> > > > > > > > > > > > features
> > > > > > > > > > > > > > we
> > > > > > > > > > > > > > > >> > desired
> > > > > > > > > > > > > > > >> > > > for
> > > > > > > > > > > > > > > >> > > > > > > REPL:
> > > > > > > > > > > > > > > >> > > > > > > - Autocomplete
> > > > > > > > > > > > > > > >> > > > > > > - Python commands disabled (the
> > user
> > > > > have
> > > > > > > only
> > > > > > > > > the
> > > > > > > > > > > > > > commands
> > > > > > > > > > > > > > > we
> > > > > > > > > > > > > > > >> > > > > provide).
> > > > > > > > > > > > > > > >> > > > > > > The other languages REPL options
> > > don't
> > > > > > have
> > > > > > > > this
> > > > > > > > > > > > > feature.
> > > > > > > > > > > > > > > >> > > > > > > - Easy to show output
> > > > > > > > > > > > > > > >> > > > > > > - Etc
> > > > > > > > > > > > > > > >> > > > > > > So, I think the language chosen
> > will
> > > > not
> > > > > > be
> > > > > > > > > > > important
> > > > > > > > > > > > > here
> > > > > > > > > > > > > > > >> > because
> > > > > > > > > > > > > > > >> > > > the
> > > > > > > > > > > > > > > >> > > > > > user
> > > > > > > > > > > > > > > >> > > > > > > will only interact with the
> > commands
> > > > > that
> > > > > > we
> > > > > > > > > > create.
> > > > > > > > > > > > > > > >> > > > > > >
> > > > > > > > > > > > > > > >> > > > > > > 2 - The "engine-generate"
> command
> > > will
> > > > > > > > download
> > > > > > > > > a
> > > > > > > > > > > > docker
> > > > > > > > > > > > > > > image
> > > > > > > > > > > > > > > >> > that
> > > > > > > > > > > > > > > >> > > > we
> > > > > > > > > > > > > > > >> > > > > > > create for that language and
> > start a
> > > > > > > container
> > > > > > > > > > with
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > basic
> > > > > > > > > > > > > > > >> > > project
> > > > > > > > > > > > > > > >> > > > > > > structure for that language.
> > > > > > > > > > > > > > > >> > > > > > >
> > > > > > > > > > > > > > > >> > > > > > > 3 - The REPL client will use the
> > > > "docker
> > > > > > > > > protocol"
> > > > > > > > > > > to
> > > > > > > > > > > > > run
> > > > > > > > > > > > > > > >> > command,
> > > > > > > > > > > > > > > >> > > > > > > start/stop services and etc
> inside
> > > the
> > > > > > > > container
> > > > > > > > > > and
> > > > > > > > > > > > it
> > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > >> > > receive
> > > > > > > > > > > > > > > >> > > > > the
> > > > > > > > > > > > > > > >> > > > > > > log stream to show. No, the REPL
> > > will
> > > > no
> > > > > > > pass
> > > > > > > > > code
> > > > > > > > > > > > > > snippets
> > > > > > > > > > > > > > > >> for
> > > > > > > > > > > > > > > >> > > > docker
> > > > > > > > > > > > > > > >> > > > > > > container (I think it will not
> be
> > > > > > necessary)
> > > > > > > > > > > > > > > >> > > > > > >
> > > > > > > > > > > > > > > >> > > > > > > 4 - Yep! Like I said on the
> first
> > > > item.
> > > > > > > > > > > > > > > >> > > > > > >
> > > > > > > > > > > > > > > >> > > > > > > [1] -
> > > > > > https://github.com/italorossi/ishell
> > > > > > > > > > > > > > > >> > > > > > >
> > > > > > > > > > > > > > > >> > > > > > > Let me know if there is any
> other
> > > > > > question!
> > > > > > > > > > > > > > > >> > > > > > > Best!
> > > > > > > > > > > > > > > >> > > > > > > Rafael J. R. Novello
> > > > > > > > > > > > > > > >> > > > > > >
> > > > > > > > > > > > > > > >> > > > > > > Skype: rafael.novello
> > > > > > > > > > > > > > > >> > > > > > > Blog:
> > > > > http://rafanovello.blogspot.com.br/
> > > > > > > > > > > > > > > >> > > > > > >
> > > > > > > > > > > > > > > >> > > > > > >
> > > > > > > > > > > > > > > >> > > > > > > Em qui, 18 de out de 2018 às
> > 17:00,
> > > > > Lucas
> > > > > > > > > Bonatto
> > > > > > > > > > > > > Miguel <
> > > > > > > > > > > > > > > >> > > > > > > lucasbm88@apache.org> escreveu:
> > > > > > > > > > > > > > > >> > > > > > >
> > > > > > > > > > > > > > > >> > > > > > > > + 1 for migrating the REPL
> repo
> > to
> > > > > > Apache
> > > > > > > > > > > > > > > >> > > > > > > >
> > > > > > > > > > > > > > > >> > > > > > > > I have a few questions about
> the
> > > > > > previous
> > > > > > > > > > > > explanation:
> > > > > > > > > > > > > > > >> > > > > > > >  1) The REPL itself, it would
> be
> > > an
> > > > > > > > > application
> > > > > > > > > > in
> > > > > > > > > > > > > which
> > > > > > > > > > > > > > > >> > > language?
> > > > > > > > > > > > > > > >> > > > > > > Remember
> > > > > > > > > > > > > > > >> > > > > > > > that the main idea is to allow
> > the
> > > > > user
> > > > > > to
> > > > > > > > > > program
> > > > > > > > > > > > on
> > > > > > > > > > > > > > his
> > > > > > > > > > > > > > > >> > > preferred
> > > > > > > > > > > > > > > >> > > > > > > > language in the REPL.
> > > > > > > > > > > > > > > >> > > > > > > >  2) Should the engine-generate
> > > > command
> > > > > > > also
> > > > > > > > > > > > generate a
> > > > > > > > > > > > > > > >> docker
> > > > > > > > > > > > > > > >> > > image
> > > > > > > > > > > > > > > >> > > > > > with
> > > > > > > > > > > > > > > >> > > > > > > > the user's application?
> > > > > > > > > > > > > > > >> > > > > > > >  3) What type of communication
> > > would
> > > > > > > happen
> > > > > > > > > > > between
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > REPL
> > > > > > > > > > > > > > > >> > and
> > > > > > > > > > > > > > > >> > > > the
> > > > > > > > > > > > > > > >> > > > > > > engine
> > > > > > > > > > > > > > > >> > > > > > > > via Docker SDK? Would the REPL
> > > pass
> > > > > > > snippets
> > > > > > > > > of
> > > > > > > > > > > code
> > > > > > > > > > > > > to
> > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > >> > > executed
> > > > > > > > > > > > > > > >> > > > > by
> > > > > > > > > > > > > > > >> > > > > > > the
> > > > > > > > > > > > > > > >> > > > > > > > docker container?
> > > > > > > > > > > > > > > >> > > > > > > >  4) Have you considered code
> > > > > completion
> > > > > > in
> > > > > > > > the
> > > > > > > > > > > REPL?
> > > > > > > > > > > > > > > >> > > > > > > >
> > > > > > > > > > > > > > > >> > > > > > > >
> > > > > > > > > > > > > > > >> > > > > > > > On Thu, Oct 18, 2018 at 7:53
> AM
> > > > Zhang
> > > > > > > Yifei
> > > > > > > > <
> > > > > > > > > > > > > > > >> > > > yifei.z.luiz@gmail.com>
> > > > > > > > > > > > > > > >> > > > > > > > wrote:
> > > > > > > > > > > > > > > >> > > > > > > >
> > > > > > > > > > > > > > > >> > > > > > > > > Ok guys,
> > > > > > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > > > > > >> > > > > > > > > The basic idea is to provide
> > > only
> > > > > one
> > > > > > > > > Toolbox
> > > > > > > > > > > for
> > > > > > > > > > > > > > > multiple
> > > > > > > > > > > > > > > >> > > > > languages.
> > > > > > > > > > > > > > > >> > > > > > > > > We are looking for
> possibility
> > > to
> > > > > > build
> > > > > > > a
> > > > > > > > > > single
> > > > > > > > > > > > > > Marvin
> > > > > > > > > > > > > > > >> Repl,
> > > > > > > > > > > > > > > >> > > > > > > > > instead of severals
> toolboxes
> > > with
> > > > > > > > > differentes
> > > > > > > > > > > > > > > interfaces
> > > > > > > > > > > > > > > >> or
> > > > > > > > > > > > > > > >> > > > > > commands.
> > > > > > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > > > > > >> > > > > > > > > In this case, the
> > > engine-generate
> > > > > > > command
> > > > > > > > > will
> > > > > > > > > > > > > > download
> > > > > > > > > > > > > > > >> and
> > > > > > > > > > > > > > > >> > > > start a
> > > > > > > > > > > > > > > >> > > > > > > > Docker
> > > > > > > > > > > > > > > >> > > > > > > > > container with basic engine
> > > > > structure
> > > > > > > > > > > > corresponding
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > >> > choosed
> > > > > > > > > > > > > > > >> > > > > > > language.
> > > > > > > > > > > > > > > >> > > > > > > > > this means we don't need to
> > > build
> > > > > > > > Toolboxes
> > > > > > > > > of
> > > > > > > > > > > > > > > differents
> > > > > > > > > > > > > > > >> > > > > languages,
> > > > > > > > > > > > > > > >> > > > > > we
> > > > > > > > > > > > > > > >> > > > > > > > > just
> > > > > > > > > > > > > > > >> > > > > > > > > build the engine template of
> > all
> > > > > > > languages
> > > > > > > > > > that
> > > > > > > > > > > we
> > > > > > > > > > > > > > want
> > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > >> > > > support
> > > > > > > > > > > > > > > >> > > > > > and
> > > > > > > > > > > > > > > >> > > > > > > > > provide it
> > > > > > > > > > > > > > > >> > > > > > > > > as Docker containers
> > > > > > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > > > > > >> > > > > > > > > We have started researches
> > > around
> > > > > our
> > > > > > > > basic
> > > > > > > > > > > > > > requirements
> > > > > > > > > > > > > > > >> > like:
> > > > > > > > > > > > > > > >> > > > > > > > > - Repl interface
> > > > > > > > > > > > > > > >> > > > > > > > > - System communication
> > > > > > > > > > > > > > > >> > > > > > > > > - Connection security
> > > > > > > > > > > > > > > >> > > > > > > > > - Tool popularity
> > > > > > > > > > > > > > > >> > > > > > > > > - Update complexity
> > > > > > > > > > > > > > > >> > > > > > > > > - Languages support
> > > > > > > > > > > > > > > >> > > > > > > > > - ......
> > > > > > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > > > > > >> > > > > > > > > And we did some POC with
> code
> > > > here:
> > > > > > > > > > > > > > > >> > > > > > > > >
> > > > > > > https://github.com/marvin-ai/marvin-repl
> > > > > > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > > > > > >> > > > > > > > > There is POC testing gRPC
> > using
> > > > > Scala
> > > > > > > and
> > > > > > > > > > > Python,
> > > > > > > > > > > > > > > >> > > > > > > > > Repl inteface and Docker SDK
> > > with
> > > > > > > NodeJS,
> > > > > > > > > > > > > > > >> > > > > > > > > Repl interface and Docker
> SDK
> > > with
> > > > > > > Python.
> > > > > > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > > > > > >> > > > > > > > > At this moment we prefer the
> > > Repl
> > > > > > > > interface
> > > > > > > > > +
> > > > > > > > > > > > Docker
> > > > > > > > > > > > > > SDK
> > > > > > > > > > > > > > > >> way,
> > > > > > > > > > > > > > > >> > > > > because
> > > > > > > > > > > > > > > >> > > > > > > > good
> > > > > > > > > > > > > > > >> > > > > > > > > part of the requirements
> > > > > > > > > > > > > > > >> > > > > > > > > will be guaranteed by
> Docker.
> > > > > > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > > > > > >> > > > > > > > > With this informations, what
> > do
> > > > you
> > > > > > > think?
> > > > > > > > > > > Should
> > > > > > > > > > > > we
> > > > > > > > > > > > > > > >> submit
> > > > > > > > > > > > > > > >> > all
> > > > > > > > > > > > > > > >> > > > > this
> > > > > > > > > > > > > > > >> > > > > > > POCs
> > > > > > > > > > > > > > > >> > > > > > > > > to Apache Repo?
> > > > > > > > > > > > > > > >> > > > > > > > > Please feel free to opine.
> > > > > > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > > > > > >> > > > > > > > > Thats all, thanks!!!
> > > > > > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > > > > > >> > > > > > > > > Em ter, 16 de out de 2018 às
> > > > 18:55,
> > > > > > > Daniel
> > > > > > > > > > > > > > Takabayashi <
> > > > > > > > > > > > > > > >> > > > > > > > >
> daniel.takabayashi@gmail.com>
> > > > > > escreveu:
> > > > > > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > > > > > >> > > > > > > > > > Zhang,
> > > > > > > > > > > > > > > >> > > > > > > > > >
> > > > > > > > > > > > > > > >> > > > > > > > > > I think the best approach
> is
> > > > give
> > > > > > us a
> > > > > > > > > > better
> > > > > > > > > > > > > > > >> explanation
> > > > > > > > > > > > > > > >> > > about
> > > > > > > > > > > > > > > >> > > > > > this
> > > > > > > > > > > > > > > >> > > > > > > > new
> > > > > > > > > > > > > > > >> > > > > > > > > > feature and how this can
> > help
> > > us
> > > > > to
> > > > > > > > > archive
> > > > > > > > > > > the
> > > > > > > > > > > > > > > desired
> > > > > > > > > > > > > > > >> > > feature
> > > > > > > > > > > > > > > >> > > > > > > > (support
> > > > > > > > > > > > > > > >> > > > > > > > > > multiple languages). Than
> if
> > > the
> > > > > > > > majority
> > > > > > > > > > > agree
> > > > > > > > > > > > > > than I
> > > > > > > > > > > > > > > >> can
> > > > > > > > > > > > > > > >> > do
> > > > > > > > > > > > > > > >> > > > the
> > > > > > > > > > > > > > > >> > > > > > > > merge,
> > > > > > > > > > > > > > > >> > > > > > > > > > just like I did before,
> but
> > in
> > > > > > another
> > > > > > > > > > branch.
> > > > > > > > > > > > > > > >> > > > > > > > > >
> > > > > > > > > > > > > > > >> > > > > > > > > > Let me know if makes
> sense,
> > > ok?
> > > > > > > > > > > > > > > >> > > > > > > > > >
> > > > > > > > > > > > > > > >> > > > > > > > > > Taka
> > > > > > > > > > > > > > > >> > > > > > > > > >
> > > > > > > > > > > > > > > >> > > > > > > > > > Em ter, 16 de out de 2018
> às
> > > > > 14:00,
> > > > > > > > > Luciano
> > > > > > > > > > > > > Resende
> > > > > > > > > > > > > > <
> > > > > > > > > > > > > > > >> > > > > > > > > luckbr1975@gmail.com>
> > > > > > > > > > > > > > > >> > > > > > > > > > escreveu:
> > > > > > > > > > > > > > > >> > > > > > > > > >
> > > > > > > > > > > > > > > >> > > > > > > > > > > So, what is the POC, is
> > it a
> > > > > > > > refactoring
> > > > > > > > > > of
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > >> existing
> > > > > > > > > > > > > > > >> > > > repo?
> > > > > > > > > > > > > > > >> > > > > Or
> > > > > > > > > > > > > > > >> > > > > > > is
> > > > > > > > > > > > > > > >> > > > > > > > > > > it a new rewrite of the
> > > repo?
> > > > > > > > > > > > > > > >> > > > > > > > > > >
> > > > > > > > > > > > > > > >> > > > > > > > > > > Just asking as it might
> > make
> > > > > sense
> > > > > > > to
> > > > > > > > > make
> > > > > > > > > > > it
> > > > > > > > > > > > a
> > > > > > > > > > > > > > > branch
> > > > > > > > > > > > > > > >> > then
> > > > > > > > > > > > > > > >> > > > > > > actually
> > > > > > > > > > > > > > > >> > > > > > > > a
> > > > > > > > > > > > > > > >> > > > > > > > > > > thing in parallel, as
> this
> > > > will
> > > > > > have
> > > > > > > > an
> > > > > > > > > > > effect
> > > > > > > > > > > > > for
> > > > > > > > > > > > > > > >> > > releases,
> > > > > > > > > > > > > > > >> > > > > etc.
> > > > > > > > > > > > > > > >> > > > > > > but
> > > > > > > > > > > > > > > >> > > > > > > > > > > you guys know more here
> > > than I
> > > > > do.
> > > > > > > > > > > > > > > >> > > > > > > > > > >
> > > > > > > > > > > > > > > >> > > > > > > > > > > Also, it's probably good
> > to
> > > > > have a
> > > > > > > > write
> > > > > > > > > > up
> > > > > > > > > > > of
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > >> main
> > > > > > > > > > > > > > > >> > > > > direction
> > > > > > > > > > > > > > > >> > > > > > > of
> > > > > > > > > > > > > > > >> > > > > > > > > > > the design that can help
> > > > people
> > > > > > get
> > > > > > > > > > familiar
> > > > > > > > > > > > > with
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > >> new
> > > > > > > > > > > > > > > >> > > > > > approach.
> > > > > > > > > > > > > > > >> > > > > > > > > > > On Tue, Oct 16, 2018 at
> > > 11:12
> > > > AM
> > > > > > > Zhang
> > > > > > > > > > > Yifei <
> > > > > > > > > > > > > > > >> > > > > > > yifei.z.luiz@gmail.com
> > > > > > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > > > > > >> > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > >> > > > > > > > > > > >
> > > > > > > > > > > > > > > >> > > > > > > > > > > > Hey guys,
> > > > > > > > > > > > > > > >> > > > > > > > > > > >
> > > > > > > > > > > > > > > >> > > > > > > > > > > > We have reorganized
> the
> > > Poc
> > > > > > repo,
> > > > > > > > and
> > > > > > > > > > want
> > > > > > > > > > > > to
> > > > > > > > > > > > > > > merge
> > > > > > > > > > > > > > > >> it
> > > > > > > > > > > > > > > >> > to
> > > > > > > > > > > > > > > >> > > > our
> > > > > > > > > > > > > > > >> > > > > > > > Apache
> > > > > > > > > > > > > > > >> > > > > > > > > > > repo.
> > > > > > > > > > > > > > > >> > > > > > > > > > > > Just making sure here
> > > before
> > > > > we
> > > > > > do
> > > > > > > > the
> > > > > > > > > > > > merge,
> > > > > > > > > > > > > > > >> because
> > > > > > > > > > > > > > > >> > im
> > > > > > > > > > > > > > > >> > > > not
> > > > > > > > > > > > > > > >> > > > > > sure
> > > > > > > > > > > > > > > >> > > > > > > > if
> > > > > > > > > > > > > > > >> > > > > > > > > > > doing
> > > > > > > > > > > > > > > >> > > > > > > > > > > > this will perserve the
> > Git
> > > > > > commit
> > > > > > > > > > history.
> > > > > > > > > > > > > > > >> > > > > > > > > > > > What we are planning
> to
> > do
> > > > is:
> > > > > > > > > > > > > > > >> > > > > > > > > > > >
> > > > > > > > > > > > > > > >> > > > > > > > > > > > git filter-branch
> > > > > > > > > --subdirectory-filter
> > > > > > > > > > > <git
> > > > > > > > > > > > > > > >> > > > > origin_repository
> > > > > > > > > > > > > > > >> > > > > > > > > > directory>
> > > > > > > > > > > > > > > >> > > > > > > > > > > > -- --all
> > > > > > > > > > > > > > > >> > > > > > > > > > > > mkdir POCs/
> > > > > > > > > > > > > > > >> > > > > > > > > > > > git mv * POCs/
> > > > > > > > > > > > > > > >> > > > > > > > > > > > git commit -m
> "colleted
> > > the
> > > > > data
> > > > > > > to
> > > > > > > > > > move"
> > > > > > > > > > > > > > > >> > > > > > > > > > > >
> > > > > > > > > > > > > > > >> > > > > > > > > > > > git clone <git
> > > > > > > > > > destination_repository_url>
> > > > > > > > > > > > cd
> > > > > > > > > > > > > > <git
> > > > > > > > > > > > > > > >> > > > > > > > > > > >
> > > > > > destination_repository_directory>
> > > > > > > > git
> > > > > > > > > > > remote
> > > > > > > > > > > > > add
> > > > > > > > > > > > > > > >> > > > > > > > > > origin_repository_branch
> > > > > > > > > > > > > > > >> > > > > > > > > > > > <git origin_repository
> > > > > > directory>
> > > > > > > > git
> > > > > > > > > > pull
> > > > > > > > > > > > > > > >> > > > > > > origin_repository_branch
> > > > > > > > > > > > > > > >> > > > > > > > > > > master
> > > > > > > > > > > > > > > >> > > > > > > > > > > >
> > > --allow-unrelated-histories
> > > > > git
> > > > > > > > remote
> > > > > > > > > > rm
> > > > > > > > > > > > > > > >> > > > > > > origin_repository_branch
> > > > > > > > > > > > > > > >> > > > > > > > > > > >
> > > > > > > > > > > > > > > >> > > > > > > > > > > >
> > > > > > > > > > > > > > > >> > > > > > > > > > > > Thanks in advance !
> > > > > > > > > > > > > > > >> > > > > > > > > > > >
> > > > > > > > > > > > > > > >> > > > > > > > > > > >
> > > > > > > > > > > > > > > >> > > > >
> > > > > > > > > > > > >
> > > > > > --------------------------------------------------------------
> > > > > > > > > > > > > > > >> > > > > > > > > > > > Zhang Yifei
> > > > > > > > > > > > > > > >> > > > > > > > > > >
> > > > > > > > > > > > > > > >> > > > > > > > > > >
> > > > > > > > > > > > > > > >> > > > > > > > > > >
> > > > > > > > > > > > > > > >> > > > > > > > > > > --
> > > > > > > > > > > > > > > >> > > > > > > > > > > Luciano Resende
> > > > > > > > > > > > > > > >> > > > > > > > > > >
> > > > http://twitter.com/lresende1975
> > > > > > > > > > > > > > > >> > > > > > > > > > >
> > > http://lresende.blogspot.com/
> > > > > > > > > > > > > > > >> > > > > > > > > > >
> > > > > > > > > > > > > > > >> > > > > > > > > >
> > > > > > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > > > > > >> > > > > > > > > --
> > > > > > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > > > > > >> >
> > > > > > > > > >
> > > --------------------------------------------------------------
> > > > > > > > > > > > > > > >> > > > > > > > > Zhang Yifei
> > > > > > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > > > > > >> > > > > > > >
> > > > > > > > > > > > > > > >> > > > > > >
> > > > > > > > > > > > > > > >> > > > > >
> > > > > > > > > > > > > > > >> > > > >
> > > > > > > > > > > > > > > >> > > >
> > > > > > > > > > > > > > > >> > >
> > > > > > > > > > > > > > > >> >
> > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> >
> > --
> > --------------------------------------------------------------
> > Zhang Yifei
> >
>

Re: POCs, Prototypes, experiments ...

Posted by Rafael Novello <ra...@gmail.com>.

Guys,

How can we organize the tasks to start working at this new version?

I believe we had the core devs approval.

Atenciosamente,
Rafael J. R. Novello

Skype: rafael.novello
Blog: http://rafanovello.blogspot.com.br/


Em ter, 30 de out de 2018 às 14:30, Zhang Yifei <yi...@gmail.com>
escreveu:

> +1
>
> Em seg, 29 de out de 2018 às 18:23, Rafael Novello <
> rafa.reis.novello@gmail.com> escreveu:
>
> > Lucas, yes! It make sense!
> >
> > In the end, CLI and REPL will share a big part of code and they will be
> > just different interfaces to the same features.
> >
> > +1 for this new concept!
> >
> > Thanks for make it clear to me!
> > Atenciosamente,
> > Rafael J. R. Novello
> >
> > Skype: rafael.novello
> > Blog: http://rafanovello.blogspot.com.br/
> >
> >
> > Em seg, 29 de out de 2018 às 17:43, Lucas Bonatto Miguel <
> > lucasbm88@apache.org> escreveu:
> >
> > > That's a good point Rafael. I believe the main motivation for adding a
> > REPL
> > > to Marvin's toolbox is to enhance user experience.
> > >
> > > Imo we can't just switch CLI per REPL. I don't think one excludes the
> > > other, both have different use cases.
> > >
> > > We may be mixing too much stuff together, but in general, I think we
> are:
> > >  - Refactoring toolbox architecture to avoid unnecessarily duplicated
> > logic
> > > when adding support to new languages.
> > >  - Adding a new interface to enable users to open an interactive
> session
> > > with the engine (i.e. REPL)
> > >  - Making it easier to add support for new languages.
> > >
> > > Does it make sense for you?
> > >
> > > On Mon, Oct 29, 2018 at 11:22 AM Rafael Novello <
> > > rafa.reis.novello@gmail.com>
> > > wrote:
> > >
> > > > I do not want to be the annoying conversation but...
> > > >
> > > > If we will have an unique CLI that will be able to handle any kind of
> > > > engines (in any language) by using docker, Why we need a REPL?
> > > >
> > > > I think we can focus on creating this new architecture using docker
> and
> > > > docker SDK and use only a CLI for that.
> > > >
> > > > Atenciosamente,
> > > > Rafael J. R. Novello
> > > >
> > > > Skype: rafael.novello
> > > > Blog: http://rafanovello.blogspot.com.br/
> > > >
> > > >
> > > > Em seg, 29 de out de 2018 às 14:58, Lucas Bonatto Miguel <
> > > > lucasbm88@apache.org> escreveu:
> > > >
> > > > > No, I think the idea is to do not create different versions of the
> > CLI.
> > > > > Let's use one single tech stack for the CLI, ideally, it will be
> > shared
> > > > > with the REPL. The way I see the CLI would be very similar to the
> > REPL,
> > > > but
> > > > > instead of opening an interactive session the user could send
> > > > > unattended commands. As an analogy, you can think on spark-shell
> (our
> > > > REPL)
> > > > > and spark-submit (our CLI).
> > > > >
> > > > > - Lucas
> > > > >
> > > > > On Mon, Oct 29, 2018 at 10:27 AM Rafael Novello <
> > > > > rafa.reis.novello@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Hi Guys!
> > > > > >
> > > > > > Let me ask about this new version that include CLI's and REPL.
> Will
> > > we
> > > > > > create different versions of CLI?
> > > > > >
> > > > > > One of the reasons to create the REPL is to avoid re-implementing
> > the
> > > > > same
> > > > > > features many times,
> > > > > > each time for a different language.
> > > > > >
> > > > > > Regards!
> > > > > > Atenciosamente,
> > > > > > Rafael J. R. Novello
> > > > > >
> > > > > > Skype: rafael.novello
> > > > > > Blog: http://rafanovello.blogspot.com.br/
> > > > > >
> > > > > >
> > > > > > Em sex, 26 de out de 2018 às 20:54, Daniel Takabayashi <
> > > > > > daniel.takabayashi@gmail.com> escreveu:
> > > > > >
> > > > > > > +1
> > > > > > >
> > > > > > > Em sex, 26 de out de 2018 às 16:47, Lucas Bonatto Miguel <
> > > > > > > lucasbm88@apache.org> escreveu:
> > > > > > >
> > > > > > > > Makes perfect sense to me. I believe that if the majority of
> > > people
> > > > > > > agrees
> > > > > > > > with this architecture we could create issues and start
> > > > implementing
> > > > > > it.
> > > > > > > >
> > > > > > > > +1
> > > > > > > >
> > > > > > > >
> > > > > > > > On Fri, Oct 26, 2018 at 4:09 PM Daniel Takabayashi <
> > > > > > > > daniel.takabayashi@gmail.com> wrote:
> > > > > > > >
> > > > > > > > > I see... what about this new version?
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://docs.google.com/drawings/d/shajxIpLJHxxMbFgDXiPuhg/image?w=602&h=468&rev=1630&ac=1&parent=1ySERHGBXbHeyCMRookq5UfTuFkzzU0ugtjvR3rF3deY
> > > > > > > > >
> > > > > > > > > I changed the toolbox ...from 1 component to 3... docker
> > > client,
> > > > > cli
> > > > > > > and
> > > > > > > > > repl. Both (cli and repl) now are using the docker client
> to
> > > > > control
> > > > > > > the
> > > > > > > > > engines.
> > > > > > > > >
> > > > > > > > > Em sex, 26 de out de 2018 às 14:49, Lucas Bonatto Miguel <
> > > > > > > > > lucasbm88@apache.org> escreveu:
> > > > > > > > >
> > > > > > > > > > For sure, both suggestions are to make sure we're not
> > killing
> > > > the
> > > > > > CLI
> > > > > > > > and
> > > > > > > > > > that we have a component (the SDK client) that is shared
> by
> > > CLI
> > > > > and
> > > > > > > > REPL.
> > > > > > > > > >
> > > > > > > > > > Thanks for clarifying where the system calls will live.
> > > > > > > > > >
> > > > > > > > > > - Lucas
> > > > > > > > > >
> > > > > > > > > > On Fri, Oct 26, 2018 at 2:39 PM Daniel Takabayashi <
> > > > > > > > > > daniel.takabayashi@gmail.com> wrote:
> > > > > > > > > >
> > > > > > > > > > > Hi Lucas, could you please justified your two
> > suggestions?
> > > > > > > > > > >
> > > > > > > > > > > The general ideia is to use the Docker DSK to execute
> > local
> > > > > > > commands,
> > > > > > > > > to
> > > > > > > > > > > consume outputs and also to execute Marvin "specific
> > > > language"
> > > > > > > > > commands.
> > > > > > > > > > > Today almost all CLIs commands in Marvin is a mixed of
> > > > systems
> > > > > > > > commands
> > > > > > > > > > and
> > > > > > > > > > > library method calls. The plan is to put all "specific
> > > > language
> > > > > > > > calls"
> > > > > > > > > > > inside the marvin-common-libs and group the systems
> call
> > on
> > > > the
> > > > > > > > toolbox
> > > > > > > > > > > (using the docker SDK in this task).
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Em qui, 25 de out de 2018 às 20:09, Lucas Bonatto
> Miguel
> > <
> > > > > > > > > > > lucasbm88@apache.org> escreveu:
> > > > > > > > > > >
> > > > > > > > > > > > Great, thanks.
> > > > > > > > > > > >
> > > > > > > > > > > > So here some suggestions:
> > > > > > > > > > > >  1. We could create one new layer between Toolbox and
> > the
> > > > > > > Engines,
> > > > > > > > > this
> > > > > > > > > > > > would be Docker client.
> > > > > > > > > > > >  2. Split the Toolbox layer in two. Toolbox CLI and
> > > Toolbox
> > > > > > REPL
> > > > > > > > > > > >
> > > > > > > > > > > > It's not clear to me if the commands listed on
> General
> > > > > Commands
> > > > > > > > will
> > > > > > > > > > need
> > > > > > > > > > > > to be written for every language, from the image I
> > > believe
> > > > > not,
> > > > > > > but
> > > > > > > > > how
> > > > > > > > > > > it
> > > > > > > > > > > > would work, since starting a notebook in Python is
> > > > different
> > > > > > than
> > > > > > > > > > > starting
> > > > > > > > > > > > in Scala.
> > > > > > > > > > > >
> > > > > > > > > > > > - Lucas
> > > > > > > > > > > > On Thu, Oct 25, 2018 at 4:51 PM Daniel Takabayashi <
> > > > > > > > > > > > daniel.takabayashi@gmail.com> wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://docs.google.com/drawings/d/shajxIpLJHxxMbFgDXiPuhg/image?w=602&h=461&rev=1423&ac=1&parent=1ySERHGBXbHeyCMRookq5UfTuFkzzU0ugtjvR3rF3deY
> > > > > > > > > > > > >
> > > > > > > > > > > > > Em qui, 25 de out de 2018 às 07:38, Rafael Novello
> <
> > > > > > > > > > > > > rafa.reis.novello@gmail.com> escreveu:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Hi Taka,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > I can't see the image! Maybe the Apache email
> list
> > > > don't
> > > > > > > > support
> > > > > > > > > > > > > > attachments.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Best!
> > > > > > > > > > > > > > Rafael J. R. Novello
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Skype: rafael.novello
> > > > > > > > > > > > > > Blog: http://rafanovello.blogspot.com.br/
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Em qua, 24 de out de 2018 às 21:14, Daniel
> > > Takabayashi
> > > > <
> > > > > > > > > > > > > > daniel.takabayashi@gmail.com> escreveu:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Hi all,
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > To try to give a little more details about this
> > > "new
> > > > > > > concept"
> > > > > > > > > > that
> > > > > > > > > > > we
> > > > > > > > > > > > > > want
> > > > > > > > > > > > > > > to bring to the marvin toolbox, I did this
> simple
> > > > > > > > architecture
> > > > > > > > > > > draw.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > [image: marvin-architecture-views (5).png]
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > The general idea here is try to transform the
> > > toolbox
> > > > > > > > something
> > > > > > > > > > > > > > > disconnected with the "language", something
> more
> > > > > > agnostic.
> > > > > > > > Also
> > > > > > > > > > in
> > > > > > > > > > > > this
> > > > > > > > > > > > > > new
> > > > > > > > > > > > > > > architecture we could use remote resource to
> > > process
> > > > > > > engines
> > > > > > > > > and
> > > > > > > > > > > make
> > > > > > > > > > > > > > easy
> > > > > > > > > > > > > > > the support for new languages.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > This "new toolbox" will be the only thing that
> a
> > > > Marvin
> > > > > > > user
> > > > > > > > > must
> > > > > > > > > > > to
> > > > > > > > > > > > > > > install and also we could start to support
> > > multiples
> > > > > O.S
> > > > > > > > (once
> > > > > > > > > > the
> > > > > > > > > > > > REPL
> > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > a dummy application that only interprets and by
> > > pass
> > > > > > > > commands).
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Regards,
> > > > > > > > > > > > > > > Taka
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Em qua, 24 de out de 2018 às 09:52, Rafael
> > Novello
> > > <
> > > > > > > > > > > > > > > rafa.reis.novello@gmail.com> escreveu:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >> Alan,
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >> Yes! We are using the Docker SDK and it's
> > possible
> > > > to
> > > > > > use
> > > > > > > > the
> > > > > > > > > > API
> > > > > > > > > > > to
> > > > > > > > > > > > > > >> automate, but it's a little bit harder than
> > > automate
> > > > > CLI
> > > > > > > > > calls.
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >> Atenciosamente,
> > > > > > > > > > > > > > >> Rafael J. R. Novello
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >> Skype: rafael.novello
> > > > > > > > > > > > > > >> Blog: http://rafanovello.blogspot.com.br/
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >> Em qua, 24 de out de 2018 às 12:02, Alan
> Silva <
> > > > > > > > > > jumpi@apache.org>
> > > > > > > > > > > > > > >> escreveu:
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >> > Hi,
> > > > > > > > > > > > > > >> >
> > > > > > > > > > > > > > >> > One question here, I understand that we
> start
> > to
> > > > use
> > > > > > > with
> > > > > > > > > this
> > > > > > > > > > > PoC
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > >> > Docker SDK API, right?
> > > > > > > > > > > > > > >> >
> > > > > > > > > > > > > > >> > Why not use the API to expose some endpoints
> > to
> > > > > permit
> > > > > > > > this
> > > > > > > > > > kind
> > > > > > > > > > > > of
> > > > > > > > > > > > > > >> > automation by devops?
> > > > > > > > > > > > > > >> >
> > > > > > > > > > > > > > >> > I think it is possible and it solves the CLI
> > > > > problem,
> > > > > > > > right?
> > > > > > > > > > > > > > >> >
> > > > > > > > > > > > > > >> >
> > > > > > > > > > > > > > >> > On Tue, Oct 23, 2018 at 1:05 PM Rafael
> > Novello <
> > > > > > > > > > > > > > >> > rafa.reis.novello@gmail.com>
> > > > > > > > > > > > > > >> > wrote:
> > > > > > > > > > > > > > >> >
> > > > > > > > > > > > > > >> > > Lucas,
> > > > > > > > > > > > > > >> > >
> > > > > > > > > > > > > > >> > > Sorry, I didn't understood your question
> > > bellow.
> > > > > > > > > > > > > > >> > >
> > > > > > > > > > > > > > >> > > "Would it make sense to use the same
> > solution
> > > > that
> > > > > > we
> > > > > > > > will
> > > > > > > > > > use
> > > > > > > > > > > > for
> > > > > > > > > > > > > > >> > having a
> > > > > > > > > > > > > > >> > > single-language REPL to have a
> > single-language
> > > > > CLI?"
> > > > > > > > > > > > > > >> > >
> > > > > > > > > > > > > > >> > > For DevOps purposes, maybe this new
> toolbox
> > > > > concept
> > > > > > is
> > > > > > > > not
> > > > > > > > > > > > ideal.
> > > > > > > > > > > > > I
> > > > > > > > > > > > > > >> think
> > > > > > > > > > > > > > >> > > we can keep the CLI inside the docker
> > > container
> > > > > but
> > > > > > it
> > > > > > > > > will
> > > > > > > > > > > not
> > > > > > > > > > > > > easy
> > > > > > > > > > > > > > >> to
> > > > > > > > > > > > > > >> > > automate
> > > > > > > > > > > > > > >> > > jobs by this way.
> > > > > > > > > > > > > > >> > >
> > > > > > > > > > > > > > >> > > How to deal with this issue? Voting?
> > > > > > > > > > > > > > >> > > Atenciosamente,
> > > > > > > > > > > > > > >> > > Rafael J. R. Novello
> > > > > > > > > > > > > > >> > >
> > > > > > > > > > > > > > >> > > Skype: rafael.novello
> > > > > > > > > > > > > > >> > > Blog: http://rafanovello.blogspot.com.br/
> > > > > > > > > > > > > > >> > >
> > > > > > > > > > > > > > >> > >
> > > > > > > > > > > > > > >> > > Em sex, 19 de out de 2018 às 19:00, Lucas
> > > > Bonatto
> > > > > > > > Miguel <
> > > > > > > > > > > > > > >> > > lucasbm88@apache.org> escreveu:
> > > > > > > > > > > > > > >> > >
> > > > > > > > > > > > > > >> > > > Got it! Thanks for clarifying.
> > > > > > > > > > > > > > >> > > >
> > > > > > > > > > > > > > >> > > > Would it make sense to use the same
> > solution
> > > > > that
> > > > > > we
> > > > > > > > > will
> > > > > > > > > > > use
> > > > > > > > > > > > > for
> > > > > > > > > > > > > > >> > having
> > > > > > > > > > > > > > >> > > a
> > > > > > > > > > > > > > >> > > > single-language REPL to have a
> > > single-language
> > > > > > CLI?
> > > > > > > My
> > > > > > > > > > only
> > > > > > > > > > > > > > concern
> > > > > > > > > > > > > > >> > with
> > > > > > > > > > > > > > >> > > > killing the CLI is that you remove an
> > > > essential
> > > > > > > > feature
> > > > > > > > > > for
> > > > > > > > > > > > > > DevOps.
> > > > > > > > > > > > > > >> > > >
> > > > > > > > > > > > > > >> > > > - Lucas
> > > > > > > > > > > > > > >> > > >
> > > > > > > > > > > > > > >> > > > On Fri, Oct 19, 2018 at 1:52 PM Rafael
> > > > Novello <
> > > > > > > > > > > > > > >> > > > rafa.reis.novello@gmail.com>
> > > > > > > > > > > > > > >> > > > wrote:
> > > > > > > > > > > > > > >> > > >
> > > > > > > > > > > > > > >> > > > > Lucas,
> > > > > > > > > > > > > > >> > > > >
> > > > > > > > > > > > > > >> > > > > The idea is that REPL will substitute
> > the
> > > > > actual
> > > > > > > > CLI.
> > > > > > > > > > It's
> > > > > > > > > > > > > > because
> > > > > > > > > > > > > > >> > with
> > > > > > > > > > > > > > >> > > > the
> > > > > > > > > > > > > > >> > > > > actual concept (using language
> specific
> > > CLI)
> > > > > we
> > > > > > > will
> > > > > > > > > > need
> > > > > > > > > > > to
> > > > > > > > > > > > > > >> > > re-implement
> > > > > > > > > > > > > > >> > > > > the same features many times and
> > probably
> > > > each
> > > > > > CLI
> > > > > > > > > will
> > > > > > > > > > > > have a
> > > > > > > > > > > > > > >> > > different
> > > > > > > > > > > > > > >> > > > > behavior because some language
> specific
> > > > > > > restrictions
> > > > > > > > > > > and/or
> > > > > > > > > > > > > > >> > > limitations.
> > > > > > > > > > > > > > >> > > > >
> > > > > > > > > > > > > > >> > > > > With this new concept, all users will
> > have
> > > > the
> > > > > > > same
> > > > > > > > > > > > experience
> > > > > > > > > > > > > > >> > > > interacting
> > > > > > > > > > > > > > >> > > > > with Marvin REPL and they will use the
> > > bests
> > > > > > tools
> > > > > > > > to
> > > > > > > > > > > > develop
> > > > > > > > > > > > > > >> their
> > > > > > > > > > > > > > >> > > > engines
> > > > > > > > > > > > > > >> > > > > (Jupyter Notebook and/or Lab with
> > > different
> > > > > > > > languages
> > > > > > > > > > > > support
> > > > > > > > > > > > > or
> > > > > > > > > > > > > > >> even
> > > > > > > > > > > > > > >> > > > > Apache Zeppelin). All the interact
> will
> > > > occur
> > > > > > > > through
> > > > > > > > > > REPL
> > > > > > > > > > > > and
> > > > > > > > > > > > > > >> docker
> > > > > > > > > > > > > > >> > > > > protocol.
> > > > > > > > > > > > > > >> > > > >
> > > > > > > > > > > > > > >> > > > > Alan, Yes! As Lucas said, the concept
> is
> > > the
> > > > > > same
> > > > > > > > but
> > > > > > > > > we
> > > > > > > > > > > can
> > > > > > > > > > > > > use
> > > > > > > > > > > > > > >> > docker
> > > > > > > > > > > > > > >> > > > to
> > > > > > > > > > > > > > >> > > > > do the same job.
> > > > > > > > > > > > > > >> > > > >
> > > > > > > > > > > > > > >> > > > > Best!
> > > > > > > > > > > > > > >> > > > >
> > > > > > > > > > > > > > >> > > > > Em sex, 19 de out de 2018 às 00:39,
> > Lucas
> > > > > > Bonatto
> > > > > > > > > > Miguel <
> > > > > > > > > > > > > > >> > > > > lucasbm88@apache.org> escreveu:
> > > > > > > > > > > > > > >> > > > >
> > > > > > > > > > > > > > >> > > > > > Thanks for the clarifications
> Rafael,
> > so
> > > > > from
> > > > > > > > what I
> > > > > > > > > > > > > > understood,
> > > > > > > > > > > > > > >> > > > Marvin's
> > > > > > > > > > > > > > >> > > > > > developers would not use the REPL to
> > > > explore
> > > > > > > data
> > > > > > > > > and
> > > > > > > > > > > test
> > > > > > > > > > > > > > >> models,
> > > > > > > > > > > > > > >> > > i.e.
> > > > > > > > > > > > > > >> > > > > > develop the engine. Is the idea to
> > build
> > > > > > > something
> > > > > > > > > > more
> > > > > > > > > > > > like
> > > > > > > > > > > > > > an
> > > > > > > > > > > > > > >> > > > > interactive
> > > > > > > > > > > > > > >> > > > > > CLI? I think an interactive CLI
> would
> > be
> > > > > > useful
> > > > > > > > for
> > > > > > > > > > the
> > > > > > > > > > > > > > >> developer
> > > > > > > > > > > > > > >> > > > > > experience, however, it's important
> to
> > > > keep
> > > > > > the
> > > > > > > > > > > unattended
> > > > > > > > > > > > > > >> > (current)
> > > > > > > > > > > > > > >> > > > CLI
> > > > > > > > > > > > > > >> > > > > in
> > > > > > > > > > > > > > >> > > > > > place for scenarios where users want
> > to
> > > > > > automate
> > > > > > > > > > Marvin
> > > > > > > > > > > > > tasks.
> > > > > > > > > > > > > > >> > > > > >
> > > > > > > > > > > > > > >> > > > > > Alan, I believe the idea is still
> the
> > > same
> > > > > as
> > > > > > > you
> > > > > > > > > > > started,
> > > > > > > > > > > > > but
> > > > > > > > > > > > > > >> > using
> > > > > > > > > > > > > > >> > > > > > docker SDK now.
> > > > > > > > > > > > > > >> > > > > >
> > > > > > > > > > > > > > >> > > > > > - Lucas
> > > > > > > > > > > > > > >> > > > > >
> > > > > > > > > > > > > > >> > > > > > On Thu, Oct 18, 2018 at 1:29 PM
> Rafael
> > > > > > Novello <
> > > > > > > > > > > > > > >> > > > > > rafa.reis.novello@gmail.com>
> > > > > > > > > > > > > > >> > > > > > wrote:
> > > > > > > > > > > > > > >> > > > > >
> > > > > > > > > > > > > > >> > > > > > > Hi Lucas!
> > > > > > > > > > > > > > >> > > > > > >
> > > > > > > > > > > > > > >> > > > > > > First of all +1 for REPL POCs to
> > > Apache!
> > > > > > > > > > > > > > >> > > > > > >
> > > > > > > > > > > > > > >> > > > > > > Let me help with some comments:
> > > > > > > > > > > > > > >> > > > > > >
> > > > > > > > > > > > > > >> > > > > > > 1 - We have tested NodeJS, Scala
> and
> > > > > Python
> > > > > > > and
> > > > > > > > > the
> > > > > > > > > > > > > easiest
> > > > > > > > > > > > > > >> one
> > > > > > > > > > > > > > >> > was
> > > > > > > > > > > > > > >> > > > > > Python.
> > > > > > > > > > > > > > >> > > > > > > We have found a small project [1]
> > that
> > > > > have
> > > > > > > all
> > > > > > > > > > > features
> > > > > > > > > > > > > we
> > > > > > > > > > > > > > >> > desired
> > > > > > > > > > > > > > >> > > > for
> > > > > > > > > > > > > > >> > > > > > > REPL:
> > > > > > > > > > > > > > >> > > > > > > - Autocomplete
> > > > > > > > > > > > > > >> > > > > > > - Python commands disabled (the
> user
> > > > have
> > > > > > only
> > > > > > > > the
> > > > > > > > > > > > > commands
> > > > > > > > > > > > > > we
> > > > > > > > > > > > > > >> > > > > provide).
> > > > > > > > > > > > > > >> > > > > > > The other languages REPL options
> > don't
> > > > > have
> > > > > > > this
> > > > > > > > > > > > feature.
> > > > > > > > > > > > > > >> > > > > > > - Easy to show output
> > > > > > > > > > > > > > >> > > > > > > - Etc
> > > > > > > > > > > > > > >> > > > > > > So, I think the language chosen
> will
> > > not
> > > > > be
> > > > > > > > > > important
> > > > > > > > > > > > here
> > > > > > > > > > > > > > >> > because
> > > > > > > > > > > > > > >> > > > the
> > > > > > > > > > > > > > >> > > > > > user
> > > > > > > > > > > > > > >> > > > > > > will only interact with the
> commands
> > > > that
> > > > > we
> > > > > > > > > create.
> > > > > > > > > > > > > > >> > > > > > >
> > > > > > > > > > > > > > >> > > > > > > 2 - The "engine-generate" command
> > will
> > > > > > > download
> > > > > > > > a
> > > > > > > > > > > docker
> > > > > > > > > > > > > > image
> > > > > > > > > > > > > > >> > that
> > > > > > > > > > > > > > >> > > > we
> > > > > > > > > > > > > > >> > > > > > > create for that language and
> start a
> > > > > > container
> > > > > > > > > with
> > > > > > > > > > > the
> > > > > > > > > > > > > > basic
> > > > > > > > > > > > > > >> > > project
> > > > > > > > > > > > > > >> > > > > > > structure for that language.
> > > > > > > > > > > > > > >> > > > > > >
> > > > > > > > > > > > > > >> > > > > > > 3 - The REPL client will use the
> > > "docker
> > > > > > > > protocol"
> > > > > > > > > > to
> > > > > > > > > > > > run
> > > > > > > > > > > > > > >> > command,
> > > > > > > > > > > > > > >> > > > > > > start/stop services and etc inside
> > the
> > > > > > > container
> > > > > > > > > and
> > > > > > > > > > > it
> > > > > > > > > > > > > will
> > > > > > > > > > > > > > >> > > receive
> > > > > > > > > > > > > > >> > > > > the
> > > > > > > > > > > > > > >> > > > > > > log stream to show. No, the REPL
> > will
> > > no
> > > > > > pass
> > > > > > > > code
> > > > > > > > > > > > > snippets
> > > > > > > > > > > > > > >> for
> > > > > > > > > > > > > > >> > > > docker
> > > > > > > > > > > > > > >> > > > > > > container (I think it will not be
> > > > > necessary)
> > > > > > > > > > > > > > >> > > > > > >
> > > > > > > > > > > > > > >> > > > > > > 4 - Yep! Like I said on the first
> > > item.
> > > > > > > > > > > > > > >> > > > > > >
> > > > > > > > > > > > > > >> > > > > > > [1] -
> > > > > https://github.com/italorossi/ishell
> > > > > > > > > > > > > > >> > > > > > >
> > > > > > > > > > > > > > >> > > > > > > Let me know if there is any other
> > > > > question!
> > > > > > > > > > > > > > >> > > > > > > Best!
> > > > > > > > > > > > > > >> > > > > > > Rafael J. R. Novello
> > > > > > > > > > > > > > >> > > > > > >
> > > > > > > > > > > > > > >> > > > > > > Skype: rafael.novello
> > > > > > > > > > > > > > >> > > > > > > Blog:
> > > > http://rafanovello.blogspot.com.br/
> > > > > > > > > > > > > > >> > > > > > >
> > > > > > > > > > > > > > >> > > > > > >
> > > > > > > > > > > > > > >> > > > > > > Em qui, 18 de out de 2018 às
> 17:00,
> > > > Lucas
> > > > > > > > Bonatto
> > > > > > > > > > > > Miguel <
> > > > > > > > > > > > > > >> > > > > > > lucasbm88@apache.org> escreveu:
> > > > > > > > > > > > > > >> > > > > > >
> > > > > > > > > > > > > > >> > > > > > > > + 1 for migrating the REPL repo
> to
> > > > > Apache
> > > > > > > > > > > > > > >> > > > > > > >
> > > > > > > > > > > > > > >> > > > > > > > I have a few questions about the
> > > > > previous
> > > > > > > > > > > explanation:
> > > > > > > > > > > > > > >> > > > > > > >  1) The REPL itself, it would be
> > an
> > > > > > > > application
> > > > > > > > > in
> > > > > > > > > > > > which
> > > > > > > > > > > > > > >> > > language?
> > > > > > > > > > > > > > >> > > > > > > Remember
> > > > > > > > > > > > > > >> > > > > > > > that the main idea is to allow
> the
> > > > user
> > > > > to
> > > > > > > > > program
> > > > > > > > > > > on
> > > > > > > > > > > > > his
> > > > > > > > > > > > > > >> > > preferred
> > > > > > > > > > > > > > >> > > > > > > > language in the REPL.
> > > > > > > > > > > > > > >> > > > > > > >  2) Should the engine-generate
> > > command
> > > > > > also
> > > > > > > > > > > generate a
> > > > > > > > > > > > > > >> docker
> > > > > > > > > > > > > > >> > > image
> > > > > > > > > > > > > > >> > > > > > with
> > > > > > > > > > > > > > >> > > > > > > > the user's application?
> > > > > > > > > > > > > > >> > > > > > > >  3) What type of communication
> > would
> > > > > > happen
> > > > > > > > > > between
> > > > > > > > > > > > the
> > > > > > > > > > > > > > REPL
> > > > > > > > > > > > > > >> > and
> > > > > > > > > > > > > > >> > > > the
> > > > > > > > > > > > > > >> > > > > > > engine
> > > > > > > > > > > > > > >> > > > > > > > via Docker SDK? Would the REPL
> > pass
> > > > > > snippets
> > > > > > > > of
> > > > > > > > > > code
> > > > > > > > > > > > to
> > > > > > > > > > > > > be
> > > > > > > > > > > > > > >> > > executed
> > > > > > > > > > > > > > >> > > > > by
> > > > > > > > > > > > > > >> > > > > > > the
> > > > > > > > > > > > > > >> > > > > > > > docker container?
> > > > > > > > > > > > > > >> > > > > > > >  4) Have you considered code
> > > > completion
> > > > > in
> > > > > > > the
> > > > > > > > > > REPL?
> > > > > > > > > > > > > > >> > > > > > > >
> > > > > > > > > > > > > > >> > > > > > > >
> > > > > > > > > > > > > > >> > > > > > > > On Thu, Oct 18, 2018 at 7:53 AM
> > > Zhang
> > > > > > Yifei
> > > > > > > <
> > > > > > > > > > > > > > >> > > > yifei.z.luiz@gmail.com>
> > > > > > > > > > > > > > >> > > > > > > > wrote:
> > > > > > > > > > > > > > >> > > > > > > >
> > > > > > > > > > > > > > >> > > > > > > > > Ok guys,
> > > > > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > > > > >> > > > > > > > > The basic idea is to provide
> > only
> > > > one
> > > > > > > > Toolbox
> > > > > > > > > > for
> > > > > > > > > > > > > > multiple
> > > > > > > > > > > > > > >> > > > > languages.
> > > > > > > > > > > > > > >> > > > > > > > > We are looking for possibility
> > to
> > > > > build
> > > > > > a
> > > > > > > > > single
> > > > > > > > > > > > > Marvin
> > > > > > > > > > > > > > >> Repl,
> > > > > > > > > > > > > > >> > > > > > > > > instead of severals toolboxes
> > with
> > > > > > > > differentes
> > > > > > > > > > > > > > interfaces
> > > > > > > > > > > > > > >> or
> > > > > > > > > > > > > > >> > > > > > commands.
> > > > > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > > > > >> > > > > > > > > In this case, the
> > engine-generate
> > > > > > command
> > > > > > > > will
> > > > > > > > > > > > > download
> > > > > > > > > > > > > > >> and
> > > > > > > > > > > > > > >> > > > start a
> > > > > > > > > > > > > > >> > > > > > > > Docker
> > > > > > > > > > > > > > >> > > > > > > > > container with basic engine
> > > > structure
> > > > > > > > > > > corresponding
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > >> > choosed
> > > > > > > > > > > > > > >> > > > > > > language.
> > > > > > > > > > > > > > >> > > > > > > > > this means we don't need to
> > build
> > > > > > > Toolboxes
> > > > > > > > of
> > > > > > > > > > > > > > differents
> > > > > > > > > > > > > > >> > > > > languages,
> > > > > > > > > > > > > > >> > > > > > we
> > > > > > > > > > > > > > >> > > > > > > > > just
> > > > > > > > > > > > > > >> > > > > > > > > build the engine template of
> all
> > > > > > languages
> > > > > > > > > that
> > > > > > > > > > we
> > > > > > > > > > > > > want
> > > > > > > > > > > > > > to
> > > > > > > > > > > > > > >> > > > support
> > > > > > > > > > > > > > >> > > > > > and
> > > > > > > > > > > > > > >> > > > > > > > > provide it
> > > > > > > > > > > > > > >> > > > > > > > > as Docker containers
> > > > > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > > > > >> > > > > > > > > We have started researches
> > around
> > > > our
> > > > > > > basic
> > > > > > > > > > > > > requirements
> > > > > > > > > > > > > > >> > like:
> > > > > > > > > > > > > > >> > > > > > > > > - Repl interface
> > > > > > > > > > > > > > >> > > > > > > > > - System communication
> > > > > > > > > > > > > > >> > > > > > > > > - Connection security
> > > > > > > > > > > > > > >> > > > > > > > > - Tool popularity
> > > > > > > > > > > > > > >> > > > > > > > > - Update complexity
> > > > > > > > > > > > > > >> > > > > > > > > - Languages support
> > > > > > > > > > > > > > >> > > > > > > > > - ......
> > > > > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > > > > >> > > > > > > > > And we did some POC with code
> > > here:
> > > > > > > > > > > > > > >> > > > > > > > >
> > > > > > https://github.com/marvin-ai/marvin-repl
> > > > > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > > > > >> > > > > > > > > There is POC testing gRPC
> using
> > > > Scala
> > > > > > and
> > > > > > > > > > Python,
> > > > > > > > > > > > > > >> > > > > > > > > Repl inteface and Docker SDK
> > with
> > > > > > NodeJS,
> > > > > > > > > > > > > > >> > > > > > > > > Repl interface and Docker SDK
> > with
> > > > > > Python.
> > > > > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > > > > >> > > > > > > > > At this moment we prefer the
> > Repl
> > > > > > > interface
> > > > > > > > +
> > > > > > > > > > > Docker
> > > > > > > > > > > > > SDK
> > > > > > > > > > > > > > >> way,
> > > > > > > > > > > > > > >> > > > > because
> > > > > > > > > > > > > > >> > > > > > > > good
> > > > > > > > > > > > > > >> > > > > > > > > part of the requirements
> > > > > > > > > > > > > > >> > > > > > > > > will be guaranteed by Docker.
> > > > > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > > > > >> > > > > > > > > With this informations, what
> do
> > > you
> > > > > > think?
> > > > > > > > > > Should
> > > > > > > > > > > we
> > > > > > > > > > > > > > >> submit
> > > > > > > > > > > > > > >> > all
> > > > > > > > > > > > > > >> > > > > this
> > > > > > > > > > > > > > >> > > > > > > POCs
> > > > > > > > > > > > > > >> > > > > > > > > to Apache Repo?
> > > > > > > > > > > > > > >> > > > > > > > > Please feel free to opine.
> > > > > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > > > > >> > > > > > > > > Thats all, thanks!!!
> > > > > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > > > > >> > > > > > > > > Em ter, 16 de out de 2018 às
> > > 18:55,
> > > > > > Daniel
> > > > > > > > > > > > > Takabayashi <
> > > > > > > > > > > > > > >> > > > > > > > > daniel.takabayashi@gmail.com>
> > > > > escreveu:
> > > > > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > > > > >> > > > > > > > > > Zhang,
> > > > > > > > > > > > > > >> > > > > > > > > >
> > > > > > > > > > > > > > >> > > > > > > > > > I think the best approach is
> > > give
> > > > > us a
> > > > > > > > > better
> > > > > > > > > > > > > > >> explanation
> > > > > > > > > > > > > > >> > > about
> > > > > > > > > > > > > > >> > > > > > this
> > > > > > > > > > > > > > >> > > > > > > > new
> > > > > > > > > > > > > > >> > > > > > > > > > feature and how this can
> help
> > us
> > > > to
> > > > > > > > archive
> > > > > > > > > > the
> > > > > > > > > > > > > > desired
> > > > > > > > > > > > > > >> > > feature
> > > > > > > > > > > > > > >> > > > > > > > (support
> > > > > > > > > > > > > > >> > > > > > > > > > multiple languages). Than if
> > the
> > > > > > > majority
> > > > > > > > > > agree
> > > > > > > > > > > > > than I
> > > > > > > > > > > > > > >> can
> > > > > > > > > > > > > > >> > do
> > > > > > > > > > > > > > >> > > > the
> > > > > > > > > > > > > > >> > > > > > > > merge,
> > > > > > > > > > > > > > >> > > > > > > > > > just like I did before, but
> in
> > > > > another
> > > > > > > > > branch.
> > > > > > > > > > > > > > >> > > > > > > > > >
> > > > > > > > > > > > > > >> > > > > > > > > > Let me know if makes sense,
> > ok?
> > > > > > > > > > > > > > >> > > > > > > > > >
> > > > > > > > > > > > > > >> > > > > > > > > > Taka
> > > > > > > > > > > > > > >> > > > > > > > > >
> > > > > > > > > > > > > > >> > > > > > > > > > Em ter, 16 de out de 2018 às
> > > > 14:00,
> > > > > > > > Luciano
> > > > > > > > > > > > Resende
> > > > > > > > > > > > > <
> > > > > > > > > > > > > > >> > > > > > > > > luckbr1975@gmail.com>
> > > > > > > > > > > > > > >> > > > > > > > > > escreveu:
> > > > > > > > > > > > > > >> > > > > > > > > >
> > > > > > > > > > > > > > >> > > > > > > > > > > So, what is the POC, is
> it a
> > > > > > > refactoring
> > > > > > > > > of
> > > > > > > > > > > the
> > > > > > > > > > > > > > >> existing
> > > > > > > > > > > > > > >> > > > repo?
> > > > > > > > > > > > > > >> > > > > Or
> > > > > > > > > > > > > > >> > > > > > > is
> > > > > > > > > > > > > > >> > > > > > > > > > > it a new rewrite of the
> > repo?
> > > > > > > > > > > > > > >> > > > > > > > > > >
> > > > > > > > > > > > > > >> > > > > > > > > > > Just asking as it might
> make
> > > > sense
> > > > > > to
> > > > > > > > make
> > > > > > > > > > it
> > > > > > > > > > > a
> > > > > > > > > > > > > > branch
> > > > > > > > > > > > > > >> > then
> > > > > > > > > > > > > > >> > > > > > > actually
> > > > > > > > > > > > > > >> > > > > > > > a
> > > > > > > > > > > > > > >> > > > > > > > > > > thing in parallel, as this
> > > will
> > > > > have
> > > > > > > an
> > > > > > > > > > effect
> > > > > > > > > > > > for
> > > > > > > > > > > > > > >> > > releases,
> > > > > > > > > > > > > > >> > > > > etc.
> > > > > > > > > > > > > > >> > > > > > > but
> > > > > > > > > > > > > > >> > > > > > > > > > > you guys know more here
> > than I
> > > > do.
> > > > > > > > > > > > > > >> > > > > > > > > > >
> > > > > > > > > > > > > > >> > > > > > > > > > > Also, it's probably good
> to
> > > > have a
> > > > > > > write
> > > > > > > > > up
> > > > > > > > > > of
> > > > > > > > > > > > the
> > > > > > > > > > > > > > >> main
> > > > > > > > > > > > > > >> > > > > direction
> > > > > > > > > > > > > > >> > > > > > > of
> > > > > > > > > > > > > > >> > > > > > > > > > > the design that can help
> > > people
> > > > > get
> > > > > > > > > familiar
> > > > > > > > > > > > with
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > >> new
> > > > > > > > > > > > > > >> > > > > > approach.
> > > > > > > > > > > > > > >> > > > > > > > > > > On Tue, Oct 16, 2018 at
> > 11:12
> > > AM
> > > > > > Zhang
> > > > > > > > > > Yifei <
> > > > > > > > > > > > > > >> > > > > > > yifei.z.luiz@gmail.com
> > > > > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > > > > >> > > > > > > > > > > wrote:
> > > > > > > > > > > > > > >> > > > > > > > > > > >
> > > > > > > > > > > > > > >> > > > > > > > > > > > Hey guys,
> > > > > > > > > > > > > > >> > > > > > > > > > > >
> > > > > > > > > > > > > > >> > > > > > > > > > > > We have reorganized the
> > Poc
> > > > > repo,
> > > > > > > and
> > > > > > > > > want
> > > > > > > > > > > to
> > > > > > > > > > > > > > merge
> > > > > > > > > > > > > > >> it
> > > > > > > > > > > > > > >> > to
> > > > > > > > > > > > > > >> > > > our
> > > > > > > > > > > > > > >> > > > > > > > Apache
> > > > > > > > > > > > > > >> > > > > > > > > > > repo.
> > > > > > > > > > > > > > >> > > > > > > > > > > > Just making sure here
> > before
> > > > we
> > > > > do
> > > > > > > the
> > > > > > > > > > > merge,
> > > > > > > > > > > > > > >> because
> > > > > > > > > > > > > > >> > im
> > > > > > > > > > > > > > >> > > > not
> > > > > > > > > > > > > > >> > > > > > sure
> > > > > > > > > > > > > > >> > > > > > > > if
> > > > > > > > > > > > > > >> > > > > > > > > > > doing
> > > > > > > > > > > > > > >> > > > > > > > > > > > this will perserve the
> Git
> > > > > commit
> > > > > > > > > history.
> > > > > > > > > > > > > > >> > > > > > > > > > > > What we are planning to
> do
> > > is:
> > > > > > > > > > > > > > >> > > > > > > > > > > >
> > > > > > > > > > > > > > >> > > > > > > > > > > > git filter-branch
> > > > > > > > --subdirectory-filter
> > > > > > > > > > <git
> > > > > > > > > > > > > > >> > > > > origin_repository
> > > > > > > > > > > > > > >> > > > > > > > > > directory>
> > > > > > > > > > > > > > >> > > > > > > > > > > > -- --all
> > > > > > > > > > > > > > >> > > > > > > > > > > > mkdir POCs/
> > > > > > > > > > > > > > >> > > > > > > > > > > > git mv * POCs/
> > > > > > > > > > > > > > >> > > > > > > > > > > > git commit -m "colleted
> > the
> > > > data
> > > > > > to
> > > > > > > > > move"
> > > > > > > > > > > > > > >> > > > > > > > > > > >
> > > > > > > > > > > > > > >> > > > > > > > > > > > git clone <git
> > > > > > > > > destination_repository_url>
> > > > > > > > > > > cd
> > > > > > > > > > > > > <git
> > > > > > > > > > > > > > >> > > > > > > > > > > >
> > > > > destination_repository_directory>
> > > > > > > git
> > > > > > > > > > remote
> > > > > > > > > > > > add
> > > > > > > > > > > > > > >> > > > > > > > > > origin_repository_branch
> > > > > > > > > > > > > > >> > > > > > > > > > > > <git origin_repository
> > > > > directory>
> > > > > > > git
> > > > > > > > > pull
> > > > > > > > > > > > > > >> > > > > > > origin_repository_branch
> > > > > > > > > > > > > > >> > > > > > > > > > > master
> > > > > > > > > > > > > > >> > > > > > > > > > > >
> > --allow-unrelated-histories
> > > > git
> > > > > > > remote
> > > > > > > > > rm
> > > > > > > > > > > > > > >> > > > > > > origin_repository_branch
> > > > > > > > > > > > > > >> > > > > > > > > > > >
> > > > > > > > > > > > > > >> > > > > > > > > > > >
> > > > > > > > > > > > > > >> > > > > > > > > > > > Thanks in advance !
> > > > > > > > > > > > > > >> > > > > > > > > > > >
> > > > > > > > > > > > > > >> > > > > > > > > > > >
> > > > > > > > > > > > > > >> > > > >
> > > > > > > > > > > >
> > > > > --------------------------------------------------------------
> > > > > > > > > > > > > > >> > > > > > > > > > > > Zhang Yifei
> > > > > > > > > > > > > > >> > > > > > > > > > >
> > > > > > > > > > > > > > >> > > > > > > > > > >
> > > > > > > > > > > > > > >> > > > > > > > > > >
> > > > > > > > > > > > > > >> > > > > > > > > > > --
> > > > > > > > > > > > > > >> > > > > > > > > > > Luciano Resende
> > > > > > > > > > > > > > >> > > > > > > > > > >
> > > http://twitter.com/lresende1975
> > > > > > > > > > > > > > >> > > > > > > > > > >
> > http://lresende.blogspot.com/
> > > > > > > > > > > > > > >> > > > > > > > > > >
> > > > > > > > > > > > > > >> > > > > > > > > >
> > > > > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > > > > >> > > > > > > > > --
> > > > > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > > > > >> >
> > > > > > > > >
> > --------------------------------------------------------------
> > > > > > > > > > > > > > >> > > > > > > > > Zhang Yifei
> > > > > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > > > > >> > > > > > > >
> > > > > > > > > > > > > > >> > > > > > >
> > > > > > > > > > > > > > >> > > > > >
> > > > > > > > > > > > > > >> > > > >
> > > > > > > > > > > > > > >> > > >
> > > > > > > > > > > > > > >> > >
> > > > > > > > > > > > > > >> >
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
>
> --
> --------------------------------------------------------------
> Zhang Yifei
>

Re: POCs, Prototypes, experiments ...

Posted by Zhang Yifei <yi...@gmail.com>.

+1

Em seg, 29 de out de 2018 às 18:23, Rafael Novello <
rafa.reis.novello@gmail.com> escreveu:

> Lucas, yes! It make sense!
>
> In the end, CLI and REPL will share a big part of code and they will be
> just different interfaces to the same features.
>
> +1 for this new concept!
>
> Thanks for make it clear to me!
> Atenciosamente,
> Rafael J. R. Novello
>
> Skype: rafael.novello
> Blog: http://rafanovello.blogspot.com.br/
>
>
> Em seg, 29 de out de 2018 às 17:43, Lucas Bonatto Miguel <
> lucasbm88@apache.org> escreveu:
>
> > That's a good point Rafael. I believe the main motivation for adding a
> REPL
> > to Marvin's toolbox is to enhance user experience.
> >
> > Imo we can't just switch CLI per REPL. I don't think one excludes the
> > other, both have different use cases.
> >
> > We may be mixing too much stuff together, but in general, I think we are:
> >  - Refactoring toolbox architecture to avoid unnecessarily duplicated
> logic
> > when adding support to new languages.
> >  - Adding a new interface to enable users to open an interactive session
> > with the engine (i.e. REPL)
> >  - Making it easier to add support for new languages.
> >
> > Does it make sense for you?
> >
> > On Mon, Oct 29, 2018 at 11:22 AM Rafael Novello <
> > rafa.reis.novello@gmail.com>
> > wrote:
> >
> > > I do not want to be the annoying conversation but...
> > >
> > > If we will have an unique CLI that will be able to handle any kind of
> > > engines (in any language) by using docker, Why we need a REPL?
> > >
> > > I think we can focus on creating this new architecture using docker and
> > > docker SDK and use only a CLI for that.
> > >
> > > Atenciosamente,
> > > Rafael J. R. Novello
> > >
> > > Skype: rafael.novello
> > > Blog: http://rafanovello.blogspot.com.br/
> > >
> > >
> > > Em seg, 29 de out de 2018 às 14:58, Lucas Bonatto Miguel <
> > > lucasbm88@apache.org> escreveu:
> > >
> > > > No, I think the idea is to do not create different versions of the
> CLI.
> > > > Let's use one single tech stack for the CLI, ideally, it will be
> shared
> > > > with the REPL. The way I see the CLI would be very similar to the
> REPL,
> > > but
> > > > instead of opening an interactive session the user could send
> > > > unattended commands. As an analogy, you can think on spark-shell (our
> > > REPL)
> > > > and spark-submit (our CLI).
> > > >
> > > > - Lucas
> > > >
> > > > On Mon, Oct 29, 2018 at 10:27 AM Rafael Novello <
> > > > rafa.reis.novello@gmail.com>
> > > > wrote:
> > > >
> > > > > Hi Guys!
> > > > >
> > > > > Let me ask about this new version that include CLI's and REPL. Will
> > we
> > > > > create different versions of CLI?
> > > > >
> > > > > One of the reasons to create the REPL is to avoid re-implementing
> the
> > > > same
> > > > > features many times,
> > > > > each time for a different language.
> > > > >
> > > > > Regards!
> > > > > Atenciosamente,
> > > > > Rafael J. R. Novello
> > > > >
> > > > > Skype: rafael.novello
> > > > > Blog: http://rafanovello.blogspot.com.br/
> > > > >
> > > > >
> > > > > Em sex, 26 de out de 2018 às 20:54, Daniel Takabayashi <
> > > > > daniel.takabayashi@gmail.com> escreveu:
> > > > >
> > > > > > +1
> > > > > >
> > > > > > Em sex, 26 de out de 2018 às 16:47, Lucas Bonatto Miguel <
> > > > > > lucasbm88@apache.org> escreveu:
> > > > > >
> > > > > > > Makes perfect sense to me. I believe that if the majority of
> > people
> > > > > > agrees
> > > > > > > with this architecture we could create issues and start
> > > implementing
> > > > > it.
> > > > > > >
> > > > > > > +1
> > > > > > >
> > > > > > >
> > > > > > > On Fri, Oct 26, 2018 at 4:09 PM Daniel Takabayashi <
> > > > > > > daniel.takabayashi@gmail.com> wrote:
> > > > > > >
> > > > > > > > I see... what about this new version?
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://docs.google.com/drawings/d/shajxIpLJHxxMbFgDXiPuhg/image?w=602&h=468&rev=1630&ac=1&parent=1ySERHGBXbHeyCMRookq5UfTuFkzzU0ugtjvR3rF3deY
> > > > > > > >
> > > > > > > > I changed the toolbox ...from 1 component to 3... docker
> > client,
> > > > cli
> > > > > > and
> > > > > > > > repl. Both (cli and repl) now are using the docker client to
> > > > control
> > > > > > the
> > > > > > > > engines.
> > > > > > > >
> > > > > > > > Em sex, 26 de out de 2018 às 14:49, Lucas Bonatto Miguel <
> > > > > > > > lucasbm88@apache.org> escreveu:
> > > > > > > >
> > > > > > > > > For sure, both suggestions are to make sure we're not
> killing
> > > the
> > > > > CLI
> > > > > > > and
> > > > > > > > > that we have a component (the SDK client) that is shared by
> > CLI
> > > > and
> > > > > > > REPL.
> > > > > > > > >
> > > > > > > > > Thanks for clarifying where the system calls will live.
> > > > > > > > >
> > > > > > > > > - Lucas
> > > > > > > > >
> > > > > > > > > On Fri, Oct 26, 2018 at 2:39 PM Daniel Takabayashi <
> > > > > > > > > daniel.takabayashi@gmail.com> wrote:
> > > > > > > > >
> > > > > > > > > > Hi Lucas, could you please justified your two
> suggestions?
> > > > > > > > > >
> > > > > > > > > > The general ideia is to use the Docker DSK to execute
> local
> > > > > > commands,
> > > > > > > > to
> > > > > > > > > > consume outputs and also to execute Marvin "specific
> > > language"
> > > > > > > > commands.
> > > > > > > > > > Today almost all CLIs commands in Marvin is a mixed of
> > > systems
> > > > > > > commands
> > > > > > > > > and
> > > > > > > > > > library method calls. The plan is to put all "specific
> > > language
> > > > > > > calls"
> > > > > > > > > > inside the marvin-common-libs and group the systems call
> on
> > > the
> > > > > > > toolbox
> > > > > > > > > > (using the docker SDK in this task).
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Em qui, 25 de out de 2018 às 20:09, Lucas Bonatto Miguel
> <
> > > > > > > > > > lucasbm88@apache.org> escreveu:
> > > > > > > > > >
> > > > > > > > > > > Great, thanks.
> > > > > > > > > > >
> > > > > > > > > > > So here some suggestions:
> > > > > > > > > > >  1. We could create one new layer between Toolbox and
> the
> > > > > > Engines,
> > > > > > > > this
> > > > > > > > > > > would be Docker client.
> > > > > > > > > > >  2. Split the Toolbox layer in two. Toolbox CLI and
> > Toolbox
> > > > > REPL
> > > > > > > > > > >
> > > > > > > > > > > It's not clear to me if the commands listed on General
> > > > Commands
> > > > > > > will
> > > > > > > > > need
> > > > > > > > > > > to be written for every language, from the image I
> > believe
> > > > not,
> > > > > > but
> > > > > > > > how
> > > > > > > > > > it
> > > > > > > > > > > would work, since starting a notebook in Python is
> > > different
> > > > > than
> > > > > > > > > > starting
> > > > > > > > > > > in Scala.
> > > > > > > > > > >
> > > > > > > > > > > - Lucas
> > > > > > > > > > > On Thu, Oct 25, 2018 at 4:51 PM Daniel Takabayashi <
> > > > > > > > > > > daniel.takabayashi@gmail.com> wrote:
> > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://docs.google.com/drawings/d/shajxIpLJHxxMbFgDXiPuhg/image?w=602&h=461&rev=1423&ac=1&parent=1ySERHGBXbHeyCMRookq5UfTuFkzzU0ugtjvR3rF3deY
> > > > > > > > > > > >
> > > > > > > > > > > > Em qui, 25 de out de 2018 às 07:38, Rafael Novello <
> > > > > > > > > > > > rafa.reis.novello@gmail.com> escreveu:
> > > > > > > > > > > >
> > > > > > > > > > > > > Hi Taka,
> > > > > > > > > > > > >
> > > > > > > > > > > > > I can't see the image! Maybe the Apache email list
> > > don't
> > > > > > > support
> > > > > > > > > > > > > attachments.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Best!
> > > > > > > > > > > > > Rafael J. R. Novello
> > > > > > > > > > > > >
> > > > > > > > > > > > > Skype: rafael.novello
> > > > > > > > > > > > > Blog: http://rafanovello.blogspot.com.br/
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > Em qua, 24 de out de 2018 às 21:14, Daniel
> > Takabayashi
> > > <
> > > > > > > > > > > > > daniel.takabayashi@gmail.com> escreveu:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Hi all,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > To try to give a little more details about this
> > "new
> > > > > > concept"
> > > > > > > > > that
> > > > > > > > > > we
> > > > > > > > > > > > > want
> > > > > > > > > > > > > > to bring to the marvin toolbox, I did this simple
> > > > > > > architecture
> > > > > > > > > > draw.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > [image: marvin-architecture-views (5).png]
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > The general idea here is try to transform the
> > toolbox
> > > > > > > something
> > > > > > > > > > > > > > disconnected with the "language", something more
> > > > > agnostic.
> > > > > > > Also
> > > > > > > > > in
> > > > > > > > > > > this
> > > > > > > > > > > > > new
> > > > > > > > > > > > > > architecture we could use remote resource to
> > process
> > > > > > engines
> > > > > > > > and
> > > > > > > > > > make
> > > > > > > > > > > > > easy
> > > > > > > > > > > > > > the support for new languages.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > This "new toolbox" will be the only thing that a
> > > Marvin
> > > > > > user
> > > > > > > > must
> > > > > > > > > > to
> > > > > > > > > > > > > > install and also we could start to support
> > multiples
> > > > O.S
> > > > > > > (once
> > > > > > > > > the
> > > > > > > > > > > REPL
> > > > > > > > > > > > > is
> > > > > > > > > > > > > > a dummy application that only interprets and by
> > pass
> > > > > > > commands).
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Regards,
> > > > > > > > > > > > > > Taka
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Em qua, 24 de out de 2018 às 09:52, Rafael
> Novello
> > <
> > > > > > > > > > > > > > rafa.reis.novello@gmail.com> escreveu:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >> Alan,
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> Yes! We are using the Docker SDK and it's
> possible
> > > to
> > > > > use
> > > > > > > the
> > > > > > > > > API
> > > > > > > > > > to
> > > > > > > > > > > > > >> automate, but it's a little bit harder than
> > automate
> > > > CLI
> > > > > > > > calls.
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> Atenciosamente,
> > > > > > > > > > > > > >> Rafael J. R. Novello
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> Skype: rafael.novello
> > > > > > > > > > > > > >> Blog: http://rafanovello.blogspot.com.br/
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> Em qua, 24 de out de 2018 às 12:02, Alan Silva <
> > > > > > > > > jumpi@apache.org>
> > > > > > > > > > > > > >> escreveu:
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> > Hi,
> > > > > > > > > > > > > >> >
> > > > > > > > > > > > > >> > One question here, I understand that we start
> to
> > > use
> > > > > > with
> > > > > > > > this
> > > > > > > > > > PoC
> > > > > > > > > > > > the
> > > > > > > > > > > > > >> > Docker SDK API, right?
> > > > > > > > > > > > > >> >
> > > > > > > > > > > > > >> > Why not use the API to expose some endpoints
> to
> > > > permit
> > > > > > > this
> > > > > > > > > kind
> > > > > > > > > > > of
> > > > > > > > > > > > > >> > automation by devops?
> > > > > > > > > > > > > >> >
> > > > > > > > > > > > > >> > I think it is possible and it solves the CLI
> > > > problem,
> > > > > > > right?
> > > > > > > > > > > > > >> >
> > > > > > > > > > > > > >> >
> > > > > > > > > > > > > >> > On Tue, Oct 23, 2018 at 1:05 PM Rafael
> Novello <
> > > > > > > > > > > > > >> > rafa.reis.novello@gmail.com>
> > > > > > > > > > > > > >> > wrote:
> > > > > > > > > > > > > >> >
> > > > > > > > > > > > > >> > > Lucas,
> > > > > > > > > > > > > >> > >
> > > > > > > > > > > > > >> > > Sorry, I didn't understood your question
> > bellow.
> > > > > > > > > > > > > >> > >
> > > > > > > > > > > > > >> > > "Would it make sense to use the same
> solution
> > > that
> > > > > we
> > > > > > > will
> > > > > > > > > use
> > > > > > > > > > > for
> > > > > > > > > > > > > >> > having a
> > > > > > > > > > > > > >> > > single-language REPL to have a
> single-language
> > > > CLI?"
> > > > > > > > > > > > > >> > >
> > > > > > > > > > > > > >> > > For DevOps purposes, maybe this new toolbox
> > > > concept
> > > > > is
> > > > > > > not
> > > > > > > > > > > ideal.
> > > > > > > > > > > > I
> > > > > > > > > > > > > >> think
> > > > > > > > > > > > > >> > > we can keep the CLI inside the docker
> > container
> > > > but
> > > > > it
> > > > > > > > will
> > > > > > > > > > not
> > > > > > > > > > > > easy
> > > > > > > > > > > > > >> to
> > > > > > > > > > > > > >> > > automate
> > > > > > > > > > > > > >> > > jobs by this way.
> > > > > > > > > > > > > >> > >
> > > > > > > > > > > > > >> > > How to deal with this issue? Voting?
> > > > > > > > > > > > > >> > > Atenciosamente,
> > > > > > > > > > > > > >> > > Rafael J. R. Novello
> > > > > > > > > > > > > >> > >
> > > > > > > > > > > > > >> > > Skype: rafael.novello
> > > > > > > > > > > > > >> > > Blog: http://rafanovello.blogspot.com.br/
> > > > > > > > > > > > > >> > >
> > > > > > > > > > > > > >> > >
> > > > > > > > > > > > > >> > > Em sex, 19 de out de 2018 às 19:00, Lucas
> > > Bonatto
> > > > > > > Miguel <
> > > > > > > > > > > > > >> > > lucasbm88@apache.org> escreveu:
> > > > > > > > > > > > > >> > >
> > > > > > > > > > > > > >> > > > Got it! Thanks for clarifying.
> > > > > > > > > > > > > >> > > >
> > > > > > > > > > > > > >> > > > Would it make sense to use the same
> solution
> > > > that
> > > > > we
> > > > > > > > will
> > > > > > > > > > use
> > > > > > > > > > > > for
> > > > > > > > > > > > > >> > having
> > > > > > > > > > > > > >> > > a
> > > > > > > > > > > > > >> > > > single-language REPL to have a
> > single-language
> > > > > CLI?
> > > > > > My
> > > > > > > > > only
> > > > > > > > > > > > > concern
> > > > > > > > > > > > > >> > with
> > > > > > > > > > > > > >> > > > killing the CLI is that you remove an
> > > essential
> > > > > > > feature
> > > > > > > > > for
> > > > > > > > > > > > > DevOps.
> > > > > > > > > > > > > >> > > >
> > > > > > > > > > > > > >> > > > - Lucas
> > > > > > > > > > > > > >> > > >
> > > > > > > > > > > > > >> > > > On Fri, Oct 19, 2018 at 1:52 PM Rafael
> > > Novello <
> > > > > > > > > > > > > >> > > > rafa.reis.novello@gmail.com>
> > > > > > > > > > > > > >> > > > wrote:
> > > > > > > > > > > > > >> > > >
> > > > > > > > > > > > > >> > > > > Lucas,
> > > > > > > > > > > > > >> > > > >
> > > > > > > > > > > > > >> > > > > The idea is that REPL will substitute
> the
> > > > actual
> > > > > > > CLI.
> > > > > > > > > It's
> > > > > > > > > > > > > because
> > > > > > > > > > > > > >> > with
> > > > > > > > > > > > > >> > > > the
> > > > > > > > > > > > > >> > > > > actual concept (using language specific
> > CLI)
> > > > we
> > > > > > will
> > > > > > > > > need
> > > > > > > > > > to
> > > > > > > > > > > > > >> > > re-implement
> > > > > > > > > > > > > >> > > > > the same features many times and
> probably
> > > each
> > > > > CLI
> > > > > > > > will
> > > > > > > > > > > have a
> > > > > > > > > > > > > >> > > different
> > > > > > > > > > > > > >> > > > > behavior because some language specific
> > > > > > restrictions
> > > > > > > > > > and/or
> > > > > > > > > > > > > >> > > limitations.
> > > > > > > > > > > > > >> > > > >
> > > > > > > > > > > > > >> > > > > With this new concept, all users will
> have
> > > the
> > > > > > same
> > > > > > > > > > > experience
> > > > > > > > > > > > > >> > > > interacting
> > > > > > > > > > > > > >> > > > > with Marvin REPL and they will use the
> > bests
> > > > > tools
> > > > > > > to
> > > > > > > > > > > develop
> > > > > > > > > > > > > >> their
> > > > > > > > > > > > > >> > > > engines
> > > > > > > > > > > > > >> > > > > (Jupyter Notebook and/or Lab with
> > different
> > > > > > > languages
> > > > > > > > > > > support
> > > > > > > > > > > > or
> > > > > > > > > > > > > >> even
> > > > > > > > > > > > > >> > > > > Apache Zeppelin). All the interact will
> > > occur
> > > > > > > through
> > > > > > > > > REPL
> > > > > > > > > > > and
> > > > > > > > > > > > > >> docker
> > > > > > > > > > > > > >> > > > > protocol.
> > > > > > > > > > > > > >> > > > >
> > > > > > > > > > > > > >> > > > > Alan, Yes! As Lucas said, the concept is
> > the
> > > > > same
> > > > > > > but
> > > > > > > > we
> > > > > > > > > > can
> > > > > > > > > > > > use
> > > > > > > > > > > > > >> > docker
> > > > > > > > > > > > > >> > > > to
> > > > > > > > > > > > > >> > > > > do the same job.
> > > > > > > > > > > > > >> > > > >
> > > > > > > > > > > > > >> > > > > Best!
> > > > > > > > > > > > > >> > > > >
> > > > > > > > > > > > > >> > > > > Em sex, 19 de out de 2018 às 00:39,
> Lucas
> > > > > Bonatto
> > > > > > > > > Miguel <
> > > > > > > > > > > > > >> > > > > lucasbm88@apache.org> escreveu:
> > > > > > > > > > > > > >> > > > >
> > > > > > > > > > > > > >> > > > > > Thanks for the clarifications Rafael,
> so
> > > > from
> > > > > > > what I
> > > > > > > > > > > > > understood,
> > > > > > > > > > > > > >> > > > Marvin's
> > > > > > > > > > > > > >> > > > > > developers would not use the REPL to
> > > explore
> > > > > > data
> > > > > > > > and
> > > > > > > > > > test
> > > > > > > > > > > > > >> models,
> > > > > > > > > > > > > >> > > i.e.
> > > > > > > > > > > > > >> > > > > > develop the engine. Is the idea to
> build
> > > > > > something
> > > > > > > > > more
> > > > > > > > > > > like
> > > > > > > > > > > > > an
> > > > > > > > > > > > > >> > > > > interactive
> > > > > > > > > > > > > >> > > > > > CLI? I think an interactive CLI would
> be
> > > > > useful
> > > > > > > for
> > > > > > > > > the
> > > > > > > > > > > > > >> developer
> > > > > > > > > > > > > >> > > > > > experience, however, it's important to
> > > keep
> > > > > the
> > > > > > > > > > unattended
> > > > > > > > > > > > > >> > (current)
> > > > > > > > > > > > > >> > > > CLI
> > > > > > > > > > > > > >> > > > > in
> > > > > > > > > > > > > >> > > > > > place for scenarios where users want
> to
> > > > > automate
> > > > > > > > > Marvin
> > > > > > > > > > > > tasks.
> > > > > > > > > > > > > >> > > > > >
> > > > > > > > > > > > > >> > > > > > Alan, I believe the idea is still the
> > same
> > > > as
> > > > > > you
> > > > > > > > > > started,
> > > > > > > > > > > > but
> > > > > > > > > > > > > >> > using
> > > > > > > > > > > > > >> > > > > > docker SDK now.
> > > > > > > > > > > > > >> > > > > >
> > > > > > > > > > > > > >> > > > > > - Lucas
> > > > > > > > > > > > > >> > > > > >
> > > > > > > > > > > > > >> > > > > > On Thu, Oct 18, 2018 at 1:29 PM Rafael
> > > > > Novello <
> > > > > > > > > > > > > >> > > > > > rafa.reis.novello@gmail.com>
> > > > > > > > > > > > > >> > > > > > wrote:
> > > > > > > > > > > > > >> > > > > >
> > > > > > > > > > > > > >> > > > > > > Hi Lucas!
> > > > > > > > > > > > > >> > > > > > >
> > > > > > > > > > > > > >> > > > > > > First of all +1 for REPL POCs to
> > Apache!
> > > > > > > > > > > > > >> > > > > > >
> > > > > > > > > > > > > >> > > > > > > Let me help with some comments:
> > > > > > > > > > > > > >> > > > > > >
> > > > > > > > > > > > > >> > > > > > > 1 - We have tested NodeJS, Scala and
> > > > Python
> > > > > > and
> > > > > > > > the
> > > > > > > > > > > > easiest
> > > > > > > > > > > > > >> one
> > > > > > > > > > > > > >> > was
> > > > > > > > > > > > > >> > > > > > Python.
> > > > > > > > > > > > > >> > > > > > > We have found a small project [1]
> that
> > > > have
> > > > > > all
> > > > > > > > > > features
> > > > > > > > > > > > we
> > > > > > > > > > > > > >> > desired
> > > > > > > > > > > > > >> > > > for
> > > > > > > > > > > > > >> > > > > > > REPL:
> > > > > > > > > > > > > >> > > > > > > - Autocomplete
> > > > > > > > > > > > > >> > > > > > > - Python commands disabled (the user
> > > have
> > > > > only
> > > > > > > the
> > > > > > > > > > > > commands
> > > > > > > > > > > > > we
> > > > > > > > > > > > > >> > > > > provide).
> > > > > > > > > > > > > >> > > > > > > The other languages REPL options
> don't
> > > > have
> > > > > > this
> > > > > > > > > > > feature.
> > > > > > > > > > > > > >> > > > > > > - Easy to show output
> > > > > > > > > > > > > >> > > > > > > - Etc
> > > > > > > > > > > > > >> > > > > > > So, I think the language chosen will
> > not
> > > > be
> > > > > > > > > important
> > > > > > > > > > > here
> > > > > > > > > > > > > >> > because
> > > > > > > > > > > > > >> > > > the
> > > > > > > > > > > > > >> > > > > > user
> > > > > > > > > > > > > >> > > > > > > will only interact with the commands
> > > that
> > > > we
> > > > > > > > create.
> > > > > > > > > > > > > >> > > > > > >
> > > > > > > > > > > > > >> > > > > > > 2 - The "engine-generate" command
> will
> > > > > > download
> > > > > > > a
> > > > > > > > > > docker
> > > > > > > > > > > > > image
> > > > > > > > > > > > > >> > that
> > > > > > > > > > > > > >> > > > we
> > > > > > > > > > > > > >> > > > > > > create for that language and start a
> > > > > container
> > > > > > > > with
> > > > > > > > > > the
> > > > > > > > > > > > > basic
> > > > > > > > > > > > > >> > > project
> > > > > > > > > > > > > >> > > > > > > structure for that language.
> > > > > > > > > > > > > >> > > > > > >
> > > > > > > > > > > > > >> > > > > > > 3 - The REPL client will use the
> > "docker
> > > > > > > protocol"
> > > > > > > > > to
> > > > > > > > > > > run
> > > > > > > > > > > > > >> > command,
> > > > > > > > > > > > > >> > > > > > > start/stop services and etc inside
> the
> > > > > > container
> > > > > > > > and
> > > > > > > > > > it
> > > > > > > > > > > > will
> > > > > > > > > > > > > >> > > receive
> > > > > > > > > > > > > >> > > > > the
> > > > > > > > > > > > > >> > > > > > > log stream to show. No, the REPL
> will
> > no
> > > > > pass
> > > > > > > code
> > > > > > > > > > > > snippets
> > > > > > > > > > > > > >> for
> > > > > > > > > > > > > >> > > > docker
> > > > > > > > > > > > > >> > > > > > > container (I think it will not be
> > > > necessary)
> > > > > > > > > > > > > >> > > > > > >
> > > > > > > > > > > > > >> > > > > > > 4 - Yep! Like I said on the first
> > item.
> > > > > > > > > > > > > >> > > > > > >
> > > > > > > > > > > > > >> > > > > > > [1] -
> > > > https://github.com/italorossi/ishell
> > > > > > > > > > > > > >> > > > > > >
> > > > > > > > > > > > > >> > > > > > > Let me know if there is any other
> > > > question!
> > > > > > > > > > > > > >> > > > > > > Best!
> > > > > > > > > > > > > >> > > > > > > Rafael J. R. Novello
> > > > > > > > > > > > > >> > > > > > >
> > > > > > > > > > > > > >> > > > > > > Skype: rafael.novello
> > > > > > > > > > > > > >> > > > > > > Blog:
> > > http://rafanovello.blogspot.com.br/
> > > > > > > > > > > > > >> > > > > > >
> > > > > > > > > > > > > >> > > > > > >
> > > > > > > > > > > > > >> > > > > > > Em qui, 18 de out de 2018 às 17:00,
> > > Lucas
> > > > > > > Bonatto
> > > > > > > > > > > Miguel <
> > > > > > > > > > > > > >> > > > > > > lucasbm88@apache.org> escreveu:
> > > > > > > > > > > > > >> > > > > > >
> > > > > > > > > > > > > >> > > > > > > > + 1 for migrating the REPL repo to
> > > > Apache
> > > > > > > > > > > > > >> > > > > > > >
> > > > > > > > > > > > > >> > > > > > > > I have a few questions about the
> > > > previous
> > > > > > > > > > explanation:
> > > > > > > > > > > > > >> > > > > > > >  1) The REPL itself, it would be
> an
> > > > > > > application
> > > > > > > > in
> > > > > > > > > > > which
> > > > > > > > > > > > > >> > > language?
> > > > > > > > > > > > > >> > > > > > > Remember
> > > > > > > > > > > > > >> > > > > > > > that the main idea is to allow the
> > > user
> > > > to
> > > > > > > > program
> > > > > > > > > > on
> > > > > > > > > > > > his
> > > > > > > > > > > > > >> > > preferred
> > > > > > > > > > > > > >> > > > > > > > language in the REPL.
> > > > > > > > > > > > > >> > > > > > > >  2) Should the engine-generate
> > command
> > > > > also
> > > > > > > > > > generate a
> > > > > > > > > > > > > >> docker
> > > > > > > > > > > > > >> > > image
> > > > > > > > > > > > > >> > > > > > with
> > > > > > > > > > > > > >> > > > > > > > the user's application?
> > > > > > > > > > > > > >> > > > > > > >  3) What type of communication
> would
> > > > > happen
> > > > > > > > > between
> > > > > > > > > > > the
> > > > > > > > > > > > > REPL
> > > > > > > > > > > > > >> > and
> > > > > > > > > > > > > >> > > > the
> > > > > > > > > > > > > >> > > > > > > engine
> > > > > > > > > > > > > >> > > > > > > > via Docker SDK? Would the REPL
> pass
> > > > > snippets
> > > > > > > of
> > > > > > > > > code
> > > > > > > > > > > to
> > > > > > > > > > > > be
> > > > > > > > > > > > > >> > > executed
> > > > > > > > > > > > > >> > > > > by
> > > > > > > > > > > > > >> > > > > > > the
> > > > > > > > > > > > > >> > > > > > > > docker container?
> > > > > > > > > > > > > >> > > > > > > >  4) Have you considered code
> > > completion
> > > > in
> > > > > > the
> > > > > > > > > REPL?
> > > > > > > > > > > > > >> > > > > > > >
> > > > > > > > > > > > > >> > > > > > > >
> > > > > > > > > > > > > >> > > > > > > > On Thu, Oct 18, 2018 at 7:53 AM
> > Zhang
> > > > > Yifei
> > > > > > <
> > > > > > > > > > > > > >> > > > yifei.z.luiz@gmail.com>
> > > > > > > > > > > > > >> > > > > > > > wrote:
> > > > > > > > > > > > > >> > > > > > > >
> > > > > > > > > > > > > >> > > > > > > > > Ok guys,
> > > > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > > > >> > > > > > > > > The basic idea is to provide
> only
> > > one
> > > > > > > Toolbox
> > > > > > > > > for
> > > > > > > > > > > > > multiple
> > > > > > > > > > > > > >> > > > > languages.
> > > > > > > > > > > > > >> > > > > > > > > We are looking for possibility
> to
> > > > build
> > > > > a
> > > > > > > > single
> > > > > > > > > > > > Marvin
> > > > > > > > > > > > > >> Repl,
> > > > > > > > > > > > > >> > > > > > > > > instead of severals toolboxes
> with
> > > > > > > differentes
> > > > > > > > > > > > > interfaces
> > > > > > > > > > > > > >> or
> > > > > > > > > > > > > >> > > > > > commands.
> > > > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > > > >> > > > > > > > > In this case, the
> engine-generate
> > > > > command
> > > > > > > will
> > > > > > > > > > > > download
> > > > > > > > > > > > > >> and
> > > > > > > > > > > > > >> > > > start a
> > > > > > > > > > > > > >> > > > > > > > Docker
> > > > > > > > > > > > > >> > > > > > > > > container with basic engine
> > > structure
> > > > > > > > > > corresponding
> > > > > > > > > > > > the
> > > > > > > > > > > > > >> > choosed
> > > > > > > > > > > > > >> > > > > > > language.
> > > > > > > > > > > > > >> > > > > > > > > this means we don't need to
> build
> > > > > > Toolboxes
> > > > > > > of
> > > > > > > > > > > > > differents
> > > > > > > > > > > > > >> > > > > languages,
> > > > > > > > > > > > > >> > > > > > we
> > > > > > > > > > > > > >> > > > > > > > > just
> > > > > > > > > > > > > >> > > > > > > > > build the engine template of all
> > > > > languages
> > > > > > > > that
> > > > > > > > > we
> > > > > > > > > > > > want
> > > > > > > > > > > > > to
> > > > > > > > > > > > > >> > > > support
> > > > > > > > > > > > > >> > > > > > and
> > > > > > > > > > > > > >> > > > > > > > > provide it
> > > > > > > > > > > > > >> > > > > > > > > as Docker containers
> > > > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > > > >> > > > > > > > > We have started researches
> around
> > > our
> > > > > > basic
> > > > > > > > > > > > requirements
> > > > > > > > > > > > > >> > like:
> > > > > > > > > > > > > >> > > > > > > > > - Repl interface
> > > > > > > > > > > > > >> > > > > > > > > - System communication
> > > > > > > > > > > > > >> > > > > > > > > - Connection security
> > > > > > > > > > > > > >> > > > > > > > > - Tool popularity
> > > > > > > > > > > > > >> > > > > > > > > - Update complexity
> > > > > > > > > > > > > >> > > > > > > > > - Languages support
> > > > > > > > > > > > > >> > > > > > > > > - ......
> > > > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > > > >> > > > > > > > > And we did some POC with code
> > here:
> > > > > > > > > > > > > >> > > > > > > > >
> > > > > https://github.com/marvin-ai/marvin-repl
> > > > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > > > >> > > > > > > > > There is POC testing gRPC using
> > > Scala
> > > > > and
> > > > > > > > > Python,
> > > > > > > > > > > > > >> > > > > > > > > Repl inteface and Docker SDK
> with
> > > > > NodeJS,
> > > > > > > > > > > > > >> > > > > > > > > Repl interface and Docker SDK
> with
> > > > > Python.
> > > > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > > > >> > > > > > > > > At this moment we prefer the
> Repl
> > > > > > interface
> > > > > > > +
> > > > > > > > > > Docker
> > > > > > > > > > > > SDK
> > > > > > > > > > > > > >> way,
> > > > > > > > > > > > > >> > > > > because
> > > > > > > > > > > > > >> > > > > > > > good
> > > > > > > > > > > > > >> > > > > > > > > part of the requirements
> > > > > > > > > > > > > >> > > > > > > > > will be guaranteed by Docker.
> > > > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > > > >> > > > > > > > > With this informations, what do
> > you
> > > > > think?
> > > > > > > > > Should
> > > > > > > > > > we
> > > > > > > > > > > > > >> submit
> > > > > > > > > > > > > >> > all
> > > > > > > > > > > > > >> > > > > this
> > > > > > > > > > > > > >> > > > > > > POCs
> > > > > > > > > > > > > >> > > > > > > > > to Apache Repo?
> > > > > > > > > > > > > >> > > > > > > > > Please feel free to opine.
> > > > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > > > >> > > > > > > > > Thats all, thanks!!!
> > > > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > > > >> > > > > > > > > Em ter, 16 de out de 2018 às
> > 18:55,
> > > > > Daniel
> > > > > > > > > > > > Takabayashi <
> > > > > > > > > > > > > >> > > > > > > > > daniel.takabayashi@gmail.com>
> > > > escreveu:
> > > > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > > > >> > > > > > > > > > Zhang,
> > > > > > > > > > > > > >> > > > > > > > > >
> > > > > > > > > > > > > >> > > > > > > > > > I think the best approach is
> > give
> > > > us a
> > > > > > > > better
> > > > > > > > > > > > > >> explanation
> > > > > > > > > > > > > >> > > about
> > > > > > > > > > > > > >> > > > > > this
> > > > > > > > > > > > > >> > > > > > > > new
> > > > > > > > > > > > > >> > > > > > > > > > feature and how this can help
> us
> > > to
> > > > > > > archive
> > > > > > > > > the
> > > > > > > > > > > > > desired
> > > > > > > > > > > > > >> > > feature
> > > > > > > > > > > > > >> > > > > > > > (support
> > > > > > > > > > > > > >> > > > > > > > > > multiple languages). Than if
> the
> > > > > > majority
> > > > > > > > > agree
> > > > > > > > > > > > than I
> > > > > > > > > > > > > >> can
> > > > > > > > > > > > > >> > do
> > > > > > > > > > > > > >> > > > the
> > > > > > > > > > > > > >> > > > > > > > merge,
> > > > > > > > > > > > > >> > > > > > > > > > just like I did before, but in
> > > > another
> > > > > > > > branch.
> > > > > > > > > > > > > >> > > > > > > > > >
> > > > > > > > > > > > > >> > > > > > > > > > Let me know if makes sense,
> ok?
> > > > > > > > > > > > > >> > > > > > > > > >
> > > > > > > > > > > > > >> > > > > > > > > > Taka
> > > > > > > > > > > > > >> > > > > > > > > >
> > > > > > > > > > > > > >> > > > > > > > > > Em ter, 16 de out de 2018 às
> > > 14:00,
> > > > > > > Luciano
> > > > > > > > > > > Resende
> > > > > > > > > > > > <
> > > > > > > > > > > > > >> > > > > > > > > luckbr1975@gmail.com>
> > > > > > > > > > > > > >> > > > > > > > > > escreveu:
> > > > > > > > > > > > > >> > > > > > > > > >
> > > > > > > > > > > > > >> > > > > > > > > > > So, what is the POC, is it a
> > > > > > refactoring
> > > > > > > > of
> > > > > > > > > > the
> > > > > > > > > > > > > >> existing
> > > > > > > > > > > > > >> > > > repo?
> > > > > > > > > > > > > >> > > > > Or
> > > > > > > > > > > > > >> > > > > > > is
> > > > > > > > > > > > > >> > > > > > > > > > > it a new rewrite of the
> repo?
> > > > > > > > > > > > > >> > > > > > > > > > >
> > > > > > > > > > > > > >> > > > > > > > > > > Just asking as it might make
> > > sense
> > > > > to
> > > > > > > make
> > > > > > > > > it
> > > > > > > > > > a
> > > > > > > > > > > > > branch
> > > > > > > > > > > > > >> > then
> > > > > > > > > > > > > >> > > > > > > actually
> > > > > > > > > > > > > >> > > > > > > > a
> > > > > > > > > > > > > >> > > > > > > > > > > thing in parallel, as this
> > will
> > > > have
> > > > > > an
> > > > > > > > > effect
> > > > > > > > > > > for
> > > > > > > > > > > > > >> > > releases,
> > > > > > > > > > > > > >> > > > > etc.
> > > > > > > > > > > > > >> > > > > > > but
> > > > > > > > > > > > > >> > > > > > > > > > > you guys know more here
> than I
> > > do.
> > > > > > > > > > > > > >> > > > > > > > > > >
> > > > > > > > > > > > > >> > > > > > > > > > > Also, it's probably good to
> > > have a
> > > > > > write
> > > > > > > > up
> > > > > > > > > of
> > > > > > > > > > > the
> > > > > > > > > > > > > >> main
> > > > > > > > > > > > > >> > > > > direction
> > > > > > > > > > > > > >> > > > > > > of
> > > > > > > > > > > > > >> > > > > > > > > > > the design that can help
> > people
> > > > get
> > > > > > > > familiar
> > > > > > > > > > > with
> > > > > > > > > > > > > the
> > > > > > > > > > > > > >> new
> > > > > > > > > > > > > >> > > > > > approach.
> > > > > > > > > > > > > >> > > > > > > > > > > On Tue, Oct 16, 2018 at
> 11:12
> > AM
> > > > > Zhang
> > > > > > > > > Yifei <
> > > > > > > > > > > > > >> > > > > > > yifei.z.luiz@gmail.com
> > > > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > > > >> > > > > > > > > > > wrote:
> > > > > > > > > > > > > >> > > > > > > > > > > >
> > > > > > > > > > > > > >> > > > > > > > > > > > Hey guys,
> > > > > > > > > > > > > >> > > > > > > > > > > >
> > > > > > > > > > > > > >> > > > > > > > > > > > We have reorganized the
> Poc
> > > > repo,
> > > > > > and
> > > > > > > > want
> > > > > > > > > > to
> > > > > > > > > > > > > merge
> > > > > > > > > > > > > >> it
> > > > > > > > > > > > > >> > to
> > > > > > > > > > > > > >> > > > our
> > > > > > > > > > > > > >> > > > > > > > Apache
> > > > > > > > > > > > > >> > > > > > > > > > > repo.
> > > > > > > > > > > > > >> > > > > > > > > > > > Just making sure here
> before
> > > we
> > > > do
> > > > > > the
> > > > > > > > > > merge,
> > > > > > > > > > > > > >> because
> > > > > > > > > > > > > >> > im
> > > > > > > > > > > > > >> > > > not
> > > > > > > > > > > > > >> > > > > > sure
> > > > > > > > > > > > > >> > > > > > > > if
> > > > > > > > > > > > > >> > > > > > > > > > > doing
> > > > > > > > > > > > > >> > > > > > > > > > > > this will perserve the Git
> > > > commit
> > > > > > > > history.
> > > > > > > > > > > > > >> > > > > > > > > > > > What we are planning to do
> > is:
> > > > > > > > > > > > > >> > > > > > > > > > > >
> > > > > > > > > > > > > >> > > > > > > > > > > > git filter-branch
> > > > > > > --subdirectory-filter
> > > > > > > > > <git
> > > > > > > > > > > > > >> > > > > origin_repository
> > > > > > > > > > > > > >> > > > > > > > > > directory>
> > > > > > > > > > > > > >> > > > > > > > > > > > -- --all
> > > > > > > > > > > > > >> > > > > > > > > > > > mkdir POCs/
> > > > > > > > > > > > > >> > > > > > > > > > > > git mv * POCs/
> > > > > > > > > > > > > >> > > > > > > > > > > > git commit -m "colleted
> the
> > > data
> > > > > to
> > > > > > > > move"
> > > > > > > > > > > > > >> > > > > > > > > > > >
> > > > > > > > > > > > > >> > > > > > > > > > > > git clone <git
> > > > > > > > destination_repository_url>
> > > > > > > > > > cd
> > > > > > > > > > > > <git
> > > > > > > > > > > > > >> > > > > > > > > > > >
> > > > destination_repository_directory>
> > > > > > git
> > > > > > > > > remote
> > > > > > > > > > > add
> > > > > > > > > > > > > >> > > > > > > > > > origin_repository_branch
> > > > > > > > > > > > > >> > > > > > > > > > > > <git origin_repository
> > > > directory>
> > > > > > git
> > > > > > > > pull
> > > > > > > > > > > > > >> > > > > > > origin_repository_branch
> > > > > > > > > > > > > >> > > > > > > > > > > master
> > > > > > > > > > > > > >> > > > > > > > > > > >
> --allow-unrelated-histories
> > > git
> > > > > > remote
> > > > > > > > rm
> > > > > > > > > > > > > >> > > > > > > origin_repository_branch
> > > > > > > > > > > > > >> > > > > > > > > > > >
> > > > > > > > > > > > > >> > > > > > > > > > > >
> > > > > > > > > > > > > >> > > > > > > > > > > > Thanks in advance !
> > > > > > > > > > > > > >> > > > > > > > > > > >
> > > > > > > > > > > > > >> > > > > > > > > > > >
> > > > > > > > > > > > > >> > > > >
> > > > > > > > > > >
> > > > --------------------------------------------------------------
> > > > > > > > > > > > > >> > > > > > > > > > > > Zhang Yifei
> > > > > > > > > > > > > >> > > > > > > > > > >
> > > > > > > > > > > > > >> > > > > > > > > > >
> > > > > > > > > > > > > >> > > > > > > > > > >
> > > > > > > > > > > > > >> > > > > > > > > > > --
> > > > > > > > > > > > > >> > > > > > > > > > > Luciano Resende
> > > > > > > > > > > > > >> > > > > > > > > > >
> > http://twitter.com/lresende1975
> > > > > > > > > > > > > >> > > > > > > > > > >
> http://lresende.blogspot.com/
> > > > > > > > > > > > > >> > > > > > > > > > >
> > > > > > > > > > > > > >> > > > > > > > > >
> > > > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > > > >> > > > > > > > > --
> > > > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > > > >> >
> > > > > > > >
> --------------------------------------------------------------
> > > > > > > > > > > > > >> > > > > > > > > Zhang Yifei
> > > > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > > > >> > > > > > > >
> > > > > > > > > > > > > >> > > > > > >
> > > > > > > > > > > > > >> > > > > >
> > > > > > > > > > > > > >> > > > >
> > > > > > > > > > > > > >> > > >
> > > > > > > > > > > > > >> > >
> > > > > > > > > > > > > >> >
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>


-- 
--------------------------------------------------------------
Zhang Yifei

Re: POCs, Prototypes, experiments ...

Posted by Rafael Novello <ra...@gmail.com>.

Lucas, yes! It make sense!

In the end, CLI and REPL will share a big part of code and they will be
just different interfaces to the same features.

+1 for this new concept!

Thanks for make it clear to me!
Atenciosamente,
Rafael J. R. Novello

Skype: rafael.novello
Blog: http://rafanovello.blogspot.com.br/


Em seg, 29 de out de 2018 às 17:43, Lucas Bonatto Miguel <
lucasbm88@apache.org> escreveu:

> That's a good point Rafael. I believe the main motivation for adding a REPL
> to Marvin's toolbox is to enhance user experience.
>
> Imo we can't just switch CLI per REPL. I don't think one excludes the
> other, both have different use cases.
>
> We may be mixing too much stuff together, but in general, I think we are:
>  - Refactoring toolbox architecture to avoid unnecessarily duplicated logic
> when adding support to new languages.
>  - Adding a new interface to enable users to open an interactive session
> with the engine (i.e. REPL)
>  - Making it easier to add support for new languages.
>
> Does it make sense for you?
>
> On Mon, Oct 29, 2018 at 11:22 AM Rafael Novello <
> rafa.reis.novello@gmail.com>
> wrote:
>
> > I do not want to be the annoying conversation but...
> >
> > If we will have an unique CLI that will be able to handle any kind of
> > engines (in any language) by using docker, Why we need a REPL?
> >
> > I think we can focus on creating this new architecture using docker and
> > docker SDK and use only a CLI for that.
> >
> > Atenciosamente,
> > Rafael J. R. Novello
> >
> > Skype: rafael.novello
> > Blog: http://rafanovello.blogspot.com.br/
> >
> >
> > Em seg, 29 de out de 2018 às 14:58, Lucas Bonatto Miguel <
> > lucasbm88@apache.org> escreveu:
> >
> > > No, I think the idea is to do not create different versions of the CLI.
> > > Let's use one single tech stack for the CLI, ideally, it will be shared
> > > with the REPL. The way I see the CLI would be very similar to the REPL,
> > but
> > > instead of opening an interactive session the user could send
> > > unattended commands. As an analogy, you can think on spark-shell (our
> > REPL)
> > > and spark-submit (our CLI).
> > >
> > > - Lucas
> > >
> > > On Mon, Oct 29, 2018 at 10:27 AM Rafael Novello <
> > > rafa.reis.novello@gmail.com>
> > > wrote:
> > >
> > > > Hi Guys!
> > > >
> > > > Let me ask about this new version that include CLI's and REPL. Will
> we
> > > > create different versions of CLI?
> > > >
> > > > One of the reasons to create the REPL is to avoid re-implementing the
> > > same
> > > > features many times,
> > > > each time for a different language.
> > > >
> > > > Regards!
> > > > Atenciosamente,
> > > > Rafael J. R. Novello
> > > >
> > > > Skype: rafael.novello
> > > > Blog: http://rafanovello.blogspot.com.br/
> > > >
> > > >
> > > > Em sex, 26 de out de 2018 às 20:54, Daniel Takabayashi <
> > > > daniel.takabayashi@gmail.com> escreveu:
> > > >
> > > > > +1
> > > > >
> > > > > Em sex, 26 de out de 2018 às 16:47, Lucas Bonatto Miguel <
> > > > > lucasbm88@apache.org> escreveu:
> > > > >
> > > > > > Makes perfect sense to me. I believe that if the majority of
> people
> > > > > agrees
> > > > > > with this architecture we could create issues and start
> > implementing
> > > > it.
> > > > > >
> > > > > > +1
> > > > > >
> > > > > >
> > > > > > On Fri, Oct 26, 2018 at 4:09 PM Daniel Takabayashi <
> > > > > > daniel.takabayashi@gmail.com> wrote:
> > > > > >
> > > > > > > I see... what about this new version?
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://docs.google.com/drawings/d/shajxIpLJHxxMbFgDXiPuhg/image?w=602&h=468&rev=1630&ac=1&parent=1ySERHGBXbHeyCMRookq5UfTuFkzzU0ugtjvR3rF3deY
> > > > > > >
> > > > > > > I changed the toolbox ...from 1 component to 3... docker
> client,
> > > cli
> > > > > and
> > > > > > > repl. Both (cli and repl) now are using the docker client to
> > > control
> > > > > the
> > > > > > > engines.
> > > > > > >
> > > > > > > Em sex, 26 de out de 2018 às 14:49, Lucas Bonatto Miguel <
> > > > > > > lucasbm88@apache.org> escreveu:
> > > > > > >
> > > > > > > > For sure, both suggestions are to make sure we're not killing
> > the
> > > > CLI
> > > > > > and
> > > > > > > > that we have a component (the SDK client) that is shared by
> CLI
> > > and
> > > > > > REPL.
> > > > > > > >
> > > > > > > > Thanks for clarifying where the system calls will live.
> > > > > > > >
> > > > > > > > - Lucas
> > > > > > > >
> > > > > > > > On Fri, Oct 26, 2018 at 2:39 PM Daniel Takabayashi <
> > > > > > > > daniel.takabayashi@gmail.com> wrote:
> > > > > > > >
> > > > > > > > > Hi Lucas, could you please justified your two suggestions?
> > > > > > > > >
> > > > > > > > > The general ideia is to use the Docker DSK to execute local
> > > > > commands,
> > > > > > > to
> > > > > > > > > consume outputs and also to execute Marvin "specific
> > language"
> > > > > > > commands.
> > > > > > > > > Today almost all CLIs commands in Marvin is a mixed of
> > systems
> > > > > > commands
> > > > > > > > and
> > > > > > > > > library method calls. The plan is to put all "specific
> > language
> > > > > > calls"
> > > > > > > > > inside the marvin-common-libs and group the systems call on
> > the
> > > > > > toolbox
> > > > > > > > > (using the docker SDK in this task).
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Em qui, 25 de out de 2018 às 20:09, Lucas Bonatto Miguel <
> > > > > > > > > lucasbm88@apache.org> escreveu:
> > > > > > > > >
> > > > > > > > > > Great, thanks.
> > > > > > > > > >
> > > > > > > > > > So here some suggestions:
> > > > > > > > > >  1. We could create one new layer between Toolbox and the
> > > > > Engines,
> > > > > > > this
> > > > > > > > > > would be Docker client.
> > > > > > > > > >  2. Split the Toolbox layer in two. Toolbox CLI and
> Toolbox
> > > > REPL
> > > > > > > > > >
> > > > > > > > > > It's not clear to me if the commands listed on General
> > > Commands
> > > > > > will
> > > > > > > > need
> > > > > > > > > > to be written for every language, from the image I
> believe
> > > not,
> > > > > but
> > > > > > > how
> > > > > > > > > it
> > > > > > > > > > would work, since starting a notebook in Python is
> > different
> > > > than
> > > > > > > > > starting
> > > > > > > > > > in Scala.
> > > > > > > > > >
> > > > > > > > > > - Lucas
> > > > > > > > > > On Thu, Oct 25, 2018 at 4:51 PM Daniel Takabayashi <
> > > > > > > > > > daniel.takabayashi@gmail.com> wrote:
> > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://docs.google.com/drawings/d/shajxIpLJHxxMbFgDXiPuhg/image?w=602&h=461&rev=1423&ac=1&parent=1ySERHGBXbHeyCMRookq5UfTuFkzzU0ugtjvR3rF3deY
> > > > > > > > > > >
> > > > > > > > > > > Em qui, 25 de out de 2018 às 07:38, Rafael Novello <
> > > > > > > > > > > rafa.reis.novello@gmail.com> escreveu:
> > > > > > > > > > >
> > > > > > > > > > > > Hi Taka,
> > > > > > > > > > > >
> > > > > > > > > > > > I can't see the image! Maybe the Apache email list
> > don't
> > > > > > support
> > > > > > > > > > > > attachments.
> > > > > > > > > > > >
> > > > > > > > > > > > Best!
> > > > > > > > > > > > Rafael J. R. Novello
> > > > > > > > > > > >
> > > > > > > > > > > > Skype: rafael.novello
> > > > > > > > > > > > Blog: http://rafanovello.blogspot.com.br/
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > Em qua, 24 de out de 2018 às 21:14, Daniel
> Takabayashi
> > <
> > > > > > > > > > > > daniel.takabayashi@gmail.com> escreveu:
> > > > > > > > > > > >
> > > > > > > > > > > > > Hi all,
> > > > > > > > > > > > >
> > > > > > > > > > > > > To try to give a little more details about this
> "new
> > > > > concept"
> > > > > > > > that
> > > > > > > > > we
> > > > > > > > > > > > want
> > > > > > > > > > > > > to bring to the marvin toolbox, I did this simple
> > > > > > architecture
> > > > > > > > > draw.
> > > > > > > > > > > > >
> > > > > > > > > > > > > [image: marvin-architecture-views (5).png]
> > > > > > > > > > > > >
> > > > > > > > > > > > > The general idea here is try to transform the
> toolbox
> > > > > > something
> > > > > > > > > > > > > disconnected with the "language", something more
> > > > agnostic.
> > > > > > Also
> > > > > > > > in
> > > > > > > > > > this
> > > > > > > > > > > > new
> > > > > > > > > > > > > architecture we could use remote resource to
> process
> > > > > engines
> > > > > > > and
> > > > > > > > > make
> > > > > > > > > > > > easy
> > > > > > > > > > > > > the support for new languages.
> > > > > > > > > > > > >
> > > > > > > > > > > > > This "new toolbox" will be the only thing that a
> > Marvin
> > > > > user
> > > > > > > must
> > > > > > > > > to
> > > > > > > > > > > > > install and also we could start to support
> multiples
> > > O.S
> > > > > > (once
> > > > > > > > the
> > > > > > > > > > REPL
> > > > > > > > > > > > is
> > > > > > > > > > > > > a dummy application that only interprets and by
> pass
> > > > > > commands).
> > > > > > > > > > > > >
> > > > > > > > > > > > > Regards,
> > > > > > > > > > > > > Taka
> > > > > > > > > > > > >
> > > > > > > > > > > > > Em qua, 24 de out de 2018 às 09:52, Rafael Novello
> <
> > > > > > > > > > > > > rafa.reis.novello@gmail.com> escreveu:
> > > > > > > > > > > > >
> > > > > > > > > > > > >> Alan,
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> Yes! We are using the Docker SDK and it's possible
> > to
> > > > use
> > > > > > the
> > > > > > > > API
> > > > > > > > > to
> > > > > > > > > > > > >> automate, but it's a little bit harder than
> automate
> > > CLI
> > > > > > > calls.
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> Atenciosamente,
> > > > > > > > > > > > >> Rafael J. R. Novello
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> Skype: rafael.novello
> > > > > > > > > > > > >> Blog: http://rafanovello.blogspot.com.br/
> > > > > > > > > > > > >>
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> Em qua, 24 de out de 2018 às 12:02, Alan Silva <
> > > > > > > > jumpi@apache.org>
> > > > > > > > > > > > >> escreveu:
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> > Hi,
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> > One question here, I understand that we start to
> > use
> > > > > with
> > > > > > > this
> > > > > > > > > PoC
> > > > > > > > > > > the
> > > > > > > > > > > > >> > Docker SDK API, right?
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> > Why not use the API to expose some endpoints to
> > > permit
> > > > > > this
> > > > > > > > kind
> > > > > > > > > > of
> > > > > > > > > > > > >> > automation by devops?
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> > I think it is possible and it solves the CLI
> > > problem,
> > > > > > right?
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> > On Tue, Oct 23, 2018 at 1:05 PM Rafael Novello <
> > > > > > > > > > > > >> > rafa.reis.novello@gmail.com>
> > > > > > > > > > > > >> > wrote:
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >> > > Lucas,
> > > > > > > > > > > > >> > >
> > > > > > > > > > > > >> > > Sorry, I didn't understood your question
> bellow.
> > > > > > > > > > > > >> > >
> > > > > > > > > > > > >> > > "Would it make sense to use the same solution
> > that
> > > > we
> > > > > > will
> > > > > > > > use
> > > > > > > > > > for
> > > > > > > > > > > > >> > having a
> > > > > > > > > > > > >> > > single-language REPL to have a single-language
> > > CLI?"
> > > > > > > > > > > > >> > >
> > > > > > > > > > > > >> > > For DevOps purposes, maybe this new toolbox
> > > concept
> > > > is
> > > > > > not
> > > > > > > > > > ideal.
> > > > > > > > > > > I
> > > > > > > > > > > > >> think
> > > > > > > > > > > > >> > > we can keep the CLI inside the docker
> container
> > > but
> > > > it
> > > > > > > will
> > > > > > > > > not
> > > > > > > > > > > easy
> > > > > > > > > > > > >> to
> > > > > > > > > > > > >> > > automate
> > > > > > > > > > > > >> > > jobs by this way.
> > > > > > > > > > > > >> > >
> > > > > > > > > > > > >> > > How to deal with this issue? Voting?
> > > > > > > > > > > > >> > > Atenciosamente,
> > > > > > > > > > > > >> > > Rafael J. R. Novello
> > > > > > > > > > > > >> > >
> > > > > > > > > > > > >> > > Skype: rafael.novello
> > > > > > > > > > > > >> > > Blog: http://rafanovello.blogspot.com.br/
> > > > > > > > > > > > >> > >
> > > > > > > > > > > > >> > >
> > > > > > > > > > > > >> > > Em sex, 19 de out de 2018 às 19:00, Lucas
> > Bonatto
> > > > > > Miguel <
> > > > > > > > > > > > >> > > lucasbm88@apache.org> escreveu:
> > > > > > > > > > > > >> > >
> > > > > > > > > > > > >> > > > Got it! Thanks for clarifying.
> > > > > > > > > > > > >> > > >
> > > > > > > > > > > > >> > > > Would it make sense to use the same solution
> > > that
> > > > we
> > > > > > > will
> > > > > > > > > use
> > > > > > > > > > > for
> > > > > > > > > > > > >> > having
> > > > > > > > > > > > >> > > a
> > > > > > > > > > > > >> > > > single-language REPL to have a
> single-language
> > > > CLI?
> > > > > My
> > > > > > > > only
> > > > > > > > > > > > concern
> > > > > > > > > > > > >> > with
> > > > > > > > > > > > >> > > > killing the CLI is that you remove an
> > essential
> > > > > > feature
> > > > > > > > for
> > > > > > > > > > > > DevOps.
> > > > > > > > > > > > >> > > >
> > > > > > > > > > > > >> > > > - Lucas
> > > > > > > > > > > > >> > > >
> > > > > > > > > > > > >> > > > On Fri, Oct 19, 2018 at 1:52 PM Rafael
> > Novello <
> > > > > > > > > > > > >> > > > rafa.reis.novello@gmail.com>
> > > > > > > > > > > > >> > > > wrote:
> > > > > > > > > > > > >> > > >
> > > > > > > > > > > > >> > > > > Lucas,
> > > > > > > > > > > > >> > > > >
> > > > > > > > > > > > >> > > > > The idea is that REPL will substitute the
> > > actual
> > > > > > CLI.
> > > > > > > > It's
> > > > > > > > > > > > because
> > > > > > > > > > > > >> > with
> > > > > > > > > > > > >> > > > the
> > > > > > > > > > > > >> > > > > actual concept (using language specific
> CLI)
> > > we
> > > > > will
> > > > > > > > need
> > > > > > > > > to
> > > > > > > > > > > > >> > > re-implement
> > > > > > > > > > > > >> > > > > the same features many times and probably
> > each
> > > > CLI
> > > > > > > will
> > > > > > > > > > have a
> > > > > > > > > > > > >> > > different
> > > > > > > > > > > > >> > > > > behavior because some language specific
> > > > > restrictions
> > > > > > > > > and/or
> > > > > > > > > > > > >> > > limitations.
> > > > > > > > > > > > >> > > > >
> > > > > > > > > > > > >> > > > > With this new concept, all users will have
> > the
> > > > > same
> > > > > > > > > > experience
> > > > > > > > > > > > >> > > > interacting
> > > > > > > > > > > > >> > > > > with Marvin REPL and they will use the
> bests
> > > > tools
> > > > > > to
> > > > > > > > > > develop
> > > > > > > > > > > > >> their
> > > > > > > > > > > > >> > > > engines
> > > > > > > > > > > > >> > > > > (Jupyter Notebook and/or Lab with
> different
> > > > > > languages
> > > > > > > > > > support
> > > > > > > > > > > or
> > > > > > > > > > > > >> even
> > > > > > > > > > > > >> > > > > Apache Zeppelin). All the interact will
> > occur
> > > > > > through
> > > > > > > > REPL
> > > > > > > > > > and
> > > > > > > > > > > > >> docker
> > > > > > > > > > > > >> > > > > protocol.
> > > > > > > > > > > > >> > > > >
> > > > > > > > > > > > >> > > > > Alan, Yes! As Lucas said, the concept is
> the
> > > > same
> > > > > > but
> > > > > > > we
> > > > > > > > > can
> > > > > > > > > > > use
> > > > > > > > > > > > >> > docker
> > > > > > > > > > > > >> > > > to
> > > > > > > > > > > > >> > > > > do the same job.
> > > > > > > > > > > > >> > > > >
> > > > > > > > > > > > >> > > > > Best!
> > > > > > > > > > > > >> > > > >
> > > > > > > > > > > > >> > > > > Em sex, 19 de out de 2018 às 00:39, Lucas
> > > > Bonatto
> > > > > > > > Miguel <
> > > > > > > > > > > > >> > > > > lucasbm88@apache.org> escreveu:
> > > > > > > > > > > > >> > > > >
> > > > > > > > > > > > >> > > > > > Thanks for the clarifications Rafael, so
> > > from
> > > > > > what I
> > > > > > > > > > > > understood,
> > > > > > > > > > > > >> > > > Marvin's
> > > > > > > > > > > > >> > > > > > developers would not use the REPL to
> > explore
> > > > > data
> > > > > > > and
> > > > > > > > > test
> > > > > > > > > > > > >> models,
> > > > > > > > > > > > >> > > i.e.
> > > > > > > > > > > > >> > > > > > develop the engine. Is the idea to build
> > > > > something
> > > > > > > > more
> > > > > > > > > > like
> > > > > > > > > > > > an
> > > > > > > > > > > > >> > > > > interactive
> > > > > > > > > > > > >> > > > > > CLI? I think an interactive CLI would be
> > > > useful
> > > > > > for
> > > > > > > > the
> > > > > > > > > > > > >> developer
> > > > > > > > > > > > >> > > > > > experience, however, it's important to
> > keep
> > > > the
> > > > > > > > > unattended
> > > > > > > > > > > > >> > (current)
> > > > > > > > > > > > >> > > > CLI
> > > > > > > > > > > > >> > > > > in
> > > > > > > > > > > > >> > > > > > place for scenarios where users want to
> > > > automate
> > > > > > > > Marvin
> > > > > > > > > > > tasks.
> > > > > > > > > > > > >> > > > > >
> > > > > > > > > > > > >> > > > > > Alan, I believe the idea is still the
> same
> > > as
> > > > > you
> > > > > > > > > started,
> > > > > > > > > > > but
> > > > > > > > > > > > >> > using
> > > > > > > > > > > > >> > > > > > docker SDK now.
> > > > > > > > > > > > >> > > > > >
> > > > > > > > > > > > >> > > > > > - Lucas
> > > > > > > > > > > > >> > > > > >
> > > > > > > > > > > > >> > > > > > On Thu, Oct 18, 2018 at 1:29 PM Rafael
> > > > Novello <
> > > > > > > > > > > > >> > > > > > rafa.reis.novello@gmail.com>
> > > > > > > > > > > > >> > > > > > wrote:
> > > > > > > > > > > > >> > > > > >
> > > > > > > > > > > > >> > > > > > > Hi Lucas!
> > > > > > > > > > > > >> > > > > > >
> > > > > > > > > > > > >> > > > > > > First of all +1 for REPL POCs to
> Apache!
> > > > > > > > > > > > >> > > > > > >
> > > > > > > > > > > > >> > > > > > > Let me help with some comments:
> > > > > > > > > > > > >> > > > > > >
> > > > > > > > > > > > >> > > > > > > 1 - We have tested NodeJS, Scala and
> > > Python
> > > > > and
> > > > > > > the
> > > > > > > > > > > easiest
> > > > > > > > > > > > >> one
> > > > > > > > > > > > >> > was
> > > > > > > > > > > > >> > > > > > Python.
> > > > > > > > > > > > >> > > > > > > We have found a small project [1] that
> > > have
> > > > > all
> > > > > > > > > features
> > > > > > > > > > > we
> > > > > > > > > > > > >> > desired
> > > > > > > > > > > > >> > > > for
> > > > > > > > > > > > >> > > > > > > REPL:
> > > > > > > > > > > > >> > > > > > > - Autocomplete
> > > > > > > > > > > > >> > > > > > > - Python commands disabled (the user
> > have
> > > > only
> > > > > > the
> > > > > > > > > > > commands
> > > > > > > > > > > > we
> > > > > > > > > > > > >> > > > > provide).
> > > > > > > > > > > > >> > > > > > > The other languages REPL options don't
> > > have
> > > > > this
> > > > > > > > > > feature.
> > > > > > > > > > > > >> > > > > > > - Easy to show output
> > > > > > > > > > > > >> > > > > > > - Etc
> > > > > > > > > > > > >> > > > > > > So, I think the language chosen will
> not
> > > be
> > > > > > > > important
> > > > > > > > > > here
> > > > > > > > > > > > >> > because
> > > > > > > > > > > > >> > > > the
> > > > > > > > > > > > >> > > > > > user
> > > > > > > > > > > > >> > > > > > > will only interact with the commands
> > that
> > > we
> > > > > > > create.
> > > > > > > > > > > > >> > > > > > >
> > > > > > > > > > > > >> > > > > > > 2 - The "engine-generate" command will
> > > > > download
> > > > > > a
> > > > > > > > > docker
> > > > > > > > > > > > image
> > > > > > > > > > > > >> > that
> > > > > > > > > > > > >> > > > we
> > > > > > > > > > > > >> > > > > > > create for that language and start a
> > > > container
> > > > > > > with
> > > > > > > > > the
> > > > > > > > > > > > basic
> > > > > > > > > > > > >> > > project
> > > > > > > > > > > > >> > > > > > > structure for that language.
> > > > > > > > > > > > >> > > > > > >
> > > > > > > > > > > > >> > > > > > > 3 - The REPL client will use the
> "docker
> > > > > > protocol"
> > > > > > > > to
> > > > > > > > > > run
> > > > > > > > > > > > >> > command,
> > > > > > > > > > > > >> > > > > > > start/stop services and etc inside the
> > > > > container
> > > > > > > and
> > > > > > > > > it
> > > > > > > > > > > will
> > > > > > > > > > > > >> > > receive
> > > > > > > > > > > > >> > > > > the
> > > > > > > > > > > > >> > > > > > > log stream to show. No, the REPL will
> no
> > > > pass
> > > > > > code
> > > > > > > > > > > snippets
> > > > > > > > > > > > >> for
> > > > > > > > > > > > >> > > > docker
> > > > > > > > > > > > >> > > > > > > container (I think it will not be
> > > necessary)
> > > > > > > > > > > > >> > > > > > >
> > > > > > > > > > > > >> > > > > > > 4 - Yep! Like I said on the first
> item.
> > > > > > > > > > > > >> > > > > > >
> > > > > > > > > > > > >> > > > > > > [1] -
> > > https://github.com/italorossi/ishell
> > > > > > > > > > > > >> > > > > > >
> > > > > > > > > > > > >> > > > > > > Let me know if there is any other
> > > question!
> > > > > > > > > > > > >> > > > > > > Best!
> > > > > > > > > > > > >> > > > > > > Rafael J. R. Novello
> > > > > > > > > > > > >> > > > > > >
> > > > > > > > > > > > >> > > > > > > Skype: rafael.novello
> > > > > > > > > > > > >> > > > > > > Blog:
> > http://rafanovello.blogspot.com.br/
> > > > > > > > > > > > >> > > > > > >
> > > > > > > > > > > > >> > > > > > >
> > > > > > > > > > > > >> > > > > > > Em qui, 18 de out de 2018 às 17:00,
> > Lucas
> > > > > > Bonatto
> > > > > > > > > > Miguel <
> > > > > > > > > > > > >> > > > > > > lucasbm88@apache.org> escreveu:
> > > > > > > > > > > > >> > > > > > >
> > > > > > > > > > > > >> > > > > > > > + 1 for migrating the REPL repo to
> > > Apache
> > > > > > > > > > > > >> > > > > > > >
> > > > > > > > > > > > >> > > > > > > > I have a few questions about the
> > > previous
> > > > > > > > > explanation:
> > > > > > > > > > > > >> > > > > > > >  1) The REPL itself, it would be an
> > > > > > application
> > > > > > > in
> > > > > > > > > > which
> > > > > > > > > > > > >> > > language?
> > > > > > > > > > > > >> > > > > > > Remember
> > > > > > > > > > > > >> > > > > > > > that the main idea is to allow the
> > user
> > > to
> > > > > > > program
> > > > > > > > > on
> > > > > > > > > > > his
> > > > > > > > > > > > >> > > preferred
> > > > > > > > > > > > >> > > > > > > > language in the REPL.
> > > > > > > > > > > > >> > > > > > > >  2) Should the engine-generate
> command
> > > > also
> > > > > > > > > generate a
> > > > > > > > > > > > >> docker
> > > > > > > > > > > > >> > > image
> > > > > > > > > > > > >> > > > > > with
> > > > > > > > > > > > >> > > > > > > > the user's application?
> > > > > > > > > > > > >> > > > > > > >  3) What type of communication would
> > > > happen
> > > > > > > > between
> > > > > > > > > > the
> > > > > > > > > > > > REPL
> > > > > > > > > > > > >> > and
> > > > > > > > > > > > >> > > > the
> > > > > > > > > > > > >> > > > > > > engine
> > > > > > > > > > > > >> > > > > > > > via Docker SDK? Would the REPL pass
> > > > snippets
> > > > > > of
> > > > > > > > code
> > > > > > > > > > to
> > > > > > > > > > > be
> > > > > > > > > > > > >> > > executed
> > > > > > > > > > > > >> > > > > by
> > > > > > > > > > > > >> > > > > > > the
> > > > > > > > > > > > >> > > > > > > > docker container?
> > > > > > > > > > > > >> > > > > > > >  4) Have you considered code
> > completion
> > > in
> > > > > the
> > > > > > > > REPL?
> > > > > > > > > > > > >> > > > > > > >
> > > > > > > > > > > > >> > > > > > > >
> > > > > > > > > > > > >> > > > > > > > On Thu, Oct 18, 2018 at 7:53 AM
> Zhang
> > > > Yifei
> > > > > <
> > > > > > > > > > > > >> > > > yifei.z.luiz@gmail.com>
> > > > > > > > > > > > >> > > > > > > > wrote:
> > > > > > > > > > > > >> > > > > > > >
> > > > > > > > > > > > >> > > > > > > > > Ok guys,
> > > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > > >> > > > > > > > > The basic idea is to provide only
> > one
> > > > > > Toolbox
> > > > > > > > for
> > > > > > > > > > > > multiple
> > > > > > > > > > > > >> > > > > languages.
> > > > > > > > > > > > >> > > > > > > > > We are looking for possibility to
> > > build
> > > > a
> > > > > > > single
> > > > > > > > > > > Marvin
> > > > > > > > > > > > >> Repl,
> > > > > > > > > > > > >> > > > > > > > > instead of severals toolboxes with
> > > > > > differentes
> > > > > > > > > > > > interfaces
> > > > > > > > > > > > >> or
> > > > > > > > > > > > >> > > > > > commands.
> > > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > > >> > > > > > > > > In this case, the engine-generate
> > > > command
> > > > > > will
> > > > > > > > > > > download
> > > > > > > > > > > > >> and
> > > > > > > > > > > > >> > > > start a
> > > > > > > > > > > > >> > > > > > > > Docker
> > > > > > > > > > > > >> > > > > > > > > container with basic engine
> > structure
> > > > > > > > > corresponding
> > > > > > > > > > > the
> > > > > > > > > > > > >> > choosed
> > > > > > > > > > > > >> > > > > > > language.
> > > > > > > > > > > > >> > > > > > > > > this means we don't need to build
> > > > > Toolboxes
> > > > > > of
> > > > > > > > > > > > differents
> > > > > > > > > > > > >> > > > > languages,
> > > > > > > > > > > > >> > > > > > we
> > > > > > > > > > > > >> > > > > > > > > just
> > > > > > > > > > > > >> > > > > > > > > build the engine template of all
> > > > languages
> > > > > > > that
> > > > > > > > we
> > > > > > > > > > > want
> > > > > > > > > > > > to
> > > > > > > > > > > > >> > > > support
> > > > > > > > > > > > >> > > > > > and
> > > > > > > > > > > > >> > > > > > > > > provide it
> > > > > > > > > > > > >> > > > > > > > > as Docker containers
> > > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > > >> > > > > > > > > We have started researches around
> > our
> > > > > basic
> > > > > > > > > > > requirements
> > > > > > > > > > > > >> > like:
> > > > > > > > > > > > >> > > > > > > > > - Repl interface
> > > > > > > > > > > > >> > > > > > > > > - System communication
> > > > > > > > > > > > >> > > > > > > > > - Connection security
> > > > > > > > > > > > >> > > > > > > > > - Tool popularity
> > > > > > > > > > > > >> > > > > > > > > - Update complexity
> > > > > > > > > > > > >> > > > > > > > > - Languages support
> > > > > > > > > > > > >> > > > > > > > > - ......
> > > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > > >> > > > > > > > > And we did some POC with code
> here:
> > > > > > > > > > > > >> > > > > > > > >
> > > > https://github.com/marvin-ai/marvin-repl
> > > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > > >> > > > > > > > > There is POC testing gRPC using
> > Scala
> > > > and
> > > > > > > > Python,
> > > > > > > > > > > > >> > > > > > > > > Repl inteface and Docker SDK with
> > > > NodeJS,
> > > > > > > > > > > > >> > > > > > > > > Repl interface and Docker SDK with
> > > > Python.
> > > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > > >> > > > > > > > > At this moment we prefer the Repl
> > > > > interface
> > > > > > +
> > > > > > > > > Docker
> > > > > > > > > > > SDK
> > > > > > > > > > > > >> way,
> > > > > > > > > > > > >> > > > > because
> > > > > > > > > > > > >> > > > > > > > good
> > > > > > > > > > > > >> > > > > > > > > part of the requirements
> > > > > > > > > > > > >> > > > > > > > > will be guaranteed by Docker.
> > > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > > >> > > > > > > > > With this informations, what do
> you
> > > > think?
> > > > > > > > Should
> > > > > > > > > we
> > > > > > > > > > > > >> submit
> > > > > > > > > > > > >> > all
> > > > > > > > > > > > >> > > > > this
> > > > > > > > > > > > >> > > > > > > POCs
> > > > > > > > > > > > >> > > > > > > > > to Apache Repo?
> > > > > > > > > > > > >> > > > > > > > > Please feel free to opine.
> > > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > > >> > > > > > > > > Thats all, thanks!!!
> > > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > > >> > > > > > > > > Em ter, 16 de out de 2018 às
> 18:55,
> > > > Daniel
> > > > > > > > > > > Takabayashi <
> > > > > > > > > > > > >> > > > > > > > > daniel.takabayashi@gmail.com>
> > > escreveu:
> > > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > > >> > > > > > > > > > Zhang,
> > > > > > > > > > > > >> > > > > > > > > >
> > > > > > > > > > > > >> > > > > > > > > > I think the best approach is
> give
> > > us a
> > > > > > > better
> > > > > > > > > > > > >> explanation
> > > > > > > > > > > > >> > > about
> > > > > > > > > > > > >> > > > > > this
> > > > > > > > > > > > >> > > > > > > > new
> > > > > > > > > > > > >> > > > > > > > > > feature and how this can help us
> > to
> > > > > > archive
> > > > > > > > the
> > > > > > > > > > > > desired
> > > > > > > > > > > > >> > > feature
> > > > > > > > > > > > >> > > > > > > > (support
> > > > > > > > > > > > >> > > > > > > > > > multiple languages). Than if the
> > > > > majority
> > > > > > > > agree
> > > > > > > > > > > than I
> > > > > > > > > > > > >> can
> > > > > > > > > > > > >> > do
> > > > > > > > > > > > >> > > > the
> > > > > > > > > > > > >> > > > > > > > merge,
> > > > > > > > > > > > >> > > > > > > > > > just like I did before, but in
> > > another
> > > > > > > branch.
> > > > > > > > > > > > >> > > > > > > > > >
> > > > > > > > > > > > >> > > > > > > > > > Let me know if makes sense, ok?
> > > > > > > > > > > > >> > > > > > > > > >
> > > > > > > > > > > > >> > > > > > > > > > Taka
> > > > > > > > > > > > >> > > > > > > > > >
> > > > > > > > > > > > >> > > > > > > > > > Em ter, 16 de out de 2018 às
> > 14:00,
> > > > > > Luciano
> > > > > > > > > > Resende
> > > > > > > > > > > <
> > > > > > > > > > > > >> > > > > > > > > luckbr1975@gmail.com>
> > > > > > > > > > > > >> > > > > > > > > > escreveu:
> > > > > > > > > > > > >> > > > > > > > > >
> > > > > > > > > > > > >> > > > > > > > > > > So, what is the POC, is it a
> > > > > refactoring
> > > > > > > of
> > > > > > > > > the
> > > > > > > > > > > > >> existing
> > > > > > > > > > > > >> > > > repo?
> > > > > > > > > > > > >> > > > > Or
> > > > > > > > > > > > >> > > > > > > is
> > > > > > > > > > > > >> > > > > > > > > > > it a new rewrite of the repo?
> > > > > > > > > > > > >> > > > > > > > > > >
> > > > > > > > > > > > >> > > > > > > > > > > Just asking as it might make
> > sense
> > > > to
> > > > > > make
> > > > > > > > it
> > > > > > > > > a
> > > > > > > > > > > > branch
> > > > > > > > > > > > >> > then
> > > > > > > > > > > > >> > > > > > > actually
> > > > > > > > > > > > >> > > > > > > > a
> > > > > > > > > > > > >> > > > > > > > > > > thing in parallel, as this
> will
> > > have
> > > > > an
> > > > > > > > effect
> > > > > > > > > > for
> > > > > > > > > > > > >> > > releases,
> > > > > > > > > > > > >> > > > > etc.
> > > > > > > > > > > > >> > > > > > > but
> > > > > > > > > > > > >> > > > > > > > > > > you guys know more here than I
> > do.
> > > > > > > > > > > > >> > > > > > > > > > >
> > > > > > > > > > > > >> > > > > > > > > > > Also, it's probably good to
> > have a
> > > > > write
> > > > > > > up
> > > > > > > > of
> > > > > > > > > > the
> > > > > > > > > > > > >> main
> > > > > > > > > > > > >> > > > > direction
> > > > > > > > > > > > >> > > > > > > of
> > > > > > > > > > > > >> > > > > > > > > > > the design that can help
> people
> > > get
> > > > > > > familiar
> > > > > > > > > > with
> > > > > > > > > > > > the
> > > > > > > > > > > > >> new
> > > > > > > > > > > > >> > > > > > approach.
> > > > > > > > > > > > >> > > > > > > > > > > On Tue, Oct 16, 2018 at 11:12
> AM
> > > > Zhang
> > > > > > > > Yifei <
> > > > > > > > > > > > >> > > > > > > yifei.z.luiz@gmail.com
> > > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > > >> > > > > > > > > > > wrote:
> > > > > > > > > > > > >> > > > > > > > > > > >
> > > > > > > > > > > > >> > > > > > > > > > > > Hey guys,
> > > > > > > > > > > > >> > > > > > > > > > > >
> > > > > > > > > > > > >> > > > > > > > > > > > We have reorganized the Poc
> > > repo,
> > > > > and
> > > > > > > want
> > > > > > > > > to
> > > > > > > > > > > > merge
> > > > > > > > > > > > >> it
> > > > > > > > > > > > >> > to
> > > > > > > > > > > > >> > > > our
> > > > > > > > > > > > >> > > > > > > > Apache
> > > > > > > > > > > > >> > > > > > > > > > > repo.
> > > > > > > > > > > > >> > > > > > > > > > > > Just making sure here before
> > we
> > > do
> > > > > the
> > > > > > > > > merge,
> > > > > > > > > > > > >> because
> > > > > > > > > > > > >> > im
> > > > > > > > > > > > >> > > > not
> > > > > > > > > > > > >> > > > > > sure
> > > > > > > > > > > > >> > > > > > > > if
> > > > > > > > > > > > >> > > > > > > > > > > doing
> > > > > > > > > > > > >> > > > > > > > > > > > this will perserve the Git
> > > commit
> > > > > > > history.
> > > > > > > > > > > > >> > > > > > > > > > > > What we are planning to do
> is:
> > > > > > > > > > > > >> > > > > > > > > > > >
> > > > > > > > > > > > >> > > > > > > > > > > > git filter-branch
> > > > > > --subdirectory-filter
> > > > > > > > <git
> > > > > > > > > > > > >> > > > > origin_repository
> > > > > > > > > > > > >> > > > > > > > > > directory>
> > > > > > > > > > > > >> > > > > > > > > > > > -- --all
> > > > > > > > > > > > >> > > > > > > > > > > > mkdir POCs/
> > > > > > > > > > > > >> > > > > > > > > > > > git mv * POCs/
> > > > > > > > > > > > >> > > > > > > > > > > > git commit -m "colleted the
> > data
> > > > to
> > > > > > > move"
> > > > > > > > > > > > >> > > > > > > > > > > >
> > > > > > > > > > > > >> > > > > > > > > > > > git clone <git
> > > > > > > destination_repository_url>
> > > > > > > > > cd
> > > > > > > > > > > <git
> > > > > > > > > > > > >> > > > > > > > > > > >
> > > destination_repository_directory>
> > > > > git
> > > > > > > > remote
> > > > > > > > > > add
> > > > > > > > > > > > >> > > > > > > > > > origin_repository_branch
> > > > > > > > > > > > >> > > > > > > > > > > > <git origin_repository
> > > directory>
> > > > > git
> > > > > > > pull
> > > > > > > > > > > > >> > > > > > > origin_repository_branch
> > > > > > > > > > > > >> > > > > > > > > > > master
> > > > > > > > > > > > >> > > > > > > > > > > > --allow-unrelated-histories
> > git
> > > > > remote
> > > > > > > rm
> > > > > > > > > > > > >> > > > > > > origin_repository_branch
> > > > > > > > > > > > >> > > > > > > > > > > >
> > > > > > > > > > > > >> > > > > > > > > > > >
> > > > > > > > > > > > >> > > > > > > > > > > > Thanks in advance !
> > > > > > > > > > > > >> > > > > > > > > > > >
> > > > > > > > > > > > >> > > > > > > > > > > >
> > > > > > > > > > > > >> > > > >
> > > > > > > > > >
> > > --------------------------------------------------------------
> > > > > > > > > > > > >> > > > > > > > > > > > Zhang Yifei
> > > > > > > > > > > > >> > > > > > > > > > >
> > > > > > > > > > > > >> > > > > > > > > > >
> > > > > > > > > > > > >> > > > > > > > > > >
> > > > > > > > > > > > >> > > > > > > > > > > --
> > > > > > > > > > > > >> > > > > > > > > > > Luciano Resende
> > > > > > > > > > > > >> > > > > > > > > > >
> http://twitter.com/lresende1975
> > > > > > > > > > > > >> > > > > > > > > > > http://lresende.blogspot.com/
> > > > > > > > > > > > >> > > > > > > > > > >
> > > > > > > > > > > > >> > > > > > > > > >
> > > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > > >> > > > > > > > > --
> > > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > > >> >
> > > > > > > --------------------------------------------------------------
> > > > > > > > > > > > >> > > > > > > > > Zhang Yifei
> > > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > > >> > > > > > > >
> > > > > > > > > > > > >> > > > > > >
> > > > > > > > > > > > >> > > > > >
> > > > > > > > > > > > >> > > > >
> > > > > > > > > > > > >> > > >
> > > > > > > > > > > > >> > >
> > > > > > > > > > > > >> >
> > > > > > > > > > > > >>
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: POCs, Prototypes, experiments ...

Posted by Lucas Bonatto Miguel <lu...@apache.org>.

That's a good point Rafael. I believe the main motivation for adding a REPL
to Marvin's toolbox is to enhance user experience.

Imo we can't just switch CLI per REPL. I don't think one excludes the
other, both have different use cases.

We may be mixing too much stuff together, but in general, I think we are:
 - Refactoring toolbox architecture to avoid unnecessarily duplicated logic
when adding support to new languages.
 - Adding a new interface to enable users to open an interactive session
with the engine (i.e. REPL)
 - Making it easier to add support for new languages.

Does it make sense for you?

On Mon, Oct 29, 2018 at 11:22 AM Rafael Novello <ra...@gmail.com>
wrote:

> I do not want to be the annoying conversation but...
>
> If we will have an unique CLI that will be able to handle any kind of
> engines (in any language) by using docker, Why we need a REPL?
>
> I think we can focus on creating this new architecture using docker and
> docker SDK and use only a CLI for that.
>
> Atenciosamente,
> Rafael J. R. Novello
>
> Skype: rafael.novello
> Blog: http://rafanovello.blogspot.com.br/
>
>
> Em seg, 29 de out de 2018 às 14:58, Lucas Bonatto Miguel <
> lucasbm88@apache.org> escreveu:
>
> > No, I think the idea is to do not create different versions of the CLI.
> > Let's use one single tech stack for the CLI, ideally, it will be shared
> > with the REPL. The way I see the CLI would be very similar to the REPL,
> but
> > instead of opening an interactive session the user could send
> > unattended commands. As an analogy, you can think on spark-shell (our
> REPL)
> > and spark-submit (our CLI).
> >
> > - Lucas
> >
> > On Mon, Oct 29, 2018 at 10:27 AM Rafael Novello <
> > rafa.reis.novello@gmail.com>
> > wrote:
> >
> > > Hi Guys!
> > >
> > > Let me ask about this new version that include CLI's and REPL. Will we
> > > create different versions of CLI?
> > >
> > > One of the reasons to create the REPL is to avoid re-implementing the
> > same
> > > features many times,
> > > each time for a different language.
> > >
> > > Regards!
> > > Atenciosamente,
> > > Rafael J. R. Novello
> > >
> > > Skype: rafael.novello
> > > Blog: http://rafanovello.blogspot.com.br/
> > >
> > >
> > > Em sex, 26 de out de 2018 às 20:54, Daniel Takabayashi <
> > > daniel.takabayashi@gmail.com> escreveu:
> > >
> > > > +1
> > > >
> > > > Em sex, 26 de out de 2018 às 16:47, Lucas Bonatto Miguel <
> > > > lucasbm88@apache.org> escreveu:
> > > >
> > > > > Makes perfect sense to me. I believe that if the majority of people
> > > > agrees
> > > > > with this architecture we could create issues and start
> implementing
> > > it.
> > > > >
> > > > > +1
> > > > >
> > > > >
> > > > > On Fri, Oct 26, 2018 at 4:09 PM Daniel Takabayashi <
> > > > > daniel.takabayashi@gmail.com> wrote:
> > > > >
> > > > > > I see... what about this new version?
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://docs.google.com/drawings/d/shajxIpLJHxxMbFgDXiPuhg/image?w=602&h=468&rev=1630&ac=1&parent=1ySERHGBXbHeyCMRookq5UfTuFkzzU0ugtjvR3rF3deY
> > > > > >
> > > > > > I changed the toolbox ...from 1 component to 3... docker client,
> > cli
> > > > and
> > > > > > repl. Both (cli and repl) now are using the docker client to
> > control
> > > > the
> > > > > > engines.
> > > > > >
> > > > > > Em sex, 26 de out de 2018 às 14:49, Lucas Bonatto Miguel <
> > > > > > lucasbm88@apache.org> escreveu:
> > > > > >
> > > > > > > For sure, both suggestions are to make sure we're not killing
> the
> > > CLI
> > > > > and
> > > > > > > that we have a component (the SDK client) that is shared by CLI
> > and
> > > > > REPL.
> > > > > > >
> > > > > > > Thanks for clarifying where the system calls will live.
> > > > > > >
> > > > > > > - Lucas
> > > > > > >
> > > > > > > On Fri, Oct 26, 2018 at 2:39 PM Daniel Takabayashi <
> > > > > > > daniel.takabayashi@gmail.com> wrote:
> > > > > > >
> > > > > > > > Hi Lucas, could you please justified your two suggestions?
> > > > > > > >
> > > > > > > > The general ideia is to use the Docker DSK to execute local
> > > > commands,
> > > > > > to
> > > > > > > > consume outputs and also to execute Marvin "specific
> language"
> > > > > > commands.
> > > > > > > > Today almost all CLIs commands in Marvin is a mixed of
> systems
> > > > > commands
> > > > > > > and
> > > > > > > > library method calls. The plan is to put all "specific
> language
> > > > > calls"
> > > > > > > > inside the marvin-common-libs and group the systems call on
> the
> > > > > toolbox
> > > > > > > > (using the docker SDK in this task).
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > Em qui, 25 de out de 2018 às 20:09, Lucas Bonatto Miguel <
> > > > > > > > lucasbm88@apache.org> escreveu:
> > > > > > > >
> > > > > > > > > Great, thanks.
> > > > > > > > >
> > > > > > > > > So here some suggestions:
> > > > > > > > >  1. We could create one new layer between Toolbox and the
> > > > Engines,
> > > > > > this
> > > > > > > > > would be Docker client.
> > > > > > > > >  2. Split the Toolbox layer in two. Toolbox CLI and Toolbox
> > > REPL
> > > > > > > > >
> > > > > > > > > It's not clear to me if the commands listed on General
> > Commands
> > > > > will
> > > > > > > need
> > > > > > > > > to be written for every language, from the image I believe
> > not,
> > > > but
> > > > > > how
> > > > > > > > it
> > > > > > > > > would work, since starting a notebook in Python is
> different
> > > than
> > > > > > > > starting
> > > > > > > > > in Scala.
> > > > > > > > >
> > > > > > > > > - Lucas
> > > > > > > > > On Thu, Oct 25, 2018 at 4:51 PM Daniel Takabayashi <
> > > > > > > > > daniel.takabayashi@gmail.com> wrote:
> > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://docs.google.com/drawings/d/shajxIpLJHxxMbFgDXiPuhg/image?w=602&h=461&rev=1423&ac=1&parent=1ySERHGBXbHeyCMRookq5UfTuFkzzU0ugtjvR3rF3deY
> > > > > > > > > >
> > > > > > > > > > Em qui, 25 de out de 2018 às 07:38, Rafael Novello <
> > > > > > > > > > rafa.reis.novello@gmail.com> escreveu:
> > > > > > > > > >
> > > > > > > > > > > Hi Taka,
> > > > > > > > > > >
> > > > > > > > > > > I can't see the image! Maybe the Apache email list
> don't
> > > > > support
> > > > > > > > > > > attachments.
> > > > > > > > > > >
> > > > > > > > > > > Best!
> > > > > > > > > > > Rafael J. R. Novello
> > > > > > > > > > >
> > > > > > > > > > > Skype: rafael.novello
> > > > > > > > > > > Blog: http://rafanovello.blogspot.com.br/
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Em qua, 24 de out de 2018 às 21:14, Daniel Takabayashi
> <
> > > > > > > > > > > daniel.takabayashi@gmail.com> escreveu:
> > > > > > > > > > >
> > > > > > > > > > > > Hi all,
> > > > > > > > > > > >
> > > > > > > > > > > > To try to give a little more details about this "new
> > > > concept"
> > > > > > > that
> > > > > > > > we
> > > > > > > > > > > want
> > > > > > > > > > > > to bring to the marvin toolbox, I did this simple
> > > > > architecture
> > > > > > > > draw.
> > > > > > > > > > > >
> > > > > > > > > > > > [image: marvin-architecture-views (5).png]
> > > > > > > > > > > >
> > > > > > > > > > > > The general idea here is try to transform the toolbox
> > > > > something
> > > > > > > > > > > > disconnected with the "language", something more
> > > agnostic.
> > > > > Also
> > > > > > > in
> > > > > > > > > this
> > > > > > > > > > > new
> > > > > > > > > > > > architecture we could use remote resource to process
> > > > engines
> > > > > > and
> > > > > > > > make
> > > > > > > > > > > easy
> > > > > > > > > > > > the support for new languages.
> > > > > > > > > > > >
> > > > > > > > > > > > This "new toolbox" will be the only thing that a
> Marvin
> > > > user
> > > > > > must
> > > > > > > > to
> > > > > > > > > > > > install and also we could start to support multiples
> > O.S
> > > > > (once
> > > > > > > the
> > > > > > > > > REPL
> > > > > > > > > > > is
> > > > > > > > > > > > a dummy application that only interprets and by pass
> > > > > commands).
> > > > > > > > > > > >
> > > > > > > > > > > > Regards,
> > > > > > > > > > > > Taka
> > > > > > > > > > > >
> > > > > > > > > > > > Em qua, 24 de out de 2018 às 09:52, Rafael Novello <
> > > > > > > > > > > > rafa.reis.novello@gmail.com> escreveu:
> > > > > > > > > > > >
> > > > > > > > > > > >> Alan,
> > > > > > > > > > > >>
> > > > > > > > > > > >> Yes! We are using the Docker SDK and it's possible
> to
> > > use
> > > > > the
> > > > > > > API
> > > > > > > > to
> > > > > > > > > > > >> automate, but it's a little bit harder than automate
> > CLI
> > > > > > calls.
> > > > > > > > > > > >>
> > > > > > > > > > > >> Atenciosamente,
> > > > > > > > > > > >> Rafael J. R. Novello
> > > > > > > > > > > >>
> > > > > > > > > > > >> Skype: rafael.novello
> > > > > > > > > > > >> Blog: http://rafanovello.blogspot.com.br/
> > > > > > > > > > > >>
> > > > > > > > > > > >>
> > > > > > > > > > > >> Em qua, 24 de out de 2018 às 12:02, Alan Silva <
> > > > > > > jumpi@apache.org>
> > > > > > > > > > > >> escreveu:
> > > > > > > > > > > >>
> > > > > > > > > > > >> > Hi,
> > > > > > > > > > > >> >
> > > > > > > > > > > >> > One question here, I understand that we start to
> use
> > > > with
> > > > > > this
> > > > > > > > PoC
> > > > > > > > > > the
> > > > > > > > > > > >> > Docker SDK API, right?
> > > > > > > > > > > >> >
> > > > > > > > > > > >> > Why not use the API to expose some endpoints to
> > permit
> > > > > this
> > > > > > > kind
> > > > > > > > > of
> > > > > > > > > > > >> > automation by devops?
> > > > > > > > > > > >> >
> > > > > > > > > > > >> > I think it is possible and it solves the CLI
> > problem,
> > > > > right?
> > > > > > > > > > > >> >
> > > > > > > > > > > >> >
> > > > > > > > > > > >> > On Tue, Oct 23, 2018 at 1:05 PM Rafael Novello <
> > > > > > > > > > > >> > rafa.reis.novello@gmail.com>
> > > > > > > > > > > >> > wrote:
> > > > > > > > > > > >> >
> > > > > > > > > > > >> > > Lucas,
> > > > > > > > > > > >> > >
> > > > > > > > > > > >> > > Sorry, I didn't understood your question bellow.
> > > > > > > > > > > >> > >
> > > > > > > > > > > >> > > "Would it make sense to use the same solution
> that
> > > we
> > > > > will
> > > > > > > use
> > > > > > > > > for
> > > > > > > > > > > >> > having a
> > > > > > > > > > > >> > > single-language REPL to have a single-language
> > CLI?"
> > > > > > > > > > > >> > >
> > > > > > > > > > > >> > > For DevOps purposes, maybe this new toolbox
> > concept
> > > is
> > > > > not
> > > > > > > > > ideal.
> > > > > > > > > > I
> > > > > > > > > > > >> think
> > > > > > > > > > > >> > > we can keep the CLI inside the docker container
> > but
> > > it
> > > > > > will
> > > > > > > > not
> > > > > > > > > > easy
> > > > > > > > > > > >> to
> > > > > > > > > > > >> > > automate
> > > > > > > > > > > >> > > jobs by this way.
> > > > > > > > > > > >> > >
> > > > > > > > > > > >> > > How to deal with this issue? Voting?
> > > > > > > > > > > >> > > Atenciosamente,
> > > > > > > > > > > >> > > Rafael J. R. Novello
> > > > > > > > > > > >> > >
> > > > > > > > > > > >> > > Skype: rafael.novello
> > > > > > > > > > > >> > > Blog: http://rafanovello.blogspot.com.br/
> > > > > > > > > > > >> > >
> > > > > > > > > > > >> > >
> > > > > > > > > > > >> > > Em sex, 19 de out de 2018 às 19:00, Lucas
> Bonatto
> > > > > Miguel <
> > > > > > > > > > > >> > > lucasbm88@apache.org> escreveu:
> > > > > > > > > > > >> > >
> > > > > > > > > > > >> > > > Got it! Thanks for clarifying.
> > > > > > > > > > > >> > > >
> > > > > > > > > > > >> > > > Would it make sense to use the same solution
> > that
> > > we
> > > > > > will
> > > > > > > > use
> > > > > > > > > > for
> > > > > > > > > > > >> > having
> > > > > > > > > > > >> > > a
> > > > > > > > > > > >> > > > single-language REPL to have a single-language
> > > CLI?
> > > > My
> > > > > > > only
> > > > > > > > > > > concern
> > > > > > > > > > > >> > with
> > > > > > > > > > > >> > > > killing the CLI is that you remove an
> essential
> > > > > feature
> > > > > > > for
> > > > > > > > > > > DevOps.
> > > > > > > > > > > >> > > >
> > > > > > > > > > > >> > > > - Lucas
> > > > > > > > > > > >> > > >
> > > > > > > > > > > >> > > > On Fri, Oct 19, 2018 at 1:52 PM Rafael
> Novello <
> > > > > > > > > > > >> > > > rafa.reis.novello@gmail.com>
> > > > > > > > > > > >> > > > wrote:
> > > > > > > > > > > >> > > >
> > > > > > > > > > > >> > > > > Lucas,
> > > > > > > > > > > >> > > > >
> > > > > > > > > > > >> > > > > The idea is that REPL will substitute the
> > actual
> > > > > CLI.
> > > > > > > It's
> > > > > > > > > > > because
> > > > > > > > > > > >> > with
> > > > > > > > > > > >> > > > the
> > > > > > > > > > > >> > > > > actual concept (using language specific CLI)
> > we
> > > > will
> > > > > > > need
> > > > > > > > to
> > > > > > > > > > > >> > > re-implement
> > > > > > > > > > > >> > > > > the same features many times and probably
> each
> > > CLI
> > > > > > will
> > > > > > > > > have a
> > > > > > > > > > > >> > > different
> > > > > > > > > > > >> > > > > behavior because some language specific
> > > > restrictions
> > > > > > > > and/or
> > > > > > > > > > > >> > > limitations.
> > > > > > > > > > > >> > > > >
> > > > > > > > > > > >> > > > > With this new concept, all users will have
> the
> > > > same
> > > > > > > > > experience
> > > > > > > > > > > >> > > > interacting
> > > > > > > > > > > >> > > > > with Marvin REPL and they will use the bests
> > > tools
> > > > > to
> > > > > > > > > develop
> > > > > > > > > > > >> their
> > > > > > > > > > > >> > > > engines
> > > > > > > > > > > >> > > > > (Jupyter Notebook and/or Lab with different
> > > > > languages
> > > > > > > > > support
> > > > > > > > > > or
> > > > > > > > > > > >> even
> > > > > > > > > > > >> > > > > Apache Zeppelin). All the interact will
> occur
> > > > > through
> > > > > > > REPL
> > > > > > > > > and
> > > > > > > > > > > >> docker
> > > > > > > > > > > >> > > > > protocol.
> > > > > > > > > > > >> > > > >
> > > > > > > > > > > >> > > > > Alan, Yes! As Lucas said, the concept is the
> > > same
> > > > > but
> > > > > > we
> > > > > > > > can
> > > > > > > > > > use
> > > > > > > > > > > >> > docker
> > > > > > > > > > > >> > > > to
> > > > > > > > > > > >> > > > > do the same job.
> > > > > > > > > > > >> > > > >
> > > > > > > > > > > >> > > > > Best!
> > > > > > > > > > > >> > > > >
> > > > > > > > > > > >> > > > > Em sex, 19 de out de 2018 às 00:39, Lucas
> > > Bonatto
> > > > > > > Miguel <
> > > > > > > > > > > >> > > > > lucasbm88@apache.org> escreveu:
> > > > > > > > > > > >> > > > >
> > > > > > > > > > > >> > > > > > Thanks for the clarifications Rafael, so
> > from
> > > > > what I
> > > > > > > > > > > understood,
> > > > > > > > > > > >> > > > Marvin's
> > > > > > > > > > > >> > > > > > developers would not use the REPL to
> explore
> > > > data
> > > > > > and
> > > > > > > > test
> > > > > > > > > > > >> models,
> > > > > > > > > > > >> > > i.e.
> > > > > > > > > > > >> > > > > > develop the engine. Is the idea to build
> > > > something
> > > > > > > more
> > > > > > > > > like
> > > > > > > > > > > an
> > > > > > > > > > > >> > > > > interactive
> > > > > > > > > > > >> > > > > > CLI? I think an interactive CLI would be
> > > useful
> > > > > for
> > > > > > > the
> > > > > > > > > > > >> developer
> > > > > > > > > > > >> > > > > > experience, however, it's important to
> keep
> > > the
> > > > > > > > unattended
> > > > > > > > > > > >> > (current)
> > > > > > > > > > > >> > > > CLI
> > > > > > > > > > > >> > > > > in
> > > > > > > > > > > >> > > > > > place for scenarios where users want to
> > > automate
> > > > > > > Marvin
> > > > > > > > > > tasks.
> > > > > > > > > > > >> > > > > >
> > > > > > > > > > > >> > > > > > Alan, I believe the idea is still the same
> > as
> > > > you
> > > > > > > > started,
> > > > > > > > > > but
> > > > > > > > > > > >> > using
> > > > > > > > > > > >> > > > > > docker SDK now.
> > > > > > > > > > > >> > > > > >
> > > > > > > > > > > >> > > > > > - Lucas
> > > > > > > > > > > >> > > > > >
> > > > > > > > > > > >> > > > > > On Thu, Oct 18, 2018 at 1:29 PM Rafael
> > > Novello <
> > > > > > > > > > > >> > > > > > rafa.reis.novello@gmail.com>
> > > > > > > > > > > >> > > > > > wrote:
> > > > > > > > > > > >> > > > > >
> > > > > > > > > > > >> > > > > > > Hi Lucas!
> > > > > > > > > > > >> > > > > > >
> > > > > > > > > > > >> > > > > > > First of all +1 for REPL POCs to Apache!
> > > > > > > > > > > >> > > > > > >
> > > > > > > > > > > >> > > > > > > Let me help with some comments:
> > > > > > > > > > > >> > > > > > >
> > > > > > > > > > > >> > > > > > > 1 - We have tested NodeJS, Scala and
> > Python
> > > > and
> > > > > > the
> > > > > > > > > > easiest
> > > > > > > > > > > >> one
> > > > > > > > > > > >> > was
> > > > > > > > > > > >> > > > > > Python.
> > > > > > > > > > > >> > > > > > > We have found a small project [1] that
> > have
> > > > all
> > > > > > > > features
> > > > > > > > > > we
> > > > > > > > > > > >> > desired
> > > > > > > > > > > >> > > > for
> > > > > > > > > > > >> > > > > > > REPL:
> > > > > > > > > > > >> > > > > > > - Autocomplete
> > > > > > > > > > > >> > > > > > > - Python commands disabled (the user
> have
> > > only
> > > > > the
> > > > > > > > > > commands
> > > > > > > > > > > we
> > > > > > > > > > > >> > > > > provide).
> > > > > > > > > > > >> > > > > > > The other languages REPL options don't
> > have
> > > > this
> > > > > > > > > feature.
> > > > > > > > > > > >> > > > > > > - Easy to show output
> > > > > > > > > > > >> > > > > > > - Etc
> > > > > > > > > > > >> > > > > > > So, I think the language chosen will not
> > be
> > > > > > > important
> > > > > > > > > here
> > > > > > > > > > > >> > because
> > > > > > > > > > > >> > > > the
> > > > > > > > > > > >> > > > > > user
> > > > > > > > > > > >> > > > > > > will only interact with the commands
> that
> > we
> > > > > > create.
> > > > > > > > > > > >> > > > > > >
> > > > > > > > > > > >> > > > > > > 2 - The "engine-generate" command will
> > > > download
> > > > > a
> > > > > > > > docker
> > > > > > > > > > > image
> > > > > > > > > > > >> > that
> > > > > > > > > > > >> > > > we
> > > > > > > > > > > >> > > > > > > create for that language and start a
> > > container
> > > > > > with
> > > > > > > > the
> > > > > > > > > > > basic
> > > > > > > > > > > >> > > project
> > > > > > > > > > > >> > > > > > > structure for that language.
> > > > > > > > > > > >> > > > > > >
> > > > > > > > > > > >> > > > > > > 3 - The REPL client will use the "docker
> > > > > protocol"
> > > > > > > to
> > > > > > > > > run
> > > > > > > > > > > >> > command,
> > > > > > > > > > > >> > > > > > > start/stop services and etc inside the
> > > > container
> > > > > > and
> > > > > > > > it
> > > > > > > > > > will
> > > > > > > > > > > >> > > receive
> > > > > > > > > > > >> > > > > the
> > > > > > > > > > > >> > > > > > > log stream to show. No, the REPL will no
> > > pass
> > > > > code
> > > > > > > > > > snippets
> > > > > > > > > > > >> for
> > > > > > > > > > > >> > > > docker
> > > > > > > > > > > >> > > > > > > container (I think it will not be
> > necessary)
> > > > > > > > > > > >> > > > > > >
> > > > > > > > > > > >> > > > > > > 4 - Yep! Like I said on the first item.
> > > > > > > > > > > >> > > > > > >
> > > > > > > > > > > >> > > > > > > [1] -
> > https://github.com/italorossi/ishell
> > > > > > > > > > > >> > > > > > >
> > > > > > > > > > > >> > > > > > > Let me know if there is any other
> > question!
> > > > > > > > > > > >> > > > > > > Best!
> > > > > > > > > > > >> > > > > > > Rafael J. R. Novello
> > > > > > > > > > > >> > > > > > >
> > > > > > > > > > > >> > > > > > > Skype: rafael.novello
> > > > > > > > > > > >> > > > > > > Blog:
> http://rafanovello.blogspot.com.br/
> > > > > > > > > > > >> > > > > > >
> > > > > > > > > > > >> > > > > > >
> > > > > > > > > > > >> > > > > > > Em qui, 18 de out de 2018 às 17:00,
> Lucas
> > > > > Bonatto
> > > > > > > > > Miguel <
> > > > > > > > > > > >> > > > > > > lucasbm88@apache.org> escreveu:
> > > > > > > > > > > >> > > > > > >
> > > > > > > > > > > >> > > > > > > > + 1 for migrating the REPL repo to
> > Apache
> > > > > > > > > > > >> > > > > > > >
> > > > > > > > > > > >> > > > > > > > I have a few questions about the
> > previous
> > > > > > > > explanation:
> > > > > > > > > > > >> > > > > > > >  1) The REPL itself, it would be an
> > > > > application
> > > > > > in
> > > > > > > > > which
> > > > > > > > > > > >> > > language?
> > > > > > > > > > > >> > > > > > > Remember
> > > > > > > > > > > >> > > > > > > > that the main idea is to allow the
> user
> > to
> > > > > > program
> > > > > > > > on
> > > > > > > > > > his
> > > > > > > > > > > >> > > preferred
> > > > > > > > > > > >> > > > > > > > language in the REPL.
> > > > > > > > > > > >> > > > > > > >  2) Should the engine-generate command
> > > also
> > > > > > > > generate a
> > > > > > > > > > > >> docker
> > > > > > > > > > > >> > > image
> > > > > > > > > > > >> > > > > > with
> > > > > > > > > > > >> > > > > > > > the user's application?
> > > > > > > > > > > >> > > > > > > >  3) What type of communication would
> > > happen
> > > > > > > between
> > > > > > > > > the
> > > > > > > > > > > REPL
> > > > > > > > > > > >> > and
> > > > > > > > > > > >> > > > the
> > > > > > > > > > > >> > > > > > > engine
> > > > > > > > > > > >> > > > > > > > via Docker SDK? Would the REPL pass
> > > snippets
> > > > > of
> > > > > > > code
> > > > > > > > > to
> > > > > > > > > > be
> > > > > > > > > > > >> > > executed
> > > > > > > > > > > >> > > > > by
> > > > > > > > > > > >> > > > > > > the
> > > > > > > > > > > >> > > > > > > > docker container?
> > > > > > > > > > > >> > > > > > > >  4) Have you considered code
> completion
> > in
> > > > the
> > > > > > > REPL?
> > > > > > > > > > > >> > > > > > > >
> > > > > > > > > > > >> > > > > > > >
> > > > > > > > > > > >> > > > > > > > On Thu, Oct 18, 2018 at 7:53 AM Zhang
> > > Yifei
> > > > <
> > > > > > > > > > > >> > > > yifei.z.luiz@gmail.com>
> > > > > > > > > > > >> > > > > > > > wrote:
> > > > > > > > > > > >> > > > > > > >
> > > > > > > > > > > >> > > > > > > > > Ok guys,
> > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > >> > > > > > > > > The basic idea is to provide only
> one
> > > > > Toolbox
> > > > > > > for
> > > > > > > > > > > multiple
> > > > > > > > > > > >> > > > > languages.
> > > > > > > > > > > >> > > > > > > > > We are looking for possibility to
> > build
> > > a
> > > > > > single
> > > > > > > > > > Marvin
> > > > > > > > > > > >> Repl,
> > > > > > > > > > > >> > > > > > > > > instead of severals toolboxes with
> > > > > differentes
> > > > > > > > > > > interfaces
> > > > > > > > > > > >> or
> > > > > > > > > > > >> > > > > > commands.
> > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > >> > > > > > > > > In this case, the engine-generate
> > > command
> > > > > will
> > > > > > > > > > download
> > > > > > > > > > > >> and
> > > > > > > > > > > >> > > > start a
> > > > > > > > > > > >> > > > > > > > Docker
> > > > > > > > > > > >> > > > > > > > > container with basic engine
> structure
> > > > > > > > corresponding
> > > > > > > > > > the
> > > > > > > > > > > >> > choosed
> > > > > > > > > > > >> > > > > > > language.
> > > > > > > > > > > >> > > > > > > > > this means we don't need to build
> > > > Toolboxes
> > > > > of
> > > > > > > > > > > differents
> > > > > > > > > > > >> > > > > languages,
> > > > > > > > > > > >> > > > > > we
> > > > > > > > > > > >> > > > > > > > > just
> > > > > > > > > > > >> > > > > > > > > build the engine template of all
> > > languages
> > > > > > that
> > > > > > > we
> > > > > > > > > > want
> > > > > > > > > > > to
> > > > > > > > > > > >> > > > support
> > > > > > > > > > > >> > > > > > and
> > > > > > > > > > > >> > > > > > > > > provide it
> > > > > > > > > > > >> > > > > > > > > as Docker containers
> > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > >> > > > > > > > > We have started researches around
> our
> > > > basic
> > > > > > > > > > requirements
> > > > > > > > > > > >> > like:
> > > > > > > > > > > >> > > > > > > > > - Repl interface
> > > > > > > > > > > >> > > > > > > > > - System communication
> > > > > > > > > > > >> > > > > > > > > - Connection security
> > > > > > > > > > > >> > > > > > > > > - Tool popularity
> > > > > > > > > > > >> > > > > > > > > - Update complexity
> > > > > > > > > > > >> > > > > > > > > - Languages support
> > > > > > > > > > > >> > > > > > > > > - ......
> > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > >> > > > > > > > > And we did some POC with code here:
> > > > > > > > > > > >> > > > > > > > >
> > > https://github.com/marvin-ai/marvin-repl
> > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > >> > > > > > > > > There is POC testing gRPC using
> Scala
> > > and
> > > > > > > Python,
> > > > > > > > > > > >> > > > > > > > > Repl inteface and Docker SDK with
> > > NodeJS,
> > > > > > > > > > > >> > > > > > > > > Repl interface and Docker SDK with
> > > Python.
> > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > >> > > > > > > > > At this moment we prefer the Repl
> > > > interface
> > > > > +
> > > > > > > > Docker
> > > > > > > > > > SDK
> > > > > > > > > > > >> way,
> > > > > > > > > > > >> > > > > because
> > > > > > > > > > > >> > > > > > > > good
> > > > > > > > > > > >> > > > > > > > > part of the requirements
> > > > > > > > > > > >> > > > > > > > > will be guaranteed by Docker.
> > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > >> > > > > > > > > With this informations, what do you
> > > think?
> > > > > > > Should
> > > > > > > > we
> > > > > > > > > > > >> submit
> > > > > > > > > > > >> > all
> > > > > > > > > > > >> > > > > this
> > > > > > > > > > > >> > > > > > > POCs
> > > > > > > > > > > >> > > > > > > > > to Apache Repo?
> > > > > > > > > > > >> > > > > > > > > Please feel free to opine.
> > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > >> > > > > > > > > Thats all, thanks!!!
> > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > >> > > > > > > > > Em ter, 16 de out de 2018 às 18:55,
> > > Daniel
> > > > > > > > > > Takabayashi <
> > > > > > > > > > > >> > > > > > > > > daniel.takabayashi@gmail.com>
> > escreveu:
> > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > >> > > > > > > > > > Zhang,
> > > > > > > > > > > >> > > > > > > > > >
> > > > > > > > > > > >> > > > > > > > > > I think the best approach is give
> > us a
> > > > > > better
> > > > > > > > > > > >> explanation
> > > > > > > > > > > >> > > about
> > > > > > > > > > > >> > > > > > this
> > > > > > > > > > > >> > > > > > > > new
> > > > > > > > > > > >> > > > > > > > > > feature and how this can help us
> to
> > > > > archive
> > > > > > > the
> > > > > > > > > > > desired
> > > > > > > > > > > >> > > feature
> > > > > > > > > > > >> > > > > > > > (support
> > > > > > > > > > > >> > > > > > > > > > multiple languages). Than if the
> > > > majority
> > > > > > > agree
> > > > > > > > > > than I
> > > > > > > > > > > >> can
> > > > > > > > > > > >> > do
> > > > > > > > > > > >> > > > the
> > > > > > > > > > > >> > > > > > > > merge,
> > > > > > > > > > > >> > > > > > > > > > just like I did before, but in
> > another
> > > > > > branch.
> > > > > > > > > > > >> > > > > > > > > >
> > > > > > > > > > > >> > > > > > > > > > Let me know if makes sense, ok?
> > > > > > > > > > > >> > > > > > > > > >
> > > > > > > > > > > >> > > > > > > > > > Taka
> > > > > > > > > > > >> > > > > > > > > >
> > > > > > > > > > > >> > > > > > > > > > Em ter, 16 de out de 2018 às
> 14:00,
> > > > > Luciano
> > > > > > > > > Resende
> > > > > > > > > > <
> > > > > > > > > > > >> > > > > > > > > luckbr1975@gmail.com>
> > > > > > > > > > > >> > > > > > > > > > escreveu:
> > > > > > > > > > > >> > > > > > > > > >
> > > > > > > > > > > >> > > > > > > > > > > So, what is the POC, is it a
> > > > refactoring
> > > > > > of
> > > > > > > > the
> > > > > > > > > > > >> existing
> > > > > > > > > > > >> > > > repo?
> > > > > > > > > > > >> > > > > Or
> > > > > > > > > > > >> > > > > > > is
> > > > > > > > > > > >> > > > > > > > > > > it a new rewrite of the repo?
> > > > > > > > > > > >> > > > > > > > > > >
> > > > > > > > > > > >> > > > > > > > > > > Just asking as it might make
> sense
> > > to
> > > > > make
> > > > > > > it
> > > > > > > > a
> > > > > > > > > > > branch
> > > > > > > > > > > >> > then
> > > > > > > > > > > >> > > > > > > actually
> > > > > > > > > > > >> > > > > > > > a
> > > > > > > > > > > >> > > > > > > > > > > thing in parallel, as this will
> > have
> > > > an
> > > > > > > effect
> > > > > > > > > for
> > > > > > > > > > > >> > > releases,
> > > > > > > > > > > >> > > > > etc.
> > > > > > > > > > > >> > > > > > > but
> > > > > > > > > > > >> > > > > > > > > > > you guys know more here than I
> do.
> > > > > > > > > > > >> > > > > > > > > > >
> > > > > > > > > > > >> > > > > > > > > > > Also, it's probably good to
> have a
> > > > write
> > > > > > up
> > > > > > > of
> > > > > > > > > the
> > > > > > > > > > > >> main
> > > > > > > > > > > >> > > > > direction
> > > > > > > > > > > >> > > > > > > of
> > > > > > > > > > > >> > > > > > > > > > > the design that can help people
> > get
> > > > > > familiar
> > > > > > > > > with
> > > > > > > > > > > the
> > > > > > > > > > > >> new
> > > > > > > > > > > >> > > > > > approach.
> > > > > > > > > > > >> > > > > > > > > > > On Tue, Oct 16, 2018 at 11:12 AM
> > > Zhang
> > > > > > > Yifei <
> > > > > > > > > > > >> > > > > > > yifei.z.luiz@gmail.com
> > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > >> > > > > > > > > > > wrote:
> > > > > > > > > > > >> > > > > > > > > > > >
> > > > > > > > > > > >> > > > > > > > > > > > Hey guys,
> > > > > > > > > > > >> > > > > > > > > > > >
> > > > > > > > > > > >> > > > > > > > > > > > We have reorganized the Poc
> > repo,
> > > > and
> > > > > > want
> > > > > > > > to
> > > > > > > > > > > merge
> > > > > > > > > > > >> it
> > > > > > > > > > > >> > to
> > > > > > > > > > > >> > > > our
> > > > > > > > > > > >> > > > > > > > Apache
> > > > > > > > > > > >> > > > > > > > > > > repo.
> > > > > > > > > > > >> > > > > > > > > > > > Just making sure here before
> we
> > do
> > > > the
> > > > > > > > merge,
> > > > > > > > > > > >> because
> > > > > > > > > > > >> > im
> > > > > > > > > > > >> > > > not
> > > > > > > > > > > >> > > > > > sure
> > > > > > > > > > > >> > > > > > > > if
> > > > > > > > > > > >> > > > > > > > > > > doing
> > > > > > > > > > > >> > > > > > > > > > > > this will perserve the Git
> > commit
> > > > > > history.
> > > > > > > > > > > >> > > > > > > > > > > > What we are planning to do is:
> > > > > > > > > > > >> > > > > > > > > > > >
> > > > > > > > > > > >> > > > > > > > > > > > git filter-branch
> > > > > --subdirectory-filter
> > > > > > > <git
> > > > > > > > > > > >> > > > > origin_repository
> > > > > > > > > > > >> > > > > > > > > > directory>
> > > > > > > > > > > >> > > > > > > > > > > > -- --all
> > > > > > > > > > > >> > > > > > > > > > > > mkdir POCs/
> > > > > > > > > > > >> > > > > > > > > > > > git mv * POCs/
> > > > > > > > > > > >> > > > > > > > > > > > git commit -m "colleted the
> data
> > > to
> > > > > > move"
> > > > > > > > > > > >> > > > > > > > > > > >
> > > > > > > > > > > >> > > > > > > > > > > > git clone <git
> > > > > > destination_repository_url>
> > > > > > > > cd
> > > > > > > > > > <git
> > > > > > > > > > > >> > > > > > > > > > > >
> > destination_repository_directory>
> > > > git
> > > > > > > remote
> > > > > > > > > add
> > > > > > > > > > > >> > > > > > > > > > origin_repository_branch
> > > > > > > > > > > >> > > > > > > > > > > > <git origin_repository
> > directory>
> > > > git
> > > > > > pull
> > > > > > > > > > > >> > > > > > > origin_repository_branch
> > > > > > > > > > > >> > > > > > > > > > > master
> > > > > > > > > > > >> > > > > > > > > > > > --allow-unrelated-histories
> git
> > > > remote
> > > > > > rm
> > > > > > > > > > > >> > > > > > > origin_repository_branch
> > > > > > > > > > > >> > > > > > > > > > > >
> > > > > > > > > > > >> > > > > > > > > > > >
> > > > > > > > > > > >> > > > > > > > > > > > Thanks in advance !
> > > > > > > > > > > >> > > > > > > > > > > >
> > > > > > > > > > > >> > > > > > > > > > > >
> > > > > > > > > > > >> > > > >
> > > > > > > > >
> > --------------------------------------------------------------
> > > > > > > > > > > >> > > > > > > > > > > > Zhang Yifei
> > > > > > > > > > > >> > > > > > > > > > >
> > > > > > > > > > > >> > > > > > > > > > >
> > > > > > > > > > > >> > > > > > > > > > >
> > > > > > > > > > > >> > > > > > > > > > > --
> > > > > > > > > > > >> > > > > > > > > > > Luciano Resende
> > > > > > > > > > > >> > > > > > > > > > > http://twitter.com/lresende1975
> > > > > > > > > > > >> > > > > > > > > > > http://lresende.blogspot.com/
> > > > > > > > > > > >> > > > > > > > > > >
> > > > > > > > > > > >> > > > > > > > > >
> > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > >> > > > > > > > > --
> > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > >> >
> > > > > > --------------------------------------------------------------
> > > > > > > > > > > >> > > > > > > > > Zhang Yifei
> > > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > > >> > > > > > > >
> > > > > > > > > > > >> > > > > > >
> > > > > > > > > > > >> > > > > >
> > > > > > > > > > > >> > > > >
> > > > > > > > > > > >> > > >
> > > > > > > > > > > >> > >
> > > > > > > > > > > >> >
> > > > > > > > > > > >>
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: POCs, Prototypes, experiments ...

Posted by Rafael Novello <ra...@gmail.com>.

I do not want to be the annoying conversation but...

If we will have an unique CLI that will be able to handle any kind of
engines (in any language) by using docker, Why we need a REPL?

I think we can focus on creating this new architecture using docker and
docker SDK and use only a CLI for that.

Atenciosamente,
Rafael J. R. Novello

Skype: rafael.novello
Blog: http://rafanovello.blogspot.com.br/


Em seg, 29 de out de 2018 às 14:58, Lucas Bonatto Miguel <
lucasbm88@apache.org> escreveu:

> No, I think the idea is to do not create different versions of the CLI.
> Let's use one single tech stack for the CLI, ideally, it will be shared
> with the REPL. The way I see the CLI would be very similar to the REPL, but
> instead of opening an interactive session the user could send
> unattended commands. As an analogy, you can think on spark-shell (our REPL)
> and spark-submit (our CLI).
>
> - Lucas
>
> On Mon, Oct 29, 2018 at 10:27 AM Rafael Novello <
> rafa.reis.novello@gmail.com>
> wrote:
>
> > Hi Guys!
> >
> > Let me ask about this new version that include CLI's and REPL. Will we
> > create different versions of CLI?
> >
> > One of the reasons to create the REPL is to avoid re-implementing the
> same
> > features many times,
> > each time for a different language.
> >
> > Regards!
> > Atenciosamente,
> > Rafael J. R. Novello
> >
> > Skype: rafael.novello
> > Blog: http://rafanovello.blogspot.com.br/
> >
> >
> > Em sex, 26 de out de 2018 às 20:54, Daniel Takabayashi <
> > daniel.takabayashi@gmail.com> escreveu:
> >
> > > +1
> > >
> > > Em sex, 26 de out de 2018 às 16:47, Lucas Bonatto Miguel <
> > > lucasbm88@apache.org> escreveu:
> > >
> > > > Makes perfect sense to me. I believe that if the majority of people
> > > agrees
> > > > with this architecture we could create issues and start implementing
> > it.
> > > >
> > > > +1
> > > >
> > > >
> > > > On Fri, Oct 26, 2018 at 4:09 PM Daniel Takabayashi <
> > > > daniel.takabayashi@gmail.com> wrote:
> > > >
> > > > > I see... what about this new version?
> > > > >
> > > > >
> > > > >
> > > >
> > >
> >
> https://docs.google.com/drawings/d/shajxIpLJHxxMbFgDXiPuhg/image?w=602&h=468&rev=1630&ac=1&parent=1ySERHGBXbHeyCMRookq5UfTuFkzzU0ugtjvR3rF3deY
> > > > >
> > > > > I changed the toolbox ...from 1 component to 3... docker client,
> cli
> > > and
> > > > > repl. Both (cli and repl) now are using the docker client to
> control
> > > the
> > > > > engines.
> > > > >
> > > > > Em sex, 26 de out de 2018 às 14:49, Lucas Bonatto Miguel <
> > > > > lucasbm88@apache.org> escreveu:
> > > > >
> > > > > > For sure, both suggestions are to make sure we're not killing the
> > CLI
> > > > and
> > > > > > that we have a component (the SDK client) that is shared by CLI
> and
> > > > REPL.
> > > > > >
> > > > > > Thanks for clarifying where the system calls will live.
> > > > > >
> > > > > > - Lucas
> > > > > >
> > > > > > On Fri, Oct 26, 2018 at 2:39 PM Daniel Takabayashi <
> > > > > > daniel.takabayashi@gmail.com> wrote:
> > > > > >
> > > > > > > Hi Lucas, could you please justified your two suggestions?
> > > > > > >
> > > > > > > The general ideia is to use the Docker DSK to execute local
> > > commands,
> > > > > to
> > > > > > > consume outputs and also to execute Marvin "specific language"
> > > > > commands.
> > > > > > > Today almost all CLIs commands in Marvin is a mixed of systems
> > > > commands
> > > > > > and
> > > > > > > library method calls. The plan is to put all "specific language
> > > > calls"
> > > > > > > inside the marvin-common-libs and group the systems call on the
> > > > toolbox
> > > > > > > (using the docker SDK in this task).
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > Em qui, 25 de out de 2018 às 20:09, Lucas Bonatto Miguel <
> > > > > > > lucasbm88@apache.org> escreveu:
> > > > > > >
> > > > > > > > Great, thanks.
> > > > > > > >
> > > > > > > > So here some suggestions:
> > > > > > > >  1. We could create one new layer between Toolbox and the
> > > Engines,
> > > > > this
> > > > > > > > would be Docker client.
> > > > > > > >  2. Split the Toolbox layer in two. Toolbox CLI and Toolbox
> > REPL
> > > > > > > >
> > > > > > > > It's not clear to me if the commands listed on General
> Commands
> > > > will
> > > > > > need
> > > > > > > > to be written for every language, from the image I believe
> not,
> > > but
> > > > > how
> > > > > > > it
> > > > > > > > would work, since starting a notebook in Python is different
> > than
> > > > > > > starting
> > > > > > > > in Scala.
> > > > > > > >
> > > > > > > > - Lucas
> > > > > > > > On Thu, Oct 25, 2018 at 4:51 PM Daniel Takabayashi <
> > > > > > > > daniel.takabayashi@gmail.com> wrote:
> > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://docs.google.com/drawings/d/shajxIpLJHxxMbFgDXiPuhg/image?w=602&h=461&rev=1423&ac=1&parent=1ySERHGBXbHeyCMRookq5UfTuFkzzU0ugtjvR3rF3deY
> > > > > > > > >
> > > > > > > > > Em qui, 25 de out de 2018 às 07:38, Rafael Novello <
> > > > > > > > > rafa.reis.novello@gmail.com> escreveu:
> > > > > > > > >
> > > > > > > > > > Hi Taka,
> > > > > > > > > >
> > > > > > > > > > I can't see the image! Maybe the Apache email list don't
> > > > support
> > > > > > > > > > attachments.
> > > > > > > > > >
> > > > > > > > > > Best!
> > > > > > > > > > Rafael J. R. Novello
> > > > > > > > > >
> > > > > > > > > > Skype: rafael.novello
> > > > > > > > > > Blog: http://rafanovello.blogspot.com.br/
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Em qua, 24 de out de 2018 às 21:14, Daniel Takabayashi <
> > > > > > > > > > daniel.takabayashi@gmail.com> escreveu:
> > > > > > > > > >
> > > > > > > > > > > Hi all,
> > > > > > > > > > >
> > > > > > > > > > > To try to give a little more details about this "new
> > > concept"
> > > > > > that
> > > > > > > we
> > > > > > > > > > want
> > > > > > > > > > > to bring to the marvin toolbox, I did this simple
> > > > architecture
> > > > > > > draw.
> > > > > > > > > > >
> > > > > > > > > > > [image: marvin-architecture-views (5).png]
> > > > > > > > > > >
> > > > > > > > > > > The general idea here is try to transform the toolbox
> > > > something
> > > > > > > > > > > disconnected with the "language", something more
> > agnostic.
> > > > Also
> > > > > > in
> > > > > > > > this
> > > > > > > > > > new
> > > > > > > > > > > architecture we could use remote resource to process
> > > engines
> > > > > and
> > > > > > > make
> > > > > > > > > > easy
> > > > > > > > > > > the support for new languages.
> > > > > > > > > > >
> > > > > > > > > > > This "new toolbox" will be the only thing that a Marvin
> > > user
> > > > > must
> > > > > > > to
> > > > > > > > > > > install and also we could start to support multiples
> O.S
> > > > (once
> > > > > > the
> > > > > > > > REPL
> > > > > > > > > > is
> > > > > > > > > > > a dummy application that only interprets and by pass
> > > > commands).
> > > > > > > > > > >
> > > > > > > > > > > Regards,
> > > > > > > > > > > Taka
> > > > > > > > > > >
> > > > > > > > > > > Em qua, 24 de out de 2018 às 09:52, Rafael Novello <
> > > > > > > > > > > rafa.reis.novello@gmail.com> escreveu:
> > > > > > > > > > >
> > > > > > > > > > >> Alan,
> > > > > > > > > > >>
> > > > > > > > > > >> Yes! We are using the Docker SDK and it's possible to
> > use
> > > > the
> > > > > > API
> > > > > > > to
> > > > > > > > > > >> automate, but it's a little bit harder than automate
> CLI
> > > > > calls.
> > > > > > > > > > >>
> > > > > > > > > > >> Atenciosamente,
> > > > > > > > > > >> Rafael J. R. Novello
> > > > > > > > > > >>
> > > > > > > > > > >> Skype: rafael.novello
> > > > > > > > > > >> Blog: http://rafanovello.blogspot.com.br/
> > > > > > > > > > >>
> > > > > > > > > > >>
> > > > > > > > > > >> Em qua, 24 de out de 2018 às 12:02, Alan Silva <
> > > > > > jumpi@apache.org>
> > > > > > > > > > >> escreveu:
> > > > > > > > > > >>
> > > > > > > > > > >> > Hi,
> > > > > > > > > > >> >
> > > > > > > > > > >> > One question here, I understand that we start to use
> > > with
> > > > > this
> > > > > > > PoC
> > > > > > > > > the
> > > > > > > > > > >> > Docker SDK API, right?
> > > > > > > > > > >> >
> > > > > > > > > > >> > Why not use the API to expose some endpoints to
> permit
> > > > this
> > > > > > kind
> > > > > > > > of
> > > > > > > > > > >> > automation by devops?
> > > > > > > > > > >> >
> > > > > > > > > > >> > I think it is possible and it solves the CLI
> problem,
> > > > right?
> > > > > > > > > > >> >
> > > > > > > > > > >> >
> > > > > > > > > > >> > On Tue, Oct 23, 2018 at 1:05 PM Rafael Novello <
> > > > > > > > > > >> > rafa.reis.novello@gmail.com>
> > > > > > > > > > >> > wrote:
> > > > > > > > > > >> >
> > > > > > > > > > >> > > Lucas,
> > > > > > > > > > >> > >
> > > > > > > > > > >> > > Sorry, I didn't understood your question bellow.
> > > > > > > > > > >> > >
> > > > > > > > > > >> > > "Would it make sense to use the same solution that
> > we
> > > > will
> > > > > > use
> > > > > > > > for
> > > > > > > > > > >> > having a
> > > > > > > > > > >> > > single-language REPL to have a single-language
> CLI?"
> > > > > > > > > > >> > >
> > > > > > > > > > >> > > For DevOps purposes, maybe this new toolbox
> concept
> > is
> > > > not
> > > > > > > > ideal.
> > > > > > > > > I
> > > > > > > > > > >> think
> > > > > > > > > > >> > > we can keep the CLI inside the docker container
> but
> > it
> > > > > will
> > > > > > > not
> > > > > > > > > easy
> > > > > > > > > > >> to
> > > > > > > > > > >> > > automate
> > > > > > > > > > >> > > jobs by this way.
> > > > > > > > > > >> > >
> > > > > > > > > > >> > > How to deal with this issue? Voting?
> > > > > > > > > > >> > > Atenciosamente,
> > > > > > > > > > >> > > Rafael J. R. Novello
> > > > > > > > > > >> > >
> > > > > > > > > > >> > > Skype: rafael.novello
> > > > > > > > > > >> > > Blog: http://rafanovello.blogspot.com.br/
> > > > > > > > > > >> > >
> > > > > > > > > > >> > >
> > > > > > > > > > >> > > Em sex, 19 de out de 2018 às 19:00, Lucas Bonatto
> > > > Miguel <
> > > > > > > > > > >> > > lucasbm88@apache.org> escreveu:
> > > > > > > > > > >> > >
> > > > > > > > > > >> > > > Got it! Thanks for clarifying.
> > > > > > > > > > >> > > >
> > > > > > > > > > >> > > > Would it make sense to use the same solution
> that
> > we
> > > > > will
> > > > > > > use
> > > > > > > > > for
> > > > > > > > > > >> > having
> > > > > > > > > > >> > > a
> > > > > > > > > > >> > > > single-language REPL to have a single-language
> > CLI?
> > > My
> > > > > > only
> > > > > > > > > > concern
> > > > > > > > > > >> > with
> > > > > > > > > > >> > > > killing the CLI is that you remove an essential
> > > > feature
> > > > > > for
> > > > > > > > > > DevOps.
> > > > > > > > > > >> > > >
> > > > > > > > > > >> > > > - Lucas
> > > > > > > > > > >> > > >
> > > > > > > > > > >> > > > On Fri, Oct 19, 2018 at 1:52 PM Rafael Novello <
> > > > > > > > > > >> > > > rafa.reis.novello@gmail.com>
> > > > > > > > > > >> > > > wrote:
> > > > > > > > > > >> > > >
> > > > > > > > > > >> > > > > Lucas,
> > > > > > > > > > >> > > > >
> > > > > > > > > > >> > > > > The idea is that REPL will substitute the
> actual
> > > > CLI.
> > > > > > It's
> > > > > > > > > > because
> > > > > > > > > > >> > with
> > > > > > > > > > >> > > > the
> > > > > > > > > > >> > > > > actual concept (using language specific CLI)
> we
> > > will
> > > > > > need
> > > > > > > to
> > > > > > > > > > >> > > re-implement
> > > > > > > > > > >> > > > > the same features many times and probably each
> > CLI
> > > > > will
> > > > > > > > have a
> > > > > > > > > > >> > > different
> > > > > > > > > > >> > > > > behavior because some language specific
> > > restrictions
> > > > > > > and/or
> > > > > > > > > > >> > > limitations.
> > > > > > > > > > >> > > > >
> > > > > > > > > > >> > > > > With this new concept, all users will have the
> > > same
> > > > > > > > experience
> > > > > > > > > > >> > > > interacting
> > > > > > > > > > >> > > > > with Marvin REPL and they will use the bests
> > tools
> > > > to
> > > > > > > > develop
> > > > > > > > > > >> their
> > > > > > > > > > >> > > > engines
> > > > > > > > > > >> > > > > (Jupyter Notebook and/or Lab with different
> > > > languages
> > > > > > > > support
> > > > > > > > > or
> > > > > > > > > > >> even
> > > > > > > > > > >> > > > > Apache Zeppelin). All the interact will occur
> > > > through
> > > > > > REPL
> > > > > > > > and
> > > > > > > > > > >> docker
> > > > > > > > > > >> > > > > protocol.
> > > > > > > > > > >> > > > >
> > > > > > > > > > >> > > > > Alan, Yes! As Lucas said, the concept is the
> > same
> > > > but
> > > > > we
> > > > > > > can
> > > > > > > > > use
> > > > > > > > > > >> > docker
> > > > > > > > > > >> > > > to
> > > > > > > > > > >> > > > > do the same job.
> > > > > > > > > > >> > > > >
> > > > > > > > > > >> > > > > Best!
> > > > > > > > > > >> > > > >
> > > > > > > > > > >> > > > > Em sex, 19 de out de 2018 às 00:39, Lucas
> > Bonatto
> > > > > > Miguel <
> > > > > > > > > > >> > > > > lucasbm88@apache.org> escreveu:
> > > > > > > > > > >> > > > >
> > > > > > > > > > >> > > > > > Thanks for the clarifications Rafael, so
> from
> > > > what I
> > > > > > > > > > understood,
> > > > > > > > > > >> > > > Marvin's
> > > > > > > > > > >> > > > > > developers would not use the REPL to explore
> > > data
> > > > > and
> > > > > > > test
> > > > > > > > > > >> models,
> > > > > > > > > > >> > > i.e.
> > > > > > > > > > >> > > > > > develop the engine. Is the idea to build
> > > something
> > > > > > more
> > > > > > > > like
> > > > > > > > > > an
> > > > > > > > > > >> > > > > interactive
> > > > > > > > > > >> > > > > > CLI? I think an interactive CLI would be
> > useful
> > > > for
> > > > > > the
> > > > > > > > > > >> developer
> > > > > > > > > > >> > > > > > experience, however, it's important to keep
> > the
> > > > > > > unattended
> > > > > > > > > > >> > (current)
> > > > > > > > > > >> > > > CLI
> > > > > > > > > > >> > > > > in
> > > > > > > > > > >> > > > > > place for scenarios where users want to
> > automate
> > > > > > Marvin
> > > > > > > > > tasks.
> > > > > > > > > > >> > > > > >
> > > > > > > > > > >> > > > > > Alan, I believe the idea is still the same
> as
> > > you
> > > > > > > started,
> > > > > > > > > but
> > > > > > > > > > >> > using
> > > > > > > > > > >> > > > > > docker SDK now.
> > > > > > > > > > >> > > > > >
> > > > > > > > > > >> > > > > > - Lucas
> > > > > > > > > > >> > > > > >
> > > > > > > > > > >> > > > > > On Thu, Oct 18, 2018 at 1:29 PM Rafael
> > Novello <
> > > > > > > > > > >> > > > > > rafa.reis.novello@gmail.com>
> > > > > > > > > > >> > > > > > wrote:
> > > > > > > > > > >> > > > > >
> > > > > > > > > > >> > > > > > > Hi Lucas!
> > > > > > > > > > >> > > > > > >
> > > > > > > > > > >> > > > > > > First of all +1 for REPL POCs to Apache!
> > > > > > > > > > >> > > > > > >
> > > > > > > > > > >> > > > > > > Let me help with some comments:
> > > > > > > > > > >> > > > > > >
> > > > > > > > > > >> > > > > > > 1 - We have tested NodeJS, Scala and
> Python
> > > and
> > > > > the
> > > > > > > > > easiest
> > > > > > > > > > >> one
> > > > > > > > > > >> > was
> > > > > > > > > > >> > > > > > Python.
> > > > > > > > > > >> > > > > > > We have found a small project [1] that
> have
> > > all
> > > > > > > features
> > > > > > > > > we
> > > > > > > > > > >> > desired
> > > > > > > > > > >> > > > for
> > > > > > > > > > >> > > > > > > REPL:
> > > > > > > > > > >> > > > > > > - Autocomplete
> > > > > > > > > > >> > > > > > > - Python commands disabled (the user have
> > only
> > > > the
> > > > > > > > > commands
> > > > > > > > > > we
> > > > > > > > > > >> > > > > provide).
> > > > > > > > > > >> > > > > > > The other languages REPL options don't
> have
> > > this
> > > > > > > > feature.
> > > > > > > > > > >> > > > > > > - Easy to show output
> > > > > > > > > > >> > > > > > > - Etc
> > > > > > > > > > >> > > > > > > So, I think the language chosen will not
> be
> > > > > > important
> > > > > > > > here
> > > > > > > > > > >> > because
> > > > > > > > > > >> > > > the
> > > > > > > > > > >> > > > > > user
> > > > > > > > > > >> > > > > > > will only interact with the commands that
> we
> > > > > create.
> > > > > > > > > > >> > > > > > >
> > > > > > > > > > >> > > > > > > 2 - The "engine-generate" command will
> > > download
> > > > a
> > > > > > > docker
> > > > > > > > > > image
> > > > > > > > > > >> > that
> > > > > > > > > > >> > > > we
> > > > > > > > > > >> > > > > > > create for that language and start a
> > container
> > > > > with
> > > > > > > the
> > > > > > > > > > basic
> > > > > > > > > > >> > > project
> > > > > > > > > > >> > > > > > > structure for that language.
> > > > > > > > > > >> > > > > > >
> > > > > > > > > > >> > > > > > > 3 - The REPL client will use the "docker
> > > > protocol"
> > > > > > to
> > > > > > > > run
> > > > > > > > > > >> > command,
> > > > > > > > > > >> > > > > > > start/stop services and etc inside the
> > > container
> > > > > and
> > > > > > > it
> > > > > > > > > will
> > > > > > > > > > >> > > receive
> > > > > > > > > > >> > > > > the
> > > > > > > > > > >> > > > > > > log stream to show. No, the REPL will no
> > pass
> > > > code
> > > > > > > > > snippets
> > > > > > > > > > >> for
> > > > > > > > > > >> > > > docker
> > > > > > > > > > >> > > > > > > container (I think it will not be
> necessary)
> > > > > > > > > > >> > > > > > >
> > > > > > > > > > >> > > > > > > 4 - Yep! Like I said on the first item.
> > > > > > > > > > >> > > > > > >
> > > > > > > > > > >> > > > > > > [1] -
> https://github.com/italorossi/ishell
> > > > > > > > > > >> > > > > > >
> > > > > > > > > > >> > > > > > > Let me know if there is any other
> question!
> > > > > > > > > > >> > > > > > > Best!
> > > > > > > > > > >> > > > > > > Rafael J. R. Novello
> > > > > > > > > > >> > > > > > >
> > > > > > > > > > >> > > > > > > Skype: rafael.novello
> > > > > > > > > > >> > > > > > > Blog: http://rafanovello.blogspot.com.br/
> > > > > > > > > > >> > > > > > >
> > > > > > > > > > >> > > > > > >
> > > > > > > > > > >> > > > > > > Em qui, 18 de out de 2018 às 17:00, Lucas
> > > > Bonatto
> > > > > > > > Miguel <
> > > > > > > > > > >> > > > > > > lucasbm88@apache.org> escreveu:
> > > > > > > > > > >> > > > > > >
> > > > > > > > > > >> > > > > > > > + 1 for migrating the REPL repo to
> Apache
> > > > > > > > > > >> > > > > > > >
> > > > > > > > > > >> > > > > > > > I have a few questions about the
> previous
> > > > > > > explanation:
> > > > > > > > > > >> > > > > > > >  1) The REPL itself, it would be an
> > > > application
> > > > > in
> > > > > > > > which
> > > > > > > > > > >> > > language?
> > > > > > > > > > >> > > > > > > Remember
> > > > > > > > > > >> > > > > > > > that the main idea is to allow the user
> to
> > > > > program
> > > > > > > on
> > > > > > > > > his
> > > > > > > > > > >> > > preferred
> > > > > > > > > > >> > > > > > > > language in the REPL.
> > > > > > > > > > >> > > > > > > >  2) Should the engine-generate command
> > also
> > > > > > > generate a
> > > > > > > > > > >> docker
> > > > > > > > > > >> > > image
> > > > > > > > > > >> > > > > > with
> > > > > > > > > > >> > > > > > > > the user's application?
> > > > > > > > > > >> > > > > > > >  3) What type of communication would
> > happen
> > > > > > between
> > > > > > > > the
> > > > > > > > > > REPL
> > > > > > > > > > >> > and
> > > > > > > > > > >> > > > the
> > > > > > > > > > >> > > > > > > engine
> > > > > > > > > > >> > > > > > > > via Docker SDK? Would the REPL pass
> > snippets
> > > > of
> > > > > > code
> > > > > > > > to
> > > > > > > > > be
> > > > > > > > > > >> > > executed
> > > > > > > > > > >> > > > > by
> > > > > > > > > > >> > > > > > > the
> > > > > > > > > > >> > > > > > > > docker container?
> > > > > > > > > > >> > > > > > > >  4) Have you considered code completion
> in
> > > the
> > > > > > REPL?
> > > > > > > > > > >> > > > > > > >
> > > > > > > > > > >> > > > > > > >
> > > > > > > > > > >> > > > > > > > On Thu, Oct 18, 2018 at 7:53 AM Zhang
> > Yifei
> > > <
> > > > > > > > > > >> > > > yifei.z.luiz@gmail.com>
> > > > > > > > > > >> > > > > > > > wrote:
> > > > > > > > > > >> > > > > > > >
> > > > > > > > > > >> > > > > > > > > Ok guys,
> > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > >> > > > > > > > > The basic idea is to provide only one
> > > > Toolbox
> > > > > > for
> > > > > > > > > > multiple
> > > > > > > > > > >> > > > > languages.
> > > > > > > > > > >> > > > > > > > > We are looking for possibility to
> build
> > a
> > > > > single
> > > > > > > > > Marvin
> > > > > > > > > > >> Repl,
> > > > > > > > > > >> > > > > > > > > instead of severals toolboxes with
> > > > differentes
> > > > > > > > > > interfaces
> > > > > > > > > > >> or
> > > > > > > > > > >> > > > > > commands.
> > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > >> > > > > > > > > In this case, the engine-generate
> > command
> > > > will
> > > > > > > > > download
> > > > > > > > > > >> and
> > > > > > > > > > >> > > > start a
> > > > > > > > > > >> > > > > > > > Docker
> > > > > > > > > > >> > > > > > > > > container with basic engine structure
> > > > > > > corresponding
> > > > > > > > > the
> > > > > > > > > > >> > choosed
> > > > > > > > > > >> > > > > > > language.
> > > > > > > > > > >> > > > > > > > > this means we don't need to build
> > > Toolboxes
> > > > of
> > > > > > > > > > differents
> > > > > > > > > > >> > > > > languages,
> > > > > > > > > > >> > > > > > we
> > > > > > > > > > >> > > > > > > > > just
> > > > > > > > > > >> > > > > > > > > build the engine template of all
> > languages
> > > > > that
> > > > > > we
> > > > > > > > > want
> > > > > > > > > > to
> > > > > > > > > > >> > > > support
> > > > > > > > > > >> > > > > > and
> > > > > > > > > > >> > > > > > > > > provide it
> > > > > > > > > > >> > > > > > > > > as Docker containers
> > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > >> > > > > > > > > We have started researches around our
> > > basic
> > > > > > > > > requirements
> > > > > > > > > > >> > like:
> > > > > > > > > > >> > > > > > > > > - Repl interface
> > > > > > > > > > >> > > > > > > > > - System communication
> > > > > > > > > > >> > > > > > > > > - Connection security
> > > > > > > > > > >> > > > > > > > > - Tool popularity
> > > > > > > > > > >> > > > > > > > > - Update complexity
> > > > > > > > > > >> > > > > > > > > - Languages support
> > > > > > > > > > >> > > > > > > > > - ......
> > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > >> > > > > > > > > And we did some POC with code here:
> > > > > > > > > > >> > > > > > > > >
> > https://github.com/marvin-ai/marvin-repl
> > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > >> > > > > > > > > There is POC testing gRPC using Scala
> > and
> > > > > > Python,
> > > > > > > > > > >> > > > > > > > > Repl inteface and Docker SDK with
> > NodeJS,
> > > > > > > > > > >> > > > > > > > > Repl interface and Docker SDK with
> > Python.
> > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > >> > > > > > > > > At this moment we prefer the Repl
> > > interface
> > > > +
> > > > > > > Docker
> > > > > > > > > SDK
> > > > > > > > > > >> way,
> > > > > > > > > > >> > > > > because
> > > > > > > > > > >> > > > > > > > good
> > > > > > > > > > >> > > > > > > > > part of the requirements
> > > > > > > > > > >> > > > > > > > > will be guaranteed by Docker.
> > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > >> > > > > > > > > With this informations, what do you
> > think?
> > > > > > Should
> > > > > > > we
> > > > > > > > > > >> submit
> > > > > > > > > > >> > all
> > > > > > > > > > >> > > > > this
> > > > > > > > > > >> > > > > > > POCs
> > > > > > > > > > >> > > > > > > > > to Apache Repo?
> > > > > > > > > > >> > > > > > > > > Please feel free to opine.
> > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > >> > > > > > > > > Thats all, thanks!!!
> > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > >> > > > > > > > > Em ter, 16 de out de 2018 às 18:55,
> > Daniel
> > > > > > > > > Takabayashi <
> > > > > > > > > > >> > > > > > > > > daniel.takabayashi@gmail.com>
> escreveu:
> > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > >> > > > > > > > > > Zhang,
> > > > > > > > > > >> > > > > > > > > >
> > > > > > > > > > >> > > > > > > > > > I think the best approach is give
> us a
> > > > > better
> > > > > > > > > > >> explanation
> > > > > > > > > > >> > > about
> > > > > > > > > > >> > > > > > this
> > > > > > > > > > >> > > > > > > > new
> > > > > > > > > > >> > > > > > > > > > feature and how this can help us to
> > > > archive
> > > > > > the
> > > > > > > > > > desired
> > > > > > > > > > >> > > feature
> > > > > > > > > > >> > > > > > > > (support
> > > > > > > > > > >> > > > > > > > > > multiple languages). Than if the
> > > majority
> > > > > > agree
> > > > > > > > > than I
> > > > > > > > > > >> can
> > > > > > > > > > >> > do
> > > > > > > > > > >> > > > the
> > > > > > > > > > >> > > > > > > > merge,
> > > > > > > > > > >> > > > > > > > > > just like I did before, but in
> another
> > > > > branch.
> > > > > > > > > > >> > > > > > > > > >
> > > > > > > > > > >> > > > > > > > > > Let me know if makes sense, ok?
> > > > > > > > > > >> > > > > > > > > >
> > > > > > > > > > >> > > > > > > > > > Taka
> > > > > > > > > > >> > > > > > > > > >
> > > > > > > > > > >> > > > > > > > > > Em ter, 16 de out de 2018 às 14:00,
> > > > Luciano
> > > > > > > > Resende
> > > > > > > > > <
> > > > > > > > > > >> > > > > > > > > luckbr1975@gmail.com>
> > > > > > > > > > >> > > > > > > > > > escreveu:
> > > > > > > > > > >> > > > > > > > > >
> > > > > > > > > > >> > > > > > > > > > > So, what is the POC, is it a
> > > refactoring
> > > > > of
> > > > > > > the
> > > > > > > > > > >> existing
> > > > > > > > > > >> > > > repo?
> > > > > > > > > > >> > > > > Or
> > > > > > > > > > >> > > > > > > is
> > > > > > > > > > >> > > > > > > > > > > it a new rewrite of the repo?
> > > > > > > > > > >> > > > > > > > > > >
> > > > > > > > > > >> > > > > > > > > > > Just asking as it might make sense
> > to
> > > > make
> > > > > > it
> > > > > > > a
> > > > > > > > > > branch
> > > > > > > > > > >> > then
> > > > > > > > > > >> > > > > > > actually
> > > > > > > > > > >> > > > > > > > a
> > > > > > > > > > >> > > > > > > > > > > thing in parallel, as this will
> have
> > > an
> > > > > > effect
> > > > > > > > for
> > > > > > > > > > >> > > releases,
> > > > > > > > > > >> > > > > etc.
> > > > > > > > > > >> > > > > > > but
> > > > > > > > > > >> > > > > > > > > > > you guys know more here than I do.
> > > > > > > > > > >> > > > > > > > > > >
> > > > > > > > > > >> > > > > > > > > > > Also, it's probably good to have a
> > > write
> > > > > up
> > > > > > of
> > > > > > > > the
> > > > > > > > > > >> main
> > > > > > > > > > >> > > > > direction
> > > > > > > > > > >> > > > > > > of
> > > > > > > > > > >> > > > > > > > > > > the design that can help people
> get
> > > > > familiar
> > > > > > > > with
> > > > > > > > > > the
> > > > > > > > > > >> new
> > > > > > > > > > >> > > > > > approach.
> > > > > > > > > > >> > > > > > > > > > > On Tue, Oct 16, 2018 at 11:12 AM
> > Zhang
> > > > > > Yifei <
> > > > > > > > > > >> > > > > > > yifei.z.luiz@gmail.com
> > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > >> > > > > > > > > > > wrote:
> > > > > > > > > > >> > > > > > > > > > > >
> > > > > > > > > > >> > > > > > > > > > > > Hey guys,
> > > > > > > > > > >> > > > > > > > > > > >
> > > > > > > > > > >> > > > > > > > > > > > We have reorganized the Poc
> repo,
> > > and
> > > > > want
> > > > > > > to
> > > > > > > > > > merge
> > > > > > > > > > >> it
> > > > > > > > > > >> > to
> > > > > > > > > > >> > > > our
> > > > > > > > > > >> > > > > > > > Apache
> > > > > > > > > > >> > > > > > > > > > > repo.
> > > > > > > > > > >> > > > > > > > > > > > Just making sure here before we
> do
> > > the
> > > > > > > merge,
> > > > > > > > > > >> because
> > > > > > > > > > >> > im
> > > > > > > > > > >> > > > not
> > > > > > > > > > >> > > > > > sure
> > > > > > > > > > >> > > > > > > > if
> > > > > > > > > > >> > > > > > > > > > > doing
> > > > > > > > > > >> > > > > > > > > > > > this will perserve the Git
> commit
> > > > > history.
> > > > > > > > > > >> > > > > > > > > > > > What we are planning to do is:
> > > > > > > > > > >> > > > > > > > > > > >
> > > > > > > > > > >> > > > > > > > > > > > git filter-branch
> > > > --subdirectory-filter
> > > > > > <git
> > > > > > > > > > >> > > > > origin_repository
> > > > > > > > > > >> > > > > > > > > > directory>
> > > > > > > > > > >> > > > > > > > > > > > -- --all
> > > > > > > > > > >> > > > > > > > > > > > mkdir POCs/
> > > > > > > > > > >> > > > > > > > > > > > git mv * POCs/
> > > > > > > > > > >> > > > > > > > > > > > git commit -m "colleted the data
> > to
> > > > > move"
> > > > > > > > > > >> > > > > > > > > > > >
> > > > > > > > > > >> > > > > > > > > > > > git clone <git
> > > > > destination_repository_url>
> > > > > > > cd
> > > > > > > > > <git
> > > > > > > > > > >> > > > > > > > > > > >
> destination_repository_directory>
> > > git
> > > > > > remote
> > > > > > > > add
> > > > > > > > > > >> > > > > > > > > > origin_repository_branch
> > > > > > > > > > >> > > > > > > > > > > > <git origin_repository
> directory>
> > > git
> > > > > pull
> > > > > > > > > > >> > > > > > > origin_repository_branch
> > > > > > > > > > >> > > > > > > > > > > master
> > > > > > > > > > >> > > > > > > > > > > > --allow-unrelated-histories git
> > > remote
> > > > > rm
> > > > > > > > > > >> > > > > > > origin_repository_branch
> > > > > > > > > > >> > > > > > > > > > > >
> > > > > > > > > > >> > > > > > > > > > > >
> > > > > > > > > > >> > > > > > > > > > > > Thanks in advance !
> > > > > > > > > > >> > > > > > > > > > > >
> > > > > > > > > > >> > > > > > > > > > > >
> > > > > > > > > > >> > > > >
> > > > > > > >
> --------------------------------------------------------------
> > > > > > > > > > >> > > > > > > > > > > > Zhang Yifei
> > > > > > > > > > >> > > > > > > > > > >
> > > > > > > > > > >> > > > > > > > > > >
> > > > > > > > > > >> > > > > > > > > > >
> > > > > > > > > > >> > > > > > > > > > > --
> > > > > > > > > > >> > > > > > > > > > > Luciano Resende
> > > > > > > > > > >> > > > > > > > > > > http://twitter.com/lresende1975
> > > > > > > > > > >> > > > > > > > > > > http://lresende.blogspot.com/
> > > > > > > > > > >> > > > > > > > > > >
> > > > > > > > > > >> > > > > > > > > >
> > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > >> > > > > > > > > --
> > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > >> >
> > > > > --------------------------------------------------------------
> > > > > > > > > > >> > > > > > > > > Zhang Yifei
> > > > > > > > > > >> > > > > > > > >
> > > > > > > > > > >> > > > > > > >
> > > > > > > > > > >> > > > > > >
> > > > > > > > > > >> > > > > >
> > > > > > > > > > >> > > > >
> > > > > > > > > > >> > > >
> > > > > > > > > > >> > >
> > > > > > > > > > >> >
> > > > > > > > > > >>
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: POCs, Prototypes, experiments ...

Posted by Lucas Bonatto Miguel <lu...@apache.org>.

No, I think the idea is to do not create different versions of the CLI.
Let's use one single tech stack for the CLI, ideally, it will be shared
with the REPL. The way I see the CLI would be very similar to the REPL, but
instead of opening an interactive session the user could send
unattended commands. As an analogy, you can think on spark-shell (our REPL)
and spark-submit (our CLI).

- Lucas

On Mon, Oct 29, 2018 at 10:27 AM Rafael Novello <ra...@gmail.com>
wrote:

> Hi Guys!
>
> Let me ask about this new version that include CLI's and REPL. Will we
> create different versions of CLI?
>
> One of the reasons to create the REPL is to avoid re-implementing the same
> features many times,
> each time for a different language.
>
> Regards!
> Atenciosamente,
> Rafael J. R. Novello
>
> Skype: rafael.novello
> Blog: http://rafanovello.blogspot.com.br/
>
>
> Em sex, 26 de out de 2018 às 20:54, Daniel Takabayashi <
> daniel.takabayashi@gmail.com> escreveu:
>
> > +1
> >
> > Em sex, 26 de out de 2018 às 16:47, Lucas Bonatto Miguel <
> > lucasbm88@apache.org> escreveu:
> >
> > > Makes perfect sense to me. I believe that if the majority of people
> > agrees
> > > with this architecture we could create issues and start implementing
> it.
> > >
> > > +1
> > >
> > >
> > > On Fri, Oct 26, 2018 at 4:09 PM Daniel Takabayashi <
> > > daniel.takabayashi@gmail.com> wrote:
> > >
> > > > I see... what about this new version?
> > > >
> > > >
> > > >
> > >
> >
> https://docs.google.com/drawings/d/shajxIpLJHxxMbFgDXiPuhg/image?w=602&h=468&rev=1630&ac=1&parent=1ySERHGBXbHeyCMRookq5UfTuFkzzU0ugtjvR3rF3deY
> > > >
> > > > I changed the toolbox ...from 1 component to 3... docker client, cli
> > and
> > > > repl. Both (cli and repl) now are using the docker client to control
> > the
> > > > engines.
> > > >
> > > > Em sex, 26 de out de 2018 às 14:49, Lucas Bonatto Miguel <
> > > > lucasbm88@apache.org> escreveu:
> > > >
> > > > > For sure, both suggestions are to make sure we're not killing the
> CLI
> > > and
> > > > > that we have a component (the SDK client) that is shared by CLI and
> > > REPL.
> > > > >
> > > > > Thanks for clarifying where the system calls will live.
> > > > >
> > > > > - Lucas
> > > > >
> > > > > On Fri, Oct 26, 2018 at 2:39 PM Daniel Takabayashi <
> > > > > daniel.takabayashi@gmail.com> wrote:
> > > > >
> > > > > > Hi Lucas, could you please justified your two suggestions?
> > > > > >
> > > > > > The general ideia is to use the Docker DSK to execute local
> > commands,
> > > > to
> > > > > > consume outputs and also to execute Marvin "specific language"
> > > > commands.
> > > > > > Today almost all CLIs commands in Marvin is a mixed of systems
> > > commands
> > > > > and
> > > > > > library method calls. The plan is to put all "specific language
> > > calls"
> > > > > > inside the marvin-common-libs and group the systems call on the
> > > toolbox
> > > > > > (using the docker SDK in this task).
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > Em qui, 25 de out de 2018 às 20:09, Lucas Bonatto Miguel <
> > > > > > lucasbm88@apache.org> escreveu:
> > > > > >
> > > > > > > Great, thanks.
> > > > > > >
> > > > > > > So here some suggestions:
> > > > > > >  1. We could create one new layer between Toolbox and the
> > Engines,
> > > > this
> > > > > > > would be Docker client.
> > > > > > >  2. Split the Toolbox layer in two. Toolbox CLI and Toolbox
> REPL
> > > > > > >
> > > > > > > It's not clear to me if the commands listed on General Commands
> > > will
> > > > > need
> > > > > > > to be written for every language, from the image I believe not,
> > but
> > > > how
> > > > > > it
> > > > > > > would work, since starting a notebook in Python is different
> than
> > > > > > starting
> > > > > > > in Scala.
> > > > > > >
> > > > > > > - Lucas
> > > > > > > On Thu, Oct 25, 2018 at 4:51 PM Daniel Takabayashi <
> > > > > > > daniel.takabayashi@gmail.com> wrote:
> > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://docs.google.com/drawings/d/shajxIpLJHxxMbFgDXiPuhg/image?w=602&h=461&rev=1423&ac=1&parent=1ySERHGBXbHeyCMRookq5UfTuFkzzU0ugtjvR3rF3deY
> > > > > > > >
> > > > > > > > Em qui, 25 de out de 2018 às 07:38, Rafael Novello <
> > > > > > > > rafa.reis.novello@gmail.com> escreveu:
> > > > > > > >
> > > > > > > > > Hi Taka,
> > > > > > > > >
> > > > > > > > > I can't see the image! Maybe the Apache email list don't
> > > support
> > > > > > > > > attachments.
> > > > > > > > >
> > > > > > > > > Best!
> > > > > > > > > Rafael J. R. Novello
> > > > > > > > >
> > > > > > > > > Skype: rafael.novello
> > > > > > > > > Blog: http://rafanovello.blogspot.com.br/
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Em qua, 24 de out de 2018 às 21:14, Daniel Takabayashi <
> > > > > > > > > daniel.takabayashi@gmail.com> escreveu:
> > > > > > > > >
> > > > > > > > > > Hi all,
> > > > > > > > > >
> > > > > > > > > > To try to give a little more details about this "new
> > concept"
> > > > > that
> > > > > > we
> > > > > > > > > want
> > > > > > > > > > to bring to the marvin toolbox, I did this simple
> > > architecture
> > > > > > draw.
> > > > > > > > > >
> > > > > > > > > > [image: marvin-architecture-views (5).png]
> > > > > > > > > >
> > > > > > > > > > The general idea here is try to transform the toolbox
> > > something
> > > > > > > > > > disconnected with the "language", something more
> agnostic.
> > > Also
> > > > > in
> > > > > > > this
> > > > > > > > > new
> > > > > > > > > > architecture we could use remote resource to process
> > engines
> > > > and
> > > > > > make
> > > > > > > > > easy
> > > > > > > > > > the support for new languages.
> > > > > > > > > >
> > > > > > > > > > This "new toolbox" will be the only thing that a Marvin
> > user
> > > > must
> > > > > > to
> > > > > > > > > > install and also we could start to support multiples O.S
> > > (once
> > > > > the
> > > > > > > REPL
> > > > > > > > > is
> > > > > > > > > > a dummy application that only interprets and by pass
> > > commands).
> > > > > > > > > >
> > > > > > > > > > Regards,
> > > > > > > > > > Taka
> > > > > > > > > >
> > > > > > > > > > Em qua, 24 de out de 2018 às 09:52, Rafael Novello <
> > > > > > > > > > rafa.reis.novello@gmail.com> escreveu:
> > > > > > > > > >
> > > > > > > > > >> Alan,
> > > > > > > > > >>
> > > > > > > > > >> Yes! We are using the Docker SDK and it's possible to
> use
> > > the
> > > > > API
> > > > > > to
> > > > > > > > > >> automate, but it's a little bit harder than automate CLI
> > > > calls.
> > > > > > > > > >>
> > > > > > > > > >> Atenciosamente,
> > > > > > > > > >> Rafael J. R. Novello
> > > > > > > > > >>
> > > > > > > > > >> Skype: rafael.novello
> > > > > > > > > >> Blog: http://rafanovello.blogspot.com.br/
> > > > > > > > > >>
> > > > > > > > > >>
> > > > > > > > > >> Em qua, 24 de out de 2018 às 12:02, Alan Silva <
> > > > > jumpi@apache.org>
> > > > > > > > > >> escreveu:
> > > > > > > > > >>
> > > > > > > > > >> > Hi,
> > > > > > > > > >> >
> > > > > > > > > >> > One question here, I understand that we start to use
> > with
> > > > this
> > > > > > PoC
> > > > > > > > the
> > > > > > > > > >> > Docker SDK API, right?
> > > > > > > > > >> >
> > > > > > > > > >> > Why not use the API to expose some endpoints to permit
> > > this
> > > > > kind
> > > > > > > of
> > > > > > > > > >> > automation by devops?
> > > > > > > > > >> >
> > > > > > > > > >> > I think it is possible and it solves the CLI problem,
> > > right?
> > > > > > > > > >> >
> > > > > > > > > >> >
> > > > > > > > > >> > On Tue, Oct 23, 2018 at 1:05 PM Rafael Novello <
> > > > > > > > > >> > rafa.reis.novello@gmail.com>
> > > > > > > > > >> > wrote:
> > > > > > > > > >> >
> > > > > > > > > >> > > Lucas,
> > > > > > > > > >> > >
> > > > > > > > > >> > > Sorry, I didn't understood your question bellow.
> > > > > > > > > >> > >
> > > > > > > > > >> > > "Would it make sense to use the same solution that
> we
> > > will
> > > > > use
> > > > > > > for
> > > > > > > > > >> > having a
> > > > > > > > > >> > > single-language REPL to have a single-language CLI?"
> > > > > > > > > >> > >
> > > > > > > > > >> > > For DevOps purposes, maybe this new toolbox concept
> is
> > > not
> > > > > > > ideal.
> > > > > > > > I
> > > > > > > > > >> think
> > > > > > > > > >> > > we can keep the CLI inside the docker container but
> it
> > > > will
> > > > > > not
> > > > > > > > easy
> > > > > > > > > >> to
> > > > > > > > > >> > > automate
> > > > > > > > > >> > > jobs by this way.
> > > > > > > > > >> > >
> > > > > > > > > >> > > How to deal with this issue? Voting?
> > > > > > > > > >> > > Atenciosamente,
> > > > > > > > > >> > > Rafael J. R. Novello
> > > > > > > > > >> > >
> > > > > > > > > >> > > Skype: rafael.novello
> > > > > > > > > >> > > Blog: http://rafanovello.blogspot.com.br/
> > > > > > > > > >> > >
> > > > > > > > > >> > >
> > > > > > > > > >> > > Em sex, 19 de out de 2018 às 19:00, Lucas Bonatto
> > > Miguel <
> > > > > > > > > >> > > lucasbm88@apache.org> escreveu:
> > > > > > > > > >> > >
> > > > > > > > > >> > > > Got it! Thanks for clarifying.
> > > > > > > > > >> > > >
> > > > > > > > > >> > > > Would it make sense to use the same solution that
> we
> > > > will
> > > > > > use
> > > > > > > > for
> > > > > > > > > >> > having
> > > > > > > > > >> > > a
> > > > > > > > > >> > > > single-language REPL to have a single-language
> CLI?
> > My
> > > > > only
> > > > > > > > > concern
> > > > > > > > > >> > with
> > > > > > > > > >> > > > killing the CLI is that you remove an essential
> > > feature
> > > > > for
> > > > > > > > > DevOps.
> > > > > > > > > >> > > >
> > > > > > > > > >> > > > - Lucas
> > > > > > > > > >> > > >
> > > > > > > > > >> > > > On Fri, Oct 19, 2018 at 1:52 PM Rafael Novello <
> > > > > > > > > >> > > > rafa.reis.novello@gmail.com>
> > > > > > > > > >> > > > wrote:
> > > > > > > > > >> > > >
> > > > > > > > > >> > > > > Lucas,
> > > > > > > > > >> > > > >
> > > > > > > > > >> > > > > The idea is that REPL will substitute the actual
> > > CLI.
> > > > > It's
> > > > > > > > > because
> > > > > > > > > >> > with
> > > > > > > > > >> > > > the
> > > > > > > > > >> > > > > actual concept (using language specific CLI) we
> > will
> > > > > need
> > > > > > to
> > > > > > > > > >> > > re-implement
> > > > > > > > > >> > > > > the same features many times and probably each
> CLI
> > > > will
> > > > > > > have a
> > > > > > > > > >> > > different
> > > > > > > > > >> > > > > behavior because some language specific
> > restrictions
> > > > > > and/or
> > > > > > > > > >> > > limitations.
> > > > > > > > > >> > > > >
> > > > > > > > > >> > > > > With this new concept, all users will have the
> > same
> > > > > > > experience
> > > > > > > > > >> > > > interacting
> > > > > > > > > >> > > > > with Marvin REPL and they will use the bests
> tools
> > > to
> > > > > > > develop
> > > > > > > > > >> their
> > > > > > > > > >> > > > engines
> > > > > > > > > >> > > > > (Jupyter Notebook and/or Lab with different
> > > languages
> > > > > > > support
> > > > > > > > or
> > > > > > > > > >> even
> > > > > > > > > >> > > > > Apache Zeppelin). All the interact will occur
> > > through
> > > > > REPL
> > > > > > > and
> > > > > > > > > >> docker
> > > > > > > > > >> > > > > protocol.
> > > > > > > > > >> > > > >
> > > > > > > > > >> > > > > Alan, Yes! As Lucas said, the concept is the
> same
> > > but
> > > > we
> > > > > > can
> > > > > > > > use
> > > > > > > > > >> > docker
> > > > > > > > > >> > > > to
> > > > > > > > > >> > > > > do the same job.
> > > > > > > > > >> > > > >
> > > > > > > > > >> > > > > Best!
> > > > > > > > > >> > > > >
> > > > > > > > > >> > > > > Em sex, 19 de out de 2018 às 00:39, Lucas
> Bonatto
> > > > > Miguel <
> > > > > > > > > >> > > > > lucasbm88@apache.org> escreveu:
> > > > > > > > > >> > > > >
> > > > > > > > > >> > > > > > Thanks for the clarifications Rafael, so from
> > > what I
> > > > > > > > > understood,
> > > > > > > > > >> > > > Marvin's
> > > > > > > > > >> > > > > > developers would not use the REPL to explore
> > data
> > > > and
> > > > > > test
> > > > > > > > > >> models,
> > > > > > > > > >> > > i.e.
> > > > > > > > > >> > > > > > develop the engine. Is the idea to build
> > something
> > > > > more
> > > > > > > like
> > > > > > > > > an
> > > > > > > > > >> > > > > interactive
> > > > > > > > > >> > > > > > CLI? I think an interactive CLI would be
> useful
> > > for
> > > > > the
> > > > > > > > > >> developer
> > > > > > > > > >> > > > > > experience, however, it's important to keep
> the
> > > > > > unattended
> > > > > > > > > >> > (current)
> > > > > > > > > >> > > > CLI
> > > > > > > > > >> > > > > in
> > > > > > > > > >> > > > > > place for scenarios where users want to
> automate
> > > > > Marvin
> > > > > > > > tasks.
> > > > > > > > > >> > > > > >
> > > > > > > > > >> > > > > > Alan, I believe the idea is still the same as
> > you
> > > > > > started,
> > > > > > > > but
> > > > > > > > > >> > using
> > > > > > > > > >> > > > > > docker SDK now.
> > > > > > > > > >> > > > > >
> > > > > > > > > >> > > > > > - Lucas
> > > > > > > > > >> > > > > >
> > > > > > > > > >> > > > > > On Thu, Oct 18, 2018 at 1:29 PM Rafael
> Novello <
> > > > > > > > > >> > > > > > rafa.reis.novello@gmail.com>
> > > > > > > > > >> > > > > > wrote:
> > > > > > > > > >> > > > > >
> > > > > > > > > >> > > > > > > Hi Lucas!
> > > > > > > > > >> > > > > > >
> > > > > > > > > >> > > > > > > First of all +1 for REPL POCs to Apache!
> > > > > > > > > >> > > > > > >
> > > > > > > > > >> > > > > > > Let me help with some comments:
> > > > > > > > > >> > > > > > >
> > > > > > > > > >> > > > > > > 1 - We have tested NodeJS, Scala and Python
> > and
> > > > the
> > > > > > > > easiest
> > > > > > > > > >> one
> > > > > > > > > >> > was
> > > > > > > > > >> > > > > > Python.
> > > > > > > > > >> > > > > > > We have found a small project [1] that have
> > all
> > > > > > features
> > > > > > > > we
> > > > > > > > > >> > desired
> > > > > > > > > >> > > > for
> > > > > > > > > >> > > > > > > REPL:
> > > > > > > > > >> > > > > > > - Autocomplete
> > > > > > > > > >> > > > > > > - Python commands disabled (the user have
> only
> > > the
> > > > > > > > commands
> > > > > > > > > we
> > > > > > > > > >> > > > > provide).
> > > > > > > > > >> > > > > > > The other languages REPL options don't have
> > this
> > > > > > > feature.
> > > > > > > > > >> > > > > > > - Easy to show output
> > > > > > > > > >> > > > > > > - Etc
> > > > > > > > > >> > > > > > > So, I think the language chosen will not be
> > > > > important
> > > > > > > here
> > > > > > > > > >> > because
> > > > > > > > > >> > > > the
> > > > > > > > > >> > > > > > user
> > > > > > > > > >> > > > > > > will only interact with the commands that we
> > > > create.
> > > > > > > > > >> > > > > > >
> > > > > > > > > >> > > > > > > 2 - The "engine-generate" command will
> > download
> > > a
> > > > > > docker
> > > > > > > > > image
> > > > > > > > > >> > that
> > > > > > > > > >> > > > we
> > > > > > > > > >> > > > > > > create for that language and start a
> container
> > > > with
> > > > > > the
> > > > > > > > > basic
> > > > > > > > > >> > > project
> > > > > > > > > >> > > > > > > structure for that language.
> > > > > > > > > >> > > > > > >
> > > > > > > > > >> > > > > > > 3 - The REPL client will use the "docker
> > > protocol"
> > > > > to
> > > > > > > run
> > > > > > > > > >> > command,
> > > > > > > > > >> > > > > > > start/stop services and etc inside the
> > container
> > > > and
> > > > > > it
> > > > > > > > will
> > > > > > > > > >> > > receive
> > > > > > > > > >> > > > > the
> > > > > > > > > >> > > > > > > log stream to show. No, the REPL will no
> pass
> > > code
> > > > > > > > snippets
> > > > > > > > > >> for
> > > > > > > > > >> > > > docker
> > > > > > > > > >> > > > > > > container (I think it will not be necessary)
> > > > > > > > > >> > > > > > >
> > > > > > > > > >> > > > > > > 4 - Yep! Like I said on the first item.
> > > > > > > > > >> > > > > > >
> > > > > > > > > >> > > > > > > [1] - https://github.com/italorossi/ishell
> > > > > > > > > >> > > > > > >
> > > > > > > > > >> > > > > > > Let me know if there is any other question!
> > > > > > > > > >> > > > > > > Best!
> > > > > > > > > >> > > > > > > Rafael J. R. Novello
> > > > > > > > > >> > > > > > >
> > > > > > > > > >> > > > > > > Skype: rafael.novello
> > > > > > > > > >> > > > > > > Blog: http://rafanovello.blogspot.com.br/
> > > > > > > > > >> > > > > > >
> > > > > > > > > >> > > > > > >
> > > > > > > > > >> > > > > > > Em qui, 18 de out de 2018 às 17:00, Lucas
> > > Bonatto
> > > > > > > Miguel <
> > > > > > > > > >> > > > > > > lucasbm88@apache.org> escreveu:
> > > > > > > > > >> > > > > > >
> > > > > > > > > >> > > > > > > > + 1 for migrating the REPL repo to Apache
> > > > > > > > > >> > > > > > > >
> > > > > > > > > >> > > > > > > > I have a few questions about the previous
> > > > > > explanation:
> > > > > > > > > >> > > > > > > >  1) The REPL itself, it would be an
> > > application
> > > > in
> > > > > > > which
> > > > > > > > > >> > > language?
> > > > > > > > > >> > > > > > > Remember
> > > > > > > > > >> > > > > > > > that the main idea is to allow the user to
> > > > program
> > > > > > on
> > > > > > > > his
> > > > > > > > > >> > > preferred
> > > > > > > > > >> > > > > > > > language in the REPL.
> > > > > > > > > >> > > > > > > >  2) Should the engine-generate command
> also
> > > > > > generate a
> > > > > > > > > >> docker
> > > > > > > > > >> > > image
> > > > > > > > > >> > > > > > with
> > > > > > > > > >> > > > > > > > the user's application?
> > > > > > > > > >> > > > > > > >  3) What type of communication would
> happen
> > > > > between
> > > > > > > the
> > > > > > > > > REPL
> > > > > > > > > >> > and
> > > > > > > > > >> > > > the
> > > > > > > > > >> > > > > > > engine
> > > > > > > > > >> > > > > > > > via Docker SDK? Would the REPL pass
> snippets
> > > of
> > > > > code
> > > > > > > to
> > > > > > > > be
> > > > > > > > > >> > > executed
> > > > > > > > > >> > > > > by
> > > > > > > > > >> > > > > > > the
> > > > > > > > > >> > > > > > > > docker container?
> > > > > > > > > >> > > > > > > >  4) Have you considered code completion in
> > the
> > > > > REPL?
> > > > > > > > > >> > > > > > > >
> > > > > > > > > >> > > > > > > >
> > > > > > > > > >> > > > > > > > On Thu, Oct 18, 2018 at 7:53 AM Zhang
> Yifei
> > <
> > > > > > > > > >> > > > yifei.z.luiz@gmail.com>
> > > > > > > > > >> > > > > > > > wrote:
> > > > > > > > > >> > > > > > > >
> > > > > > > > > >> > > > > > > > > Ok guys,
> > > > > > > > > >> > > > > > > > >
> > > > > > > > > >> > > > > > > > > The basic idea is to provide only one
> > > Toolbox
> > > > > for
> > > > > > > > > multiple
> > > > > > > > > >> > > > > languages.
> > > > > > > > > >> > > > > > > > > We are looking for possibility to build
> a
> > > > single
> > > > > > > > Marvin
> > > > > > > > > >> Repl,
> > > > > > > > > >> > > > > > > > > instead of severals toolboxes with
> > > differentes
> > > > > > > > > interfaces
> > > > > > > > > >> or
> > > > > > > > > >> > > > > > commands.
> > > > > > > > > >> > > > > > > > >
> > > > > > > > > >> > > > > > > > > In this case, the engine-generate
> command
> > > will
> > > > > > > > download
> > > > > > > > > >> and
> > > > > > > > > >> > > > start a
> > > > > > > > > >> > > > > > > > Docker
> > > > > > > > > >> > > > > > > > > container with basic engine structure
> > > > > > corresponding
> > > > > > > > the
> > > > > > > > > >> > choosed
> > > > > > > > > >> > > > > > > language.
> > > > > > > > > >> > > > > > > > > this means we don't need to build
> > Toolboxes
> > > of
> > > > > > > > > differents
> > > > > > > > > >> > > > > languages,
> > > > > > > > > >> > > > > > we
> > > > > > > > > >> > > > > > > > > just
> > > > > > > > > >> > > > > > > > > build the engine template of all
> languages
> > > > that
> > > > > we
> > > > > > > > want
> > > > > > > > > to
> > > > > > > > > >> > > > support
> > > > > > > > > >> > > > > > and
> > > > > > > > > >> > > > > > > > > provide it
> > > > > > > > > >> > > > > > > > > as Docker containers
> > > > > > > > > >> > > > > > > > >
> > > > > > > > > >> > > > > > > > > We have started researches around our
> > basic
> > > > > > > > requirements
> > > > > > > > > >> > like:
> > > > > > > > > >> > > > > > > > > - Repl interface
> > > > > > > > > >> > > > > > > > > - System communication
> > > > > > > > > >> > > > > > > > > - Connection security
> > > > > > > > > >> > > > > > > > > - Tool popularity
> > > > > > > > > >> > > > > > > > > - Update complexity
> > > > > > > > > >> > > > > > > > > - Languages support
> > > > > > > > > >> > > > > > > > > - ......
> > > > > > > > > >> > > > > > > > >
> > > > > > > > > >> > > > > > > > > And we did some POC with code here:
> > > > > > > > > >> > > > > > > > >
> https://github.com/marvin-ai/marvin-repl
> > > > > > > > > >> > > > > > > > >
> > > > > > > > > >> > > > > > > > > There is POC testing gRPC using Scala
> and
> > > > > Python,
> > > > > > > > > >> > > > > > > > > Repl inteface and Docker SDK with
> NodeJS,
> > > > > > > > > >> > > > > > > > > Repl interface and Docker SDK with
> Python.
> > > > > > > > > >> > > > > > > > >
> > > > > > > > > >> > > > > > > > > At this moment we prefer the Repl
> > interface
> > > +
> > > > > > Docker
> > > > > > > > SDK
> > > > > > > > > >> way,
> > > > > > > > > >> > > > > because
> > > > > > > > > >> > > > > > > > good
> > > > > > > > > >> > > > > > > > > part of the requirements
> > > > > > > > > >> > > > > > > > > will be guaranteed by Docker.
> > > > > > > > > >> > > > > > > > >
> > > > > > > > > >> > > > > > > > > With this informations, what do you
> think?
> > > > > Should
> > > > > > we
> > > > > > > > > >> submit
> > > > > > > > > >> > all
> > > > > > > > > >> > > > > this
> > > > > > > > > >> > > > > > > POCs
> > > > > > > > > >> > > > > > > > > to Apache Repo?
> > > > > > > > > >> > > > > > > > > Please feel free to opine.
> > > > > > > > > >> > > > > > > > >
> > > > > > > > > >> > > > > > > > >
> > > > > > > > > >> > > > > > > > > Thats all, thanks!!!
> > > > > > > > > >> > > > > > > > >
> > > > > > > > > >> > > > > > > > >
> > > > > > > > > >> > > > > > > > > Em ter, 16 de out de 2018 às 18:55,
> Daniel
> > > > > > > > Takabayashi <
> > > > > > > > > >> > > > > > > > > daniel.takabayashi@gmail.com> escreveu:
> > > > > > > > > >> > > > > > > > >
> > > > > > > > > >> > > > > > > > > > Zhang,
> > > > > > > > > >> > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > I think the best approach is give us a
> > > > better
> > > > > > > > > >> explanation
> > > > > > > > > >> > > about
> > > > > > > > > >> > > > > > this
> > > > > > > > > >> > > > > > > > new
> > > > > > > > > >> > > > > > > > > > feature and how this can help us to
> > > archive
> > > > > the
> > > > > > > > > desired
> > > > > > > > > >> > > feature
> > > > > > > > > >> > > > > > > > (support
> > > > > > > > > >> > > > > > > > > > multiple languages). Than if the
> > majority
> > > > > agree
> > > > > > > > than I
> > > > > > > > > >> can
> > > > > > > > > >> > do
> > > > > > > > > >> > > > the
> > > > > > > > > >> > > > > > > > merge,
> > > > > > > > > >> > > > > > > > > > just like I did before, but in another
> > > > branch.
> > > > > > > > > >> > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > Let me know if makes sense, ok?
> > > > > > > > > >> > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > Taka
> > > > > > > > > >> > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > Em ter, 16 de out de 2018 às 14:00,
> > > Luciano
> > > > > > > Resende
> > > > > > > > <
> > > > > > > > > >> > > > > > > > > luckbr1975@gmail.com>
> > > > > > > > > >> > > > > > > > > > escreveu:
> > > > > > > > > >> > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > So, what is the POC, is it a
> > refactoring
> > > > of
> > > > > > the
> > > > > > > > > >> existing
> > > > > > > > > >> > > > repo?
> > > > > > > > > >> > > > > Or
> > > > > > > > > >> > > > > > > is
> > > > > > > > > >> > > > > > > > > > > it a new rewrite of the repo?
> > > > > > > > > >> > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > Just asking as it might make sense
> to
> > > make
> > > > > it
> > > > > > a
> > > > > > > > > branch
> > > > > > > > > >> > then
> > > > > > > > > >> > > > > > > actually
> > > > > > > > > >> > > > > > > > a
> > > > > > > > > >> > > > > > > > > > > thing in parallel, as this will have
> > an
> > > > > effect
> > > > > > > for
> > > > > > > > > >> > > releases,
> > > > > > > > > >> > > > > etc.
> > > > > > > > > >> > > > > > > but
> > > > > > > > > >> > > > > > > > > > > you guys know more here than I do.
> > > > > > > > > >> > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > Also, it's probably good to have a
> > write
> > > > up
> > > > > of
> > > > > > > the
> > > > > > > > > >> main
> > > > > > > > > >> > > > > direction
> > > > > > > > > >> > > > > > > of
> > > > > > > > > >> > > > > > > > > > > the design that can help people get
> > > > familiar
> > > > > > > with
> > > > > > > > > the
> > > > > > > > > >> new
> > > > > > > > > >> > > > > > approach.
> > > > > > > > > >> > > > > > > > > > > On Tue, Oct 16, 2018 at 11:12 AM
> Zhang
> > > > > Yifei <
> > > > > > > > > >> > > > > > > yifei.z.luiz@gmail.com
> > > > > > > > > >> > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > wrote:
> > > > > > > > > >> > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > Hey guys,
> > > > > > > > > >> > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > We have reorganized the Poc repo,
> > and
> > > > want
> > > > > > to
> > > > > > > > > merge
> > > > > > > > > >> it
> > > > > > > > > >> > to
> > > > > > > > > >> > > > our
> > > > > > > > > >> > > > > > > > Apache
> > > > > > > > > >> > > > > > > > > > > repo.
> > > > > > > > > >> > > > > > > > > > > > Just making sure here before we do
> > the
> > > > > > merge,
> > > > > > > > > >> because
> > > > > > > > > >> > im
> > > > > > > > > >> > > > not
> > > > > > > > > >> > > > > > sure
> > > > > > > > > >> > > > > > > > if
> > > > > > > > > >> > > > > > > > > > > doing
> > > > > > > > > >> > > > > > > > > > > > this will perserve the Git commit
> > > > history.
> > > > > > > > > >> > > > > > > > > > > > What we are planning to do is:
> > > > > > > > > >> > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > git filter-branch
> > > --subdirectory-filter
> > > > > <git
> > > > > > > > > >> > > > > origin_repository
> > > > > > > > > >> > > > > > > > > > directory>
> > > > > > > > > >> > > > > > > > > > > > -- --all
> > > > > > > > > >> > > > > > > > > > > > mkdir POCs/
> > > > > > > > > >> > > > > > > > > > > > git mv * POCs/
> > > > > > > > > >> > > > > > > > > > > > git commit -m "colleted the data
> to
> > > > move"
> > > > > > > > > >> > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > git clone <git
> > > > destination_repository_url>
> > > > > > cd
> > > > > > > > <git
> > > > > > > > > >> > > > > > > > > > > > destination_repository_directory>
> > git
> > > > > remote
> > > > > > > add
> > > > > > > > > >> > > > > > > > > > origin_repository_branch
> > > > > > > > > >> > > > > > > > > > > > <git origin_repository directory>
> > git
> > > > pull
> > > > > > > > > >> > > > > > > origin_repository_branch
> > > > > > > > > >> > > > > > > > > > > master
> > > > > > > > > >> > > > > > > > > > > > --allow-unrelated-histories git
> > remote
> > > > rm
> > > > > > > > > >> > > > > > > origin_repository_branch
> > > > > > > > > >> > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > > Thanks in advance !
> > > > > > > > > >> > > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > >
> > > > > > > > > >> > > > >
> > > > > > > --------------------------------------------------------------
> > > > > > > > > >> > > > > > > > > > > > Zhang Yifei
> > > > > > > > > >> > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > > > --
> > > > > > > > > >> > > > > > > > > > > Luciano Resende
> > > > > > > > > >> > > > > > > > > > > http://twitter.com/lresende1975
> > > > > > > > > >> > > > > > > > > > > http://lresende.blogspot.com/
> > > > > > > > > >> > > > > > > > > > >
> > > > > > > > > >> > > > > > > > > >
> > > > > > > > > >> > > > > > > > >
> > > > > > > > > >> > > > > > > > >
> > > > > > > > > >> > > > > > > > > --
> > > > > > > > > >> > > > > > > > >
> > > > > > > > > >> >
> > > > --------------------------------------------------------------
> > > > > > > > > >> > > > > > > > > Zhang Yifei
> > > > > > > > > >> > > > > > > > >
> > > > > > > > > >> > > > > > > >
> > > > > > > > > >> > > > > > >
> > > > > > > > > >> > > > > >
> > > > > > > > > >> > > > >
> > > > > > > > > >> > > >
> > > > > > > > > >> > >
> > > > > > > > > >> >
> > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: POCs, Prototypes, experiments ...

Posted by Rafael Novello <ra...@gmail.com>.

Hi Guys!

Let me ask about this new version that include CLI's and REPL. Will we
create different versions of CLI?

One of the reasons to create the REPL is to avoid re-implementing the same
features many times,
each time for a different language.

Regards!
Atenciosamente,
Rafael J. R. Novello

Skype: rafael.novello
Blog: http://rafanovello.blogspot.com.br/


Em sex, 26 de out de 2018 às 20:54, Daniel Takabayashi <
daniel.takabayashi@gmail.com> escreveu:

> +1
>
> Em sex, 26 de out de 2018 às 16:47, Lucas Bonatto Miguel <
> lucasbm88@apache.org> escreveu:
>
> > Makes perfect sense to me. I believe that if the majority of people
> agrees
> > with this architecture we could create issues and start implementing it.
> >
> > +1
> >
> >
> > On Fri, Oct 26, 2018 at 4:09 PM Daniel Takabayashi <
> > daniel.takabayashi@gmail.com> wrote:
> >
> > > I see... what about this new version?
> > >
> > >
> > >
> >
> https://docs.google.com/drawings/d/shajxIpLJHxxMbFgDXiPuhg/image?w=602&h=468&rev=1630&ac=1&parent=1ySERHGBXbHeyCMRookq5UfTuFkzzU0ugtjvR3rF3deY
> > >
> > > I changed the toolbox ...from 1 component to 3... docker client, cli
> and
> > > repl. Both (cli and repl) now are using the docker client to control
> the
> > > engines.
> > >
> > > Em sex, 26 de out de 2018 às 14:49, Lucas Bonatto Miguel <
> > > lucasbm88@apache.org> escreveu:
> > >
> > > > For sure, both suggestions are to make sure we're not killing the CLI
> > and
> > > > that we have a component (the SDK client) that is shared by CLI and
> > REPL.
> > > >
> > > > Thanks for clarifying where the system calls will live.
> > > >
> > > > - Lucas
> > > >
> > > > On Fri, Oct 26, 2018 at 2:39 PM Daniel Takabayashi <
> > > > daniel.takabayashi@gmail.com> wrote:
> > > >
> > > > > Hi Lucas, could you please justified your two suggestions?
> > > > >
> > > > > The general ideia is to use the Docker DSK to execute local
> commands,
> > > to
> > > > > consume outputs and also to execute Marvin "specific language"
> > > commands.
> > > > > Today almost all CLIs commands in Marvin is a mixed of systems
> > commands
> > > > and
> > > > > library method calls. The plan is to put all "specific language
> > calls"
> > > > > inside the marvin-common-libs and group the systems call on the
> > toolbox
> > > > > (using the docker SDK in this task).
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > Em qui, 25 de out de 2018 às 20:09, Lucas Bonatto Miguel <
> > > > > lucasbm88@apache.org> escreveu:
> > > > >
> > > > > > Great, thanks.
> > > > > >
> > > > > > So here some suggestions:
> > > > > >  1. We could create one new layer between Toolbox and the
> Engines,
> > > this
> > > > > > would be Docker client.
> > > > > >  2. Split the Toolbox layer in two. Toolbox CLI and Toolbox REPL
> > > > > >
> > > > > > It's not clear to me if the commands listed on General Commands
> > will
> > > > need
> > > > > > to be written for every language, from the image I believe not,
> but
> > > how
> > > > > it
> > > > > > would work, since starting a notebook in Python is different than
> > > > > starting
> > > > > > in Scala.
> > > > > >
> > > > > > - Lucas
> > > > > > On Thu, Oct 25, 2018 at 4:51 PM Daniel Takabayashi <
> > > > > > daniel.takabayashi@gmail.com> wrote:
> > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://docs.google.com/drawings/d/shajxIpLJHxxMbFgDXiPuhg/image?w=602&h=461&rev=1423&ac=1&parent=1ySERHGBXbHeyCMRookq5UfTuFkzzU0ugtjvR3rF3deY
> > > > > > >
> > > > > > > Em qui, 25 de out de 2018 às 07:38, Rafael Novello <
> > > > > > > rafa.reis.novello@gmail.com> escreveu:
> > > > > > >
> > > > > > > > Hi Taka,
> > > > > > > >
> > > > > > > > I can't see the image! Maybe the Apache email list don't
> > support
> > > > > > > > attachments.
> > > > > > > >
> > > > > > > > Best!
> > > > > > > > Rafael J. R. Novello
> > > > > > > >
> > > > > > > > Skype: rafael.novello
> > > > > > > > Blog: http://rafanovello.blogspot.com.br/
> > > > > > > >
> > > > > > > >
> > > > > > > > Em qua, 24 de out de 2018 às 21:14, Daniel Takabayashi <
> > > > > > > > daniel.takabayashi@gmail.com> escreveu:
> > > > > > > >
> > > > > > > > > Hi all,
> > > > > > > > >
> > > > > > > > > To try to give a little more details about this "new
> concept"
> > > > that
> > > > > we
> > > > > > > > want
> > > > > > > > > to bring to the marvin toolbox, I did this simple
> > architecture
> > > > > draw.
> > > > > > > > >
> > > > > > > > > [image: marvin-architecture-views (5).png]
> > > > > > > > >
> > > > > > > > > The general idea here is try to transform the toolbox
> > something
> > > > > > > > > disconnected with the "language", something more agnostic.
> > Also
> > > > in
> > > > > > this
> > > > > > > > new
> > > > > > > > > architecture we could use remote resource to process
> engines
> > > and
> > > > > make
> > > > > > > > easy
> > > > > > > > > the support for new languages.
> > > > > > > > >
> > > > > > > > > This "new toolbox" will be the only thing that a Marvin
> user
> > > must
> > > > > to
> > > > > > > > > install and also we could start to support multiples O.S
> > (once
> > > > the
> > > > > > REPL
> > > > > > > > is
> > > > > > > > > a dummy application that only interprets and by pass
> > commands).
> > > > > > > > >
> > > > > > > > > Regards,
> > > > > > > > > Taka
> > > > > > > > >
> > > > > > > > > Em qua, 24 de out de 2018 às 09:52, Rafael Novello <
> > > > > > > > > rafa.reis.novello@gmail.com> escreveu:
> > > > > > > > >
> > > > > > > > >> Alan,
> > > > > > > > >>
> > > > > > > > >> Yes! We are using the Docker SDK and it's possible to use
> > the
> > > > API
> > > > > to
> > > > > > > > >> automate, but it's a little bit harder than automate CLI
> > > calls.
> > > > > > > > >>
> > > > > > > > >> Atenciosamente,
> > > > > > > > >> Rafael J. R. Novello
> > > > > > > > >>
> > > > > > > > >> Skype: rafael.novello
> > > > > > > > >> Blog: http://rafanovello.blogspot.com.br/
> > > > > > > > >>
> > > > > > > > >>
> > > > > > > > >> Em qua, 24 de out de 2018 às 12:02, Alan Silva <
> > > > jumpi@apache.org>
> > > > > > > > >> escreveu:
> > > > > > > > >>
> > > > > > > > >> > Hi,
> > > > > > > > >> >
> > > > > > > > >> > One question here, I understand that we start to use
> with
> > > this
> > > > > PoC
> > > > > > > the
> > > > > > > > >> > Docker SDK API, right?
> > > > > > > > >> >
> > > > > > > > >> > Why not use the API to expose some endpoints to permit
> > this
> > > > kind
> > > > > > of
> > > > > > > > >> > automation by devops?
> > > > > > > > >> >
> > > > > > > > >> > I think it is possible and it solves the CLI problem,
> > right?
> > > > > > > > >> >
> > > > > > > > >> >
> > > > > > > > >> > On Tue, Oct 23, 2018 at 1:05 PM Rafael Novello <
> > > > > > > > >> > rafa.reis.novello@gmail.com>
> > > > > > > > >> > wrote:
> > > > > > > > >> >
> > > > > > > > >> > > Lucas,
> > > > > > > > >> > >
> > > > > > > > >> > > Sorry, I didn't understood your question bellow.
> > > > > > > > >> > >
> > > > > > > > >> > > "Would it make sense to use the same solution that we
> > will
> > > > use
> > > > > > for
> > > > > > > > >> > having a
> > > > > > > > >> > > single-language REPL to have a single-language CLI?"
> > > > > > > > >> > >
> > > > > > > > >> > > For DevOps purposes, maybe this new toolbox concept is
> > not
> > > > > > ideal.
> > > > > > > I
> > > > > > > > >> think
> > > > > > > > >> > > we can keep the CLI inside the docker container but it
> > > will
> > > > > not
> > > > > > > easy
> > > > > > > > >> to
> > > > > > > > >> > > automate
> > > > > > > > >> > > jobs by this way.
> > > > > > > > >> > >
> > > > > > > > >> > > How to deal with this issue? Voting?
> > > > > > > > >> > > Atenciosamente,
> > > > > > > > >> > > Rafael J. R. Novello
> > > > > > > > >> > >
> > > > > > > > >> > > Skype: rafael.novello
> > > > > > > > >> > > Blog: http://rafanovello.blogspot.com.br/
> > > > > > > > >> > >
> > > > > > > > >> > >
> > > > > > > > >> > > Em sex, 19 de out de 2018 às 19:00, Lucas Bonatto
> > Miguel <
> > > > > > > > >> > > lucasbm88@apache.org> escreveu:
> > > > > > > > >> > >
> > > > > > > > >> > > > Got it! Thanks for clarifying.
> > > > > > > > >> > > >
> > > > > > > > >> > > > Would it make sense to use the same solution that we
> > > will
> > > > > use
> > > > > > > for
> > > > > > > > >> > having
> > > > > > > > >> > > a
> > > > > > > > >> > > > single-language REPL to have a single-language CLI?
> My
> > > > only
> > > > > > > > concern
> > > > > > > > >> > with
> > > > > > > > >> > > > killing the CLI is that you remove an essential
> > feature
> > > > for
> > > > > > > > DevOps.
> > > > > > > > >> > > >
> > > > > > > > >> > > > - Lucas
> > > > > > > > >> > > >
> > > > > > > > >> > > > On Fri, Oct 19, 2018 at 1:52 PM Rafael Novello <
> > > > > > > > >> > > > rafa.reis.novello@gmail.com>
> > > > > > > > >> > > > wrote:
> > > > > > > > >> > > >
> > > > > > > > >> > > > > Lucas,
> > > > > > > > >> > > > >
> > > > > > > > >> > > > > The idea is that REPL will substitute the actual
> > CLI.
> > > > It's
> > > > > > > > because
> > > > > > > > >> > with
> > > > > > > > >> > > > the
> > > > > > > > >> > > > > actual concept (using language specific CLI) we
> will
> > > > need
> > > > > to
> > > > > > > > >> > > re-implement
> > > > > > > > >> > > > > the same features many times and probably each CLI
> > > will
> > > > > > have a
> > > > > > > > >> > > different
> > > > > > > > >> > > > > behavior because some language specific
> restrictions
> > > > > and/or
> > > > > > > > >> > > limitations.
> > > > > > > > >> > > > >
> > > > > > > > >> > > > > With this new concept, all users will have the
> same
> > > > > > experience
> > > > > > > > >> > > > interacting
> > > > > > > > >> > > > > with Marvin REPL and they will use the bests tools
> > to
> > > > > > develop
> > > > > > > > >> their
> > > > > > > > >> > > > engines
> > > > > > > > >> > > > > (Jupyter Notebook and/or Lab with different
> > languages
> > > > > > support
> > > > > > > or
> > > > > > > > >> even
> > > > > > > > >> > > > > Apache Zeppelin). All the interact will occur
> > through
> > > > REPL
> > > > > > and
> > > > > > > > >> docker
> > > > > > > > >> > > > > protocol.
> > > > > > > > >> > > > >
> > > > > > > > >> > > > > Alan, Yes! As Lucas said, the concept is the same
> > but
> > > we
> > > > > can
> > > > > > > use
> > > > > > > > >> > docker
> > > > > > > > >> > > > to
> > > > > > > > >> > > > > do the same job.
> > > > > > > > >> > > > >
> > > > > > > > >> > > > > Best!
> > > > > > > > >> > > > >
> > > > > > > > >> > > > > Em sex, 19 de out de 2018 às 00:39, Lucas Bonatto
> > > > Miguel <
> > > > > > > > >> > > > > lucasbm88@apache.org> escreveu:
> > > > > > > > >> > > > >
> > > > > > > > >> > > > > > Thanks for the clarifications Rafael, so from
> > what I
> > > > > > > > understood,
> > > > > > > > >> > > > Marvin's
> > > > > > > > >> > > > > > developers would not use the REPL to explore
> data
> > > and
> > > > > test
> > > > > > > > >> models,
> > > > > > > > >> > > i.e.
> > > > > > > > >> > > > > > develop the engine. Is the idea to build
> something
> > > > more
> > > > > > like
> > > > > > > > an
> > > > > > > > >> > > > > interactive
> > > > > > > > >> > > > > > CLI? I think an interactive CLI would be useful
> > for
> > > > the
> > > > > > > > >> developer
> > > > > > > > >> > > > > > experience, however, it's important to keep the
> > > > > unattended
> > > > > > > > >> > (current)
> > > > > > > > >> > > > CLI
> > > > > > > > >> > > > > in
> > > > > > > > >> > > > > > place for scenarios where users want to automate
> > > > Marvin
> > > > > > > tasks.
> > > > > > > > >> > > > > >
> > > > > > > > >> > > > > > Alan, I believe the idea is still the same as
> you
> > > > > started,
> > > > > > > but
> > > > > > > > >> > using
> > > > > > > > >> > > > > > docker SDK now.
> > > > > > > > >> > > > > >
> > > > > > > > >> > > > > > - Lucas
> > > > > > > > >> > > > > >
> > > > > > > > >> > > > > > On Thu, Oct 18, 2018 at 1:29 PM Rafael Novello <
> > > > > > > > >> > > > > > rafa.reis.novello@gmail.com>
> > > > > > > > >> > > > > > wrote:
> > > > > > > > >> > > > > >
> > > > > > > > >> > > > > > > Hi Lucas!
> > > > > > > > >> > > > > > >
> > > > > > > > >> > > > > > > First of all +1 for REPL POCs to Apache!
> > > > > > > > >> > > > > > >
> > > > > > > > >> > > > > > > Let me help with some comments:
> > > > > > > > >> > > > > > >
> > > > > > > > >> > > > > > > 1 - We have tested NodeJS, Scala and Python
> and
> > > the
> > > > > > > easiest
> > > > > > > > >> one
> > > > > > > > >> > was
> > > > > > > > >> > > > > > Python.
> > > > > > > > >> > > > > > > We have found a small project [1] that have
> all
> > > > > features
> > > > > > > we
> > > > > > > > >> > desired
> > > > > > > > >> > > > for
> > > > > > > > >> > > > > > > REPL:
> > > > > > > > >> > > > > > > - Autocomplete
> > > > > > > > >> > > > > > > - Python commands disabled (the user have only
> > the
> > > > > > > commands
> > > > > > > > we
> > > > > > > > >> > > > > provide).
> > > > > > > > >> > > > > > > The other languages REPL options don't have
> this
> > > > > > feature.
> > > > > > > > >> > > > > > > - Easy to show output
> > > > > > > > >> > > > > > > - Etc
> > > > > > > > >> > > > > > > So, I think the language chosen will not be
> > > > important
> > > > > > here
> > > > > > > > >> > because
> > > > > > > > >> > > > the
> > > > > > > > >> > > > > > user
> > > > > > > > >> > > > > > > will only interact with the commands that we
> > > create.
> > > > > > > > >> > > > > > >
> > > > > > > > >> > > > > > > 2 - The "engine-generate" command will
> download
> > a
> > > > > docker
> > > > > > > > image
> > > > > > > > >> > that
> > > > > > > > >> > > > we
> > > > > > > > >> > > > > > > create for that language and start a container
> > > with
> > > > > the
> > > > > > > > basic
> > > > > > > > >> > > project
> > > > > > > > >> > > > > > > structure for that language.
> > > > > > > > >> > > > > > >
> > > > > > > > >> > > > > > > 3 - The REPL client will use the "docker
> > protocol"
> > > > to
> > > > > > run
> > > > > > > > >> > command,
> > > > > > > > >> > > > > > > start/stop services and etc inside the
> container
> > > and
> > > > > it
> > > > > > > will
> > > > > > > > >> > > receive
> > > > > > > > >> > > > > the
> > > > > > > > >> > > > > > > log stream to show. No, the REPL will no pass
> > code
> > > > > > > snippets
> > > > > > > > >> for
> > > > > > > > >> > > > docker
> > > > > > > > >> > > > > > > container (I think it will not be necessary)
> > > > > > > > >> > > > > > >
> > > > > > > > >> > > > > > > 4 - Yep! Like I said on the first item.
> > > > > > > > >> > > > > > >
> > > > > > > > >> > > > > > > [1] - https://github.com/italorossi/ishell
> > > > > > > > >> > > > > > >
> > > > > > > > >> > > > > > > Let me know if there is any other question!
> > > > > > > > >> > > > > > > Best!
> > > > > > > > >> > > > > > > Rafael J. R. Novello
> > > > > > > > >> > > > > > >
> > > > > > > > >> > > > > > > Skype: rafael.novello
> > > > > > > > >> > > > > > > Blog: http://rafanovello.blogspot.com.br/
> > > > > > > > >> > > > > > >
> > > > > > > > >> > > > > > >
> > > > > > > > >> > > > > > > Em qui, 18 de out de 2018 às 17:00, Lucas
> > Bonatto
> > > > > > Miguel <
> > > > > > > > >> > > > > > > lucasbm88@apache.org> escreveu:
> > > > > > > > >> > > > > > >
> > > > > > > > >> > > > > > > > + 1 for migrating the REPL repo to Apache
> > > > > > > > >> > > > > > > >
> > > > > > > > >> > > > > > > > I have a few questions about the previous
> > > > > explanation:
> > > > > > > > >> > > > > > > >  1) The REPL itself, it would be an
> > application
> > > in
> > > > > > which
> > > > > > > > >> > > language?
> > > > > > > > >> > > > > > > Remember
> > > > > > > > >> > > > > > > > that the main idea is to allow the user to
> > > program
> > > > > on
> > > > > > > his
> > > > > > > > >> > > preferred
> > > > > > > > >> > > > > > > > language in the REPL.
> > > > > > > > >> > > > > > > >  2) Should the engine-generate command also
> > > > > generate a
> > > > > > > > >> docker
> > > > > > > > >> > > image
> > > > > > > > >> > > > > > with
> > > > > > > > >> > > > > > > > the user's application?
> > > > > > > > >> > > > > > > >  3) What type of communication would happen
> > > > between
> > > > > > the
> > > > > > > > REPL
> > > > > > > > >> > and
> > > > > > > > >> > > > the
> > > > > > > > >> > > > > > > engine
> > > > > > > > >> > > > > > > > via Docker SDK? Would the REPL pass snippets
> > of
> > > > code
> > > > > > to
> > > > > > > be
> > > > > > > > >> > > executed
> > > > > > > > >> > > > > by
> > > > > > > > >> > > > > > > the
> > > > > > > > >> > > > > > > > docker container?
> > > > > > > > >> > > > > > > >  4) Have you considered code completion in
> the
> > > > REPL?
> > > > > > > > >> > > > > > > >
> > > > > > > > >> > > > > > > >
> > > > > > > > >> > > > > > > > On Thu, Oct 18, 2018 at 7:53 AM Zhang Yifei
> <
> > > > > > > > >> > > > yifei.z.luiz@gmail.com>
> > > > > > > > >> > > > > > > > wrote:
> > > > > > > > >> > > > > > > >
> > > > > > > > >> > > > > > > > > Ok guys,
> > > > > > > > >> > > > > > > > >
> > > > > > > > >> > > > > > > > > The basic idea is to provide only one
> > Toolbox
> > > > for
> > > > > > > > multiple
> > > > > > > > >> > > > > languages.
> > > > > > > > >> > > > > > > > > We are looking for possibility to build a
> > > single
> > > > > > > Marvin
> > > > > > > > >> Repl,
> > > > > > > > >> > > > > > > > > instead of severals toolboxes with
> > differentes
> > > > > > > > interfaces
> > > > > > > > >> or
> > > > > > > > >> > > > > > commands.
> > > > > > > > >> > > > > > > > >
> > > > > > > > >> > > > > > > > > In this case, the engine-generate command
> > will
> > > > > > > download
> > > > > > > > >> and
> > > > > > > > >> > > > start a
> > > > > > > > >> > > > > > > > Docker
> > > > > > > > >> > > > > > > > > container with basic engine structure
> > > > > corresponding
> > > > > > > the
> > > > > > > > >> > choosed
> > > > > > > > >> > > > > > > language.
> > > > > > > > >> > > > > > > > > this means we don't need to build
> Toolboxes
> > of
> > > > > > > > differents
> > > > > > > > >> > > > > languages,
> > > > > > > > >> > > > > > we
> > > > > > > > >> > > > > > > > > just
> > > > > > > > >> > > > > > > > > build the engine template of all languages
> > > that
> > > > we
> > > > > > > want
> > > > > > > > to
> > > > > > > > >> > > > support
> > > > > > > > >> > > > > > and
> > > > > > > > >> > > > > > > > > provide it
> > > > > > > > >> > > > > > > > > as Docker containers
> > > > > > > > >> > > > > > > > >
> > > > > > > > >> > > > > > > > > We have started researches around our
> basic
> > > > > > > requirements
> > > > > > > > >> > like:
> > > > > > > > >> > > > > > > > > - Repl interface
> > > > > > > > >> > > > > > > > > - System communication
> > > > > > > > >> > > > > > > > > - Connection security
> > > > > > > > >> > > > > > > > > - Tool popularity
> > > > > > > > >> > > > > > > > > - Update complexity
> > > > > > > > >> > > > > > > > > - Languages support
> > > > > > > > >> > > > > > > > > - ......
> > > > > > > > >> > > > > > > > >
> > > > > > > > >> > > > > > > > > And we did some POC with code here:
> > > > > > > > >> > > > > > > > > https://github.com/marvin-ai/marvin-repl
> > > > > > > > >> > > > > > > > >
> > > > > > > > >> > > > > > > > > There is POC testing gRPC using Scala and
> > > > Python,
> > > > > > > > >> > > > > > > > > Repl inteface and Docker SDK with NodeJS,
> > > > > > > > >> > > > > > > > > Repl interface and Docker SDK with Python.
> > > > > > > > >> > > > > > > > >
> > > > > > > > >> > > > > > > > > At this moment we prefer the Repl
> interface
> > +
> > > > > Docker
> > > > > > > SDK
> > > > > > > > >> way,
> > > > > > > > >> > > > > because
> > > > > > > > >> > > > > > > > good
> > > > > > > > >> > > > > > > > > part of the requirements
> > > > > > > > >> > > > > > > > > will be guaranteed by Docker.
> > > > > > > > >> > > > > > > > >
> > > > > > > > >> > > > > > > > > With this informations, what do you think?
> > > > Should
> > > > > we
> > > > > > > > >> submit
> > > > > > > > >> > all
> > > > > > > > >> > > > > this
> > > > > > > > >> > > > > > > POCs
> > > > > > > > >> > > > > > > > > to Apache Repo?
> > > > > > > > >> > > > > > > > > Please feel free to opine.
> > > > > > > > >> > > > > > > > >
> > > > > > > > >> > > > > > > > >
> > > > > > > > >> > > > > > > > > Thats all, thanks!!!
> > > > > > > > >> > > > > > > > >
> > > > > > > > >> > > > > > > > >
> > > > > > > > >> > > > > > > > > Em ter, 16 de out de 2018 às 18:55, Daniel
> > > > > > > Takabayashi <
> > > > > > > > >> > > > > > > > > daniel.takabayashi@gmail.com> escreveu:
> > > > > > > > >> > > > > > > > >
> > > > > > > > >> > > > > > > > > > Zhang,
> > > > > > > > >> > > > > > > > > >
> > > > > > > > >> > > > > > > > > > I think the best approach is give us a
> > > better
> > > > > > > > >> explanation
> > > > > > > > >> > > about
> > > > > > > > >> > > > > > this
> > > > > > > > >> > > > > > > > new
> > > > > > > > >> > > > > > > > > > feature and how this can help us to
> > archive
> > > > the
> > > > > > > > desired
> > > > > > > > >> > > feature
> > > > > > > > >> > > > > > > > (support
> > > > > > > > >> > > > > > > > > > multiple languages). Than if the
> majority
> > > > agree
> > > > > > > than I
> > > > > > > > >> can
> > > > > > > > >> > do
> > > > > > > > >> > > > the
> > > > > > > > >> > > > > > > > merge,
> > > > > > > > >> > > > > > > > > > just like I did before, but in another
> > > branch.
> > > > > > > > >> > > > > > > > > >
> > > > > > > > >> > > > > > > > > > Let me know if makes sense, ok?
> > > > > > > > >> > > > > > > > > >
> > > > > > > > >> > > > > > > > > > Taka
> > > > > > > > >> > > > > > > > > >
> > > > > > > > >> > > > > > > > > > Em ter, 16 de out de 2018 às 14:00,
> > Luciano
> > > > > > Resende
> > > > > > > <
> > > > > > > > >> > > > > > > > > luckbr1975@gmail.com>
> > > > > > > > >> > > > > > > > > > escreveu:
> > > > > > > > >> > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > So, what is the POC, is it a
> refactoring
> > > of
> > > > > the
> > > > > > > > >> existing
> > > > > > > > >> > > > repo?
> > > > > > > > >> > > > > Or
> > > > > > > > >> > > > > > > is
> > > > > > > > >> > > > > > > > > > > it a new rewrite of the repo?
> > > > > > > > >> > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > Just asking as it might make sense to
> > make
> > > > it
> > > > > a
> > > > > > > > branch
> > > > > > > > >> > then
> > > > > > > > >> > > > > > > actually
> > > > > > > > >> > > > > > > > a
> > > > > > > > >> > > > > > > > > > > thing in parallel, as this will have
> an
> > > > effect
> > > > > > for
> > > > > > > > >> > > releases,
> > > > > > > > >> > > > > etc.
> > > > > > > > >> > > > > > > but
> > > > > > > > >> > > > > > > > > > > you guys know more here than I do.
> > > > > > > > >> > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > Also, it's probably good to have a
> write
> > > up
> > > > of
> > > > > > the
> > > > > > > > >> main
> > > > > > > > >> > > > > direction
> > > > > > > > >> > > > > > > of
> > > > > > > > >> > > > > > > > > > > the design that can help people get
> > > familiar
> > > > > > with
> > > > > > > > the
> > > > > > > > >> new
> > > > > > > > >> > > > > > approach.
> > > > > > > > >> > > > > > > > > > > On Tue, Oct 16, 2018 at 11:12 AM Zhang
> > > > Yifei <
> > > > > > > > >> > > > > > > yifei.z.luiz@gmail.com
> > > > > > > > >> > > > > > > > >
> > > > > > > > >> > > > > > > > > > > wrote:
> > > > > > > > >> > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > Hey guys,
> > > > > > > > >> > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > We have reorganized the Poc repo,
> and
> > > want
> > > > > to
> > > > > > > > merge
> > > > > > > > >> it
> > > > > > > > >> > to
> > > > > > > > >> > > > our
> > > > > > > > >> > > > > > > > Apache
> > > > > > > > >> > > > > > > > > > > repo.
> > > > > > > > >> > > > > > > > > > > > Just making sure here before we do
> the
> > > > > merge,
> > > > > > > > >> because
> > > > > > > > >> > im
> > > > > > > > >> > > > not
> > > > > > > > >> > > > > > sure
> > > > > > > > >> > > > > > > > if
> > > > > > > > >> > > > > > > > > > > doing
> > > > > > > > >> > > > > > > > > > > > this will perserve the Git commit
> > > history.
> > > > > > > > >> > > > > > > > > > > > What we are planning to do is:
> > > > > > > > >> > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > git filter-branch
> > --subdirectory-filter
> > > > <git
> > > > > > > > >> > > > > origin_repository
> > > > > > > > >> > > > > > > > > > directory>
> > > > > > > > >> > > > > > > > > > > > -- --all
> > > > > > > > >> > > > > > > > > > > > mkdir POCs/
> > > > > > > > >> > > > > > > > > > > > git mv * POCs/
> > > > > > > > >> > > > > > > > > > > > git commit -m "colleted the data to
> > > move"
> > > > > > > > >> > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > git clone <git
> > > destination_repository_url>
> > > > > cd
> > > > > > > <git
> > > > > > > > >> > > > > > > > > > > > destination_repository_directory>
> git
> > > > remote
> > > > > > add
> > > > > > > > >> > > > > > > > > > origin_repository_branch
> > > > > > > > >> > > > > > > > > > > > <git origin_repository directory>
> git
> > > pull
> > > > > > > > >> > > > > > > origin_repository_branch
> > > > > > > > >> > > > > > > > > > > master
> > > > > > > > >> > > > > > > > > > > > --allow-unrelated-histories git
> remote
> > > rm
> > > > > > > > >> > > > > > > origin_repository_branch
> > > > > > > > >> > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > > Thanks in advance !
> > > > > > > > >> > > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > >
> > > > > > > > >> > > > >
> > > > > > --------------------------------------------------------------
> > > > > > > > >> > > > > > > > > > > > Zhang Yifei
> > > > > > > > >> > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > >
> > > > > > > > >> > > > > > > > > > > --
> > > > > > > > >> > > > > > > > > > > Luciano Resende
> > > > > > > > >> > > > > > > > > > > http://twitter.com/lresende1975
> > > > > > > > >> > > > > > > > > > > http://lresende.blogspot.com/
> > > > > > > > >> > > > > > > > > > >
> > > > > > > > >> > > > > > > > > >
> > > > > > > > >> > > > > > > > >
> > > > > > > > >> > > > > > > > >
> > > > > > > > >> > > > > > > > > --
> > > > > > > > >> > > > > > > > >
> > > > > > > > >> >
> > > --------------------------------------------------------------
> > > > > > > > >> > > > > > > > > Zhang Yifei
> > > > > > > > >> > > > > > > > >
> > > > > > > > >> > > > > > > >
> > > > > > > > >> > > > > > >
> > > > > > > > >> > > > > >
> > > > > > > > >> > > > >
> > > > > > > > >> > > >
> > > > > > > > >> > >
> > > > > > > > >> >
> > > > > > > > >>
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: POCs, Prototypes, experiments ...

Posted by Daniel Takabayashi <da...@gmail.com>.

+1

Em sex, 26 de out de 2018 às 16:47, Lucas Bonatto Miguel <
lucasbm88@apache.org> escreveu:

> Makes perfect sense to me. I believe that if the majority of people agrees
> with this architecture we could create issues and start implementing it.
>
> +1
>
>
> On Fri, Oct 26, 2018 at 4:09 PM Daniel Takabayashi <
> daniel.takabayashi@gmail.com> wrote:
>
> > I see... what about this new version?
> >
> >
> >
> https://docs.google.com/drawings/d/shajxIpLJHxxMbFgDXiPuhg/image?w=602&h=468&rev=1630&ac=1&parent=1ySERHGBXbHeyCMRookq5UfTuFkzzU0ugtjvR3rF3deY
> >
> > I changed the toolbox ...from 1 component to 3... docker client, cli and
> > repl. Both (cli and repl) now are using the docker client to control the
> > engines.
> >
> > Em sex, 26 de out de 2018 às 14:49, Lucas Bonatto Miguel <
> > lucasbm88@apache.org> escreveu:
> >
> > > For sure, both suggestions are to make sure we're not killing the CLI
> and
> > > that we have a component (the SDK client) that is shared by CLI and
> REPL.
> > >
> > > Thanks for clarifying where the system calls will live.
> > >
> > > - Lucas
> > >
> > > On Fri, Oct 26, 2018 at 2:39 PM Daniel Takabayashi <
> > > daniel.takabayashi@gmail.com> wrote:
> > >
> > > > Hi Lucas, could you please justified your two suggestions?
> > > >
> > > > The general ideia is to use the Docker DSK to execute local commands,
> > to
> > > > consume outputs and also to execute Marvin "specific language"
> > commands.
> > > > Today almost all CLIs commands in Marvin is a mixed of systems
> commands
> > > and
> > > > library method calls. The plan is to put all "specific language
> calls"
> > > > inside the marvin-common-libs and group the systems call on the
> toolbox
> > > > (using the docker SDK in this task).
> > > >
> > > >
> > > >
> > > >
> > > > Em qui, 25 de out de 2018 às 20:09, Lucas Bonatto Miguel <
> > > > lucasbm88@apache.org> escreveu:
> > > >
> > > > > Great, thanks.
> > > > >
> > > > > So here some suggestions:
> > > > >  1. We could create one new layer between Toolbox and the Engines,
> > this
> > > > > would be Docker client.
> > > > >  2. Split the Toolbox layer in two. Toolbox CLI and Toolbox REPL
> > > > >
> > > > > It's not clear to me if the commands listed on General Commands
> will
> > > need
> > > > > to be written for every language, from the image I believe not, but
> > how
> > > > it
> > > > > would work, since starting a notebook in Python is different than
> > > > starting
> > > > > in Scala.
> > > > >
> > > > > - Lucas
> > > > > On Thu, Oct 25, 2018 at 4:51 PM Daniel Takabayashi <
> > > > > daniel.takabayashi@gmail.com> wrote:
> > > > >
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://docs.google.com/drawings/d/shajxIpLJHxxMbFgDXiPuhg/image?w=602&h=461&rev=1423&ac=1&parent=1ySERHGBXbHeyCMRookq5UfTuFkzzU0ugtjvR3rF3deY
> > > > > >
> > > > > > Em qui, 25 de out de 2018 às 07:38, Rafael Novello <
> > > > > > rafa.reis.novello@gmail.com> escreveu:
> > > > > >
> > > > > > > Hi Taka,
> > > > > > >
> > > > > > > I can't see the image! Maybe the Apache email list don't
> support
> > > > > > > attachments.
> > > > > > >
> > > > > > > Best!
> > > > > > > Rafael J. R. Novello
> > > > > > >
> > > > > > > Skype: rafael.novello
> > > > > > > Blog: http://rafanovello.blogspot.com.br/
> > > > > > >
> > > > > > >
> > > > > > > Em qua, 24 de out de 2018 às 21:14, Daniel Takabayashi <
> > > > > > > daniel.takabayashi@gmail.com> escreveu:
> > > > > > >
> > > > > > > > Hi all,
> > > > > > > >
> > > > > > > > To try to give a little more details about this "new concept"
> > > that
> > > > we
> > > > > > > want
> > > > > > > > to bring to the marvin toolbox, I did this simple
> architecture
> > > > draw.
> > > > > > > >
> > > > > > > > [image: marvin-architecture-views (5).png]
> > > > > > > >
> > > > > > > > The general idea here is try to transform the toolbox
> something
> > > > > > > > disconnected with the "language", something more agnostic.
> Also
> > > in
> > > > > this
> > > > > > > new
> > > > > > > > architecture we could use remote resource to process engines
> > and
> > > > make
> > > > > > > easy
> > > > > > > > the support for new languages.
> > > > > > > >
> > > > > > > > This "new toolbox" will be the only thing that a Marvin user
> > must
> > > > to
> > > > > > > > install and also we could start to support multiples O.S
> (once
> > > the
> > > > > REPL
> > > > > > > is
> > > > > > > > a dummy application that only interprets and by pass
> commands).
> > > > > > > >
> > > > > > > > Regards,
> > > > > > > > Taka
> > > > > > > >
> > > > > > > > Em qua, 24 de out de 2018 às 09:52, Rafael Novello <
> > > > > > > > rafa.reis.novello@gmail.com> escreveu:
> > > > > > > >
> > > > > > > >> Alan,
> > > > > > > >>
> > > > > > > >> Yes! We are using the Docker SDK and it's possible to use
> the
> > > API
> > > > to
> > > > > > > >> automate, but it's a little bit harder than automate CLI
> > calls.
> > > > > > > >>
> > > > > > > >> Atenciosamente,
> > > > > > > >> Rafael J. R. Novello
> > > > > > > >>
> > > > > > > >> Skype: rafael.novello
> > > > > > > >> Blog: http://rafanovello.blogspot.com.br/
> > > > > > > >>
> > > > > > > >>
> > > > > > > >> Em qua, 24 de out de 2018 às 12:02, Alan Silva <
> > > jumpi@apache.org>
> > > > > > > >> escreveu:
> > > > > > > >>
> > > > > > > >> > Hi,
> > > > > > > >> >
> > > > > > > >> > One question here, I understand that we start to use with
> > this
> > > > PoC
> > > > > > the
> > > > > > > >> > Docker SDK API, right?
> > > > > > > >> >
> > > > > > > >> > Why not use the API to expose some endpoints to permit
> this
> > > kind
> > > > > of
> > > > > > > >> > automation by devops?
> > > > > > > >> >
> > > > > > > >> > I think it is possible and it solves the CLI problem,
> right?
> > > > > > > >> >
> > > > > > > >> >
> > > > > > > >> > On Tue, Oct 23, 2018 at 1:05 PM Rafael Novello <
> > > > > > > >> > rafa.reis.novello@gmail.com>
> > > > > > > >> > wrote:
> > > > > > > >> >
> > > > > > > >> > > Lucas,
> > > > > > > >> > >
> > > > > > > >> > > Sorry, I didn't understood your question bellow.
> > > > > > > >> > >
> > > > > > > >> > > "Would it make sense to use the same solution that we
> will
> > > use
> > > > > for
> > > > > > > >> > having a
> > > > > > > >> > > single-language REPL to have a single-language CLI?"
> > > > > > > >> > >
> > > > > > > >> > > For DevOps purposes, maybe this new toolbox concept is
> not
> > > > > ideal.
> > > > > > I
> > > > > > > >> think
> > > > > > > >> > > we can keep the CLI inside the docker container but it
> > will
> > > > not
> > > > > > easy
> > > > > > > >> to
> > > > > > > >> > > automate
> > > > > > > >> > > jobs by this way.
> > > > > > > >> > >
> > > > > > > >> > > How to deal with this issue? Voting?
> > > > > > > >> > > Atenciosamente,
> > > > > > > >> > > Rafael J. R. Novello
> > > > > > > >> > >
> > > > > > > >> > > Skype: rafael.novello
> > > > > > > >> > > Blog: http://rafanovello.blogspot.com.br/
> > > > > > > >> > >
> > > > > > > >> > >
> > > > > > > >> > > Em sex, 19 de out de 2018 às 19:00, Lucas Bonatto
> Miguel <
> > > > > > > >> > > lucasbm88@apache.org> escreveu:
> > > > > > > >> > >
> > > > > > > >> > > > Got it! Thanks for clarifying.
> > > > > > > >> > > >
> > > > > > > >> > > > Would it make sense to use the same solution that we
> > will
> > > > use
> > > > > > for
> > > > > > > >> > having
> > > > > > > >> > > a
> > > > > > > >> > > > single-language REPL to have a single-language CLI? My
> > > only
> > > > > > > concern
> > > > > > > >> > with
> > > > > > > >> > > > killing the CLI is that you remove an essential
> feature
> > > for
> > > > > > > DevOps.
> > > > > > > >> > > >
> > > > > > > >> > > > - Lucas
> > > > > > > >> > > >
> > > > > > > >> > > > On Fri, Oct 19, 2018 at 1:52 PM Rafael Novello <
> > > > > > > >> > > > rafa.reis.novello@gmail.com>
> > > > > > > >> > > > wrote:
> > > > > > > >> > > >
> > > > > > > >> > > > > Lucas,
> > > > > > > >> > > > >
> > > > > > > >> > > > > The idea is that REPL will substitute the actual
> CLI.
> > > It's
> > > > > > > because
> > > > > > > >> > with
> > > > > > > >> > > > the
> > > > > > > >> > > > > actual concept (using language specific CLI) we will
> > > need
> > > > to
> > > > > > > >> > > re-implement
> > > > > > > >> > > > > the same features many times and probably each CLI
> > will
> > > > > have a
> > > > > > > >> > > different
> > > > > > > >> > > > > behavior because some language specific restrictions
> > > > and/or
> > > > > > > >> > > limitations.
> > > > > > > >> > > > >
> > > > > > > >> > > > > With this new concept, all users will have the same
> > > > > experience
> > > > > > > >> > > > interacting
> > > > > > > >> > > > > with Marvin REPL and they will use the bests tools
> to
> > > > > develop
> > > > > > > >> their
> > > > > > > >> > > > engines
> > > > > > > >> > > > > (Jupyter Notebook and/or Lab with different
> languages
> > > > > support
> > > > > > or
> > > > > > > >> even
> > > > > > > >> > > > > Apache Zeppelin). All the interact will occur
> through
> > > REPL
> > > > > and
> > > > > > > >> docker
> > > > > > > >> > > > > protocol.
> > > > > > > >> > > > >
> > > > > > > >> > > > > Alan, Yes! As Lucas said, the concept is the same
> but
> > we
> > > > can
> > > > > > use
> > > > > > > >> > docker
> > > > > > > >> > > > to
> > > > > > > >> > > > > do the same job.
> > > > > > > >> > > > >
> > > > > > > >> > > > > Best!
> > > > > > > >> > > > >
> > > > > > > >> > > > > Em sex, 19 de out de 2018 às 00:39, Lucas Bonatto
> > > Miguel <
> > > > > > > >> > > > > lucasbm88@apache.org> escreveu:
> > > > > > > >> > > > >
> > > > > > > >> > > > > > Thanks for the clarifications Rafael, so from
> what I
> > > > > > > understood,
> > > > > > > >> > > > Marvin's
> > > > > > > >> > > > > > developers would not use the REPL to explore data
> > and
> > > > test
> > > > > > > >> models,
> > > > > > > >> > > i.e.
> > > > > > > >> > > > > > develop the engine. Is the idea to build something
> > > more
> > > > > like
> > > > > > > an
> > > > > > > >> > > > > interactive
> > > > > > > >> > > > > > CLI? I think an interactive CLI would be useful
> for
> > > the
> > > > > > > >> developer
> > > > > > > >> > > > > > experience, however, it's important to keep the
> > > > unattended
> > > > > > > >> > (current)
> > > > > > > >> > > > CLI
> > > > > > > >> > > > > in
> > > > > > > >> > > > > > place for scenarios where users want to automate
> > > Marvin
> > > > > > tasks.
> > > > > > > >> > > > > >
> > > > > > > >> > > > > > Alan, I believe the idea is still the same as you
> > > > started,
> > > > > > but
> > > > > > > >> > using
> > > > > > > >> > > > > > docker SDK now.
> > > > > > > >> > > > > >
> > > > > > > >> > > > > > - Lucas
> > > > > > > >> > > > > >
> > > > > > > >> > > > > > On Thu, Oct 18, 2018 at 1:29 PM Rafael Novello <
> > > > > > > >> > > > > > rafa.reis.novello@gmail.com>
> > > > > > > >> > > > > > wrote:
> > > > > > > >> > > > > >
> > > > > > > >> > > > > > > Hi Lucas!
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > > > First of all +1 for REPL POCs to Apache!
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > > > Let me help with some comments:
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > > > 1 - We have tested NodeJS, Scala and Python and
> > the
> > > > > > easiest
> > > > > > > >> one
> > > > > > > >> > was
> > > > > > > >> > > > > > Python.
> > > > > > > >> > > > > > > We have found a small project [1] that have all
> > > > features
> > > > > > we
> > > > > > > >> > desired
> > > > > > > >> > > > for
> > > > > > > >> > > > > > > REPL:
> > > > > > > >> > > > > > > - Autocomplete
> > > > > > > >> > > > > > > - Python commands disabled (the user have only
> the
> > > > > > commands
> > > > > > > we
> > > > > > > >> > > > > provide).
> > > > > > > >> > > > > > > The other languages REPL options don't have this
> > > > > feature.
> > > > > > > >> > > > > > > - Easy to show output
> > > > > > > >> > > > > > > - Etc
> > > > > > > >> > > > > > > So, I think the language chosen will not be
> > > important
> > > > > here
> > > > > > > >> > because
> > > > > > > >> > > > the
> > > > > > > >> > > > > > user
> > > > > > > >> > > > > > > will only interact with the commands that we
> > create.
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > > > 2 - The "engine-generate" command will download
> a
> > > > docker
> > > > > > > image
> > > > > > > >> > that
> > > > > > > >> > > > we
> > > > > > > >> > > > > > > create for that language and start a container
> > with
> > > > the
> > > > > > > basic
> > > > > > > >> > > project
> > > > > > > >> > > > > > > structure for that language.
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > > > 3 - The REPL client will use the "docker
> protocol"
> > > to
> > > > > run
> > > > > > > >> > command,
> > > > > > > >> > > > > > > start/stop services and etc inside the container
> > and
> > > > it
> > > > > > will
> > > > > > > >> > > receive
> > > > > > > >> > > > > the
> > > > > > > >> > > > > > > log stream to show. No, the REPL will no pass
> code
> > > > > > snippets
> > > > > > > >> for
> > > > > > > >> > > > docker
> > > > > > > >> > > > > > > container (I think it will not be necessary)
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > > > 4 - Yep! Like I said on the first item.
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > > > [1] - https://github.com/italorossi/ishell
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > > > Let me know if there is any other question!
> > > > > > > >> > > > > > > Best!
> > > > > > > >> > > > > > > Rafael J. R. Novello
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > > > Skype: rafael.novello
> > > > > > > >> > > > > > > Blog: http://rafanovello.blogspot.com.br/
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > > > Em qui, 18 de out de 2018 às 17:00, Lucas
> Bonatto
> > > > > Miguel <
> > > > > > > >> > > > > > > lucasbm88@apache.org> escreveu:
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > > > > + 1 for migrating the REPL repo to Apache
> > > > > > > >> > > > > > > >
> > > > > > > >> > > > > > > > I have a few questions about the previous
> > > > explanation:
> > > > > > > >> > > > > > > >  1) The REPL itself, it would be an
> application
> > in
> > > > > which
> > > > > > > >> > > language?
> > > > > > > >> > > > > > > Remember
> > > > > > > >> > > > > > > > that the main idea is to allow the user to
> > program
> > > > on
> > > > > > his
> > > > > > > >> > > preferred
> > > > > > > >> > > > > > > > language in the REPL.
> > > > > > > >> > > > > > > >  2) Should the engine-generate command also
> > > > generate a
> > > > > > > >> docker
> > > > > > > >> > > image
> > > > > > > >> > > > > > with
> > > > > > > >> > > > > > > > the user's application?
> > > > > > > >> > > > > > > >  3) What type of communication would happen
> > > between
> > > > > the
> > > > > > > REPL
> > > > > > > >> > and
> > > > > > > >> > > > the
> > > > > > > >> > > > > > > engine
> > > > > > > >> > > > > > > > via Docker SDK? Would the REPL pass snippets
> of
> > > code
> > > > > to
> > > > > > be
> > > > > > > >> > > executed
> > > > > > > >> > > > > by
> > > > > > > >> > > > > > > the
> > > > > > > >> > > > > > > > docker container?
> > > > > > > >> > > > > > > >  4) Have you considered code completion in the
> > > REPL?
> > > > > > > >> > > > > > > >
> > > > > > > >> > > > > > > >
> > > > > > > >> > > > > > > > On Thu, Oct 18, 2018 at 7:53 AM Zhang Yifei <
> > > > > > > >> > > > yifei.z.luiz@gmail.com>
> > > > > > > >> > > > > > > > wrote:
> > > > > > > >> > > > > > > >
> > > > > > > >> > > > > > > > > Ok guys,
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > > The basic idea is to provide only one
> Toolbox
> > > for
> > > > > > > multiple
> > > > > > > >> > > > > languages.
> > > > > > > >> > > > > > > > > We are looking for possibility to build a
> > single
> > > > > > Marvin
> > > > > > > >> Repl,
> > > > > > > >> > > > > > > > > instead of severals toolboxes with
> differentes
> > > > > > > interfaces
> > > > > > > >> or
> > > > > > > >> > > > > > commands.
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > > In this case, the engine-generate command
> will
> > > > > > download
> > > > > > > >> and
> > > > > > > >> > > > start a
> > > > > > > >> > > > > > > > Docker
> > > > > > > >> > > > > > > > > container with basic engine structure
> > > > corresponding
> > > > > > the
> > > > > > > >> > choosed
> > > > > > > >> > > > > > > language.
> > > > > > > >> > > > > > > > > this means we don't need to build Toolboxes
> of
> > > > > > > differents
> > > > > > > >> > > > > languages,
> > > > > > > >> > > > > > we
> > > > > > > >> > > > > > > > > just
> > > > > > > >> > > > > > > > > build the engine template of all languages
> > that
> > > we
> > > > > > want
> > > > > > > to
> > > > > > > >> > > > support
> > > > > > > >> > > > > > and
> > > > > > > >> > > > > > > > > provide it
> > > > > > > >> > > > > > > > > as Docker containers
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > > We have started researches around our basic
> > > > > > requirements
> > > > > > > >> > like:
> > > > > > > >> > > > > > > > > - Repl interface
> > > > > > > >> > > > > > > > > - System communication
> > > > > > > >> > > > > > > > > - Connection security
> > > > > > > >> > > > > > > > > - Tool popularity
> > > > > > > >> > > > > > > > > - Update complexity
> > > > > > > >> > > > > > > > > - Languages support
> > > > > > > >> > > > > > > > > - ......
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > > And we did some POC with code here:
> > > > > > > >> > > > > > > > > https://github.com/marvin-ai/marvin-repl
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > > There is POC testing gRPC using Scala and
> > > Python,
> > > > > > > >> > > > > > > > > Repl inteface and Docker SDK with NodeJS,
> > > > > > > >> > > > > > > > > Repl interface and Docker SDK with Python.
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > > At this moment we prefer the Repl interface
> +
> > > > Docker
> > > > > > SDK
> > > > > > > >> way,
> > > > > > > >> > > > > because
> > > > > > > >> > > > > > > > good
> > > > > > > >> > > > > > > > > part of the requirements
> > > > > > > >> > > > > > > > > will be guaranteed by Docker.
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > > With this informations, what do you think?
> > > Should
> > > > we
> > > > > > > >> submit
> > > > > > > >> > all
> > > > > > > >> > > > > this
> > > > > > > >> > > > > > > POCs
> > > > > > > >> > > > > > > > > to Apache Repo?
> > > > > > > >> > > > > > > > > Please feel free to opine.
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > > Thats all, thanks!!!
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > > Em ter, 16 de out de 2018 às 18:55, Daniel
> > > > > > Takabayashi <
> > > > > > > >> > > > > > > > > daniel.takabayashi@gmail.com> escreveu:
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > > > Zhang,
> > > > > > > >> > > > > > > > > >
> > > > > > > >> > > > > > > > > > I think the best approach is give us a
> > better
> > > > > > > >> explanation
> > > > > > > >> > > about
> > > > > > > >> > > > > > this
> > > > > > > >> > > > > > > > new
> > > > > > > >> > > > > > > > > > feature and how this can help us to
> archive
> > > the
> > > > > > > desired
> > > > > > > >> > > feature
> > > > > > > >> > > > > > > > (support
> > > > > > > >> > > > > > > > > > multiple languages). Than if the majority
> > > agree
> > > > > > than I
> > > > > > > >> can
> > > > > > > >> > do
> > > > > > > >> > > > the
> > > > > > > >> > > > > > > > merge,
> > > > > > > >> > > > > > > > > > just like I did before, but in another
> > branch.
> > > > > > > >> > > > > > > > > >
> > > > > > > >> > > > > > > > > > Let me know if makes sense, ok?
> > > > > > > >> > > > > > > > > >
> > > > > > > >> > > > > > > > > > Taka
> > > > > > > >> > > > > > > > > >
> > > > > > > >> > > > > > > > > > Em ter, 16 de out de 2018 às 14:00,
> Luciano
> > > > > Resende
> > > > > > <
> > > > > > > >> > > > > > > > > luckbr1975@gmail.com>
> > > > > > > >> > > > > > > > > > escreveu:
> > > > > > > >> > > > > > > > > >
> > > > > > > >> > > > > > > > > > > So, what is the POC, is it a refactoring
> > of
> > > > the
> > > > > > > >> existing
> > > > > > > >> > > > repo?
> > > > > > > >> > > > > Or
> > > > > > > >> > > > > > > is
> > > > > > > >> > > > > > > > > > > it a new rewrite of the repo?
> > > > > > > >> > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > Just asking as it might make sense to
> make
> > > it
> > > > a
> > > > > > > branch
> > > > > > > >> > then
> > > > > > > >> > > > > > > actually
> > > > > > > >> > > > > > > > a
> > > > > > > >> > > > > > > > > > > thing in parallel, as this will have an
> > > effect
> > > > > for
> > > > > > > >> > > releases,
> > > > > > > >> > > > > etc.
> > > > > > > >> > > > > > > but
> > > > > > > >> > > > > > > > > > > you guys know more here than I do.
> > > > > > > >> > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > Also, it's probably good to have a write
> > up
> > > of
> > > > > the
> > > > > > > >> main
> > > > > > > >> > > > > direction
> > > > > > > >> > > > > > > of
> > > > > > > >> > > > > > > > > > > the design that can help people get
> > familiar
> > > > > with
> > > > > > > the
> > > > > > > >> new
> > > > > > > >> > > > > > approach.
> > > > > > > >> > > > > > > > > > > On Tue, Oct 16, 2018 at 11:12 AM Zhang
> > > Yifei <
> > > > > > > >> > > > > > > yifei.z.luiz@gmail.com
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > > > > wrote:
> > > > > > > >> > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > Hey guys,
> > > > > > > >> > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > We have reorganized the Poc repo, and
> > want
> > > > to
> > > > > > > merge
> > > > > > > >> it
> > > > > > > >> > to
> > > > > > > >> > > > our
> > > > > > > >> > > > > > > > Apache
> > > > > > > >> > > > > > > > > > > repo.
> > > > > > > >> > > > > > > > > > > > Just making sure here before we do the
> > > > merge,
> > > > > > > >> because
> > > > > > > >> > im
> > > > > > > >> > > > not
> > > > > > > >> > > > > > sure
> > > > > > > >> > > > > > > > if
> > > > > > > >> > > > > > > > > > > doing
> > > > > > > >> > > > > > > > > > > > this will perserve the Git commit
> > history.
> > > > > > > >> > > > > > > > > > > > What we are planning to do is:
> > > > > > > >> > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > git filter-branch
> --subdirectory-filter
> > > <git
> > > > > > > >> > > > > origin_repository
> > > > > > > >> > > > > > > > > > directory>
> > > > > > > >> > > > > > > > > > > > -- --all
> > > > > > > >> > > > > > > > > > > > mkdir POCs/
> > > > > > > >> > > > > > > > > > > > git mv * POCs/
> > > > > > > >> > > > > > > > > > > > git commit -m "colleted the data to
> > move"
> > > > > > > >> > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > git clone <git
> > destination_repository_url>
> > > > cd
> > > > > > <git
> > > > > > > >> > > > > > > > > > > > destination_repository_directory> git
> > > remote
> > > > > add
> > > > > > > >> > > > > > > > > > origin_repository_branch
> > > > > > > >> > > > > > > > > > > > <git origin_repository directory> git
> > pull
> > > > > > > >> > > > > > > origin_repository_branch
> > > > > > > >> > > > > > > > > > > master
> > > > > > > >> > > > > > > > > > > > --allow-unrelated-histories git remote
> > rm
> > > > > > > >> > > > > > > origin_repository_branch
> > > > > > > >> > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > > Thanks in advance !
> > > > > > > >> > > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > >
> > > > > > > >> > > > >
> > > > > --------------------------------------------------------------
> > > > > > > >> > > > > > > > > > > > Zhang Yifei
> > > > > > > >> > > > > > > > > > >
> > > > > > > >> > > > > > > > > > >
> > > > > > > >> > > > > > > > > > >
> > > > > > > >> > > > > > > > > > > --
> > > > > > > >> > > > > > > > > > > Luciano Resende
> > > > > > > >> > > > > > > > > > > http://twitter.com/lresende1975
> > > > > > > >> > > > > > > > > > > http://lresende.blogspot.com/
> > > > > > > >> > > > > > > > > > >
> > > > > > > >> > > > > > > > > >
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > > > --
> > > > > > > >> > > > > > > > >
> > > > > > > >> >
> > --------------------------------------------------------------
> > > > > > > >> > > > > > > > > Zhang Yifei
> > > > > > > >> > > > > > > > >
> > > > > > > >> > > > > > > >
> > > > > > > >> > > > > > >
> > > > > > > >> > > > > >
> > > > > > > >> > > > >
> > > > > > > >> > > >
> > > > > > > >> > >
> > > > > > > >> >
> > > > > > > >>
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: POCs, Prototypes, experiments ...

Posted by Lucas Bonatto Miguel <lu...@apache.org>.

Makes perfect sense to me. I believe that if the majority of people agrees
with this architecture we could create issues and start implementing it.

+1


On Fri, Oct 26, 2018 at 4:09 PM Daniel Takabayashi <
daniel.takabayashi@gmail.com> wrote:

> I see... what about this new version?
>
>
> https://docs.google.com/drawings/d/shajxIpLJHxxMbFgDXiPuhg/image?w=602&h=468&rev=1630&ac=1&parent=1ySERHGBXbHeyCMRookq5UfTuFkzzU0ugtjvR3rF3deY
>
> I changed the toolbox ...from 1 component to 3... docker client, cli and
> repl. Both (cli and repl) now are using the docker client to control the
> engines.
>
> Em sex, 26 de out de 2018 às 14:49, Lucas Bonatto Miguel <
> lucasbm88@apache.org> escreveu:
>
> > For sure, both suggestions are to make sure we're not killing the CLI and
> > that we have a component (the SDK client) that is shared by CLI and REPL.
> >
> > Thanks for clarifying where the system calls will live.
> >
> > - Lucas
> >
> > On Fri, Oct 26, 2018 at 2:39 PM Daniel Takabayashi <
> > daniel.takabayashi@gmail.com> wrote:
> >
> > > Hi Lucas, could you please justified your two suggestions?
> > >
> > > The general ideia is to use the Docker DSK to execute local commands,
> to
> > > consume outputs and also to execute Marvin "specific language"
> commands.
> > > Today almost all CLIs commands in Marvin is a mixed of systems commands
> > and
> > > library method calls. The plan is to put all "specific language calls"
> > > inside the marvin-common-libs and group the systems call on the toolbox
> > > (using the docker SDK in this task).
> > >
> > >
> > >
> > >
> > > Em qui, 25 de out de 2018 às 20:09, Lucas Bonatto Miguel <
> > > lucasbm88@apache.org> escreveu:
> > >
> > > > Great, thanks.
> > > >
> > > > So here some suggestions:
> > > >  1. We could create one new layer between Toolbox and the Engines,
> this
> > > > would be Docker client.
> > > >  2. Split the Toolbox layer in two. Toolbox CLI and Toolbox REPL
> > > >
> > > > It's not clear to me if the commands listed on General Commands will
> > need
> > > > to be written for every language, from the image I believe not, but
> how
> > > it
> > > > would work, since starting a notebook in Python is different than
> > > starting
> > > > in Scala.
> > > >
> > > > - Lucas
> > > > On Thu, Oct 25, 2018 at 4:51 PM Daniel Takabayashi <
> > > > daniel.takabayashi@gmail.com> wrote:
> > > >
> > > > >
> > > > >
> > > >
> > >
> >
> https://docs.google.com/drawings/d/shajxIpLJHxxMbFgDXiPuhg/image?w=602&h=461&rev=1423&ac=1&parent=1ySERHGBXbHeyCMRookq5UfTuFkzzU0ugtjvR3rF3deY
> > > > >
> > > > > Em qui, 25 de out de 2018 às 07:38, Rafael Novello <
> > > > > rafa.reis.novello@gmail.com> escreveu:
> > > > >
> > > > > > Hi Taka,
> > > > > >
> > > > > > I can't see the image! Maybe the Apache email list don't support
> > > > > > attachments.
> > > > > >
> > > > > > Best!
> > > > > > Rafael J. R. Novello
> > > > > >
> > > > > > Skype: rafael.novello
> > > > > > Blog: http://rafanovello.blogspot.com.br/
> > > > > >
> > > > > >
> > > > > > Em qua, 24 de out de 2018 às 21:14, Daniel Takabayashi <
> > > > > > daniel.takabayashi@gmail.com> escreveu:
> > > > > >
> > > > > > > Hi all,
> > > > > > >
> > > > > > > To try to give a little more details about this "new concept"
> > that
> > > we
> > > > > > want
> > > > > > > to bring to the marvin toolbox, I did this simple architecture
> > > draw.
> > > > > > >
> > > > > > > [image: marvin-architecture-views (5).png]
> > > > > > >
> > > > > > > The general idea here is try to transform the toolbox something
> > > > > > > disconnected with the "language", something more agnostic. Also
> > in
> > > > this
> > > > > > new
> > > > > > > architecture we could use remote resource to process engines
> and
> > > make
> > > > > > easy
> > > > > > > the support for new languages.
> > > > > > >
> > > > > > > This "new toolbox" will be the only thing that a Marvin user
> must
> > > to
> > > > > > > install and also we could start to support multiples O.S (once
> > the
> > > > REPL
> > > > > > is
> > > > > > > a dummy application that only interprets and by pass commands).
> > > > > > >
> > > > > > > Regards,
> > > > > > > Taka
> > > > > > >
> > > > > > > Em qua, 24 de out de 2018 às 09:52, Rafael Novello <
> > > > > > > rafa.reis.novello@gmail.com> escreveu:
> > > > > > >
> > > > > > >> Alan,
> > > > > > >>
> > > > > > >> Yes! We are using the Docker SDK and it's possible to use the
> > API
> > > to
> > > > > > >> automate, but it's a little bit harder than automate CLI
> calls.
> > > > > > >>
> > > > > > >> Atenciosamente,
> > > > > > >> Rafael J. R. Novello
> > > > > > >>
> > > > > > >> Skype: rafael.novello
> > > > > > >> Blog: http://rafanovello.blogspot.com.br/
> > > > > > >>
> > > > > > >>
> > > > > > >> Em qua, 24 de out de 2018 às 12:02, Alan Silva <
> > jumpi@apache.org>
> > > > > > >> escreveu:
> > > > > > >>
> > > > > > >> > Hi,
> > > > > > >> >
> > > > > > >> > One question here, I understand that we start to use with
> this
> > > PoC
> > > > > the
> > > > > > >> > Docker SDK API, right?
> > > > > > >> >
> > > > > > >> > Why not use the API to expose some endpoints to permit this
> > kind
> > > > of
> > > > > > >> > automation by devops?
> > > > > > >> >
> > > > > > >> > I think it is possible and it solves the CLI problem, right?
> > > > > > >> >
> > > > > > >> >
> > > > > > >> > On Tue, Oct 23, 2018 at 1:05 PM Rafael Novello <
> > > > > > >> > rafa.reis.novello@gmail.com>
> > > > > > >> > wrote:
> > > > > > >> >
> > > > > > >> > > Lucas,
> > > > > > >> > >
> > > > > > >> > > Sorry, I didn't understood your question bellow.
> > > > > > >> > >
> > > > > > >> > > "Would it make sense to use the same solution that we will
> > use
> > > > for
> > > > > > >> > having a
> > > > > > >> > > single-language REPL to have a single-language CLI?"
> > > > > > >> > >
> > > > > > >> > > For DevOps purposes, maybe this new toolbox concept is not
> > > > ideal.
> > > > > I
> > > > > > >> think
> > > > > > >> > > we can keep the CLI inside the docker container but it
> will
> > > not
> > > > > easy
> > > > > > >> to
> > > > > > >> > > automate
> > > > > > >> > > jobs by this way.
> > > > > > >> > >
> > > > > > >> > > How to deal with this issue? Voting?
> > > > > > >> > > Atenciosamente,
> > > > > > >> > > Rafael J. R. Novello
> > > > > > >> > >
> > > > > > >> > > Skype: rafael.novello
> > > > > > >> > > Blog: http://rafanovello.blogspot.com.br/
> > > > > > >> > >
> > > > > > >> > >
> > > > > > >> > > Em sex, 19 de out de 2018 às 19:00, Lucas Bonatto Miguel <
> > > > > > >> > > lucasbm88@apache.org> escreveu:
> > > > > > >> > >
> > > > > > >> > > > Got it! Thanks for clarifying.
> > > > > > >> > > >
> > > > > > >> > > > Would it make sense to use the same solution that we
> will
> > > use
> > > > > for
> > > > > > >> > having
> > > > > > >> > > a
> > > > > > >> > > > single-language REPL to have a single-language CLI? My
> > only
> > > > > > concern
> > > > > > >> > with
> > > > > > >> > > > killing the CLI is that you remove an essential feature
> > for
> > > > > > DevOps.
> > > > > > >> > > >
> > > > > > >> > > > - Lucas
> > > > > > >> > > >
> > > > > > >> > > > On Fri, Oct 19, 2018 at 1:52 PM Rafael Novello <
> > > > > > >> > > > rafa.reis.novello@gmail.com>
> > > > > > >> > > > wrote:
> > > > > > >> > > >
> > > > > > >> > > > > Lucas,
> > > > > > >> > > > >
> > > > > > >> > > > > The idea is that REPL will substitute the actual CLI.
> > It's
> > > > > > because
> > > > > > >> > with
> > > > > > >> > > > the
> > > > > > >> > > > > actual concept (using language specific CLI) we will
> > need
> > > to
> > > > > > >> > > re-implement
> > > > > > >> > > > > the same features many times and probably each CLI
> will
> > > > have a
> > > > > > >> > > different
> > > > > > >> > > > > behavior because some language specific restrictions
> > > and/or
> > > > > > >> > > limitations.
> > > > > > >> > > > >
> > > > > > >> > > > > With this new concept, all users will have the same
> > > > experience
> > > > > > >> > > > interacting
> > > > > > >> > > > > with Marvin REPL and they will use the bests tools to
> > > > develop
> > > > > > >> their
> > > > > > >> > > > engines
> > > > > > >> > > > > (Jupyter Notebook and/or Lab with different languages
> > > > support
> > > > > or
> > > > > > >> even
> > > > > > >> > > > > Apache Zeppelin). All the interact will occur through
> > REPL
> > > > and
> > > > > > >> docker
> > > > > > >> > > > > protocol.
> > > > > > >> > > > >
> > > > > > >> > > > > Alan, Yes! As Lucas said, the concept is the same but
> we
> > > can
> > > > > use
> > > > > > >> > docker
> > > > > > >> > > > to
> > > > > > >> > > > > do the same job.
> > > > > > >> > > > >
> > > > > > >> > > > > Best!
> > > > > > >> > > > >
> > > > > > >> > > > > Em sex, 19 de out de 2018 às 00:39, Lucas Bonatto
> > Miguel <
> > > > > > >> > > > > lucasbm88@apache.org> escreveu:
> > > > > > >> > > > >
> > > > > > >> > > > > > Thanks for the clarifications Rafael, so from what I
> > > > > > understood,
> > > > > > >> > > > Marvin's
> > > > > > >> > > > > > developers would not use the REPL to explore data
> and
> > > test
> > > > > > >> models,
> > > > > > >> > > i.e.
> > > > > > >> > > > > > develop the engine. Is the idea to build something
> > more
> > > > like
> > > > > > an
> > > > > > >> > > > > interactive
> > > > > > >> > > > > > CLI? I think an interactive CLI would be useful for
> > the
> > > > > > >> developer
> > > > > > >> > > > > > experience, however, it's important to keep the
> > > unattended
> > > > > > >> > (current)
> > > > > > >> > > > CLI
> > > > > > >> > > > > in
> > > > > > >> > > > > > place for scenarios where users want to automate
> > Marvin
> > > > > tasks.
> > > > > > >> > > > > >
> > > > > > >> > > > > > Alan, I believe the idea is still the same as you
> > > started,
> > > > > but
> > > > > > >> > using
> > > > > > >> > > > > > docker SDK now.
> > > > > > >> > > > > >
> > > > > > >> > > > > > - Lucas
> > > > > > >> > > > > >
> > > > > > >> > > > > > On Thu, Oct 18, 2018 at 1:29 PM Rafael Novello <
> > > > > > >> > > > > > rafa.reis.novello@gmail.com>
> > > > > > >> > > > > > wrote:
> > > > > > >> > > > > >
> > > > > > >> > > > > > > Hi Lucas!
> > > > > > >> > > > > > >
> > > > > > >> > > > > > > First of all +1 for REPL POCs to Apache!
> > > > > > >> > > > > > >
> > > > > > >> > > > > > > Let me help with some comments:
> > > > > > >> > > > > > >
> > > > > > >> > > > > > > 1 - We have tested NodeJS, Scala and Python and
> the
> > > > > easiest
> > > > > > >> one
> > > > > > >> > was
> > > > > > >> > > > > > Python.
> > > > > > >> > > > > > > We have found a small project [1] that have all
> > > features
> > > > > we
> > > > > > >> > desired
> > > > > > >> > > > for
> > > > > > >> > > > > > > REPL:
> > > > > > >> > > > > > > - Autocomplete
> > > > > > >> > > > > > > - Python commands disabled (the user have only the
> > > > > commands
> > > > > > we
> > > > > > >> > > > > provide).
> > > > > > >> > > > > > > The other languages REPL options don't have this
> > > > feature.
> > > > > > >> > > > > > > - Easy to show output
> > > > > > >> > > > > > > - Etc
> > > > > > >> > > > > > > So, I think the language chosen will not be
> > important
> > > > here
> > > > > > >> > because
> > > > > > >> > > > the
> > > > > > >> > > > > > user
> > > > > > >> > > > > > > will only interact with the commands that we
> create.
> > > > > > >> > > > > > >
> > > > > > >> > > > > > > 2 - The "engine-generate" command will download a
> > > docker
> > > > > > image
> > > > > > >> > that
> > > > > > >> > > > we
> > > > > > >> > > > > > > create for that language and start a container
> with
> > > the
> > > > > > basic
> > > > > > >> > > project
> > > > > > >> > > > > > > structure for that language.
> > > > > > >> > > > > > >
> > > > > > >> > > > > > > 3 - The REPL client will use the "docker protocol"
> > to
> > > > run
> > > > > > >> > command,
> > > > > > >> > > > > > > start/stop services and etc inside the container
> and
> > > it
> > > > > will
> > > > > > >> > > receive
> > > > > > >> > > > > the
> > > > > > >> > > > > > > log stream to show. No, the REPL will no pass code
> > > > > snippets
> > > > > > >> for
> > > > > > >> > > > docker
> > > > > > >> > > > > > > container (I think it will not be necessary)
> > > > > > >> > > > > > >
> > > > > > >> > > > > > > 4 - Yep! Like I said on the first item.
> > > > > > >> > > > > > >
> > > > > > >> > > > > > > [1] - https://github.com/italorossi/ishell
> > > > > > >> > > > > > >
> > > > > > >> > > > > > > Let me know if there is any other question!
> > > > > > >> > > > > > > Best!
> > > > > > >> > > > > > > Rafael J. R. Novello
> > > > > > >> > > > > > >
> > > > > > >> > > > > > > Skype: rafael.novello
> > > > > > >> > > > > > > Blog: http://rafanovello.blogspot.com.br/
> > > > > > >> > > > > > >
> > > > > > >> > > > > > >
> > > > > > >> > > > > > > Em qui, 18 de out de 2018 às 17:00, Lucas Bonatto
> > > > Miguel <
> > > > > > >> > > > > > > lucasbm88@apache.org> escreveu:
> > > > > > >> > > > > > >
> > > > > > >> > > > > > > > + 1 for migrating the REPL repo to Apache
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > > > I have a few questions about the previous
> > > explanation:
> > > > > > >> > > > > > > >  1) The REPL itself, it would be an application
> in
> > > > which
> > > > > > >> > > language?
> > > > > > >> > > > > > > Remember
> > > > > > >> > > > > > > > that the main idea is to allow the user to
> program
> > > on
> > > > > his
> > > > > > >> > > preferred
> > > > > > >> > > > > > > > language in the REPL.
> > > > > > >> > > > > > > >  2) Should the engine-generate command also
> > > generate a
> > > > > > >> docker
> > > > > > >> > > image
> > > > > > >> > > > > > with
> > > > > > >> > > > > > > > the user's application?
> > > > > > >> > > > > > > >  3) What type of communication would happen
> > between
> > > > the
> > > > > > REPL
> > > > > > >> > and
> > > > > > >> > > > the
> > > > > > >> > > > > > > engine
> > > > > > >> > > > > > > > via Docker SDK? Would the REPL pass snippets of
> > code
> > > > to
> > > > > be
> > > > > > >> > > executed
> > > > > > >> > > > > by
> > > > > > >> > > > > > > the
> > > > > > >> > > > > > > > docker container?
> > > > > > >> > > > > > > >  4) Have you considered code completion in the
> > REPL?
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > > > On Thu, Oct 18, 2018 at 7:53 AM Zhang Yifei <
> > > > > > >> > > > yifei.z.luiz@gmail.com>
> > > > > > >> > > > > > > > wrote:
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > > > > Ok guys,
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > > The basic idea is to provide only one Toolbox
> > for
> > > > > > multiple
> > > > > > >> > > > > languages.
> > > > > > >> > > > > > > > > We are looking for possibility to build a
> single
> > > > > Marvin
> > > > > > >> Repl,
> > > > > > >> > > > > > > > > instead of severals toolboxes with differentes
> > > > > > interfaces
> > > > > > >> or
> > > > > > >> > > > > > commands.
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > > In this case, the engine-generate command will
> > > > > download
> > > > > > >> and
> > > > > > >> > > > start a
> > > > > > >> > > > > > > > Docker
> > > > > > >> > > > > > > > > container with basic engine structure
> > > corresponding
> > > > > the
> > > > > > >> > choosed
> > > > > > >> > > > > > > language.
> > > > > > >> > > > > > > > > this means we don't need to build Toolboxes of
> > > > > > differents
> > > > > > >> > > > > languages,
> > > > > > >> > > > > > we
> > > > > > >> > > > > > > > > just
> > > > > > >> > > > > > > > > build the engine template of all languages
> that
> > we
> > > > > want
> > > > > > to
> > > > > > >> > > > support
> > > > > > >> > > > > > and
> > > > > > >> > > > > > > > > provide it
> > > > > > >> > > > > > > > > as Docker containers
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > > We have started researches around our basic
> > > > > requirements
> > > > > > >> > like:
> > > > > > >> > > > > > > > > - Repl interface
> > > > > > >> > > > > > > > > - System communication
> > > > > > >> > > > > > > > > - Connection security
> > > > > > >> > > > > > > > > - Tool popularity
> > > > > > >> > > > > > > > > - Update complexity
> > > > > > >> > > > > > > > > - Languages support
> > > > > > >> > > > > > > > > - ......
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > > And we did some POC with code here:
> > > > > > >> > > > > > > > > https://github.com/marvin-ai/marvin-repl
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > > There is POC testing gRPC using Scala and
> > Python,
> > > > > > >> > > > > > > > > Repl inteface and Docker SDK with NodeJS,
> > > > > > >> > > > > > > > > Repl interface and Docker SDK with Python.
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > > At this moment we prefer the Repl interface +
> > > Docker
> > > > > SDK
> > > > > > >> way,
> > > > > > >> > > > > because
> > > > > > >> > > > > > > > good
> > > > > > >> > > > > > > > > part of the requirements
> > > > > > >> > > > > > > > > will be guaranteed by Docker.
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > > With this informations, what do you think?
> > Should
> > > we
> > > > > > >> submit
> > > > > > >> > all
> > > > > > >> > > > > this
> > > > > > >> > > > > > > POCs
> > > > > > >> > > > > > > > > to Apache Repo?
> > > > > > >> > > > > > > > > Please feel free to opine.
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > > Thats all, thanks!!!
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > > Em ter, 16 de out de 2018 às 18:55, Daniel
> > > > > Takabayashi <
> > > > > > >> > > > > > > > > daniel.takabayashi@gmail.com> escreveu:
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > > > Zhang,
> > > > > > >> > > > > > > > > >
> > > > > > >> > > > > > > > > > I think the best approach is give us a
> better
> > > > > > >> explanation
> > > > > > >> > > about
> > > > > > >> > > > > > this
> > > > > > >> > > > > > > > new
> > > > > > >> > > > > > > > > > feature and how this can help us to archive
> > the
> > > > > > desired
> > > > > > >> > > feature
> > > > > > >> > > > > > > > (support
> > > > > > >> > > > > > > > > > multiple languages). Than if the majority
> > agree
> > > > > than I
> > > > > > >> can
> > > > > > >> > do
> > > > > > >> > > > the
> > > > > > >> > > > > > > > merge,
> > > > > > >> > > > > > > > > > just like I did before, but in another
> branch.
> > > > > > >> > > > > > > > > >
> > > > > > >> > > > > > > > > > Let me know if makes sense, ok?
> > > > > > >> > > > > > > > > >
> > > > > > >> > > > > > > > > > Taka
> > > > > > >> > > > > > > > > >
> > > > > > >> > > > > > > > > > Em ter, 16 de out de 2018 às 14:00, Luciano
> > > > Resende
> > > > > <
> > > > > > >> > > > > > > > > luckbr1975@gmail.com>
> > > > > > >> > > > > > > > > > escreveu:
> > > > > > >> > > > > > > > > >
> > > > > > >> > > > > > > > > > > So, what is the POC, is it a refactoring
> of
> > > the
> > > > > > >> existing
> > > > > > >> > > > repo?
> > > > > > >> > > > > Or
> > > > > > >> > > > > > > is
> > > > > > >> > > > > > > > > > > it a new rewrite of the repo?
> > > > > > >> > > > > > > > > > >
> > > > > > >> > > > > > > > > > > Just asking as it might make sense to make
> > it
> > > a
> > > > > > branch
> > > > > > >> > then
> > > > > > >> > > > > > > actually
> > > > > > >> > > > > > > > a
> > > > > > >> > > > > > > > > > > thing in parallel, as this will have an
> > effect
> > > > for
> > > > > > >> > > releases,
> > > > > > >> > > > > etc.
> > > > > > >> > > > > > > but
> > > > > > >> > > > > > > > > > > you guys know more here than I do.
> > > > > > >> > > > > > > > > > >
> > > > > > >> > > > > > > > > > > Also, it's probably good to have a write
> up
> > of
> > > > the
> > > > > > >> main
> > > > > > >> > > > > direction
> > > > > > >> > > > > > > of
> > > > > > >> > > > > > > > > > > the design that can help people get
> familiar
> > > > with
> > > > > > the
> > > > > > >> new
> > > > > > >> > > > > > approach.
> > > > > > >> > > > > > > > > > > On Tue, Oct 16, 2018 at 11:12 AM Zhang
> > Yifei <
> > > > > > >> > > > > > > yifei.z.luiz@gmail.com
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > > > > wrote:
> > > > > > >> > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > Hey guys,
> > > > > > >> > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > We have reorganized the Poc repo, and
> want
> > > to
> > > > > > merge
> > > > > > >> it
> > > > > > >> > to
> > > > > > >> > > > our
> > > > > > >> > > > > > > > Apache
> > > > > > >> > > > > > > > > > > repo.
> > > > > > >> > > > > > > > > > > > Just making sure here before we do the
> > > merge,
> > > > > > >> because
> > > > > > >> > im
> > > > > > >> > > > not
> > > > > > >> > > > > > sure
> > > > > > >> > > > > > > > if
> > > > > > >> > > > > > > > > > > doing
> > > > > > >> > > > > > > > > > > > this will perserve the Git commit
> history.
> > > > > > >> > > > > > > > > > > > What we are planning to do is:
> > > > > > >> > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > git filter-branch --subdirectory-filter
> > <git
> > > > > > >> > > > > origin_repository
> > > > > > >> > > > > > > > > > directory>
> > > > > > >> > > > > > > > > > > > -- --all
> > > > > > >> > > > > > > > > > > > mkdir POCs/
> > > > > > >> > > > > > > > > > > > git mv * POCs/
> > > > > > >> > > > > > > > > > > > git commit -m "colleted the data to
> move"
> > > > > > >> > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > git clone <git
> destination_repository_url>
> > > cd
> > > > > <git
> > > > > > >> > > > > > > > > > > > destination_repository_directory> git
> > remote
> > > > add
> > > > > > >> > > > > > > > > > origin_repository_branch
> > > > > > >> > > > > > > > > > > > <git origin_repository directory> git
> pull
> > > > > > >> > > > > > > origin_repository_branch
> > > > > > >> > > > > > > > > > > master
> > > > > > >> > > > > > > > > > > > --allow-unrelated-histories git remote
> rm
> > > > > > >> > > > > > > origin_repository_branch
> > > > > > >> > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > > Thanks in advance !
> > > > > > >> > > > > > > > > > > >
> > > > > > >> > > > > > > > > > > >
> > > > > > >> > > > >
> > > > --------------------------------------------------------------
> > > > > > >> > > > > > > > > > > > Zhang Yifei
> > > > > > >> > > > > > > > > > >
> > > > > > >> > > > > > > > > > >
> > > > > > >> > > > > > > > > > >
> > > > > > >> > > > > > > > > > > --
> > > > > > >> > > > > > > > > > > Luciano Resende
> > > > > > >> > > > > > > > > > > http://twitter.com/lresende1975
> > > > > > >> > > > > > > > > > > http://lresende.blogspot.com/
> > > > > > >> > > > > > > > > > >
> > > > > > >> > > > > > > > > >
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > > > --
> > > > > > >> > > > > > > > >
> > > > > > >> >
> --------------------------------------------------------------
> > > > > > >> > > > > > > > > Zhang Yifei
> > > > > > >> > > > > > > > >
> > > > > > >> > > > > > > >
> > > > > > >> > > > > > >
> > > > > > >> > > > > >
> > > > > > >> > > > >
> > > > > > >> > > >
> > > > > > >> > >
> > > > > > >> >
> > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: POCs, Prototypes, experiments ...

Posted by Daniel Takabayashi <da...@gmail.com>.

I see... what about this new version?

https://docs.google.com/drawings/d/shajxIpLJHxxMbFgDXiPuhg/image?w=602&h=468&rev=1630&ac=1&parent=1ySERHGBXbHeyCMRookq5UfTuFkzzU0ugtjvR3rF3deY

I changed the toolbox ...from 1 component to 3... docker client, cli and
repl. Both (cli and repl) now are using the docker client to control the
engines.

Em sex, 26 de out de 2018 às 14:49, Lucas Bonatto Miguel <
lucasbm88@apache.org> escreveu:

> For sure, both suggestions are to make sure we're not killing the CLI and
> that we have a component (the SDK client) that is shared by CLI and REPL.
>
> Thanks for clarifying where the system calls will live.
>
> - Lucas
>
> On Fri, Oct 26, 2018 at 2:39 PM Daniel Takabayashi <
> daniel.takabayashi@gmail.com> wrote:
>
> > Hi Lucas, could you please justified your two suggestions?
> >
> > The general ideia is to use the Docker DSK to execute local commands, to
> > consume outputs and also to execute Marvin "specific language" commands.
> > Today almost all CLIs commands in Marvin is a mixed of systems commands
> and
> > library method calls. The plan is to put all "specific language calls"
> > inside the marvin-common-libs and group the systems call on the toolbox
> > (using the docker SDK in this task).
> >
> >
> >
> >
> > Em qui, 25 de out de 2018 às 20:09, Lucas Bonatto Miguel <
> > lucasbm88@apache.org> escreveu:
> >
> > > Great, thanks.
> > >
> > > So here some suggestions:
> > >  1. We could create one new layer between Toolbox and the Engines, this
> > > would be Docker client.
> > >  2. Split the Toolbox layer in two. Toolbox CLI and Toolbox REPL
> > >
> > > It's not clear to me if the commands listed on General Commands will
> need
> > > to be written for every language, from the image I believe not, but how
> > it
> > > would work, since starting a notebook in Python is different than
> > starting
> > > in Scala.
> > >
> > > - Lucas
> > > On Thu, Oct 25, 2018 at 4:51 PM Daniel Takabayashi <
> > > daniel.takabayashi@gmail.com> wrote:
> > >
> > > >
> > > >
> > >
> >
> https://docs.google.com/drawings/d/shajxIpLJHxxMbFgDXiPuhg/image?w=602&h=461&rev=1423&ac=1&parent=1ySERHGBXbHeyCMRookq5UfTuFkzzU0ugtjvR3rF3deY
> > > >
> > > > Em qui, 25 de out de 2018 às 07:38, Rafael Novello <
> > > > rafa.reis.novello@gmail.com> escreveu:
> > > >
> > > > > Hi Taka,
> > > > >
> > > > > I can't see the image! Maybe the Apache email list don't support
> > > > > attachments.
> > > > >
> > > > > Best!
> > > > > Rafael J. R. Novello
> > > > >
> > > > > Skype: rafael.novello
> > > > > Blog: http://rafanovello.blogspot.com.br/
> > > > >
> > > > >
> > > > > Em qua, 24 de out de 2018 às 21:14, Daniel Takabayashi <
> > > > > daniel.takabayashi@gmail.com> escreveu:
> > > > >
> > > > > > Hi all,
> > > > > >
> > > > > > To try to give a little more details about this "new concept"
> that
> > we
> > > > > want
> > > > > > to bring to the marvin toolbox, I did this simple architecture
> > draw.
> > > > > >
> > > > > > [image: marvin-architecture-views (5).png]
> > > > > >
> > > > > > The general idea here is try to transform the toolbox something
> > > > > > disconnected with the "language", something more agnostic. Also
> in
> > > this
> > > > > new
> > > > > > architecture we could use remote resource to process engines and
> > make
> > > > > easy
> > > > > > the support for new languages.
> > > > > >
> > > > > > This "new toolbox" will be the only thing that a Marvin user must
> > to
> > > > > > install and also we could start to support multiples O.S (once
> the
> > > REPL
> > > > > is
> > > > > > a dummy application that only interprets and by pass commands).
> > > > > >
> > > > > > Regards,
> > > > > > Taka
> > > > > >
> > > > > > Em qua, 24 de out de 2018 às 09:52, Rafael Novello <
> > > > > > rafa.reis.novello@gmail.com> escreveu:
> > > > > >
> > > > > >> Alan,
> > > > > >>
> > > > > >> Yes! We are using the Docker SDK and it's possible to use the
> API
> > to
> > > > > >> automate, but it's a little bit harder than automate CLI calls.
> > > > > >>
> > > > > >> Atenciosamente,
> > > > > >> Rafael J. R. Novello
> > > > > >>
> > > > > >> Skype: rafael.novello
> > > > > >> Blog: http://rafanovello.blogspot.com.br/
> > > > > >>
> > > > > >>
> > > > > >> Em qua, 24 de out de 2018 às 12:02, Alan Silva <
> jumpi@apache.org>
> > > > > >> escreveu:
> > > > > >>
> > > > > >> > Hi,
> > > > > >> >
> > > > > >> > One question here, I understand that we start to use with this
> > PoC
> > > > the
> > > > > >> > Docker SDK API, right?
> > > > > >> >
> > > > > >> > Why not use the API to expose some endpoints to permit this
> kind
> > > of
> > > > > >> > automation by devops?
> > > > > >> >
> > > > > >> > I think it is possible and it solves the CLI problem, right?
> > > > > >> >
> > > > > >> >
> > > > > >> > On Tue, Oct 23, 2018 at 1:05 PM Rafael Novello <
> > > > > >> > rafa.reis.novello@gmail.com>
> > > > > >> > wrote:
> > > > > >> >
> > > > > >> > > Lucas,
> > > > > >> > >
> > > > > >> > > Sorry, I didn't understood your question bellow.
> > > > > >> > >
> > > > > >> > > "Would it make sense to use the same solution that we will
> use
> > > for
> > > > > >> > having a
> > > > > >> > > single-language REPL to have a single-language CLI?"
> > > > > >> > >
> > > > > >> > > For DevOps purposes, maybe this new toolbox concept is not
> > > ideal.
> > > > I
> > > > > >> think
> > > > > >> > > we can keep the CLI inside the docker container but it will
> > not
> > > > easy
> > > > > >> to
> > > > > >> > > automate
> > > > > >> > > jobs by this way.
> > > > > >> > >
> > > > > >> > > How to deal with this issue? Voting?
> > > > > >> > > Atenciosamente,
> > > > > >> > > Rafael J. R. Novello
> > > > > >> > >
> > > > > >> > > Skype: rafael.novello
> > > > > >> > > Blog: http://rafanovello.blogspot.com.br/
> > > > > >> > >
> > > > > >> > >
> > > > > >> > > Em sex, 19 de out de 2018 às 19:00, Lucas Bonatto Miguel <
> > > > > >> > > lucasbm88@apache.org> escreveu:
> > > > > >> > >
> > > > > >> > > > Got it! Thanks for clarifying.
> > > > > >> > > >
> > > > > >> > > > Would it make sense to use the same solution that we will
> > use
> > > > for
> > > > > >> > having
> > > > > >> > > a
> > > > > >> > > > single-language REPL to have a single-language CLI? My
> only
> > > > > concern
> > > > > >> > with
> > > > > >> > > > killing the CLI is that you remove an essential feature
> for
> > > > > DevOps.
> > > > > >> > > >
> > > > > >> > > > - Lucas
> > > > > >> > > >
> > > > > >> > > > On Fri, Oct 19, 2018 at 1:52 PM Rafael Novello <
> > > > > >> > > > rafa.reis.novello@gmail.com>
> > > > > >> > > > wrote:
> > > > > >> > > >
> > > > > >> > > > > Lucas,
> > > > > >> > > > >
> > > > > >> > > > > The idea is that REPL will substitute the actual CLI.
> It's
> > > > > because
> > > > > >> > with
> > > > > >> > > > the
> > > > > >> > > > > actual concept (using language specific CLI) we will
> need
> > to
> > > > > >> > > re-implement
> > > > > >> > > > > the same features many times and probably each CLI will
> > > have a
> > > > > >> > > different
> > > > > >> > > > > behavior because some language specific restrictions
> > and/or
> > > > > >> > > limitations.
> > > > > >> > > > >
> > > > > >> > > > > With this new concept, all users will have the same
> > > experience
> > > > > >> > > > interacting
> > > > > >> > > > > with Marvin REPL and they will use the bests tools to
> > > develop
> > > > > >> their
> > > > > >> > > > engines
> > > > > >> > > > > (Jupyter Notebook and/or Lab with different languages
> > > support
> > > > or
> > > > > >> even
> > > > > >> > > > > Apache Zeppelin). All the interact will occur through
> REPL
> > > and
> > > > > >> docker
> > > > > >> > > > > protocol.
> > > > > >> > > > >
> > > > > >> > > > > Alan, Yes! As Lucas said, the concept is the same but we
> > can
> > > > use
> > > > > >> > docker
> > > > > >> > > > to
> > > > > >> > > > > do the same job.
> > > > > >> > > > >
> > > > > >> > > > > Best!
> > > > > >> > > > >
> > > > > >> > > > > Em sex, 19 de out de 2018 às 00:39, Lucas Bonatto
> Miguel <
> > > > > >> > > > > lucasbm88@apache.org> escreveu:
> > > > > >> > > > >
> > > > > >> > > > > > Thanks for the clarifications Rafael, so from what I
> > > > > understood,
> > > > > >> > > > Marvin's
> > > > > >> > > > > > developers would not use the REPL to explore data and
> > test
> > > > > >> models,
> > > > > >> > > i.e.
> > > > > >> > > > > > develop the engine. Is the idea to build something
> more
> > > like
> > > > > an
> > > > > >> > > > > interactive
> > > > > >> > > > > > CLI? I think an interactive CLI would be useful for
> the
> > > > > >> developer
> > > > > >> > > > > > experience, however, it's important to keep the
> > unattended
> > > > > >> > (current)
> > > > > >> > > > CLI
> > > > > >> > > > > in
> > > > > >> > > > > > place for scenarios where users want to automate
> Marvin
> > > > tasks.
> > > > > >> > > > > >
> > > > > >> > > > > > Alan, I believe the idea is still the same as you
> > started,
> > > > but
> > > > > >> > using
> > > > > >> > > > > > docker SDK now.
> > > > > >> > > > > >
> > > > > >> > > > > > - Lucas
> > > > > >> > > > > >
> > > > > >> > > > > > On Thu, Oct 18, 2018 at 1:29 PM Rafael Novello <
> > > > > >> > > > > > rafa.reis.novello@gmail.com>
> > > > > >> > > > > > wrote:
> > > > > >> > > > > >
> > > > > >> > > > > > > Hi Lucas!
> > > > > >> > > > > > >
> > > > > >> > > > > > > First of all +1 for REPL POCs to Apache!
> > > > > >> > > > > > >
> > > > > >> > > > > > > Let me help with some comments:
> > > > > >> > > > > > >
> > > > > >> > > > > > > 1 - We have tested NodeJS, Scala and Python and the
> > > > easiest
> > > > > >> one
> > > > > >> > was
> > > > > >> > > > > > Python.
> > > > > >> > > > > > > We have found a small project [1] that have all
> > features
> > > > we
> > > > > >> > desired
> > > > > >> > > > for
> > > > > >> > > > > > > REPL:
> > > > > >> > > > > > > - Autocomplete
> > > > > >> > > > > > > - Python commands disabled (the user have only the
> > > > commands
> > > > > we
> > > > > >> > > > > provide).
> > > > > >> > > > > > > The other languages REPL options don't have this
> > > feature.
> > > > > >> > > > > > > - Easy to show output
> > > > > >> > > > > > > - Etc
> > > > > >> > > > > > > So, I think the language chosen will not be
> important
> > > here
> > > > > >> > because
> > > > > >> > > > the
> > > > > >> > > > > > user
> > > > > >> > > > > > > will only interact with the commands that we create.
> > > > > >> > > > > > >
> > > > > >> > > > > > > 2 - The "engine-generate" command will download a
> > docker
> > > > > image
> > > > > >> > that
> > > > > >> > > > we
> > > > > >> > > > > > > create for that language and start a container with
> > the
> > > > > basic
> > > > > >> > > project
> > > > > >> > > > > > > structure for that language.
> > > > > >> > > > > > >
> > > > > >> > > > > > > 3 - The REPL client will use the "docker protocol"
> to
> > > run
> > > > > >> > command,
> > > > > >> > > > > > > start/stop services and etc inside the container and
> > it
> > > > will
> > > > > >> > > receive
> > > > > >> > > > > the
> > > > > >> > > > > > > log stream to show. No, the REPL will no pass code
> > > > snippets
> > > > > >> for
> > > > > >> > > > docker
> > > > > >> > > > > > > container (I think it will not be necessary)
> > > > > >> > > > > > >
> > > > > >> > > > > > > 4 - Yep! Like I said on the first item.
> > > > > >> > > > > > >
> > > > > >> > > > > > > [1] - https://github.com/italorossi/ishell
> > > > > >> > > > > > >
> > > > > >> > > > > > > Let me know if there is any other question!
> > > > > >> > > > > > > Best!
> > > > > >> > > > > > > Rafael J. R. Novello
> > > > > >> > > > > > >
> > > > > >> > > > > > > Skype: rafael.novello
> > > > > >> > > > > > > Blog: http://rafanovello.blogspot.com.br/
> > > > > >> > > > > > >
> > > > > >> > > > > > >
> > > > > >> > > > > > > Em qui, 18 de out de 2018 às 17:00, Lucas Bonatto
> > > Miguel <
> > > > > >> > > > > > > lucasbm88@apache.org> escreveu:
> > > > > >> > > > > > >
> > > > > >> > > > > > > > + 1 for migrating the REPL repo to Apache
> > > > > >> > > > > > > >
> > > > > >> > > > > > > > I have a few questions about the previous
> > explanation:
> > > > > >> > > > > > > >  1) The REPL itself, it would be an application in
> > > which
> > > > > >> > > language?
> > > > > >> > > > > > > Remember
> > > > > >> > > > > > > > that the main idea is to allow the user to program
> > on
> > > > his
> > > > > >> > > preferred
> > > > > >> > > > > > > > language in the REPL.
> > > > > >> > > > > > > >  2) Should the engine-generate command also
> > generate a
> > > > > >> docker
> > > > > >> > > image
> > > > > >> > > > > > with
> > > > > >> > > > > > > > the user's application?
> > > > > >> > > > > > > >  3) What type of communication would happen
> between
> > > the
> > > > > REPL
> > > > > >> > and
> > > > > >> > > > the
> > > > > >> > > > > > > engine
> > > > > >> > > > > > > > via Docker SDK? Would the REPL pass snippets of
> code
> > > to
> > > > be
> > > > > >> > > executed
> > > > > >> > > > > by
> > > > > >> > > > > > > the
> > > > > >> > > > > > > > docker container?
> > > > > >> > > > > > > >  4) Have you considered code completion in the
> REPL?
> > > > > >> > > > > > > >
> > > > > >> > > > > > > >
> > > > > >> > > > > > > > On Thu, Oct 18, 2018 at 7:53 AM Zhang Yifei <
> > > > > >> > > > yifei.z.luiz@gmail.com>
> > > > > >> > > > > > > > wrote:
> > > > > >> > > > > > > >
> > > > > >> > > > > > > > > Ok guys,
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > > The basic idea is to provide only one Toolbox
> for
> > > > > multiple
> > > > > >> > > > > languages.
> > > > > >> > > > > > > > > We are looking for possibility to build a single
> > > > Marvin
> > > > > >> Repl,
> > > > > >> > > > > > > > > instead of severals toolboxes with differentes
> > > > > interfaces
> > > > > >> or
> > > > > >> > > > > > commands.
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > > In this case, the engine-generate command will
> > > > download
> > > > > >> and
> > > > > >> > > > start a
> > > > > >> > > > > > > > Docker
> > > > > >> > > > > > > > > container with basic engine structure
> > corresponding
> > > > the
> > > > > >> > choosed
> > > > > >> > > > > > > language.
> > > > > >> > > > > > > > > this means we don't need to build Toolboxes of
> > > > > differents
> > > > > >> > > > > languages,
> > > > > >> > > > > > we
> > > > > >> > > > > > > > > just
> > > > > >> > > > > > > > > build the engine template of all languages that
> we
> > > > want
> > > > > to
> > > > > >> > > > support
> > > > > >> > > > > > and
> > > > > >> > > > > > > > > provide it
> > > > > >> > > > > > > > > as Docker containers
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > > We have started researches around our basic
> > > > requirements
> > > > > >> > like:
> > > > > >> > > > > > > > > - Repl interface
> > > > > >> > > > > > > > > - System communication
> > > > > >> > > > > > > > > - Connection security
> > > > > >> > > > > > > > > - Tool popularity
> > > > > >> > > > > > > > > - Update complexity
> > > > > >> > > > > > > > > - Languages support
> > > > > >> > > > > > > > > - ......
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > > And we did some POC with code here:
> > > > > >> > > > > > > > > https://github.com/marvin-ai/marvin-repl
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > > There is POC testing gRPC using Scala and
> Python,
> > > > > >> > > > > > > > > Repl inteface and Docker SDK with NodeJS,
> > > > > >> > > > > > > > > Repl interface and Docker SDK with Python.
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > > At this moment we prefer the Repl interface +
> > Docker
> > > > SDK
> > > > > >> way,
> > > > > >> > > > > because
> > > > > >> > > > > > > > good
> > > > > >> > > > > > > > > part of the requirements
> > > > > >> > > > > > > > > will be guaranteed by Docker.
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > > With this informations, what do you think?
> Should
> > we
> > > > > >> submit
> > > > > >> > all
> > > > > >> > > > > this
> > > > > >> > > > > > > POCs
> > > > > >> > > > > > > > > to Apache Repo?
> > > > > >> > > > > > > > > Please feel free to opine.
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > > Thats all, thanks!!!
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > > Em ter, 16 de out de 2018 às 18:55, Daniel
> > > > Takabayashi <
> > > > > >> > > > > > > > > daniel.takabayashi@gmail.com> escreveu:
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > > > Zhang,
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > > > I think the best approach is give us a better
> > > > > >> explanation
> > > > > >> > > about
> > > > > >> > > > > > this
> > > > > >> > > > > > > > new
> > > > > >> > > > > > > > > > feature and how this can help us to archive
> the
> > > > > desired
> > > > > >> > > feature
> > > > > >> > > > > > > > (support
> > > > > >> > > > > > > > > > multiple languages). Than if the majority
> agree
> > > > than I
> > > > > >> can
> > > > > >> > do
> > > > > >> > > > the
> > > > > >> > > > > > > > merge,
> > > > > >> > > > > > > > > > just like I did before, but in another branch.
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > > > Let me know if makes sense, ok?
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > > > Taka
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > > > Em ter, 16 de out de 2018 às 14:00, Luciano
> > > Resende
> > > > <
> > > > > >> > > > > > > > > luckbr1975@gmail.com>
> > > > > >> > > > > > > > > > escreveu:
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > > > > So, what is the POC, is it a refactoring of
> > the
> > > > > >> existing
> > > > > >> > > > repo?
> > > > > >> > > > > Or
> > > > > >> > > > > > > is
> > > > > >> > > > > > > > > > > it a new rewrite of the repo?
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > > > Just asking as it might make sense to make
> it
> > a
> > > > > branch
> > > > > >> > then
> > > > > >> > > > > > > actually
> > > > > >> > > > > > > > a
> > > > > >> > > > > > > > > > > thing in parallel, as this will have an
> effect
> > > for
> > > > > >> > > releases,
> > > > > >> > > > > etc.
> > > > > >> > > > > > > but
> > > > > >> > > > > > > > > > > you guys know more here than I do.
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > > > Also, it's probably good to have a write up
> of
> > > the
> > > > > >> main
> > > > > >> > > > > direction
> > > > > >> > > > > > > of
> > > > > >> > > > > > > > > > > the design that can help people get familiar
> > > with
> > > > > the
> > > > > >> new
> > > > > >> > > > > > approach.
> > > > > >> > > > > > > > > > > On Tue, Oct 16, 2018 at 11:12 AM Zhang
> Yifei <
> > > > > >> > > > > > > yifei.z.luiz@gmail.com
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > > > > wrote:
> > > > > >> > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > Hey guys,
> > > > > >> > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > We have reorganized the Poc repo, and want
> > to
> > > > > merge
> > > > > >> it
> > > > > >> > to
> > > > > >> > > > our
> > > > > >> > > > > > > > Apache
> > > > > >> > > > > > > > > > > repo.
> > > > > >> > > > > > > > > > > > Just making sure here before we do the
> > merge,
> > > > > >> because
> > > > > >> > im
> > > > > >> > > > not
> > > > > >> > > > > > sure
> > > > > >> > > > > > > > if
> > > > > >> > > > > > > > > > > doing
> > > > > >> > > > > > > > > > > > this will perserve the Git commit history.
> > > > > >> > > > > > > > > > > > What we are planning to do is:
> > > > > >> > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > git filter-branch --subdirectory-filter
> <git
> > > > > >> > > > > origin_repository
> > > > > >> > > > > > > > > > directory>
> > > > > >> > > > > > > > > > > > -- --all
> > > > > >> > > > > > > > > > > > mkdir POCs/
> > > > > >> > > > > > > > > > > > git mv * POCs/
> > > > > >> > > > > > > > > > > > git commit -m "colleted the data to move"
> > > > > >> > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > git clone <git destination_repository_url>
> > cd
> > > > <git
> > > > > >> > > > > > > > > > > > destination_repository_directory> git
> remote
> > > add
> > > > > >> > > > > > > > > > origin_repository_branch
> > > > > >> > > > > > > > > > > > <git origin_repository directory> git pull
> > > > > >> > > > > > > origin_repository_branch
> > > > > >> > > > > > > > > > > master
> > > > > >> > > > > > > > > > > > --allow-unrelated-histories git remote rm
> > > > > >> > > > > > > origin_repository_branch
> > > > > >> > > > > > > > > > > >
> > > > > >> > > > > > > > > > > >
> > > > > >> > > > > > > > > > > > Thanks in advance !
> > > > > >> > > > > > > > > > > >
> > > > > >> > > > > > > > > > > >
> > > > > >> > > > >
> > > --------------------------------------------------------------
> > > > > >> > > > > > > > > > > > Zhang Yifei
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > > > --
> > > > > >> > > > > > > > > > > Luciano Resende
> > > > > >> > > > > > > > > > > http://twitter.com/lresende1975
> > > > > >> > > > > > > > > > > http://lresende.blogspot.com/
> > > > > >> > > > > > > > > > >
> > > > > >> > > > > > > > > >
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > > --
> > > > > >> > > > > > > > >
> > > > > >> > --------------------------------------------------------------
> > > > > >> > > > > > > > > Zhang Yifei
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > >
> > > > > >> > > > > > >
> > > > > >> > > > > >
> > > > > >> > > > >
> > > > > >> > > >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: POCs, Prototypes, experiments ...

Posted by Lucas Bonatto Miguel <lu...@apache.org>.

For sure, both suggestions are to make sure we're not killing the CLI and
that we have a component (the SDK client) that is shared by CLI and REPL.

Thanks for clarifying where the system calls will live.

- Lucas

On Fri, Oct 26, 2018 at 2:39 PM Daniel Takabayashi <
daniel.takabayashi@gmail.com> wrote:

> Hi Lucas, could you please justified your two suggestions?
>
> The general ideia is to use the Docker DSK to execute local commands, to
> consume outputs and also to execute Marvin "specific language" commands.
> Today almost all CLIs commands in Marvin is a mixed of systems commands and
> library method calls. The plan is to put all "specific language calls"
> inside the marvin-common-libs and group the systems call on the toolbox
> (using the docker SDK in this task).
>
>
>
>
> Em qui, 25 de out de 2018 às 20:09, Lucas Bonatto Miguel <
> lucasbm88@apache.org> escreveu:
>
> > Great, thanks.
> >
> > So here some suggestions:
> >  1. We could create one new layer between Toolbox and the Engines, this
> > would be Docker client.
> >  2. Split the Toolbox layer in two. Toolbox CLI and Toolbox REPL
> >
> > It's not clear to me if the commands listed on General Commands will need
> > to be written for every language, from the image I believe not, but how
> it
> > would work, since starting a notebook in Python is different than
> starting
> > in Scala.
> >
> > - Lucas
> > On Thu, Oct 25, 2018 at 4:51 PM Daniel Takabayashi <
> > daniel.takabayashi@gmail.com> wrote:
> >
> > >
> > >
> >
> https://docs.google.com/drawings/d/shajxIpLJHxxMbFgDXiPuhg/image?w=602&h=461&rev=1423&ac=1&parent=1ySERHGBXbHeyCMRookq5UfTuFkzzU0ugtjvR3rF3deY
> > >
> > > Em qui, 25 de out de 2018 às 07:38, Rafael Novello <
> > > rafa.reis.novello@gmail.com> escreveu:
> > >
> > > > Hi Taka,
> > > >
> > > > I can't see the image! Maybe the Apache email list don't support
> > > > attachments.
> > > >
> > > > Best!
> > > > Rafael J. R. Novello
> > > >
> > > > Skype: rafael.novello
> > > > Blog: http://rafanovello.blogspot.com.br/
> > > >
> > > >
> > > > Em qua, 24 de out de 2018 às 21:14, Daniel Takabayashi <
> > > > daniel.takabayashi@gmail.com> escreveu:
> > > >
> > > > > Hi all,
> > > > >
> > > > > To try to give a little more details about this "new concept" that
> we
> > > > want
> > > > > to bring to the marvin toolbox, I did this simple architecture
> draw.
> > > > >
> > > > > [image: marvin-architecture-views (5).png]
> > > > >
> > > > > The general idea here is try to transform the toolbox something
> > > > > disconnected with the "language", something more agnostic. Also in
> > this
> > > > new
> > > > > architecture we could use remote resource to process engines and
> make
> > > > easy
> > > > > the support for new languages.
> > > > >
> > > > > This "new toolbox" will be the only thing that a Marvin user must
> to
> > > > > install and also we could start to support multiples O.S (once the
> > REPL
> > > > is
> > > > > a dummy application that only interprets and by pass commands).
> > > > >
> > > > > Regards,
> > > > > Taka
> > > > >
> > > > > Em qua, 24 de out de 2018 às 09:52, Rafael Novello <
> > > > > rafa.reis.novello@gmail.com> escreveu:
> > > > >
> > > > >> Alan,
> > > > >>
> > > > >> Yes! We are using the Docker SDK and it's possible to use the API
> to
> > > > >> automate, but it's a little bit harder than automate CLI calls.
> > > > >>
> > > > >> Atenciosamente,
> > > > >> Rafael J. R. Novello
> > > > >>
> > > > >> Skype: rafael.novello
> > > > >> Blog: http://rafanovello.blogspot.com.br/
> > > > >>
> > > > >>
> > > > >> Em qua, 24 de out de 2018 às 12:02, Alan Silva <ju...@apache.org>
> > > > >> escreveu:
> > > > >>
> > > > >> > Hi,
> > > > >> >
> > > > >> > One question here, I understand that we start to use with this
> PoC
> > > the
> > > > >> > Docker SDK API, right?
> > > > >> >
> > > > >> > Why not use the API to expose some endpoints to permit this kind
> > of
> > > > >> > automation by devops?
> > > > >> >
> > > > >> > I think it is possible and it solves the CLI problem, right?
> > > > >> >
> > > > >> >
> > > > >> > On Tue, Oct 23, 2018 at 1:05 PM Rafael Novello <
> > > > >> > rafa.reis.novello@gmail.com>
> > > > >> > wrote:
> > > > >> >
> > > > >> > > Lucas,
> > > > >> > >
> > > > >> > > Sorry, I didn't understood your question bellow.
> > > > >> > >
> > > > >> > > "Would it make sense to use the same solution that we will use
> > for
> > > > >> > having a
> > > > >> > > single-language REPL to have a single-language CLI?"
> > > > >> > >
> > > > >> > > For DevOps purposes, maybe this new toolbox concept is not
> > ideal.
> > > I
> > > > >> think
> > > > >> > > we can keep the CLI inside the docker container but it will
> not
> > > easy
> > > > >> to
> > > > >> > > automate
> > > > >> > > jobs by this way.
> > > > >> > >
> > > > >> > > How to deal with this issue? Voting?
> > > > >> > > Atenciosamente,
> > > > >> > > Rafael J. R. Novello
> > > > >> > >
> > > > >> > > Skype: rafael.novello
> > > > >> > > Blog: http://rafanovello.blogspot.com.br/
> > > > >> > >
> > > > >> > >
> > > > >> > > Em sex, 19 de out de 2018 às 19:00, Lucas Bonatto Miguel <
> > > > >> > > lucasbm88@apache.org> escreveu:
> > > > >> > >
> > > > >> > > > Got it! Thanks for clarifying.
> > > > >> > > >
> > > > >> > > > Would it make sense to use the same solution that we will
> use
> > > for
> > > > >> > having
> > > > >> > > a
> > > > >> > > > single-language REPL to have a single-language CLI? My only
> > > > concern
> > > > >> > with
> > > > >> > > > killing the CLI is that you remove an essential feature for
> > > > DevOps.
> > > > >> > > >
> > > > >> > > > - Lucas
> > > > >> > > >
> > > > >> > > > On Fri, Oct 19, 2018 at 1:52 PM Rafael Novello <
> > > > >> > > > rafa.reis.novello@gmail.com>
> > > > >> > > > wrote:
> > > > >> > > >
> > > > >> > > > > Lucas,
> > > > >> > > > >
> > > > >> > > > > The idea is that REPL will substitute the actual CLI. It's
> > > > because
> > > > >> > with
> > > > >> > > > the
> > > > >> > > > > actual concept (using language specific CLI) we will need
> to
> > > > >> > > re-implement
> > > > >> > > > > the same features many times and probably each CLI will
> > have a
> > > > >> > > different
> > > > >> > > > > behavior because some language specific restrictions
> and/or
> > > > >> > > limitations.
> > > > >> > > > >
> > > > >> > > > > With this new concept, all users will have the same
> > experience
> > > > >> > > > interacting
> > > > >> > > > > with Marvin REPL and they will use the bests tools to
> > develop
> > > > >> their
> > > > >> > > > engines
> > > > >> > > > > (Jupyter Notebook and/or Lab with different languages
> > support
> > > or
> > > > >> even
> > > > >> > > > > Apache Zeppelin). All the interact will occur through REPL
> > and
> > > > >> docker
> > > > >> > > > > protocol.
> > > > >> > > > >
> > > > >> > > > > Alan, Yes! As Lucas said, the concept is the same but we
> can
> > > use
> > > > >> > docker
> > > > >> > > > to
> > > > >> > > > > do the same job.
> > > > >> > > > >
> > > > >> > > > > Best!
> > > > >> > > > >
> > > > >> > > > > Em sex, 19 de out de 2018 às 00:39, Lucas Bonatto Miguel <
> > > > >> > > > > lucasbm88@apache.org> escreveu:
> > > > >> > > > >
> > > > >> > > > > > Thanks for the clarifications Rafael, so from what I
> > > > understood,
> > > > >> > > > Marvin's
> > > > >> > > > > > developers would not use the REPL to explore data and
> test
> > > > >> models,
> > > > >> > > i.e.
> > > > >> > > > > > develop the engine. Is the idea to build something more
> > like
> > > > an
> > > > >> > > > > interactive
> > > > >> > > > > > CLI? I think an interactive CLI would be useful for the
> > > > >> developer
> > > > >> > > > > > experience, however, it's important to keep the
> unattended
> > > > >> > (current)
> > > > >> > > > CLI
> > > > >> > > > > in
> > > > >> > > > > > place for scenarios where users want to automate Marvin
> > > tasks.
> > > > >> > > > > >
> > > > >> > > > > > Alan, I believe the idea is still the same as you
> started,
> > > but
> > > > >> > using
> > > > >> > > > > > docker SDK now.
> > > > >> > > > > >
> > > > >> > > > > > - Lucas
> > > > >> > > > > >
> > > > >> > > > > > On Thu, Oct 18, 2018 at 1:29 PM Rafael Novello <
> > > > >> > > > > > rafa.reis.novello@gmail.com>
> > > > >> > > > > > wrote:
> > > > >> > > > > >
> > > > >> > > > > > > Hi Lucas!
> > > > >> > > > > > >
> > > > >> > > > > > > First of all +1 for REPL POCs to Apache!
> > > > >> > > > > > >
> > > > >> > > > > > > Let me help with some comments:
> > > > >> > > > > > >
> > > > >> > > > > > > 1 - We have tested NodeJS, Scala and Python and the
> > > easiest
> > > > >> one
> > > > >> > was
> > > > >> > > > > > Python.
> > > > >> > > > > > > We have found a small project [1] that have all
> features
> > > we
> > > > >> > desired
> > > > >> > > > for
> > > > >> > > > > > > REPL:
> > > > >> > > > > > > - Autocomplete
> > > > >> > > > > > > - Python commands disabled (the user have only the
> > > commands
> > > > we
> > > > >> > > > > provide).
> > > > >> > > > > > > The other languages REPL options don't have this
> > feature.
> > > > >> > > > > > > - Easy to show output
> > > > >> > > > > > > - Etc
> > > > >> > > > > > > So, I think the language chosen will not be important
> > here
> > > > >> > because
> > > > >> > > > the
> > > > >> > > > > > user
> > > > >> > > > > > > will only interact with the commands that we create.
> > > > >> > > > > > >
> > > > >> > > > > > > 2 - The "engine-generate" command will download a
> docker
> > > > image
> > > > >> > that
> > > > >> > > > we
> > > > >> > > > > > > create for that language and start a container with
> the
> > > > basic
> > > > >> > > project
> > > > >> > > > > > > structure for that language.
> > > > >> > > > > > >
> > > > >> > > > > > > 3 - The REPL client will use the "docker protocol" to
> > run
> > > > >> > command,
> > > > >> > > > > > > start/stop services and etc inside the container and
> it
> > > will
> > > > >> > > receive
> > > > >> > > > > the
> > > > >> > > > > > > log stream to show. No, the REPL will no pass code
> > > snippets
> > > > >> for
> > > > >> > > > docker
> > > > >> > > > > > > container (I think it will not be necessary)
> > > > >> > > > > > >
> > > > >> > > > > > > 4 - Yep! Like I said on the first item.
> > > > >> > > > > > >
> > > > >> > > > > > > [1] - https://github.com/italorossi/ishell
> > > > >> > > > > > >
> > > > >> > > > > > > Let me know if there is any other question!
> > > > >> > > > > > > Best!
> > > > >> > > > > > > Rafael J. R. Novello
> > > > >> > > > > > >
> > > > >> > > > > > > Skype: rafael.novello
> > > > >> > > > > > > Blog: http://rafanovello.blogspot.com.br/
> > > > >> > > > > > >
> > > > >> > > > > > >
> > > > >> > > > > > > Em qui, 18 de out de 2018 às 17:00, Lucas Bonatto
> > Miguel <
> > > > >> > > > > > > lucasbm88@apache.org> escreveu:
> > > > >> > > > > > >
> > > > >> > > > > > > > + 1 for migrating the REPL repo to Apache
> > > > >> > > > > > > >
> > > > >> > > > > > > > I have a few questions about the previous
> explanation:
> > > > >> > > > > > > >  1) The REPL itself, it would be an application in
> > which
> > > > >> > > language?
> > > > >> > > > > > > Remember
> > > > >> > > > > > > > that the main idea is to allow the user to program
> on
> > > his
> > > > >> > > preferred
> > > > >> > > > > > > > language in the REPL.
> > > > >> > > > > > > >  2) Should the engine-generate command also
> generate a
> > > > >> docker
> > > > >> > > image
> > > > >> > > > > > with
> > > > >> > > > > > > > the user's application?
> > > > >> > > > > > > >  3) What type of communication would happen between
> > the
> > > > REPL
> > > > >> > and
> > > > >> > > > the
> > > > >> > > > > > > engine
> > > > >> > > > > > > > via Docker SDK? Would the REPL pass snippets of code
> > to
> > > be
> > > > >> > > executed
> > > > >> > > > > by
> > > > >> > > > > > > the
> > > > >> > > > > > > > docker container?
> > > > >> > > > > > > >  4) Have you considered code completion in the REPL?
> > > > >> > > > > > > >
> > > > >> > > > > > > >
> > > > >> > > > > > > > On Thu, Oct 18, 2018 at 7:53 AM Zhang Yifei <
> > > > >> > > > yifei.z.luiz@gmail.com>
> > > > >> > > > > > > > wrote:
> > > > >> > > > > > > >
> > > > >> > > > > > > > > Ok guys,
> > > > >> > > > > > > > >
> > > > >> > > > > > > > > The basic idea is to provide only one Toolbox for
> > > > multiple
> > > > >> > > > > languages.
> > > > >> > > > > > > > > We are looking for possibility to build a single
> > > Marvin
> > > > >> Repl,
> > > > >> > > > > > > > > instead of severals toolboxes with differentes
> > > > interfaces
> > > > >> or
> > > > >> > > > > > commands.
> > > > >> > > > > > > > >
> > > > >> > > > > > > > > In this case, the engine-generate command will
> > > download
> > > > >> and
> > > > >> > > > start a
> > > > >> > > > > > > > Docker
> > > > >> > > > > > > > > container with basic engine structure
> corresponding
> > > the
> > > > >> > choosed
> > > > >> > > > > > > language.
> > > > >> > > > > > > > > this means we don't need to build Toolboxes of
> > > > differents
> > > > >> > > > > languages,
> > > > >> > > > > > we
> > > > >> > > > > > > > > just
> > > > >> > > > > > > > > build the engine template of all languages that we
> > > want
> > > > to
> > > > >> > > > support
> > > > >> > > > > > and
> > > > >> > > > > > > > > provide it
> > > > >> > > > > > > > > as Docker containers
> > > > >> > > > > > > > >
> > > > >> > > > > > > > > We have started researches around our basic
> > > requirements
> > > > >> > like:
> > > > >> > > > > > > > > - Repl interface
> > > > >> > > > > > > > > - System communication
> > > > >> > > > > > > > > - Connection security
> > > > >> > > > > > > > > - Tool popularity
> > > > >> > > > > > > > > - Update complexity
> > > > >> > > > > > > > > - Languages support
> > > > >> > > > > > > > > - ......
> > > > >> > > > > > > > >
> > > > >> > > > > > > > > And we did some POC with code here:
> > > > >> > > > > > > > > https://github.com/marvin-ai/marvin-repl
> > > > >> > > > > > > > >
> > > > >> > > > > > > > > There is POC testing gRPC using Scala and Python,
> > > > >> > > > > > > > > Repl inteface and Docker SDK with NodeJS,
> > > > >> > > > > > > > > Repl interface and Docker SDK with Python.
> > > > >> > > > > > > > >
> > > > >> > > > > > > > > At this moment we prefer the Repl interface +
> Docker
> > > SDK
> > > > >> way,
> > > > >> > > > > because
> > > > >> > > > > > > > good
> > > > >> > > > > > > > > part of the requirements
> > > > >> > > > > > > > > will be guaranteed by Docker.
> > > > >> > > > > > > > >
> > > > >> > > > > > > > > With this informations, what do you think? Should
> we
> > > > >> submit
> > > > >> > all
> > > > >> > > > > this
> > > > >> > > > > > > POCs
> > > > >> > > > > > > > > to Apache Repo?
> > > > >> > > > > > > > > Please feel free to opine.
> > > > >> > > > > > > > >
> > > > >> > > > > > > > >
> > > > >> > > > > > > > > Thats all, thanks!!!
> > > > >> > > > > > > > >
> > > > >> > > > > > > > >
> > > > >> > > > > > > > > Em ter, 16 de out de 2018 às 18:55, Daniel
> > > Takabayashi <
> > > > >> > > > > > > > > daniel.takabayashi@gmail.com> escreveu:
> > > > >> > > > > > > > >
> > > > >> > > > > > > > > > Zhang,
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > > > I think the best approach is give us a better
> > > > >> explanation
> > > > >> > > about
> > > > >> > > > > > this
> > > > >> > > > > > > > new
> > > > >> > > > > > > > > > feature and how this can help us to archive the
> > > > desired
> > > > >> > > feature
> > > > >> > > > > > > > (support
> > > > >> > > > > > > > > > multiple languages). Than if the majority agree
> > > than I
> > > > >> can
> > > > >> > do
> > > > >> > > > the
> > > > >> > > > > > > > merge,
> > > > >> > > > > > > > > > just like I did before, but in another branch.
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > > > Let me know if makes sense, ok?
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > > > Taka
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > > > Em ter, 16 de out de 2018 às 14:00, Luciano
> > Resende
> > > <
> > > > >> > > > > > > > > luckbr1975@gmail.com>
> > > > >> > > > > > > > > > escreveu:
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > > > > So, what is the POC, is it a refactoring of
> the
> > > > >> existing
> > > > >> > > > repo?
> > > > >> > > > > Or
> > > > >> > > > > > > is
> > > > >> > > > > > > > > > > it a new rewrite of the repo?
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > > > > > Just asking as it might make sense to make it
> a
> > > > branch
> > > > >> > then
> > > > >> > > > > > > actually
> > > > >> > > > > > > > a
> > > > >> > > > > > > > > > > thing in parallel, as this will have an effect
> > for
> > > > >> > > releases,
> > > > >> > > > > etc.
> > > > >> > > > > > > but
> > > > >> > > > > > > > > > > you guys know more here than I do.
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > > > > > Also, it's probably good to have a write up of
> > the
> > > > >> main
> > > > >> > > > > direction
> > > > >> > > > > > > of
> > > > >> > > > > > > > > > > the design that can help people get familiar
> > with
> > > > the
> > > > >> new
> > > > >> > > > > > approach.
> > > > >> > > > > > > > > > > On Tue, Oct 16, 2018 at 11:12 AM Zhang Yifei <
> > > > >> > > > > > > yifei.z.luiz@gmail.com
> > > > >> > > > > > > > >
> > > > >> > > > > > > > > > > wrote:
> > > > >> > > > > > > > > > > >
> > > > >> > > > > > > > > > > > Hey guys,
> > > > >> > > > > > > > > > > >
> > > > >> > > > > > > > > > > > We have reorganized the Poc repo, and want
> to
> > > > merge
> > > > >> it
> > > > >> > to
> > > > >> > > > our
> > > > >> > > > > > > > Apache
> > > > >> > > > > > > > > > > repo.
> > > > >> > > > > > > > > > > > Just making sure here before we do the
> merge,
> > > > >> because
> > > > >> > im
> > > > >> > > > not
> > > > >> > > > > > sure
> > > > >> > > > > > > > if
> > > > >> > > > > > > > > > > doing
> > > > >> > > > > > > > > > > > this will perserve the Git commit history.
> > > > >> > > > > > > > > > > > What we are planning to do is:
> > > > >> > > > > > > > > > > >
> > > > >> > > > > > > > > > > > git filter-branch --subdirectory-filter <git
> > > > >> > > > > origin_repository
> > > > >> > > > > > > > > > directory>
> > > > >> > > > > > > > > > > > -- --all
> > > > >> > > > > > > > > > > > mkdir POCs/
> > > > >> > > > > > > > > > > > git mv * POCs/
> > > > >> > > > > > > > > > > > git commit -m "colleted the data to move"
> > > > >> > > > > > > > > > > >
> > > > >> > > > > > > > > > > > git clone <git destination_repository_url>
> cd
> > > <git
> > > > >> > > > > > > > > > > > destination_repository_directory> git remote
> > add
> > > > >> > > > > > > > > > origin_repository_branch
> > > > >> > > > > > > > > > > > <git origin_repository directory> git pull
> > > > >> > > > > > > origin_repository_branch
> > > > >> > > > > > > > > > > master
> > > > >> > > > > > > > > > > > --allow-unrelated-histories git remote rm
> > > > >> > > > > > > origin_repository_branch
> > > > >> > > > > > > > > > > >
> > > > >> > > > > > > > > > > >
> > > > >> > > > > > > > > > > > Thanks in advance !
> > > > >> > > > > > > > > > > >
> > > > >> > > > > > > > > > > >
> > > > >> > > > >
> > --------------------------------------------------------------
> > > > >> > > > > > > > > > > > Zhang Yifei
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > > > > > --
> > > > >> > > > > > > > > > > Luciano Resende
> > > > >> > > > > > > > > > > http://twitter.com/lresende1975
> > > > >> > > > > > > > > > > http://lresende.blogspot.com/
> > > > >> > > > > > > > > > >
> > > > >> > > > > > > > > >
> > > > >> > > > > > > > >
> > > > >> > > > > > > > >
> > > > >> > > > > > > > > --
> > > > >> > > > > > > > >
> > > > >> > --------------------------------------------------------------
> > > > >> > > > > > > > > Zhang Yifei
> > > > >> > > > > > > > >
> > > > >> > > > > > > >
> > > > >> > > > > > >
> > > > >> > > > > >
> > > > >> > > > >
> > > > >> > > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > > >
> > > >
> > >
> >
>

Re: POCs, Prototypes, experiments ...

Posted by Daniel Takabayashi <da...@gmail.com>.

Hi Lucas, could you please justified your two suggestions?

The general ideia is to use the Docker DSK to execute local commands, to
consume outputs and also to execute Marvin "specific language" commands.
Today almost all CLIs commands in Marvin is a mixed of systems commands and
library method calls. The plan is to put all "specific language calls"
inside the marvin-common-libs and group the systems call on the toolbox
(using the docker SDK in this task).




Em qui, 25 de out de 2018 às 20:09, Lucas Bonatto Miguel <
lucasbm88@apache.org> escreveu:

> Great, thanks.
>
> So here some suggestions:
>  1. We could create one new layer between Toolbox and the Engines, this
> would be Docker client.
>  2. Split the Toolbox layer in two. Toolbox CLI and Toolbox REPL
>
> It's not clear to me if the commands listed on General Commands will need
> to be written for every language, from the image I believe not, but how it
> would work, since starting a notebook in Python is different than starting
> in Scala.
>
> - Lucas
> On Thu, Oct 25, 2018 at 4:51 PM Daniel Takabayashi <
> daniel.takabayashi@gmail.com> wrote:
>
> >
> >
> https://docs.google.com/drawings/d/shajxIpLJHxxMbFgDXiPuhg/image?w=602&h=461&rev=1423&ac=1&parent=1ySERHGBXbHeyCMRookq5UfTuFkzzU0ugtjvR3rF3deY
> >
> > Em qui, 25 de out de 2018 às 07:38, Rafael Novello <
> > rafa.reis.novello@gmail.com> escreveu:
> >
> > > Hi Taka,
> > >
> > > I can't see the image! Maybe the Apache email list don't support
> > > attachments.
> > >
> > > Best!
> > > Rafael J. R. Novello
> > >
> > > Skype: rafael.novello
> > > Blog: http://rafanovello.blogspot.com.br/
> > >
> > >
> > > Em qua, 24 de out de 2018 às 21:14, Daniel Takabayashi <
> > > daniel.takabayashi@gmail.com> escreveu:
> > >
> > > > Hi all,
> > > >
> > > > To try to give a little more details about this "new concept" that we
> > > want
> > > > to bring to the marvin toolbox, I did this simple architecture draw.
> > > >
> > > > [image: marvin-architecture-views (5).png]
> > > >
> > > > The general idea here is try to transform the toolbox something
> > > > disconnected with the "language", something more agnostic. Also in
> this
> > > new
> > > > architecture we could use remote resource to process engines and make
> > > easy
> > > > the support for new languages.
> > > >
> > > > This "new toolbox" will be the only thing that a Marvin user must to
> > > > install and also we could start to support multiples O.S (once the
> REPL
> > > is
> > > > a dummy application that only interprets and by pass commands).
> > > >
> > > > Regards,
> > > > Taka
> > > >
> > > > Em qua, 24 de out de 2018 às 09:52, Rafael Novello <
> > > > rafa.reis.novello@gmail.com> escreveu:
> > > >
> > > >> Alan,
> > > >>
> > > >> Yes! We are using the Docker SDK and it's possible to use the API to
> > > >> automate, but it's a little bit harder than automate CLI calls.
> > > >>
> > > >> Atenciosamente,
> > > >> Rafael J. R. Novello
> > > >>
> > > >> Skype: rafael.novello
> > > >> Blog: http://rafanovello.blogspot.com.br/
> > > >>
> > > >>
> > > >> Em qua, 24 de out de 2018 às 12:02, Alan Silva <ju...@apache.org>
> > > >> escreveu:
> > > >>
> > > >> > Hi,
> > > >> >
> > > >> > One question here, I understand that we start to use with this PoC
> > the
> > > >> > Docker SDK API, right?
> > > >> >
> > > >> > Why not use the API to expose some endpoints to permit this kind
> of
> > > >> > automation by devops?
> > > >> >
> > > >> > I think it is possible and it solves the CLI problem, right?
> > > >> >
> > > >> >
> > > >> > On Tue, Oct 23, 2018 at 1:05 PM Rafael Novello <
> > > >> > rafa.reis.novello@gmail.com>
> > > >> > wrote:
> > > >> >
> > > >> > > Lucas,
> > > >> > >
> > > >> > > Sorry, I didn't understood your question bellow.
> > > >> > >
> > > >> > > "Would it make sense to use the same solution that we will use
> for
> > > >> > having a
> > > >> > > single-language REPL to have a single-language CLI?"
> > > >> > >
> > > >> > > For DevOps purposes, maybe this new toolbox concept is not
> ideal.
> > I
> > > >> think
> > > >> > > we can keep the CLI inside the docker container but it will not
> > easy
> > > >> to
> > > >> > > automate
> > > >> > > jobs by this way.
> > > >> > >
> > > >> > > How to deal with this issue? Voting?
> > > >> > > Atenciosamente,
> > > >> > > Rafael J. R. Novello
> > > >> > >
> > > >> > > Skype: rafael.novello
> > > >> > > Blog: http://rafanovello.blogspot.com.br/
> > > >> > >
> > > >> > >
> > > >> > > Em sex, 19 de out de 2018 às 19:00, Lucas Bonatto Miguel <
> > > >> > > lucasbm88@apache.org> escreveu:
> > > >> > >
> > > >> > > > Got it! Thanks for clarifying.
> > > >> > > >
> > > >> > > > Would it make sense to use the same solution that we will use
> > for
> > > >> > having
> > > >> > > a
> > > >> > > > single-language REPL to have a single-language CLI? My only
> > > concern
> > > >> > with
> > > >> > > > killing the CLI is that you remove an essential feature for
> > > DevOps.
> > > >> > > >
> > > >> > > > - Lucas
> > > >> > > >
> > > >> > > > On Fri, Oct 19, 2018 at 1:52 PM Rafael Novello <
> > > >> > > > rafa.reis.novello@gmail.com>
> > > >> > > > wrote:
> > > >> > > >
> > > >> > > > > Lucas,
> > > >> > > > >
> > > >> > > > > The idea is that REPL will substitute the actual CLI. It's
> > > because
> > > >> > with
> > > >> > > > the
> > > >> > > > > actual concept (using language specific CLI) we will need to
> > > >> > > re-implement
> > > >> > > > > the same features many times and probably each CLI will
> have a
> > > >> > > different
> > > >> > > > > behavior because some language specific restrictions and/or
> > > >> > > limitations.
> > > >> > > > >
> > > >> > > > > With this new concept, all users will have the same
> experience
> > > >> > > > interacting
> > > >> > > > > with Marvin REPL and they will use the bests tools to
> develop
> > > >> their
> > > >> > > > engines
> > > >> > > > > (Jupyter Notebook and/or Lab with different languages
> support
> > or
> > > >> even
> > > >> > > > > Apache Zeppelin). All the interact will occur through REPL
> and
> > > >> docker
> > > >> > > > > protocol.
> > > >> > > > >
> > > >> > > > > Alan, Yes! As Lucas said, the concept is the same but we can
> > use
> > > >> > docker
> > > >> > > > to
> > > >> > > > > do the same job.
> > > >> > > > >
> > > >> > > > > Best!
> > > >> > > > >
> > > >> > > > > Em sex, 19 de out de 2018 às 00:39, Lucas Bonatto Miguel <
> > > >> > > > > lucasbm88@apache.org> escreveu:
> > > >> > > > >
> > > >> > > > > > Thanks for the clarifications Rafael, so from what I
> > > understood,
> > > >> > > > Marvin's
> > > >> > > > > > developers would not use the REPL to explore data and test
> > > >> models,
> > > >> > > i.e.
> > > >> > > > > > develop the engine. Is the idea to build something more
> like
> > > an
> > > >> > > > > interactive
> > > >> > > > > > CLI? I think an interactive CLI would be useful for the
> > > >> developer
> > > >> > > > > > experience, however, it's important to keep the unattended
> > > >> > (current)
> > > >> > > > CLI
> > > >> > > > > in
> > > >> > > > > > place for scenarios where users want to automate Marvin
> > tasks.
> > > >> > > > > >
> > > >> > > > > > Alan, I believe the idea is still the same as you started,
> > but
> > > >> > using
> > > >> > > > > > docker SDK now.
> > > >> > > > > >
> > > >> > > > > > - Lucas
> > > >> > > > > >
> > > >> > > > > > On Thu, Oct 18, 2018 at 1:29 PM Rafael Novello <
> > > >> > > > > > rafa.reis.novello@gmail.com>
> > > >> > > > > > wrote:
> > > >> > > > > >
> > > >> > > > > > > Hi Lucas!
> > > >> > > > > > >
> > > >> > > > > > > First of all +1 for REPL POCs to Apache!
> > > >> > > > > > >
> > > >> > > > > > > Let me help with some comments:
> > > >> > > > > > >
> > > >> > > > > > > 1 - We have tested NodeJS, Scala and Python and the
> > easiest
> > > >> one
> > > >> > was
> > > >> > > > > > Python.
> > > >> > > > > > > We have found a small project [1] that have all features
> > we
> > > >> > desired
> > > >> > > > for
> > > >> > > > > > > REPL:
> > > >> > > > > > > - Autocomplete
> > > >> > > > > > > - Python commands disabled (the user have only the
> > commands
> > > we
> > > >> > > > > provide).
> > > >> > > > > > > The other languages REPL options don't have this
> feature.
> > > >> > > > > > > - Easy to show output
> > > >> > > > > > > - Etc
> > > >> > > > > > > So, I think the language chosen will not be important
> here
> > > >> > because
> > > >> > > > the
> > > >> > > > > > user
> > > >> > > > > > > will only interact with the commands that we create.
> > > >> > > > > > >
> > > >> > > > > > > 2 - The "engine-generate" command will download a docker
> > > image
> > > >> > that
> > > >> > > > we
> > > >> > > > > > > create for that language and start a container with the
> > > basic
> > > >> > > project
> > > >> > > > > > > structure for that language.
> > > >> > > > > > >
> > > >> > > > > > > 3 - The REPL client will use the "docker protocol" to
> run
> > > >> > command,
> > > >> > > > > > > start/stop services and etc inside the container and it
> > will
> > > >> > > receive
> > > >> > > > > the
> > > >> > > > > > > log stream to show. No, the REPL will no pass code
> > snippets
> > > >> for
> > > >> > > > docker
> > > >> > > > > > > container (I think it will not be necessary)
> > > >> > > > > > >
> > > >> > > > > > > 4 - Yep! Like I said on the first item.
> > > >> > > > > > >
> > > >> > > > > > > [1] - https://github.com/italorossi/ishell
> > > >> > > > > > >
> > > >> > > > > > > Let me know if there is any other question!
> > > >> > > > > > > Best!
> > > >> > > > > > > Rafael J. R. Novello
> > > >> > > > > > >
> > > >> > > > > > > Skype: rafael.novello
> > > >> > > > > > > Blog: http://rafanovello.blogspot.com.br/
> > > >> > > > > > >
> > > >> > > > > > >
> > > >> > > > > > > Em qui, 18 de out de 2018 às 17:00, Lucas Bonatto
> Miguel <
> > > >> > > > > > > lucasbm88@apache.org> escreveu:
> > > >> > > > > > >
> > > >> > > > > > > > + 1 for migrating the REPL repo to Apache
> > > >> > > > > > > >
> > > >> > > > > > > > I have a few questions about the previous explanation:
> > > >> > > > > > > >  1) The REPL itself, it would be an application in
> which
> > > >> > > language?
> > > >> > > > > > > Remember
> > > >> > > > > > > > that the main idea is to allow the user to program on
> > his
> > > >> > > preferred
> > > >> > > > > > > > language in the REPL.
> > > >> > > > > > > >  2) Should the engine-generate command also generate a
> > > >> docker
> > > >> > > image
> > > >> > > > > > with
> > > >> > > > > > > > the user's application?
> > > >> > > > > > > >  3) What type of communication would happen between
> the
> > > REPL
> > > >> > and
> > > >> > > > the
> > > >> > > > > > > engine
> > > >> > > > > > > > via Docker SDK? Would the REPL pass snippets of code
> to
> > be
> > > >> > > executed
> > > >> > > > > by
> > > >> > > > > > > the
> > > >> > > > > > > > docker container?
> > > >> > > > > > > >  4) Have you considered code completion in the REPL?
> > > >> > > > > > > >
> > > >> > > > > > > >
> > > >> > > > > > > > On Thu, Oct 18, 2018 at 7:53 AM Zhang Yifei <
> > > >> > > > yifei.z.luiz@gmail.com>
> > > >> > > > > > > > wrote:
> > > >> > > > > > > >
> > > >> > > > > > > > > Ok guys,
> > > >> > > > > > > > >
> > > >> > > > > > > > > The basic idea is to provide only one Toolbox for
> > > multiple
> > > >> > > > > languages.
> > > >> > > > > > > > > We are looking for possibility to build a single
> > Marvin
> > > >> Repl,
> > > >> > > > > > > > > instead of severals toolboxes with differentes
> > > interfaces
> > > >> or
> > > >> > > > > > commands.
> > > >> > > > > > > > >
> > > >> > > > > > > > > In this case, the engine-generate command will
> > download
> > > >> and
> > > >> > > > start a
> > > >> > > > > > > > Docker
> > > >> > > > > > > > > container with basic engine structure corresponding
> > the
> > > >> > choosed
> > > >> > > > > > > language.
> > > >> > > > > > > > > this means we don't need to build Toolboxes of
> > > differents
> > > >> > > > > languages,
> > > >> > > > > > we
> > > >> > > > > > > > > just
> > > >> > > > > > > > > build the engine template of all languages that we
> > want
> > > to
> > > >> > > > support
> > > >> > > > > > and
> > > >> > > > > > > > > provide it
> > > >> > > > > > > > > as Docker containers
> > > >> > > > > > > > >
> > > >> > > > > > > > > We have started researches around our basic
> > requirements
> > > >> > like:
> > > >> > > > > > > > > - Repl interface
> > > >> > > > > > > > > - System communication
> > > >> > > > > > > > > - Connection security
> > > >> > > > > > > > > - Tool popularity
> > > >> > > > > > > > > - Update complexity
> > > >> > > > > > > > > - Languages support
> > > >> > > > > > > > > - ......
> > > >> > > > > > > > >
> > > >> > > > > > > > > And we did some POC with code here:
> > > >> > > > > > > > > https://github.com/marvin-ai/marvin-repl
> > > >> > > > > > > > >
> > > >> > > > > > > > > There is POC testing gRPC using Scala and Python,
> > > >> > > > > > > > > Repl inteface and Docker SDK with NodeJS,
> > > >> > > > > > > > > Repl interface and Docker SDK with Python.
> > > >> > > > > > > > >
> > > >> > > > > > > > > At this moment we prefer the Repl interface + Docker
> > SDK
> > > >> way,
> > > >> > > > > because
> > > >> > > > > > > > good
> > > >> > > > > > > > > part of the requirements
> > > >> > > > > > > > > will be guaranteed by Docker.
> > > >> > > > > > > > >
> > > >> > > > > > > > > With this informations, what do you think? Should we
> > > >> submit
> > > >> > all
> > > >> > > > > this
> > > >> > > > > > > POCs
> > > >> > > > > > > > > to Apache Repo?
> > > >> > > > > > > > > Please feel free to opine.
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > > > Thats all, thanks!!!
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > > > Em ter, 16 de out de 2018 às 18:55, Daniel
> > Takabayashi <
> > > >> > > > > > > > > daniel.takabayashi@gmail.com> escreveu:
> > > >> > > > > > > > >
> > > >> > > > > > > > > > Zhang,
> > > >> > > > > > > > > >
> > > >> > > > > > > > > > I think the best approach is give us a better
> > > >> explanation
> > > >> > > about
> > > >> > > > > > this
> > > >> > > > > > > > new
> > > >> > > > > > > > > > feature and how this can help us to archive the
> > > desired
> > > >> > > feature
> > > >> > > > > > > > (support
> > > >> > > > > > > > > > multiple languages). Than if the majority agree
> > than I
> > > >> can
> > > >> > do
> > > >> > > > the
> > > >> > > > > > > > merge,
> > > >> > > > > > > > > > just like I did before, but in another branch.
> > > >> > > > > > > > > >
> > > >> > > > > > > > > > Let me know if makes sense, ok?
> > > >> > > > > > > > > >
> > > >> > > > > > > > > > Taka
> > > >> > > > > > > > > >
> > > >> > > > > > > > > > Em ter, 16 de out de 2018 às 14:00, Luciano
> Resende
> > <
> > > >> > > > > > > > > luckbr1975@gmail.com>
> > > >> > > > > > > > > > escreveu:
> > > >> > > > > > > > > >
> > > >> > > > > > > > > > > So, what is the POC, is it a refactoring of the
> > > >> existing
> > > >> > > > repo?
> > > >> > > > > Or
> > > >> > > > > > > is
> > > >> > > > > > > > > > > it a new rewrite of the repo?
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > > Just asking as it might make sense to make it a
> > > branch
> > > >> > then
> > > >> > > > > > > actually
> > > >> > > > > > > > a
> > > >> > > > > > > > > > > thing in parallel, as this will have an effect
> for
> > > >> > > releases,
> > > >> > > > > etc.
> > > >> > > > > > > but
> > > >> > > > > > > > > > > you guys know more here than I do.
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > > Also, it's probably good to have a write up of
> the
> > > >> main
> > > >> > > > > direction
> > > >> > > > > > > of
> > > >> > > > > > > > > > > the design that can help people get familiar
> with
> > > the
> > > >> new
> > > >> > > > > > approach.
> > > >> > > > > > > > > > > On Tue, Oct 16, 2018 at 11:12 AM Zhang Yifei <
> > > >> > > > > > > yifei.z.luiz@gmail.com
> > > >> > > > > > > > >
> > > >> > > > > > > > > > > wrote:
> > > >> > > > > > > > > > > >
> > > >> > > > > > > > > > > > Hey guys,
> > > >> > > > > > > > > > > >
> > > >> > > > > > > > > > > > We have reorganized the Poc repo, and want to
> > > merge
> > > >> it
> > > >> > to
> > > >> > > > our
> > > >> > > > > > > > Apache
> > > >> > > > > > > > > > > repo.
> > > >> > > > > > > > > > > > Just making sure here before we do the merge,
> > > >> because
> > > >> > im
> > > >> > > > not
> > > >> > > > > > sure
> > > >> > > > > > > > if
> > > >> > > > > > > > > > > doing
> > > >> > > > > > > > > > > > this will perserve the Git commit history.
> > > >> > > > > > > > > > > > What we are planning to do is:
> > > >> > > > > > > > > > > >
> > > >> > > > > > > > > > > > git filter-branch --subdirectory-filter <git
> > > >> > > > > origin_repository
> > > >> > > > > > > > > > directory>
> > > >> > > > > > > > > > > > -- --all
> > > >> > > > > > > > > > > > mkdir POCs/
> > > >> > > > > > > > > > > > git mv * POCs/
> > > >> > > > > > > > > > > > git commit -m "colleted the data to move"
> > > >> > > > > > > > > > > >
> > > >> > > > > > > > > > > > git clone <git destination_repository_url> cd
> > <git
> > > >> > > > > > > > > > > > destination_repository_directory> git remote
> add
> > > >> > > > > > > > > > origin_repository_branch
> > > >> > > > > > > > > > > > <git origin_repository directory> git pull
> > > >> > > > > > > origin_repository_branch
> > > >> > > > > > > > > > > master
> > > >> > > > > > > > > > > > --allow-unrelated-histories git remote rm
> > > >> > > > > > > origin_repository_branch
> > > >> > > > > > > > > > > >
> > > >> > > > > > > > > > > >
> > > >> > > > > > > > > > > > Thanks in advance !
> > > >> > > > > > > > > > > >
> > > >> > > > > > > > > > > >
> > > >> > > > >
> --------------------------------------------------------------
> > > >> > > > > > > > > > > > Zhang Yifei
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > > > --
> > > >> > > > > > > > > > > Luciano Resende
> > > >> > > > > > > > > > > http://twitter.com/lresende1975
> > > >> > > > > > > > > > > http://lresende.blogspot.com/
> > > >> > > > > > > > > > >
> > > >> > > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > > >
> > > >> > > > > > > > > --
> > > >> > > > > > > > >
> > > >> > --------------------------------------------------------------
> > > >> > > > > > > > > Zhang Yifei
> > > >> > > > > > > > >
> > > >> > > > > > > >
> > > >> > > > > > >
> > > >> > > > > >
> > > >> > > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > > >
> > >
> >
>

Re: POCs, Prototypes, experiments ...

Posted by Lucas Bonatto Miguel <lu...@apache.org>.

Great, thanks.

So here some suggestions:
 1. We could create one new layer between Toolbox and the Engines, this
would be Docker client.
 2. Split the Toolbox layer in two. Toolbox CLI and Toolbox REPL

It's not clear to me if the commands listed on General Commands will need
to be written for every language, from the image I believe not, but how it
would work, since starting a notebook in Python is different than starting
in Scala.

- Lucas
On Thu, Oct 25, 2018 at 4:51 PM Daniel Takabayashi <
daniel.takabayashi@gmail.com> wrote:

>
> https://docs.google.com/drawings/d/shajxIpLJHxxMbFgDXiPuhg/image?w=602&h=461&rev=1423&ac=1&parent=1ySERHGBXbHeyCMRookq5UfTuFkzzU0ugtjvR3rF3deY
>
> Em qui, 25 de out de 2018 às 07:38, Rafael Novello <
> rafa.reis.novello@gmail.com> escreveu:
>
> > Hi Taka,
> >
> > I can't see the image! Maybe the Apache email list don't support
> > attachments.
> >
> > Best!
> > Rafael J. R. Novello
> >
> > Skype: rafael.novello
> > Blog: http://rafanovello.blogspot.com.br/
> >
> >
> > Em qua, 24 de out de 2018 às 21:14, Daniel Takabayashi <
> > daniel.takabayashi@gmail.com> escreveu:
> >
> > > Hi all,
> > >
> > > To try to give a little more details about this "new concept" that we
> > want
> > > to bring to the marvin toolbox, I did this simple architecture draw.
> > >
> > > [image: marvin-architecture-views (5).png]
> > >
> > > The general idea here is try to transform the toolbox something
> > > disconnected with the "language", something more agnostic. Also in this
> > new
> > > architecture we could use remote resource to process engines and make
> > easy
> > > the support for new languages.
> > >
> > > This "new toolbox" will be the only thing that a Marvin user must to
> > > install and also we could start to support multiples O.S (once the REPL
> > is
> > > a dummy application that only interprets and by pass commands).
> > >
> > > Regards,
> > > Taka
> > >
> > > Em qua, 24 de out de 2018 às 09:52, Rafael Novello <
> > > rafa.reis.novello@gmail.com> escreveu:
> > >
> > >> Alan,
> > >>
> > >> Yes! We are using the Docker SDK and it's possible to use the API to
> > >> automate, but it's a little bit harder than automate CLI calls.
> > >>
> > >> Atenciosamente,
> > >> Rafael J. R. Novello
> > >>
> > >> Skype: rafael.novello
> > >> Blog: http://rafanovello.blogspot.com.br/
> > >>
> > >>
> > >> Em qua, 24 de out de 2018 às 12:02, Alan Silva <ju...@apache.org>
> > >> escreveu:
> > >>
> > >> > Hi,
> > >> >
> > >> > One question here, I understand that we start to use with this PoC
> the
> > >> > Docker SDK API, right?
> > >> >
> > >> > Why not use the API to expose some endpoints to permit this kind of
> > >> > automation by devops?
> > >> >
> > >> > I think it is possible and it solves the CLI problem, right?
> > >> >
> > >> >
> > >> > On Tue, Oct 23, 2018 at 1:05 PM Rafael Novello <
> > >> > rafa.reis.novello@gmail.com>
> > >> > wrote:
> > >> >
> > >> > > Lucas,
> > >> > >
> > >> > > Sorry, I didn't understood your question bellow.
> > >> > >
> > >> > > "Would it make sense to use the same solution that we will use for
> > >> > having a
> > >> > > single-language REPL to have a single-language CLI?"
> > >> > >
> > >> > > For DevOps purposes, maybe this new toolbox concept is not ideal.
> I
> > >> think
> > >> > > we can keep the CLI inside the docker container but it will not
> easy
> > >> to
> > >> > > automate
> > >> > > jobs by this way.
> > >> > >
> > >> > > How to deal with this issue? Voting?
> > >> > > Atenciosamente,
> > >> > > Rafael J. R. Novello
> > >> > >
> > >> > > Skype: rafael.novello
> > >> > > Blog: http://rafanovello.blogspot.com.br/
> > >> > >
> > >> > >
> > >> > > Em sex, 19 de out de 2018 às 19:00, Lucas Bonatto Miguel <
> > >> > > lucasbm88@apache.org> escreveu:
> > >> > >
> > >> > > > Got it! Thanks for clarifying.
> > >> > > >
> > >> > > > Would it make sense to use the same solution that we will use
> for
> > >> > having
> > >> > > a
> > >> > > > single-language REPL to have a single-language CLI? My only
> > concern
> > >> > with
> > >> > > > killing the CLI is that you remove an essential feature for
> > DevOps.
> > >> > > >
> > >> > > > - Lucas
> > >> > > >
> > >> > > > On Fri, Oct 19, 2018 at 1:52 PM Rafael Novello <
> > >> > > > rafa.reis.novello@gmail.com>
> > >> > > > wrote:
> > >> > > >
> > >> > > > > Lucas,
> > >> > > > >
> > >> > > > > The idea is that REPL will substitute the actual CLI. It's
> > because
> > >> > with
> > >> > > > the
> > >> > > > > actual concept (using language specific CLI) we will need to
> > >> > > re-implement
> > >> > > > > the same features many times and probably each CLI will have a
> > >> > > different
> > >> > > > > behavior because some language specific restrictions and/or
> > >> > > limitations.
> > >> > > > >
> > >> > > > > With this new concept, all users will have the same experience
> > >> > > > interacting
> > >> > > > > with Marvin REPL and they will use the bests tools to develop
> > >> their
> > >> > > > engines
> > >> > > > > (Jupyter Notebook and/or Lab with different languages support
> or
> > >> even
> > >> > > > > Apache Zeppelin). All the interact will occur through REPL and
> > >> docker
> > >> > > > > protocol.
> > >> > > > >
> > >> > > > > Alan, Yes! As Lucas said, the concept is the same but we can
> use
> > >> > docker
> > >> > > > to
> > >> > > > > do the same job.
> > >> > > > >
> > >> > > > > Best!
> > >> > > > >
> > >> > > > > Em sex, 19 de out de 2018 às 00:39, Lucas Bonatto Miguel <
> > >> > > > > lucasbm88@apache.org> escreveu:
> > >> > > > >
> > >> > > > > > Thanks for the clarifications Rafael, so from what I
> > understood,
> > >> > > > Marvin's
> > >> > > > > > developers would not use the REPL to explore data and test
> > >> models,
> > >> > > i.e.
> > >> > > > > > develop the engine. Is the idea to build something more like
> > an
> > >> > > > > interactive
> > >> > > > > > CLI? I think an interactive CLI would be useful for the
> > >> developer
> > >> > > > > > experience, however, it's important to keep the unattended
> > >> > (current)
> > >> > > > CLI
> > >> > > > > in
> > >> > > > > > place for scenarios where users want to automate Marvin
> tasks.
> > >> > > > > >
> > >> > > > > > Alan, I believe the idea is still the same as you started,
> but
> > >> > using
> > >> > > > > > docker SDK now.
> > >> > > > > >
> > >> > > > > > - Lucas
> > >> > > > > >
> > >> > > > > > On Thu, Oct 18, 2018 at 1:29 PM Rafael Novello <
> > >> > > > > > rafa.reis.novello@gmail.com>
> > >> > > > > > wrote:
> > >> > > > > >
> > >> > > > > > > Hi Lucas!
> > >> > > > > > >
> > >> > > > > > > First of all +1 for REPL POCs to Apache!
> > >> > > > > > >
> > >> > > > > > > Let me help with some comments:
> > >> > > > > > >
> > >> > > > > > > 1 - We have tested NodeJS, Scala and Python and the
> easiest
> > >> one
> > >> > was
> > >> > > > > > Python.
> > >> > > > > > > We have found a small project [1] that have all features
> we
> > >> > desired
> > >> > > > for
> > >> > > > > > > REPL:
> > >> > > > > > > - Autocomplete
> > >> > > > > > > - Python commands disabled (the user have only the
> commands
> > we
> > >> > > > > provide).
> > >> > > > > > > The other languages REPL options don't have this feature.
> > >> > > > > > > - Easy to show output
> > >> > > > > > > - Etc
> > >> > > > > > > So, I think the language chosen will not be important here
> > >> > because
> > >> > > > the
> > >> > > > > > user
> > >> > > > > > > will only interact with the commands that we create.
> > >> > > > > > >
> > >> > > > > > > 2 - The "engine-generate" command will download a docker
> > image
> > >> > that
> > >> > > > we
> > >> > > > > > > create for that language and start a container with the
> > basic
> > >> > > project
> > >> > > > > > > structure for that language.
> > >> > > > > > >
> > >> > > > > > > 3 - The REPL client will use the "docker protocol" to run
> > >> > command,
> > >> > > > > > > start/stop services and etc inside the container and it
> will
> > >> > > receive
> > >> > > > > the
> > >> > > > > > > log stream to show. No, the REPL will no pass code
> snippets
> > >> for
> > >> > > > docker
> > >> > > > > > > container (I think it will not be necessary)
> > >> > > > > > >
> > >> > > > > > > 4 - Yep! Like I said on the first item.
> > >> > > > > > >
> > >> > > > > > > [1] - https://github.com/italorossi/ishell
> > >> > > > > > >
> > >> > > > > > > Let me know if there is any other question!
> > >> > > > > > > Best!
> > >> > > > > > > Rafael J. R. Novello
> > >> > > > > > >
> > >> > > > > > > Skype: rafael.novello
> > >> > > > > > > Blog: http://rafanovello.blogspot.com.br/
> > >> > > > > > >
> > >> > > > > > >
> > >> > > > > > > Em qui, 18 de out de 2018 às 17:00, Lucas Bonatto Miguel <
> > >> > > > > > > lucasbm88@apache.org> escreveu:
> > >> > > > > > >
> > >> > > > > > > > + 1 for migrating the REPL repo to Apache
> > >> > > > > > > >
> > >> > > > > > > > I have a few questions about the previous explanation:
> > >> > > > > > > >  1) The REPL itself, it would be an application in which
> > >> > > language?
> > >> > > > > > > Remember
> > >> > > > > > > > that the main idea is to allow the user to program on
> his
> > >> > > preferred
> > >> > > > > > > > language in the REPL.
> > >> > > > > > > >  2) Should the engine-generate command also generate a
> > >> docker
> > >> > > image
> > >> > > > > > with
> > >> > > > > > > > the user's application?
> > >> > > > > > > >  3) What type of communication would happen between the
> > REPL
> > >> > and
> > >> > > > the
> > >> > > > > > > engine
> > >> > > > > > > > via Docker SDK? Would the REPL pass snippets of code to
> be
> > >> > > executed
> > >> > > > > by
> > >> > > > > > > the
> > >> > > > > > > > docker container?
> > >> > > > > > > >  4) Have you considered code completion in the REPL?
> > >> > > > > > > >
> > >> > > > > > > >
> > >> > > > > > > > On Thu, Oct 18, 2018 at 7:53 AM Zhang Yifei <
> > >> > > > yifei.z.luiz@gmail.com>
> > >> > > > > > > > wrote:
> > >> > > > > > > >
> > >> > > > > > > > > Ok guys,
> > >> > > > > > > > >
> > >> > > > > > > > > The basic idea is to provide only one Toolbox for
> > multiple
> > >> > > > > languages.
> > >> > > > > > > > > We are looking for possibility to build a single
> Marvin
> > >> Repl,
> > >> > > > > > > > > instead of severals toolboxes with differentes
> > interfaces
> > >> or
> > >> > > > > > commands.
> > >> > > > > > > > >
> > >> > > > > > > > > In this case, the engine-generate command will
> download
> > >> and
> > >> > > > start a
> > >> > > > > > > > Docker
> > >> > > > > > > > > container with basic engine structure corresponding
> the
> > >> > choosed
> > >> > > > > > > language.
> > >> > > > > > > > > this means we don't need to build Toolboxes of
> > differents
> > >> > > > > languages,
> > >> > > > > > we
> > >> > > > > > > > > just
> > >> > > > > > > > > build the engine template of all languages that we
> want
> > to
> > >> > > > support
> > >> > > > > > and
> > >> > > > > > > > > provide it
> > >> > > > > > > > > as Docker containers
> > >> > > > > > > > >
> > >> > > > > > > > > We have started researches around our basic
> requirements
> > >> > like:
> > >> > > > > > > > > - Repl interface
> > >> > > > > > > > > - System communication
> > >> > > > > > > > > - Connection security
> > >> > > > > > > > > - Tool popularity
> > >> > > > > > > > > - Update complexity
> > >> > > > > > > > > - Languages support
> > >> > > > > > > > > - ......
> > >> > > > > > > > >
> > >> > > > > > > > > And we did some POC with code here:
> > >> > > > > > > > > https://github.com/marvin-ai/marvin-repl
> > >> > > > > > > > >
> > >> > > > > > > > > There is POC testing gRPC using Scala and Python,
> > >> > > > > > > > > Repl inteface and Docker SDK with NodeJS,
> > >> > > > > > > > > Repl interface and Docker SDK with Python.
> > >> > > > > > > > >
> > >> > > > > > > > > At this moment we prefer the Repl interface + Docker
> SDK
> > >> way,
> > >> > > > > because
> > >> > > > > > > > good
> > >> > > > > > > > > part of the requirements
> > >> > > > > > > > > will be guaranteed by Docker.
> > >> > > > > > > > >
> > >> > > > > > > > > With this informations, what do you think? Should we
> > >> submit
> > >> > all
> > >> > > > > this
> > >> > > > > > > POCs
> > >> > > > > > > > > to Apache Repo?
> > >> > > > > > > > > Please feel free to opine.
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > > Thats all, thanks!!!
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > > Em ter, 16 de out de 2018 às 18:55, Daniel
> Takabayashi <
> > >> > > > > > > > > daniel.takabayashi@gmail.com> escreveu:
> > >> > > > > > > > >
> > >> > > > > > > > > > Zhang,
> > >> > > > > > > > > >
> > >> > > > > > > > > > I think the best approach is give us a better
> > >> explanation
> > >> > > about
> > >> > > > > > this
> > >> > > > > > > > new
> > >> > > > > > > > > > feature and how this can help us to archive the
> > desired
> > >> > > feature
> > >> > > > > > > > (support
> > >> > > > > > > > > > multiple languages). Than if the majority agree
> than I
> > >> can
> > >> > do
> > >> > > > the
> > >> > > > > > > > merge,
> > >> > > > > > > > > > just like I did before, but in another branch.
> > >> > > > > > > > > >
> > >> > > > > > > > > > Let me know if makes sense, ok?
> > >> > > > > > > > > >
> > >> > > > > > > > > > Taka
> > >> > > > > > > > > >
> > >> > > > > > > > > > Em ter, 16 de out de 2018 às 14:00, Luciano Resende
> <
> > >> > > > > > > > > luckbr1975@gmail.com>
> > >> > > > > > > > > > escreveu:
> > >> > > > > > > > > >
> > >> > > > > > > > > > > So, what is the POC, is it a refactoring of the
> > >> existing
> > >> > > > repo?
> > >> > > > > Or
> > >> > > > > > > is
> > >> > > > > > > > > > > it a new rewrite of the repo?
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > Just asking as it might make sense to make it a
> > branch
> > >> > then
> > >> > > > > > > actually
> > >> > > > > > > > a
> > >> > > > > > > > > > > thing in parallel, as this will have an effect for
> > >> > > releases,
> > >> > > > > etc.
> > >> > > > > > > but
> > >> > > > > > > > > > > you guys know more here than I do.
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > Also, it's probably good to have a write up of the
> > >> main
> > >> > > > > direction
> > >> > > > > > > of
> > >> > > > > > > > > > > the design that can help people get familiar with
> > the
> > >> new
> > >> > > > > > approach.
> > >> > > > > > > > > > > On Tue, Oct 16, 2018 at 11:12 AM Zhang Yifei <
> > >> > > > > > > yifei.z.luiz@gmail.com
> > >> > > > > > > > >
> > >> > > > > > > > > > > wrote:
> > >> > > > > > > > > > > >
> > >> > > > > > > > > > > > Hey guys,
> > >> > > > > > > > > > > >
> > >> > > > > > > > > > > > We have reorganized the Poc repo, and want to
> > merge
> > >> it
> > >> > to
> > >> > > > our
> > >> > > > > > > > Apache
> > >> > > > > > > > > > > repo.
> > >> > > > > > > > > > > > Just making sure here before we do the merge,
> > >> because
> > >> > im
> > >> > > > not
> > >> > > > > > sure
> > >> > > > > > > > if
> > >> > > > > > > > > > > doing
> > >> > > > > > > > > > > > this will perserve the Git commit history.
> > >> > > > > > > > > > > > What we are planning to do is:
> > >> > > > > > > > > > > >
> > >> > > > > > > > > > > > git filter-branch --subdirectory-filter <git
> > >> > > > > origin_repository
> > >> > > > > > > > > > directory>
> > >> > > > > > > > > > > > -- --all
> > >> > > > > > > > > > > > mkdir POCs/
> > >> > > > > > > > > > > > git mv * POCs/
> > >> > > > > > > > > > > > git commit -m "colleted the data to move"
> > >> > > > > > > > > > > >
> > >> > > > > > > > > > > > git clone <git destination_repository_url> cd
> <git
> > >> > > > > > > > > > > > destination_repository_directory> git remote add
> > >> > > > > > > > > > origin_repository_branch
> > >> > > > > > > > > > > > <git origin_repository directory> git pull
> > >> > > > > > > origin_repository_branch
> > >> > > > > > > > > > > master
> > >> > > > > > > > > > > > --allow-unrelated-histories git remote rm
> > >> > > > > > > origin_repository_branch
> > >> > > > > > > > > > > >
> > >> > > > > > > > > > > >
> > >> > > > > > > > > > > > Thanks in advance !
> > >> > > > > > > > > > > >
> > >> > > > > > > > > > > >
> > >> > > > > --------------------------------------------------------------
> > >> > > > > > > > > > > > Zhang Yifei
> > >> > > > > > > > > > >
> > >> > > > > > > > > > >
> > >> > > > > > > > > > >
> > >> > > > > > > > > > > --
> > >> > > > > > > > > > > Luciano Resende
> > >> > > > > > > > > > > http://twitter.com/lresende1975
> > >> > > > > > > > > > > http://lresende.blogspot.com/
> > >> > > > > > > > > > >
> > >> > > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > >
> > >> > > > > > > > > --
> > >> > > > > > > > >
> > >> > --------------------------------------------------------------
> > >> > > > > > > > > Zhang Yifei
> > >> > > > > > > > >
> > >> > > > > > > >
> > >> > > > > > >
> > >> > > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> > >
> >
>

Re: POCs, Prototypes, experiments ...

Posted by Daniel Takabayashi <da...@gmail.com>.

https://docs.google.com/drawings/d/shajxIpLJHxxMbFgDXiPuhg/image?w=602&h=461&rev=1423&ac=1&parent=1ySERHGBXbHeyCMRookq5UfTuFkzzU0ugtjvR3rF3deY

Em qui, 25 de out de 2018 às 07:38, Rafael Novello <
rafa.reis.novello@gmail.com> escreveu:

> Hi Taka,
>
> I can't see the image! Maybe the Apache email list don't support
> attachments.
>
> Best!
> Rafael J. R. Novello
>
> Skype: rafael.novello
> Blog: http://rafanovello.blogspot.com.br/
>
>
> Em qua, 24 de out de 2018 às 21:14, Daniel Takabayashi <
> daniel.takabayashi@gmail.com> escreveu:
>
> > Hi all,
> >
> > To try to give a little more details about this "new concept" that we
> want
> > to bring to the marvin toolbox, I did this simple architecture draw.
> >
> > [image: marvin-architecture-views (5).png]
> >
> > The general idea here is try to transform the toolbox something
> > disconnected with the "language", something more agnostic. Also in this
> new
> > architecture we could use remote resource to process engines and make
> easy
> > the support for new languages.
> >
> > This "new toolbox" will be the only thing that a Marvin user must to
> > install and also we could start to support multiples O.S (once the REPL
> is
> > a dummy application that only interprets and by pass commands).
> >
> > Regards,
> > Taka
> >
> > Em qua, 24 de out de 2018 às 09:52, Rafael Novello <
> > rafa.reis.novello@gmail.com> escreveu:
> >
> >> Alan,
> >>
> >> Yes! We are using the Docker SDK and it's possible to use the API to
> >> automate, but it's a little bit harder than automate CLI calls.
> >>
> >> Atenciosamente,
> >> Rafael J. R. Novello
> >>
> >> Skype: rafael.novello
> >> Blog: http://rafanovello.blogspot.com.br/
> >>
> >>
> >> Em qua, 24 de out de 2018 às 12:02, Alan Silva <ju...@apache.org>
> >> escreveu:
> >>
> >> > Hi,
> >> >
> >> > One question here, I understand that we start to use with this PoC the
> >> > Docker SDK API, right?
> >> >
> >> > Why not use the API to expose some endpoints to permit this kind of
> >> > automation by devops?
> >> >
> >> > I think it is possible and it solves the CLI problem, right?
> >> >
> >> >
> >> > On Tue, Oct 23, 2018 at 1:05 PM Rafael Novello <
> >> > rafa.reis.novello@gmail.com>
> >> > wrote:
> >> >
> >> > > Lucas,
> >> > >
> >> > > Sorry, I didn't understood your question bellow.
> >> > >
> >> > > "Would it make sense to use the same solution that we will use for
> >> > having a
> >> > > single-language REPL to have a single-language CLI?"
> >> > >
> >> > > For DevOps purposes, maybe this new toolbox concept is not ideal. I
> >> think
> >> > > we can keep the CLI inside the docker container but it will not easy
> >> to
> >> > > automate
> >> > > jobs by this way.
> >> > >
> >> > > How to deal with this issue? Voting?
> >> > > Atenciosamente,
> >> > > Rafael J. R. Novello
> >> > >
> >> > > Skype: rafael.novello
> >> > > Blog: http://rafanovello.blogspot.com.br/
> >> > >
> >> > >
> >> > > Em sex, 19 de out de 2018 às 19:00, Lucas Bonatto Miguel <
> >> > > lucasbm88@apache.org> escreveu:
> >> > >
> >> > > > Got it! Thanks for clarifying.
> >> > > >
> >> > > > Would it make sense to use the same solution that we will use for
> >> > having
> >> > > a
> >> > > > single-language REPL to have a single-language CLI? My only
> concern
> >> > with
> >> > > > killing the CLI is that you remove an essential feature for
> DevOps.
> >> > > >
> >> > > > - Lucas
> >> > > >
> >> > > > On Fri, Oct 19, 2018 at 1:52 PM Rafael Novello <
> >> > > > rafa.reis.novello@gmail.com>
> >> > > > wrote:
> >> > > >
> >> > > > > Lucas,
> >> > > > >
> >> > > > > The idea is that REPL will substitute the actual CLI. It's
> because
> >> > with
> >> > > > the
> >> > > > > actual concept (using language specific CLI) we will need to
> >> > > re-implement
> >> > > > > the same features many times and probably each CLI will have a
> >> > > different
> >> > > > > behavior because some language specific restrictions and/or
> >> > > limitations.
> >> > > > >
> >> > > > > With this new concept, all users will have the same experience
> >> > > > interacting
> >> > > > > with Marvin REPL and they will use the bests tools to develop
> >> their
> >> > > > engines
> >> > > > > (Jupyter Notebook and/or Lab with different languages support or
> >> even
> >> > > > > Apache Zeppelin). All the interact will occur through REPL and
> >> docker
> >> > > > > protocol.
> >> > > > >
> >> > > > > Alan, Yes! As Lucas said, the concept is the same but we can use
> >> > docker
> >> > > > to
> >> > > > > do the same job.
> >> > > > >
> >> > > > > Best!
> >> > > > >
> >> > > > > Em sex, 19 de out de 2018 às 00:39, Lucas Bonatto Miguel <
> >> > > > > lucasbm88@apache.org> escreveu:
> >> > > > >
> >> > > > > > Thanks for the clarifications Rafael, so from what I
> understood,
> >> > > > Marvin's
> >> > > > > > developers would not use the REPL to explore data and test
> >> models,
> >> > > i.e.
> >> > > > > > develop the engine. Is the idea to build something more like
> an
> >> > > > > interactive
> >> > > > > > CLI? I think an interactive CLI would be useful for the
> >> developer
> >> > > > > > experience, however, it's important to keep the unattended
> >> > (current)
> >> > > > CLI
> >> > > > > in
> >> > > > > > place for scenarios where users want to automate Marvin tasks.
> >> > > > > >
> >> > > > > > Alan, I believe the idea is still the same as you started, but
> >> > using
> >> > > > > > docker SDK now.
> >> > > > > >
> >> > > > > > - Lucas
> >> > > > > >
> >> > > > > > On Thu, Oct 18, 2018 at 1:29 PM Rafael Novello <
> >> > > > > > rafa.reis.novello@gmail.com>
> >> > > > > > wrote:
> >> > > > > >
> >> > > > > > > Hi Lucas!
> >> > > > > > >
> >> > > > > > > First of all +1 for REPL POCs to Apache!
> >> > > > > > >
> >> > > > > > > Let me help with some comments:
> >> > > > > > >
> >> > > > > > > 1 - We have tested NodeJS, Scala and Python and the easiest
> >> one
> >> > was
> >> > > > > > Python.
> >> > > > > > > We have found a small project [1] that have all features we
> >> > desired
> >> > > > for
> >> > > > > > > REPL:
> >> > > > > > > - Autocomplete
> >> > > > > > > - Python commands disabled (the user have only the commands
> we
> >> > > > > provide).
> >> > > > > > > The other languages REPL options don't have this feature.
> >> > > > > > > - Easy to show output
> >> > > > > > > - Etc
> >> > > > > > > So, I think the language chosen will not be important here
> >> > because
> >> > > > the
> >> > > > > > user
> >> > > > > > > will only interact with the commands that we create.
> >> > > > > > >
> >> > > > > > > 2 - The "engine-generate" command will download a docker
> image
> >> > that
> >> > > > we
> >> > > > > > > create for that language and start a container with the
> basic
> >> > > project
> >> > > > > > > structure for that language.
> >> > > > > > >
> >> > > > > > > 3 - The REPL client will use the "docker protocol" to run
> >> > command,
> >> > > > > > > start/stop services and etc inside the container and it will
> >> > > receive
> >> > > > > the
> >> > > > > > > log stream to show. No, the REPL will no pass code snippets
> >> for
> >> > > > docker
> >> > > > > > > container (I think it will not be necessary)
> >> > > > > > >
> >> > > > > > > 4 - Yep! Like I said on the first item.
> >> > > > > > >
> >> > > > > > > [1] - https://github.com/italorossi/ishell
> >> > > > > > >
> >> > > > > > > Let me know if there is any other question!
> >> > > > > > > Best!
> >> > > > > > > Rafael J. R. Novello
> >> > > > > > >
> >> > > > > > > Skype: rafael.novello
> >> > > > > > > Blog: http://rafanovello.blogspot.com.br/
> >> > > > > > >
> >> > > > > > >
> >> > > > > > > Em qui, 18 de out de 2018 às 17:00, Lucas Bonatto Miguel <
> >> > > > > > > lucasbm88@apache.org> escreveu:
> >> > > > > > >
> >> > > > > > > > + 1 for migrating the REPL repo to Apache
> >> > > > > > > >
> >> > > > > > > > I have a few questions about the previous explanation:
> >> > > > > > > >  1) The REPL itself, it would be an application in which
> >> > > language?
> >> > > > > > > Remember
> >> > > > > > > > that the main idea is to allow the user to program on his
> >> > > preferred
> >> > > > > > > > language in the REPL.
> >> > > > > > > >  2) Should the engine-generate command also generate a
> >> docker
> >> > > image
> >> > > > > > with
> >> > > > > > > > the user's application?
> >> > > > > > > >  3) What type of communication would happen between the
> REPL
> >> > and
> >> > > > the
> >> > > > > > > engine
> >> > > > > > > > via Docker SDK? Would the REPL pass snippets of code to be
> >> > > executed
> >> > > > > by
> >> > > > > > > the
> >> > > > > > > > docker container?
> >> > > > > > > >  4) Have you considered code completion in the REPL?
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > > > On Thu, Oct 18, 2018 at 7:53 AM Zhang Yifei <
> >> > > > yifei.z.luiz@gmail.com>
> >> > > > > > > > wrote:
> >> > > > > > > >
> >> > > > > > > > > Ok guys,
> >> > > > > > > > >
> >> > > > > > > > > The basic idea is to provide only one Toolbox for
> multiple
> >> > > > > languages.
> >> > > > > > > > > We are looking for possibility to build a single Marvin
> >> Repl,
> >> > > > > > > > > instead of severals toolboxes with differentes
> interfaces
> >> or
> >> > > > > > commands.
> >> > > > > > > > >
> >> > > > > > > > > In this case, the engine-generate command will download
> >> and
> >> > > > start a
> >> > > > > > > > Docker
> >> > > > > > > > > container with basic engine structure corresponding the
> >> > choosed
> >> > > > > > > language.
> >> > > > > > > > > this means we don't need to build Toolboxes of
> differents
> >> > > > > languages,
> >> > > > > > we
> >> > > > > > > > > just
> >> > > > > > > > > build the engine template of all languages that we want
> to
> >> > > > support
> >> > > > > > and
> >> > > > > > > > > provide it
> >> > > > > > > > > as Docker containers
> >> > > > > > > > >
> >> > > > > > > > > We have started researches around our basic requirements
> >> > like:
> >> > > > > > > > > - Repl interface
> >> > > > > > > > > - System communication
> >> > > > > > > > > - Connection security
> >> > > > > > > > > - Tool popularity
> >> > > > > > > > > - Update complexity
> >> > > > > > > > > - Languages support
> >> > > > > > > > > - ......
> >> > > > > > > > >
> >> > > > > > > > > And we did some POC with code here:
> >> > > > > > > > > https://github.com/marvin-ai/marvin-repl
> >> > > > > > > > >
> >> > > > > > > > > There is POC testing gRPC using Scala and Python,
> >> > > > > > > > > Repl inteface and Docker SDK with NodeJS,
> >> > > > > > > > > Repl interface and Docker SDK with Python.
> >> > > > > > > > >
> >> > > > > > > > > At this moment we prefer the Repl interface + Docker SDK
> >> way,
> >> > > > > because
> >> > > > > > > > good
> >> > > > > > > > > part of the requirements
> >> > > > > > > > > will be guaranteed by Docker.
> >> > > > > > > > >
> >> > > > > > > > > With this informations, what do you think? Should we
> >> submit
> >> > all
> >> > > > > this
> >> > > > > > > POCs
> >> > > > > > > > > to Apache Repo?
> >> > > > > > > > > Please feel free to opine.
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > > Thats all, thanks!!!
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > > Em ter, 16 de out de 2018 às 18:55, Daniel Takabayashi <
> >> > > > > > > > > daniel.takabayashi@gmail.com> escreveu:
> >> > > > > > > > >
> >> > > > > > > > > > Zhang,
> >> > > > > > > > > >
> >> > > > > > > > > > I think the best approach is give us a better
> >> explanation
> >> > > about
> >> > > > > > this
> >> > > > > > > > new
> >> > > > > > > > > > feature and how this can help us to archive the
> desired
> >> > > feature
> >> > > > > > > > (support
> >> > > > > > > > > > multiple languages). Than if the majority agree than I
> >> can
> >> > do
> >> > > > the
> >> > > > > > > > merge,
> >> > > > > > > > > > just like I did before, but in another branch.
> >> > > > > > > > > >
> >> > > > > > > > > > Let me know if makes sense, ok?
> >> > > > > > > > > >
> >> > > > > > > > > > Taka
> >> > > > > > > > > >
> >> > > > > > > > > > Em ter, 16 de out de 2018 às 14:00, Luciano Resende <
> >> > > > > > > > > luckbr1975@gmail.com>
> >> > > > > > > > > > escreveu:
> >> > > > > > > > > >
> >> > > > > > > > > > > So, what is the POC, is it a refactoring of the
> >> existing
> >> > > > repo?
> >> > > > > Or
> >> > > > > > > is
> >> > > > > > > > > > > it a new rewrite of the repo?
> >> > > > > > > > > > >
> >> > > > > > > > > > > Just asking as it might make sense to make it a
> branch
> >> > then
> >> > > > > > > actually
> >> > > > > > > > a
> >> > > > > > > > > > > thing in parallel, as this will have an effect for
> >> > > releases,
> >> > > > > etc.
> >> > > > > > > but
> >> > > > > > > > > > > you guys know more here than I do.
> >> > > > > > > > > > >
> >> > > > > > > > > > > Also, it's probably good to have a write up of the
> >> main
> >> > > > > direction
> >> > > > > > > of
> >> > > > > > > > > > > the design that can help people get familiar with
> the
> >> new
> >> > > > > > approach.
> >> > > > > > > > > > > On Tue, Oct 16, 2018 at 11:12 AM Zhang Yifei <
> >> > > > > > > yifei.z.luiz@gmail.com
> >> > > > > > > > >
> >> > > > > > > > > > > wrote:
> >> > > > > > > > > > > >
> >> > > > > > > > > > > > Hey guys,
> >> > > > > > > > > > > >
> >> > > > > > > > > > > > We have reorganized the Poc repo, and want to
> merge
> >> it
> >> > to
> >> > > > our
> >> > > > > > > > Apache
> >> > > > > > > > > > > repo.
> >> > > > > > > > > > > > Just making sure here before we do the merge,
> >> because
> >> > im
> >> > > > not
> >> > > > > > sure
> >> > > > > > > > if
> >> > > > > > > > > > > doing
> >> > > > > > > > > > > > this will perserve the Git commit history.
> >> > > > > > > > > > > > What we are planning to do is:
> >> > > > > > > > > > > >
> >> > > > > > > > > > > > git filter-branch --subdirectory-filter <git
> >> > > > > origin_repository
> >> > > > > > > > > > directory>
> >> > > > > > > > > > > > -- --all
> >> > > > > > > > > > > > mkdir POCs/
> >> > > > > > > > > > > > git mv * POCs/
> >> > > > > > > > > > > > git commit -m "colleted the data to move"
> >> > > > > > > > > > > >
> >> > > > > > > > > > > > git clone <git destination_repository_url> cd <git
> >> > > > > > > > > > > > destination_repository_directory> git remote add
> >> > > > > > > > > > origin_repository_branch
> >> > > > > > > > > > > > <git origin_repository directory> git pull
> >> > > > > > > origin_repository_branch
> >> > > > > > > > > > > master
> >> > > > > > > > > > > > --allow-unrelated-histories git remote rm
> >> > > > > > > origin_repository_branch
> >> > > > > > > > > > > >
> >> > > > > > > > > > > >
> >> > > > > > > > > > > > Thanks in advance !
> >> > > > > > > > > > > >
> >> > > > > > > > > > > >
> >> > > > > --------------------------------------------------------------
> >> > > > > > > > > > > > Zhang Yifei
> >> > > > > > > > > > >
> >> > > > > > > > > > >
> >> > > > > > > > > > >
> >> > > > > > > > > > > --
> >> > > > > > > > > > > Luciano Resende
> >> > > > > > > > > > > http://twitter.com/lresende1975
> >> > > > > > > > > > > http://lresende.blogspot.com/
> >> > > > > > > > > > >
> >> > > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > >
> >> > > > > > > > > --
> >> > > > > > > > >
> >> > --------------------------------------------------------------
> >> > > > > > > > > Zhang Yifei
> >> > > > > > > > >
> >> > > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> >
>

Re: POCs, Prototypes, experiments ...

Posted by Rafael Novello <ra...@gmail.com>.

Hi Taka,

I can't see the image! Maybe the Apache email list don't support
attachments.

Best!
Rafael J. R. Novello

Skype: rafael.novello
Blog: http://rafanovello.blogspot.com.br/


Em qua, 24 de out de 2018 às 21:14, Daniel Takabayashi <
daniel.takabayashi@gmail.com> escreveu:

> Hi all,
>
> To try to give a little more details about this "new concept" that we want
> to bring to the marvin toolbox, I did this simple architecture draw.
>
> [image: marvin-architecture-views (5).png]
>
> The general idea here is try to transform the toolbox something
> disconnected with the "language", something more agnostic. Also in this new
> architecture we could use remote resource to process engines and make easy
> the support for new languages.
>
> This "new toolbox" will be the only thing that a Marvin user must to
> install and also we could start to support multiples O.S (once the REPL is
> a dummy application that only interprets and by pass commands).
>
> Regards,
> Taka
>
> Em qua, 24 de out de 2018 às 09:52, Rafael Novello <
> rafa.reis.novello@gmail.com> escreveu:
>
>> Alan,
>>
>> Yes! We are using the Docker SDK and it's possible to use the API to
>> automate, but it's a little bit harder than automate CLI calls.
>>
>> Atenciosamente,
>> Rafael J. R. Novello
>>
>> Skype: rafael.novello
>> Blog: http://rafanovello.blogspot.com.br/
>>
>>
>> Em qua, 24 de out de 2018 às 12:02, Alan Silva <ju...@apache.org>
>> escreveu:
>>
>> > Hi,
>> >
>> > One question here, I understand that we start to use with this PoC the
>> > Docker SDK API, right?
>> >
>> > Why not use the API to expose some endpoints to permit this kind of
>> > automation by devops?
>> >
>> > I think it is possible and it solves the CLI problem, right?
>> >
>> >
>> > On Tue, Oct 23, 2018 at 1:05 PM Rafael Novello <
>> > rafa.reis.novello@gmail.com>
>> > wrote:
>> >
>> > > Lucas,
>> > >
>> > > Sorry, I didn't understood your question bellow.
>> > >
>> > > "Would it make sense to use the same solution that we will use for
>> > having a
>> > > single-language REPL to have a single-language CLI?"
>> > >
>> > > For DevOps purposes, maybe this new toolbox concept is not ideal. I
>> think
>> > > we can keep the CLI inside the docker container but it will not easy
>> to
>> > > automate
>> > > jobs by this way.
>> > >
>> > > How to deal with this issue? Voting?
>> > > Atenciosamente,
>> > > Rafael J. R. Novello
>> > >
>> > > Skype: rafael.novello
>> > > Blog: http://rafanovello.blogspot.com.br/
>> > >
>> > >
>> > > Em sex, 19 de out de 2018 às 19:00, Lucas Bonatto Miguel <
>> > > lucasbm88@apache.org> escreveu:
>> > >
>> > > > Got it! Thanks for clarifying.
>> > > >
>> > > > Would it make sense to use the same solution that we will use for
>> > having
>> > > a
>> > > > single-language REPL to have a single-language CLI? My only concern
>> > with
>> > > > killing the CLI is that you remove an essential feature for DevOps.
>> > > >
>> > > > - Lucas
>> > > >
>> > > > On Fri, Oct 19, 2018 at 1:52 PM Rafael Novello <
>> > > > rafa.reis.novello@gmail.com>
>> > > > wrote:
>> > > >
>> > > > > Lucas,
>> > > > >
>> > > > > The idea is that REPL will substitute the actual CLI. It's because
>> > with
>> > > > the
>> > > > > actual concept (using language specific CLI) we will need to
>> > > re-implement
>> > > > > the same features many times and probably each CLI will have a
>> > > different
>> > > > > behavior because some language specific restrictions and/or
>> > > limitations.
>> > > > >
>> > > > > With this new concept, all users will have the same experience
>> > > > interacting
>> > > > > with Marvin REPL and they will use the bests tools to develop
>> their
>> > > > engines
>> > > > > (Jupyter Notebook and/or Lab with different languages support or
>> even
>> > > > > Apache Zeppelin). All the interact will occur through REPL and
>> docker
>> > > > > protocol.
>> > > > >
>> > > > > Alan, Yes! As Lucas said, the concept is the same but we can use
>> > docker
>> > > > to
>> > > > > do the same job.
>> > > > >
>> > > > > Best!
>> > > > >
>> > > > > Em sex, 19 de out de 2018 às 00:39, Lucas Bonatto Miguel <
>> > > > > lucasbm88@apache.org> escreveu:
>> > > > >
>> > > > > > Thanks for the clarifications Rafael, so from what I understood,
>> > > > Marvin's
>> > > > > > developers would not use the REPL to explore data and test
>> models,
>> > > i.e.
>> > > > > > develop the engine. Is the idea to build something more like an
>> > > > > interactive
>> > > > > > CLI? I think an interactive CLI would be useful for the
>> developer
>> > > > > > experience, however, it's important to keep the unattended
>> > (current)
>> > > > CLI
>> > > > > in
>> > > > > > place for scenarios where users want to automate Marvin tasks.
>> > > > > >
>> > > > > > Alan, I believe the idea is still the same as you started, but
>> > using
>> > > > > > docker SDK now.
>> > > > > >
>> > > > > > - Lucas
>> > > > > >
>> > > > > > On Thu, Oct 18, 2018 at 1:29 PM Rafael Novello <
>> > > > > > rafa.reis.novello@gmail.com>
>> > > > > > wrote:
>> > > > > >
>> > > > > > > Hi Lucas!
>> > > > > > >
>> > > > > > > First of all +1 for REPL POCs to Apache!
>> > > > > > >
>> > > > > > > Let me help with some comments:
>> > > > > > >
>> > > > > > > 1 - We have tested NodeJS, Scala and Python and the easiest
>> one
>> > was
>> > > > > > Python.
>> > > > > > > We have found a small project [1] that have all features we
>> > desired
>> > > > for
>> > > > > > > REPL:
>> > > > > > > - Autocomplete
>> > > > > > > - Python commands disabled (the user have only the commands we
>> > > > > provide).
>> > > > > > > The other languages REPL options don't have this feature.
>> > > > > > > - Easy to show output
>> > > > > > > - Etc
>> > > > > > > So, I think the language chosen will not be important here
>> > because
>> > > > the
>> > > > > > user
>> > > > > > > will only interact with the commands that we create.
>> > > > > > >
>> > > > > > > 2 - The "engine-generate" command will download a docker image
>> > that
>> > > > we
>> > > > > > > create for that language and start a container with the basic
>> > > project
>> > > > > > > structure for that language.
>> > > > > > >
>> > > > > > > 3 - The REPL client will use the "docker protocol" to run
>> > command,
>> > > > > > > start/stop services and etc inside the container and it will
>> > > receive
>> > > > > the
>> > > > > > > log stream to show. No, the REPL will no pass code snippets
>> for
>> > > > docker
>> > > > > > > container (I think it will not be necessary)
>> > > > > > >
>> > > > > > > 4 - Yep! Like I said on the first item.
>> > > > > > >
>> > > > > > > [1] - https://github.com/italorossi/ishell
>> > > > > > >
>> > > > > > > Let me know if there is any other question!
>> > > > > > > Best!
>> > > > > > > Rafael J. R. Novello
>> > > > > > >
>> > > > > > > Skype: rafael.novello
>> > > > > > > Blog: http://rafanovello.blogspot.com.br/
>> > > > > > >
>> > > > > > >
>> > > > > > > Em qui, 18 de out de 2018 às 17:00, Lucas Bonatto Miguel <
>> > > > > > > lucasbm88@apache.org> escreveu:
>> > > > > > >
>> > > > > > > > + 1 for migrating the REPL repo to Apache
>> > > > > > > >
>> > > > > > > > I have a few questions about the previous explanation:
>> > > > > > > >  1) The REPL itself, it would be an application in which
>> > > language?
>> > > > > > > Remember
>> > > > > > > > that the main idea is to allow the user to program on his
>> > > preferred
>> > > > > > > > language in the REPL.
>> > > > > > > >  2) Should the engine-generate command also generate a
>> docker
>> > > image
>> > > > > > with
>> > > > > > > > the user's application?
>> > > > > > > >  3) What type of communication would happen between the REPL
>> > and
>> > > > the
>> > > > > > > engine
>> > > > > > > > via Docker SDK? Would the REPL pass snippets of code to be
>> > > executed
>> > > > > by
>> > > > > > > the
>> > > > > > > > docker container?
>> > > > > > > >  4) Have you considered code completion in the REPL?
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > On Thu, Oct 18, 2018 at 7:53 AM Zhang Yifei <
>> > > > yifei.z.luiz@gmail.com>
>> > > > > > > > wrote:
>> > > > > > > >
>> > > > > > > > > Ok guys,
>> > > > > > > > >
>> > > > > > > > > The basic idea is to provide only one Toolbox for multiple
>> > > > > languages.
>> > > > > > > > > We are looking for possibility to build a single Marvin
>> Repl,
>> > > > > > > > > instead of severals toolboxes with differentes interfaces
>> or
>> > > > > > commands.
>> > > > > > > > >
>> > > > > > > > > In this case, the engine-generate command will download
>> and
>> > > > start a
>> > > > > > > > Docker
>> > > > > > > > > container with basic engine structure corresponding the
>> > choosed
>> > > > > > > language.
>> > > > > > > > > this means we don't need to build Toolboxes of differents
>> > > > > languages,
>> > > > > > we
>> > > > > > > > > just
>> > > > > > > > > build the engine template of all languages that we want to
>> > > > support
>> > > > > > and
>> > > > > > > > > provide it
>> > > > > > > > > as Docker containers
>> > > > > > > > >
>> > > > > > > > > We have started researches around our basic requirements
>> > like:
>> > > > > > > > > - Repl interface
>> > > > > > > > > - System communication
>> > > > > > > > > - Connection security
>> > > > > > > > > - Tool popularity
>> > > > > > > > > - Update complexity
>> > > > > > > > > - Languages support
>> > > > > > > > > - ......
>> > > > > > > > >
>> > > > > > > > > And we did some POC with code here:
>> > > > > > > > > https://github.com/marvin-ai/marvin-repl
>> > > > > > > > >
>> > > > > > > > > There is POC testing gRPC using Scala and Python,
>> > > > > > > > > Repl inteface and Docker SDK with NodeJS,
>> > > > > > > > > Repl interface and Docker SDK with Python.
>> > > > > > > > >
>> > > > > > > > > At this moment we prefer the Repl interface + Docker SDK
>> way,
>> > > > > because
>> > > > > > > > good
>> > > > > > > > > part of the requirements
>> > > > > > > > > will be guaranteed by Docker.
>> > > > > > > > >
>> > > > > > > > > With this informations, what do you think? Should we
>> submit
>> > all
>> > > > > this
>> > > > > > > POCs
>> > > > > > > > > to Apache Repo?
>> > > > > > > > > Please feel free to opine.
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > > Thats all, thanks!!!
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > > Em ter, 16 de out de 2018 às 18:55, Daniel Takabayashi <
>> > > > > > > > > daniel.takabayashi@gmail.com> escreveu:
>> > > > > > > > >
>> > > > > > > > > > Zhang,
>> > > > > > > > > >
>> > > > > > > > > > I think the best approach is give us a better
>> explanation
>> > > about
>> > > > > > this
>> > > > > > > > new
>> > > > > > > > > > feature and how this can help us to archive the desired
>> > > feature
>> > > > > > > > (support
>> > > > > > > > > > multiple languages). Than if the majority agree than I
>> can
>> > do
>> > > > the
>> > > > > > > > merge,
>> > > > > > > > > > just like I did before, but in another branch.
>> > > > > > > > > >
>> > > > > > > > > > Let me know if makes sense, ok?
>> > > > > > > > > >
>> > > > > > > > > > Taka
>> > > > > > > > > >
>> > > > > > > > > > Em ter, 16 de out de 2018 às 14:00, Luciano Resende <
>> > > > > > > > > luckbr1975@gmail.com>
>> > > > > > > > > > escreveu:
>> > > > > > > > > >
>> > > > > > > > > > > So, what is the POC, is it a refactoring of the
>> existing
>> > > > repo?
>> > > > > Or
>> > > > > > > is
>> > > > > > > > > > > it a new rewrite of the repo?
>> > > > > > > > > > >
>> > > > > > > > > > > Just asking as it might make sense to make it a branch
>> > then
>> > > > > > > actually
>> > > > > > > > a
>> > > > > > > > > > > thing in parallel, as this will have an effect for
>> > > releases,
>> > > > > etc.
>> > > > > > > but
>> > > > > > > > > > > you guys know more here than I do.
>> > > > > > > > > > >
>> > > > > > > > > > > Also, it's probably good to have a write up of the
>> main
>> > > > > direction
>> > > > > > > of
>> > > > > > > > > > > the design that can help people get familiar with the
>> new
>> > > > > > approach.
>> > > > > > > > > > > On Tue, Oct 16, 2018 at 11:12 AM Zhang Yifei <
>> > > > > > > yifei.z.luiz@gmail.com
>> > > > > > > > >
>> > > > > > > > > > > wrote:
>> > > > > > > > > > > >
>> > > > > > > > > > > > Hey guys,
>> > > > > > > > > > > >
>> > > > > > > > > > > > We have reorganized the Poc repo, and want to merge
>> it
>> > to
>> > > > our
>> > > > > > > > Apache
>> > > > > > > > > > > repo.
>> > > > > > > > > > > > Just making sure here before we do the merge,
>> because
>> > im
>> > > > not
>> > > > > > sure
>> > > > > > > > if
>> > > > > > > > > > > doing
>> > > > > > > > > > > > this will perserve the Git commit history.
>> > > > > > > > > > > > What we are planning to do is:
>> > > > > > > > > > > >
>> > > > > > > > > > > > git filter-branch --subdirectory-filter <git
>> > > > > origin_repository
>> > > > > > > > > > directory>
>> > > > > > > > > > > > -- --all
>> > > > > > > > > > > > mkdir POCs/
>> > > > > > > > > > > > git mv * POCs/
>> > > > > > > > > > > > git commit -m "colleted the data to move"
>> > > > > > > > > > > >
>> > > > > > > > > > > > git clone <git destination_repository_url> cd <git
>> > > > > > > > > > > > destination_repository_directory> git remote add
>> > > > > > > > > > origin_repository_branch
>> > > > > > > > > > > > <git origin_repository directory> git pull
>> > > > > > > origin_repository_branch
>> > > > > > > > > > > master
>> > > > > > > > > > > > --allow-unrelated-histories git remote rm
>> > > > > > > origin_repository_branch
>> > > > > > > > > > > >
>> > > > > > > > > > > >
>> > > > > > > > > > > > Thanks in advance !
>> > > > > > > > > > > >
>> > > > > > > > > > > >
>> > > > > --------------------------------------------------------------
>> > > > > > > > > > > > Zhang Yifei
>> > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > > > --
>> > > > > > > > > > > Luciano Resende
>> > > > > > > > > > > http://twitter.com/lresende1975
>> > > > > > > > > > > http://lresende.blogspot.com/
>> > > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > > --
>> > > > > > > > >
>> > --------------------------------------------------------------
>> > > > > > > > > Zhang Yifei
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
>

Re: POCs, Prototypes, experiments ...

Posted by Daniel Takabayashi <da...@gmail.com>.

Hi all,

To try to give a little more details about this "new concept" that we want
to bring to the marvin toolbox, I did this simple architecture draw.

[image: marvin-architecture-views (5).png]

The general idea here is try to transform the toolbox something
disconnected with the "language", something more agnostic. Also in this new
architecture we could use remote resource to process engines and make easy
the support for new languages.

This "new toolbox" will be the only thing that a Marvin user must to
install and also we could start to support multiples O.S (once the REPL is
a dummy application that only interprets and by pass commands).

Regards,
Taka

Em qua, 24 de out de 2018 às 09:52, Rafael Novello <
rafa.reis.novello@gmail.com> escreveu:

> Alan,
>
> Yes! We are using the Docker SDK and it's possible to use the API to
> automate, but it's a little bit harder than automate CLI calls.
>
> Atenciosamente,
> Rafael J. R. Novello
>
> Skype: rafael.novello
> Blog: http://rafanovello.blogspot.com.br/
>
>
> Em qua, 24 de out de 2018 às 12:02, Alan Silva <ju...@apache.org>
> escreveu:
>
> > Hi,
> >
> > One question here, I understand that we start to use with this PoC the
> > Docker SDK API, right?
> >
> > Why not use the API to expose some endpoints to permit this kind of
> > automation by devops?
> >
> > I think it is possible and it solves the CLI problem, right?
> >
> >
> > On Tue, Oct 23, 2018 at 1:05 PM Rafael Novello <
> > rafa.reis.novello@gmail.com>
> > wrote:
> >
> > > Lucas,
> > >
> > > Sorry, I didn't understood your question bellow.
> > >
> > > "Would it make sense to use the same solution that we will use for
> > having a
> > > single-language REPL to have a single-language CLI?"
> > >
> > > For DevOps purposes, maybe this new toolbox concept is not ideal. I
> think
> > > we can keep the CLI inside the docker container but it will not easy to
> > > automate
> > > jobs by this way.
> > >
> > > How to deal with this issue? Voting?
> > > Atenciosamente,
> > > Rafael J. R. Novello
> > >
> > > Skype: rafael.novello
> > > Blog: http://rafanovello.blogspot.com.br/
> > >
> > >
> > > Em sex, 19 de out de 2018 às 19:00, Lucas Bonatto Miguel <
> > > lucasbm88@apache.org> escreveu:
> > >
> > > > Got it! Thanks for clarifying.
> > > >
> > > > Would it make sense to use the same solution that we will use for
> > having
> > > a
> > > > single-language REPL to have a single-language CLI? My only concern
> > with
> > > > killing the CLI is that you remove an essential feature for DevOps.
> > > >
> > > > - Lucas
> > > >
> > > > On Fri, Oct 19, 2018 at 1:52 PM Rafael Novello <
> > > > rafa.reis.novello@gmail.com>
> > > > wrote:
> > > >
> > > > > Lucas,
> > > > >
> > > > > The idea is that REPL will substitute the actual CLI. It's because
> > with
> > > > the
> > > > > actual concept (using language specific CLI) we will need to
> > > re-implement
> > > > > the same features many times and probably each CLI will have a
> > > different
> > > > > behavior because some language specific restrictions and/or
> > > limitations.
> > > > >
> > > > > With this new concept, all users will have the same experience
> > > > interacting
> > > > > with Marvin REPL and they will use the bests tools to develop their
> > > > engines
> > > > > (Jupyter Notebook and/or Lab with different languages support or
> even
> > > > > Apache Zeppelin). All the interact will occur through REPL and
> docker
> > > > > protocol.
> > > > >
> > > > > Alan, Yes! As Lucas said, the concept is the same but we can use
> > docker
> > > > to
> > > > > do the same job.
> > > > >
> > > > > Best!
> > > > >
> > > > > Em sex, 19 de out de 2018 às 00:39, Lucas Bonatto Miguel <
> > > > > lucasbm88@apache.org> escreveu:
> > > > >
> > > > > > Thanks for the clarifications Rafael, so from what I understood,
> > > > Marvin's
> > > > > > developers would not use the REPL to explore data and test
> models,
> > > i.e.
> > > > > > develop the engine. Is the idea to build something more like an
> > > > > interactive
> > > > > > CLI? I think an interactive CLI would be useful for the developer
> > > > > > experience, however, it's important to keep the unattended
> > (current)
> > > > CLI
> > > > > in
> > > > > > place for scenarios where users want to automate Marvin tasks.
> > > > > >
> > > > > > Alan, I believe the idea is still the same as you started, but
> > using
> > > > > > docker SDK now.
> > > > > >
> > > > > > - Lucas
> > > > > >
> > > > > > On Thu, Oct 18, 2018 at 1:29 PM Rafael Novello <
> > > > > > rafa.reis.novello@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Hi Lucas!
> > > > > > >
> > > > > > > First of all +1 for REPL POCs to Apache!
> > > > > > >
> > > > > > > Let me help with some comments:
> > > > > > >
> > > > > > > 1 - We have tested NodeJS, Scala and Python and the easiest one
> > was
> > > > > > Python.
> > > > > > > We have found a small project [1] that have all features we
> > desired
> > > > for
> > > > > > > REPL:
> > > > > > > - Autocomplete
> > > > > > > - Python commands disabled (the user have only the commands we
> > > > > provide).
> > > > > > > The other languages REPL options don't have this feature.
> > > > > > > - Easy to show output
> > > > > > > - Etc
> > > > > > > So, I think the language chosen will not be important here
> > because
> > > > the
> > > > > > user
> > > > > > > will only interact with the commands that we create.
> > > > > > >
> > > > > > > 2 - The "engine-generate" command will download a docker image
> > that
> > > > we
> > > > > > > create for that language and start a container with the basic
> > > project
> > > > > > > structure for that language.
> > > > > > >
> > > > > > > 3 - The REPL client will use the "docker protocol" to run
> > command,
> > > > > > > start/stop services and etc inside the container and it will
> > > receive
> > > > > the
> > > > > > > log stream to show. No, the REPL will no pass code snippets for
> > > > docker
> > > > > > > container (I think it will not be necessary)
> > > > > > >
> > > > > > > 4 - Yep! Like I said on the first item.
> > > > > > >
> > > > > > > [1] - https://github.com/italorossi/ishell
> > > > > > >
> > > > > > > Let me know if there is any other question!
> > > > > > > Best!
> > > > > > > Rafael J. R. Novello
> > > > > > >
> > > > > > > Skype: rafael.novello
> > > > > > > Blog: http://rafanovello.blogspot.com.br/
> > > > > > >
> > > > > > >
> > > > > > > Em qui, 18 de out de 2018 às 17:00, Lucas Bonatto Miguel <
> > > > > > > lucasbm88@apache.org> escreveu:
> > > > > > >
> > > > > > > > + 1 for migrating the REPL repo to Apache
> > > > > > > >
> > > > > > > > I have a few questions about the previous explanation:
> > > > > > > >  1) The REPL itself, it would be an application in which
> > > language?
> > > > > > > Remember
> > > > > > > > that the main idea is to allow the user to program on his
> > > preferred
> > > > > > > > language in the REPL.
> > > > > > > >  2) Should the engine-generate command also generate a docker
> > > image
> > > > > > with
> > > > > > > > the user's application?
> > > > > > > >  3) What type of communication would happen between the REPL
> > and
> > > > the
> > > > > > > engine
> > > > > > > > via Docker SDK? Would the REPL pass snippets of code to be
> > > executed
> > > > > by
> > > > > > > the
> > > > > > > > docker container?
> > > > > > > >  4) Have you considered code completion in the REPL?
> > > > > > > >
> > > > > > > >
> > > > > > > > On Thu, Oct 18, 2018 at 7:53 AM Zhang Yifei <
> > > > yifei.z.luiz@gmail.com>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Ok guys,
> > > > > > > > >
> > > > > > > > > The basic idea is to provide only one Toolbox for multiple
> > > > > languages.
> > > > > > > > > We are looking for possibility to build a single Marvin
> Repl,
> > > > > > > > > instead of severals toolboxes with differentes interfaces
> or
> > > > > > commands.
> > > > > > > > >
> > > > > > > > > In this case, the engine-generate command will download and
> > > > start a
> > > > > > > > Docker
> > > > > > > > > container with basic engine structure corresponding the
> > choosed
> > > > > > > language.
> > > > > > > > > this means we don't need to build Toolboxes of differents
> > > > > languages,
> > > > > > we
> > > > > > > > > just
> > > > > > > > > build the engine template of all languages that we want to
> > > > support
> > > > > > and
> > > > > > > > > provide it
> > > > > > > > > as Docker containers
> > > > > > > > >
> > > > > > > > > We have started researches around our basic requirements
> > like:
> > > > > > > > > - Repl interface
> > > > > > > > > - System communication
> > > > > > > > > - Connection security
> > > > > > > > > - Tool popularity
> > > > > > > > > - Update complexity
> > > > > > > > > - Languages support
> > > > > > > > > - ......
> > > > > > > > >
> > > > > > > > > And we did some POC with code here:
> > > > > > > > > https://github.com/marvin-ai/marvin-repl
> > > > > > > > >
> > > > > > > > > There is POC testing gRPC using Scala and Python,
> > > > > > > > > Repl inteface and Docker SDK with NodeJS,
> > > > > > > > > Repl interface and Docker SDK with Python.
> > > > > > > > >
> > > > > > > > > At this moment we prefer the Repl interface + Docker SDK
> way,
> > > > > because
> > > > > > > > good
> > > > > > > > > part of the requirements
> > > > > > > > > will be guaranteed by Docker.
> > > > > > > > >
> > > > > > > > > With this informations, what do you think? Should we submit
> > all
> > > > > this
> > > > > > > POCs
> > > > > > > > > to Apache Repo?
> > > > > > > > > Please feel free to opine.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Thats all, thanks!!!
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Em ter, 16 de out de 2018 às 18:55, Daniel Takabayashi <
> > > > > > > > > daniel.takabayashi@gmail.com> escreveu:
> > > > > > > > >
> > > > > > > > > > Zhang,
> > > > > > > > > >
> > > > > > > > > > I think the best approach is give us a better explanation
> > > about
> > > > > > this
> > > > > > > > new
> > > > > > > > > > feature and how this can help us to archive the desired
> > > feature
> > > > > > > > (support
> > > > > > > > > > multiple languages). Than if the majority agree than I
> can
> > do
> > > > the
> > > > > > > > merge,
> > > > > > > > > > just like I did before, but in another branch.
> > > > > > > > > >
> > > > > > > > > > Let me know if makes sense, ok?
> > > > > > > > > >
> > > > > > > > > > Taka
> > > > > > > > > >
> > > > > > > > > > Em ter, 16 de out de 2018 às 14:00, Luciano Resende <
> > > > > > > > > luckbr1975@gmail.com>
> > > > > > > > > > escreveu:
> > > > > > > > > >
> > > > > > > > > > > So, what is the POC, is it a refactoring of the
> existing
> > > > repo?
> > > > > Or
> > > > > > > is
> > > > > > > > > > > it a new rewrite of the repo?
> > > > > > > > > > >
> > > > > > > > > > > Just asking as it might make sense to make it a branch
> > then
> > > > > > > actually
> > > > > > > > a
> > > > > > > > > > > thing in parallel, as this will have an effect for
> > > releases,
> > > > > etc.
> > > > > > > but
> > > > > > > > > > > you guys know more here than I do.
> > > > > > > > > > >
> > > > > > > > > > > Also, it's probably good to have a write up of the main
> > > > > direction
> > > > > > > of
> > > > > > > > > > > the design that can help people get familiar with the
> new
> > > > > > approach.
> > > > > > > > > > > On Tue, Oct 16, 2018 at 11:12 AM Zhang Yifei <
> > > > > > > yifei.z.luiz@gmail.com
> > > > > > > > >
> > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > Hey guys,
> > > > > > > > > > > >
> > > > > > > > > > > > We have reorganized the Poc repo, and want to merge
> it
> > to
> > > > our
> > > > > > > > Apache
> > > > > > > > > > > repo.
> > > > > > > > > > > > Just making sure here before we do the merge, because
> > im
> > > > not
> > > > > > sure
> > > > > > > > if
> > > > > > > > > > > doing
> > > > > > > > > > > > this will perserve the Git commit history.
> > > > > > > > > > > > What we are planning to do is:
> > > > > > > > > > > >
> > > > > > > > > > > > git filter-branch --subdirectory-filter <git
> > > > > origin_repository
> > > > > > > > > > directory>
> > > > > > > > > > > > -- --all
> > > > > > > > > > > > mkdir POCs/
> > > > > > > > > > > > git mv * POCs/
> > > > > > > > > > > > git commit -m "colleted the data to move"
> > > > > > > > > > > >
> > > > > > > > > > > > git clone <git destination_repository_url> cd <git
> > > > > > > > > > > > destination_repository_directory> git remote add
> > > > > > > > > > origin_repository_branch
> > > > > > > > > > > > <git origin_repository directory> git pull
> > > > > > > origin_repository_branch
> > > > > > > > > > > master
> > > > > > > > > > > > --allow-unrelated-histories git remote rm
> > > > > > > origin_repository_branch
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks in advance !
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > --------------------------------------------------------------
> > > > > > > > > > > > Zhang Yifei
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > --
> > > > > > > > > > > Luciano Resende
> > > > > > > > > > > http://twitter.com/lresende1975
> > > > > > > > > > > http://lresende.blogspot.com/
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > --
> > > > > > > > >
> > --------------------------------------------------------------
> > > > > > > > > Zhang Yifei
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: POCs, Prototypes, experiments ...

Posted by Rafael Novello <ra...@gmail.com>.

Alan,

Yes! We are using the Docker SDK and it's possible to use the API to
automate, but it's a little bit harder than automate CLI calls.

Atenciosamente,
Rafael J. R. Novello

Skype: rafael.novello
Blog: http://rafanovello.blogspot.com.br/


Em qua, 24 de out de 2018 às 12:02, Alan Silva <ju...@apache.org> escreveu:

> Hi,
>
> One question here, I understand that we start to use with this PoC the
> Docker SDK API, right?
>
> Why not use the API to expose some endpoints to permit this kind of
> automation by devops?
>
> I think it is possible and it solves the CLI problem, right?
>
>
> On Tue, Oct 23, 2018 at 1:05 PM Rafael Novello <
> rafa.reis.novello@gmail.com>
> wrote:
>
> > Lucas,
> >
> > Sorry, I didn't understood your question bellow.
> >
> > "Would it make sense to use the same solution that we will use for
> having a
> > single-language REPL to have a single-language CLI?"
> >
> > For DevOps purposes, maybe this new toolbox concept is not ideal. I think
> > we can keep the CLI inside the docker container but it will not easy to
> > automate
> > jobs by this way.
> >
> > How to deal with this issue? Voting?
> > Atenciosamente,
> > Rafael J. R. Novello
> >
> > Skype: rafael.novello
> > Blog: http://rafanovello.blogspot.com.br/
> >
> >
> > Em sex, 19 de out de 2018 às 19:00, Lucas Bonatto Miguel <
> > lucasbm88@apache.org> escreveu:
> >
> > > Got it! Thanks for clarifying.
> > >
> > > Would it make sense to use the same solution that we will use for
> having
> > a
> > > single-language REPL to have a single-language CLI? My only concern
> with
> > > killing the CLI is that you remove an essential feature for DevOps.
> > >
> > > - Lucas
> > >
> > > On Fri, Oct 19, 2018 at 1:52 PM Rafael Novello <
> > > rafa.reis.novello@gmail.com>
> > > wrote:
> > >
> > > > Lucas,
> > > >
> > > > The idea is that REPL will substitute the actual CLI. It's because
> with
> > > the
> > > > actual concept (using language specific CLI) we will need to
> > re-implement
> > > > the same features many times and probably each CLI will have a
> > different
> > > > behavior because some language specific restrictions and/or
> > limitations.
> > > >
> > > > With this new concept, all users will have the same experience
> > > interacting
> > > > with Marvin REPL and they will use the bests tools to develop their
> > > engines
> > > > (Jupyter Notebook and/or Lab with different languages support or even
> > > > Apache Zeppelin). All the interact will occur through REPL and docker
> > > > protocol.
> > > >
> > > > Alan, Yes! As Lucas said, the concept is the same but we can use
> docker
> > > to
> > > > do the same job.
> > > >
> > > > Best!
> > > >
> > > > Em sex, 19 de out de 2018 às 00:39, Lucas Bonatto Miguel <
> > > > lucasbm88@apache.org> escreveu:
> > > >
> > > > > Thanks for the clarifications Rafael, so from what I understood,
> > > Marvin's
> > > > > developers would not use the REPL to explore data and test models,
> > i.e.
> > > > > develop the engine. Is the idea to build something more like an
> > > > interactive
> > > > > CLI? I think an interactive CLI would be useful for the developer
> > > > > experience, however, it's important to keep the unattended
> (current)
> > > CLI
> > > > in
> > > > > place for scenarios where users want to automate Marvin tasks.
> > > > >
> > > > > Alan, I believe the idea is still the same as you started, but
> using
> > > > > docker SDK now.
> > > > >
> > > > > - Lucas
> > > > >
> > > > > On Thu, Oct 18, 2018 at 1:29 PM Rafael Novello <
> > > > > rafa.reis.novello@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Hi Lucas!
> > > > > >
> > > > > > First of all +1 for REPL POCs to Apache!
> > > > > >
> > > > > > Let me help with some comments:
> > > > > >
> > > > > > 1 - We have tested NodeJS, Scala and Python and the easiest one
> was
> > > > > Python.
> > > > > > We have found a small project [1] that have all features we
> desired
> > > for
> > > > > > REPL:
> > > > > > - Autocomplete
> > > > > > - Python commands disabled (the user have only the commands we
> > > > provide).
> > > > > > The other languages REPL options don't have this feature.
> > > > > > - Easy to show output
> > > > > > - Etc
> > > > > > So, I think the language chosen will not be important here
> because
> > > the
> > > > > user
> > > > > > will only interact with the commands that we create.
> > > > > >
> > > > > > 2 - The "engine-generate" command will download a docker image
> that
> > > we
> > > > > > create for that language and start a container with the basic
> > project
> > > > > > structure for that language.
> > > > > >
> > > > > > 3 - The REPL client will use the "docker protocol" to run
> command,
> > > > > > start/stop services and etc inside the container and it will
> > receive
> > > > the
> > > > > > log stream to show. No, the REPL will no pass code snippets for
> > > docker
> > > > > > container (I think it will not be necessary)
> > > > > >
> > > > > > 4 - Yep! Like I said on the first item.
> > > > > >
> > > > > > [1] - https://github.com/italorossi/ishell
> > > > > >
> > > > > > Let me know if there is any other question!
> > > > > > Best!
> > > > > > Rafael J. R. Novello
> > > > > >
> > > > > > Skype: rafael.novello
> > > > > > Blog: http://rafanovello.blogspot.com.br/
> > > > > >
> > > > > >
> > > > > > Em qui, 18 de out de 2018 às 17:00, Lucas Bonatto Miguel <
> > > > > > lucasbm88@apache.org> escreveu:
> > > > > >
> > > > > > > + 1 for migrating the REPL repo to Apache
> > > > > > >
> > > > > > > I have a few questions about the previous explanation:
> > > > > > >  1) The REPL itself, it would be an application in which
> > language?
> > > > > > Remember
> > > > > > > that the main idea is to allow the user to program on his
> > preferred
> > > > > > > language in the REPL.
> > > > > > >  2) Should the engine-generate command also generate a docker
> > image
> > > > > with
> > > > > > > the user's application?
> > > > > > >  3) What type of communication would happen between the REPL
> and
> > > the
> > > > > > engine
> > > > > > > via Docker SDK? Would the REPL pass snippets of code to be
> > executed
> > > > by
> > > > > > the
> > > > > > > docker container?
> > > > > > >  4) Have you considered code completion in the REPL?
> > > > > > >
> > > > > > >
> > > > > > > On Thu, Oct 18, 2018 at 7:53 AM Zhang Yifei <
> > > yifei.z.luiz@gmail.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Ok guys,
> > > > > > > >
> > > > > > > > The basic idea is to provide only one Toolbox for multiple
> > > > languages.
> > > > > > > > We are looking for possibility to build a single Marvin Repl,
> > > > > > > > instead of severals toolboxes with differentes interfaces or
> > > > > commands.
> > > > > > > >
> > > > > > > > In this case, the engine-generate command will download and
> > > start a
> > > > > > > Docker
> > > > > > > > container with basic engine structure corresponding the
> choosed
> > > > > > language.
> > > > > > > > this means we don't need to build Toolboxes of differents
> > > > languages,
> > > > > we
> > > > > > > > just
> > > > > > > > build the engine template of all languages that we want to
> > > support
> > > > > and
> > > > > > > > provide it
> > > > > > > > as Docker containers
> > > > > > > >
> > > > > > > > We have started researches around our basic requirements
> like:
> > > > > > > > - Repl interface
> > > > > > > > - System communication
> > > > > > > > - Connection security
> > > > > > > > - Tool popularity
> > > > > > > > - Update complexity
> > > > > > > > - Languages support
> > > > > > > > - ......
> > > > > > > >
> > > > > > > > And we did some POC with code here:
> > > > > > > > https://github.com/marvin-ai/marvin-repl
> > > > > > > >
> > > > > > > > There is POC testing gRPC using Scala and Python,
> > > > > > > > Repl inteface and Docker SDK with NodeJS,
> > > > > > > > Repl interface and Docker SDK with Python.
> > > > > > > >
> > > > > > > > At this moment we prefer the Repl interface + Docker SDK way,
> > > > because
> > > > > > > good
> > > > > > > > part of the requirements
> > > > > > > > will be guaranteed by Docker.
> > > > > > > >
> > > > > > > > With this informations, what do you think? Should we submit
> all
> > > > this
> > > > > > POCs
> > > > > > > > to Apache Repo?
> > > > > > > > Please feel free to opine.
> > > > > > > >
> > > > > > > >
> > > > > > > > Thats all, thanks!!!
> > > > > > > >
> > > > > > > >
> > > > > > > > Em ter, 16 de out de 2018 às 18:55, Daniel Takabayashi <
> > > > > > > > daniel.takabayashi@gmail.com> escreveu:
> > > > > > > >
> > > > > > > > > Zhang,
> > > > > > > > >
> > > > > > > > > I think the best approach is give us a better explanation
> > about
> > > > > this
> > > > > > > new
> > > > > > > > > feature and how this can help us to archive the desired
> > feature
> > > > > > > (support
> > > > > > > > > multiple languages). Than if the majority agree than I can
> do
> > > the
> > > > > > > merge,
> > > > > > > > > just like I did before, but in another branch.
> > > > > > > > >
> > > > > > > > > Let me know if makes sense, ok?
> > > > > > > > >
> > > > > > > > > Taka
> > > > > > > > >
> > > > > > > > > Em ter, 16 de out de 2018 às 14:00, Luciano Resende <
> > > > > > > > luckbr1975@gmail.com>
> > > > > > > > > escreveu:
> > > > > > > > >
> > > > > > > > > > So, what is the POC, is it a refactoring of the existing
> > > repo?
> > > > Or
> > > > > > is
> > > > > > > > > > it a new rewrite of the repo?
> > > > > > > > > >
> > > > > > > > > > Just asking as it might make sense to make it a branch
> then
> > > > > > actually
> > > > > > > a
> > > > > > > > > > thing in parallel, as this will have an effect for
> > releases,
> > > > etc.
> > > > > > but
> > > > > > > > > > you guys know more here than I do.
> > > > > > > > > >
> > > > > > > > > > Also, it's probably good to have a write up of the main
> > > > direction
> > > > > > of
> > > > > > > > > > the design that can help people get familiar with the new
> > > > > approach.
> > > > > > > > > > On Tue, Oct 16, 2018 at 11:12 AM Zhang Yifei <
> > > > > > yifei.z.luiz@gmail.com
> > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > Hey guys,
> > > > > > > > > > >
> > > > > > > > > > > We have reorganized the Poc repo, and want to merge it
> to
> > > our
> > > > > > > Apache
> > > > > > > > > > repo.
> > > > > > > > > > > Just making sure here before we do the merge, because
> im
> > > not
> > > > > sure
> > > > > > > if
> > > > > > > > > > doing
> > > > > > > > > > > this will perserve the Git commit history.
> > > > > > > > > > > What we are planning to do is:
> > > > > > > > > > >
> > > > > > > > > > > git filter-branch --subdirectory-filter <git
> > > > origin_repository
> > > > > > > > > directory>
> > > > > > > > > > > -- --all
> > > > > > > > > > > mkdir POCs/
> > > > > > > > > > > git mv * POCs/
> > > > > > > > > > > git commit -m "colleted the data to move"
> > > > > > > > > > >
> > > > > > > > > > > git clone <git destination_repository_url> cd <git
> > > > > > > > > > > destination_repository_directory> git remote add
> > > > > > > > > origin_repository_branch
> > > > > > > > > > > <git origin_repository directory> git pull
> > > > > > origin_repository_branch
> > > > > > > > > > master
> > > > > > > > > > > --allow-unrelated-histories git remote rm
> > > > > > origin_repository_branch
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Thanks in advance !
> > > > > > > > > > >
> > > > > > > > > > >
> > > > --------------------------------------------------------------
> > > > > > > > > > > Zhang Yifei
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > --
> > > > > > > > > > Luciano Resende
> > > > > > > > > > http://twitter.com/lresende1975
> > > > > > > > > > http://lresende.blogspot.com/
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > >
> --------------------------------------------------------------
> > > > > > > > Zhang Yifei
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: POCs, Prototypes, experiments ...

Posted by Alan Silva <ju...@apache.org>.

Hi,

One question here, I understand that we start to use with this PoC the
Docker SDK API, right?

Why not use the API to expose some endpoints to permit this kind of
automation by devops?

I think it is possible and it solves the CLI problem, right?


On Tue, Oct 23, 2018 at 1:05 PM Rafael Novello <ra...@gmail.com>
wrote:

> Lucas,
>
> Sorry, I didn't understood your question bellow.
>
> "Would it make sense to use the same solution that we will use for having a
> single-language REPL to have a single-language CLI?"
>
> For DevOps purposes, maybe this new toolbox concept is not ideal. I think
> we can keep the CLI inside the docker container but it will not easy to
> automate
> jobs by this way.
>
> How to deal with this issue? Voting?
> Atenciosamente,
> Rafael J. R. Novello
>
> Skype: rafael.novello
> Blog: http://rafanovello.blogspot.com.br/
>
>
> Em sex, 19 de out de 2018 às 19:00, Lucas Bonatto Miguel <
> lucasbm88@apache.org> escreveu:
>
> > Got it! Thanks for clarifying.
> >
> > Would it make sense to use the same solution that we will use for having
> a
> > single-language REPL to have a single-language CLI? My only concern with
> > killing the CLI is that you remove an essential feature for DevOps.
> >
> > - Lucas
> >
> > On Fri, Oct 19, 2018 at 1:52 PM Rafael Novello <
> > rafa.reis.novello@gmail.com>
> > wrote:
> >
> > > Lucas,
> > >
> > > The idea is that REPL will substitute the actual CLI. It's because with
> > the
> > > actual concept (using language specific CLI) we will need to
> re-implement
> > > the same features many times and probably each CLI will have a
> different
> > > behavior because some language specific restrictions and/or
> limitations.
> > >
> > > With this new concept, all users will have the same experience
> > interacting
> > > with Marvin REPL and they will use the bests tools to develop their
> > engines
> > > (Jupyter Notebook and/or Lab with different languages support or even
> > > Apache Zeppelin). All the interact will occur through REPL and docker
> > > protocol.
> > >
> > > Alan, Yes! As Lucas said, the concept is the same but we can use docker
> > to
> > > do the same job.
> > >
> > > Best!
> > >
> > > Em sex, 19 de out de 2018 às 00:39, Lucas Bonatto Miguel <
> > > lucasbm88@apache.org> escreveu:
> > >
> > > > Thanks for the clarifications Rafael, so from what I understood,
> > Marvin's
> > > > developers would not use the REPL to explore data and test models,
> i.e.
> > > > develop the engine. Is the idea to build something more like an
> > > interactive
> > > > CLI? I think an interactive CLI would be useful for the developer
> > > > experience, however, it's important to keep the unattended (current)
> > CLI
> > > in
> > > > place for scenarios where users want to automate Marvin tasks.
> > > >
> > > > Alan, I believe the idea is still the same as you started, but using
> > > > docker SDK now.
> > > >
> > > > - Lucas
> > > >
> > > > On Thu, Oct 18, 2018 at 1:29 PM Rafael Novello <
> > > > rafa.reis.novello@gmail.com>
> > > > wrote:
> > > >
> > > > > Hi Lucas!
> > > > >
> > > > > First of all +1 for REPL POCs to Apache!
> > > > >
> > > > > Let me help with some comments:
> > > > >
> > > > > 1 - We have tested NodeJS, Scala and Python and the easiest one was
> > > > Python.
> > > > > We have found a small project [1] that have all features we desired
> > for
> > > > > REPL:
> > > > > - Autocomplete
> > > > > - Python commands disabled (the user have only the commands we
> > > provide).
> > > > > The other languages REPL options don't have this feature.
> > > > > - Easy to show output
> > > > > - Etc
> > > > > So, I think the language chosen will not be important here because
> > the
> > > > user
> > > > > will only interact with the commands that we create.
> > > > >
> > > > > 2 - The "engine-generate" command will download a docker image that
> > we
> > > > > create for that language and start a container with the basic
> project
> > > > > structure for that language.
> > > > >
> > > > > 3 - The REPL client will use the "docker protocol" to run command,
> > > > > start/stop services and etc inside the container and it will
> receive
> > > the
> > > > > log stream to show. No, the REPL will no pass code snippets for
> > docker
> > > > > container (I think it will not be necessary)
> > > > >
> > > > > 4 - Yep! Like I said on the first item.
> > > > >
> > > > > [1] - https://github.com/italorossi/ishell
> > > > >
> > > > > Let me know if there is any other question!
> > > > > Best!
> > > > > Rafael J. R. Novello
> > > > >
> > > > > Skype: rafael.novello
> > > > > Blog: http://rafanovello.blogspot.com.br/
> > > > >
> > > > >
> > > > > Em qui, 18 de out de 2018 às 17:00, Lucas Bonatto Miguel <
> > > > > lucasbm88@apache.org> escreveu:
> > > > >
> > > > > > + 1 for migrating the REPL repo to Apache
> > > > > >
> > > > > > I have a few questions about the previous explanation:
> > > > > >  1) The REPL itself, it would be an application in which
> language?
> > > > > Remember
> > > > > > that the main idea is to allow the user to program on his
> preferred
> > > > > > language in the REPL.
> > > > > >  2) Should the engine-generate command also generate a docker
> image
> > > > with
> > > > > > the user's application?
> > > > > >  3) What type of communication would happen between the REPL and
> > the
> > > > > engine
> > > > > > via Docker SDK? Would the REPL pass snippets of code to be
> executed
> > > by
> > > > > the
> > > > > > docker container?
> > > > > >  4) Have you considered code completion in the REPL?
> > > > > >
> > > > > >
> > > > > > On Thu, Oct 18, 2018 at 7:53 AM Zhang Yifei <
> > yifei.z.luiz@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Ok guys,
> > > > > > >
> > > > > > > The basic idea is to provide only one Toolbox for multiple
> > > languages.
> > > > > > > We are looking for possibility to build a single Marvin Repl,
> > > > > > > instead of severals toolboxes with differentes interfaces or
> > > > commands.
> > > > > > >
> > > > > > > In this case, the engine-generate command will download and
> > start a
> > > > > > Docker
> > > > > > > container with basic engine structure corresponding the choosed
> > > > > language.
> > > > > > > this means we don't need to build Toolboxes of differents
> > > languages,
> > > > we
> > > > > > > just
> > > > > > > build the engine template of all languages that we want to
> > support
> > > > and
> > > > > > > provide it
> > > > > > > as Docker containers
> > > > > > >
> > > > > > > We have started researches around our basic requirements like:
> > > > > > > - Repl interface
> > > > > > > - System communication
> > > > > > > - Connection security
> > > > > > > - Tool popularity
> > > > > > > - Update complexity
> > > > > > > - Languages support
> > > > > > > - ......
> > > > > > >
> > > > > > > And we did some POC with code here:
> > > > > > > https://github.com/marvin-ai/marvin-repl
> > > > > > >
> > > > > > > There is POC testing gRPC using Scala and Python,
> > > > > > > Repl inteface and Docker SDK with NodeJS,
> > > > > > > Repl interface and Docker SDK with Python.
> > > > > > >
> > > > > > > At this moment we prefer the Repl interface + Docker SDK way,
> > > because
> > > > > > good
> > > > > > > part of the requirements
> > > > > > > will be guaranteed by Docker.
> > > > > > >
> > > > > > > With this informations, what do you think? Should we submit all
> > > this
> > > > > POCs
> > > > > > > to Apache Repo?
> > > > > > > Please feel free to opine.
> > > > > > >
> > > > > > >
> > > > > > > Thats all, thanks!!!
> > > > > > >
> > > > > > >
> > > > > > > Em ter, 16 de out de 2018 às 18:55, Daniel Takabayashi <
> > > > > > > daniel.takabayashi@gmail.com> escreveu:
> > > > > > >
> > > > > > > > Zhang,
> > > > > > > >
> > > > > > > > I think the best approach is give us a better explanation
> about
> > > > this
> > > > > > new
> > > > > > > > feature and how this can help us to archive the desired
> feature
> > > > > > (support
> > > > > > > > multiple languages). Than if the majority agree than I can do
> > the
> > > > > > merge,
> > > > > > > > just like I did before, but in another branch.
> > > > > > > >
> > > > > > > > Let me know if makes sense, ok?
> > > > > > > >
> > > > > > > > Taka
> > > > > > > >
> > > > > > > > Em ter, 16 de out de 2018 às 14:00, Luciano Resende <
> > > > > > > luckbr1975@gmail.com>
> > > > > > > > escreveu:
> > > > > > > >
> > > > > > > > > So, what is the POC, is it a refactoring of the existing
> > repo?
> > > Or
> > > > > is
> > > > > > > > > it a new rewrite of the repo?
> > > > > > > > >
> > > > > > > > > Just asking as it might make sense to make it a branch then
> > > > > actually
> > > > > > a
> > > > > > > > > thing in parallel, as this will have an effect for
> releases,
> > > etc.
> > > > > but
> > > > > > > > > you guys know more here than I do.
> > > > > > > > >
> > > > > > > > > Also, it's probably good to have a write up of the main
> > > direction
> > > > > of
> > > > > > > > > the design that can help people get familiar with the new
> > > > approach.
> > > > > > > > > On Tue, Oct 16, 2018 at 11:12 AM Zhang Yifei <
> > > > > yifei.z.luiz@gmail.com
> > > > > > >
> > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > Hey guys,
> > > > > > > > > >
> > > > > > > > > > We have reorganized the Poc repo, and want to merge it to
> > our
> > > > > > Apache
> > > > > > > > > repo.
> > > > > > > > > > Just making sure here before we do the merge, because im
> > not
> > > > sure
> > > > > > if
> > > > > > > > > doing
> > > > > > > > > > this will perserve the Git commit history.
> > > > > > > > > > What we are planning to do is:
> > > > > > > > > >
> > > > > > > > > > git filter-branch --subdirectory-filter <git
> > > origin_repository
> > > > > > > > directory>
> > > > > > > > > > -- --all
> > > > > > > > > > mkdir POCs/
> > > > > > > > > > git mv * POCs/
> > > > > > > > > > git commit -m "colleted the data to move"
> > > > > > > > > >
> > > > > > > > > > git clone <git destination_repository_url> cd <git
> > > > > > > > > > destination_repository_directory> git remote add
> > > > > > > > origin_repository_branch
> > > > > > > > > > <git origin_repository directory> git pull
> > > > > origin_repository_branch
> > > > > > > > > master
> > > > > > > > > > --allow-unrelated-histories git remote rm
> > > > > origin_repository_branch
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Thanks in advance !
> > > > > > > > > >
> > > > > > > > > >
> > > --------------------------------------------------------------
> > > > > > > > > > Zhang Yifei
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > --
> > > > > > > > > Luciano Resende
> > > > > > > > > http://twitter.com/lresende1975
> > > > > > > > > http://lresende.blogspot.com/
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > --------------------------------------------------------------
> > > > > > > Zhang Yifei
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: POCs, Prototypes, experiments ...

Posted by Rafael Novello <ra...@gmail.com>.

Lucas,

Sorry, I didn't understood your question bellow.

"Would it make sense to use the same solution that we will use for having a
single-language REPL to have a single-language CLI?"

For DevOps purposes, maybe this new toolbox concept is not ideal. I think
we can keep the CLI inside the docker container but it will not easy to
automate
jobs by this way.

How to deal with this issue? Voting?
Atenciosamente,
Rafael J. R. Novello

Skype: rafael.novello
Blog: http://rafanovello.blogspot.com.br/


Em sex, 19 de out de 2018 às 19:00, Lucas Bonatto Miguel <
lucasbm88@apache.org> escreveu:

> Got it! Thanks for clarifying.
>
> Would it make sense to use the same solution that we will use for having a
> single-language REPL to have a single-language CLI? My only concern with
> killing the CLI is that you remove an essential feature for DevOps.
>
> - Lucas
>
> On Fri, Oct 19, 2018 at 1:52 PM Rafael Novello <
> rafa.reis.novello@gmail.com>
> wrote:
>
> > Lucas,
> >
> > The idea is that REPL will substitute the actual CLI. It's because with
> the
> > actual concept (using language specific CLI) we will need to re-implement
> > the same features many times and probably each CLI will have a different
> > behavior because some language specific restrictions and/or limitations.
> >
> > With this new concept, all users will have the same experience
> interacting
> > with Marvin REPL and they will use the bests tools to develop their
> engines
> > (Jupyter Notebook and/or Lab with different languages support or even
> > Apache Zeppelin). All the interact will occur through REPL and docker
> > protocol.
> >
> > Alan, Yes! As Lucas said, the concept is the same but we can use docker
> to
> > do the same job.
> >
> > Best!
> >
> > Em sex, 19 de out de 2018 às 00:39, Lucas Bonatto Miguel <
> > lucasbm88@apache.org> escreveu:
> >
> > > Thanks for the clarifications Rafael, so from what I understood,
> Marvin's
> > > developers would not use the REPL to explore data and test models, i.e.
> > > develop the engine. Is the idea to build something more like an
> > interactive
> > > CLI? I think an interactive CLI would be useful for the developer
> > > experience, however, it's important to keep the unattended (current)
> CLI
> > in
> > > place for scenarios where users want to automate Marvin tasks.
> > >
> > > Alan, I believe the idea is still the same as you started, but using
> > > docker SDK now.
> > >
> > > - Lucas
> > >
> > > On Thu, Oct 18, 2018 at 1:29 PM Rafael Novello <
> > > rafa.reis.novello@gmail.com>
> > > wrote:
> > >
> > > > Hi Lucas!
> > > >
> > > > First of all +1 for REPL POCs to Apache!
> > > >
> > > > Let me help with some comments:
> > > >
> > > > 1 - We have tested NodeJS, Scala and Python and the easiest one was
> > > Python.
> > > > We have found a small project [1] that have all features we desired
> for
> > > > REPL:
> > > > - Autocomplete
> > > > - Python commands disabled (the user have only the commands we
> > provide).
> > > > The other languages REPL options don't have this feature.
> > > > - Easy to show output
> > > > - Etc
> > > > So, I think the language chosen will not be important here because
> the
> > > user
> > > > will only interact with the commands that we create.
> > > >
> > > > 2 - The "engine-generate" command will download a docker image that
> we
> > > > create for that language and start a container with the basic project
> > > > structure for that language.
> > > >
> > > > 3 - The REPL client will use the "docker protocol" to run command,
> > > > start/stop services and etc inside the container and it will receive
> > the
> > > > log stream to show. No, the REPL will no pass code snippets for
> docker
> > > > container (I think it will not be necessary)
> > > >
> > > > 4 - Yep! Like I said on the first item.
> > > >
> > > > [1] - https://github.com/italorossi/ishell
> > > >
> > > > Let me know if there is any other question!
> > > > Best!
> > > > Rafael J. R. Novello
> > > >
> > > > Skype: rafael.novello
> > > > Blog: http://rafanovello.blogspot.com.br/
> > > >
> > > >
> > > > Em qui, 18 de out de 2018 às 17:00, Lucas Bonatto Miguel <
> > > > lucasbm88@apache.org> escreveu:
> > > >
> > > > > + 1 for migrating the REPL repo to Apache
> > > > >
> > > > > I have a few questions about the previous explanation:
> > > > >  1) The REPL itself, it would be an application in which language?
> > > > Remember
> > > > > that the main idea is to allow the user to program on his preferred
> > > > > language in the REPL.
> > > > >  2) Should the engine-generate command also generate a docker image
> > > with
> > > > > the user's application?
> > > > >  3) What type of communication would happen between the REPL and
> the
> > > > engine
> > > > > via Docker SDK? Would the REPL pass snippets of code to be executed
> > by
> > > > the
> > > > > docker container?
> > > > >  4) Have you considered code completion in the REPL?
> > > > >
> > > > >
> > > > > On Thu, Oct 18, 2018 at 7:53 AM Zhang Yifei <
> yifei.z.luiz@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Ok guys,
> > > > > >
> > > > > > The basic idea is to provide only one Toolbox for multiple
> > languages.
> > > > > > We are looking for possibility to build a single Marvin Repl,
> > > > > > instead of severals toolboxes with differentes interfaces or
> > > commands.
> > > > > >
> > > > > > In this case, the engine-generate command will download and
> start a
> > > > > Docker
> > > > > > container with basic engine structure corresponding the choosed
> > > > language.
> > > > > > this means we don't need to build Toolboxes of differents
> > languages,
> > > we
> > > > > > just
> > > > > > build the engine template of all languages that we want to
> support
> > > and
> > > > > > provide it
> > > > > > as Docker containers
> > > > > >
> > > > > > We have started researches around our basic requirements like:
> > > > > > - Repl interface
> > > > > > - System communication
> > > > > > - Connection security
> > > > > > - Tool popularity
> > > > > > - Update complexity
> > > > > > - Languages support
> > > > > > - ......
> > > > > >
> > > > > > And we did some POC with code here:
> > > > > > https://github.com/marvin-ai/marvin-repl
> > > > > >
> > > > > > There is POC testing gRPC using Scala and Python,
> > > > > > Repl inteface and Docker SDK with NodeJS,
> > > > > > Repl interface and Docker SDK with Python.
> > > > > >
> > > > > > At this moment we prefer the Repl interface + Docker SDK way,
> > because
> > > > > good
> > > > > > part of the requirements
> > > > > > will be guaranteed by Docker.
> > > > > >
> > > > > > With this informations, what do you think? Should we submit all
> > this
> > > > POCs
> > > > > > to Apache Repo?
> > > > > > Please feel free to opine.
> > > > > >
> > > > > >
> > > > > > Thats all, thanks!!!
> > > > > >
> > > > > >
> > > > > > Em ter, 16 de out de 2018 às 18:55, Daniel Takabayashi <
> > > > > > daniel.takabayashi@gmail.com> escreveu:
> > > > > >
> > > > > > > Zhang,
> > > > > > >
> > > > > > > I think the best approach is give us a better explanation about
> > > this
> > > > > new
> > > > > > > feature and how this can help us to archive the desired feature
> > > > > (support
> > > > > > > multiple languages). Than if the majority agree than I can do
> the
> > > > > merge,
> > > > > > > just like I did before, but in another branch.
> > > > > > >
> > > > > > > Let me know if makes sense, ok?
> > > > > > >
> > > > > > > Taka
> > > > > > >
> > > > > > > Em ter, 16 de out de 2018 às 14:00, Luciano Resende <
> > > > > > luckbr1975@gmail.com>
> > > > > > > escreveu:
> > > > > > >
> > > > > > > > So, what is the POC, is it a refactoring of the existing
> repo?
> > Or
> > > > is
> > > > > > > > it a new rewrite of the repo?
> > > > > > > >
> > > > > > > > Just asking as it might make sense to make it a branch then
> > > > actually
> > > > > a
> > > > > > > > thing in parallel, as this will have an effect for releases,
> > etc.
> > > > but
> > > > > > > > you guys know more here than I do.
> > > > > > > >
> > > > > > > > Also, it's probably good to have a write up of the main
> > direction
> > > > of
> > > > > > > > the design that can help people get familiar with the new
> > > approach.
> > > > > > > > On Tue, Oct 16, 2018 at 11:12 AM Zhang Yifei <
> > > > yifei.z.luiz@gmail.com
> > > > > >
> > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > Hey guys,
> > > > > > > > >
> > > > > > > > > We have reorganized the Poc repo, and want to merge it to
> our
> > > > > Apache
> > > > > > > > repo.
> > > > > > > > > Just making sure here before we do the merge, because im
> not
> > > sure
> > > > > if
> > > > > > > > doing
> > > > > > > > > this will perserve the Git commit history.
> > > > > > > > > What we are planning to do is:
> > > > > > > > >
> > > > > > > > > git filter-branch --subdirectory-filter <git
> > origin_repository
> > > > > > > directory>
> > > > > > > > > -- --all
> > > > > > > > > mkdir POCs/
> > > > > > > > > git mv * POCs/
> > > > > > > > > git commit -m "colleted the data to move"
> > > > > > > > >
> > > > > > > > > git clone <git destination_repository_url> cd <git
> > > > > > > > > destination_repository_directory> git remote add
> > > > > > > origin_repository_branch
> > > > > > > > > <git origin_repository directory> git pull
> > > > origin_repository_branch
> > > > > > > > master
> > > > > > > > > --allow-unrelated-histories git remote rm
> > > > origin_repository_branch
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Thanks in advance !
> > > > > > > > >
> > > > > > > > >
> > --------------------------------------------------------------
> > > > > > > > > Zhang Yifei
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > > Luciano Resende
> > > > > > > > http://twitter.com/lresende1975
> > > > > > > > http://lresende.blogspot.com/
> > > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > --------------------------------------------------------------
> > > > > > Zhang Yifei
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: POCs, Prototypes, experiments ...

Posted by Lucas Bonatto Miguel <lu...@apache.org>.

Got it! Thanks for clarifying.

Would it make sense to use the same solution that we will use for having a
single-language REPL to have a single-language CLI? My only concern with
killing the CLI is that you remove an essential feature for DevOps.

- Lucas

On Fri, Oct 19, 2018 at 1:52 PM Rafael Novello <ra...@gmail.com>
wrote:

> Lucas,
>
> The idea is that REPL will substitute the actual CLI. It's because with the
> actual concept (using language specific CLI) we will need to re-implement
> the same features many times and probably each CLI will have a different
> behavior because some language specific restrictions and/or limitations.
>
> With this new concept, all users will have the same experience interacting
> with Marvin REPL and they will use the bests tools to develop their engines
> (Jupyter Notebook and/or Lab with different languages support or even
> Apache Zeppelin). All the interact will occur through REPL and docker
> protocol.
>
> Alan, Yes! As Lucas said, the concept is the same but we can use docker to
> do the same job.
>
> Best!
>
> Em sex, 19 de out de 2018 às 00:39, Lucas Bonatto Miguel <
> lucasbm88@apache.org> escreveu:
>
> > Thanks for the clarifications Rafael, so from what I understood, Marvin's
> > developers would not use the REPL to explore data and test models, i.e.
> > develop the engine. Is the idea to build something more like an
> interactive
> > CLI? I think an interactive CLI would be useful for the developer
> > experience, however, it's important to keep the unattended (current) CLI
> in
> > place for scenarios where users want to automate Marvin tasks.
> >
> > Alan, I believe the idea is still the same as you started, but using
> > docker SDK now.
> >
> > - Lucas
> >
> > On Thu, Oct 18, 2018 at 1:29 PM Rafael Novello <
> > rafa.reis.novello@gmail.com>
> > wrote:
> >
> > > Hi Lucas!
> > >
> > > First of all +1 for REPL POCs to Apache!
> > >
> > > Let me help with some comments:
> > >
> > > 1 - We have tested NodeJS, Scala and Python and the easiest one was
> > Python.
> > > We have found a small project [1] that have all features we desired for
> > > REPL:
> > > - Autocomplete
> > > - Python commands disabled (the user have only the commands we
> provide).
> > > The other languages REPL options don't have this feature.
> > > - Easy to show output
> > > - Etc
> > > So, I think the language chosen will not be important here because the
> > user
> > > will only interact with the commands that we create.
> > >
> > > 2 - The "engine-generate" command will download a docker image that we
> > > create for that language and start a container with the basic project
> > > structure for that language.
> > >
> > > 3 - The REPL client will use the "docker protocol" to run command,
> > > start/stop services and etc inside the container and it will receive
> the
> > > log stream to show. No, the REPL will no pass code snippets for docker
> > > container (I think it will not be necessary)
> > >
> > > 4 - Yep! Like I said on the first item.
> > >
> > > [1] - https://github.com/italorossi/ishell
> > >
> > > Let me know if there is any other question!
> > > Best!
> > > Rafael J. R. Novello
> > >
> > > Skype: rafael.novello
> > > Blog: http://rafanovello.blogspot.com.br/
> > >
> > >
> > > Em qui, 18 de out de 2018 às 17:00, Lucas Bonatto Miguel <
> > > lucasbm88@apache.org> escreveu:
> > >
> > > > + 1 for migrating the REPL repo to Apache
> > > >
> > > > I have a few questions about the previous explanation:
> > > >  1) The REPL itself, it would be an application in which language?
> > > Remember
> > > > that the main idea is to allow the user to program on his preferred
> > > > language in the REPL.
> > > >  2) Should the engine-generate command also generate a docker image
> > with
> > > > the user's application?
> > > >  3) What type of communication would happen between the REPL and the
> > > engine
> > > > via Docker SDK? Would the REPL pass snippets of code to be executed
> by
> > > the
> > > > docker container?
> > > >  4) Have you considered code completion in the REPL?
> > > >
> > > >
> > > > On Thu, Oct 18, 2018 at 7:53 AM Zhang Yifei <yi...@gmail.com>
> > > > wrote:
> > > >
> > > > > Ok guys,
> > > > >
> > > > > The basic idea is to provide only one Toolbox for multiple
> languages.
> > > > > We are looking for possibility to build a single Marvin Repl,
> > > > > instead of severals toolboxes with differentes interfaces or
> > commands.
> > > > >
> > > > > In this case, the engine-generate command will download and start a
> > > > Docker
> > > > > container with basic engine structure corresponding the choosed
> > > language.
> > > > > this means we don't need to build Toolboxes of differents
> languages,
> > we
> > > > > just
> > > > > build the engine template of all languages that we want to support
> > and
> > > > > provide it
> > > > > as Docker containers
> > > > >
> > > > > We have started researches around our basic requirements like:
> > > > > - Repl interface
> > > > > - System communication
> > > > > - Connection security
> > > > > - Tool popularity
> > > > > - Update complexity
> > > > > - Languages support
> > > > > - ......
> > > > >
> > > > > And we did some POC with code here:
> > > > > https://github.com/marvin-ai/marvin-repl
> > > > >
> > > > > There is POC testing gRPC using Scala and Python,
> > > > > Repl inteface and Docker SDK with NodeJS,
> > > > > Repl interface and Docker SDK with Python.
> > > > >
> > > > > At this moment we prefer the Repl interface + Docker SDK way,
> because
> > > > good
> > > > > part of the requirements
> > > > > will be guaranteed by Docker.
> > > > >
> > > > > With this informations, what do you think? Should we submit all
> this
> > > POCs
> > > > > to Apache Repo?
> > > > > Please feel free to opine.
> > > > >
> > > > >
> > > > > Thats all, thanks!!!
> > > > >
> > > > >
> > > > > Em ter, 16 de out de 2018 às 18:55, Daniel Takabayashi <
> > > > > daniel.takabayashi@gmail.com> escreveu:
> > > > >
> > > > > > Zhang,
> > > > > >
> > > > > > I think the best approach is give us a better explanation about
> > this
> > > > new
> > > > > > feature and how this can help us to archive the desired feature
> > > > (support
> > > > > > multiple languages). Than if the majority agree than I can do the
> > > > merge,
> > > > > > just like I did before, but in another branch.
> > > > > >
> > > > > > Let me know if makes sense, ok?
> > > > > >
> > > > > > Taka
> > > > > >
> > > > > > Em ter, 16 de out de 2018 às 14:00, Luciano Resende <
> > > > > luckbr1975@gmail.com>
> > > > > > escreveu:
> > > > > >
> > > > > > > So, what is the POC, is it a refactoring of the existing repo?
> Or
> > > is
> > > > > > > it a new rewrite of the repo?
> > > > > > >
> > > > > > > Just asking as it might make sense to make it a branch then
> > > actually
> > > > a
> > > > > > > thing in parallel, as this will have an effect for releases,
> etc.
> > > but
> > > > > > > you guys know more here than I do.
> > > > > > >
> > > > > > > Also, it's probably good to have a write up of the main
> direction
> > > of
> > > > > > > the design that can help people get familiar with the new
> > approach.
> > > > > > > On Tue, Oct 16, 2018 at 11:12 AM Zhang Yifei <
> > > yifei.z.luiz@gmail.com
> > > > >
> > > > > > > wrote:
> > > > > > > >
> > > > > > > > Hey guys,
> > > > > > > >
> > > > > > > > We have reorganized the Poc repo, and want to merge it to our
> > > > Apache
> > > > > > > repo.
> > > > > > > > Just making sure here before we do the merge, because im not
> > sure
> > > > if
> > > > > > > doing
> > > > > > > > this will perserve the Git commit history.
> > > > > > > > What we are planning to do is:
> > > > > > > >
> > > > > > > > git filter-branch --subdirectory-filter <git
> origin_repository
> > > > > > directory>
> > > > > > > > -- --all
> > > > > > > > mkdir POCs/
> > > > > > > > git mv * POCs/
> > > > > > > > git commit -m "colleted the data to move"
> > > > > > > >
> > > > > > > > git clone <git destination_repository_url> cd <git
> > > > > > > > destination_repository_directory> git remote add
> > > > > > origin_repository_branch
> > > > > > > > <git origin_repository directory> git pull
> > > origin_repository_branch
> > > > > > > master
> > > > > > > > --allow-unrelated-histories git remote rm
> > > origin_repository_branch
> > > > > > > >
> > > > > > > >
> > > > > > > > Thanks in advance !
> > > > > > > >
> > > > > > > >
> --------------------------------------------------------------
> > > > > > > > Zhang Yifei
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > Luciano Resende
> > > > > > > http://twitter.com/lresende1975
> > > > > > > http://lresende.blogspot.com/
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > > --
> > > > > --------------------------------------------------------------
> > > > > Zhang Yifei
> > > > >
> > > >
> > >
> >
>

Re: POCs, Prototypes, experiments ...

Posted by Rafael Novello <ra...@gmail.com>.

Lucas,

The idea is that REPL will substitute the actual CLI. It's because with the
actual concept (using language specific CLI) we will need to re-implement
the same features many times and probably each CLI will have a different
behavior because some language specific restrictions and/or limitations.

With this new concept, all users will have the same experience interacting
with Marvin REPL and they will use the bests tools to develop their engines
(Jupyter Notebook and/or Lab with different languages support or even
Apache Zeppelin). All the interact will occur through REPL and docker
protocol.

Alan, Yes! As Lucas said, the concept is the same but we can use docker to
do the same job.

Best!

Em sex, 19 de out de 2018 às 00:39, Lucas Bonatto Miguel <
lucasbm88@apache.org> escreveu:

> Thanks for the clarifications Rafael, so from what I understood, Marvin's
> developers would not use the REPL to explore data and test models, i.e.
> develop the engine. Is the idea to build something more like an interactive
> CLI? I think an interactive CLI would be useful for the developer
> experience, however, it's important to keep the unattended (current) CLI in
> place for scenarios where users want to automate Marvin tasks.
>
> Alan, I believe the idea is still the same as you started, but using
> docker SDK now.
>
> - Lucas
>
> On Thu, Oct 18, 2018 at 1:29 PM Rafael Novello <
> rafa.reis.novello@gmail.com>
> wrote:
>
> > Hi Lucas!
> >
> > First of all +1 for REPL POCs to Apache!
> >
> > Let me help with some comments:
> >
> > 1 - We have tested NodeJS, Scala and Python and the easiest one was
> Python.
> > We have found a small project [1] that have all features we desired for
> > REPL:
> > - Autocomplete
> > - Python commands disabled (the user have only the commands we provide).
> > The other languages REPL options don't have this feature.
> > - Easy to show output
> > - Etc
> > So, I think the language chosen will not be important here because the
> user
> > will only interact with the commands that we create.
> >
> > 2 - The "engine-generate" command will download a docker image that we
> > create for that language and start a container with the basic project
> > structure for that language.
> >
> > 3 - The REPL client will use the "docker protocol" to run command,
> > start/stop services and etc inside the container and it will receive the
> > log stream to show. No, the REPL will no pass code snippets for docker
> > container (I think it will not be necessary)
> >
> > 4 - Yep! Like I said on the first item.
> >
> > [1] - https://github.com/italorossi/ishell
> >
> > Let me know if there is any other question!
> > Best!
> > Rafael J. R. Novello
> >
> > Skype: rafael.novello
> > Blog: http://rafanovello.blogspot.com.br/
> >
> >
> > Em qui, 18 de out de 2018 às 17:00, Lucas Bonatto Miguel <
> > lucasbm88@apache.org> escreveu:
> >
> > > + 1 for migrating the REPL repo to Apache
> > >
> > > I have a few questions about the previous explanation:
> > >  1) The REPL itself, it would be an application in which language?
> > Remember
> > > that the main idea is to allow the user to program on his preferred
> > > language in the REPL.
> > >  2) Should the engine-generate command also generate a docker image
> with
> > > the user's application?
> > >  3) What type of communication would happen between the REPL and the
> > engine
> > > via Docker SDK? Would the REPL pass snippets of code to be executed by
> > the
> > > docker container?
> > >  4) Have you considered code completion in the REPL?
> > >
> > >
> > > On Thu, Oct 18, 2018 at 7:53 AM Zhang Yifei <yi...@gmail.com>
> > > wrote:
> > >
> > > > Ok guys,
> > > >
> > > > The basic idea is to provide only one Toolbox for multiple languages.
> > > > We are looking for possibility to build a single Marvin Repl,
> > > > instead of severals toolboxes with differentes interfaces or
> commands.
> > > >
> > > > In this case, the engine-generate command will download and start a
> > > Docker
> > > > container with basic engine structure corresponding the choosed
> > language.
> > > > this means we don't need to build Toolboxes of differents languages,
> we
> > > > just
> > > > build the engine template of all languages that we want to support
> and
> > > > provide it
> > > > as Docker containers
> > > >
> > > > We have started researches around our basic requirements like:
> > > > - Repl interface
> > > > - System communication
> > > > - Connection security
> > > > - Tool popularity
> > > > - Update complexity
> > > > - Languages support
> > > > - ......
> > > >
> > > > And we did some POC with code here:
> > > > https://github.com/marvin-ai/marvin-repl
> > > >
> > > > There is POC testing gRPC using Scala and Python,
> > > > Repl inteface and Docker SDK with NodeJS,
> > > > Repl interface and Docker SDK with Python.
> > > >
> > > > At this moment we prefer the Repl interface + Docker SDK way, because
> > > good
> > > > part of the requirements
> > > > will be guaranteed by Docker.
> > > >
> > > > With this informations, what do you think? Should we submit all this
> > POCs
> > > > to Apache Repo?
> > > > Please feel free to opine.
> > > >
> > > >
> > > > Thats all, thanks!!!
> > > >
> > > >
> > > > Em ter, 16 de out de 2018 às 18:55, Daniel Takabayashi <
> > > > daniel.takabayashi@gmail.com> escreveu:
> > > >
> > > > > Zhang,
> > > > >
> > > > > I think the best approach is give us a better explanation about
> this
> > > new
> > > > > feature and how this can help us to archive the desired feature
> > > (support
> > > > > multiple languages). Than if the majority agree than I can do the
> > > merge,
> > > > > just like I did before, but in another branch.
> > > > >
> > > > > Let me know if makes sense, ok?
> > > > >
> > > > > Taka
> > > > >
> > > > > Em ter, 16 de out de 2018 às 14:00, Luciano Resende <
> > > > luckbr1975@gmail.com>
> > > > > escreveu:
> > > > >
> > > > > > So, what is the POC, is it a refactoring of the existing repo? Or
> > is
> > > > > > it a new rewrite of the repo?
> > > > > >
> > > > > > Just asking as it might make sense to make it a branch then
> > actually
> > > a
> > > > > > thing in parallel, as this will have an effect for releases, etc.
> > but
> > > > > > you guys know more here than I do.
> > > > > >
> > > > > > Also, it's probably good to have a write up of the main direction
> > of
> > > > > > the design that can help people get familiar with the new
> approach.
> > > > > > On Tue, Oct 16, 2018 at 11:12 AM Zhang Yifei <
> > yifei.z.luiz@gmail.com
> > > >
> > > > > > wrote:
> > > > > > >
> > > > > > > Hey guys,
> > > > > > >
> > > > > > > We have reorganized the Poc repo, and want to merge it to our
> > > Apache
> > > > > > repo.
> > > > > > > Just making sure here before we do the merge, because im not
> sure
> > > if
> > > > > > doing
> > > > > > > this will perserve the Git commit history.
> > > > > > > What we are planning to do is:
> > > > > > >
> > > > > > > git filter-branch --subdirectory-filter <git origin_repository
> > > > > directory>
> > > > > > > -- --all
> > > > > > > mkdir POCs/
> > > > > > > git mv * POCs/
> > > > > > > git commit -m "colleted the data to move"
> > > > > > >
> > > > > > > git clone <git destination_repository_url> cd <git
> > > > > > > destination_repository_directory> git remote add
> > > > > origin_repository_branch
> > > > > > > <git origin_repository directory> git pull
> > origin_repository_branch
> > > > > > master
> > > > > > > --allow-unrelated-histories git remote rm
> > origin_repository_branch
> > > > > > >
> > > > > > >
> > > > > > > Thanks in advance !
> > > > > > >
> > > > > > > --------------------------------------------------------------
> > > > > > > Zhang Yifei
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Luciano Resende
> > > > > > http://twitter.com/lresende1975
> > > > > > http://lresende.blogspot.com/
> > > > > >
> > > > >
> > > >
> > > >
> > > > --
> > > > --------------------------------------------------------------
> > > > Zhang Yifei
> > > >
> > >
> >
>

Re: POCs, Prototypes, experiments ...

Posted by Lucas Bonatto Miguel <lu...@apache.org>.

Thanks for the clarifications Rafael, so from what I understood, Marvin's
developers would not use the REPL to explore data and test models, i.e.
develop the engine. Is the idea to build something more like an interactive
CLI? I think an interactive CLI would be useful for the developer
experience, however, it's important to keep the unattended (current) CLI in
place for scenarios where users want to automate Marvin tasks.

Alan, I believe the idea is still the same as you started, but using
docker SDK now.

- Lucas

On Thu, Oct 18, 2018 at 1:29 PM Rafael Novello <ra...@gmail.com>
wrote:

> Hi Lucas!
>
> First of all +1 for REPL POCs to Apache!
>
> Let me help with some comments:
>
> 1 - We have tested NodeJS, Scala and Python and the easiest one was Python.
> We have found a small project [1] that have all features we desired for
> REPL:
> - Autocomplete
> - Python commands disabled (the user have only the commands we provide).
> The other languages REPL options don't have this feature.
> - Easy to show output
> - Etc
> So, I think the language chosen will not be important here because the user
> will only interact with the commands that we create.
>
> 2 - The "engine-generate" command will download a docker image that we
> create for that language and start a container with the basic project
> structure for that language.
>
> 3 - The REPL client will use the "docker protocol" to run command,
> start/stop services and etc inside the container and it will receive the
> log stream to show. No, the REPL will no pass code snippets for docker
> container (I think it will not be necessary)
>
> 4 - Yep! Like I said on the first item.
>
> [1] - https://github.com/italorossi/ishell
>
> Let me know if there is any other question!
> Best!
> Rafael J. R. Novello
>
> Skype: rafael.novello
> Blog: http://rafanovello.blogspot.com.br/
>
>
> Em qui, 18 de out de 2018 às 17:00, Lucas Bonatto Miguel <
> lucasbm88@apache.org> escreveu:
>
> > + 1 for migrating the REPL repo to Apache
> >
> > I have a few questions about the previous explanation:
> >  1) The REPL itself, it would be an application in which language?
> Remember
> > that the main idea is to allow the user to program on his preferred
> > language in the REPL.
> >  2) Should the engine-generate command also generate a docker image with
> > the user's application?
> >  3) What type of communication would happen between the REPL and the
> engine
> > via Docker SDK? Would the REPL pass snippets of code to be executed by
> the
> > docker container?
> >  4) Have you considered code completion in the REPL?
> >
> >
> > On Thu, Oct 18, 2018 at 7:53 AM Zhang Yifei <yi...@gmail.com>
> > wrote:
> >
> > > Ok guys,
> > >
> > > The basic idea is to provide only one Toolbox for multiple languages.
> > > We are looking for possibility to build a single Marvin Repl,
> > > instead of severals toolboxes with differentes interfaces or commands.
> > >
> > > In this case, the engine-generate command will download and start a
> > Docker
> > > container with basic engine structure corresponding the choosed
> language.
> > > this means we don't need to build Toolboxes of differents languages, we
> > > just
> > > build the engine template of all languages that we want to support and
> > > provide it
> > > as Docker containers
> > >
> > > We have started researches around our basic requirements like:
> > > - Repl interface
> > > - System communication
> > > - Connection security
> > > - Tool popularity
> > > - Update complexity
> > > - Languages support
> > > - ......
> > >
> > > And we did some POC with code here:
> > > https://github.com/marvin-ai/marvin-repl
> > >
> > > There is POC testing gRPC using Scala and Python,
> > > Repl inteface and Docker SDK with NodeJS,
> > > Repl interface and Docker SDK with Python.
> > >
> > > At this moment we prefer the Repl interface + Docker SDK way, because
> > good
> > > part of the requirements
> > > will be guaranteed by Docker.
> > >
> > > With this informations, what do you think? Should we submit all this
> POCs
> > > to Apache Repo?
> > > Please feel free to opine.
> > >
> > >
> > > Thats all, thanks!!!
> > >
> > >
> > > Em ter, 16 de out de 2018 às 18:55, Daniel Takabayashi <
> > > daniel.takabayashi@gmail.com> escreveu:
> > >
> > > > Zhang,
> > > >
> > > > I think the best approach is give us a better explanation about this
> > new
> > > > feature and how this can help us to archive the desired feature
> > (support
> > > > multiple languages). Than if the majority agree than I can do the
> > merge,
> > > > just like I did before, but in another branch.
> > > >
> > > > Let me know if makes sense, ok?
> > > >
> > > > Taka
> > > >
> > > > Em ter, 16 de out de 2018 às 14:00, Luciano Resende <
> > > luckbr1975@gmail.com>
> > > > escreveu:
> > > >
> > > > > So, what is the POC, is it a refactoring of the existing repo? Or
> is
> > > > > it a new rewrite of the repo?
> > > > >
> > > > > Just asking as it might make sense to make it a branch then
> actually
> > a
> > > > > thing in parallel, as this will have an effect for releases, etc.
> but
> > > > > you guys know more here than I do.
> > > > >
> > > > > Also, it's probably good to have a write up of the main direction
> of
> > > > > the design that can help people get familiar with the new approach.
> > > > > On Tue, Oct 16, 2018 at 11:12 AM Zhang Yifei <
> yifei.z.luiz@gmail.com
> > >
> > > > > wrote:
> > > > > >
> > > > > > Hey guys,
> > > > > >
> > > > > > We have reorganized the Poc repo, and want to merge it to our
> > Apache
> > > > > repo.
> > > > > > Just making sure here before we do the merge, because im not sure
> > if
> > > > > doing
> > > > > > this will perserve the Git commit history.
> > > > > > What we are planning to do is:
> > > > > >
> > > > > > git filter-branch --subdirectory-filter <git origin_repository
> > > > directory>
> > > > > > -- --all
> > > > > > mkdir POCs/
> > > > > > git mv * POCs/
> > > > > > git commit -m "colleted the data to move"
> > > > > >
> > > > > > git clone <git destination_repository_url> cd <git
> > > > > > destination_repository_directory> git remote add
> > > > origin_repository_branch
> > > > > > <git origin_repository directory> git pull
> origin_repository_branch
> > > > > master
> > > > > > --allow-unrelated-histories git remote rm
> origin_repository_branch
> > > > > >
> > > > > >
> > > > > > Thanks in advance !
> > > > > >
> > > > > > --------------------------------------------------------------
> > > > > > Zhang Yifei
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Luciano Resende
> > > > > http://twitter.com/lresende1975
> > > > > http://lresende.blogspot.com/
> > > > >
> > > >
> > >
> > >
> > > --
> > > --------------------------------------------------------------
> > > Zhang Yifei
> > >
> >
>

Re: POCs, Prototypes, experiments ...

Posted by Rafael Novello <ra...@gmail.com>.

Hi Lucas!

First of all +1 for REPL POCs to Apache!

Let me help with some comments:

1 - We have tested NodeJS, Scala and Python and the easiest one was Python.
We have found a small project [1] that have all features we desired for
REPL:
- Autocomplete
- Python commands disabled (the user have only the commands we provide).
The other languages REPL options don't have this feature.
- Easy to show output
- Etc
So, I think the language chosen will not be important here because the user
will only interact with the commands that we create.

2 - The "engine-generate" command will download a docker image that we
create for that language and start a container with the basic project
structure for that language.

3 - The REPL client will use the "docker protocol" to run command,
start/stop services and etc inside the container and it will receive the
log stream to show. No, the REPL will no pass code snippets for docker
container (I think it will not be necessary)

4 - Yep! Like I said on the first item.

[1] - https://github.com/italorossi/ishell

Let me know if there is any other question!
Best!
Rafael J. R. Novello

Skype: rafael.novello
Blog: http://rafanovello.blogspot.com.br/


Em qui, 18 de out de 2018 às 17:00, Lucas Bonatto Miguel <
lucasbm88@apache.org> escreveu:

> + 1 for migrating the REPL repo to Apache
>
> I have a few questions about the previous explanation:
>  1) The REPL itself, it would be an application in which language? Remember
> that the main idea is to allow the user to program on his preferred
> language in the REPL.
>  2) Should the engine-generate command also generate a docker image with
> the user's application?
>  3) What type of communication would happen between the REPL and the engine
> via Docker SDK? Would the REPL pass snippets of code to be executed by the
> docker container?
>  4) Have you considered code completion in the REPL?
>
>
> On Thu, Oct 18, 2018 at 7:53 AM Zhang Yifei <yi...@gmail.com>
> wrote:
>
> > Ok guys,
> >
> > The basic idea is to provide only one Toolbox for multiple languages.
> > We are looking for possibility to build a single Marvin Repl,
> > instead of severals toolboxes with differentes interfaces or commands.
> >
> > In this case, the engine-generate command will download and start a
> Docker
> > container with basic engine structure corresponding the choosed language.
> > this means we don't need to build Toolboxes of differents languages, we
> > just
> > build the engine template of all languages that we want to support and
> > provide it
> > as Docker containers
> >
> > We have started researches around our basic requirements like:
> > - Repl interface
> > - System communication
> > - Connection security
> > - Tool popularity
> > - Update complexity
> > - Languages support
> > - ......
> >
> > And we did some POC with code here:
> > https://github.com/marvin-ai/marvin-repl
> >
> > There is POC testing gRPC using Scala and Python,
> > Repl inteface and Docker SDK with NodeJS,
> > Repl interface and Docker SDK with Python.
> >
> > At this moment we prefer the Repl interface + Docker SDK way, because
> good
> > part of the requirements
> > will be guaranteed by Docker.
> >
> > With this informations, what do you think? Should we submit all this POCs
> > to Apache Repo?
> > Please feel free to opine.
> >
> >
> > Thats all, thanks!!!
> >
> >
> > Em ter, 16 de out de 2018 às 18:55, Daniel Takabayashi <
> > daniel.takabayashi@gmail.com> escreveu:
> >
> > > Zhang,
> > >
> > > I think the best approach is give us a better explanation about this
> new
> > > feature and how this can help us to archive the desired feature
> (support
> > > multiple languages). Than if the majority agree than I can do the
> merge,
> > > just like I did before, but in another branch.
> > >
> > > Let me know if makes sense, ok?
> > >
> > > Taka
> > >
> > > Em ter, 16 de out de 2018 às 14:00, Luciano Resende <
> > luckbr1975@gmail.com>
> > > escreveu:
> > >
> > > > So, what is the POC, is it a refactoring of the existing repo? Or is
> > > > it a new rewrite of the repo?
> > > >
> > > > Just asking as it might make sense to make it a branch then actually
> a
> > > > thing in parallel, as this will have an effect for releases, etc. but
> > > > you guys know more here than I do.
> > > >
> > > > Also, it's probably good to have a write up of the main direction of
> > > > the design that can help people get familiar with the new approach.
> > > > On Tue, Oct 16, 2018 at 11:12 AM Zhang Yifei <yifei.z.luiz@gmail.com
> >
> > > > wrote:
> > > > >
> > > > > Hey guys,
> > > > >
> > > > > We have reorganized the Poc repo, and want to merge it to our
> Apache
> > > > repo.
> > > > > Just making sure here before we do the merge, because im not sure
> if
> > > > doing
> > > > > this will perserve the Git commit history.
> > > > > What we are planning to do is:
> > > > >
> > > > > git filter-branch --subdirectory-filter <git origin_repository
> > > directory>
> > > > > -- --all
> > > > > mkdir POCs/
> > > > > git mv * POCs/
> > > > > git commit -m "colleted the data to move"
> > > > >
> > > > > git clone <git destination_repository_url> cd <git
> > > > > destination_repository_directory> git remote add
> > > origin_repository_branch
> > > > > <git origin_repository directory> git pull origin_repository_branch
> > > > master
> > > > > --allow-unrelated-histories git remote rm origin_repository_branch
> > > > >
> > > > >
> > > > > Thanks in advance !
> > > > >
> > > > > --------------------------------------------------------------
> > > > > Zhang Yifei
> > > >
> > > >
> > > >
> > > > --
> > > > Luciano Resende
> > > > http://twitter.com/lresende1975
> > > > http://lresende.blogspot.com/
> > > >
> > >
> >
> >
> > --
> > --------------------------------------------------------------
> > Zhang Yifei
> >
>

Re: POCs, Prototypes, experiments ...

Posted by Alan Silva <al...@cloudera.com.INVALID>.

Hi,

I had started work on this feature about system communication, and make
some PoCs that I commit in the past at marvin-repl and basically the idea
is is to leverage existing engine resources using scala and python via gRPC
as stated in the previous email from yifei, but this is my initial idea! If
you like to discuss further about a better architecture and resources of
this feature, seems good to me too.

My basic idea consists in this basic diagram:



     +-------------------+                      +--------------------+
     |    Basic REPL |  +--------->      |        Docker       |
     |    Interface      |  <-gRPC-+     |  Instantiation     |
     +-------------------+                       +-------------------+







On Thu, Oct 18, 2018 at 3:00 PM Lucas Bonatto Miguel <lu...@apache.org>
wrote:

> + 1 for migrating the REPL repo to Apache
>
>  1) The REPL itself, it would be an application in which language? Remember
> that the main idea is to allow the user to program on his preferred
> language in the REPL.
>




>  2) Should the engine-generate command also generate a docker image with
> the user's application?
>



>  3) What type of communication would happen between the REPL and the engine
> via Docker SDK? Would the REPL pass snippets of code to be executed by the
> docker container?
>



>  4) Have you considered code completion in the REPL?
>
>
> On Thu, Oct 18, 2018 at 7:53 AM Zhang Yifei <yi...@gmail.com>
> wrote:
>
> > Ok guys,
> >
> > The basic idea is to provide only one Toolbox for multiple languages.
> > We are looking for possibility to build a single Marvin Repl,
> > instead of severals toolboxes with differentes interfaces or commands.
> >
> > In this case, the engine-generate command will download and start a
> Docker
> > container with basic engine structure corresponding the choosed language.
> > this means we don't need to build Toolboxes of differents languages, we
> > just
> > build the engine template of all languages that we want to support and
> > provide it
> > as Docker containers
> >
> > We have started researches around our basic requirements like:
> > - Repl interface
> > - System communication
> > - Connection security
> > - Tool popularity
> > - Update complexity
> > - Languages support
> > - ......
> >
> > And we did some POC with code here:
> > https://github.com/marvin-ai/marvin-repl
> >
> > There is POC testing gRPC using Scala and Python,
> > Repl inteface and Docker SDK with NodeJS,
> > Repl interface and Docker SDK with Python.
> >
> > At this moment we prefer the Repl interface + Docker SDK way, because
> good
> > part of the requirements
> > will be guaranteed by Docker.
> >
> > With this informations, what do you think? Should we submit all this POCs
> > to Apache Repo?
> > Please feel free to opine.
> >
> >
> > Thats all, thanks!!!
> >
> >
> > Em ter, 16 de out de 2018 às 18:55, Daniel Takabayashi <
> > daniel.takabayashi@gmail.com> escreveu:
> >
> > > Zhang,
> > >
> > > I think the best approach is give us a better explanation about this
> new
> > > feature and how this can help us to archive the desired feature
> (support
> > > multiple languages). Than if the majority agree than I can do the
> merge,
> > > just like I did before, but in another branch.
> > >
> > > Let me know if makes sense, ok?
> > >
> > > Taka
> > >
> > > Em ter, 16 de out de 2018 às 14:00, Luciano Resende <
> > luckbr1975@gmail.com>
> > > escreveu:
> > >
> > > > So, what is the POC, is it a refactoring of the existing repo? Or is
> > > > it a new rewrite of the repo?
> > > >
> > > > Just asking as it might make sense to make it a branch then actually
> a
> > > > thing in parallel, as this will have an effect for releases, etc. but
> > > > you guys know more here than I do.
> > > >
> > > > Also, it's probably good to have a write up of the main direction of
> > > > the design that can help people get familiar with the new approach.
> > > > On Tue, Oct 16, 2018 at 11:12 AM Zhang Yifei <yifei.z.luiz@gmail.com
> >
> > > > wrote:
> > > > >
> > > > > Hey guys,
> > > > >
> > > > > We have reorganized the Poc repo, and want to merge it to our
> Apache
> > > > repo.
> > > > > Just making sure here before we do the merge, because im not sure
> if
> > > > doing
> > > > > this will perserve the Git commit history.
> > > > > What we are planning to do is:
> > > > >
> > > > > git filter-branch --subdirectory-filter <git origin_repository
> > > directory>
> > > > > -- --all
> > > > > mkdir POCs/
> > > > > git mv * POCs/
> > > > > git commit -m "colleted the data to move"
> > > > >
> > > > > git clone <git destination_repository_url> cd <git
> > > > > destination_repository_directory> git remote add
> > > origin_repository_branch
> > > > > <git origin_repository directory> git pull origin_repository_branch
> > > > master
> > > > > --allow-unrelated-histories git remote rm origin_repository_branch
> > > > >
> > > > >
> > > > > Thanks in advance !
> > > > >
> > > > > --------------------------------------------------------------
> > > > > Zhang Yifei
> > > >
> > > >
> > > >
> > > > --
> > > > Luciano Resende
> > > > http://twitter.com/lresende1975
> > > > http://lresende.blogspot.com/
> > > >
> > >
> >
> >
> > --
> > --------------------------------------------------------------
> > Zhang Yifei
> >
>


-- 
*Alan Silva* | Solutions Architect
t. +55 (11) 99951- <0000000000>5242 e. alan.silva@cloudera.com
cloudera.com <http://www.cloudera.com>

[image: Cloudera] <http://www.cloudera.com/>

[image: Cloudera on Twitter] <https://twitter.com/cloudera> [image:
Cloudera on Facebook] <https://www.facebook.com/cloudera> [image: Cloudera
on LinkedIn] <https://www.linkedin.com/company/cloudera>

Re: POCs, Prototypes, experiments ...

Posted by Lucas Bonatto Miguel <lu...@apache.org>.

+ 1 for migrating the REPL repo to Apache

I have a few questions about the previous explanation:
 1) The REPL itself, it would be an application in which language? Remember
that the main idea is to allow the user to program on his preferred
language in the REPL.
 2) Should the engine-generate command also generate a docker image with
the user's application?
 3) What type of communication would happen between the REPL and the engine
via Docker SDK? Would the REPL pass snippets of code to be executed by the
docker container?
 4) Have you considered code completion in the REPL?


On Thu, Oct 18, 2018 at 7:53 AM Zhang Yifei <yi...@gmail.com> wrote:

> Ok guys,
>
> The basic idea is to provide only one Toolbox for multiple languages.
> We are looking for possibility to build a single Marvin Repl,
> instead of severals toolboxes with differentes interfaces or commands.
>
> In this case, the engine-generate command will download and start a Docker
> container with basic engine structure corresponding the choosed language.
> this means we don't need to build Toolboxes of differents languages, we
> just
> build the engine template of all languages that we want to support and
> provide it
> as Docker containers
>
> We have started researches around our basic requirements like:
> - Repl interface
> - System communication
> - Connection security
> - Tool popularity
> - Update complexity
> - Languages support
> - ......
>
> And we did some POC with code here:
> https://github.com/marvin-ai/marvin-repl
>
> There is POC testing gRPC using Scala and Python,
> Repl inteface and Docker SDK with NodeJS,
> Repl interface and Docker SDK with Python.
>
> At this moment we prefer the Repl interface + Docker SDK way, because  good
> part of the requirements
> will be guaranteed by Docker.
>
> With this informations, what do you think? Should we submit all this POCs
> to Apache Repo?
> Please feel free to opine.
>
>
> Thats all, thanks!!!
>
>
> Em ter, 16 de out de 2018 às 18:55, Daniel Takabayashi <
> daniel.takabayashi@gmail.com> escreveu:
>
> > Zhang,
> >
> > I think the best approach is give us a better explanation about this new
> > feature and how this can help us to archive the desired feature (support
> > multiple languages). Than if the majority agree than I can do the merge,
> > just like I did before, but in another branch.
> >
> > Let me know if makes sense, ok?
> >
> > Taka
> >
> > Em ter, 16 de out de 2018 às 14:00, Luciano Resende <
> luckbr1975@gmail.com>
> > escreveu:
> >
> > > So, what is the POC, is it a refactoring of the existing repo? Or is
> > > it a new rewrite of the repo?
> > >
> > > Just asking as it might make sense to make it a branch then actually a
> > > thing in parallel, as this will have an effect for releases, etc. but
> > > you guys know more here than I do.
> > >
> > > Also, it's probably good to have a write up of the main direction of
> > > the design that can help people get familiar with the new approach.
> > > On Tue, Oct 16, 2018 at 11:12 AM Zhang Yifei <yi...@gmail.com>
> > > wrote:
> > > >
> > > > Hey guys,
> > > >
> > > > We have reorganized the Poc repo, and want to merge it to our Apache
> > > repo.
> > > > Just making sure here before we do the merge, because im not sure if
> > > doing
> > > > this will perserve the Git commit history.
> > > > What we are planning to do is:
> > > >
> > > > git filter-branch --subdirectory-filter <git origin_repository
> > directory>
> > > > -- --all
> > > > mkdir POCs/
> > > > git mv * POCs/
> > > > git commit -m "colleted the data to move"
> > > >
> > > > git clone <git destination_repository_url> cd <git
> > > > destination_repository_directory> git remote add
> > origin_repository_branch
> > > > <git origin_repository directory> git pull origin_repository_branch
> > > master
> > > > --allow-unrelated-histories git remote rm origin_repository_branch
> > > >
> > > >
> > > > Thanks in advance !
> > > >
> > > > --------------------------------------------------------------
> > > > Zhang Yifei
> > >
> > >
> > >
> > > --
> > > Luciano Resende
> > > http://twitter.com/lresende1975
> > > http://lresende.blogspot.com/
> > >
> >
>
>
> --
> --------------------------------------------------------------
> Zhang Yifei
>

Re: POCs, Prototypes, experiments ...

Posted by Zhang Yifei <yi...@gmail.com>.

Ok guys,

The basic idea is to provide only one Toolbox for multiple languages.
We are looking for possibility to build a single Marvin Repl,
instead of severals toolboxes with differentes interfaces or commands.

In this case, the engine-generate command will download and start a Docker
container with basic engine structure corresponding the choosed language.
this means we don't need to build Toolboxes of differents languages, we just
build the engine template of all languages that we want to support and
provide it
as Docker containers

We have started researches around our basic requirements like:
- Repl interface
- System communication
- Connection security
- Tool popularity
- Update complexity
- Languages support
- ......

And we did some POC with code here: https://github.com/marvin-ai/marvin-repl

There is POC testing gRPC using Scala and Python,
Repl inteface and Docker SDK with NodeJS,
Repl interface and Docker SDK with Python.

At this moment we prefer the Repl interface + Docker SDK way, because  good
part of the requirements
will be guaranteed by Docker.

With this informations, what do you think? Should we submit all this POCs
to Apache Repo?
Please feel free to opine.

Thats all, thanks!!!

Em ter, 16 de out de 2018 às 18:55, Daniel Takabayashi <
daniel.takabayashi@gmail.com> escreveu:

> Zhang,
>
> I think the best approach is give us a better explanation about this new
> feature and how this can help us to archive the desired feature (support
> multiple languages). Than if the majority agree than I can do the merge,
> just like I did before, but in another branch.
>
> Let me know if makes sense, ok?
>
> Taka
>
> Em ter, 16 de out de 2018 às 14:00, Luciano Resende <lu...@gmail.com>
> escreveu:
>
> > So, what is the POC, is it a refactoring of the existing repo? Or is
> > it a new rewrite of the repo?
> >
> > Just asking as it might make sense to make it a branch then actually a
> > thing in parallel, as this will have an effect for releases, etc. but
> > you guys know more here than I do.
> >
> > Also, it's probably good to have a write up of the main direction of
> > the design that can help people get familiar with the new approach.
> > On Tue, Oct 16, 2018 at 11:12 AM Zhang Yifei <yi...@gmail.com>
> > wrote:
> > >
> > > Hey guys,
> > >
> > > We have reorganized the Poc repo, and want to merge it to our Apache
> > repo.
> > > Just making sure here before we do the merge, because im not sure if
> > doing
> > > this will perserve the Git commit history.
> > > What we are planning to do is:
> > >
> > > git filter-branch --subdirectory-filter <git origin_repository
> directory>
> > > -- --all
> > > mkdir POCs/
> > > git mv * POCs/
> > > git commit -m "colleted the data to move"
> > >
> > > git clone <git destination_repository_url> cd <git
> > > destination_repository_directory> git remote add
> origin_repository_branch
> > > <git origin_repository directory> git pull origin_repository_branch
> > master
> > > --allow-unrelated-histories git remote rm origin_repository_branch
> > >
> > >
> > > Thanks in advance !
> > >
> > > --------------------------------------------------------------
> > > Zhang Yifei
> >
> >
> >
> > --
> > Luciano Resende
> > http://twitter.com/lresende1975
> > http://lresende.blogspot.com/
> >
>

-- 
--------------------------------------------------------------
Zhang Yifei

Re: POCs, Prototypes, experiments ...

Posted by Daniel Takabayashi <da...@gmail.com>.

Zhang,

I think the best approach is give us a better explanation about this new
feature and how this can help us to archive the desired feature (support
multiple languages). Than if the majority agree than I can do the merge,
just like I did before, but in another branch.

Let me know if makes sense, ok?

Taka

Em ter, 16 de out de 2018 às 14:00, Luciano Resende <lu...@gmail.com>
escreveu:

> So, what is the POC, is it a refactoring of the existing repo? Or is
> it a new rewrite of the repo?
>
> Just asking as it might make sense to make it a branch then actually a
> thing in parallel, as this will have an effect for releases, etc. but
> you guys know more here than I do.
>
> Also, it's probably good to have a write up of the main direction of
> the design that can help people get familiar with the new approach.
> On Tue, Oct 16, 2018 at 11:12 AM Zhang Yifei <yi...@gmail.com>
> wrote:
> >
> > Hey guys,
> >
> > We have reorganized the Poc repo, and want to merge it to our Apache
> repo.
> > Just making sure here before we do the merge, because im not sure if
> doing
> > this will perserve the Git commit history.
> > What we are planning to do is:
> >
> > git filter-branch --subdirectory-filter <git origin_repository directory>
> > -- --all
> > mkdir POCs/
> > git mv * POCs/
> > git commit -m "colleted the data to move"
> >
> > git clone <git destination_repository_url> cd <git
> > destination_repository_directory> git remote add origin_repository_branch
> > <git origin_repository directory> git pull origin_repository_branch
> master
> > --allow-unrelated-histories git remote rm origin_repository_branch
> >
> >
> > Thanks in advance !
> >
> > --------------------------------------------------------------
> > Zhang Yifei
>
>
>
> --
> Luciano Resende
> http://twitter.com/lresende1975
> http://lresende.blogspot.com/
>

Re: POCs, Prototypes, experiments ...

Posted by Luciano Resende <lu...@gmail.com>.

So, what is the POC, is it a refactoring of the existing repo? Or is
it a new rewrite of the repo?

Just asking as it might make sense to make it a branch then actually a
thing in parallel, as this will have an effect for releases, etc. but
you guys know more here than I do.

Also, it's probably good to have a write up of the main direction of
the design that can help people get familiar with the new approach.
On Tue, Oct 16, 2018 at 11:12 AM Zhang Yifei <yi...@gmail.com> wrote:
>
> Hey guys,
>
> We have reorganized the Poc repo, and want to merge it to our Apache repo.
> Just making sure here before we do the merge, because im not sure if doing
> this will perserve the Git commit history.
> What we are planning to do is:
>
> git filter-branch --subdirectory-filter <git origin_repository directory>
> -- --all
> mkdir POCs/
> git mv * POCs/
> git commit -m "colleted the data to move"
>
> git clone <git destination_repository_url> cd <git
> destination_repository_directory> git remote add origin_repository_branch
> <git origin_repository directory> git pull origin_repository_branch master
> --allow-unrelated-histories git remote rm origin_repository_branch
>
>
> Thanks in advance !
>
> --------------------------------------------------------------
> Zhang Yifei



-- 
Luciano Resende
http://twitter.com/lresende1975
http://lresende.blogspot.com/

Re: POCs, Prototypes, experiments ...

Posted by Zhang Yifei <yi...@gmail.com>.

Guys,

The commands is poorly formatted in my last e-mail,
I reorganized it here: https://pastebin.com/guMLtgJZ

Em ter, 16 de out de 2018 às 15:12, Zhang Yifei <yi...@gmail.com>
escreveu:

> Hey guys,
>
> We have reorganized the Poc repo, and want to merge it to our Apache repo.
> Just making sure here before we do the merge, because im not sure if doing
> this will perserve the Git commit history.
> What we are planning to do is:
>
> git filter-branch --subdirectory-filter <git origin_repository directory>
> -- --all
> mkdir POCs/
> git mv * POCs/
> git commit -m "colleted the data to move"
>
> git clone <git destination_repository_url> cd <git
> destination_repository_directory> git remote add origin_repository_branch
> <git origin_repository directory> git pull origin_repository_branch master
> --allow-unrelated-histories git remote rm origin_repository_branch
>
>
> Thanks in advance !
>
> --------------------------------------------------------------
> Zhang Yifei
>


-- 
--------------------------------------------------------------
Zhang Yifei

Re: POCs, Prototypes, experiments ...

Posted by Zhang Yifei <yi...@gmail.com>.

Hey guys,

We have reorganized the Poc repo, and want to merge it to our Apache repo.
Just making sure here before we do the merge, because im not sure if doing
this will perserve the Git commit history.
What we are planning to do is:

git filter-branch --subdirectory-filter <git origin_repository directory>
-- --all
mkdir POCs/
git mv * POCs/
git commit -m "colleted the data to move"

git clone <git destination_repository_url> cd <git
destination_repository_directory> git remote add origin_repository_branch
<git origin_repository directory> git pull origin_repository_branch master
--allow-unrelated-histories git remote rm origin_repository_branch


Thanks in advance !

--------------------------------------------------------------
Zhang Yifei

Re: POCs, Prototypes, experiments ...

Posted by Zhang Yifei <yi...@gmail.com>.

+1

Yes we have started those POCs of the new toolbox concept for a while,
and we never clarified this concept to all Marvin contributors.

Em seg, 15 de out de 2018 às 19:29, Daniel Takabayashi <
daniel.takabayashi@gmail.com> escreveu:

> +1
>
> The best way is create this branch and start to tell a little more about
> this new "concept" and how we are planning to do this.
>
> Em seg, 15 de out de 2018 às 15:16, Lucas Bonatto Miguel <
> lucasbm88@apache.org> escreveu:
>
> > +1
> > Rafael, do you think you could create a branch with the POC code? That
> > would be a big step towards working as a community.
> >
> > Together we're stronger!
> >
> > On Mon, Oct 15, 2018 at 12:10 PM Luciano Resende <lu...@gmail.com>
> > wrote:
> >
> > > Now that Marvin is an Apache open source project, I would like to
> > > promote more and more open development and recommend that items like
> > > the one mentioned below (e.g. prototypes, experiments, POCs, etc) be
> > > done in the open as branches or PRs for the projects which will enable
> > > others on the community to be aware of the new ideas and also be able
> > > to comment/participate on the early design and even help on the
> > > implementation.
> > >
> > > Thanks
> > >
> > > ---------- Forwarded message ---------
> > > From: Rafael Novello <ra...@gmail.com>
> > > Date: Mon, Oct 15, 2018 at 11:56 AM
> > > Subject: Re: Future support to R
> > > To: <de...@marvin.apache.org>
> > >
> > > We are working (doing some POCs and trials) on a new toolbox concept to
> > > facilitate the inclusion of new languages support. We could discuss
> this
> > in
> > > more details but I believe this kind of detail is not the focus for
> this
> > > thread.
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > --
> > > Luciano Resende
> > > http://twitter.com/lresende1975
> > > http://lresende.blogspot.com/
> > >
> >
>


-- 
--------------------------------------------------------------
Zhang Yifei

Re: POCs, Prototypes, experiments ...

Posted by Daniel Takabayashi <da...@gmail.com>.

+1

The best way is create this branch and start to tell a little more about
this new "concept" and how we are planning to do this.

Em seg, 15 de out de 2018 às 15:16, Lucas Bonatto Miguel <
lucasbm88@apache.org> escreveu:

> +1
> Rafael, do you think you could create a branch with the POC code? That
> would be a big step towards working as a community.
>
> Together we're stronger!
>
> On Mon, Oct 15, 2018 at 12:10 PM Luciano Resende <lu...@gmail.com>
> wrote:
>
> > Now that Marvin is an Apache open source project, I would like to
> > promote more and more open development and recommend that items like
> > the one mentioned below (e.g. prototypes, experiments, POCs, etc) be
> > done in the open as branches or PRs for the projects which will enable
> > others on the community to be aware of the new ideas and also be able
> > to comment/participate on the early design and even help on the
> > implementation.
> >
> > Thanks
> >
> > ---------- Forwarded message ---------
> > From: Rafael Novello <ra...@gmail.com>
> > Date: Mon, Oct 15, 2018 at 11:56 AM
> > Subject: Re: Future support to R
> > To: <de...@marvin.apache.org>
> >
> > We are working (doing some POCs and trials) on a new toolbox concept to
> > facilitate the inclusion of new languages support. We could discuss this
> in
> > more details but I believe this kind of detail is not the focus for this
> > thread.
> >
> >
> >
> >
> >
> >
> >
> > --
> > Luciano Resende
> > http://twitter.com/lresende1975
> > http://lresende.blogspot.com/
> >
>

Re: POCs, Prototypes, experiments ...

Posted by Lucas Bonatto Miguel <lu...@apache.org>.

+1
Rafael, do you think you could create a branch with the POC code? That
would be a big step towards working as a community.

Together we're stronger!

On Mon, Oct 15, 2018 at 12:10 PM Luciano Resende <lu...@gmail.com>
wrote:

> Now that Marvin is an Apache open source project, I would like to
> promote more and more open development and recommend that items like
> the one mentioned below (e.g. prototypes, experiments, POCs, etc) be
> done in the open as branches or PRs for the projects which will enable
> others on the community to be aware of the new ideas and also be able
> to comment/participate on the early design and even help on the
> implementation.
>
> Thanks
>
> ---------- Forwarded message ---------
> From: Rafael Novello <ra...@gmail.com>
> Date: Mon, Oct 15, 2018 at 11:56 AM
> Subject: Re: Future support to R
> To: <de...@marvin.apache.org>
>
> We are working (doing some POCs and trials) on a new toolbox concept to
> facilitate the inclusion of new languages support. We could discuss this in
> more details but I believe this kind of detail is not the focus for this
> thread.
>
>
>
>
>
>
>
> --
> Luciano Resende
> http://twitter.com/lresende1975
> http://lresende.blogspot.com/
>

POCs, Prototypes, experiments ...

Posted by Luciano Resende <lu...@gmail.com>.

Now that Marvin is an Apache open source project, I would like to
promote more and more open development and recommend that items like
the one mentioned below (e.g. prototypes, experiments, POCs, etc) be
done in the open as branches or PRs for the projects which will enable
others on the community to be aware of the new ideas and also be able
to comment/participate on the early design and even help on the
implementation.

Thanks

---------- Forwarded message ---------
From: Rafael Novello <ra...@gmail.com>
Date: Mon, Oct 15, 2018 at 11:56 AM
Subject: Re: Future support to R
To: <de...@marvin.apache.org>

We are working (doing some POCs and trials) on a new toolbox concept to
facilitate the inclusion of new languages support. We could discuss this in
more details but I believe this kind of detail is not the focus for this
thread.







-- 
Luciano Resende
http://twitter.com/lresende1975
http://lresende.blogspot.com/

Re: Future support to R

Posted by Daniel Takabayashi <da...@gmail.com>.

Guys, anyone has question about what should be done?

Push more code ???? Lol

Sent from my iPhone

> On May 23, 2019, at 1:33 PM, Lucas Bonatto Miguel <lu...@apache.org> wrote:
> 
> All the issue have description, or am I wrong?
> 
> Before upadting, it's good to hear if anyone has any questions about what
> needs to be done. I think it's relatively clear.
> 
> IMO, better use of your time would be pushing some code to help us to move
> forward.
> I believe this one is a strong candidate:
> https://issues.apache.org/jira/projects/MARVIN/issues/MARVIN-3?filter=allopenissues
> 
> - Lucas
> 
> 
> On Thu, May 23, 2019 at 1:24 PM Daniel Takabayashi <
> daniel.takabayashi@gmail.com> wrote:
> 
>> Hi, Guys
>> 
>> To help ourselves to avoid more misunderstanding I will update all the
>> issues we need to do to archive this big feature, everybody is ok with
>> this?
>> 
>> Em sex, 9 de nov de 2018 às 00:25, Lucas Bonatto Miguel <
>> lucasbonatto@gmail.com> escreveu:
>> 
>>> Alan, the issues that will help you implement the changes to support R
>> are
>>> created in our Jira
>> https://issues.apache.org/jira/projects/MARVIN/issues
>>> 
>>> 
>>>> On Mon, Oct 29, 2018 at 10:59 AM Alan Silva <ju...@apache.org> wrote:
>>>> 
>>>> Ok, guys
>>>> 
>>>> I will like to get this and work with a start on this task this week
>>> based
>>>> today in your python-engine and I'll tell you about the progress.
>>>> 
>>>> Thanks,
>>>> 
>>>> 
>>>> On Thu, Oct 25, 2018 at 9:24 PM Daniel Takabayashi <
>>>> daniel.takabayashi@gmail.com> wrote:
>>>> 
>>>>> +1
>>>>> 
>>>>> Em qui, 25 de out de 2018 às 17:19, Lucas Bonatto Miguel <
>>>>> lucasbm88@apache.org> escreveu:
>>>>> 
>>>>>> This discussion is now duplicated at the thread with title "POCs,
>>>>>> Prototypes, experiments"
>>>>>> 
>>>>>> Let's continue the discussion there.
>>>>>> 
>>>>>> EOF
>>>>>> 
>>>>>> On Thu, Oct 25, 2018 at 4:50 PM Daniel Takabayashi <
>>>>>> daniel.takabayashi@gmail.com> wrote:
>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>>> 
>> https://docs.google.com/drawings/d/shajxIpLJHxxMbFgDXiPuhg/image?w=602&h=461&rev=1423&ac=1&parent=1ySERHGBXbHeyCMRookq5UfTuFkzzU0ugtjvR3rF3deY
>>>>>>> 
>>>>>>> Em qui, 25 de out de 2018 às 11:20, Lucas Bonatto Miguel <
>>>>>>> lucasbm88@apache.org> escreveu:
>>>>>>> 
>>>>>>>> Hey Daniel, I can't open your attachment. It is broken for me.
>>>> Could
>>>>>> you
>>>>>>>> add a link to the image?
>>>>>>>> 
>>>>>>>> Thanks,
>>>>>>>> Lucas
>>>>>>>> 
>>>>>>>> On Wed, Oct 24, 2018 at 5:11 PM Daniel Takabayashi <
>>>>>>>> daniel.takabayashi@gmail.com> wrote:
>>>>>>>> 
>>>>>>>>> [image: marvin-architecture-views (5).png]
>>>>>>>>> attached the arch draw..
>>>>>>>>> 
>>>>>>>>> Em qua, 24 de out de 2018 às 17:09, Daniel Takabayashi <
>>>>>>>>> daniel.takabayashi@gmail.com> escreveu:
>>>>>>>>> 
>>>>>>>>>> Hi all,
>>>>>>>>>> 
>>>>>>>>>> To try to give a little more details about this "new
>> concept"
>>>> that
>>>>>> we
>>>>>>>>>> want to bring to the marvin toolbox, I did this simple
>>>>> architecture
>>>>>>>> draw.
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> The general idea here is try to transform the toolbox
>>> something
>>>>>>>>>> disconnected with the "language", something more agnostic.
>>> Also
>>>> in
>>>>>>> this
>>>>>>>> new
>>>>>>>>>> architecture we could use remote resource to process engines
>>> and
>>>>>> make
>>>>>>>> easy
>>>>>>>>>> the support for new languages.
>>>>>>>>>> 
>>>>>>>>>> This "new toolbox" will be the only thing that a Marvin user
>>>> must
>>>>> to
>>>>>>>>>> install and also we could start to support multiples O.S
>> (once
>>>> the
>>>>>>> REPL
>>>>>>>> is
>>>>>>>>>> a dummy application that only interprets and by pass
>>> commands).
>>>>>>>>>> 
>>>>>>>>>> Regards,
>>>>>>>>>> Taka
>>>>>>>>>> 
>>>>>>>>>> Em qua, 24 de out de 2018 às 12:44, Luciano Resende <
>>>>>>>> luckbr1975@gmail.com>
>>>>>>>>>> escreveu:
>>>>>>>>>> 
>>>>>>>>>>> On Wed, Oct 24, 2018 at 5:56 PM Rafael Novello
>>>>>>>>>>> <ra...@gmail.com> wrote:
>>>>>>>>>>>> 
>>>>>>>>>>>> Alan,
>>>>>>>>>>>> 
>>>>>>>>>>>> To be honest, I don't know. The new toolbox concept help
>> us
>>>>> with
>>>>>>> some
>>>>>>>>>>>> problems
>>>>>>>>>>>> but it create others. I believe we need to vote if the
>> new
>>>>>> concept
>>>>>>>>>>> will be
>>>>>>>>>>>> the future
>>>>>>>>>>>> or not. Am I right?
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> We usually only require a vote when there is no consensus
>> on
>>>> the
>>>>>>>>>>> community. If the community agrees with the new direction,
>>> then
>>>>>> there
>>>>>>>>>>> isn't a need for a formal vote. In case there isn't
>>> agreement,
>>>> a
>>>>>> vote
>>>>>>>>>>> can be a possibility to resolve the impasse.
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> --
>>>>>>>>>>> Luciano Resende
>>>>>>>>>>> http://twitter.com/lresende1975
>>>>>>>>>>> http://lresende.blogspot.com/
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>>> 
>>

Re: Future support to R

Posted by Lucas Bonatto Miguel <lu...@apache.org>.

All the issue have description, or am I wrong?

Before upadting, it's good to hear if anyone has any questions about what
needs to be done. I think it's relatively clear.

IMO, better use of your time would be pushing some code to help us to move
forward.
I believe this one is a strong candidate:
https://issues.apache.org/jira/projects/MARVIN/issues/MARVIN-3?filter=allopenissues

- Lucas


On Thu, May 23, 2019 at 1:24 PM Daniel Takabayashi <
daniel.takabayashi@gmail.com> wrote:

> Hi, Guys
>
> To help ourselves to avoid more misunderstanding I will update all the
> issues we need to do to archive this big feature, everybody is ok with
> this?
>
> Em sex, 9 de nov de 2018 às 00:25, Lucas Bonatto Miguel <
> lucasbonatto@gmail.com> escreveu:
>
> > Alan, the issues that will help you implement the changes to support R
> are
> > created in our Jira
> https://issues.apache.org/jira/projects/MARVIN/issues
> >
> >
> > On Mon, Oct 29, 2018 at 10:59 AM Alan Silva <ju...@apache.org> wrote:
> >
> > > Ok, guys
> > >
> > > I will like to get this and work with a start on this task this week
> > based
> > > today in your python-engine and I'll tell you about the progress.
> > >
> > > Thanks,
> > >
> > >
> > > On Thu, Oct 25, 2018 at 9:24 PM Daniel Takabayashi <
> > > daniel.takabayashi@gmail.com> wrote:
> > >
> > > > +1
> > > >
> > > > Em qui, 25 de out de 2018 às 17:19, Lucas Bonatto Miguel <
> > > > lucasbm88@apache.org> escreveu:
> > > >
> > > > > This discussion is now duplicated at the thread with title "POCs,
> > > > > Prototypes, experiments"
> > > > >
> > > > > Let's continue the discussion there.
> > > > >
> > > > > EOF
> > > > >
> > > > > On Thu, Oct 25, 2018 at 4:50 PM Daniel Takabayashi <
> > > > > daniel.takabayashi@gmail.com> wrote:
> > > > >
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://docs.google.com/drawings/d/shajxIpLJHxxMbFgDXiPuhg/image?w=602&h=461&rev=1423&ac=1&parent=1ySERHGBXbHeyCMRookq5UfTuFkzzU0ugtjvR3rF3deY
> > > > > >
> > > > > > Em qui, 25 de out de 2018 às 11:20, Lucas Bonatto Miguel <
> > > > > > lucasbm88@apache.org> escreveu:
> > > > > >
> > > > > > > Hey Daniel, I can't open your attachment. It is broken for me.
> > > Could
> > > > > you
> > > > > > > add a link to the image?
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Lucas
> > > > > > >
> > > > > > > On Wed, Oct 24, 2018 at 5:11 PM Daniel Takabayashi <
> > > > > > > daniel.takabayashi@gmail.com> wrote:
> > > > > > >
> > > > > > > > [image: marvin-architecture-views (5).png]
> > > > > > > > attached the arch draw..
> > > > > > > >
> > > > > > > > Em qua, 24 de out de 2018 às 17:09, Daniel Takabayashi <
> > > > > > > > daniel.takabayashi@gmail.com> escreveu:
> > > > > > > >
> > > > > > > >> Hi all,
> > > > > > > >>
> > > > > > > >> To try to give a little more details about this "new
> concept"
> > > that
> > > > > we
> > > > > > > >> want to bring to the marvin toolbox, I did this simple
> > > > architecture
> > > > > > > draw.
> > > > > > > >>
> > > > > > > >>
> > > > > > > >> The general idea here is try to transform the toolbox
> > something
> > > > > > > >> disconnected with the "language", something more agnostic.
> > Also
> > > in
> > > > > > this
> > > > > > > new
> > > > > > > >> architecture we could use remote resource to process engines
> > and
> > > > > make
> > > > > > > easy
> > > > > > > >> the support for new languages.
> > > > > > > >>
> > > > > > > >> This "new toolbox" will be the only thing that a Marvin user
> > > must
> > > > to
> > > > > > > >> install and also we could start to support multiples O.S
> (once
> > > the
> > > > > > REPL
> > > > > > > is
> > > > > > > >> a dummy application that only interprets and by pass
> > commands).
> > > > > > > >>
> > > > > > > >> Regards,
> > > > > > > >> Taka
> > > > > > > >>
> > > > > > > >> Em qua, 24 de out de 2018 às 12:44, Luciano Resende <
> > > > > > > luckbr1975@gmail.com>
> > > > > > > >> escreveu:
> > > > > > > >>
> > > > > > > >>> On Wed, Oct 24, 2018 at 5:56 PM Rafael Novello
> > > > > > > >>> <ra...@gmail.com> wrote:
> > > > > > > >>> >
> > > > > > > >>> > Alan,
> > > > > > > >>> >
> > > > > > > >>> > To be honest, I don't know. The new toolbox concept help
> us
> > > > with
> > > > > > some
> > > > > > > >>> > problems
> > > > > > > >>> > but it create others. I believe we need to vote if the
> new
> > > > > concept
> > > > > > > >>> will be
> > > > > > > >>> > the future
> > > > > > > >>> > or not. Am I right?
> > > > > > > >>> >
> > > > > > > >>>
> > > > > > > >>> We usually only require a vote when there is no consensus
> on
> > > the
> > > > > > > >>> community. If the community agrees with the new direction,
> > then
> > > > > there
> > > > > > > >>> isn't a need for a formal vote. In case there isn't
> > agreement,
> > > a
> > > > > vote
> > > > > > > >>> can be a possibility to resolve the impasse.
> > > > > > > >>>
> > > > > > > >>>
> > > > > > > >>> --
> > > > > > > >>> Luciano Resende
> > > > > > > >>> http://twitter.com/lresende1975
> > > > > > > >>> http://lresende.blogspot.com/
> > > > > > > >>>
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: Future support to R

Posted by Daniel Takabayashi <da...@gmail.com>.

Hi, Guys

To help ourselves to avoid more misunderstanding I will update all the
issues we need to do to archive this big feature, everybody is ok with this?

Em sex, 9 de nov de 2018 às 00:25, Lucas Bonatto Miguel <
lucasbonatto@gmail.com> escreveu:

> Alan, the issues that will help you implement the changes to support R are
> created in our Jira https://issues.apache.org/jira/projects/MARVIN/issues
>
>
> On Mon, Oct 29, 2018 at 10:59 AM Alan Silva <ju...@apache.org> wrote:
>
> > Ok, guys
> >
> > I will like to get this and work with a start on this task this week
> based
> > today in your python-engine and I'll tell you about the progress.
> >
> > Thanks,
> >
> >
> > On Thu, Oct 25, 2018 at 9:24 PM Daniel Takabayashi <
> > daniel.takabayashi@gmail.com> wrote:
> >
> > > +1
> > >
> > > Em qui, 25 de out de 2018 às 17:19, Lucas Bonatto Miguel <
> > > lucasbm88@apache.org> escreveu:
> > >
> > > > This discussion is now duplicated at the thread with title "POCs,
> > > > Prototypes, experiments"
> > > >
> > > > Let's continue the discussion there.
> > > >
> > > > EOF
> > > >
> > > > On Thu, Oct 25, 2018 at 4:50 PM Daniel Takabayashi <
> > > > daniel.takabayashi@gmail.com> wrote:
> > > >
> > > > >
> > > > >
> > > >
> > >
> >
> https://docs.google.com/drawings/d/shajxIpLJHxxMbFgDXiPuhg/image?w=602&h=461&rev=1423&ac=1&parent=1ySERHGBXbHeyCMRookq5UfTuFkzzU0ugtjvR3rF3deY
> > > > >
> > > > > Em qui, 25 de out de 2018 às 11:20, Lucas Bonatto Miguel <
> > > > > lucasbm88@apache.org> escreveu:
> > > > >
> > > > > > Hey Daniel, I can't open your attachment. It is broken for me.
> > Could
> > > > you
> > > > > > add a link to the image?
> > > > > >
> > > > > > Thanks,
> > > > > > Lucas
> > > > > >
> > > > > > On Wed, Oct 24, 2018 at 5:11 PM Daniel Takabayashi <
> > > > > > daniel.takabayashi@gmail.com> wrote:
> > > > > >
> > > > > > > [image: marvin-architecture-views (5).png]
> > > > > > > attached the arch draw..
> > > > > > >
> > > > > > > Em qua, 24 de out de 2018 às 17:09, Daniel Takabayashi <
> > > > > > > daniel.takabayashi@gmail.com> escreveu:
> > > > > > >
> > > > > > >> Hi all,
> > > > > > >>
> > > > > > >> To try to give a little more details about this "new concept"
> > that
> > > > we
> > > > > > >> want to bring to the marvin toolbox, I did this simple
> > > architecture
> > > > > > draw.
> > > > > > >>
> > > > > > >>
> > > > > > >> The general idea here is try to transform the toolbox
> something
> > > > > > >> disconnected with the "language", something more agnostic.
> Also
> > in
> > > > > this
> > > > > > new
> > > > > > >> architecture we could use remote resource to process engines
> and
> > > > make
> > > > > > easy
> > > > > > >> the support for new languages.
> > > > > > >>
> > > > > > >> This "new toolbox" will be the only thing that a Marvin user
> > must
> > > to
> > > > > > >> install and also we could start to support multiples O.S (once
> > the
> > > > > REPL
> > > > > > is
> > > > > > >> a dummy application that only interprets and by pass
> commands).
> > > > > > >>
> > > > > > >> Regards,
> > > > > > >> Taka
> > > > > > >>
> > > > > > >> Em qua, 24 de out de 2018 às 12:44, Luciano Resende <
> > > > > > luckbr1975@gmail.com>
> > > > > > >> escreveu:
> > > > > > >>
> > > > > > >>> On Wed, Oct 24, 2018 at 5:56 PM Rafael Novello
> > > > > > >>> <ra...@gmail.com> wrote:
> > > > > > >>> >
> > > > > > >>> > Alan,
> > > > > > >>> >
> > > > > > >>> > To be honest, I don't know. The new toolbox concept help us
> > > with
> > > > > some
> > > > > > >>> > problems
> > > > > > >>> > but it create others. I believe we need to vote if the new
> > > > concept
> > > > > > >>> will be
> > > > > > >>> > the future
> > > > > > >>> > or not. Am I right?
> > > > > > >>> >
> > > > > > >>>
> > > > > > >>> We usually only require a vote when there is no consensus on
> > the
> > > > > > >>> community. If the community agrees with the new direction,
> then
> > > > there
> > > > > > >>> isn't a need for a formal vote. In case there isn't
> agreement,
> > a
> > > > vote
> > > > > > >>> can be a possibility to resolve the impasse.
> > > > > > >>>
> > > > > > >>>
> > > > > > >>> --
> > > > > > >>> Luciano Resende
> > > > > > >>> http://twitter.com/lresende1975
> > > > > > >>> http://lresende.blogspot.com/
> > > > > > >>>
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: Future support to R

Posted by Lucas Bonatto Miguel <lu...@gmail.com>.

Alan, the issues that will help you implement the changes to support R are
created in our Jira https://issues.apache.org/jira/projects/MARVIN/issues


On Mon, Oct 29, 2018 at 10:59 AM Alan Silva <ju...@apache.org> wrote:

> Ok, guys
>
> I will like to get this and work with a start on this task this week based
> today in your python-engine and I'll tell you about the progress.
>
> Thanks,
>
>
> On Thu, Oct 25, 2018 at 9:24 PM Daniel Takabayashi <
> daniel.takabayashi@gmail.com> wrote:
>
> > +1
> >
> > Em qui, 25 de out de 2018 às 17:19, Lucas Bonatto Miguel <
> > lucasbm88@apache.org> escreveu:
> >
> > > This discussion is now duplicated at the thread with title "POCs,
> > > Prototypes, experiments"
> > >
> > > Let's continue the discussion there.
> > >
> > > EOF
> > >
> > > On Thu, Oct 25, 2018 at 4:50 PM Daniel Takabayashi <
> > > daniel.takabayashi@gmail.com> wrote:
> > >
> > > >
> > > >
> > >
> >
> https://docs.google.com/drawings/d/shajxIpLJHxxMbFgDXiPuhg/image?w=602&h=461&rev=1423&ac=1&parent=1ySERHGBXbHeyCMRookq5UfTuFkzzU0ugtjvR3rF3deY
> > > >
> > > > Em qui, 25 de out de 2018 às 11:20, Lucas Bonatto Miguel <
> > > > lucasbm88@apache.org> escreveu:
> > > >
> > > > > Hey Daniel, I can't open your attachment. It is broken for me.
> Could
> > > you
> > > > > add a link to the image?
> > > > >
> > > > > Thanks,
> > > > > Lucas
> > > > >
> > > > > On Wed, Oct 24, 2018 at 5:11 PM Daniel Takabayashi <
> > > > > daniel.takabayashi@gmail.com> wrote:
> > > > >
> > > > > > [image: marvin-architecture-views (5).png]
> > > > > > attached the arch draw..
> > > > > >
> > > > > > Em qua, 24 de out de 2018 às 17:09, Daniel Takabayashi <
> > > > > > daniel.takabayashi@gmail.com> escreveu:
> > > > > >
> > > > > >> Hi all,
> > > > > >>
> > > > > >> To try to give a little more details about this "new concept"
> that
> > > we
> > > > > >> want to bring to the marvin toolbox, I did this simple
> > architecture
> > > > > draw.
> > > > > >>
> > > > > >>
> > > > > >> The general idea here is try to transform the toolbox something
> > > > > >> disconnected with the "language", something more agnostic. Also
> in
> > > > this
> > > > > new
> > > > > >> architecture we could use remote resource to process engines and
> > > make
> > > > > easy
> > > > > >> the support for new languages.
> > > > > >>
> > > > > >> This "new toolbox" will be the only thing that a Marvin user
> must
> > to
> > > > > >> install and also we could start to support multiples O.S (once
> the
> > > > REPL
> > > > > is
> > > > > >> a dummy application that only interprets and by pass commands).
> > > > > >>
> > > > > >> Regards,
> > > > > >> Taka
> > > > > >>
> > > > > >> Em qua, 24 de out de 2018 às 12:44, Luciano Resende <
> > > > > luckbr1975@gmail.com>
> > > > > >> escreveu:
> > > > > >>
> > > > > >>> On Wed, Oct 24, 2018 at 5:56 PM Rafael Novello
> > > > > >>> <ra...@gmail.com> wrote:
> > > > > >>> >
> > > > > >>> > Alan,
> > > > > >>> >
> > > > > >>> > To be honest, I don't know. The new toolbox concept help us
> > with
> > > > some
> > > > > >>> > problems
> > > > > >>> > but it create others. I believe we need to vote if the new
> > > concept
> > > > > >>> will be
> > > > > >>> > the future
> > > > > >>> > or not. Am I right?
> > > > > >>> >
> > > > > >>>
> > > > > >>> We usually only require a vote when there is no consensus on
> the
> > > > > >>> community. If the community agrees with the new direction, then
> > > there
> > > > > >>> isn't a need for a formal vote. In case there isn't agreement,
> a
> > > vote
> > > > > >>> can be a possibility to resolve the impasse.
> > > > > >>>
> > > > > >>>
> > > > > >>> --
> > > > > >>> Luciano Resende
> > > > > >>> http://twitter.com/lresende1975
> > > > > >>> http://lresende.blogspot.com/
> > > > > >>>
> > > > > >>
> > > > >
> > > >
> > >
> >
>

Re: Future support to R

Posted by Alan Silva <ju...@apache.org>.

Ok, guys

I will like to get this and work with a start on this task this week based
today in your python-engine and I'll tell you about the progress.

Thanks,


On Thu, Oct 25, 2018 at 9:24 PM Daniel Takabayashi <
daniel.takabayashi@gmail.com> wrote:

> +1
>
> Em qui, 25 de out de 2018 às 17:19, Lucas Bonatto Miguel <
> lucasbm88@apache.org> escreveu:
>
> > This discussion is now duplicated at the thread with title "POCs,
> > Prototypes, experiments"
> >
> > Let's continue the discussion there.
> >
> > EOF
> >
> > On Thu, Oct 25, 2018 at 4:50 PM Daniel Takabayashi <
> > daniel.takabayashi@gmail.com> wrote:
> >
> > >
> > >
> >
> https://docs.google.com/drawings/d/shajxIpLJHxxMbFgDXiPuhg/image?w=602&h=461&rev=1423&ac=1&parent=1ySERHGBXbHeyCMRookq5UfTuFkzzU0ugtjvR3rF3deY
> > >
> > > Em qui, 25 de out de 2018 às 11:20, Lucas Bonatto Miguel <
> > > lucasbm88@apache.org> escreveu:
> > >
> > > > Hey Daniel, I can't open your attachment. It is broken for me. Could
> > you
> > > > add a link to the image?
> > > >
> > > > Thanks,
> > > > Lucas
> > > >
> > > > On Wed, Oct 24, 2018 at 5:11 PM Daniel Takabayashi <
> > > > daniel.takabayashi@gmail.com> wrote:
> > > >
> > > > > [image: marvin-architecture-views (5).png]
> > > > > attached the arch draw..
> > > > >
> > > > > Em qua, 24 de out de 2018 às 17:09, Daniel Takabayashi <
> > > > > daniel.takabayashi@gmail.com> escreveu:
> > > > >
> > > > >> Hi all,
> > > > >>
> > > > >> To try to give a little more details about this "new concept" that
> > we
> > > > >> want to bring to the marvin toolbox, I did this simple
> architecture
> > > > draw.
> > > > >>
> > > > >>
> > > > >> The general idea here is try to transform the toolbox something
> > > > >> disconnected with the "language", something more agnostic. Also in
> > > this
> > > > new
> > > > >> architecture we could use remote resource to process engines and
> > make
> > > > easy
> > > > >> the support for new languages.
> > > > >>
> > > > >> This "new toolbox" will be the only thing that a Marvin user must
> to
> > > > >> install and also we could start to support multiples O.S (once the
> > > REPL
> > > > is
> > > > >> a dummy application that only interprets and by pass commands).
> > > > >>
> > > > >> Regards,
> > > > >> Taka
> > > > >>
> > > > >> Em qua, 24 de out de 2018 às 12:44, Luciano Resende <
> > > > luckbr1975@gmail.com>
> > > > >> escreveu:
> > > > >>
> > > > >>> On Wed, Oct 24, 2018 at 5:56 PM Rafael Novello
> > > > >>> <ra...@gmail.com> wrote:
> > > > >>> >
> > > > >>> > Alan,
> > > > >>> >
> > > > >>> > To be honest, I don't know. The new toolbox concept help us
> with
> > > some
> > > > >>> > problems
> > > > >>> > but it create others. I believe we need to vote if the new
> > concept
> > > > >>> will be
> > > > >>> > the future
> > > > >>> > or not. Am I right?
> > > > >>> >
> > > > >>>
> > > > >>> We usually only require a vote when there is no consensus on the
> > > > >>> community. If the community agrees with the new direction, then
> > there
> > > > >>> isn't a need for a formal vote. In case there isn't agreement, a
> > vote
> > > > >>> can be a possibility to resolve the impasse.
> > > > >>>
> > > > >>>
> > > > >>> --
> > > > >>> Luciano Resende
> > > > >>> http://twitter.com/lresende1975
> > > > >>> http://lresende.blogspot.com/
> > > > >>>
> > > > >>
> > > >
> > >
> >
>

Re: Future support to R

Posted by Daniel Takabayashi <da...@gmail.com>.

+1

Em qui, 25 de out de 2018 às 17:19, Lucas Bonatto Miguel <
lucasbm88@apache.org> escreveu:

> This discussion is now duplicated at the thread with title "POCs,
> Prototypes, experiments"
>
> Let's continue the discussion there.
>
> EOF
>
> On Thu, Oct 25, 2018 at 4:50 PM Daniel Takabayashi <
> daniel.takabayashi@gmail.com> wrote:
>
> >
> >
> https://docs.google.com/drawings/d/shajxIpLJHxxMbFgDXiPuhg/image?w=602&h=461&rev=1423&ac=1&parent=1ySERHGBXbHeyCMRookq5UfTuFkzzU0ugtjvR3rF3deY
> >
> > Em qui, 25 de out de 2018 às 11:20, Lucas Bonatto Miguel <
> > lucasbm88@apache.org> escreveu:
> >
> > > Hey Daniel, I can't open your attachment. It is broken for me. Could
> you
> > > add a link to the image?
> > >
> > > Thanks,
> > > Lucas
> > >
> > > On Wed, Oct 24, 2018 at 5:11 PM Daniel Takabayashi <
> > > daniel.takabayashi@gmail.com> wrote:
> > >
> > > > [image: marvin-architecture-views (5).png]
> > > > attached the arch draw..
> > > >
> > > > Em qua, 24 de out de 2018 às 17:09, Daniel Takabayashi <
> > > > daniel.takabayashi@gmail.com> escreveu:
> > > >
> > > >> Hi all,
> > > >>
> > > >> To try to give a little more details about this "new concept" that
> we
> > > >> want to bring to the marvin toolbox, I did this simple architecture
> > > draw.
> > > >>
> > > >>
> > > >> The general idea here is try to transform the toolbox something
> > > >> disconnected with the "language", something more agnostic. Also in
> > this
> > > new
> > > >> architecture we could use remote resource to process engines and
> make
> > > easy
> > > >> the support for new languages.
> > > >>
> > > >> This "new toolbox" will be the only thing that a Marvin user must to
> > > >> install and also we could start to support multiples O.S (once the
> > REPL
> > > is
> > > >> a dummy application that only interprets and by pass commands).
> > > >>
> > > >> Regards,
> > > >> Taka
> > > >>
> > > >> Em qua, 24 de out de 2018 às 12:44, Luciano Resende <
> > > luckbr1975@gmail.com>
> > > >> escreveu:
> > > >>
> > > >>> On Wed, Oct 24, 2018 at 5:56 PM Rafael Novello
> > > >>> <ra...@gmail.com> wrote:
> > > >>> >
> > > >>> > Alan,
> > > >>> >
> > > >>> > To be honest, I don't know. The new toolbox concept help us with
> > some
> > > >>> > problems
> > > >>> > but it create others. I believe we need to vote if the new
> concept
> > > >>> will be
> > > >>> > the future
> > > >>> > or not. Am I right?
> > > >>> >
> > > >>>
> > > >>> We usually only require a vote when there is no consensus on the
> > > >>> community. If the community agrees with the new direction, then
> there
> > > >>> isn't a need for a formal vote. In case there isn't agreement, a
> vote
> > > >>> can be a possibility to resolve the impasse.
> > > >>>
> > > >>>
> > > >>> --
> > > >>> Luciano Resende
> > > >>> http://twitter.com/lresende1975
> > > >>> http://lresende.blogspot.com/
> > > >>>
> > > >>
> > >
> >
>

Re: Future support to R

Posted by Lucas Bonatto Miguel <lu...@apache.org>.

This discussion is now duplicated at the thread with title "POCs,
Prototypes, experiments"

Let's continue the discussion there.

EOF

On Thu, Oct 25, 2018 at 4:50 PM Daniel Takabayashi <
daniel.takabayashi@gmail.com> wrote:

>
> https://docs.google.com/drawings/d/shajxIpLJHxxMbFgDXiPuhg/image?w=602&h=461&rev=1423&ac=1&parent=1ySERHGBXbHeyCMRookq5UfTuFkzzU0ugtjvR3rF3deY
>
> Em qui, 25 de out de 2018 às 11:20, Lucas Bonatto Miguel <
> lucasbm88@apache.org> escreveu:
>
> > Hey Daniel, I can't open your attachment. It is broken for me. Could you
> > add a link to the image?
> >
> > Thanks,
> > Lucas
> >
> > On Wed, Oct 24, 2018 at 5:11 PM Daniel Takabayashi <
> > daniel.takabayashi@gmail.com> wrote:
> >
> > > [image: marvin-architecture-views (5).png]
> > > attached the arch draw..
> > >
> > > Em qua, 24 de out de 2018 às 17:09, Daniel Takabayashi <
> > > daniel.takabayashi@gmail.com> escreveu:
> > >
> > >> Hi all,
> > >>
> > >> To try to give a little more details about this "new concept" that we
> > >> want to bring to the marvin toolbox, I did this simple architecture
> > draw.
> > >>
> > >>
> > >> The general idea here is try to transform the toolbox something
> > >> disconnected with the "language", something more agnostic. Also in
> this
> > new
> > >> architecture we could use remote resource to process engines and make
> > easy
> > >> the support for new languages.
> > >>
> > >> This "new toolbox" will be the only thing that a Marvin user must to
> > >> install and also we could start to support multiples O.S (once the
> REPL
> > is
> > >> a dummy application that only interprets and by pass commands).
> > >>
> > >> Regards,
> > >> Taka
> > >>
> > >> Em qua, 24 de out de 2018 às 12:44, Luciano Resende <
> > luckbr1975@gmail.com>
> > >> escreveu:
> > >>
> > >>> On Wed, Oct 24, 2018 at 5:56 PM Rafael Novello
> > >>> <ra...@gmail.com> wrote:
> > >>> >
> > >>> > Alan,
> > >>> >
> > >>> > To be honest, I don't know. The new toolbox concept help us with
> some
> > >>> > problems
> > >>> > but it create others. I believe we need to vote if the new concept
> > >>> will be
> > >>> > the future
> > >>> > or not. Am I right?
> > >>> >
> > >>>
> > >>> We usually only require a vote when there is no consensus on the
> > >>> community. If the community agrees with the new direction, then there
> > >>> isn't a need for a formal vote. In case there isn't agreement, a vote
> > >>> can be a possibility to resolve the impasse.
> > >>>
> > >>>
> > >>> --
> > >>> Luciano Resende
> > >>> http://twitter.com/lresende1975
> > >>> http://lresende.blogspot.com/
> > >>>
> > >>
> >
>

Re: Future support to R

Posted by Daniel Takabayashi <da...@gmail.com>.

https://docs.google.com/drawings/d/shajxIpLJHxxMbFgDXiPuhg/image?w=602&h=461&rev=1423&ac=1&parent=1ySERHGBXbHeyCMRookq5UfTuFkzzU0ugtjvR3rF3deY

Em qui, 25 de out de 2018 às 11:20, Lucas Bonatto Miguel <
lucasbm88@apache.org> escreveu:

> Hey Daniel, I can't open your attachment. It is broken for me. Could you
> add a link to the image?
>
> Thanks,
> Lucas
>
> On Wed, Oct 24, 2018 at 5:11 PM Daniel Takabayashi <
> daniel.takabayashi@gmail.com> wrote:
>
> > [image: marvin-architecture-views (5).png]
> > attached the arch draw..
> >
> > Em qua, 24 de out de 2018 às 17:09, Daniel Takabayashi <
> > daniel.takabayashi@gmail.com> escreveu:
> >
> >> Hi all,
> >>
> >> To try to give a little more details about this "new concept" that we
> >> want to bring to the marvin toolbox, I did this simple architecture
> draw.
> >>
> >>
> >> The general idea here is try to transform the toolbox something
> >> disconnected with the "language", something more agnostic. Also in this
> new
> >> architecture we could use remote resource to process engines and make
> easy
> >> the support for new languages.
> >>
> >> This "new toolbox" will be the only thing that a Marvin user must to
> >> install and also we could start to support multiples O.S (once the REPL
> is
> >> a dummy application that only interprets and by pass commands).
> >>
> >> Regards,
> >> Taka
> >>
> >> Em qua, 24 de out de 2018 às 12:44, Luciano Resende <
> luckbr1975@gmail.com>
> >> escreveu:
> >>
> >>> On Wed, Oct 24, 2018 at 5:56 PM Rafael Novello
> >>> <ra...@gmail.com> wrote:
> >>> >
> >>> > Alan,
> >>> >
> >>> > To be honest, I don't know. The new toolbox concept help us with some
> >>> > problems
> >>> > but it create others. I believe we need to vote if the new concept
> >>> will be
> >>> > the future
> >>> > or not. Am I right?
> >>> >
> >>>
> >>> We usually only require a vote when there is no consensus on the
> >>> community. If the community agrees with the new direction, then there
> >>> isn't a need for a formal vote. In case there isn't agreement, a vote
> >>> can be a possibility to resolve the impasse.
> >>>
> >>>
> >>> --
> >>> Luciano Resende
> >>> http://twitter.com/lresende1975
> >>> http://lresende.blogspot.com/
> >>>
> >>
>

Re: Future support to R

Posted by Lucas Bonatto Miguel <lu...@apache.org>.

Hey Daniel, I can't open your attachment. It is broken for me. Could you
add a link to the image?

Thanks,
Lucas

On Wed, Oct 24, 2018 at 5:11 PM Daniel Takabayashi <
daniel.takabayashi@gmail.com> wrote:

> [image: marvin-architecture-views (5).png]
> attached the arch draw..
>
> Em qua, 24 de out de 2018 às 17:09, Daniel Takabayashi <
> daniel.takabayashi@gmail.com> escreveu:
>
>> Hi all,
>>
>> To try to give a little more details about this "new concept" that we
>> want to bring to the marvin toolbox, I did this simple architecture draw.
>>
>>
>> The general idea here is try to transform the toolbox something
>> disconnected with the "language", something more agnostic. Also in this new
>> architecture we could use remote resource to process engines and make easy
>> the support for new languages.
>>
>> This "new toolbox" will be the only thing that a Marvin user must to
>> install and also we could start to support multiples O.S (once the REPL is
>> a dummy application that only interprets and by pass commands).
>>
>> Regards,
>> Taka
>>
>> Em qua, 24 de out de 2018 às 12:44, Luciano Resende <lu...@gmail.com>
>> escreveu:
>>
>>> On Wed, Oct 24, 2018 at 5:56 PM Rafael Novello
>>> <ra...@gmail.com> wrote:
>>> >
>>> > Alan,
>>> >
>>> > To be honest, I don't know. The new toolbox concept help us with some
>>> > problems
>>> > but it create others. I believe we need to vote if the new concept
>>> will be
>>> > the future
>>> > or not. Am I right?
>>> >
>>>
>>> We usually only require a vote when there is no consensus on the
>>> community. If the community agrees with the new direction, then there
>>> isn't a need for a formal vote. In case there isn't agreement, a vote
>>> can be a possibility to resolve the impasse.
>>>
>>>
>>> --
>>> Luciano Resende
>>> http://twitter.com/lresende1975
>>> http://lresende.blogspot.com/
>>>
>>

Re: Future support to R

Posted by Daniel Takabayashi <da...@gmail.com>.

[image: marvin-architecture-views (5).png]
attached the arch draw..

Em qua, 24 de out de 2018 às 17:09, Daniel Takabayashi <
daniel.takabayashi@gmail.com> escreveu:

> Hi all,
>
> To try to give a little more details about this "new concept" that we want
> to bring to the marvin toolbox, I did this simple architecture draw.
>
>
> The general idea here is try to transform the toolbox something
> disconnected with the "language", something more agnostic. Also in this new
> architecture we could use remote resource to process engines and make easy
> the support for new languages.
>
> This "new toolbox" will be the only thing that a Marvin user must to
> install and also we could start to support multiples O.S (once the REPL is
> a dummy application that only interprets and by pass commands).
>
> Regards,
> Taka
>
> Em qua, 24 de out de 2018 às 12:44, Luciano Resende <lu...@gmail.com>
> escreveu:
>
>> On Wed, Oct 24, 2018 at 5:56 PM Rafael Novello
>> <ra...@gmail.com> wrote:
>> >
>> > Alan,
>> >
>> > To be honest, I don't know. The new toolbox concept help us with some
>> > problems
>> > but it create others. I believe we need to vote if the new concept will
>> be
>> > the future
>> > or not. Am I right?
>> >
>>
>> We usually only require a vote when there is no consensus on the
>> community. If the community agrees with the new direction, then there
>> isn't a need for a formal vote. In case there isn't agreement, a vote
>> can be a possibility to resolve the impasse.
>>
>>
>> --
>> Luciano Resende
>> http://twitter.com/lresende1975
>> http://lresende.blogspot.com/
>>
>

Re: Future support to R

Posted by Daniel Takabayashi <da...@gmail.com>.

Hi all,

To try to give a little more details about this "new concept" that we want
to bring to the marvin toolbox, I did this simple architecture draw.

The general idea here is try to transform the toolbox something
disconnected with the "language", something more agnostic. Also in this new
architecture we could use remote resource to process engines and make easy
the support for new languages.

This "new toolbox" will be the only thing that a Marvin user must to
install and also we could start to support multiples O.S (once the REPL is
a dummy application that only interprets and by pass commands).

Regards,
Taka

Em qua, 24 de out de 2018 às 12:44, Luciano Resende <lu...@gmail.com>
escreveu:

> On Wed, Oct 24, 2018 at 5:56 PM Rafael Novello
> <ra...@gmail.com> wrote:
> >
> > Alan,
> >
> > To be honest, I don't know. The new toolbox concept help us with some
> > problems
> > but it create others. I believe we need to vote if the new concept will
> be
> > the future
> > or not. Am I right?
> >
>
> We usually only require a vote when there is no consensus on the
> community. If the community agrees with the new direction, then there
> isn't a need for a formal vote. In case there isn't agreement, a vote
> can be a possibility to resolve the impasse.
>
>
> --
> Luciano Resende
> http://twitter.com/lresende1975
> http://lresende.blogspot.com/
>

Re: Future support to R

Posted by Luciano Resende <lu...@gmail.com>.

On Wed, Oct 24, 2018 at 5:56 PM Rafael Novello
<ra...@gmail.com> wrote:
>
> Alan,
>
> To be honest, I don't know. The new toolbox concept help us with some
> problems
> but it create others. I believe we need to vote if the new concept will be
> the future
> or not. Am I right?
>

We usually only require a vote when there is no consensus on the
community. If the community agrees with the new direction, then there
isn't a need for a formal vote. In case there isn't agreement, a vote
can be a possibility to resolve the impasse.

-- 
Luciano Resende
http://twitter.com/lresende1975
http://lresende.blogspot.com/

Re: Future support to R

Posted by Rafael Novello <ra...@gmail.com>.

Alan,

To be honest, I don't know. The new toolbox concept help us with some
problems
but it create others. I believe we need to vote if the new concept will be
the future
or not. Am I right?

Best
Atenciosamente,
Rafael J. R. Novello

Skype: rafael.novello
Blog: http://rafanovello.blogspot.com.br/


Em qua, 24 de out de 2018 às 12:05, Alan Silva <ju...@apache.org> escreveu:

> Hi,
>
> One question here, if I will start with development about this R support at
> our engine, I will make it based on that we have today or in this new
> concept?
>
> Because I understood the concepts change a lot, right?
>
>
>
> On Mon, Oct 15, 2018 at 1:56 PM Rafael Novello <
> rafa.reis.novello@gmail.com>
> wrote:
>
> > Hi guys!
> >
> > Despite issues related to performance and scalability I believe we need
> > to support R engines to cover the most part of data science/machine
> > learning
> > projects. Sometimes the code performance isn't the main question and the
> > team performance (how fast the team can delivery some value) is more
> > important.
> >
> > We are working (doing some POCs and trials) on a new toolbox concept to
> > facilitate the inclusion of new languages support. We could discuss this
> in
> > more details but I believe this kind of detail is not the focus for this
> > thread.
> >
> > +1 for R support
> >
> > Best!
> > Atenciosamente,
> > Rafael J. R. Novello
> >
> > Skype: rafael.novello
> > Blog: http://rafanovello.blogspot.com.br/
> >
> >
> > Em qui, 11 de out de 2018 às 23:01, Lucas Bonatto Miguel <
> > lucasbonatto@gmail.com> escreveu:
> >
> > > The idea is to create the R toolbox, providing a similar user
> experience
> > > that we provide in Python today.
> > >
> > > Regards,
> > > Lucas
> > >
> > > On Thu, Oct 11, 2018 at 6:30 PM Alan Silva <alan.silva@cloudera.com
> > > .invalid>
> > > wrote:
> > >
> > > > Hi to all,
> > > >
> > > > Yes, R would be a language like Python that I believe we should have
> in
> > > our
> > > > toolset and even as I had already raised it, I'm also very interested
> > in
> > > > working on this part of making a frontend with R.
> > > >
> > > > However, am I understanding after your last mail that the idea here
> is
> > to
> > > > do something different from what happens with Python today? Was not
> > that
> > > > clear to me? Or would it make a frontend similar to what we have in
> > > Python
> > > > and in the future for Scala too, correct?
> > > >
> > > >
> > > > On Thu, Oct 11, 2018 at 8:21 PM Lucas Bonatto Miguel <
> > > lucasbm88@apache.org
> > > > >
> > > > wrote:
> > > >
> > > > > Hi community, I already participated in several discussions with
> some
> > > of
> > > > > you regarding adding support to R on Marvin. Allowing users to
> write
> > > > > engines in R would be a great step for Marvin since that's the
> > primary
> > > > > language for many statisticians, physicists and math people.
> > > > >
> > > > > One of the points that always arise during those conversations is
> the
> > > > fact
> > > > > that R is not fast nor scalable, and it would be hard to either
> make
> > it
> > > > > process large chunks of data for training or make it support high
> > > > > throughput and have low latency for predictions. I want to start a
> > > thread
> > > > > on whether we should invest more time trying to add R support to
> > Marvin
> > > > or
> > > > > not. I invite you all to read about this related work:
> > > > > https://medium.com/graalvm/faster-r-with-fastr-4b8db0e0dceb
> > > > >
> > > > > I am looking forward to hearing opinions and ideas.
> > > > >
> > > > > Thanks,
> > > > > Lucas
> > > > >
> > > >
> > > >
> > > > --
> > > > *Alan Silva* | Solutions Architect
> > > > t. +55 (11) 99951- <0000000000>5242 e. alan.silva@cloudera.com
> > > > cloudera.com <http://www.cloudera.com>
> > > >
> > > > [image: Cloudera] <http://www.cloudera.com/>
> > > >
> > > > [image: Cloudera on Twitter] <https://twitter.com/cloudera> [image:
> > > > Cloudera on Facebook] <https://www.facebook.com/cloudera> [image:
> > > Cloudera
> > > > on LinkedIn] <https://www.linkedin.com/company/cloudera>
> > > >
> > >
> >
>

Re: Future support to R

Posted by Alan Silva <ju...@apache.org>.

Hi,

One question here, if I will start with development about this R support at
our engine, I will make it based on that we have today or in this new
concept?

Because I understood the concepts change a lot, right?



On Mon, Oct 15, 2018 at 1:56 PM Rafael Novello <ra...@gmail.com>
wrote:

> Hi guys!
>
> Despite issues related to performance and scalability I believe we need
> to support R engines to cover the most part of data science/machine
> learning
> projects. Sometimes the code performance isn't the main question and the
> team performance (how fast the team can delivery some value) is more
> important.
>
> We are working (doing some POCs and trials) on a new toolbox concept to
> facilitate the inclusion of new languages support. We could discuss this in
> more details but I believe this kind of detail is not the focus for this
> thread.
>
> +1 for R support
>
> Best!
> Atenciosamente,
> Rafael J. R. Novello
>
> Skype: rafael.novello
> Blog: http://rafanovello.blogspot.com.br/
>
>
> Em qui, 11 de out de 2018 às 23:01, Lucas Bonatto Miguel <
> lucasbonatto@gmail.com> escreveu:
>
> > The idea is to create the R toolbox, providing a similar user experience
> > that we provide in Python today.
> >
> > Regards,
> > Lucas
> >
> > On Thu, Oct 11, 2018 at 6:30 PM Alan Silva <alan.silva@cloudera.com
> > .invalid>
> > wrote:
> >
> > > Hi to all,
> > >
> > > Yes, R would be a language like Python that I believe we should have in
> > our
> > > toolset and even as I had already raised it, I'm also very interested
> in
> > > working on this part of making a frontend with R.
> > >
> > > However, am I understanding after your last mail that the idea here is
> to
> > > do something different from what happens with Python today? Was not
> that
> > > clear to me? Or would it make a frontend similar to what we have in
> > Python
> > > and in the future for Scala too, correct?
> > >
> > >
> > > On Thu, Oct 11, 2018 at 8:21 PM Lucas Bonatto Miguel <
> > lucasbm88@apache.org
> > > >
> > > wrote:
> > >
> > > > Hi community, I already participated in several discussions with some
> > of
> > > > you regarding adding support to R on Marvin. Allowing users to write
> > > > engines in R would be a great step for Marvin since that's the
> primary
> > > > language for many statisticians, physicists and math people.
> > > >
> > > > One of the points that always arise during those conversations is the
> > > fact
> > > > that R is not fast nor scalable, and it would be hard to either make
> it
> > > > process large chunks of data for training or make it support high
> > > > throughput and have low latency for predictions. I want to start a
> > thread
> > > > on whether we should invest more time trying to add R support to
> Marvin
> > > or
> > > > not. I invite you all to read about this related work:
> > > > https://medium.com/graalvm/faster-r-with-fastr-4b8db0e0dceb
> > > >
> > > > I am looking forward to hearing opinions and ideas.
> > > >
> > > > Thanks,
> > > > Lucas
> > > >
> > >
> > >
> > > --
> > > *Alan Silva* | Solutions Architect
> > > t. +55 (11) 99951- <0000000000>5242 e. alan.silva@cloudera.com
> > > cloudera.com <http://www.cloudera.com>
> > >
> > > [image: Cloudera] <http://www.cloudera.com/>
> > >
> > > [image: Cloudera on Twitter] <https://twitter.com/cloudera> [image:
> > > Cloudera on Facebook] <https://www.facebook.com/cloudera> [image:
> > Cloudera
> > > on LinkedIn] <https://www.linkedin.com/company/cloudera>
> > >
> >
>

Re: Future support to R

Posted by Rafael Novello <ra...@gmail.com>.

Hi guys!

Despite issues related to performance and scalability I believe we need
to support R engines to cover the most part of data science/machine learning
projects. Sometimes the code performance isn't the main question and the
team performance (how fast the team can delivery some value) is more
important.

We are working (doing some POCs and trials) on a new toolbox concept to
facilitate the inclusion of new languages support. We could discuss this in
more details but I believe this kind of detail is not the focus for this
thread.

+1 for R support

Best!
Atenciosamente,
Rafael J. R. Novello

Skype: rafael.novello
Blog: http://rafanovello.blogspot.com.br/


Em qui, 11 de out de 2018 às 23:01, Lucas Bonatto Miguel <
lucasbonatto@gmail.com> escreveu:

> The idea is to create the R toolbox, providing a similar user experience
> that we provide in Python today.
>
> Regards,
> Lucas
>
> On Thu, Oct 11, 2018 at 6:30 PM Alan Silva <alan.silva@cloudera.com
> .invalid>
> wrote:
>
> > Hi to all,
> >
> > Yes, R would be a language like Python that I believe we should have in
> our
> > toolset and even as I had already raised it, I'm also very interested in
> > working on this part of making a frontend with R.
> >
> > However, am I understanding after your last mail that the idea here is to
> > do something different from what happens with Python today? Was not that
> > clear to me? Or would it make a frontend similar to what we have in
> Python
> > and in the future for Scala too, correct?
> >
> >
> > On Thu, Oct 11, 2018 at 8:21 PM Lucas Bonatto Miguel <
> lucasbm88@apache.org
> > >
> > wrote:
> >
> > > Hi community, I already participated in several discussions with some
> of
> > > you regarding adding support to R on Marvin. Allowing users to write
> > > engines in R would be a great step for Marvin since that's the primary
> > > language for many statisticians, physicists and math people.
> > >
> > > One of the points that always arise during those conversations is the
> > fact
> > > that R is not fast nor scalable, and it would be hard to either make it
> > > process large chunks of data for training or make it support high
> > > throughput and have low latency for predictions. I want to start a
> thread
> > > on whether we should invest more time trying to add R support to Marvin
> > or
> > > not. I invite you all to read about this related work:
> > > https://medium.com/graalvm/faster-r-with-fastr-4b8db0e0dceb
> > >
> > > I am looking forward to hearing opinions and ideas.
> > >
> > > Thanks,
> > > Lucas
> > >
> >
> >
> > --
> > *Alan Silva* | Solutions Architect
> > t. +55 (11) 99951- <0000000000>5242 e. alan.silva@cloudera.com
> > cloudera.com <http://www.cloudera.com>
> >
> > [image: Cloudera] <http://www.cloudera.com/>
> >
> > [image: Cloudera on Twitter] <https://twitter.com/cloudera> [image:
> > Cloudera on Facebook] <https://www.facebook.com/cloudera> [image:
> Cloudera
> > on LinkedIn] <https://www.linkedin.com/company/cloudera>
> >
>

Re: Future support to R

Posted by Lucas Bonatto Miguel <lu...@gmail.com>.

The idea is to create the R toolbox, providing a similar user experience
that we provide in Python today.

Regards,
Lucas

On Thu, Oct 11, 2018 at 6:30 PM Alan Silva <al...@cloudera.com.invalid>
wrote:

> Hi to all,
>
> Yes, R would be a language like Python that I believe we should have in our
> toolset and even as I had already raised it, I'm also very interested in
> working on this part of making a frontend with R.
>
> However, am I understanding after your last mail that the idea here is to
> do something different from what happens with Python today? Was not that
> clear to me? Or would it make a frontend similar to what we have in Python
> and in the future for Scala too, correct?
>
>
> On Thu, Oct 11, 2018 at 8:21 PM Lucas Bonatto Miguel <lucasbm88@apache.org
> >
> wrote:
>
> > Hi community, I already participated in several discussions with some of
> > you regarding adding support to R on Marvin. Allowing users to write
> > engines in R would be a great step for Marvin since that's the primary
> > language for many statisticians, physicists and math people.
> >
> > One of the points that always arise during those conversations is the
> fact
> > that R is not fast nor scalable, and it would be hard to either make it
> > process large chunks of data for training or make it support high
> > throughput and have low latency for predictions. I want to start a thread
> > on whether we should invest more time trying to add R support to Marvin
> or
> > not. I invite you all to read about this related work:
> > https://medium.com/graalvm/faster-r-with-fastr-4b8db0e0dceb
> >
> > I am looking forward to hearing opinions and ideas.
> >
> > Thanks,
> > Lucas
> >
>
>
> --
> *Alan Silva* | Solutions Architect
> t. +55 (11) 99951- <0000000000>5242 e. alan.silva@cloudera.com
> cloudera.com <http://www.cloudera.com>
>
> [image: Cloudera] <http://www.cloudera.com/>
>
> [image: Cloudera on Twitter] <https://twitter.com/cloudera> [image:
> Cloudera on Facebook] <https://www.facebook.com/cloudera> [image: Cloudera
> on LinkedIn] <https://www.linkedin.com/company/cloudera>
>

Re: Future support to R

Posted by Alan Silva <al...@cloudera.com.INVALID>.

Hi to all,

Yes, R would be a language like Python that I believe we should have in our
toolset and even as I had already raised it, I'm also very interested in
working on this part of making a frontend with R.

However, am I understanding after your last mail that the idea here is to
do something different from what happens with Python today? Was not that
clear to me? Or would it make a frontend similar to what we have in Python
and in the future for Scala too, correct?

On Thu, Oct 11, 2018 at 8:21 PM Lucas Bonatto Miguel <lu...@apache.org>
wrote:

> Hi community, I already participated in several discussions with some of
> you regarding adding support to R on Marvin. Allowing users to write
> engines in R would be a great step for Marvin since that's the primary
> language for many statisticians, physicists and math people.
>
> One of the points that always arise during those conversations is the fact
> that R is not fast nor scalable, and it would be hard to either make it
> process large chunks of data for training or make it support high
> throughput and have low latency for predictions. I want to start a thread
> on whether we should invest more time trying to add R support to Marvin or
> not. I invite you all to read about this related work:
> https://medium.com/graalvm/faster-r-with-fastr-4b8db0e0dceb
>
> I am looking forward to hearing opinions and ideas.
>
> Thanks,
> Lucas
>

-- 
*Alan Silva* | Solutions Architect
t. +55 (11) 99951- <0000000000>5242 e. alan.silva@cloudera.com
cloudera.com <http://www.cloudera.com>

[image: Cloudera] <http://www.cloudera.com/>

[image: Cloudera on Twitter] <https://twitter.com/cloudera> [image:
Cloudera on Facebook] <https://www.facebook.com/cloudera> [image: Cloudera
on LinkedIn] <https://www.linkedin.com/company/cloudera>