[DISCUSS] Review and discussions thread for Amaterasu-45 PR

[Nadav Har Tzvi <na...@gmail.com>]

Hey everyone,

I opened a draft PR for Amaterasu 45 (Python SDK, pyspark runtime, python
runtime). Do mind that this is still a WIP. I would like you to review this
PR from time to time as it evolves. The sooner I get inputs, the sooner
this feature will be out.

PR: https://github.com/apache/incubator-amaterasu/pull/44

Yaniv, Eyal, Arun. Please let's now define the final MVP of this feature so
I can know which reviews to defer and which to attend to before this PR is
closed and merged.

Arun, this is most important, if I ended up implementing a design that is
vastly different than the scala runtime design, we should talk ASAP.

I would like the community to give any feedback. Did I miss something? Did
I make some stupid mistake? Does the design look awkward? etc.

Also a note on the python runtime. It is currently February 2019. As of
January 1st 2020 Python2 is EOL. Many third party Python packages already
dropped support entirely for Python2. Because of that, the entire SDK and
runtimes are implemented in Python >=3.4 (Python <= 3.3.x is EOL)
If you think this is problematic, please raise a flag.

Cheers,
Nadav

Re: [DISCUSS] Review and discussions thread for Amaterasu-45 PR

[Nadav Har Tzvi <na...@gmail.com>]

Ok,
I will do an E2E test on a pyspark only flow tonight. I think that after
rebasing onto master with the changes from Amaterasu-8, I will have to make
some adjustments in regard to the integration between the leader and the
executor. If there are any, we should be coordinated.

Cheers,
Nadav



On Wed, 20 Feb 2019 at 13:10, Arun Manivannan <ar...@arunma.com> wrote:

> Hi Nadav,
>
> Had a look at the datasets.yml and the bindings for it.  This aligns
> perfectly with the consensus that we had on the call that we had the day
> before.
>
> I'll have it implemented for the JVM by the end of this week.
>
> Cheers,
> Arun
>
> On Wed, Feb 20, 2019 at 3:12 AM Nadav Har Tzvi <na...@gmail.com>
> wrote:
>
> > Hey everyone,
> >
> > I opened a draft PR for Amaterasu 45 (Python SDK, pyspark runtime, python
> > runtime). Do mind that this is still a WIP. I would like you to review
> this
> > PR from time to time as it evolves. The sooner I get inputs, the sooner
> > this feature will be out.
> >
> > PR: https://github.com/apache/incubator-amaterasu/pull/44
> >
> > Yaniv, Eyal, Arun. Please let's now define the final MVP of this feature
> so
> > I can know which reviews to defer and which to attend to before this PR
> is
> > closed and merged.
> >
> > Arun, this is most important, if I ended up implementing a design that is
> > vastly different than the scala runtime design, we should talk ASAP.
> >
> > I would like the community to give any feedback. Did I miss something?
> Did
> > I make some stupid mistake? Does the design look awkward? etc.
> >
> > Also a note on the python runtime. It is currently February 2019. As of
> > January 1st 2020 Python2 is EOL. Many third party Python packages already
> > dropped support entirely for Python2. Because of that, the entire SDK and
> > runtimes are implemented in Python >=3.4 (Python <= 3.3.x is EOL)
> > If you think this is problematic, please raise a flag.
> >
> > Cheers,
> > Nadav
> >
>

Re: [DISCUSS] Review and discussions thread for Amaterasu-45 PR

[Arun Manivannan <ar...@arunma.com>]

Hi Nadav,

Had a look at the datasets.yml and the bindings for it.  This aligns
perfectly with the consensus that we had on the call that we had the day
before.

I'll have it implemented for the JVM by the end of this week.

Cheers,
Arun

On Wed, Feb 20, 2019 at 3:12 AM Nadav Har Tzvi <na...@gmail.com>
wrote:

> Hey everyone,
>
> I opened a draft PR for Amaterasu 45 (Python SDK, pyspark runtime, python
> runtime). Do mind that this is still a WIP. I would like you to review this
> PR from time to time as it evolves. The sooner I get inputs, the sooner
> this feature will be out.
>
> PR: https://github.com/apache/incubator-amaterasu/pull/44
>
> Yaniv, Eyal, Arun. Please let's now define the final MVP of this feature so
> I can know which reviews to defer and which to attend to before this PR is
> closed and merged.
>
> Arun, this is most important, if I ended up implementing a design that is
> vastly different than the scala runtime design, we should talk ASAP.
>
> I would like the community to give any feedback. Did I miss something? Did
> I make some stupid mistake? Does the design look awkward? etc.
>
> Also a note on the python runtime. It is currently February 2019. As of
> January 1st 2020 Python2 is EOL. Many third party Python packages already
> dropped support entirely for Python2. Because of that, the entire SDK and
> runtimes are implemented in Python >=3.4 (Python <= 3.3.x is EOL)
> If you think this is problematic, please raise a flag.
>
> Cheers,
> Nadav
>