Python Client - Roadmap

[Jordan Birdsell <jo...@gmail.com>]

Hey folks,

I know I've thrown around a few ideas lately regarding the python client as
well as started trying to bring it up to speed with the rest of the kudu
world, so I thought i'd try and bring it all back together with a road map
of sorts and get you alls thoughts.

For starters, I've had a change in heart for now about a complete rebuild,
unless anyone objects. I think we can get the desired pythonic feel without
really any breaking changes and if there are any, we can phase out the old
functionality. I'm now favoring this over a full rebuild since we'd have to
support old/new for a while anyways.

Also, in the mean time, there is a lot of "catching up" to do to get on par
with everything else.  In this catch up phase, i'd like to propose that we
release outside of the ASF release schedule. Since the python package isn't
really tied to ASF releases this would help speed things up I think.

To put it into a list, heres kind of what I'm thinking:

   - Catch up python client with more rapid releases
   - Need to add a intro/example page on the site with some simple usage
   scenarios
   - Setup API doc generation in there somewhere
   - Improve dev ergonomics
   - Pyspark?

Thoughts?

Thanks,
Jordan

Re: Python Client - Roadmap

[Todd Lipcon <to...@cloudera.com>]

Sounds great. Thanks for all the work, Jordan! Hopefully we can recruit
some more Python devs from the community to lend a hand

-Todd

On Wed, Nov 9, 2016 at 3:48 PM, Jordan Birdsell <jt...@apache.org>
wrote:

> Hey folks,
>
> Just wanted to give an update on this.  With 1.1, the Python client will
> basically be at parity with the Java and C++ clients. There are a few minor
> things missing here and there, some of which i'm not sure really need
> exposed in Python anyways. The only missing piece that I'd like to
> eventually get in there is the async capabilities, however I dont think
> that will be all that pressing for python devs. Note that because it's now
> caught up, we'll be bumping the python client version to 1.1 (instead of
> 0.4) this release.
>
> Next steps for 1.2:
>
>    - Pydoc (top priority)
>    - Pandas integration
>    - Additional "pythonic" improvements
>    - Keep pace with other changes, particularly new security features
>
> Let me know if you have any opinions or questions.
>
> Thanks,
> Jordan
>
> On Thu, Oct 6, 2016 at 12:16 PM Todd Lipcon <to...@cloudera.com> wrote:
>
> > On Tue, Sep 27, 2016 at 4:04 PM, Jordan Birdsell <
> > jordantbirdsell@gmail.com>
> > wrote:
> >
> > > Hey folks,
> > >
> > > I know I've thrown around a few ideas lately regarding the python
> client
> > as
> > > well as started trying to bring it up to speed with the rest of the
> kudu
> > > world, so I thought i'd try and bring it all back together with a road
> > map
> > > of sorts and get you alls thoughts.
> > >
> > > For starters, I've had a change in heart for now about a complete
> > rebuild,
> > > unless anyone objects. I think we can get the desired pythonic feel
> > without
> > > really any breaking changes and if there are any, we can phase out the
> > old
> > > functionality. I'm now favoring this over a full rebuild since we'd
> have
> > to
> > > support old/new for a while anyways.
> > >
> > >
> > Yea, if there aren't breaking changes I think doing it incrementally is
> > probably the easier approach.
> >
> > Also, in the mean time, there is a lot of "catching up" to do to get on
> par
> > > with everything else.  In this catch up phase, i'd like to propose that
> > we
> > > release outside of the ASF release schedule. Since the python package
> > isn't
> > > really tied to ASF releases this would help speed things up I think.
> > >
> > >
> > As an ASF project we can't do official releases without going through a
> > normal VOTE process. That said, we're permitted to release multiple
> > different artifacts from the same project -- i.e we can call a vote for
> > just a Python client release if we like.
> >
> > That said, there are some potential issues: the Python client wraps the
> C++
> > one, and so if we were to release a new Python client today, it wouldn't
> > work with the older C++ client due to missing features (eg IN-list
> > predicate support). With enough effort this could be worked around,
> > probably (eg by adding #Ifdefs into the Python client to compile out
> things
> > that aren't present in your installed C++ one) but I don't know that it's
> > worth the effort.
> >
> > We generally have been doing releases once every 2-3 months, so my
> personal
> > preference is to just wait until the next project-wide release rather
> than
> > doing intermediary ones.
> >
> > -Todd
> > --
> > Todd Lipcon
> > Software Engineer, Cloudera
> >
>



-- 
Todd Lipcon
Software Engineer, Cloudera

Re: Python Client - Roadmap

[Jordan Birdsell <jt...@apache.org>]

Hey folks,

Just wanted to give an update on this.  With 1.1, the Python client will
basically be at parity with the Java and C++ clients. There are a few minor
things missing here and there, some of which i'm not sure really need
exposed in Python anyways. The only missing piece that I'd like to
eventually get in there is the async capabilities, however I dont think
that will be all that pressing for python devs. Note that because it's now
caught up, we'll be bumping the python client version to 1.1 (instead of
0.4) this release.

Next steps for 1.2:

   - Pydoc (top priority)
   - Pandas integration
   - Additional "pythonic" improvements
   - Keep pace with other changes, particularly new security features

Let me know if you have any opinions or questions.

Thanks,
Jordan

On Thu, Oct 6, 2016 at 12:16 PM Todd Lipcon <to...@cloudera.com> wrote:

> On Tue, Sep 27, 2016 at 4:04 PM, Jordan Birdsell <
> jordantbirdsell@gmail.com>
> wrote:
>
> > Hey folks,
> >
> > I know I've thrown around a few ideas lately regarding the python client
> as
> > well as started trying to bring it up to speed with the rest of the kudu
> > world, so I thought i'd try and bring it all back together with a road
> map
> > of sorts and get you alls thoughts.
> >
> > For starters, I've had a change in heart for now about a complete
> rebuild,
> > unless anyone objects. I think we can get the desired pythonic feel
> without
> > really any breaking changes and if there are any, we can phase out the
> old
> > functionality. I'm now favoring this over a full rebuild since we'd have
> to
> > support old/new for a while anyways.
> >
> >
> Yea, if there aren't breaking changes I think doing it incrementally is
> probably the easier approach.
>
> Also, in the mean time, there is a lot of "catching up" to do to get on par
> > with everything else.  In this catch up phase, i'd like to propose that
> we
> > release outside of the ASF release schedule. Since the python package
> isn't
> > really tied to ASF releases this would help speed things up I think.
> >
> >
> As an ASF project we can't do official releases without going through a
> normal VOTE process. That said, we're permitted to release multiple
> different artifacts from the same project -- i.e we can call a vote for
> just a Python client release if we like.
>
> That said, there are some potential issues: the Python client wraps the C++
> one, and so if we were to release a new Python client today, it wouldn't
> work with the older C++ client due to missing features (eg IN-list
> predicate support). With enough effort this could be worked around,
> probably (eg by adding #Ifdefs into the Python client to compile out things
> that aren't present in your installed C++ one) but I don't know that it's
> worth the effort.
>
> We generally have been doing releases once every 2-3 months, so my personal
> preference is to just wait until the next project-wide release rather than
> doing intermediary ones.
>
> -Todd
> --
> Todd Lipcon
> Software Engineer, Cloudera
>

Re: Python Client - Roadmap

[Todd Lipcon <to...@cloudera.com>]

On Tue, Sep 27, 2016 at 4:04 PM, Jordan Birdsell <jo...@gmail.com>
wrote:

> Hey folks,
>
> I know I've thrown around a few ideas lately regarding the python client as
> well as started trying to bring it up to speed with the rest of the kudu
> world, so I thought i'd try and bring it all back together with a road map
> of sorts and get you alls thoughts.
>
> For starters, I've had a change in heart for now about a complete rebuild,
> unless anyone objects. I think we can get the desired pythonic feel without
> really any breaking changes and if there are any, we can phase out the old
> functionality. I'm now favoring this over a full rebuild since we'd have to
> support old/new for a while anyways.
>
>
Yea, if there aren't breaking changes I think doing it incrementally is
probably the easier approach.

Also, in the mean time, there is a lot of "catching up" to do to get on par
> with everything else.  In this catch up phase, i'd like to propose that we
> release outside of the ASF release schedule. Since the python package isn't
> really tied to ASF releases this would help speed things up I think.
>
>
As an ASF project we can't do official releases without going through a
normal VOTE process. That said, we're permitted to release multiple
different artifacts from the same project -- i.e we can call a vote for
just a Python client release if we like.

That said, there are some potential issues: the Python client wraps the C++
one, and so if we were to release a new Python client today, it wouldn't
work with the older C++ client due to missing features (eg IN-list
predicate support). With enough effort this could be worked around,
probably (eg by adding #Ifdefs into the Python client to compile out things
that aren't present in your installed C++ one) but I don't know that it's
worth the effort.

We generally have been doing releases once every 2-3 months, so my personal
preference is to just wait until the next project-wide release rather than
doing intermediary ones.

-Todd
-- 
Todd Lipcon
Software Engineer, Cloudera