You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kudu.apache.org by David Alves <da...@gmail.com> on 2016/03/10 00:49:32 UTC
Kudu python client versioning
Hi All
We'll need to release a new version of the python client to pypi and we
were discussing versioning strategies.
Currently the python's client version does not match the version of the
c++ client/server (0.1.0 versus 0.7.1) and we can either stick with that
and keep different versions, or change it to match the latest Kudu release.
Making the versions match would have the advantage of making it clear to
developers if the server/client versions match or not, which might help
weeding out compatibility bugs. On the other hand keeping different
versions would likely make our lives easier if we want to release the
python client at a different cadence of regular Kudu releases.
What do people think?
Best
David
Re: Kudu python client versioning
Posted by Jean-Daniel Cryans <jd...@apache.org>.
Hey David,
I just checked in pypi and didn't see the new version. You still planning
on doing this?
Thx,
J-D
On Tue, Mar 15, 2016 at 10:43 AM, David Alves <da...@gmail.com> wrote:
> Thanks, Wes.
> Ok, I'll bump the version to 0.2.0 and release what we have, which hasn't
> changed since the last kudu release.
> If we go a few releases without needing different releases for the python
> client then we can make it match.
> I'll do the release to pypi today.
>
> -david
>
> On Tue, Mar 15, 2016 at 10:39 AM, Wes McKinney <we...@cloudera.com> wrote:
>
> > I'm fine with releasing them at the same time, but I would propose
> > that we use separate version numbering to avoid potential confusion in
> > the future should the releases decouple. If different version numbers
> > cause some concern, then using the same number for now is OK as long
> > as we aren't closing the door on faster releases (for example: urgent
> > bug fixes — I suspect that the Python client will in general have
> > strictly more bugs than the C++ client) in the future.
> >
> > On Tue, Mar 15, 2016 at 10:32 AM, David Alves <da...@gmail.com>
> > wrote:
> > > If we're bumping and if everyone agrees I propose we just bum the
> version
> > > to match kudu and, for now at least, go with the plan of releasing the
> > > python client at the same time as the rest of kudu.
> > > Wes: you ok with that?
> > >
> > > -david
> > >
> > > On Mon, Mar 14, 2016 at 12:40 PM, Todd Lipcon <to...@cloudera.com>
> wrote:
> > >
> > >> So, right now our PyPI package is incompatible with the latest
> releases
> > >> due to some client ABI changes. For now, I'd propose we bump the
> > version to
> > >> 0.2.0.
> > >>
> > >> Regarding release cadence, keep in mind that we need to do a vote
> before
> > >> any public release (even if it's just the python client). So, I'm not
> > sure
> > >> how realistic it is to have a significantly faster python release
> > cadence
> > >> than the rest of the software.
> > >>
> > >> -Todd
> > >>
> > >> On Wed, Mar 9, 2016 at 4:27 PM, Adar Dembo <ad...@cloudera.com> wrote:
> > >>
> > >>> Wes and I discussed this very issue in
> > >>> http://gerrit.cloudera.org:8080/#/c/1593 two months ago.
> > >>>
> > >>> On Wed, Mar 9, 2016 at 4:25 PM, Casey Ching <ca...@cloudera.com>
> > wrote:
> > >>>
> > >>> > I’d also go with separating the version numbers. Hopefully
> eventually
> > >>> the
> > >>> > client api will be stable and won’t change much, then a new Kudu
> > release
> > >>> > won’t require a python release. If the version are made to match,
> > there
> > >>> > should come a time when the python version number needs to be
> bumped
> > >>> just
> > >>> > to keep the versions in sync though no actual python/client changes
> > were
> > >>> > made.
> > >>> >
> > >>> > If you are worried about client/server compatibility, maybe there
> is
> > >>> > something that can be done such that the client/server verify that
> > they
> > >>> are
> > >>> > in fact compatible through some api call.
> > >>> >
> > >>> > Casey
> > >>> > On March 9, 2016 at 4:10:40 PM, David Alves (davidralves@gmail.com
> )
> > >>> wrote:
> > >>> >
> > >>> > We intend to produce Kudu releases every couple of months.
> > >>> > It's also worth noting that while we can keep a different release
> > >>> cadence
> > >>> > for the python client we still need to go through a vote and do a
> > proper
> > >>> > release.
> > >>> > Do you feel like its worth the additional work to increase the
> > cadence
> > >>> or
> > >>> > do you feel that 2 months is likely enough?
> > >>> > As an alternative we could have a primary plan to make the
> > major/minor
> > >>> > versions match and release in the same cadence, but reserve the
> right
> > >>> to do
> > >>> > ad-hoc micro releases occasionally.
> > >>> > Thoughts?
> > >>> >
> > >>> > Best
> > >>> > David
> > >>> >
> > >>> > On Wed, Mar 9, 2016 at 3:56 PM, Wes McKinney <we...@cloudera.com>
> > wrote:
> > >>> >
> > >>> > > Since I expect the Python client to evolve faster from a user
> > >>> > > perspective (as opposed to Kudu core, where users will see
> > basically a
> > >>> > > stable client API and wire protocol in general) I'd prefer to
> make
> > >>> > > Python releases separate from Kudu major/minor releases. For
> > example,
> > >>> > > sometime in the next couple months I would like to add low-level
> > >>> > > (C-level) interoperability between pandas, NumPy, and Kudu. It
> > would
> > >>> > > be nice to be able to ship this basically as soon as it's
> > available in
> > >>> > > trunk. This is somewhat independent from the version numbering
> > >>> > > question, but it would probably be better for the Python client
> and
> > >>> > > interop tools to have their own version number. If the Kudu
> > committers
> > >>> > > feel otherwise, it's not an issue -- some of this depends on the
> > >>> > > expected release cadence of Kudu itself.
> > >>> > >
> > >>> > > best,
> > >>> > > Wes
> > >>> > >
> > >>> > > On Wed, Mar 9, 2016 at 3:49 PM, David Alves <
> davidralves@gmail.com
> > >
> > >>> > wrote:
> > >>> > > > Hi All
> > >>> > > >
> > >>> > > > We'll need to release a new version of the python client to
> pypi
> > >>> and we
> > >>> > > > were discussing versioning strategies.
> > >>> > > > Currently the python's client version does not match the
> version
> > of
> > >>> the
> > >>> > > > c++ client/server (0.1.0 versus 0.7.1) and we can either stick
> > with
> > >>> > that
> > >>> > > > and keep different versions, or change it to match the latest
> > Kudu
> > >>> > > release.
> > >>> > > > Making the versions match would have the advantage of making it
> > >>> clear
> > >>> > > to
> > >>> > > > developers if the server/client versions match or not, which
> > might
> > >>> help
> > >>> > > > weeding out compatibility bugs. On the other hand keeping
> > different
> > >>> > > > versions would likely make our lives easier if we want to
> release
> > >>> the
> > >>> > > > python client at a different cadence of regular Kudu releases.
> > >>> > > > What do people think?
> > >>> > > >
> > >>> > > > Best
> > >>> > > > David
> > >>> > >
> > >>> >
> > >>>
> > >>
> > >>
> > >>
> > >> --
> > >> Todd Lipcon
> > >> Software Engineer, Cloudera
> > >>
> >
>
Re: Kudu python client versioning
Posted by David Alves <da...@gmail.com>.
Thanks, Wes.
Ok, I'll bump the version to 0.2.0 and release what we have, which hasn't
changed since the last kudu release.
If we go a few releases without needing different releases for the python
client then we can make it match.
I'll do the release to pypi today.
-david
On Tue, Mar 15, 2016 at 10:39 AM, Wes McKinney <we...@cloudera.com> wrote:
> I'm fine with releasing them at the same time, but I would propose
> that we use separate version numbering to avoid potential confusion in
> the future should the releases decouple. If different version numbers
> cause some concern, then using the same number for now is OK as long
> as we aren't closing the door on faster releases (for example: urgent
> bug fixes — I suspect that the Python client will in general have
> strictly more bugs than the C++ client) in the future.
>
> On Tue, Mar 15, 2016 at 10:32 AM, David Alves <da...@gmail.com>
> wrote:
> > If we're bumping and if everyone agrees I propose we just bum the version
> > to match kudu and, for now at least, go with the plan of releasing the
> > python client at the same time as the rest of kudu.
> > Wes: you ok with that?
> >
> > -david
> >
> > On Mon, Mar 14, 2016 at 12:40 PM, Todd Lipcon <to...@cloudera.com> wrote:
> >
> >> So, right now our PyPI package is incompatible with the latest releases
> >> due to some client ABI changes. For now, I'd propose we bump the
> version to
> >> 0.2.0.
> >>
> >> Regarding release cadence, keep in mind that we need to do a vote before
> >> any public release (even if it's just the python client). So, I'm not
> sure
> >> how realistic it is to have a significantly faster python release
> cadence
> >> than the rest of the software.
> >>
> >> -Todd
> >>
> >> On Wed, Mar 9, 2016 at 4:27 PM, Adar Dembo <ad...@cloudera.com> wrote:
> >>
> >>> Wes and I discussed this very issue in
> >>> http://gerrit.cloudera.org:8080/#/c/1593 two months ago.
> >>>
> >>> On Wed, Mar 9, 2016 at 4:25 PM, Casey Ching <ca...@cloudera.com>
> wrote:
> >>>
> >>> > I’d also go with separating the version numbers. Hopefully eventually
> >>> the
> >>> > client api will be stable and won’t change much, then a new Kudu
> release
> >>> > won’t require a python release. If the version are made to match,
> there
> >>> > should come a time when the python version number needs to be bumped
> >>> just
> >>> > to keep the versions in sync though no actual python/client changes
> were
> >>> > made.
> >>> >
> >>> > If you are worried about client/server compatibility, maybe there is
> >>> > something that can be done such that the client/server verify that
> they
> >>> are
> >>> > in fact compatible through some api call.
> >>> >
> >>> > Casey
> >>> > On March 9, 2016 at 4:10:40 PM, David Alves (davidralves@gmail.com)
> >>> wrote:
> >>> >
> >>> > We intend to produce Kudu releases every couple of months.
> >>> > It's also worth noting that while we can keep a different release
> >>> cadence
> >>> > for the python client we still need to go through a vote and do a
> proper
> >>> > release.
> >>> > Do you feel like its worth the additional work to increase the
> cadence
> >>> or
> >>> > do you feel that 2 months is likely enough?
> >>> > As an alternative we could have a primary plan to make the
> major/minor
> >>> > versions match and release in the same cadence, but reserve the right
> >>> to do
> >>> > ad-hoc micro releases occasionally.
> >>> > Thoughts?
> >>> >
> >>> > Best
> >>> > David
> >>> >
> >>> > On Wed, Mar 9, 2016 at 3:56 PM, Wes McKinney <we...@cloudera.com>
> wrote:
> >>> >
> >>> > > Since I expect the Python client to evolve faster from a user
> >>> > > perspective (as opposed to Kudu core, where users will see
> basically a
> >>> > > stable client API and wire protocol in general) I'd prefer to make
> >>> > > Python releases separate from Kudu major/minor releases. For
> example,
> >>> > > sometime in the next couple months I would like to add low-level
> >>> > > (C-level) interoperability between pandas, NumPy, and Kudu. It
> would
> >>> > > be nice to be able to ship this basically as soon as it's
> available in
> >>> > > trunk. This is somewhat independent from the version numbering
> >>> > > question, but it would probably be better for the Python client and
> >>> > > interop tools to have their own version number. If the Kudu
> committers
> >>> > > feel otherwise, it's not an issue -- some of this depends on the
> >>> > > expected release cadence of Kudu itself.
> >>> > >
> >>> > > best,
> >>> > > Wes
> >>> > >
> >>> > > On Wed, Mar 9, 2016 at 3:49 PM, David Alves <davidralves@gmail.com
> >
> >>> > wrote:
> >>> > > > Hi All
> >>> > > >
> >>> > > > We'll need to release a new version of the python client to pypi
> >>> and we
> >>> > > > were discussing versioning strategies.
> >>> > > > Currently the python's client version does not match the version
> of
> >>> the
> >>> > > > c++ client/server (0.1.0 versus 0.7.1) and we can either stick
> with
> >>> > that
> >>> > > > and keep different versions, or change it to match the latest
> Kudu
> >>> > > release.
> >>> > > > Making the versions match would have the advantage of making it
> >>> clear
> >>> > > to
> >>> > > > developers if the server/client versions match or not, which
> might
> >>> help
> >>> > > > weeding out compatibility bugs. On the other hand keeping
> different
> >>> > > > versions would likely make our lives easier if we want to release
> >>> the
> >>> > > > python client at a different cadence of regular Kudu releases.
> >>> > > > What do people think?
> >>> > > >
> >>> > > > Best
> >>> > > > David
> >>> > >
> >>> >
> >>>
> >>
> >>
> >>
> >> --
> >> Todd Lipcon
> >> Software Engineer, Cloudera
> >>
>
Re: Kudu python client versioning
Posted by Wes McKinney <we...@cloudera.com>.
I'm fine with releasing them at the same time, but I would propose
that we use separate version numbering to avoid potential confusion in
the future should the releases decouple. If different version numbers
cause some concern, then using the same number for now is OK as long
as we aren't closing the door on faster releases (for example: urgent
bug fixes — I suspect that the Python client will in general have
strictly more bugs than the C++ client) in the future.
On Tue, Mar 15, 2016 at 10:32 AM, David Alves <da...@gmail.com> wrote:
> If we're bumping and if everyone agrees I propose we just bum the version
> to match kudu and, for now at least, go with the plan of releasing the
> python client at the same time as the rest of kudu.
> Wes: you ok with that?
>
> -david
>
> On Mon, Mar 14, 2016 at 12:40 PM, Todd Lipcon <to...@cloudera.com> wrote:
>
>> So, right now our PyPI package is incompatible with the latest releases
>> due to some client ABI changes. For now, I'd propose we bump the version to
>> 0.2.0.
>>
>> Regarding release cadence, keep in mind that we need to do a vote before
>> any public release (even if it's just the python client). So, I'm not sure
>> how realistic it is to have a significantly faster python release cadence
>> than the rest of the software.
>>
>> -Todd
>>
>> On Wed, Mar 9, 2016 at 4:27 PM, Adar Dembo <ad...@cloudera.com> wrote:
>>
>>> Wes and I discussed this very issue in
>>> http://gerrit.cloudera.org:8080/#/c/1593 two months ago.
>>>
>>> On Wed, Mar 9, 2016 at 4:25 PM, Casey Ching <ca...@cloudera.com> wrote:
>>>
>>> > I’d also go with separating the version numbers. Hopefully eventually
>>> the
>>> > client api will be stable and won’t change much, then a new Kudu release
>>> > won’t require a python release. If the version are made to match, there
>>> > should come a time when the python version number needs to be bumped
>>> just
>>> > to keep the versions in sync though no actual python/client changes were
>>> > made.
>>> >
>>> > If you are worried about client/server compatibility, maybe there is
>>> > something that can be done such that the client/server verify that they
>>> are
>>> > in fact compatible through some api call.
>>> >
>>> > Casey
>>> > On March 9, 2016 at 4:10:40 PM, David Alves (davidralves@gmail.com)
>>> wrote:
>>> >
>>> > We intend to produce Kudu releases every couple of months.
>>> > It's also worth noting that while we can keep a different release
>>> cadence
>>> > for the python client we still need to go through a vote and do a proper
>>> > release.
>>> > Do you feel like its worth the additional work to increase the cadence
>>> or
>>> > do you feel that 2 months is likely enough?
>>> > As an alternative we could have a primary plan to make the major/minor
>>> > versions match and release in the same cadence, but reserve the right
>>> to do
>>> > ad-hoc micro releases occasionally.
>>> > Thoughts?
>>> >
>>> > Best
>>> > David
>>> >
>>> > On Wed, Mar 9, 2016 at 3:56 PM, Wes McKinney <we...@cloudera.com> wrote:
>>> >
>>> > > Since I expect the Python client to evolve faster from a user
>>> > > perspective (as opposed to Kudu core, where users will see basically a
>>> > > stable client API and wire protocol in general) I'd prefer to make
>>> > > Python releases separate from Kudu major/minor releases. For example,
>>> > > sometime in the next couple months I would like to add low-level
>>> > > (C-level) interoperability between pandas, NumPy, and Kudu. It would
>>> > > be nice to be able to ship this basically as soon as it's available in
>>> > > trunk. This is somewhat independent from the version numbering
>>> > > question, but it would probably be better for the Python client and
>>> > > interop tools to have their own version number. If the Kudu committers
>>> > > feel otherwise, it's not an issue -- some of this depends on the
>>> > > expected release cadence of Kudu itself.
>>> > >
>>> > > best,
>>> > > Wes
>>> > >
>>> > > On Wed, Mar 9, 2016 at 3:49 PM, David Alves <da...@gmail.com>
>>> > wrote:
>>> > > > Hi All
>>> > > >
>>> > > > We'll need to release a new version of the python client to pypi
>>> and we
>>> > > > were discussing versioning strategies.
>>> > > > Currently the python's client version does not match the version of
>>> the
>>> > > > c++ client/server (0.1.0 versus 0.7.1) and we can either stick with
>>> > that
>>> > > > and keep different versions, or change it to match the latest Kudu
>>> > > release.
>>> > > > Making the versions match would have the advantage of making it
>>> clear
>>> > > to
>>> > > > developers if the server/client versions match or not, which might
>>> help
>>> > > > weeding out compatibility bugs. On the other hand keeping different
>>> > > > versions would likely make our lives easier if we want to release
>>> the
>>> > > > python client at a different cadence of regular Kudu releases.
>>> > > > What do people think?
>>> > > >
>>> > > > Best
>>> > > > David
>>> > >
>>> >
>>>
>>
>>
>>
>> --
>> Todd Lipcon
>> Software Engineer, Cloudera
>>
Re: Kudu python client versioning
Posted by David Alves <da...@gmail.com>.
If we're bumping and if everyone agrees I propose we just bum the version
to match kudu and, for now at least, go with the plan of releasing the
python client at the same time as the rest of kudu.
Wes: you ok with that?
-david
On Mon, Mar 14, 2016 at 12:40 PM, Todd Lipcon <to...@cloudera.com> wrote:
> So, right now our PyPI package is incompatible with the latest releases
> due to some client ABI changes. For now, I'd propose we bump the version to
> 0.2.0.
>
> Regarding release cadence, keep in mind that we need to do a vote before
> any public release (even if it's just the python client). So, I'm not sure
> how realistic it is to have a significantly faster python release cadence
> than the rest of the software.
>
> -Todd
>
> On Wed, Mar 9, 2016 at 4:27 PM, Adar Dembo <ad...@cloudera.com> wrote:
>
>> Wes and I discussed this very issue in
>> http://gerrit.cloudera.org:8080/#/c/1593 two months ago.
>>
>> On Wed, Mar 9, 2016 at 4:25 PM, Casey Ching <ca...@cloudera.com> wrote:
>>
>> > I’d also go with separating the version numbers. Hopefully eventually
>> the
>> > client api will be stable and won’t change much, then a new Kudu release
>> > won’t require a python release. If the version are made to match, there
>> > should come a time when the python version number needs to be bumped
>> just
>> > to keep the versions in sync though no actual python/client changes were
>> > made.
>> >
>> > If you are worried about client/server compatibility, maybe there is
>> > something that can be done such that the client/server verify that they
>> are
>> > in fact compatible through some api call.
>> >
>> > Casey
>> > On March 9, 2016 at 4:10:40 PM, David Alves (davidralves@gmail.com)
>> wrote:
>> >
>> > We intend to produce Kudu releases every couple of months.
>> > It's also worth noting that while we can keep a different release
>> cadence
>> > for the python client we still need to go through a vote and do a proper
>> > release.
>> > Do you feel like its worth the additional work to increase the cadence
>> or
>> > do you feel that 2 months is likely enough?
>> > As an alternative we could have a primary plan to make the major/minor
>> > versions match and release in the same cadence, but reserve the right
>> to do
>> > ad-hoc micro releases occasionally.
>> > Thoughts?
>> >
>> > Best
>> > David
>> >
>> > On Wed, Mar 9, 2016 at 3:56 PM, Wes McKinney <we...@cloudera.com> wrote:
>> >
>> > > Since I expect the Python client to evolve faster from a user
>> > > perspective (as opposed to Kudu core, where users will see basically a
>> > > stable client API and wire protocol in general) I'd prefer to make
>> > > Python releases separate from Kudu major/minor releases. For example,
>> > > sometime in the next couple months I would like to add low-level
>> > > (C-level) interoperability between pandas, NumPy, and Kudu. It would
>> > > be nice to be able to ship this basically as soon as it's available in
>> > > trunk. This is somewhat independent from the version numbering
>> > > question, but it would probably be better for the Python client and
>> > > interop tools to have their own version number. If the Kudu committers
>> > > feel otherwise, it's not an issue -- some of this depends on the
>> > > expected release cadence of Kudu itself.
>> > >
>> > > best,
>> > > Wes
>> > >
>> > > On Wed, Mar 9, 2016 at 3:49 PM, David Alves <da...@gmail.com>
>> > wrote:
>> > > > Hi All
>> > > >
>> > > > We'll need to release a new version of the python client to pypi
>> and we
>> > > > were discussing versioning strategies.
>> > > > Currently the python's client version does not match the version of
>> the
>> > > > c++ client/server (0.1.0 versus 0.7.1) and we can either stick with
>> > that
>> > > > and keep different versions, or change it to match the latest Kudu
>> > > release.
>> > > > Making the versions match would have the advantage of making it
>> clear
>> > > to
>> > > > developers if the server/client versions match or not, which might
>> help
>> > > > weeding out compatibility bugs. On the other hand keeping different
>> > > > versions would likely make our lives easier if we want to release
>> the
>> > > > python client at a different cadence of regular Kudu releases.
>> > > > What do people think?
>> > > >
>> > > > Best
>> > > > David
>> > >
>> >
>>
>
>
>
> --
> Todd Lipcon
> Software Engineer, Cloudera
>
Re: Kudu python client versioning
Posted by Todd Lipcon <to...@cloudera.com>.
So, right now our PyPI package is incompatible with the latest releases due
to some client ABI changes. For now, I'd propose we bump the version to
0.2.0.
Regarding release cadence, keep in mind that we need to do a vote before
any public release (even if it's just the python client). So, I'm not sure
how realistic it is to have a significantly faster python release cadence
than the rest of the software.
-Todd
On Wed, Mar 9, 2016 at 4:27 PM, Adar Dembo <ad...@cloudera.com> wrote:
> Wes and I discussed this very issue in
> http://gerrit.cloudera.org:8080/#/c/1593 two months ago.
>
> On Wed, Mar 9, 2016 at 4:25 PM, Casey Ching <ca...@cloudera.com> wrote:
>
> > I’d also go with separating the version numbers. Hopefully eventually the
> > client api will be stable and won’t change much, then a new Kudu release
> > won’t require a python release. If the version are made to match, there
> > should come a time when the python version number needs to be bumped just
> > to keep the versions in sync though no actual python/client changes were
> > made.
> >
> > If you are worried about client/server compatibility, maybe there is
> > something that can be done such that the client/server verify that they
> are
> > in fact compatible through some api call.
> >
> > Casey
> > On March 9, 2016 at 4:10:40 PM, David Alves (davidralves@gmail.com)
> wrote:
> >
> > We intend to produce Kudu releases every couple of months.
> > It's also worth noting that while we can keep a different release cadence
> > for the python client we still need to go through a vote and do a proper
> > release.
> > Do you feel like its worth the additional work to increase the cadence or
> > do you feel that 2 months is likely enough?
> > As an alternative we could have a primary plan to make the major/minor
> > versions match and release in the same cadence, but reserve the right to
> do
> > ad-hoc micro releases occasionally.
> > Thoughts?
> >
> > Best
> > David
> >
> > On Wed, Mar 9, 2016 at 3:56 PM, Wes McKinney <we...@cloudera.com> wrote:
> >
> > > Since I expect the Python client to evolve faster from a user
> > > perspective (as opposed to Kudu core, where users will see basically a
> > > stable client API and wire protocol in general) I'd prefer to make
> > > Python releases separate from Kudu major/minor releases. For example,
> > > sometime in the next couple months I would like to add low-level
> > > (C-level) interoperability between pandas, NumPy, and Kudu. It would
> > > be nice to be able to ship this basically as soon as it's available in
> > > trunk. This is somewhat independent from the version numbering
> > > question, but it would probably be better for the Python client and
> > > interop tools to have their own version number. If the Kudu committers
> > > feel otherwise, it's not an issue -- some of this depends on the
> > > expected release cadence of Kudu itself.
> > >
> > > best,
> > > Wes
> > >
> > > On Wed, Mar 9, 2016 at 3:49 PM, David Alves <da...@gmail.com>
> > wrote:
> > > > Hi All
> > > >
> > > > We'll need to release a new version of the python client to pypi and
> we
> > > > were discussing versioning strategies.
> > > > Currently the python's client version does not match the version of
> the
> > > > c++ client/server (0.1.0 versus 0.7.1) and we can either stick with
> > that
> > > > and keep different versions, or change it to match the latest Kudu
> > > release.
> > > > Making the versions match would have the advantage of making it clear
> > > to
> > > > developers if the server/client versions match or not, which might
> help
> > > > weeding out compatibility bugs. On the other hand keeping different
> > > > versions would likely make our lives easier if we want to release the
> > > > python client at a different cadence of regular Kudu releases.
> > > > What do people think?
> > > >
> > > > Best
> > > > David
> > >
> >
>
--
Todd Lipcon
Software Engineer, Cloudera
Re: Kudu python client versioning
Posted by Adar Dembo <ad...@cloudera.com>.
Wes and I discussed this very issue in
http://gerrit.cloudera.org:8080/#/c/1593 two months ago.
On Wed, Mar 9, 2016 at 4:25 PM, Casey Ching <ca...@cloudera.com> wrote:
> I’d also go with separating the version numbers. Hopefully eventually the
> client api will be stable and won’t change much, then a new Kudu release
> won’t require a python release. If the version are made to match, there
> should come a time when the python version number needs to be bumped just
> to keep the versions in sync though no actual python/client changes were
> made.
>
> If you are worried about client/server compatibility, maybe there is
> something that can be done such that the client/server verify that they are
> in fact compatible through some api call.
>
> Casey
> On March 9, 2016 at 4:10:40 PM, David Alves (davidralves@gmail.com) wrote:
>
> We intend to produce Kudu releases every couple of months.
> It's also worth noting that while we can keep a different release cadence
> for the python client we still need to go through a vote and do a proper
> release.
> Do you feel like its worth the additional work to increase the cadence or
> do you feel that 2 months is likely enough?
> As an alternative we could have a primary plan to make the major/minor
> versions match and release in the same cadence, but reserve the right to do
> ad-hoc micro releases occasionally.
> Thoughts?
>
> Best
> David
>
> On Wed, Mar 9, 2016 at 3:56 PM, Wes McKinney <we...@cloudera.com> wrote:
>
> > Since I expect the Python client to evolve faster from a user
> > perspective (as opposed to Kudu core, where users will see basically a
> > stable client API and wire protocol in general) I'd prefer to make
> > Python releases separate from Kudu major/minor releases. For example,
> > sometime in the next couple months I would like to add low-level
> > (C-level) interoperability between pandas, NumPy, and Kudu. It would
> > be nice to be able to ship this basically as soon as it's available in
> > trunk. This is somewhat independent from the version numbering
> > question, but it would probably be better for the Python client and
> > interop tools to have their own version number. If the Kudu committers
> > feel otherwise, it's not an issue -- some of this depends on the
> > expected release cadence of Kudu itself.
> >
> > best,
> > Wes
> >
> > On Wed, Mar 9, 2016 at 3:49 PM, David Alves <da...@gmail.com>
> wrote:
> > > Hi All
> > >
> > > We'll need to release a new version of the python client to pypi and we
> > > were discussing versioning strategies.
> > > Currently the python's client version does not match the version of the
> > > c++ client/server (0.1.0 versus 0.7.1) and we can either stick with
> that
> > > and keep different versions, or change it to match the latest Kudu
> > release.
> > > Making the versions match would have the advantage of making it clear
> > to
> > > developers if the server/client versions match or not, which might help
> > > weeding out compatibility bugs. On the other hand keeping different
> > > versions would likely make our lives easier if we want to release the
> > > python client at a different cadence of regular Kudu releases.
> > > What do people think?
> > >
> > > Best
> > > David
> >
>
Re: Kudu python client versioning
Posted by Casey Ching <ca...@cloudera.com>.
I’d also go with separating the version numbers. Hopefully eventually the client api will be stable and won’t change much, then a new Kudu release won’t require a python release. If the version are made to match, there should come a time when the python version number needs to be bumped just to keep the versions in sync though no actual python/client changes were made.
If you are worried about client/server compatibility, maybe there is something that can be done such that the client/server verify that they are in fact compatible through some api call.
Casey
On March 9, 2016 at 4:10:40 PM, David Alves (davidralves@gmail.com) wrote:
We intend to produce Kudu releases every couple of months.
It's also worth noting that while we can keep a different release cadence
for the python client we still need to go through a vote and do a proper
release.
Do you feel like its worth the additional work to increase the cadence or
do you feel that 2 months is likely enough?
As an alternative we could have a primary plan to make the major/minor
versions match and release in the same cadence, but reserve the right to do
ad-hoc micro releases occasionally.
Thoughts?
Best
David
On Wed, Mar 9, 2016 at 3:56 PM, Wes McKinney <we...@cloudera.com> wrote:
> Since I expect the Python client to evolve faster from a user
> perspective (as opposed to Kudu core, where users will see basically a
> stable client API and wire protocol in general) I'd prefer to make
> Python releases separate from Kudu major/minor releases. For example,
> sometime in the next couple months I would like to add low-level
> (C-level) interoperability between pandas, NumPy, and Kudu. It would
> be nice to be able to ship this basically as soon as it's available in
> trunk. This is somewhat independent from the version numbering
> question, but it would probably be better for the Python client and
> interop tools to have their own version number. If the Kudu committers
> feel otherwise, it's not an issue -- some of this depends on the
> expected release cadence of Kudu itself.
>
> best,
> Wes
>
> On Wed, Mar 9, 2016 at 3:49 PM, David Alves <da...@gmail.com> wrote:
> > Hi All
> >
> > We'll need to release a new version of the python client to pypi and we
> > were discussing versioning strategies.
> > Currently the python's client version does not match the version of the
> > c++ client/server (0.1.0 versus 0.7.1) and we can either stick with that
> > and keep different versions, or change it to match the latest Kudu
> release.
> > Making the versions match would have the advantage of making it clear
> to
> > developers if the server/client versions match or not, which might help
> > weeding out compatibility bugs. On the other hand keeping different
> > versions would likely make our lives easier if we want to release the
> > python client at a different cadence of regular Kudu releases.
> > What do people think?
> >
> > Best
> > David
>
Re: Kudu python client versioning
Posted by David Alves <da...@gmail.com>.
We intend to produce Kudu releases every couple of months.
It's also worth noting that while we can keep a different release cadence
for the python client we still need to go through a vote and do a proper
release.
Do you feel like its worth the additional work to increase the cadence or
do you feel that 2 months is likely enough?
As an alternative we could have a primary plan to make the major/minor
versions match and release in the same cadence, but reserve the right to do
ad-hoc micro releases occasionally.
Thoughts?
Best
David
On Wed, Mar 9, 2016 at 3:56 PM, Wes McKinney <we...@cloudera.com> wrote:
> Since I expect the Python client to evolve faster from a user
> perspective (as opposed to Kudu core, where users will see basically a
> stable client API and wire protocol in general) I'd prefer to make
> Python releases separate from Kudu major/minor releases. For example,
> sometime in the next couple months I would like to add low-level
> (C-level) interoperability between pandas, NumPy, and Kudu. It would
> be nice to be able to ship this basically as soon as it's available in
> trunk. This is somewhat independent from the version numbering
> question, but it would probably be better for the Python client and
> interop tools to have their own version number. If the Kudu committers
> feel otherwise, it's not an issue -- some of this depends on the
> expected release cadence of Kudu itself.
>
> best,
> Wes
>
> On Wed, Mar 9, 2016 at 3:49 PM, David Alves <da...@gmail.com> wrote:
> > Hi All
> >
> > We'll need to release a new version of the python client to pypi and we
> > were discussing versioning strategies.
> > Currently the python's client version does not match the version of the
> > c++ client/server (0.1.0 versus 0.7.1) and we can either stick with that
> > and keep different versions, or change it to match the latest Kudu
> release.
> > Making the versions match would have the advantage of making it clear
> to
> > developers if the server/client versions match or not, which might help
> > weeding out compatibility bugs. On the other hand keeping different
> > versions would likely make our lives easier if we want to release the
> > python client at a different cadence of regular Kudu releases.
> > What do people think?
> >
> > Best
> > David
>
Re: confusion about " the primary key intervals of different RowSets
may intersect"
Posted by 曾 杰南 <ze...@hotmail.com>.
We need't pre-split DRS.
for example:
roll DRSsize after it's size > 3 (row)
row5 means row with primary key 5
1. init
MRS: {}
DRS0(min, max): {}
2. insert three rows
MRS: {row5, row7, row8}
DRS0(min, max): {}
2. flush
MRS: {}
DRS0(min, max): {row5, row6, row8}
3. insert
MRS: {row1, row9, row10, row11}
DRS0(min, max): {row5, row6, row8}
4. flush && split
MRS: {}
DRS0(min, max): {row1,row5, row6, row8, row9, row10, row11} ->
DRS0(min, 6]:{row1, row5, row6} DSR1(6, 10]:{row8, row9, row10} DSR2(11, max):{row11}
negative side effects:
1.fragment of DiskRowSets
2. redo log split is complicated
于 2016年03月17日 11:42, Binglin Chang 写道:
How can this be "bootstrapped"?
At beginning, there is no DRS, only one MRS.
It's hard to do pre-split DRS, if you don't know distribution, and key
distribution may change along time.
On Thu, Mar 17, 2016 at 11:36 AM, 曾 杰南 <ze...@hotmail.com> wrote:
Hi all:
I learn Kudu's paper "Kudu: Storage for Fast Analytics on Fast Data" very
hard to find
why performance of hbase' random query is superior to kudu. "the primary
key intervals
of different RowSets may intersect" may be one of the reasons.
My confusion is why not keep DiskRowSets ordered on primary key globally.
When flush MemRowSet,
the rows of MemRowSet dispatch to deltaMemStore of correspanding
DiskRowSets. And negative side
effects is fragment of DiskRowSets, but it is worth for globally orderd of
DiskRowSets.
best
jie
Re: confusion about " the primary key intervals of different RowSets
may intersect"
Posted by Binglin Chang <de...@gmail.com>.
Not an expert here, but my guess auto split & merge DRS is not available
right now and probably will be very complex. so compaction approach seems
more simple and reasonable?
Re: confusion about " the primary key intervals of different RowSets
may intersect"
Posted by 曾 杰南 <ze...@hotmail.com>.
We need't pre-split DRS.
for example:
roll DRSsize after it's size > 3 (row)
row5 means row with primary key 5
1. init
MRS: {}
DRS0(min, max): {}
2. insert three rows
MRS: {row5, row7, row8}
DRS0(min, max): {}
2. flush
MRS: {}
DRS0(min, max): {row5, row6, row8}
3. insert
MRS: {row1, row9, row10, row11}
DRS0(min, max): {row5, row6, row8}
4. flush && split
MRS: {}
DRS0(min, max): {row1,row5, row6, row8, row9, row10, row11} ->
DRS0(min, 6]:{row1, row5, row6} DSR1(6, 10]:{row8, row9, row10} DSR2(11, max):{row11}
negative side effects:
1.fragment of DiskRowSets
2. redo log split is complicated
于 2016年03月17日 11:42, Binglin Chang 写道:
How can this be "bootstrapped"?
At beginning, there is no DRS, only one MRS.
It's hard to do pre-split DRS, if you don't know distribution, and key
distribution may change along time.
On Thu, Mar 17, 2016 at 11:36 AM, 曾 杰南 <ze...@hotmail.com> wrote:
Hi all:
I learn Kudu's paper "Kudu: Storage for Fast Analytics on Fast Data" very
hard to find
why performance of hbase' random query is superior to kudu. "the primary
key intervals
of different RowSets may intersect" may be one of the reasons.
My confusion is why not keep DiskRowSets ordered on primary key globally.
When flush MemRowSet,
the rows of MemRowSet dispatch to deltaMemStore of correspanding
DiskRowSets. And negative side
effects is fragment of DiskRowSets, but it is worth for globally orderd of
DiskRowSets.
best
jie
Re: confusion about " the primary key intervals of different RowSets
may intersect"
Posted by Binglin Chang <de...@gmail.com>.
How can this be "bootstrapped"?
At beginning, there is no DRS, only one MRS.
It's hard to do pre-split DRS, if you don't know distribution, and key
distribution may change along time.
On Thu, Mar 17, 2016 at 11:36 AM, 曾 杰南 <ze...@hotmail.com> wrote:
>
> Hi all:
> I learn Kudu's paper "Kudu: Storage for Fast Analytics on Fast Data" very
> hard to find
> why performance of hbase' random query is superior to kudu. "the primary
> key intervals
> of different RowSets may intersect" may be one of the reasons.
>
> My confusion is why not keep DiskRowSets ordered on primary key globally.
> When flush MemRowSet,
> the rows of MemRowSet dispatch to deltaMemStore of correspanding
> DiskRowSets. And negative side
> effects is fragment of DiskRowSets, but it is worth for globally orderd of
> DiskRowSets.
>
> best
> jie
>
>
confusion about " the primary key intervals of different RowSets may
intersect"
Posted by 曾 杰南 <ze...@hotmail.com>.
Hi all:
I learn Kudu's paper "Kudu: Storage for Fast Analytics on Fast Data" very hard to find
why performance of hbase' random query is superior to kudu. "the primary key intervals
of different RowSets may intersect" may be one of the reasons.
My confusion is why not keep DiskRowSets ordered on primary key globally. When flush MemRowSet,
the rows of MemRowSet dispatch to deltaMemStore of correspanding DiskRowSets. And negative side
effects is fragment of DiskRowSets, but it is worth for globally orderd of DiskRowSets.
best
jie
Re: Kudu python client versioning
Posted by Wes McKinney <we...@cloudera.com>.
Since I expect the Python client to evolve faster from a user
perspective (as opposed to Kudu core, where users will see basically a
stable client API and wire protocol in general) I'd prefer to make
Python releases separate from Kudu major/minor releases. For example,
sometime in the next couple months I would like to add low-level
(C-level) interoperability between pandas, NumPy, and Kudu. It would
be nice to be able to ship this basically as soon as it's available in
trunk. This is somewhat independent from the version numbering
question, but it would probably be better for the Python client and
interop tools to have their own version number. If the Kudu committers
feel otherwise, it's not an issue -- some of this depends on the
expected release cadence of Kudu itself.
best,
Wes
On Wed, Mar 9, 2016 at 3:49 PM, David Alves <da...@gmail.com> wrote:
> Hi All
>
> We'll need to release a new version of the python client to pypi and we
> were discussing versioning strategies.
> Currently the python's client version does not match the version of the
> c++ client/server (0.1.0 versus 0.7.1) and we can either stick with that
> and keep different versions, or change it to match the latest Kudu release.
> Making the versions match would have the advantage of making it clear to
> developers if the server/client versions match or not, which might help
> weeding out compatibility bugs. On the other hand keeping different
> versions would likely make our lives easier if we want to release the
> python client at a different cadence of regular Kudu releases.
> What do people think?
>
> Best
> David