Drop 0. from the version

[Roman Leventov <le...@apache.org>]

It doesn't seem to me that Druid API is going to stabilize in the near
future (if ever), because there are so many extension points and something
is broken in every release. On the other hand, Druid is not Hadoop or
Spark, which have applications API. Druid API for extensions, not
applications. It is used by people who are closer to Druid development and
fixing their extensions is routine.

With that, I think it make sense to drop "0." from the Druid version and
call it Druid 14, Druid 15, etc.

Re: Drop 0. from the version

[Julian Hyde <jh...@apache.org>]

Our messages crossed, so I didn’t see the message where you said “versioning doesn’t need to be semantic”.

I did not say that Druid should do SemVer; I said that you should have a policy, and discuss how that policy differs from SemVer.

Julian

> On Dec 21, 2018, at 10:57 AM, Roman Leventov <le...@gmail.com> wrote:
> 
> Julian, I mentioned semantic versioning in the previous message. The
> comparison with IntelliJ may seem shocking at first, but actually Druid may
> be semantically closer to IntelliJ than to Hadoop or Spark. Druid is a
> sever-side *application*, not a library or framework. Like Druid has
> extensions API, IntelliJ has plugin API, that is also very unstable and
> broken in every IntelliJ release, as far as I know.
> 
> Comparison with Guava doesn't make a lot of sense - exactly because of
> that. Guava is a library, half of the Java world depends on it. Almost
> nothing depends on Druid.
> 
> On Fri, 21 Dec 2018 at 19:46, Julian Hyde <jh...@apache.org> wrote:
> 
>> No one has mentioned Semantic Versioning (semver [1]) yet, so I will.
>> I don't know whether the Druid developers think in terms of semver,
>> but a lot of your audience will. In semver, the shift from 0. to 1. is
>> a big event, because the "only remove APIs in major releases" rule
>> does not apply for versions < 1.0.
>> 
>> It would be good if Druid had a policy for how long APIs and will be
>> around. It does not need to be based on semver, but if it isn't, it
>> should explain how it is different than semver.
>> 
>> It should also spell out the planned release cadence. It sounds like
>> you're thinking of two major versions per year, which sounds great.
>> But note that Guava did exactly that, and got flak from a lot of
>> people because features would move from supported to deprecated to
>> gone in just six months.
>> 
>> Regarding combining release 1.0 with Druid's graduation. My gut says
>> no. A lot of people mistakenly think that graduation is a sign of
>> product maturity, whereas it's actually a sign of *community*
>> maturity. You don't want to play into those misconceptions and make
>> people think that Druid is less mature than it really is. (For the
>> record, I think Druid's product and community are both very mature.) I
>> think of 1.0 and graduation as two separate opportunities to generate
>> some news. For 1.0 you can talk about how Druid is industry strength,
>> has wide adoption at large scales; for graduation you can talk about
>> community and what that brings to governance, and adapting to market
>> needs.
>> 
>> Also regarding that "1.0" thing. How about going straight to 14.0?
>> 
>> Julian
>> 
>> [1] https://semver.org/
>> On Fri, Dec 21, 2018 at 10:10 AM Charles Allen <cr...@apache.org> wrote:
>>> 
>>> If I'm greedily honest, I don't want to maintain that many backport
>>> channels. I'd rather have "If you want XYZ backport for version 14, then
>>> you have to take the latest minor version for 14" and have a policy to
>>> where someone can upgrade from 14.x --> 14.latest with (hopefully) no
>>> config changes.
>>> 
>>> 
>>> 
>>> 
>>> On Fri, Dec 21, 2018 at 9:03 AM David Glasser <glasser@apollographql.com
>>> 
>>> wrote:
>>> 
>>>> One nice advantage to moving out of 0.x is that it frees up a digit on
>> the
>>>> right side to more cleanly differentiate between "minor release (a
>> random
>>>> assortment of bug fixes, small features, etc)" and "patch release
>>>> (literally the minimum delta to give you a security fix or high impact
>> bug
>>>> fix)".
>>>> 
>>>> --dave
>>>> 
>>>> On Fri, Dec 21, 2018 at 8:58 AM Gian Merlino <gi...@apache.org> wrote:
>>>> 
>>>>> I'm not too fussy around whether we do a 1.0 or simply drop the 0.
>> and
>>>> have
>>>>> it be a 14.0 or 15.0 or 16.0 or wherever we are at the time we do
>> it. I
>>>>> also like the quarterly cadence of release-from-master we had before
>> we
>>>> got
>>>>> blocked on the ASF transition, and would like to pick that back up
>> again
>>>>> (with the next branch cut from master at the end of January, since
>> we did
>>>>> the 0.13.0 branch cut in late October).
>>>>> 
>>>>> Seems to me that good points of discussion are, what should we use
>> as the
>>>>> rule for incrementing the major version? Do we do what we've been
>> doing
>>>>> (incrementing whenever there's either an incompatible change in
>> extension
>>>>> APIs, or in query APIs, or when necessary to preserve the ability to
>>>> always
>>>>> be able to roll forward/back one major release). Or do we do
>> something
>>>> else
>>>>> (Roman seems to be suggesting dropping extension APIs from
>>>> consideration).
>>>>> 
>>>>> And also, what does 1.0 or 14.0 or 15.0 or what-have-you mean to us?
>> Is
>>>> it
>>>>> something that should be tied to ASF graduation? Completeness of
>> vision?
>>>>> Stability of APIs or operational characteristics? Something else?
>> You are
>>>>> right that it is sort of a marketing/mentality thing, so it's an
>>>>> opportunity for us to declare that we feel Druid has reached some
>>>>> milestone. My feeling at this time is probably ASF graduation or
>>>>> completeness of vision (see my earlier mail for thoughts there) are
>> the
>>>>> ones that make most sense to me.
>>>>> 
>>>>> On Fri, Dec 21, 2018 at 10:41 AM Charles Allen <cr...@apache.org>
>>>> wrote:
>>>>> 
>>>>>> Is there any feeling in the community that the logic behind the
>>>> releases
>>>>>> needs to change?
>>>>>> 
>>>>>> If so then I think we should discuss what that release cadence
>> needs to
>>>>>> look like.
>>>>>> 
>>>>>> If not then dropping the 0. prefix is a marketing / mental item.
>> Kind
>>>> of
>>>>>> like the 3.x->4.x Linux kernel upgrade. If this is the case then
>> would
>>>> we
>>>>>> even want to go with 1.x? I think Roman's proposal would work fine
>> in
>>>>> this
>>>>>> case. Where we just call it Apache Druid 14 (or 15 or whatever it
>> is
>>>> when
>>>>>> we get there) and just keep the same logic for when we release
>> stuff,
>>>>> which
>>>>>> has been something like:
>>>>>> 
>>>>>> For a X.Y release, going to a X.? release should be very straight
>>>> forward
>>>>>> for anyone running stock Druid.
>>>>>> For a X.Y release, going to a (X+1).? or from a (X+1).? back to an
>> X.Y
>>>>>> release should be feasible. It might require running a tool
>> supported
>>>> by
>>>>>> the community.
>>>>>> For a X.Y release, going to an (X+2).? or an (X-2).? is not
>> supported.
>>>>> Some
>>>>>> things that will not have tools might have warning logs printed
>> that
>>>> the
>>>>>> functionality will change (should we change these to alerts?)
>>>>>> 
>>>>>> If this sounds reasonable then jumping straight to Apache Druid 14
>> on
>>>> the
>>>>>> first official apache release would make a lot of sense.
>>>>>> 
>>>>>> Cheers,
>>>>>> Charles Allen
>>>>>> 
>>>>>> 
>>>>>> On Thu, Dec 20, 2018 at 11:07 PM Gian Merlino <gi...@apache.org>
>> wrote:
>>>>>> 
>>>>>>> I think it's a good point. Culturally we have been willing to
>> break
>>>>>>> extension APIs for relatively small benefits. But we have
>> generally
>>>>> been
>>>>>>> unwilling to make breaking changes on the operations side quite
>> so
>>>>>>> liberally. Also, most cluster operators don't have their own
>> custom
>>>>>>> extensions, in my experience. So it does make sense to
>> differentiate
>>>>>> them.
>>>>>>> I'm not sure how it makes sense to differentiate them, though. It
>>>> could
>>>>>> be
>>>>>>> done through the version number (only increment the major
>> version for
>>>>>>> operations breaking changes) or it could be done through an
>>>> "upgrading"
>>>>>>> guide in the documentation (increment the major version for
>>>> operations
>>>>> or
>>>>>>> extension breaking changes, but, have a guide that tells people
>> which
>>>>>>> versions have operations breaking changes to aid in upgrades).
>>>>>>> 
>>>>>>> Coming back to the question in the subject of your mail: IMO, for
>>>>>>> "graduation" out of 0.x, we should talk as a community about what
>>>> that
>>>>>>> means to us. It is a milestone that on the one hand, doesn't mean
>>>> much,
>>>>>> but
>>>>>>> on the other hand, can be deeply symbolic. Some things that it
>> has
>>>>> meant
>>>>>> to
>>>>>>> other projects:
>>>>>>> 
>>>>>>> 1) Production readiness. Obviously Druid is well past this. If
>> this
>>>> is
>>>>>> what
>>>>>>> dropping the 0. means, then we should do it immediately.
>>>>>>> 
>>>>>>> 2) Belief that the APIs have become relatively stable. Like you
>> said,
>>>>> the
>>>>>>> extension APIs don't seem particularly close to stable, but maybe
>>>>> that's
>>>>>>> okay. However, the pace of breaking changes on the operations and
>>>> query
>>>>>>> side for non-experimental features has been relatively calm for
>> the
>>>>> past
>>>>>>> couple of years, so if we focus on that then we can make a case
>> here.
>>>>>>> 
>>>>>>> 3) Completeness of vision. This one is the most interesting to
>> me. I
>>>>>>> suspect that different people in the community have different
>> visions
>>>>> for
>>>>>>> Druid. It is also the kind of project that may never truly be
>>>> complete
>>>>> in
>>>>>>> vision (in principle, the platform could become a competitive
>> data
>>>>>>> warehouse, search engine, etc, …). For what it's worth, my
>> vision of
>>>>>> Druid
>>>>>>> for the next year at least involves robust stream ingestion
>> being a
>>>>> first
>>>>>>> class ingestion method (Kafka / Kinesis indexing service style)
>> and
>>>> SQL
>>>>>>> being a first class query language. These are both, today, still
>>>>>>> experimental features. So are lookups. All of these 3 features,
>> from
>>>>>> what I
>>>>>>> can see, are quite popular amongst Druid users despite being
>>>>>> experimental.
>>>>>>> For a 'completeness of vision' based 1.0 I would want to lift
>> all of
>>>>>> those
>>>>>>> out of experimental status and, for SQL in particular, to have
>> its
>>>>>>> functionality rounded out a bit more (to support the native query
>>>>>> features
>>>>>>> it doesn't currently support, like multi-value dimensions,
>>>>> datasketches,
>>>>>>> etc).
>>>>>>> 
>>>>>>> 4) Marketing / timing. Like, doing a 1.0 around the time we
>> graduate
>>>>> from
>>>>>>> the Incubator. Not sure how much this really matters, but
>> projects do
>>>>> it
>>>>>>> sometimes.
>>>>>>> 
>>>>>>> Another question is, how often do we intend to rev the version?
>> At
>>>> the
>>>>>> rate
>>>>>>> we're going, we rev 2-3 major versions a year. Would we intend to
>>>> keep
>>>>>> that
>>>>>>> up, or slow it down by making more of an effort to avoid breaking
>>>>>> changes?
>>>>>>> 
>>>>>>> On Thu, Dec 20, 2018 at 2:17 PM Roman Leventov <
>>>> leventov.ru@gmail.com>
>>>>>>> wrote:
>>>>>>> 
>>>>>>>> It may also make sense to distinguish "operations" breaking
>> changes
>>>>>> from
>>>>>>>> API breaking changes. Operations breaking changes establish the
>>>>> minimum
>>>>>>>> cadence of Druid cluster upgrades, that allow rolling Druid
>>>> versions
>>>>>> back
>>>>>>>> and forward. I. e. it's related to segment format, the format
>> of
>>>> the
>>>>>> data
>>>>>>>> kept in ZooKeeper and the SQL database, or events such as
>> stopping
>>>>>>> support
>>>>>>>> of ZooKeeper for certain things (e. g. forcing using of HTTP
>>>>>>>> announcements). So Druid cluster operators cannot update Druid
>> from
>>>>>>> version
>>>>>>>> X to version Z skipping the version Y, if both Y and Z have
>> some
>>>>>>> operations
>>>>>>>> breaking changes. (Any such changes should support rollback
>> options
>>>>> at
>>>>>>>> least until the next version with operations breaking changes.)
>>>>>>>> 
>>>>>>>> API breaking changes are just changes in Druid extensions APIs.
>>>> Druid
>>>>>>>> cluster operators could skip any number of releases with such
>>>>> breaking
>>>>>>>> changes, as long as their extension's code is updated for the
>>>> latest
>>>>>>>> version of API.
>>>>>>>> 
>>>>>>>> On Thu, 20 Dec 2018 at 20:03, Roman Leventov <
>> leventov@apache.org>
>>>>>>> wrote:
>>>>>>>> 
>>>>>>>>> It doesn't seem to me that Druid API is going to stabilize
>> in the
>>>>>> near
>>>>>>>>> future (if ever), because there are so many extension points
>> and
>>>>>>>> something
>>>>>>>>> is broken in every release. On the other hand, Druid is not
>>>> Hadoop
>>>>> or
>>>>>>>>> Spark, which have applications API. Druid API for
>> extensions, not
>>>>>>>>> applications. It is used by people who are closer to Druid
>>>>>> development
>>>>>>>> and
>>>>>>>>> fixing their extensions is routine.
>>>>>>>>> 
>>>>>>>>> With that, I think it make sense to drop "0." from the Druid
>>>>> version
>>>>>>> and
>>>>>>>>> call it Druid 14, Druid 15, etc.
>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>> 
>>>> 
>> 
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@druid.apache.org
>> For additional commands, e-mail: dev-help@druid.apache.org
>> 
>> 


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@druid.apache.org
For additional commands, e-mail: dev-help@druid.apache.org

Re: Drop 0. from the version

[Roman Leventov <le...@gmail.com>]

Julian, I mentioned semantic versioning in the previous message. The
comparison with IntelliJ may seem shocking at first, but actually Druid may
be semantically closer to IntelliJ than to Hadoop or Spark. Druid is a
sever-side *application*, not a library or framework. Like Druid has
extensions API, IntelliJ has plugin API, that is also very unstable and
broken in every IntelliJ release, as far as I know.

Comparison with Guava doesn't make a lot of sense - exactly because of
that. Guava is a library, half of the Java world depends on it. Almost
nothing depends on Druid.

On Fri, 21 Dec 2018 at 19:46, Julian Hyde <jh...@apache.org> wrote:

> No one has mentioned Semantic Versioning (semver [1]) yet, so I will.
> I don't know whether the Druid developers think in terms of semver,
> but a lot of your audience will. In semver, the shift from 0. to 1. is
> a big event, because the "only remove APIs in major releases" rule
> does not apply for versions < 1.0.
>
> It would be good if Druid had a policy for how long APIs and will be
> around. It does not need to be based on semver, but if it isn't, it
> should explain how it is different than semver.
>
> It should also spell out the planned release cadence. It sounds like
> you're thinking of two major versions per year, which sounds great.
> But note that Guava did exactly that, and got flak from a lot of
> people because features would move from supported to deprecated to
> gone in just six months.
>
> Regarding combining release 1.0 with Druid's graduation. My gut says
> no. A lot of people mistakenly think that graduation is a sign of
> product maturity, whereas it's actually a sign of *community*
> maturity. You don't want to play into those misconceptions and make
> people think that Druid is less mature than it really is. (For the
> record, I think Druid's product and community are both very mature.) I
> think of 1.0 and graduation as two separate opportunities to generate
> some news. For 1.0 you can talk about how Druid is industry strength,
> has wide adoption at large scales; for graduation you can talk about
> community and what that brings to governance, and adapting to market
> needs.
>
> Also regarding that "1.0" thing. How about going straight to 14.0?
>
> Julian
>
> [1] https://semver.org/
> On Fri, Dec 21, 2018 at 10:10 AM Charles Allen <cr...@apache.org> wrote:
> >
> > If I'm greedily honest, I don't want to maintain that many backport
> > channels. I'd rather have "If you want XYZ backport for version 14, then
> > you have to take the latest minor version for 14" and have a policy to
> > where someone can upgrade from 14.x --> 14.latest with (hopefully) no
> > config changes.
> >
> >
> >
> >
> > On Fri, Dec 21, 2018 at 9:03 AM David Glasser <glasser@apollographql.com
> >
> > wrote:
> >
> > > One nice advantage to moving out of 0.x is that it frees up a digit on
> the
> > > right side to more cleanly differentiate between "minor release (a
> random
> > > assortment of bug fixes, small features, etc)" and "patch release
> > > (literally the minimum delta to give you a security fix or high impact
> bug
> > > fix)".
> > >
> > > --dave
> > >
> > > On Fri, Dec 21, 2018 at 8:58 AM Gian Merlino <gi...@apache.org> wrote:
> > >
> > > > I'm not too fussy around whether we do a 1.0 or simply drop the 0.
> and
> > > have
> > > > it be a 14.0 or 15.0 or 16.0 or wherever we are at the time we do
> it. I
> > > > also like the quarterly cadence of release-from-master we had before
> we
> > > got
> > > > blocked on the ASF transition, and would like to pick that back up
> again
> > > > (with the next branch cut from master at the end of January, since
> we did
> > > > the 0.13.0 branch cut in late October).
> > > >
> > > > Seems to me that good points of discussion are, what should we use
> as the
> > > > rule for incrementing the major version? Do we do what we've been
> doing
> > > > (incrementing whenever there's either an incompatible change in
> extension
> > > > APIs, or in query APIs, or when necessary to preserve the ability to
> > > always
> > > > be able to roll forward/back one major release). Or do we do
> something
> > > else
> > > > (Roman seems to be suggesting dropping extension APIs from
> > > consideration).
> > > >
> > > > And also, what does 1.0 or 14.0 or 15.0 or what-have-you mean to us?
> Is
> > > it
> > > > something that should be tied to ASF graduation? Completeness of
> vision?
> > > > Stability of APIs or operational characteristics? Something else?
> You are
> > > > right that it is sort of a marketing/mentality thing, so it's an
> > > > opportunity for us to declare that we feel Druid has reached some
> > > > milestone. My feeling at this time is probably ASF graduation or
> > > > completeness of vision (see my earlier mail for thoughts there) are
> the
> > > > ones that make most sense to me.
> > > >
> > > > On Fri, Dec 21, 2018 at 10:41 AM Charles Allen <cr...@apache.org>
> > > wrote:
> > > >
> > > > > Is there any feeling in the community that the logic behind the
> > > releases
> > > > > needs to change?
> > > > >
> > > > > If so then I think we should discuss what that release cadence
> needs to
> > > > > look like.
> > > > >
> > > > > If not then dropping the 0. prefix is a marketing / mental item.
> Kind
> > > of
> > > > > like the 3.x->4.x Linux kernel upgrade. If this is the case then
> would
> > > we
> > > > > even want to go with 1.x? I think Roman's proposal would work fine
> in
> > > > this
> > > > > case. Where we just call it Apache Druid 14 (or 15 or whatever it
> is
> > > when
> > > > > we get there) and just keep the same logic for when we release
> stuff,
> > > > which
> > > > > has been something like:
> > > > >
> > > > > For a X.Y release, going to a X.? release should be very straight
> > > forward
> > > > > for anyone running stock Druid.
> > > > > For a X.Y release, going to a (X+1).? or from a (X+1).? back to an
> X.Y
> > > > > release should be feasible. It might require running a tool
> supported
> > > by
> > > > > the community.
> > > > > For a X.Y release, going to an (X+2).? or an (X-2).? is not
> supported.
> > > > Some
> > > > > things that will not have tools might have warning logs printed
> that
> > > the
> > > > > functionality will change (should we change these to alerts?)
> > > > >
> > > > > If this sounds reasonable then jumping straight to Apache Druid 14
> on
> > > the
> > > > > first official apache release would make a lot of sense.
> > > > >
> > > > > Cheers,
> > > > > Charles Allen
> > > > >
> > > > >
> > > > > On Thu, Dec 20, 2018 at 11:07 PM Gian Merlino <gi...@apache.org>
> wrote:
> > > > >
> > > > > > I think it's a good point. Culturally we have been willing to
> break
> > > > > > extension APIs for relatively small benefits. But we have
> generally
> > > > been
> > > > > > unwilling to make breaking changes on the operations side quite
> so
> > > > > > liberally. Also, most cluster operators don't have their own
> custom
> > > > > > extensions, in my experience. So it does make sense to
> differentiate
> > > > > them.
> > > > > > I'm not sure how it makes sense to differentiate them, though. It
> > > could
> > > > > be
> > > > > > done through the version number (only increment the major
> version for
> > > > > > operations breaking changes) or it could be done through an
> > > "upgrading"
> > > > > > guide in the documentation (increment the major version for
> > > operations
> > > > or
> > > > > > extension breaking changes, but, have a guide that tells people
> which
> > > > > > versions have operations breaking changes to aid in upgrades).
> > > > > >
> > > > > > Coming back to the question in the subject of your mail: IMO, for
> > > > > > "graduation" out of 0.x, we should talk as a community about what
> > > that
> > > > > > means to us. It is a milestone that on the one hand, doesn't mean
> > > much,
> > > > > but
> > > > > > on the other hand, can be deeply symbolic. Some things that it
> has
> > > > meant
> > > > > to
> > > > > > other projects:
> > > > > >
> > > > > > 1) Production readiness. Obviously Druid is well past this. If
> this
> > > is
> > > > > what
> > > > > > dropping the 0. means, then we should do it immediately.
> > > > > >
> > > > > > 2) Belief that the APIs have become relatively stable. Like you
> said,
> > > > the
> > > > > > extension APIs don't seem particularly close to stable, but maybe
> > > > that's
> > > > > > okay. However, the pace of breaking changes on the operations and
> > > query
> > > > > > side for non-experimental features has been relatively calm for
> the
> > > > past
> > > > > > couple of years, so if we focus on that then we can make a case
> here.
> > > > > >
> > > > > > 3) Completeness of vision. This one is the most interesting to
> me. I
> > > > > > suspect that different people in the community have different
> visions
> > > > for
> > > > > > Druid. It is also the kind of project that may never truly be
> > > complete
> > > > in
> > > > > > vision (in principle, the platform could become a competitive
> data
> > > > > > warehouse, search engine, etc, …). For what it's worth, my
> vision of
> > > > > Druid
> > > > > > for the next year at least involves robust stream ingestion
> being a
> > > > first
> > > > > > class ingestion method (Kafka / Kinesis indexing service style)
> and
> > > SQL
> > > > > > being a first class query language. These are both, today, still
> > > > > > experimental features. So are lookups. All of these 3 features,
> from
> > > > > what I
> > > > > > can see, are quite popular amongst Druid users despite being
> > > > > experimental.
> > > > > > For a 'completeness of vision' based 1.0 I would want to lift
> all of
> > > > > those
> > > > > > out of experimental status and, for SQL in particular, to have
> its
> > > > > > functionality rounded out a bit more (to support the native query
> > > > > features
> > > > > > it doesn't currently support, like multi-value dimensions,
> > > > datasketches,
> > > > > > etc).
> > > > > >
> > > > > > 4) Marketing / timing. Like, doing a 1.0 around the time we
> graduate
> > > > from
> > > > > > the Incubator. Not sure how much this really matters, but
> projects do
> > > > it
> > > > > > sometimes.
> > > > > >
> > > > > > Another question is, how often do we intend to rev the version?
> At
> > > the
> > > > > rate
> > > > > > we're going, we rev 2-3 major versions a year. Would we intend to
> > > keep
> > > > > that
> > > > > > up, or slow it down by making more of an effort to avoid breaking
> > > > > changes?
> > > > > >
> > > > > > On Thu, Dec 20, 2018 at 2:17 PM Roman Leventov <
> > > leventov.ru@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > It may also make sense to distinguish "operations" breaking
> changes
> > > > > from
> > > > > > > API breaking changes. Operations breaking changes establish the
> > > > minimum
> > > > > > > cadence of Druid cluster upgrades, that allow rolling Druid
> > > versions
> > > > > back
> > > > > > > and forward. I. e. it's related to segment format, the format
> of
> > > the
> > > > > data
> > > > > > > kept in ZooKeeper and the SQL database, or events such as
> stopping
> > > > > > support
> > > > > > > of ZooKeeper for certain things (e. g. forcing using of HTTP
> > > > > > > announcements). So Druid cluster operators cannot update Druid
> from
> > > > > > version
> > > > > > > X to version Z skipping the version Y, if both Y and Z have
> some
> > > > > > operations
> > > > > > > breaking changes. (Any such changes should support rollback
> options
> > > > at
> > > > > > > least until the next version with operations breaking changes.)
> > > > > > >
> > > > > > > API breaking changes are just changes in Druid extensions APIs.
> > > Druid
> > > > > > > cluster operators could skip any number of releases with such
> > > > breaking
> > > > > > > changes, as long as their extension's code is updated for the
> > > latest
> > > > > > > version of API.
> > > > > > >
> > > > > > > On Thu, 20 Dec 2018 at 20:03, Roman Leventov <
> leventov@apache.org>
> > > > > > wrote:
> > > > > > >
> > > > > > > > It doesn't seem to me that Druid API is going to stabilize
> in the
> > > > > near
> > > > > > > > future (if ever), because there are so many extension points
> and
> > > > > > > something
> > > > > > > > is broken in every release. On the other hand, Druid is not
> > > Hadoop
> > > > or
> > > > > > > > Spark, which have applications API. Druid API for
> extensions, not
> > > > > > > > applications. It is used by people who are closer to Druid
> > > > > development
> > > > > > > and
> > > > > > > > fixing their extensions is routine.
> > > > > > > >
> > > > > > > > With that, I think it make sense to drop "0." from the Druid
> > > > version
> > > > > > and
> > > > > > > > call it Druid 14, Druid 15, etc.
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@druid.apache.org
> For additional commands, e-mail: dev-help@druid.apache.org
>
>

Re: Drop 0. from the version

[Julian Hyde <jh...@apache.org>]

No one has mentioned Semantic Versioning (semver [1]) yet, so I will.
I don't know whether the Druid developers think in terms of semver,
but a lot of your audience will. In semver, the shift from 0. to 1. is
a big event, because the "only remove APIs in major releases" rule
does not apply for versions < 1.0.

It would be good if Druid had a policy for how long APIs and will be
around. It does not need to be based on semver, but if it isn't, it
should explain how it is different than semver.

It should also spell out the planned release cadence. It sounds like
you're thinking of two major versions per year, which sounds great.
But note that Guava did exactly that, and got flak from a lot of
people because features would move from supported to deprecated to
gone in just six months.

Regarding combining release 1.0 with Druid's graduation. My gut says
no. A lot of people mistakenly think that graduation is a sign of
product maturity, whereas it's actually a sign of *community*
maturity. You don't want to play into those misconceptions and make
people think that Druid is less mature than it really is. (For the
record, I think Druid's product and community are both very mature.) I
think of 1.0 and graduation as two separate opportunities to generate
some news. For 1.0 you can talk about how Druid is industry strength,
has wide adoption at large scales; for graduation you can talk about
community and what that brings to governance, and adapting to market
needs.

Also regarding that "1.0" thing. How about going straight to 14.0?

Julian

[1] https://semver.org/
On Fri, Dec 21, 2018 at 10:10 AM Charles Allen <cr...@apache.org> wrote:
>
> If I'm greedily honest, I don't want to maintain that many backport
> channels. I'd rather have "If you want XYZ backport for version 14, then
> you have to take the latest minor version for 14" and have a policy to
> where someone can upgrade from 14.x --> 14.latest with (hopefully) no
> config changes.
>
>
>
>
> On Fri, Dec 21, 2018 at 9:03 AM David Glasser <gl...@apollographql.com>
> wrote:
>
> > One nice advantage to moving out of 0.x is that it frees up a digit on the
> > right side to more cleanly differentiate between "minor release (a random
> > assortment of bug fixes, small features, etc)" and "patch release
> > (literally the minimum delta to give you a security fix or high impact bug
> > fix)".
> >
> > --dave
> >
> > On Fri, Dec 21, 2018 at 8:58 AM Gian Merlino <gi...@apache.org> wrote:
> >
> > > I'm not too fussy around whether we do a 1.0 or simply drop the 0. and
> > have
> > > it be a 14.0 or 15.0 or 16.0 or wherever we are at the time we do it. I
> > > also like the quarterly cadence of release-from-master we had before we
> > got
> > > blocked on the ASF transition, and would like to pick that back up again
> > > (with the next branch cut from master at the end of January, since we did
> > > the 0.13.0 branch cut in late October).
> > >
> > > Seems to me that good points of discussion are, what should we use as the
> > > rule for incrementing the major version? Do we do what we've been doing
> > > (incrementing whenever there's either an incompatible change in extension
> > > APIs, or in query APIs, or when necessary to preserve the ability to
> > always
> > > be able to roll forward/back one major release). Or do we do something
> > else
> > > (Roman seems to be suggesting dropping extension APIs from
> > consideration).
> > >
> > > And also, what does 1.0 or 14.0 or 15.0 or what-have-you mean to us? Is
> > it
> > > something that should be tied to ASF graduation? Completeness of vision?
> > > Stability of APIs or operational characteristics? Something else? You are
> > > right that it is sort of a marketing/mentality thing, so it's an
> > > opportunity for us to declare that we feel Druid has reached some
> > > milestone. My feeling at this time is probably ASF graduation or
> > > completeness of vision (see my earlier mail for thoughts there) are the
> > > ones that make most sense to me.
> > >
> > > On Fri, Dec 21, 2018 at 10:41 AM Charles Allen <cr...@apache.org>
> > wrote:
> > >
> > > > Is there any feeling in the community that the logic behind the
> > releases
> > > > needs to change?
> > > >
> > > > If so then I think we should discuss what that release cadence needs to
> > > > look like.
> > > >
> > > > If not then dropping the 0. prefix is a marketing / mental item. Kind
> > of
> > > > like the 3.x->4.x Linux kernel upgrade. If this is the case then would
> > we
> > > > even want to go with 1.x? I think Roman's proposal would work fine in
> > > this
> > > > case. Where we just call it Apache Druid 14 (or 15 or whatever it is
> > when
> > > > we get there) and just keep the same logic for when we release stuff,
> > > which
> > > > has been something like:
> > > >
> > > > For a X.Y release, going to a X.? release should be very straight
> > forward
> > > > for anyone running stock Druid.
> > > > For a X.Y release, going to a (X+1).? or from a (X+1).? back to an X.Y
> > > > release should be feasible. It might require running a tool supported
> > by
> > > > the community.
> > > > For a X.Y release, going to an (X+2).? or an (X-2).? is not supported.
> > > Some
> > > > things that will not have tools might have warning logs printed that
> > the
> > > > functionality will change (should we change these to alerts?)
> > > >
> > > > If this sounds reasonable then jumping straight to Apache Druid 14 on
> > the
> > > > first official apache release would make a lot of sense.
> > > >
> > > > Cheers,
> > > > Charles Allen
> > > >
> > > >
> > > > On Thu, Dec 20, 2018 at 11:07 PM Gian Merlino <gi...@apache.org> wrote:
> > > >
> > > > > I think it's a good point. Culturally we have been willing to break
> > > > > extension APIs for relatively small benefits. But we have generally
> > > been
> > > > > unwilling to make breaking changes on the operations side quite so
> > > > > liberally. Also, most cluster operators don't have their own custom
> > > > > extensions, in my experience. So it does make sense to differentiate
> > > > them.
> > > > > I'm not sure how it makes sense to differentiate them, though. It
> > could
> > > > be
> > > > > done through the version number (only increment the major version for
> > > > > operations breaking changes) or it could be done through an
> > "upgrading"
> > > > > guide in the documentation (increment the major version for
> > operations
> > > or
> > > > > extension breaking changes, but, have a guide that tells people which
> > > > > versions have operations breaking changes to aid in upgrades).
> > > > >
> > > > > Coming back to the question in the subject of your mail: IMO, for
> > > > > "graduation" out of 0.x, we should talk as a community about what
> > that
> > > > > means to us. It is a milestone that on the one hand, doesn't mean
> > much,
> > > > but
> > > > > on the other hand, can be deeply symbolic. Some things that it has
> > > meant
> > > > to
> > > > > other projects:
> > > > >
> > > > > 1) Production readiness. Obviously Druid is well past this. If this
> > is
> > > > what
> > > > > dropping the 0. means, then we should do it immediately.
> > > > >
> > > > > 2) Belief that the APIs have become relatively stable. Like you said,
> > > the
> > > > > extension APIs don't seem particularly close to stable, but maybe
> > > that's
> > > > > okay. However, the pace of breaking changes on the operations and
> > query
> > > > > side for non-experimental features has been relatively calm for the
> > > past
> > > > > couple of years, so if we focus on that then we can make a case here.
> > > > >
> > > > > 3) Completeness of vision. This one is the most interesting to me. I
> > > > > suspect that different people in the community have different visions
> > > for
> > > > > Druid. It is also the kind of project that may never truly be
> > complete
> > > in
> > > > > vision (in principle, the platform could become a competitive data
> > > > > warehouse, search engine, etc, …). For what it's worth, my vision of
> > > > Druid
> > > > > for the next year at least involves robust stream ingestion being a
> > > first
> > > > > class ingestion method (Kafka / Kinesis indexing service style) and
> > SQL
> > > > > being a first class query language. These are both, today, still
> > > > > experimental features. So are lookups. All of these 3 features, from
> > > > what I
> > > > > can see, are quite popular amongst Druid users despite being
> > > > experimental.
> > > > > For a 'completeness of vision' based 1.0 I would want to lift all of
> > > > those
> > > > > out of experimental status and, for SQL in particular, to have its
> > > > > functionality rounded out a bit more (to support the native query
> > > > features
> > > > > it doesn't currently support, like multi-value dimensions,
> > > datasketches,
> > > > > etc).
> > > > >
> > > > > 4) Marketing / timing. Like, doing a 1.0 around the time we graduate
> > > from
> > > > > the Incubator. Not sure how much this really matters, but projects do
> > > it
> > > > > sometimes.
> > > > >
> > > > > Another question is, how often do we intend to rev the version? At
> > the
> > > > rate
> > > > > we're going, we rev 2-3 major versions a year. Would we intend to
> > keep
> > > > that
> > > > > up, or slow it down by making more of an effort to avoid breaking
> > > > changes?
> > > > >
> > > > > On Thu, Dec 20, 2018 at 2:17 PM Roman Leventov <
> > leventov.ru@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > It may also make sense to distinguish "operations" breaking changes
> > > > from
> > > > > > API breaking changes. Operations breaking changes establish the
> > > minimum
> > > > > > cadence of Druid cluster upgrades, that allow rolling Druid
> > versions
> > > > back
> > > > > > and forward. I. e. it's related to segment format, the format of
> > the
> > > > data
> > > > > > kept in ZooKeeper and the SQL database, or events such as stopping
> > > > > support
> > > > > > of ZooKeeper for certain things (e. g. forcing using of HTTP
> > > > > > announcements). So Druid cluster operators cannot update Druid from
> > > > > version
> > > > > > X to version Z skipping the version Y, if both Y and Z have some
> > > > > operations
> > > > > > breaking changes. (Any such changes should support rollback options
> > > at
> > > > > > least until the next version with operations breaking changes.)
> > > > > >
> > > > > > API breaking changes are just changes in Druid extensions APIs.
> > Druid
> > > > > > cluster operators could skip any number of releases with such
> > > breaking
> > > > > > changes, as long as their extension's code is updated for the
> > latest
> > > > > > version of API.
> > > > > >
> > > > > > On Thu, 20 Dec 2018 at 20:03, Roman Leventov <le...@apache.org>
> > > > > wrote:
> > > > > >
> > > > > > > It doesn't seem to me that Druid API is going to stabilize in the
> > > > near
> > > > > > > future (if ever), because there are so many extension points and
> > > > > > something
> > > > > > > is broken in every release. On the other hand, Druid is not
> > Hadoop
> > > or
> > > > > > > Spark, which have applications API. Druid API for extensions, not
> > > > > > > applications. It is used by people who are closer to Druid
> > > > development
> > > > > > and
> > > > > > > fixing their extensions is routine.
> > > > > > >
> > > > > > > With that, I think it make sense to drop "0." from the Druid
> > > version
> > > > > and
> > > > > > > call it Druid 14, Druid 15, etc.
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@druid.apache.org
For additional commands, e-mail: dev-help@druid.apache.org

Re: Drop 0. from the version

[Roman Leventov <le...@gmail.com>]

I start to think that Druid versioning doesn't need to be semantic and
doesn't need a distinction between major and minor. A release is just a
release, like IntelliJ's 2018.1, 2018.2, 2018.3. Every release could have
some extensions API broken. If there is an operations breaking change, we
could commit to support rollback strategies e. g. for 1 year of releases
after that, that means in the next four releases (if we stick to 3-month
cadence).

On Fri, 21 Dec 2018 at 19:10, Charles Allen <cr...@apache.org> wrote:

> If I'm greedily honest, I don't want to maintain that many backport
> channels. I'd rather have "If you want XYZ backport for version 14, then
> you have to take the latest minor version for 14" and have a policy to
> where someone can upgrade from 14.x --> 14.latest with (hopefully) no
> config changes.
>
>
>
>
> On Fri, Dec 21, 2018 at 9:03 AM David Glasser <gl...@apollographql.com>
> wrote:
>
> > One nice advantage to moving out of 0.x is that it frees up a digit on
> the
> > right side to more cleanly differentiate between "minor release (a random
> > assortment of bug fixes, small features, etc)" and "patch release
> > (literally the minimum delta to give you a security fix or high impact
> bug
> > fix)".
> >
> > --dave
> >
> > On Fri, Dec 21, 2018 at 8:58 AM Gian Merlino <gi...@apache.org> wrote:
> >
> > > I'm not too fussy around whether we do a 1.0 or simply drop the 0. and
> > have
> > > it be a 14.0 or 15.0 or 16.0 or wherever we are at the time we do it. I
> > > also like the quarterly cadence of release-from-master we had before we
> > got
> > > blocked on the ASF transition, and would like to pick that back up
> again
> > > (with the next branch cut from master at the end of January, since we
> did
> > > the 0.13.0 branch cut in late October).
> > >
> > > Seems to me that good points of discussion are, what should we use as
> the
> > > rule for incrementing the major version? Do we do what we've been doing
> > > (incrementing whenever there's either an incompatible change in
> extension
> > > APIs, or in query APIs, or when necessary to preserve the ability to
> > always
> > > be able to roll forward/back one major release). Or do we do something
> > else
> > > (Roman seems to be suggesting dropping extension APIs from
> > consideration).
> > >
> > > And also, what does 1.0 or 14.0 or 15.0 or what-have-you mean to us? Is
> > it
> > > something that should be tied to ASF graduation? Completeness of
> vision?
> > > Stability of APIs or operational characteristics? Something else? You
> are
> > > right that it is sort of a marketing/mentality thing, so it's an
> > > opportunity for us to declare that we feel Druid has reached some
> > > milestone. My feeling at this time is probably ASF graduation or
> > > completeness of vision (see my earlier mail for thoughts there) are the
> > > ones that make most sense to me.
> > >
> > > On Fri, Dec 21, 2018 at 10:41 AM Charles Allen <cr...@apache.org>
> > wrote:
> > >
> > > > Is there any feeling in the community that the logic behind the
> > releases
> > > > needs to change?
> > > >
> > > > If so then I think we should discuss what that release cadence needs
> to
> > > > look like.
> > > >
> > > > If not then dropping the 0. prefix is a marketing / mental item. Kind
> > of
> > > > like the 3.x->4.x Linux kernel upgrade. If this is the case then
> would
> > we
> > > > even want to go with 1.x? I think Roman's proposal would work fine in
> > > this
> > > > case. Where we just call it Apache Druid 14 (or 15 or whatever it is
> > when
> > > > we get there) and just keep the same logic for when we release stuff,
> > > which
> > > > has been something like:
> > > >
> > > > For a X.Y release, going to a X.? release should be very straight
> > forward
> > > > for anyone running stock Druid.
> > > > For a X.Y release, going to a (X+1).? or from a (X+1).? back to an
> X.Y
> > > > release should be feasible. It might require running a tool supported
> > by
> > > > the community.
> > > > For a X.Y release, going to an (X+2).? or an (X-2).? is not
> supported.
> > > Some
> > > > things that will not have tools might have warning logs printed that
> > the
> > > > functionality will change (should we change these to alerts?)
> > > >
> > > > If this sounds reasonable then jumping straight to Apache Druid 14 on
> > the
> > > > first official apache release would make a lot of sense.
> > > >
> > > > Cheers,
> > > > Charles Allen
> > > >
> > > >
> > > > On Thu, Dec 20, 2018 at 11:07 PM Gian Merlino <gi...@apache.org>
> wrote:
> > > >
> > > > > I think it's a good point. Culturally we have been willing to break
> > > > > extension APIs for relatively small benefits. But we have generally
> > > been
> > > > > unwilling to make breaking changes on the operations side quite so
> > > > > liberally. Also, most cluster operators don't have their own custom
> > > > > extensions, in my experience. So it does make sense to
> differentiate
> > > > them.
> > > > > I'm not sure how it makes sense to differentiate them, though. It
> > could
> > > > be
> > > > > done through the version number (only increment the major version
> for
> > > > > operations breaking changes) or it could be done through an
> > "upgrading"
> > > > > guide in the documentation (increment the major version for
> > operations
> > > or
> > > > > extension breaking changes, but, have a guide that tells people
> which
> > > > > versions have operations breaking changes to aid in upgrades).
> > > > >
> > > > > Coming back to the question in the subject of your mail: IMO, for
> > > > > "graduation" out of 0.x, we should talk as a community about what
> > that
> > > > > means to us. It is a milestone that on the one hand, doesn't mean
> > much,
> > > > but
> > > > > on the other hand, can be deeply symbolic. Some things that it has
> > > meant
> > > > to
> > > > > other projects:
> > > > >
> > > > > 1) Production readiness. Obviously Druid is well past this. If this
> > is
> > > > what
> > > > > dropping the 0. means, then we should do it immediately.
> > > > >
> > > > > 2) Belief that the APIs have become relatively stable. Like you
> said,
> > > the
> > > > > extension APIs don't seem particularly close to stable, but maybe
> > > that's
> > > > > okay. However, the pace of breaking changes on the operations and
> > query
> > > > > side for non-experimental features has been relatively calm for the
> > > past
> > > > > couple of years, so if we focus on that then we can make a case
> here.
> > > > >
> > > > > 3) Completeness of vision. This one is the most interesting to me.
> I
> > > > > suspect that different people in the community have different
> visions
> > > for
> > > > > Druid. It is also the kind of project that may never truly be
> > complete
> > > in
> > > > > vision (in principle, the platform could become a competitive data
> > > > > warehouse, search engine, etc, …). For what it's worth, my vision
> of
> > > > Druid
> > > > > for the next year at least involves robust stream ingestion being a
> > > first
> > > > > class ingestion method (Kafka / Kinesis indexing service style) and
> > SQL
> > > > > being a first class query language. These are both, today, still
> > > > > experimental features. So are lookups. All of these 3 features,
> from
> > > > what I
> > > > > can see, are quite popular amongst Druid users despite being
> > > > experimental.
> > > > > For a 'completeness of vision' based 1.0 I would want to lift all
> of
> > > > those
> > > > > out of experimental status and, for SQL in particular, to have its
> > > > > functionality rounded out a bit more (to support the native query
> > > > features
> > > > > it doesn't currently support, like multi-value dimensions,
> > > datasketches,
> > > > > etc).
> > > > >
> > > > > 4) Marketing / timing. Like, doing a 1.0 around the time we
> graduate
> > > from
> > > > > the Incubator. Not sure how much this really matters, but projects
> do
> > > it
> > > > > sometimes.
> > > > >
> > > > > Another question is, how often do we intend to rev the version? At
> > the
> > > > rate
> > > > > we're going, we rev 2-3 major versions a year. Would we intend to
> > keep
> > > > that
> > > > > up, or slow it down by making more of an effort to avoid breaking
> > > > changes?
> > > > >
> > > > > On Thu, Dec 20, 2018 at 2:17 PM Roman Leventov <
> > leventov.ru@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > It may also make sense to distinguish "operations" breaking
> changes
> > > > from
> > > > > > API breaking changes. Operations breaking changes establish the
> > > minimum
> > > > > > cadence of Druid cluster upgrades, that allow rolling Druid
> > versions
> > > > back
> > > > > > and forward. I. e. it's related to segment format, the format of
> > the
> > > > data
> > > > > > kept in ZooKeeper and the SQL database, or events such as
> stopping
> > > > > support
> > > > > > of ZooKeeper for certain things (e. g. forcing using of HTTP
> > > > > > announcements). So Druid cluster operators cannot update Druid
> from
> > > > > version
> > > > > > X to version Z skipping the version Y, if both Y and Z have some
> > > > > operations
> > > > > > breaking changes. (Any such changes should support rollback
> options
> > > at
> > > > > > least until the next version with operations breaking changes.)
> > > > > >
> > > > > > API breaking changes are just changes in Druid extensions APIs.
> > Druid
> > > > > > cluster operators could skip any number of releases with such
> > > breaking
> > > > > > changes, as long as their extension's code is updated for the
> > latest
> > > > > > version of API.
> > > > > >
> > > > > > On Thu, 20 Dec 2018 at 20:03, Roman Leventov <
> leventov@apache.org>
> > > > > wrote:
> > > > > >
> > > > > > > It doesn't seem to me that Druid API is going to stabilize in
> the
> > > > near
> > > > > > > future (if ever), because there are so many extension points
> and
> > > > > > something
> > > > > > > is broken in every release. On the other hand, Druid is not
> > Hadoop
> > > or
> > > > > > > Spark, which have applications API. Druid API for extensions,
> not
> > > > > > > applications. It is used by people who are closer to Druid
> > > > development
> > > > > > and
> > > > > > > fixing their extensions is routine.
> > > > > > >
> > > > > > > With that, I think it make sense to drop "0." from the Druid
> > > version
> > > > > and
> > > > > > > call it Druid 14, Druid 15, etc.
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: Drop 0. from the version

[Charles Allen <cr...@apache.org>]

If I'm greedily honest, I don't want to maintain that many backport
channels. I'd rather have "If you want XYZ backport for version 14, then
you have to take the latest minor version for 14" and have a policy to
where someone can upgrade from 14.x --> 14.latest with (hopefully) no
config changes.




On Fri, Dec 21, 2018 at 9:03 AM David Glasser <gl...@apollographql.com>
wrote:

> One nice advantage to moving out of 0.x is that it frees up a digit on the
> right side to more cleanly differentiate between "minor release (a random
> assortment of bug fixes, small features, etc)" and "patch release
> (literally the minimum delta to give you a security fix or high impact bug
> fix)".
>
> --dave
>
> On Fri, Dec 21, 2018 at 8:58 AM Gian Merlino <gi...@apache.org> wrote:
>
> > I'm not too fussy around whether we do a 1.0 or simply drop the 0. and
> have
> > it be a 14.0 or 15.0 or 16.0 or wherever we are at the time we do it. I
> > also like the quarterly cadence of release-from-master we had before we
> got
> > blocked on the ASF transition, and would like to pick that back up again
> > (with the next branch cut from master at the end of January, since we did
> > the 0.13.0 branch cut in late October).
> >
> > Seems to me that good points of discussion are, what should we use as the
> > rule for incrementing the major version? Do we do what we've been doing
> > (incrementing whenever there's either an incompatible change in extension
> > APIs, or in query APIs, or when necessary to preserve the ability to
> always
> > be able to roll forward/back one major release). Or do we do something
> else
> > (Roman seems to be suggesting dropping extension APIs from
> consideration).
> >
> > And also, what does 1.0 or 14.0 or 15.0 or what-have-you mean to us? Is
> it
> > something that should be tied to ASF graduation? Completeness of vision?
> > Stability of APIs or operational characteristics? Something else? You are
> > right that it is sort of a marketing/mentality thing, so it's an
> > opportunity for us to declare that we feel Druid has reached some
> > milestone. My feeling at this time is probably ASF graduation or
> > completeness of vision (see my earlier mail for thoughts there) are the
> > ones that make most sense to me.
> >
> > On Fri, Dec 21, 2018 at 10:41 AM Charles Allen <cr...@apache.org>
> wrote:
> >
> > > Is there any feeling in the community that the logic behind the
> releases
> > > needs to change?
> > >
> > > If so then I think we should discuss what that release cadence needs to
> > > look like.
> > >
> > > If not then dropping the 0. prefix is a marketing / mental item. Kind
> of
> > > like the 3.x->4.x Linux kernel upgrade. If this is the case then would
> we
> > > even want to go with 1.x? I think Roman's proposal would work fine in
> > this
> > > case. Where we just call it Apache Druid 14 (or 15 or whatever it is
> when
> > > we get there) and just keep the same logic for when we release stuff,
> > which
> > > has been something like:
> > >
> > > For a X.Y release, going to a X.? release should be very straight
> forward
> > > for anyone running stock Druid.
> > > For a X.Y release, going to a (X+1).? or from a (X+1).? back to an X.Y
> > > release should be feasible. It might require running a tool supported
> by
> > > the community.
> > > For a X.Y release, going to an (X+2).? or an (X-2).? is not supported.
> > Some
> > > things that will not have tools might have warning logs printed that
> the
> > > functionality will change (should we change these to alerts?)
> > >
> > > If this sounds reasonable then jumping straight to Apache Druid 14 on
> the
> > > first official apache release would make a lot of sense.
> > >
> > > Cheers,
> > > Charles Allen
> > >
> > >
> > > On Thu, Dec 20, 2018 at 11:07 PM Gian Merlino <gi...@apache.org> wrote:
> > >
> > > > I think it's a good point. Culturally we have been willing to break
> > > > extension APIs for relatively small benefits. But we have generally
> > been
> > > > unwilling to make breaking changes on the operations side quite so
> > > > liberally. Also, most cluster operators don't have their own custom
> > > > extensions, in my experience. So it does make sense to differentiate
> > > them.
> > > > I'm not sure how it makes sense to differentiate them, though. It
> could
> > > be
> > > > done through the version number (only increment the major version for
> > > > operations breaking changes) or it could be done through an
> "upgrading"
> > > > guide in the documentation (increment the major version for
> operations
> > or
> > > > extension breaking changes, but, have a guide that tells people which
> > > > versions have operations breaking changes to aid in upgrades).
> > > >
> > > > Coming back to the question in the subject of your mail: IMO, for
> > > > "graduation" out of 0.x, we should talk as a community about what
> that
> > > > means to us. It is a milestone that on the one hand, doesn't mean
> much,
> > > but
> > > > on the other hand, can be deeply symbolic. Some things that it has
> > meant
> > > to
> > > > other projects:
> > > >
> > > > 1) Production readiness. Obviously Druid is well past this. If this
> is
> > > what
> > > > dropping the 0. means, then we should do it immediately.
> > > >
> > > > 2) Belief that the APIs have become relatively stable. Like you said,
> > the
> > > > extension APIs don't seem particularly close to stable, but maybe
> > that's
> > > > okay. However, the pace of breaking changes on the operations and
> query
> > > > side for non-experimental features has been relatively calm for the
> > past
> > > > couple of years, so if we focus on that then we can make a case here.
> > > >
> > > > 3) Completeness of vision. This one is the most interesting to me. I
> > > > suspect that different people in the community have different visions
> > for
> > > > Druid. It is also the kind of project that may never truly be
> complete
> > in
> > > > vision (in principle, the platform could become a competitive data
> > > > warehouse, search engine, etc, …). For what it's worth, my vision of
> > > Druid
> > > > for the next year at least involves robust stream ingestion being a
> > first
> > > > class ingestion method (Kafka / Kinesis indexing service style) and
> SQL
> > > > being a first class query language. These are both, today, still
> > > > experimental features. So are lookups. All of these 3 features, from
> > > what I
> > > > can see, are quite popular amongst Druid users despite being
> > > experimental.
> > > > For a 'completeness of vision' based 1.0 I would want to lift all of
> > > those
> > > > out of experimental status and, for SQL in particular, to have its
> > > > functionality rounded out a bit more (to support the native query
> > > features
> > > > it doesn't currently support, like multi-value dimensions,
> > datasketches,
> > > > etc).
> > > >
> > > > 4) Marketing / timing. Like, doing a 1.0 around the time we graduate
> > from
> > > > the Incubator. Not sure how much this really matters, but projects do
> > it
> > > > sometimes.
> > > >
> > > > Another question is, how often do we intend to rev the version? At
> the
> > > rate
> > > > we're going, we rev 2-3 major versions a year. Would we intend to
> keep
> > > that
> > > > up, or slow it down by making more of an effort to avoid breaking
> > > changes?
> > > >
> > > > On Thu, Dec 20, 2018 at 2:17 PM Roman Leventov <
> leventov.ru@gmail.com>
> > > > wrote:
> > > >
> > > > > It may also make sense to distinguish "operations" breaking changes
> > > from
> > > > > API breaking changes. Operations breaking changes establish the
> > minimum
> > > > > cadence of Druid cluster upgrades, that allow rolling Druid
> versions
> > > back
> > > > > and forward. I. e. it's related to segment format, the format of
> the
> > > data
> > > > > kept in ZooKeeper and the SQL database, or events such as stopping
> > > > support
> > > > > of ZooKeeper for certain things (e. g. forcing using of HTTP
> > > > > announcements). So Druid cluster operators cannot update Druid from
> > > > version
> > > > > X to version Z skipping the version Y, if both Y and Z have some
> > > > operations
> > > > > breaking changes. (Any such changes should support rollback options
> > at
> > > > > least until the next version with operations breaking changes.)
> > > > >
> > > > > API breaking changes are just changes in Druid extensions APIs.
> Druid
> > > > > cluster operators could skip any number of releases with such
> > breaking
> > > > > changes, as long as their extension's code is updated for the
> latest
> > > > > version of API.
> > > > >
> > > > > On Thu, 20 Dec 2018 at 20:03, Roman Leventov <le...@apache.org>
> > > > wrote:
> > > > >
> > > > > > It doesn't seem to me that Druid API is going to stabilize in the
> > > near
> > > > > > future (if ever), because there are so many extension points and
> > > > > something
> > > > > > is broken in every release. On the other hand, Druid is not
> Hadoop
> > or
> > > > > > Spark, which have applications API. Druid API for extensions, not
> > > > > > applications. It is used by people who are closer to Druid
> > > development
> > > > > and
> > > > > > fixing their extensions is routine.
> > > > > >
> > > > > > With that, I think it make sense to drop "0." from the Druid
> > version
> > > > and
> > > > > > call it Druid 14, Druid 15, etc.
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: Drop 0. from the version

[David Glasser <gl...@apollographql.com>]

One nice advantage to moving out of 0.x is that it frees up a digit on the
right side to more cleanly differentiate between "minor release (a random
assortment of bug fixes, small features, etc)" and "patch release
(literally the minimum delta to give you a security fix or high impact bug
fix)".

--dave

On Fri, Dec 21, 2018 at 8:58 AM Gian Merlino <gi...@apache.org> wrote:

> I'm not too fussy around whether we do a 1.0 or simply drop the 0. and have
> it be a 14.0 or 15.0 or 16.0 or wherever we are at the time we do it. I
> also like the quarterly cadence of release-from-master we had before we got
> blocked on the ASF transition, and would like to pick that back up again
> (with the next branch cut from master at the end of January, since we did
> the 0.13.0 branch cut in late October).
>
> Seems to me that good points of discussion are, what should we use as the
> rule for incrementing the major version? Do we do what we've been doing
> (incrementing whenever there's either an incompatible change in extension
> APIs, or in query APIs, or when necessary to preserve the ability to always
> be able to roll forward/back one major release). Or do we do something else
> (Roman seems to be suggesting dropping extension APIs from consideration).
>
> And also, what does 1.0 or 14.0 or 15.0 or what-have-you mean to us? Is it
> something that should be tied to ASF graduation? Completeness of vision?
> Stability of APIs or operational characteristics? Something else? You are
> right that it is sort of a marketing/mentality thing, so it's an
> opportunity for us to declare that we feel Druid has reached some
> milestone. My feeling at this time is probably ASF graduation or
> completeness of vision (see my earlier mail for thoughts there) are the
> ones that make most sense to me.
>
> On Fri, Dec 21, 2018 at 10:41 AM Charles Allen <cr...@apache.org> wrote:
>
> > Is there any feeling in the community that the logic behind the releases
> > needs to change?
> >
> > If so then I think we should discuss what that release cadence needs to
> > look like.
> >
> > If not then dropping the 0. prefix is a marketing / mental item. Kind of
> > like the 3.x->4.x Linux kernel upgrade. If this is the case then would we
> > even want to go with 1.x? I think Roman's proposal would work fine in
> this
> > case. Where we just call it Apache Druid 14 (or 15 or whatever it is when
> > we get there) and just keep the same logic for when we release stuff,
> which
> > has been something like:
> >
> > For a X.Y release, going to a X.? release should be very straight forward
> > for anyone running stock Druid.
> > For a X.Y release, going to a (X+1).? or from a (X+1).? back to an X.Y
> > release should be feasible. It might require running a tool supported by
> > the community.
> > For a X.Y release, going to an (X+2).? or an (X-2).? is not supported.
> Some
> > things that will not have tools might have warning logs printed that the
> > functionality will change (should we change these to alerts?)
> >
> > If this sounds reasonable then jumping straight to Apache Druid 14 on the
> > first official apache release would make a lot of sense.
> >
> > Cheers,
> > Charles Allen
> >
> >
> > On Thu, Dec 20, 2018 at 11:07 PM Gian Merlino <gi...@apache.org> wrote:
> >
> > > I think it's a good point. Culturally we have been willing to break
> > > extension APIs for relatively small benefits. But we have generally
> been
> > > unwilling to make breaking changes on the operations side quite so
> > > liberally. Also, most cluster operators don't have their own custom
> > > extensions, in my experience. So it does make sense to differentiate
> > them.
> > > I'm not sure how it makes sense to differentiate them, though. It could
> > be
> > > done through the version number (only increment the major version for
> > > operations breaking changes) or it could be done through an "upgrading"
> > > guide in the documentation (increment the major version for operations
> or
> > > extension breaking changes, but, have a guide that tells people which
> > > versions have operations breaking changes to aid in upgrades).
> > >
> > > Coming back to the question in the subject of your mail: IMO, for
> > > "graduation" out of 0.x, we should talk as a community about what that
> > > means to us. It is a milestone that on the one hand, doesn't mean much,
> > but
> > > on the other hand, can be deeply symbolic. Some things that it has
> meant
> > to
> > > other projects:
> > >
> > > 1) Production readiness. Obviously Druid is well past this. If this is
> > what
> > > dropping the 0. means, then we should do it immediately.
> > >
> > > 2) Belief that the APIs have become relatively stable. Like you said,
> the
> > > extension APIs don't seem particularly close to stable, but maybe
> that's
> > > okay. However, the pace of breaking changes on the operations and query
> > > side for non-experimental features has been relatively calm for the
> past
> > > couple of years, so if we focus on that then we can make a case here.
> > >
> > > 3) Completeness of vision. This one is the most interesting to me. I
> > > suspect that different people in the community have different visions
> for
> > > Druid. It is also the kind of project that may never truly be complete
> in
> > > vision (in principle, the platform could become a competitive data
> > > warehouse, search engine, etc, …). For what it's worth, my vision of
> > Druid
> > > for the next year at least involves robust stream ingestion being a
> first
> > > class ingestion method (Kafka / Kinesis indexing service style) and SQL
> > > being a first class query language. These are both, today, still
> > > experimental features. So are lookups. All of these 3 features, from
> > what I
> > > can see, are quite popular amongst Druid users despite being
> > experimental.
> > > For a 'completeness of vision' based 1.0 I would want to lift all of
> > those
> > > out of experimental status and, for SQL in particular, to have its
> > > functionality rounded out a bit more (to support the native query
> > features
> > > it doesn't currently support, like multi-value dimensions,
> datasketches,
> > > etc).
> > >
> > > 4) Marketing / timing. Like, doing a 1.0 around the time we graduate
> from
> > > the Incubator. Not sure how much this really matters, but projects do
> it
> > > sometimes.
> > >
> > > Another question is, how often do we intend to rev the version? At the
> > rate
> > > we're going, we rev 2-3 major versions a year. Would we intend to keep
> > that
> > > up, or slow it down by making more of an effort to avoid breaking
> > changes?
> > >
> > > On Thu, Dec 20, 2018 at 2:17 PM Roman Leventov <le...@gmail.com>
> > > wrote:
> > >
> > > > It may also make sense to distinguish "operations" breaking changes
> > from
> > > > API breaking changes. Operations breaking changes establish the
> minimum
> > > > cadence of Druid cluster upgrades, that allow rolling Druid versions
> > back
> > > > and forward. I. e. it's related to segment format, the format of the
> > data
> > > > kept in ZooKeeper and the SQL database, or events such as stopping
> > > support
> > > > of ZooKeeper for certain things (e. g. forcing using of HTTP
> > > > announcements). So Druid cluster operators cannot update Druid from
> > > version
> > > > X to version Z skipping the version Y, if both Y and Z have some
> > > operations
> > > > breaking changes. (Any such changes should support rollback options
> at
> > > > least until the next version with operations breaking changes.)
> > > >
> > > > API breaking changes are just changes in Druid extensions APIs. Druid
> > > > cluster operators could skip any number of releases with such
> breaking
> > > > changes, as long as their extension's code is updated for the latest
> > > > version of API.
> > > >
> > > > On Thu, 20 Dec 2018 at 20:03, Roman Leventov <le...@apache.org>
> > > wrote:
> > > >
> > > > > It doesn't seem to me that Druid API is going to stabilize in the
> > near
> > > > > future (if ever), because there are so many extension points and
> > > > something
> > > > > is broken in every release. On the other hand, Druid is not Hadoop
> or
> > > > > Spark, which have applications API. Druid API for extensions, not
> > > > > applications. It is used by people who are closer to Druid
> > development
> > > > and
> > > > > fixing their extensions is routine.
> > > > >
> > > > > With that, I think it make sense to drop "0." from the Druid
> version
> > > and
> > > > > call it Druid 14, Druid 15, etc.
> > > > >
> > > >
> > >
> >
>

Re: Drop 0. from the version

[Gian Merlino <gi...@apache.org>]

I'm not too fussy around whether we do a 1.0 or simply drop the 0. and have
it be a 14.0 or 15.0 or 16.0 or wherever we are at the time we do it. I
also like the quarterly cadence of release-from-master we had before we got
blocked on the ASF transition, and would like to pick that back up again
(with the next branch cut from master at the end of January, since we did
the 0.13.0 branch cut in late October).

Seems to me that good points of discussion are, what should we use as the
rule for incrementing the major version? Do we do what we've been doing
(incrementing whenever there's either an incompatible change in extension
APIs, or in query APIs, or when necessary to preserve the ability to always
be able to roll forward/back one major release). Or do we do something else
(Roman seems to be suggesting dropping extension APIs from consideration).

And also, what does 1.0 or 14.0 or 15.0 or what-have-you mean to us? Is it
something that should be tied to ASF graduation? Completeness of vision?
Stability of APIs or operational characteristics? Something else? You are
right that it is sort of a marketing/mentality thing, so it's an
opportunity for us to declare that we feel Druid has reached some
milestone. My feeling at this time is probably ASF graduation or
completeness of vision (see my earlier mail for thoughts there) are the
ones that make most sense to me.

On Fri, Dec 21, 2018 at 10:41 AM Charles Allen <cr...@apache.org> wrote:

> Is there any feeling in the community that the logic behind the releases
> needs to change?
>
> If so then I think we should discuss what that release cadence needs to
> look like.
>
> If not then dropping the 0. prefix is a marketing / mental item. Kind of
> like the 3.x->4.x Linux kernel upgrade. If this is the case then would we
> even want to go with 1.x? I think Roman's proposal would work fine in this
> case. Where we just call it Apache Druid 14 (or 15 or whatever it is when
> we get there) and just keep the same logic for when we release stuff, which
> has been something like:
>
> For a X.Y release, going to a X.? release should be very straight forward
> for anyone running stock Druid.
> For a X.Y release, going to a (X+1).? or from a (X+1).? back to an X.Y
> release should be feasible. It might require running a tool supported by
> the community.
> For a X.Y release, going to an (X+2).? or an (X-2).? is not supported. Some
> things that will not have tools might have warning logs printed that the
> functionality will change (should we change these to alerts?)
>
> If this sounds reasonable then jumping straight to Apache Druid 14 on the
> first official apache release would make a lot of sense.
>
> Cheers,
> Charles Allen
>
>
> On Thu, Dec 20, 2018 at 11:07 PM Gian Merlino <gi...@apache.org> wrote:
>
> > I think it's a good point. Culturally we have been willing to break
> > extension APIs for relatively small benefits. But we have generally been
> > unwilling to make breaking changes on the operations side quite so
> > liberally. Also, most cluster operators don't have their own custom
> > extensions, in my experience. So it does make sense to differentiate
> them.
> > I'm not sure how it makes sense to differentiate them, though. It could
> be
> > done through the version number (only increment the major version for
> > operations breaking changes) or it could be done through an "upgrading"
> > guide in the documentation (increment the major version for operations or
> > extension breaking changes, but, have a guide that tells people which
> > versions have operations breaking changes to aid in upgrades).
> >
> > Coming back to the question in the subject of your mail: IMO, for
> > "graduation" out of 0.x, we should talk as a community about what that
> > means to us. It is a milestone that on the one hand, doesn't mean much,
> but
> > on the other hand, can be deeply symbolic. Some things that it has meant
> to
> > other projects:
> >
> > 1) Production readiness. Obviously Druid is well past this. If this is
> what
> > dropping the 0. means, then we should do it immediately.
> >
> > 2) Belief that the APIs have become relatively stable. Like you said, the
> > extension APIs don't seem particularly close to stable, but maybe that's
> > okay. However, the pace of breaking changes on the operations and query
> > side for non-experimental features has been relatively calm for the past
> > couple of years, so if we focus on that then we can make a case here.
> >
> > 3) Completeness of vision. This one is the most interesting to me. I
> > suspect that different people in the community have different visions for
> > Druid. It is also the kind of project that may never truly be complete in
> > vision (in principle, the platform could become a competitive data
> > warehouse, search engine, etc, …). For what it's worth, my vision of
> Druid
> > for the next year at least involves robust stream ingestion being a first
> > class ingestion method (Kafka / Kinesis indexing service style) and SQL
> > being a first class query language. These are both, today, still
> > experimental features. So are lookups. All of these 3 features, from
> what I
> > can see, are quite popular amongst Druid users despite being
> experimental.
> > For a 'completeness of vision' based 1.0 I would want to lift all of
> those
> > out of experimental status and, for SQL in particular, to have its
> > functionality rounded out a bit more (to support the native query
> features
> > it doesn't currently support, like multi-value dimensions, datasketches,
> > etc).
> >
> > 4) Marketing / timing. Like, doing a 1.0 around the time we graduate from
> > the Incubator. Not sure how much this really matters, but projects do it
> > sometimes.
> >
> > Another question is, how often do we intend to rev the version? At the
> rate
> > we're going, we rev 2-3 major versions a year. Would we intend to keep
> that
> > up, or slow it down by making more of an effort to avoid breaking
> changes?
> >
> > On Thu, Dec 20, 2018 at 2:17 PM Roman Leventov <le...@gmail.com>
> > wrote:
> >
> > > It may also make sense to distinguish "operations" breaking changes
> from
> > > API breaking changes. Operations breaking changes establish the minimum
> > > cadence of Druid cluster upgrades, that allow rolling Druid versions
> back
> > > and forward. I. e. it's related to segment format, the format of the
> data
> > > kept in ZooKeeper and the SQL database, or events such as stopping
> > support
> > > of ZooKeeper for certain things (e. g. forcing using of HTTP
> > > announcements). So Druid cluster operators cannot update Druid from
> > version
> > > X to version Z skipping the version Y, if both Y and Z have some
> > operations
> > > breaking changes. (Any such changes should support rollback options at
> > > least until the next version with operations breaking changes.)
> > >
> > > API breaking changes are just changes in Druid extensions APIs. Druid
> > > cluster operators could skip any number of releases with such breaking
> > > changes, as long as their extension's code is updated for the latest
> > > version of API.
> > >
> > > On Thu, 20 Dec 2018 at 20:03, Roman Leventov <le...@apache.org>
> > wrote:
> > >
> > > > It doesn't seem to me that Druid API is going to stabilize in the
> near
> > > > future (if ever), because there are so many extension points and
> > > something
> > > > is broken in every release. On the other hand, Druid is not Hadoop or
> > > > Spark, which have applications API. Druid API for extensions, not
> > > > applications. It is used by people who are closer to Druid
> development
> > > and
> > > > fixing their extensions is routine.
> > > >
> > > > With that, I think it make sense to drop "0." from the Druid version
> > and
> > > > call it Druid 14, Druid 15, etc.
> > > >
> > >
> >
>

Re: Drop 0. from the version

[Charles Allen <cr...@apache.org>]

Is there any feeling in the community that the logic behind the releases
needs to change?

If so then I think we should discuss what that release cadence needs to
look like.

If not then dropping the 0. prefix is a marketing / mental item. Kind of
like the 3.x->4.x Linux kernel upgrade. If this is the case then would we
even want to go with 1.x? I think Roman's proposal would work fine in this
case. Where we just call it Apache Druid 14 (or 15 or whatever it is when
we get there) and just keep the same logic for when we release stuff, which
has been something like:

For a X.Y release, going to a X.? release should be very straight forward
for anyone running stock Druid.
For a X.Y release, going to a (X+1).? or from a (X+1).? back to an X.Y
release should be feasible. It might require running a tool supported by
the community.
For a X.Y release, going to an (X+2).? or an (X-2).? is not supported. Some
things that will not have tools might have warning logs printed that the
functionality will change (should we change these to alerts?)

If this sounds reasonable then jumping straight to Apache Druid 14 on the
first official apache release would make a lot of sense.

Cheers,
Charles Allen


On Thu, Dec 20, 2018 at 11:07 PM Gian Merlino <gi...@apache.org> wrote:

> I think it's a good point. Culturally we have been willing to break
> extension APIs for relatively small benefits. But we have generally been
> unwilling to make breaking changes on the operations side quite so
> liberally. Also, most cluster operators don't have their own custom
> extensions, in my experience. So it does make sense to differentiate them.
> I'm not sure how it makes sense to differentiate them, though. It could be
> done through the version number (only increment the major version for
> operations breaking changes) or it could be done through an "upgrading"
> guide in the documentation (increment the major version for operations or
> extension breaking changes, but, have a guide that tells people which
> versions have operations breaking changes to aid in upgrades).
>
> Coming back to the question in the subject of your mail: IMO, for
> "graduation" out of 0.x, we should talk as a community about what that
> means to us. It is a milestone that on the one hand, doesn't mean much, but
> on the other hand, can be deeply symbolic. Some things that it has meant to
> other projects:
>
> 1) Production readiness. Obviously Druid is well past this. If this is what
> dropping the 0. means, then we should do it immediately.
>
> 2) Belief that the APIs have become relatively stable. Like you said, the
> extension APIs don't seem particularly close to stable, but maybe that's
> okay. However, the pace of breaking changes on the operations and query
> side for non-experimental features has been relatively calm for the past
> couple of years, so if we focus on that then we can make a case here.
>
> 3) Completeness of vision. This one is the most interesting to me. I
> suspect that different people in the community have different visions for
> Druid. It is also the kind of project that may never truly be complete in
> vision (in principle, the platform could become a competitive data
> warehouse, search engine, etc, …). For what it's worth, my vision of Druid
> for the next year at least involves robust stream ingestion being a first
> class ingestion method (Kafka / Kinesis indexing service style) and SQL
> being a first class query language. These are both, today, still
> experimental features. So are lookups. All of these 3 features, from what I
> can see, are quite popular amongst Druid users despite being experimental.
> For a 'completeness of vision' based 1.0 I would want to lift all of those
> out of experimental status and, for SQL in particular, to have its
> functionality rounded out a bit more (to support the native query features
> it doesn't currently support, like multi-value dimensions, datasketches,
> etc).
>
> 4) Marketing / timing. Like, doing a 1.0 around the time we graduate from
> the Incubator. Not sure how much this really matters, but projects do it
> sometimes.
>
> Another question is, how often do we intend to rev the version? At the rate
> we're going, we rev 2-3 major versions a year. Would we intend to keep that
> up, or slow it down by making more of an effort to avoid breaking changes?
>
> On Thu, Dec 20, 2018 at 2:17 PM Roman Leventov <le...@gmail.com>
> wrote:
>
> > It may also make sense to distinguish "operations" breaking changes from
> > API breaking changes. Operations breaking changes establish the minimum
> > cadence of Druid cluster upgrades, that allow rolling Druid versions back
> > and forward. I. e. it's related to segment format, the format of the data
> > kept in ZooKeeper and the SQL database, or events such as stopping
> support
> > of ZooKeeper for certain things (e. g. forcing using of HTTP
> > announcements). So Druid cluster operators cannot update Druid from
> version
> > X to version Z skipping the version Y, if both Y and Z have some
> operations
> > breaking changes. (Any such changes should support rollback options at
> > least until the next version with operations breaking changes.)
> >
> > API breaking changes are just changes in Druid extensions APIs. Druid
> > cluster operators could skip any number of releases with such breaking
> > changes, as long as their extension's code is updated for the latest
> > version of API.
> >
> > On Thu, 20 Dec 2018 at 20:03, Roman Leventov <le...@apache.org>
> wrote:
> >
> > > It doesn't seem to me that Druid API is going to stabilize in the near
> > > future (if ever), because there are so many extension points and
> > something
> > > is broken in every release. On the other hand, Druid is not Hadoop or
> > > Spark, which have applications API. Druid API for extensions, not
> > > applications. It is used by people who are closer to Druid development
> > and
> > > fixing their extensions is routine.
> > >
> > > With that, I think it make sense to drop "0." from the Druid version
> and
> > > call it Druid 14, Druid 15, etc.
> > >
> >
>

Re: Drop 0. from the version

[Gian Merlino <gi...@apache.org>]

I think it's a good point. Culturally we have been willing to break
extension APIs for relatively small benefits. But we have generally been
unwilling to make breaking changes on the operations side quite so
liberally. Also, most cluster operators don't have their own custom
extensions, in my experience. So it does make sense to differentiate them.
I'm not sure how it makes sense to differentiate them, though. It could be
done through the version number (only increment the major version for
operations breaking changes) or it could be done through an "upgrading"
guide in the documentation (increment the major version for operations or
extension breaking changes, but, have a guide that tells people which
versions have operations breaking changes to aid in upgrades).

Coming back to the question in the subject of your mail: IMO, for
"graduation" out of 0.x, we should talk as a community about what that
means to us. It is a milestone that on the one hand, doesn't mean much, but
on the other hand, can be deeply symbolic. Some things that it has meant to
other projects:

1) Production readiness. Obviously Druid is well past this. If this is what
dropping the 0. means, then we should do it immediately.

2) Belief that the APIs have become relatively stable. Like you said, the
extension APIs don't seem particularly close to stable, but maybe that's
okay. However, the pace of breaking changes on the operations and query
side for non-experimental features has been relatively calm for the past
couple of years, so if we focus on that then we can make a case here.

3) Completeness of vision. This one is the most interesting to me. I
suspect that different people in the community have different visions for
Druid. It is also the kind of project that may never truly be complete in
vision (in principle, the platform could become a competitive data
warehouse, search engine, etc, …). For what it's worth, my vision of Druid
for the next year at least involves robust stream ingestion being a first
class ingestion method (Kafka / Kinesis indexing service style) and SQL
being a first class query language. These are both, today, still
experimental features. So are lookups. All of these 3 features, from what I
can see, are quite popular amongst Druid users despite being experimental.
For a 'completeness of vision' based 1.0 I would want to lift all of those
out of experimental status and, for SQL in particular, to have its
functionality rounded out a bit more (to support the native query features
it doesn't currently support, like multi-value dimensions, datasketches,
etc).

4) Marketing / timing. Like, doing a 1.0 around the time we graduate from
the Incubator. Not sure how much this really matters, but projects do it
sometimes.

Another question is, how often do we intend to rev the version? At the rate
we're going, we rev 2-3 major versions a year. Would we intend to keep that
up, or slow it down by making more of an effort to avoid breaking changes?

On Thu, Dec 20, 2018 at 2:17 PM Roman Leventov <le...@gmail.com>
wrote:

> It may also make sense to distinguish "operations" breaking changes from
> API breaking changes. Operations breaking changes establish the minimum
> cadence of Druid cluster upgrades, that allow rolling Druid versions back
> and forward. I. e. it's related to segment format, the format of the data
> kept in ZooKeeper and the SQL database, or events such as stopping support
> of ZooKeeper for certain things (e. g. forcing using of HTTP
> announcements). So Druid cluster operators cannot update Druid from version
> X to version Z skipping the version Y, if both Y and Z have some operations
> breaking changes. (Any such changes should support rollback options at
> least until the next version with operations breaking changes.)
>
> API breaking changes are just changes in Druid extensions APIs. Druid
> cluster operators could skip any number of releases with such breaking
> changes, as long as their extension's code is updated for the latest
> version of API.
>
> On Thu, 20 Dec 2018 at 20:03, Roman Leventov <le...@apache.org> wrote:
>
> > It doesn't seem to me that Druid API is going to stabilize in the near
> > future (if ever), because there are so many extension points and
> something
> > is broken in every release. On the other hand, Druid is not Hadoop or
> > Spark, which have applications API. Druid API for extensions, not
> > applications. It is used by people who are closer to Druid development
> and
> > fixing their extensions is routine.
> >
> > With that, I think it make sense to drop "0." from the Druid version and
> > call it Druid 14, Druid 15, etc.
> >
>

Re: Drop 0. from the version

[Roman Leventov <le...@gmail.com>]

It may also make sense to distinguish "operations" breaking changes from
API breaking changes. Operations breaking changes establish the minimum
cadence of Druid cluster upgrades, that allow rolling Druid versions back
and forward. I. e. it's related to segment format, the format of the data
kept in ZooKeeper and the SQL database, or events such as stopping support
of ZooKeeper for certain things (e. g. forcing using of HTTP
announcements). So Druid cluster operators cannot update Druid from version
X to version Z skipping the version Y, if both Y and Z have some operations
breaking changes. (Any such changes should support rollback options at
least until the next version with operations breaking changes.)

API breaking changes are just changes in Druid extensions APIs. Druid
cluster operators could skip any number of releases with such breaking
changes, as long as their extension's code is updated for the latest
version of API.

On Thu, 20 Dec 2018 at 20:03, Roman Leventov <le...@apache.org> wrote:

> It doesn't seem to me that Druid API is going to stabilize in the near
> future (if ever), because there are so many extension points and something
> is broken in every release. On the other hand, Druid is not Hadoop or
> Spark, which have applications API. Druid API for extensions, not
> applications. It is used by people who are closer to Druid development and
> fixing their extensions is routine.
>
> With that, I think it make sense to drop "0." from the Druid version and
> call it Druid 14, Druid 15, etc.
>