You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@druid.apache.org by Gian Merlino <gi...@apache.org> on 2019/01/14 22:18:33 UTC
Experimental feature 'graduation' in 0.14
I'd like to propose graduating a couple of features out of 'experimental'
status in 0.14. Both are popular features (judging by mailing list & github
issue/PR activity). Both have been around for a while and have attained a
good level of quality and stability of API & behavior. I believe removing
the 'experimental' banner from these features would more accurately reflect
reality, and be a good signal to the user community.
1) Kafka indexing service. First introduced in Druid 0.9.1, it went through
a major protocol change in Druid 0.12.0 that added incremental publishing,
& 'mixing' of data from different partitions. Subjectively, quality appears
to be getting more solid, based on frequency of bug reports and also based
on our own experiences running this in production. Finally- I believe it is
already much more robust than Tranquility, the only 'stable' alternative.
2) Druid SQL. First introduced in Druid 0.10.0. It isn't feature complete
yet (multi-value dimensions, datasketches, etc, remain unsupported) but the
API and behavior have been generally stable. No major issues around memory
/ performance / etc regressions relative to native Druid queries are
outstanding. IMO, it is well on its way to becoming a first class way to
query Druid, and it is a good time to remove the 'experimental' banner.
Re: Experimental feature 'graduation' in 0.14
Posted by Gian Merlino <gi...@apache.org>.
I haven't seen objection, so, I will do a PR to make this change soon.
On Mon, Apr 22, 2019 at 7:04 PM Gian Merlino <gi...@apache.org> wrote:
> I think it's fine to release new query features that don't have Druid SQL
> support, although I would certainly encourage people to add SQL support
> even if it's not required. In the long run I wish that SQL could become the
> primary query language for Druid, because in Druid's space (analytical
> databases) SQL is what users generally expect to see.
>
> On Mon, Apr 22, 2019 at 6:43 PM Eyal Yurman
> <ey...@verizonmedia.com.invalid> wrote:
>
>> What does this mean for release of new query features without Druid SQL
>> support?
>>
>> On Mon, Apr 22, 2019 at 2:07 PM Jihoon Son <ji...@apache.org> wrote:
>>
>> > +1
>> > I'm mostly using only SQL.
>> >
>> > Jihoon
>> >
>> > On Mon, Apr 22, 2019 at 12:24 PM Jonathan Wei <jo...@apache.org>
>> wrote:
>> >
>> > > +1, I think it has enough feature parity with the native JSON queries,
>> > and
>> > > it's stable enough to be moved out of experimental.
>> > >
>> > > On Thu, Apr 18, 2019 at 6:23 PM Fangjin Yang <fa...@imply.io>
>> wrote:
>> > >
>> > > > Strong +1
>> > > >
>> > > > I think there's been enough production usage of Druid SQL, it
>> matches
>> > > what
>> > > > native JSON-over-http can do, and it should no longer be labeled as
>> > > > experimental.
>> > > >
>> > > > On Thu, Apr 18, 2019 at 6:06 PM Gian Merlino <gi...@apache.org>
>> wrote:
>> > > >
>> > > > > Hey all,
>> > > > >
>> > > > > Reviving this thread. Now that
>> > > > > https://github.com/apache/incubator-druid/pull/6742 and
>> > > > > https://github.com/apache/incubator-druid/pull/6862 have been
>> > released
>> > > > in
>> > > > > 0.14, I'm re-proposing graduating Druid SQL from experimental
>> status
>> > in
>> > > > the
>> > > > > next release, 0.15. I don't think we need a formal vote on this,
>> but
>> > if
>> > > > > there seems to be general consensus, I'll do a PR before the next
>> > > > 3-monthly
>> > > > > 0.15 code freeze (which is in about 2 weeks).
>> > > > >
>> > > > > On Thu, Jan 31, 2019 at 9:20 AM Gian Merlino <gi...@apache.org>
>> > wrote:
>> > > > >
>> > > > > > It sounds like the general feeling is +1 on Kafka and maybe wait
>> > > > another
>> > > > > > release for SQL. I will do a PR to mark Kafka ingest as
>> > > > non-experimental,
>> > > > > > then, and on SQL we'll see whether #6742 and #6862 look solid in
>> > > 0.14.
>> > > > > >
>> > > > > > On Tue, Jan 15, 2019 at 8:39 AM Gian Merlino <gi...@apache.org>
>> > > wrote:
>> > > > > >
>> > > > > >> Hi Mat,
>> > > > > >>
>> > > > > >> Ah, right. IMO
>> > https://github.com/apache/incubator-druid/pull/6742
>> > > > is a
>> > > > > >> decent workaround towards making #6176 less of a problem. It
>> would
>> > > > > prevent
>> > > > > >> incorrect results from happening (the broker will not start up
>> its
>> > > > http
>> > > > > >> server & announce itself, and so it won't get picked up by
>> > clients,
>> > > if
>> > > > > it
>> > > > > >> never got the initialization event). If paired with monitoring
>> > that
>> > > > > >> restarts unhealthy brokers, the issue should be fully
>> > worked-around
>> > > in
>> > > > > >> practice.
>> > > > > >>
>> > > > > >> Even though there's an (imo) viable workaround, it would still
>> be
>> > > good
>> > > > > to
>> > > > > >> fix the root cause of #6176. I just raised
>> > > > > >> https://github.com/apache/incubator-druid/pull/6862 to update
>> > > Curator
>> > > > > >> and see if that helps -- there is a bug fixed in the latest
>> > release
>> > > > that
>> > > > > >> looks like it could cause the behavior we're seeing (
>> > > > > >> https://issues.apache.org/jira/browse/CURATOR-476).
>> > > > > >>
>> > > > > >> My feeling is that it's still reasonable to remove the
>> > experimental
>> > > > > label
>> > > > > >> from Druid SQL in 0.14, especially since #6742 will make SQL
>> and
>> > > > native
>> > > > > >> queries behave at parity (initialization getting missed will
>> delay
>> > > > > broker
>> > > > > >> startup for _both_ cases). So in that sense they are at least
>> on
>> > the
>> > > > > same
>> > > > > >> footing. And hopefully #6862 will fix them both, together.
>> > > > > >>
>> > > > > >> On Tue, Jan 15, 2019 at 7:56 AM Pierre-Emile Ferron <
>> > > > > pe.ferron@gmail.com>
>> > > > > >> wrote:
>> > > > > >>
>> > > > > >>> A remaining issue with SQL is
>> > > > > >>> https://github.com/apache/incubator-druid/issues/6176
>> > > > > >>>
>> > > > > >>> We've seen it happen several times in production on 0.12,
>> where
>> > > > > >>> thankfully
>> > > > > >>> SQL doesn't power anything critical. The current workarounds
>> are:
>> > > > > >>> 1. Restart the broker. Obviously not a good solution.
>> > > > > >>> 2. Migrate to HTTP segment discovery. I'm fine with that, and
>> we
>> > > are
>> > > > > >>> actually planning to do it soon in our clusters, but I'm still
>> > > > > concerned
>> > > > > >>> about other Druid users—the default setting is still ZK, which
>> > > means
>> > > > > that
>> > > > > >>> SQL would still have this issue by default.
>> > > > > >>>
>> > > > > >>> Before marking SQL as non-experimental, I'd suggest either
>> fixing
>> > > the
>> > > > > >>> root
>> > > > > >>> cause, or making HTTP segment discovery the default and then
>> > > > explicitly
>> > > > > >>> deprecating ZK segment discovery.
>> > > > > >>>
>> > > > > >>>
>> > > > > >>> On Mon, Jan 14, 2019 at 2:18 PM Gian Merlino <gian@apache.org
>> >
>> > > > wrote:
>> > > > > >>>
>> > > > > >>> > I'd like to propose graduating a couple of features out of
>> > > > > >>> 'experimental'
>> > > > > >>> > status in 0.14. Both are popular features (judging by
>> mailing
>> > > list
>> > > > &
>> > > > > >>> github
>> > > > > >>> > issue/PR activity). Both have been around for a while and
>> have
>> > > > > >>> attained a
>> > > > > >>> > good level of quality and stability of API & behavior. I
>> > believe
>> > > > > >>> removing
>> > > > > >>> > the 'experimental' banner from these features would more
>> > > accurately
>> > > > > >>> reflect
>> > > > > >>> > reality, and be a good signal to the user community.
>> > > > > >>> >
>> > > > > >>> > 1) Kafka indexing service. First introduced in Druid 0.9.1,
>> it
>> > > went
>> > > > > >>> through
>> > > > > >>> > a major protocol change in Druid 0.12.0 that added
>> incremental
>> > > > > >>> publishing,
>> > > > > >>> > & 'mixing' of data from different partitions. Subjectively,
>> > > quality
>> > > > > >>> appears
>> > > > > >>> > to be getting more solid, based on frequency of bug reports
>> and
>> > > > also
>> > > > > >>> based
>> > > > > >>> > on our own experiences running this in production. Finally-
>> I
>> > > > believe
>> > > > > >>> it is
>> > > > > >>> > already much more robust than Tranquility, the only 'stable'
>> > > > > >>> alternative.
>> > > > > >>> >
>> > > > > >>> > 2) Druid SQL. First introduced in Druid 0.10.0. It isn't
>> > feature
>> > > > > >>> complete
>> > > > > >>> > yet (multi-value dimensions, datasketches, etc, remain
>> > > unsupported)
>> > > > > >>> but the
>> > > > > >>> > API and behavior have been generally stable. No major issues
>> > > around
>> > > > > >>> memory
>> > > > > >>> > / performance / etc regressions relative to native Druid
>> > queries
>> > > > are
>> > > > > >>> > outstanding. IMO, it is well on its way to becoming a first
>> > class
>> > > > way
>> > > > > >>> to
>> > > > > >>> > query Druid, and it is a good time to remove the
>> 'experimental'
>> > > > > banner.
>> > > > > >>> >
>> > > > > >>>
>> > > > > >>
>> > > > >
>> > > >
>> > >
>> >
>>
>
Re: Experimental feature 'graduation' in 0.14
Posted by Atul Mohan <at...@gmail.com>.
Hello Rajiv,
We're working on a Druid Presto connector within our company and hope to
contribute it once it's implemented and adequately tested.
Thanks,
Atul
Atul Mohan
On Sun, Apr 28, 2019, 17:35 Rajiv Mordani <rm...@vmware.com.invalid>
wrote:
> Has there been any thought given to integration with Presto?
>
> Rajiv.
> ________________________________
> From: Gian Merlino <gi...@apache.org>
> Sent: Monday, April 22, 2019 7:04:25 PM
> To: dev@druid.apache.org; eyurman14@oath.com
> Subject: Re: Experimental feature 'graduation' in 0.14
>
> I think it's fine to release new query features that don't have Druid SQL
> support, although I would certainly encourage people to add SQL support
> even if it's not required. In the long run I wish that SQL could become the
> primary query language for Druid, because in Druid's space (analytical
> databases) SQL is what users generally expect to see.
>
> On Mon, Apr 22, 2019 at 6:43 PM Eyal Yurman
> <ey...@verizonmedia.com.invalid> wrote:
>
> > What does this mean for release of new query features without Druid SQL
> > support?
> >
> > On Mon, Apr 22, 2019 at 2:07 PM Jihoon Son <ji...@apache.org> wrote:
> >
> > > +1
> > > I'm mostly using only SQL.
> > >
> > > Jihoon
> > >
> > > On Mon, Apr 22, 2019 at 12:24 PM Jonathan Wei <jo...@apache.org>
> wrote:
> > >
> > > > +1, I think it has enough feature parity with the native JSON
> queries,
> > > and
> > > > it's stable enough to be moved out of experimental.
> > > >
> > > > On Thu, Apr 18, 2019 at 6:23 PM Fangjin Yang <fa...@imply.io>
> wrote:
> > > >
> > > > > Strong +1
> > > > >
> > > > > I think there's been enough production usage of Druid SQL, it
> matches
> > > > what
> > > > > native JSON-over-http can do, and it should no longer be labeled as
> > > > > experimental.
> > > > >
> > > > > On Thu, Apr 18, 2019 at 6:06 PM Gian Merlino <gi...@apache.org>
> > wrote:
> > > > >
> > > > > > Hey all,
> > > > > >
> > > > > > Reviving this thread. Now that
> > > > > >
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fincubator-druid%2Fpull%2F6742&data=02%7C01%7Crmordani%40vmware.com%7Cf0ec534491994fa97b9008d6c7900db7%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C636915818853605364&sdata=Pu5WmBqEkSBEaSrN3RJxgQY4rYhxyAiB%2BD5seh2Frco%3D&reserved=0
> and
> > > > > >
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fincubator-druid%2Fpull%2F6862&data=02%7C01%7Crmordani%40vmware.com%7Cf0ec534491994fa97b9008d6c7900db7%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C636915818853615377&sdata=yBNGjp8xdqIAXxSNhs99%2BzkDdiluoMvI6jalFuZHmvU%3D&reserved=0
> have been
> > > released
> > > > > in
> > > > > > 0.14, I'm re-proposing graduating Druid SQL from experimental
> > status
> > > in
> > > > > the
> > > > > > next release, 0.15. I don't think we need a formal vote on this,
> > but
> > > if
> > > > > > there seems to be general consensus, I'll do a PR before the next
> > > > > 3-monthly
> > > > > > 0.15 code freeze (which is in about 2 weeks).
> > > > > >
> > > > > > On Thu, Jan 31, 2019 at 9:20 AM Gian Merlino <gi...@apache.org>
> > > wrote:
> > > > > >
> > > > > > > It sounds like the general feeling is +1 on Kafka and maybe
> wait
> > > > > another
> > > > > > > release for SQL. I will do a PR to mark Kafka ingest as
> > > > > non-experimental,
> > > > > > > then, and on SQL we'll see whether #6742 and #6862 look solid
> in
> > > > 0.14.
> > > > > > >
> > > > > > > On Tue, Jan 15, 2019 at 8:39 AM Gian Merlino <gi...@apache.org>
> > > > wrote:
> > > > > > >
> > > > > > >> Hi Mat,
> > > > > > >>
> > > > > > >> Ah, right. IMO
> > >
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fincubator-druid%2Fpull%2F6742&data=02%7C01%7Crmordani%40vmware.com%7Cf0ec534491994fa97b9008d6c7900db7%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C636915818853615377&sdata=cewxTcZ%2Biq7rEfJBTIZLMu50JdqLiNfs8f0aGpCCOoI%3D&reserved=0
> > > > > is a
> > > > > > >> decent workaround towards making #6176 less of a problem. It
> > would
> > > > > > prevent
> > > > > > >> incorrect results from happening (the broker will not start up
> > its
> > > > > http
> > > > > > >> server & announce itself, and so it won't get picked up by
> > > clients,
> > > > if
> > > > > > it
> > > > > > >> never got the initialization event). If paired with monitoring
> > > that
> > > > > > >> restarts unhealthy brokers, the issue should be fully
> > > worked-around
> > > > in
> > > > > > >> practice.
> > > > > > >>
> > > > > > >> Even though there's an (imo) viable workaround, it would still
> > be
> > > > good
> > > > > > to
> > > > > > >> fix the root cause of #6176. I just raised
> > > > > > >>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fincubator-druid%2Fpull%2F6862&data=02%7C01%7Crmordani%40vmware.com%7Cf0ec534491994fa97b9008d6c7900db7%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C636915818853615377&sdata=yBNGjp8xdqIAXxSNhs99%2BzkDdiluoMvI6jalFuZHmvU%3D&reserved=0
> to update
> > > > Curator
> > > > > > >> and see if that helps -- there is a bug fixed in the latest
> > > release
> > > > > that
> > > > > > >> looks like it could cause the behavior we're seeing (
> > > > > > >>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCURATOR-476&data=02%7C01%7Crmordani%40vmware.com%7Cf0ec534491994fa97b9008d6c7900db7%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C636915818853615377&sdata=e4TNiP88KIGGbGVPzlU8HRtsAT5J72bCRFLre%2BjSAVY%3D&reserved=0
> ).
> > > > > > >>
> > > > > > >> My feeling is that it's still reasonable to remove the
> > > experimental
> > > > > > label
> > > > > > >> from Druid SQL in 0.14, especially since #6742 will make SQL
> and
> > > > > native
> > > > > > >> queries behave at parity (initialization getting missed will
> > delay
> > > > > > broker
> > > > > > >> startup for _both_ cases). So in that sense they are at least
> on
> > > the
> > > > > > same
> > > > > > >> footing. And hopefully #6862 will fix them both, together.
> > > > > > >>
> > > > > > >> On Tue, Jan 15, 2019 at 7:56 AM Pierre-Emile Ferron <
> > > > > > pe.ferron@gmail.com>
> > > > > > >> wrote:
> > > > > > >>
> > > > > > >>> A remaining issue with SQL is
> > > > > > >>>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fincubator-druid%2Fissues%2F6176&data=02%7C01%7Crmordani%40vmware.com%7Cf0ec534491994fa97b9008d6c7900db7%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C636915818853615377&sdata=sQ%2ByhEXaLx7jtB0w2LZTs5WOLSO2liG8bzWqEnv%2Bxyc%3D&reserved=0
> > > > > > >>>
> > > > > > >>> We've seen it happen several times in production on 0.12,
> where
> > > > > > >>> thankfully
> > > > > > >>> SQL doesn't power anything critical. The current workarounds
> > are:
> > > > > > >>> 1. Restart the broker. Obviously not a good solution.
> > > > > > >>> 2. Migrate to HTTP segment discovery. I'm fine with that, and
> > we
> > > > are
> > > > > > >>> actually planning to do it soon in our clusters, but I'm
> still
> > > > > > concerned
> > > > > > >>> about other Druid users—the default setting is still ZK,
> which
> > > > means
> > > > > > that
> > > > > > >>> SQL would still have this issue by default.
> > > > > > >>>
> > > > > > >>> Before marking SQL as non-experimental, I'd suggest either
> > fixing
> > > > the
> > > > > > >>> root
> > > > > > >>> cause, or making HTTP segment discovery the default and then
> > > > > explicitly
> > > > > > >>> deprecating ZK segment discovery.
> > > > > > >>>
> > > > > > >>>
> > > > > > >>> On Mon, Jan 14, 2019 at 2:18 PM Gian Merlino <
> gian@apache.org>
> > > > > wrote:
> > > > > > >>>
> > > > > > >>> > I'd like to propose graduating a couple of features out of
> > > > > > >>> 'experimental'
> > > > > > >>> > status in 0.14. Both are popular features (judging by
> mailing
> > > > list
> > > > > &
> > > > > > >>> github
> > > > > > >>> > issue/PR activity). Both have been around for a while and
> > have
> > > > > > >>> attained a
> > > > > > >>> > good level of quality and stability of API & behavior. I
> > > believe
> > > > > > >>> removing
> > > > > > >>> > the 'experimental' banner from these features would more
> > > > accurately
> > > > > > >>> reflect
> > > > > > >>> > reality, and be a good signal to the user community.
> > > > > > >>> >
> > > > > > >>> > 1) Kafka indexing service. First introduced in Druid 0.9.1,
> > it
> > > > went
> > > > > > >>> through
> > > > > > >>> > a major protocol change in Druid 0.12.0 that added
> > incremental
> > > > > > >>> publishing,
> > > > > > >>> > & 'mixing' of data from different partitions. Subjectively,
> > > > quality
> > > > > > >>> appears
> > > > > > >>> > to be getting more solid, based on frequency of bug reports
> > and
> > > > > also
> > > > > > >>> based
> > > > > > >>> > on our own experiences running this in production.
> Finally- I
> > > > > believe
> > > > > > >>> it is
> > > > > > >>> > already much more robust than Tranquility, the only
> 'stable'
> > > > > > >>> alternative.
> > > > > > >>> >
> > > > > > >>> > 2) Druid SQL. First introduced in Druid 0.10.0. It isn't
> > > feature
> > > > > > >>> complete
> > > > > > >>> > yet (multi-value dimensions, datasketches, etc, remain
> > > > unsupported)
> > > > > > >>> but the
> > > > > > >>> > API and behavior have been generally stable. No major
> issues
> > > > around
> > > > > > >>> memory
> > > > > > >>> > / performance / etc regressions relative to native Druid
> > > queries
> > > > > are
> > > > > > >>> > outstanding. IMO, it is well on its way to becoming a first
> > > class
> > > > > way
> > > > > > >>> to
> > > > > > >>> > query Druid, and it is a good time to remove the
> > 'experimental'
> > > > > > banner.
> > > > > > >>> >
> > > > > > >>>
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Experimental feature 'graduation' in 0.14
Posted by Rajiv Mordani <rm...@vmware.com.INVALID>.
Has there been any thought given to integration with Presto?
Rajiv.
________________________________
From: Gian Merlino <gi...@apache.org>
Sent: Monday, April 22, 2019 7:04:25 PM
To: dev@druid.apache.org; eyurman14@oath.com
Subject: Re: Experimental feature 'graduation' in 0.14
I think it's fine to release new query features that don't have Druid SQL
support, although I would certainly encourage people to add SQL support
even if it's not required. In the long run I wish that SQL could become the
primary query language for Druid, because in Druid's space (analytical
databases) SQL is what users generally expect to see.
On Mon, Apr 22, 2019 at 6:43 PM Eyal Yurman
<ey...@verizonmedia.com.invalid> wrote:
> What does this mean for release of new query features without Druid SQL
> support?
>
> On Mon, Apr 22, 2019 at 2:07 PM Jihoon Son <ji...@apache.org> wrote:
>
> > +1
> > I'm mostly using only SQL.
> >
> > Jihoon
> >
> > On Mon, Apr 22, 2019 at 12:24 PM Jonathan Wei <jo...@apache.org> wrote:
> >
> > > +1, I think it has enough feature parity with the native JSON queries,
> > and
> > > it's stable enough to be moved out of experimental.
> > >
> > > On Thu, Apr 18, 2019 at 6:23 PM Fangjin Yang <fa...@imply.io> wrote:
> > >
> > > > Strong +1
> > > >
> > > > I think there's been enough production usage of Druid SQL, it matches
> > > what
> > > > native JSON-over-http can do, and it should no longer be labeled as
> > > > experimental.
> > > >
> > > > On Thu, Apr 18, 2019 at 6:06 PM Gian Merlino <gi...@apache.org>
> wrote:
> > > >
> > > > > Hey all,
> > > > >
> > > > > Reviving this thread. Now that
> > > > > https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fincubator-druid%2Fpull%2F6742&data=02%7C01%7Crmordani%40vmware.com%7Cf0ec534491994fa97b9008d6c7900db7%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C636915818853605364&sdata=Pu5WmBqEkSBEaSrN3RJxgQY4rYhxyAiB%2BD5seh2Frco%3D&reserved=0 and
> > > > > https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fincubator-druid%2Fpull%2F6862&data=02%7C01%7Crmordani%40vmware.com%7Cf0ec534491994fa97b9008d6c7900db7%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C636915818853615377&sdata=yBNGjp8xdqIAXxSNhs99%2BzkDdiluoMvI6jalFuZHmvU%3D&reserved=0 have been
> > released
> > > > in
> > > > > 0.14, I'm re-proposing graduating Druid SQL from experimental
> status
> > in
> > > > the
> > > > > next release, 0.15. I don't think we need a formal vote on this,
> but
> > if
> > > > > there seems to be general consensus, I'll do a PR before the next
> > > > 3-monthly
> > > > > 0.15 code freeze (which is in about 2 weeks).
> > > > >
> > > > > On Thu, Jan 31, 2019 at 9:20 AM Gian Merlino <gi...@apache.org>
> > wrote:
> > > > >
> > > > > > It sounds like the general feeling is +1 on Kafka and maybe wait
> > > > another
> > > > > > release for SQL. I will do a PR to mark Kafka ingest as
> > > > non-experimental,
> > > > > > then, and on SQL we'll see whether #6742 and #6862 look solid in
> > > 0.14.
> > > > > >
> > > > > > On Tue, Jan 15, 2019 at 8:39 AM Gian Merlino <gi...@apache.org>
> > > wrote:
> > > > > >
> > > > > >> Hi Mat,
> > > > > >>
> > > > > >> Ah, right. IMO
> > https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fincubator-druid%2Fpull%2F6742&data=02%7C01%7Crmordani%40vmware.com%7Cf0ec534491994fa97b9008d6c7900db7%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C636915818853615377&sdata=cewxTcZ%2Biq7rEfJBTIZLMu50JdqLiNfs8f0aGpCCOoI%3D&reserved=0
> > > > is a
> > > > > >> decent workaround towards making #6176 less of a problem. It
> would
> > > > > prevent
> > > > > >> incorrect results from happening (the broker will not start up
> its
> > > > http
> > > > > >> server & announce itself, and so it won't get picked up by
> > clients,
> > > if
> > > > > it
> > > > > >> never got the initialization event). If paired with monitoring
> > that
> > > > > >> restarts unhealthy brokers, the issue should be fully
> > worked-around
> > > in
> > > > > >> practice.
> > > > > >>
> > > > > >> Even though there's an (imo) viable workaround, it would still
> be
> > > good
> > > > > to
> > > > > >> fix the root cause of #6176. I just raised
> > > > > >> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fincubator-druid%2Fpull%2F6862&data=02%7C01%7Crmordani%40vmware.com%7Cf0ec534491994fa97b9008d6c7900db7%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C636915818853615377&sdata=yBNGjp8xdqIAXxSNhs99%2BzkDdiluoMvI6jalFuZHmvU%3D&reserved=0 to update
> > > Curator
> > > > > >> and see if that helps -- there is a bug fixed in the latest
> > release
> > > > that
> > > > > >> looks like it could cause the behavior we're seeing (
> > > > > >> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCURATOR-476&data=02%7C01%7Crmordani%40vmware.com%7Cf0ec534491994fa97b9008d6c7900db7%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C636915818853615377&sdata=e4TNiP88KIGGbGVPzlU8HRtsAT5J72bCRFLre%2BjSAVY%3D&reserved=0).
> > > > > >>
> > > > > >> My feeling is that it's still reasonable to remove the
> > experimental
> > > > > label
> > > > > >> from Druid SQL in 0.14, especially since #6742 will make SQL and
> > > > native
> > > > > >> queries behave at parity (initialization getting missed will
> delay
> > > > > broker
> > > > > >> startup for _both_ cases). So in that sense they are at least on
> > the
> > > > > same
> > > > > >> footing. And hopefully #6862 will fix them both, together.
> > > > > >>
> > > > > >> On Tue, Jan 15, 2019 at 7:56 AM Pierre-Emile Ferron <
> > > > > pe.ferron@gmail.com>
> > > > > >> wrote:
> > > > > >>
> > > > > >>> A remaining issue with SQL is
> > > > > >>> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fincubator-druid%2Fissues%2F6176&data=02%7C01%7Crmordani%40vmware.com%7Cf0ec534491994fa97b9008d6c7900db7%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C636915818853615377&sdata=sQ%2ByhEXaLx7jtB0w2LZTs5WOLSO2liG8bzWqEnv%2Bxyc%3D&reserved=0
> > > > > >>>
> > > > > >>> We've seen it happen several times in production on 0.12, where
> > > > > >>> thankfully
> > > > > >>> SQL doesn't power anything critical. The current workarounds
> are:
> > > > > >>> 1. Restart the broker. Obviously not a good solution.
> > > > > >>> 2. Migrate to HTTP segment discovery. I'm fine with that, and
> we
> > > are
> > > > > >>> actually planning to do it soon in our clusters, but I'm still
> > > > > concerned
> > > > > >>> about other Druid users—the default setting is still ZK, which
> > > means
> > > > > that
> > > > > >>> SQL would still have this issue by default.
> > > > > >>>
> > > > > >>> Before marking SQL as non-experimental, I'd suggest either
> fixing
> > > the
> > > > > >>> root
> > > > > >>> cause, or making HTTP segment discovery the default and then
> > > > explicitly
> > > > > >>> deprecating ZK segment discovery.
> > > > > >>>
> > > > > >>>
> > > > > >>> On Mon, Jan 14, 2019 at 2:18 PM Gian Merlino <gi...@apache.org>
> > > > wrote:
> > > > > >>>
> > > > > >>> > I'd like to propose graduating a couple of features out of
> > > > > >>> 'experimental'
> > > > > >>> > status in 0.14. Both are popular features (judging by mailing
> > > list
> > > > &
> > > > > >>> github
> > > > > >>> > issue/PR activity). Both have been around for a while and
> have
> > > > > >>> attained a
> > > > > >>> > good level of quality and stability of API & behavior. I
> > believe
> > > > > >>> removing
> > > > > >>> > the 'experimental' banner from these features would more
> > > accurately
> > > > > >>> reflect
> > > > > >>> > reality, and be a good signal to the user community.
> > > > > >>> >
> > > > > >>> > 1) Kafka indexing service. First introduced in Druid 0.9.1,
> it
> > > went
> > > > > >>> through
> > > > > >>> > a major protocol change in Druid 0.12.0 that added
> incremental
> > > > > >>> publishing,
> > > > > >>> > & 'mixing' of data from different partitions. Subjectively,
> > > quality
> > > > > >>> appears
> > > > > >>> > to be getting more solid, based on frequency of bug reports
> and
> > > > also
> > > > > >>> based
> > > > > >>> > on our own experiences running this in production. Finally- I
> > > > believe
> > > > > >>> it is
> > > > > >>> > already much more robust than Tranquility, the only 'stable'
> > > > > >>> alternative.
> > > > > >>> >
> > > > > >>> > 2) Druid SQL. First introduced in Druid 0.10.0. It isn't
> > feature
> > > > > >>> complete
> > > > > >>> > yet (multi-value dimensions, datasketches, etc, remain
> > > unsupported)
> > > > > >>> but the
> > > > > >>> > API and behavior have been generally stable. No major issues
> > > around
> > > > > >>> memory
> > > > > >>> > / performance / etc regressions relative to native Druid
> > queries
> > > > are
> > > > > >>> > outstanding. IMO, it is well on its way to becoming a first
> > class
> > > > way
> > > > > >>> to
> > > > > >>> > query Druid, and it is a good time to remove the
> 'experimental'
> > > > > banner.
> > > > > >>> >
> > > > > >>>
> > > > > >>
> > > > >
> > > >
> > >
> >
>
Re: Experimental feature 'graduation' in 0.14
Posted by Gian Merlino <gi...@apache.org>.
I think it's fine to release new query features that don't have Druid SQL
support, although I would certainly encourage people to add SQL support
even if it's not required. In the long run I wish that SQL could become the
primary query language for Druid, because in Druid's space (analytical
databases) SQL is what users generally expect to see.
On Mon, Apr 22, 2019 at 6:43 PM Eyal Yurman
<ey...@verizonmedia.com.invalid> wrote:
> What does this mean for release of new query features without Druid SQL
> support?
>
> On Mon, Apr 22, 2019 at 2:07 PM Jihoon Son <ji...@apache.org> wrote:
>
> > +1
> > I'm mostly using only SQL.
> >
> > Jihoon
> >
> > On Mon, Apr 22, 2019 at 12:24 PM Jonathan Wei <jo...@apache.org> wrote:
> >
> > > +1, I think it has enough feature parity with the native JSON queries,
> > and
> > > it's stable enough to be moved out of experimental.
> > >
> > > On Thu, Apr 18, 2019 at 6:23 PM Fangjin Yang <fa...@imply.io> wrote:
> > >
> > > > Strong +1
> > > >
> > > > I think there's been enough production usage of Druid SQL, it matches
> > > what
> > > > native JSON-over-http can do, and it should no longer be labeled as
> > > > experimental.
> > > >
> > > > On Thu, Apr 18, 2019 at 6:06 PM Gian Merlino <gi...@apache.org>
> wrote:
> > > >
> > > > > Hey all,
> > > > >
> > > > > Reviving this thread. Now that
> > > > > https://github.com/apache/incubator-druid/pull/6742 and
> > > > > https://github.com/apache/incubator-druid/pull/6862 have been
> > released
> > > > in
> > > > > 0.14, I'm re-proposing graduating Druid SQL from experimental
> status
> > in
> > > > the
> > > > > next release, 0.15. I don't think we need a formal vote on this,
> but
> > if
> > > > > there seems to be general consensus, I'll do a PR before the next
> > > > 3-monthly
> > > > > 0.15 code freeze (which is in about 2 weeks).
> > > > >
> > > > > On Thu, Jan 31, 2019 at 9:20 AM Gian Merlino <gi...@apache.org>
> > wrote:
> > > > >
> > > > > > It sounds like the general feeling is +1 on Kafka and maybe wait
> > > > another
> > > > > > release for SQL. I will do a PR to mark Kafka ingest as
> > > > non-experimental,
> > > > > > then, and on SQL we'll see whether #6742 and #6862 look solid in
> > > 0.14.
> > > > > >
> > > > > > On Tue, Jan 15, 2019 at 8:39 AM Gian Merlino <gi...@apache.org>
> > > wrote:
> > > > > >
> > > > > >> Hi Mat,
> > > > > >>
> > > > > >> Ah, right. IMO
> > https://github.com/apache/incubator-druid/pull/6742
> > > > is a
> > > > > >> decent workaround towards making #6176 less of a problem. It
> would
> > > > > prevent
> > > > > >> incorrect results from happening (the broker will not start up
> its
> > > > http
> > > > > >> server & announce itself, and so it won't get picked up by
> > clients,
> > > if
> > > > > it
> > > > > >> never got the initialization event). If paired with monitoring
> > that
> > > > > >> restarts unhealthy brokers, the issue should be fully
> > worked-around
> > > in
> > > > > >> practice.
> > > > > >>
> > > > > >> Even though there's an (imo) viable workaround, it would still
> be
> > > good
> > > > > to
> > > > > >> fix the root cause of #6176. I just raised
> > > > > >> https://github.com/apache/incubator-druid/pull/6862 to update
> > > Curator
> > > > > >> and see if that helps -- there is a bug fixed in the latest
> > release
> > > > that
> > > > > >> looks like it could cause the behavior we're seeing (
> > > > > >> https://issues.apache.org/jira/browse/CURATOR-476).
> > > > > >>
> > > > > >> My feeling is that it's still reasonable to remove the
> > experimental
> > > > > label
> > > > > >> from Druid SQL in 0.14, especially since #6742 will make SQL and
> > > > native
> > > > > >> queries behave at parity (initialization getting missed will
> delay
> > > > > broker
> > > > > >> startup for _both_ cases). So in that sense they are at least on
> > the
> > > > > same
> > > > > >> footing. And hopefully #6862 will fix them both, together.
> > > > > >>
> > > > > >> On Tue, Jan 15, 2019 at 7:56 AM Pierre-Emile Ferron <
> > > > > pe.ferron@gmail.com>
> > > > > >> wrote:
> > > > > >>
> > > > > >>> A remaining issue with SQL is
> > > > > >>> https://github.com/apache/incubator-druid/issues/6176
> > > > > >>>
> > > > > >>> We've seen it happen several times in production on 0.12, where
> > > > > >>> thankfully
> > > > > >>> SQL doesn't power anything critical. The current workarounds
> are:
> > > > > >>> 1. Restart the broker. Obviously not a good solution.
> > > > > >>> 2. Migrate to HTTP segment discovery. I'm fine with that, and
> we
> > > are
> > > > > >>> actually planning to do it soon in our clusters, but I'm still
> > > > > concerned
> > > > > >>> about other Druid users—the default setting is still ZK, which
> > > means
> > > > > that
> > > > > >>> SQL would still have this issue by default.
> > > > > >>>
> > > > > >>> Before marking SQL as non-experimental, I'd suggest either
> fixing
> > > the
> > > > > >>> root
> > > > > >>> cause, or making HTTP segment discovery the default and then
> > > > explicitly
> > > > > >>> deprecating ZK segment discovery.
> > > > > >>>
> > > > > >>>
> > > > > >>> On Mon, Jan 14, 2019 at 2:18 PM Gian Merlino <gi...@apache.org>
> > > > wrote:
> > > > > >>>
> > > > > >>> > I'd like to propose graduating a couple of features out of
> > > > > >>> 'experimental'
> > > > > >>> > status in 0.14. Both are popular features (judging by mailing
> > > list
> > > > &
> > > > > >>> github
> > > > > >>> > issue/PR activity). Both have been around for a while and
> have
> > > > > >>> attained a
> > > > > >>> > good level of quality and stability of API & behavior. I
> > believe
> > > > > >>> removing
> > > > > >>> > the 'experimental' banner from these features would more
> > > accurately
> > > > > >>> reflect
> > > > > >>> > reality, and be a good signal to the user community.
> > > > > >>> >
> > > > > >>> > 1) Kafka indexing service. First introduced in Druid 0.9.1,
> it
> > > went
> > > > > >>> through
> > > > > >>> > a major protocol change in Druid 0.12.0 that added
> incremental
> > > > > >>> publishing,
> > > > > >>> > & 'mixing' of data from different partitions. Subjectively,
> > > quality
> > > > > >>> appears
> > > > > >>> > to be getting more solid, based on frequency of bug reports
> and
> > > > also
> > > > > >>> based
> > > > > >>> > on our own experiences running this in production. Finally- I
> > > > believe
> > > > > >>> it is
> > > > > >>> > already much more robust than Tranquility, the only 'stable'
> > > > > >>> alternative.
> > > > > >>> >
> > > > > >>> > 2) Druid SQL. First introduced in Druid 0.10.0. It isn't
> > feature
> > > > > >>> complete
> > > > > >>> > yet (multi-value dimensions, datasketches, etc, remain
> > > unsupported)
> > > > > >>> but the
> > > > > >>> > API and behavior have been generally stable. No major issues
> > > around
> > > > > >>> memory
> > > > > >>> > / performance / etc regressions relative to native Druid
> > queries
> > > > are
> > > > > >>> > outstanding. IMO, it is well on its way to becoming a first
> > class
> > > > way
> > > > > >>> to
> > > > > >>> > query Druid, and it is a good time to remove the
> 'experimental'
> > > > > banner.
> > > > > >>> >
> > > > > >>>
> > > > > >>
> > > > >
> > > >
> > >
> >
>
Re: Experimental feature 'graduation' in 0.14
Posted by Eyal Yurman <ey...@verizonmedia.com.INVALID>.
What does this mean for release of new query features without Druid SQL
support?
On Mon, Apr 22, 2019 at 2:07 PM Jihoon Son <ji...@apache.org> wrote:
> +1
> I'm mostly using only SQL.
>
> Jihoon
>
> On Mon, Apr 22, 2019 at 12:24 PM Jonathan Wei <jo...@apache.org> wrote:
>
> > +1, I think it has enough feature parity with the native JSON queries,
> and
> > it's stable enough to be moved out of experimental.
> >
> > On Thu, Apr 18, 2019 at 6:23 PM Fangjin Yang <fa...@imply.io> wrote:
> >
> > > Strong +1
> > >
> > > I think there's been enough production usage of Druid SQL, it matches
> > what
> > > native JSON-over-http can do, and it should no longer be labeled as
> > > experimental.
> > >
> > > On Thu, Apr 18, 2019 at 6:06 PM Gian Merlino <gi...@apache.org> wrote:
> > >
> > > > Hey all,
> > > >
> > > > Reviving this thread. Now that
> > > > https://github.com/apache/incubator-druid/pull/6742 and
> > > > https://github.com/apache/incubator-druid/pull/6862 have been
> released
> > > in
> > > > 0.14, I'm re-proposing graduating Druid SQL from experimental status
> in
> > > the
> > > > next release, 0.15. I don't think we need a formal vote on this, but
> if
> > > > there seems to be general consensus, I'll do a PR before the next
> > > 3-monthly
> > > > 0.15 code freeze (which is in about 2 weeks).
> > > >
> > > > On Thu, Jan 31, 2019 at 9:20 AM Gian Merlino <gi...@apache.org>
> wrote:
> > > >
> > > > > It sounds like the general feeling is +1 on Kafka and maybe wait
> > > another
> > > > > release for SQL. I will do a PR to mark Kafka ingest as
> > > non-experimental,
> > > > > then, and on SQL we'll see whether #6742 and #6862 look solid in
> > 0.14.
> > > > >
> > > > > On Tue, Jan 15, 2019 at 8:39 AM Gian Merlino <gi...@apache.org>
> > wrote:
> > > > >
> > > > >> Hi Mat,
> > > > >>
> > > > >> Ah, right. IMO
> https://github.com/apache/incubator-druid/pull/6742
> > > is a
> > > > >> decent workaround towards making #6176 less of a problem. It would
> > > > prevent
> > > > >> incorrect results from happening (the broker will not start up its
> > > http
> > > > >> server & announce itself, and so it won't get picked up by
> clients,
> > if
> > > > it
> > > > >> never got the initialization event). If paired with monitoring
> that
> > > > >> restarts unhealthy brokers, the issue should be fully
> worked-around
> > in
> > > > >> practice.
> > > > >>
> > > > >> Even though there's an (imo) viable workaround, it would still be
> > good
> > > > to
> > > > >> fix the root cause of #6176. I just raised
> > > > >> https://github.com/apache/incubator-druid/pull/6862 to update
> > Curator
> > > > >> and see if that helps -- there is a bug fixed in the latest
> release
> > > that
> > > > >> looks like it could cause the behavior we're seeing (
> > > > >> https://issues.apache.org/jira/browse/CURATOR-476).
> > > > >>
> > > > >> My feeling is that it's still reasonable to remove the
> experimental
> > > > label
> > > > >> from Druid SQL in 0.14, especially since #6742 will make SQL and
> > > native
> > > > >> queries behave at parity (initialization getting missed will delay
> > > > broker
> > > > >> startup for _both_ cases). So in that sense they are at least on
> the
> > > > same
> > > > >> footing. And hopefully #6862 will fix them both, together.
> > > > >>
> > > > >> On Tue, Jan 15, 2019 at 7:56 AM Pierre-Emile Ferron <
> > > > pe.ferron@gmail.com>
> > > > >> wrote:
> > > > >>
> > > > >>> A remaining issue with SQL is
> > > > >>> https://github.com/apache/incubator-druid/issues/6176
> > > > >>>
> > > > >>> We've seen it happen several times in production on 0.12, where
> > > > >>> thankfully
> > > > >>> SQL doesn't power anything critical. The current workarounds are:
> > > > >>> 1. Restart the broker. Obviously not a good solution.
> > > > >>> 2. Migrate to HTTP segment discovery. I'm fine with that, and we
> > are
> > > > >>> actually planning to do it soon in our clusters, but I'm still
> > > > concerned
> > > > >>> about other Druid users—the default setting is still ZK, which
> > means
> > > > that
> > > > >>> SQL would still have this issue by default.
> > > > >>>
> > > > >>> Before marking SQL as non-experimental, I'd suggest either fixing
> > the
> > > > >>> root
> > > > >>> cause, or making HTTP segment discovery the default and then
> > > explicitly
> > > > >>> deprecating ZK segment discovery.
> > > > >>>
> > > > >>>
> > > > >>> On Mon, Jan 14, 2019 at 2:18 PM Gian Merlino <gi...@apache.org>
> > > wrote:
> > > > >>>
> > > > >>> > I'd like to propose graduating a couple of features out of
> > > > >>> 'experimental'
> > > > >>> > status in 0.14. Both are popular features (judging by mailing
> > list
> > > &
> > > > >>> github
> > > > >>> > issue/PR activity). Both have been around for a while and have
> > > > >>> attained a
> > > > >>> > good level of quality and stability of API & behavior. I
> believe
> > > > >>> removing
> > > > >>> > the 'experimental' banner from these features would more
> > accurately
> > > > >>> reflect
> > > > >>> > reality, and be a good signal to the user community.
> > > > >>> >
> > > > >>> > 1) Kafka indexing service. First introduced in Druid 0.9.1, it
> > went
> > > > >>> through
> > > > >>> > a major protocol change in Druid 0.12.0 that added incremental
> > > > >>> publishing,
> > > > >>> > & 'mixing' of data from different partitions. Subjectively,
> > quality
> > > > >>> appears
> > > > >>> > to be getting more solid, based on frequency of bug reports and
> > > also
> > > > >>> based
> > > > >>> > on our own experiences running this in production. Finally- I
> > > believe
> > > > >>> it is
> > > > >>> > already much more robust than Tranquility, the only 'stable'
> > > > >>> alternative.
> > > > >>> >
> > > > >>> > 2) Druid SQL. First introduced in Druid 0.10.0. It isn't
> feature
> > > > >>> complete
> > > > >>> > yet (multi-value dimensions, datasketches, etc, remain
> > unsupported)
> > > > >>> but the
> > > > >>> > API and behavior have been generally stable. No major issues
> > around
> > > > >>> memory
> > > > >>> > / performance / etc regressions relative to native Druid
> queries
> > > are
> > > > >>> > outstanding. IMO, it is well on its way to becoming a first
> class
> > > way
> > > > >>> to
> > > > >>> > query Druid, and it is a good time to remove the 'experimental'
> > > > banner.
> > > > >>> >
> > > > >>>
> > > > >>
> > > >
> > >
> >
>
Re: Experimental feature 'graduation' in 0.14
Posted by Jihoon Son <ji...@apache.org>.
+1
I'm mostly using only SQL.
Jihoon
On Mon, Apr 22, 2019 at 12:24 PM Jonathan Wei <jo...@apache.org> wrote:
> +1, I think it has enough feature parity with the native JSON queries, and
> it's stable enough to be moved out of experimental.
>
> On Thu, Apr 18, 2019 at 6:23 PM Fangjin Yang <fa...@imply.io> wrote:
>
> > Strong +1
> >
> > I think there's been enough production usage of Druid SQL, it matches
> what
> > native JSON-over-http can do, and it should no longer be labeled as
> > experimental.
> >
> > On Thu, Apr 18, 2019 at 6:06 PM Gian Merlino <gi...@apache.org> wrote:
> >
> > > Hey all,
> > >
> > > Reviving this thread. Now that
> > > https://github.com/apache/incubator-druid/pull/6742 and
> > > https://github.com/apache/incubator-druid/pull/6862 have been released
> > in
> > > 0.14, I'm re-proposing graduating Druid SQL from experimental status in
> > the
> > > next release, 0.15. I don't think we need a formal vote on this, but if
> > > there seems to be general consensus, I'll do a PR before the next
> > 3-monthly
> > > 0.15 code freeze (which is in about 2 weeks).
> > >
> > > On Thu, Jan 31, 2019 at 9:20 AM Gian Merlino <gi...@apache.org> wrote:
> > >
> > > > It sounds like the general feeling is +1 on Kafka and maybe wait
> > another
> > > > release for SQL. I will do a PR to mark Kafka ingest as
> > non-experimental,
> > > > then, and on SQL we'll see whether #6742 and #6862 look solid in
> 0.14.
> > > >
> > > > On Tue, Jan 15, 2019 at 8:39 AM Gian Merlino <gi...@apache.org>
> wrote:
> > > >
> > > >> Hi Mat,
> > > >>
> > > >> Ah, right. IMO https://github.com/apache/incubator-druid/pull/6742
> > is a
> > > >> decent workaround towards making #6176 less of a problem. It would
> > > prevent
> > > >> incorrect results from happening (the broker will not start up its
> > http
> > > >> server & announce itself, and so it won't get picked up by clients,
> if
> > > it
> > > >> never got the initialization event). If paired with monitoring that
> > > >> restarts unhealthy brokers, the issue should be fully worked-around
> in
> > > >> practice.
> > > >>
> > > >> Even though there's an (imo) viable workaround, it would still be
> good
> > > to
> > > >> fix the root cause of #6176. I just raised
> > > >> https://github.com/apache/incubator-druid/pull/6862 to update
> Curator
> > > >> and see if that helps -- there is a bug fixed in the latest release
> > that
> > > >> looks like it could cause the behavior we're seeing (
> > > >> https://issues.apache.org/jira/browse/CURATOR-476).
> > > >>
> > > >> My feeling is that it's still reasonable to remove the experimental
> > > label
> > > >> from Druid SQL in 0.14, especially since #6742 will make SQL and
> > native
> > > >> queries behave at parity (initialization getting missed will delay
> > > broker
> > > >> startup for _both_ cases). So in that sense they are at least on the
> > > same
> > > >> footing. And hopefully #6862 will fix them both, together.
> > > >>
> > > >> On Tue, Jan 15, 2019 at 7:56 AM Pierre-Emile Ferron <
> > > pe.ferron@gmail.com>
> > > >> wrote:
> > > >>
> > > >>> A remaining issue with SQL is
> > > >>> https://github.com/apache/incubator-druid/issues/6176
> > > >>>
> > > >>> We've seen it happen several times in production on 0.12, where
> > > >>> thankfully
> > > >>> SQL doesn't power anything critical. The current workarounds are:
> > > >>> 1. Restart the broker. Obviously not a good solution.
> > > >>> 2. Migrate to HTTP segment discovery. I'm fine with that, and we
> are
> > > >>> actually planning to do it soon in our clusters, but I'm still
> > > concerned
> > > >>> about other Druid users—the default setting is still ZK, which
> means
> > > that
> > > >>> SQL would still have this issue by default.
> > > >>>
> > > >>> Before marking SQL as non-experimental, I'd suggest either fixing
> the
> > > >>> root
> > > >>> cause, or making HTTP segment discovery the default and then
> > explicitly
> > > >>> deprecating ZK segment discovery.
> > > >>>
> > > >>>
> > > >>> On Mon, Jan 14, 2019 at 2:18 PM Gian Merlino <gi...@apache.org>
> > wrote:
> > > >>>
> > > >>> > I'd like to propose graduating a couple of features out of
> > > >>> 'experimental'
> > > >>> > status in 0.14. Both are popular features (judging by mailing
> list
> > &
> > > >>> github
> > > >>> > issue/PR activity). Both have been around for a while and have
> > > >>> attained a
> > > >>> > good level of quality and stability of API & behavior. I believe
> > > >>> removing
> > > >>> > the 'experimental' banner from these features would more
> accurately
> > > >>> reflect
> > > >>> > reality, and be a good signal to the user community.
> > > >>> >
> > > >>> > 1) Kafka indexing service. First introduced in Druid 0.9.1, it
> went
> > > >>> through
> > > >>> > a major protocol change in Druid 0.12.0 that added incremental
> > > >>> publishing,
> > > >>> > & 'mixing' of data from different partitions. Subjectively,
> quality
> > > >>> appears
> > > >>> > to be getting more solid, based on frequency of bug reports and
> > also
> > > >>> based
> > > >>> > on our own experiences running this in production. Finally- I
> > believe
> > > >>> it is
> > > >>> > already much more robust than Tranquility, the only 'stable'
> > > >>> alternative.
> > > >>> >
> > > >>> > 2) Druid SQL. First introduced in Druid 0.10.0. It isn't feature
> > > >>> complete
> > > >>> > yet (multi-value dimensions, datasketches, etc, remain
> unsupported)
> > > >>> but the
> > > >>> > API and behavior have been generally stable. No major issues
> around
> > > >>> memory
> > > >>> > / performance / etc regressions relative to native Druid queries
> > are
> > > >>> > outstanding. IMO, it is well on its way to becoming a first class
> > way
> > > >>> to
> > > >>> > query Druid, and it is a good time to remove the 'experimental'
> > > banner.
> > > >>> >
> > > >>>
> > > >>
> > >
> >
>
Re: Experimental feature 'graduation' in 0.14
Posted by Jonathan Wei <jo...@apache.org>.
+1, I think it has enough feature parity with the native JSON queries, and
it's stable enough to be moved out of experimental.
On Thu, Apr 18, 2019 at 6:23 PM Fangjin Yang <fa...@imply.io> wrote:
> Strong +1
>
> I think there's been enough production usage of Druid SQL, it matches what
> native JSON-over-http can do, and it should no longer be labeled as
> experimental.
>
> On Thu, Apr 18, 2019 at 6:06 PM Gian Merlino <gi...@apache.org> wrote:
>
> > Hey all,
> >
> > Reviving this thread. Now that
> > https://github.com/apache/incubator-druid/pull/6742 and
> > https://github.com/apache/incubator-druid/pull/6862 have been released
> in
> > 0.14, I'm re-proposing graduating Druid SQL from experimental status in
> the
> > next release, 0.15. I don't think we need a formal vote on this, but if
> > there seems to be general consensus, I'll do a PR before the next
> 3-monthly
> > 0.15 code freeze (which is in about 2 weeks).
> >
> > On Thu, Jan 31, 2019 at 9:20 AM Gian Merlino <gi...@apache.org> wrote:
> >
> > > It sounds like the general feeling is +1 on Kafka and maybe wait
> another
> > > release for SQL. I will do a PR to mark Kafka ingest as
> non-experimental,
> > > then, and on SQL we'll see whether #6742 and #6862 look solid in 0.14.
> > >
> > > On Tue, Jan 15, 2019 at 8:39 AM Gian Merlino <gi...@apache.org> wrote:
> > >
> > >> Hi Mat,
> > >>
> > >> Ah, right. IMO https://github.com/apache/incubator-druid/pull/6742
> is a
> > >> decent workaround towards making #6176 less of a problem. It would
> > prevent
> > >> incorrect results from happening (the broker will not start up its
> http
> > >> server & announce itself, and so it won't get picked up by clients, if
> > it
> > >> never got the initialization event). If paired with monitoring that
> > >> restarts unhealthy brokers, the issue should be fully worked-around in
> > >> practice.
> > >>
> > >> Even though there's an (imo) viable workaround, it would still be good
> > to
> > >> fix the root cause of #6176. I just raised
> > >> https://github.com/apache/incubator-druid/pull/6862 to update Curator
> > >> and see if that helps -- there is a bug fixed in the latest release
> that
> > >> looks like it could cause the behavior we're seeing (
> > >> https://issues.apache.org/jira/browse/CURATOR-476).
> > >>
> > >> My feeling is that it's still reasonable to remove the experimental
> > label
> > >> from Druid SQL in 0.14, especially since #6742 will make SQL and
> native
> > >> queries behave at parity (initialization getting missed will delay
> > broker
> > >> startup for _both_ cases). So in that sense they are at least on the
> > same
> > >> footing. And hopefully #6862 will fix them both, together.
> > >>
> > >> On Tue, Jan 15, 2019 at 7:56 AM Pierre-Emile Ferron <
> > pe.ferron@gmail.com>
> > >> wrote:
> > >>
> > >>> A remaining issue with SQL is
> > >>> https://github.com/apache/incubator-druid/issues/6176
> > >>>
> > >>> We've seen it happen several times in production on 0.12, where
> > >>> thankfully
> > >>> SQL doesn't power anything critical. The current workarounds are:
> > >>> 1. Restart the broker. Obviously not a good solution.
> > >>> 2. Migrate to HTTP segment discovery. I'm fine with that, and we are
> > >>> actually planning to do it soon in our clusters, but I'm still
> > concerned
> > >>> about other Druid users—the default setting is still ZK, which means
> > that
> > >>> SQL would still have this issue by default.
> > >>>
> > >>> Before marking SQL as non-experimental, I'd suggest either fixing the
> > >>> root
> > >>> cause, or making HTTP segment discovery the default and then
> explicitly
> > >>> deprecating ZK segment discovery.
> > >>>
> > >>>
> > >>> On Mon, Jan 14, 2019 at 2:18 PM Gian Merlino <gi...@apache.org>
> wrote:
> > >>>
> > >>> > I'd like to propose graduating a couple of features out of
> > >>> 'experimental'
> > >>> > status in 0.14. Both are popular features (judging by mailing list
> &
> > >>> github
> > >>> > issue/PR activity). Both have been around for a while and have
> > >>> attained a
> > >>> > good level of quality and stability of API & behavior. I believe
> > >>> removing
> > >>> > the 'experimental' banner from these features would more accurately
> > >>> reflect
> > >>> > reality, and be a good signal to the user community.
> > >>> >
> > >>> > 1) Kafka indexing service. First introduced in Druid 0.9.1, it went
> > >>> through
> > >>> > a major protocol change in Druid 0.12.0 that added incremental
> > >>> publishing,
> > >>> > & 'mixing' of data from different partitions. Subjectively, quality
> > >>> appears
> > >>> > to be getting more solid, based on frequency of bug reports and
> also
> > >>> based
> > >>> > on our own experiences running this in production. Finally- I
> believe
> > >>> it is
> > >>> > already much more robust than Tranquility, the only 'stable'
> > >>> alternative.
> > >>> >
> > >>> > 2) Druid SQL. First introduced in Druid 0.10.0. It isn't feature
> > >>> complete
> > >>> > yet (multi-value dimensions, datasketches, etc, remain unsupported)
> > >>> but the
> > >>> > API and behavior have been generally stable. No major issues around
> > >>> memory
> > >>> > / performance / etc regressions relative to native Druid queries
> are
> > >>> > outstanding. IMO, it is well on its way to becoming a first class
> way
> > >>> to
> > >>> > query Druid, and it is a good time to remove the 'experimental'
> > banner.
> > >>> >
> > >>>
> > >>
> >
>
Re: Experimental feature 'graduation' in 0.14
Posted by Fangjin Yang <fa...@imply.io>.
Strong +1
I think there's been enough production usage of Druid SQL, it matches what
native JSON-over-http can do, and it should no longer be labeled as
experimental.
On Thu, Apr 18, 2019 at 6:06 PM Gian Merlino <gi...@apache.org> wrote:
> Hey all,
>
> Reviving this thread. Now that
> https://github.com/apache/incubator-druid/pull/6742 and
> https://github.com/apache/incubator-druid/pull/6862 have been released in
> 0.14, I'm re-proposing graduating Druid SQL from experimental status in the
> next release, 0.15. I don't think we need a formal vote on this, but if
> there seems to be general consensus, I'll do a PR before the next 3-monthly
> 0.15 code freeze (which is in about 2 weeks).
>
> On Thu, Jan 31, 2019 at 9:20 AM Gian Merlino <gi...@apache.org> wrote:
>
> > It sounds like the general feeling is +1 on Kafka and maybe wait another
> > release for SQL. I will do a PR to mark Kafka ingest as non-experimental,
> > then, and on SQL we'll see whether #6742 and #6862 look solid in 0.14.
> >
> > On Tue, Jan 15, 2019 at 8:39 AM Gian Merlino <gi...@apache.org> wrote:
> >
> >> Hi Mat,
> >>
> >> Ah, right. IMO https://github.com/apache/incubator-druid/pull/6742 is a
> >> decent workaround towards making #6176 less of a problem. It would
> prevent
> >> incorrect results from happening (the broker will not start up its http
> >> server & announce itself, and so it won't get picked up by clients, if
> it
> >> never got the initialization event). If paired with monitoring that
> >> restarts unhealthy brokers, the issue should be fully worked-around in
> >> practice.
> >>
> >> Even though there's an (imo) viable workaround, it would still be good
> to
> >> fix the root cause of #6176. I just raised
> >> https://github.com/apache/incubator-druid/pull/6862 to update Curator
> >> and see if that helps -- there is a bug fixed in the latest release that
> >> looks like it could cause the behavior we're seeing (
> >> https://issues.apache.org/jira/browse/CURATOR-476).
> >>
> >> My feeling is that it's still reasonable to remove the experimental
> label
> >> from Druid SQL in 0.14, especially since #6742 will make SQL and native
> >> queries behave at parity (initialization getting missed will delay
> broker
> >> startup for _both_ cases). So in that sense they are at least on the
> same
> >> footing. And hopefully #6862 will fix them both, together.
> >>
> >> On Tue, Jan 15, 2019 at 7:56 AM Pierre-Emile Ferron <
> pe.ferron@gmail.com>
> >> wrote:
> >>
> >>> A remaining issue with SQL is
> >>> https://github.com/apache/incubator-druid/issues/6176
> >>>
> >>> We've seen it happen several times in production on 0.12, where
> >>> thankfully
> >>> SQL doesn't power anything critical. The current workarounds are:
> >>> 1. Restart the broker. Obviously not a good solution.
> >>> 2. Migrate to HTTP segment discovery. I'm fine with that, and we are
> >>> actually planning to do it soon in our clusters, but I'm still
> concerned
> >>> about other Druid users—the default setting is still ZK, which means
> that
> >>> SQL would still have this issue by default.
> >>>
> >>> Before marking SQL as non-experimental, I'd suggest either fixing the
> >>> root
> >>> cause, or making HTTP segment discovery the default and then explicitly
> >>> deprecating ZK segment discovery.
> >>>
> >>>
> >>> On Mon, Jan 14, 2019 at 2:18 PM Gian Merlino <gi...@apache.org> wrote:
> >>>
> >>> > I'd like to propose graduating a couple of features out of
> >>> 'experimental'
> >>> > status in 0.14. Both are popular features (judging by mailing list &
> >>> github
> >>> > issue/PR activity). Both have been around for a while and have
> >>> attained a
> >>> > good level of quality and stability of API & behavior. I believe
> >>> removing
> >>> > the 'experimental' banner from these features would more accurately
> >>> reflect
> >>> > reality, and be a good signal to the user community.
> >>> >
> >>> > 1) Kafka indexing service. First introduced in Druid 0.9.1, it went
> >>> through
> >>> > a major protocol change in Druid 0.12.0 that added incremental
> >>> publishing,
> >>> > & 'mixing' of data from different partitions. Subjectively, quality
> >>> appears
> >>> > to be getting more solid, based on frequency of bug reports and also
> >>> based
> >>> > on our own experiences running this in production. Finally- I believe
> >>> it is
> >>> > already much more robust than Tranquility, the only 'stable'
> >>> alternative.
> >>> >
> >>> > 2) Druid SQL. First introduced in Druid 0.10.0. It isn't feature
> >>> complete
> >>> > yet (multi-value dimensions, datasketches, etc, remain unsupported)
> >>> but the
> >>> > API and behavior have been generally stable. No major issues around
> >>> memory
> >>> > / performance / etc regressions relative to native Druid queries are
> >>> > outstanding. IMO, it is well on its way to becoming a first class way
> >>> to
> >>> > query Druid, and it is a good time to remove the 'experimental'
> banner.
> >>> >
> >>>
> >>
>
Re: Experimental feature 'graduation' in 0.14
Posted by Gian Merlino <gi...@apache.org>.
Hey all,
Reviving this thread. Now that
https://github.com/apache/incubator-druid/pull/6742 and
https://github.com/apache/incubator-druid/pull/6862 have been released in
0.14, I'm re-proposing graduating Druid SQL from experimental status in the
next release, 0.15. I don't think we need a formal vote on this, but if
there seems to be general consensus, I'll do a PR before the next 3-monthly
0.15 code freeze (which is in about 2 weeks).
On Thu, Jan 31, 2019 at 9:20 AM Gian Merlino <gi...@apache.org> wrote:
> It sounds like the general feeling is +1 on Kafka and maybe wait another
> release for SQL. I will do a PR to mark Kafka ingest as non-experimental,
> then, and on SQL we'll see whether #6742 and #6862 look solid in 0.14.
>
> On Tue, Jan 15, 2019 at 8:39 AM Gian Merlino <gi...@apache.org> wrote:
>
>> Hi Mat,
>>
>> Ah, right. IMO https://github.com/apache/incubator-druid/pull/6742 is a
>> decent workaround towards making #6176 less of a problem. It would prevent
>> incorrect results from happening (the broker will not start up its http
>> server & announce itself, and so it won't get picked up by clients, if it
>> never got the initialization event). If paired with monitoring that
>> restarts unhealthy brokers, the issue should be fully worked-around in
>> practice.
>>
>> Even though there's an (imo) viable workaround, it would still be good to
>> fix the root cause of #6176. I just raised
>> https://github.com/apache/incubator-druid/pull/6862 to update Curator
>> and see if that helps -- there is a bug fixed in the latest release that
>> looks like it could cause the behavior we're seeing (
>> https://issues.apache.org/jira/browse/CURATOR-476).
>>
>> My feeling is that it's still reasonable to remove the experimental label
>> from Druid SQL in 0.14, especially since #6742 will make SQL and native
>> queries behave at parity (initialization getting missed will delay broker
>> startup for _both_ cases). So in that sense they are at least on the same
>> footing. And hopefully #6862 will fix them both, together.
>>
>> On Tue, Jan 15, 2019 at 7:56 AM Pierre-Emile Ferron <pe...@gmail.com>
>> wrote:
>>
>>> A remaining issue with SQL is
>>> https://github.com/apache/incubator-druid/issues/6176
>>>
>>> We've seen it happen several times in production on 0.12, where
>>> thankfully
>>> SQL doesn't power anything critical. The current workarounds are:
>>> 1. Restart the broker. Obviously not a good solution.
>>> 2. Migrate to HTTP segment discovery. I'm fine with that, and we are
>>> actually planning to do it soon in our clusters, but I'm still concerned
>>> about other Druid users—the default setting is still ZK, which means that
>>> SQL would still have this issue by default.
>>>
>>> Before marking SQL as non-experimental, I'd suggest either fixing the
>>> root
>>> cause, or making HTTP segment discovery the default and then explicitly
>>> deprecating ZK segment discovery.
>>>
>>>
>>> On Mon, Jan 14, 2019 at 2:18 PM Gian Merlino <gi...@apache.org> wrote:
>>>
>>> > I'd like to propose graduating a couple of features out of
>>> 'experimental'
>>> > status in 0.14. Both are popular features (judging by mailing list &
>>> github
>>> > issue/PR activity). Both have been around for a while and have
>>> attained a
>>> > good level of quality and stability of API & behavior. I believe
>>> removing
>>> > the 'experimental' banner from these features would more accurately
>>> reflect
>>> > reality, and be a good signal to the user community.
>>> >
>>> > 1) Kafka indexing service. First introduced in Druid 0.9.1, it went
>>> through
>>> > a major protocol change in Druid 0.12.0 that added incremental
>>> publishing,
>>> > & 'mixing' of data from different partitions. Subjectively, quality
>>> appears
>>> > to be getting more solid, based on frequency of bug reports and also
>>> based
>>> > on our own experiences running this in production. Finally- I believe
>>> it is
>>> > already much more robust than Tranquility, the only 'stable'
>>> alternative.
>>> >
>>> > 2) Druid SQL. First introduced in Druid 0.10.0. It isn't feature
>>> complete
>>> > yet (multi-value dimensions, datasketches, etc, remain unsupported)
>>> but the
>>> > API and behavior have been generally stable. No major issues around
>>> memory
>>> > / performance / etc regressions relative to native Druid queries are
>>> > outstanding. IMO, it is well on its way to becoming a first class way
>>> to
>>> > query Druid, and it is a good time to remove the 'experimental' banner.
>>> >
>>>
>>
Re: Experimental feature 'graduation' in 0.14
Posted by Gian Merlino <gi...@apache.org>.
It sounds like the general feeling is +1 on Kafka and maybe wait another
release for SQL. I will do a PR to mark Kafka ingest as non-experimental,
then, and on SQL we'll see whether #6742 and #6862 look solid in 0.14.
On Tue, Jan 15, 2019 at 8:39 AM Gian Merlino <gi...@apache.org> wrote:
> Hi Mat,
>
> Ah, right. IMO https://github.com/apache/incubator-druid/pull/6742 is a
> decent workaround towards making #6176 less of a problem. It would prevent
> incorrect results from happening (the broker will not start up its http
> server & announce itself, and so it won't get picked up by clients, if it
> never got the initialization event). If paired with monitoring that
> restarts unhealthy brokers, the issue should be fully worked-around in
> practice.
>
> Even though there's an (imo) viable workaround, it would still be good to
> fix the root cause of #6176. I just raised
> https://github.com/apache/incubator-druid/pull/6862 to update Curator and
> see if that helps -- there is a bug fixed in the latest release that looks
> like it could cause the behavior we're seeing (
> https://issues.apache.org/jira/browse/CURATOR-476).
>
> My feeling is that it's still reasonable to remove the experimental label
> from Druid SQL in 0.14, especially since #6742 will make SQL and native
> queries behave at parity (initialization getting missed will delay broker
> startup for _both_ cases). So in that sense they are at least on the same
> footing. And hopefully #6862 will fix them both, together.
>
> On Tue, Jan 15, 2019 at 7:56 AM Pierre-Emile Ferron <pe...@gmail.com>
> wrote:
>
>> A remaining issue with SQL is
>> https://github.com/apache/incubator-druid/issues/6176
>>
>> We've seen it happen several times in production on 0.12, where thankfully
>> SQL doesn't power anything critical. The current workarounds are:
>> 1. Restart the broker. Obviously not a good solution.
>> 2. Migrate to HTTP segment discovery. I'm fine with that, and we are
>> actually planning to do it soon in our clusters, but I'm still concerned
>> about other Druid users—the default setting is still ZK, which means that
>> SQL would still have this issue by default.
>>
>> Before marking SQL as non-experimental, I'd suggest either fixing the root
>> cause, or making HTTP segment discovery the default and then explicitly
>> deprecating ZK segment discovery.
>>
>>
>> On Mon, Jan 14, 2019 at 2:18 PM Gian Merlino <gi...@apache.org> wrote:
>>
>> > I'd like to propose graduating a couple of features out of
>> 'experimental'
>> > status in 0.14. Both are popular features (judging by mailing list &
>> github
>> > issue/PR activity). Both have been around for a while and have attained
>> a
>> > good level of quality and stability of API & behavior. I believe
>> removing
>> > the 'experimental' banner from these features would more accurately
>> reflect
>> > reality, and be a good signal to the user community.
>> >
>> > 1) Kafka indexing service. First introduced in Druid 0.9.1, it went
>> through
>> > a major protocol change in Druid 0.12.0 that added incremental
>> publishing,
>> > & 'mixing' of data from different partitions. Subjectively, quality
>> appears
>> > to be getting more solid, based on frequency of bug reports and also
>> based
>> > on our own experiences running this in production. Finally- I believe
>> it is
>> > already much more robust than Tranquility, the only 'stable'
>> alternative.
>> >
>> > 2) Druid SQL. First introduced in Druid 0.10.0. It isn't feature
>> complete
>> > yet (multi-value dimensions, datasketches, etc, remain unsupported) but
>> the
>> > API and behavior have been generally stable. No major issues around
>> memory
>> > / performance / etc regressions relative to native Druid queries are
>> > outstanding. IMO, it is well on its way to becoming a first class way to
>> > query Druid, and it is a good time to remove the 'experimental' banner.
>> >
>>
>
Re: Experimental feature 'graduation' in 0.14
Posted by Gian Merlino <gi...@apache.org>.
Hi Mat,
Ah, right. IMO https://github.com/apache/incubator-druid/pull/6742 is a
decent workaround towards making #6176 less of a problem. It would prevent
incorrect results from happening (the broker will not start up its http
server & announce itself, and so it won't get picked up by clients, if it
never got the initialization event). If paired with monitoring that
restarts unhealthy brokers, the issue should be fully worked-around in
practice.
Even though there's an (imo) viable workaround, it would still be good to
fix the root cause of #6176. I just raised
https://github.com/apache/incubator-druid/pull/6862 to update Curator and
see if that helps -- there is a bug fixed in the latest release that looks
like it could cause the behavior we're seeing (
https://issues.apache.org/jira/browse/CURATOR-476).
My feeling is that it's still reasonable to remove the experimental label
from Druid SQL in 0.14, especially since #6742 will make SQL and native
queries behave at parity (initialization getting missed will delay broker
startup for _both_ cases). So in that sense they are at least on the same
footing. And hopefully #6862 will fix them both, together.
On Tue, Jan 15, 2019 at 7:56 AM Pierre-Emile Ferron <pe...@gmail.com>
wrote:
> A remaining issue with SQL is
> https://github.com/apache/incubator-druid/issues/6176
>
> We've seen it happen several times in production on 0.12, where thankfully
> SQL doesn't power anything critical. The current workarounds are:
> 1. Restart the broker. Obviously not a good solution.
> 2. Migrate to HTTP segment discovery. I'm fine with that, and we are
> actually planning to do it soon in our clusters, but I'm still concerned
> about other Druid users—the default setting is still ZK, which means that
> SQL would still have this issue by default.
>
> Before marking SQL as non-experimental, I'd suggest either fixing the root
> cause, or making HTTP segment discovery the default and then explicitly
> deprecating ZK segment discovery.
>
>
> On Mon, Jan 14, 2019 at 2:18 PM Gian Merlino <gi...@apache.org> wrote:
>
> > I'd like to propose graduating a couple of features out of 'experimental'
> > status in 0.14. Both are popular features (judging by mailing list &
> github
> > issue/PR activity). Both have been around for a while and have attained a
> > good level of quality and stability of API & behavior. I believe removing
> > the 'experimental' banner from these features would more accurately
> reflect
> > reality, and be a good signal to the user community.
> >
> > 1) Kafka indexing service. First introduced in Druid 0.9.1, it went
> through
> > a major protocol change in Druid 0.12.0 that added incremental
> publishing,
> > & 'mixing' of data from different partitions. Subjectively, quality
> appears
> > to be getting more solid, based on frequency of bug reports and also
> based
> > on our own experiences running this in production. Finally- I believe it
> is
> > already much more robust than Tranquility, the only 'stable' alternative.
> >
> > 2) Druid SQL. First introduced in Druid 0.10.0. It isn't feature complete
> > yet (multi-value dimensions, datasketches, etc, remain unsupported) but
> the
> > API and behavior have been generally stable. No major issues around
> memory
> > / performance / etc regressions relative to native Druid queries are
> > outstanding. IMO, it is well on its way to becoming a first class way to
> > query Druid, and it is a good time to remove the 'experimental' banner.
> >
>
Re: Experimental feature 'graduation' in 0.14
Posted by Pierre-Emile Ferron <pe...@gmail.com>.
A remaining issue with SQL is
https://github.com/apache/incubator-druid/issues/6176
We've seen it happen several times in production on 0.12, where thankfully
SQL doesn't power anything critical. The current workarounds are:
1. Restart the broker. Obviously not a good solution.
2. Migrate to HTTP segment discovery. I'm fine with that, and we are
actually planning to do it soon in our clusters, but I'm still concerned
about other Druid users—the default setting is still ZK, which means that
SQL would still have this issue by default.
Before marking SQL as non-experimental, I'd suggest either fixing the root
cause, or making HTTP segment discovery the default and then explicitly
deprecating ZK segment discovery.
On Mon, Jan 14, 2019 at 2:18 PM Gian Merlino <gi...@apache.org> wrote:
> I'd like to propose graduating a couple of features out of 'experimental'
> status in 0.14. Both are popular features (judging by mailing list & github
> issue/PR activity). Both have been around for a while and have attained a
> good level of quality and stability of API & behavior. I believe removing
> the 'experimental' banner from these features would more accurately reflect
> reality, and be a good signal to the user community.
>
> 1) Kafka indexing service. First introduced in Druid 0.9.1, it went through
> a major protocol change in Druid 0.12.0 that added incremental publishing,
> & 'mixing' of data from different partitions. Subjectively, quality appears
> to be getting more solid, based on frequency of bug reports and also based
> on our own experiences running this in production. Finally- I believe it is
> already much more robust than Tranquility, the only 'stable' alternative.
>
> 2) Druid SQL. First introduced in Druid 0.10.0. It isn't feature complete
> yet (multi-value dimensions, datasketches, etc, remain unsupported) but the
> API and behavior have been generally stable. No major issues around memory
> / performance / etc regressions relative to native Druid queries are
> outstanding. IMO, it is well on its way to becoming a first class way to
> query Druid, and it is a good time to remove the 'experimental' banner.
>
Re: Experimental feature 'graduation' in 0.14
Posted by Gian Merlino <gi...@apache.org>.
I am ok with officially deprecating or sunsetting Tranquility. Sunsetting
may make more sense since a few of us are still committed to fixing
critical bugs with it, but I don't know of anyone that is actively working
on new features.
It was never part of the main Druid repo, so IMO all that really needs to
be done is updating the Druid webpage (which references Tranquility in a
few places), and the Tranquility repo itself, to say that Tranquility is
deprecated/sunsetted and the recommended alternative is either the Kafka or
Kinesis indexing service. We should include some rationale as to why that
decision was made, too. To me the rationale boils down to: KIS style
ingestion doesn't have a windowPeriod restriction, doesn't lose data when
tasks fail, is lower footprint when reading from an external stream hub
like Kafka/Kinesis, and has generally proven to be easier to set up and
debug.
On Tue, Jan 15, 2019 at 4:03 AM Dylan Wylie <dy...@apache.org> wrote:
> Can attest on our clusters KIS has performed well enough to be considered
> non-experimental.
>
> As part of its promotion, can we consider "officially" deprecating
> Tranquility? Its status is a little uncertain post-apache and if there's
> consensus on deprecating it it'd be good opportunity to collate what work
> needs done to get there.
>
>
>
> On Tue, 15 Jan 2019 at 11:31, 邱明明 <cs...@gmail.com> wrote:
>
> > +1.
> >
> > Gian Merlino <gi...@apache.org> 于2019年1月15日周二 上午6:18写道：
> > >
> > > I'd like to propose graduating a couple of features out of
> 'experimental'
> > > status in 0.14. Both are popular features (judging by mailing list &
> > github
> > > issue/PR activity). Both have been around for a while and have
> attained a
> > > good level of quality and stability of API & behavior. I believe
> removing
> > > the 'experimental' banner from these features would more accurately
> > reflect
> > > reality, and be a good signal to the user community.
> > >
> > > 1) Kafka indexing service. First introduced in Druid 0.9.1, it went
> > through
> > > a major protocol change in Druid 0.12.0 that added incremental
> > publishing,
> > > & 'mixing' of data from different partitions. Subjectively, quality
> > appears
> > > to be getting more solid, based on frequency of bug reports and also
> > based
> > > on our own experiences running this in production. Finally- I believe
> it
> > is
> > > already much more robust than Tranquility, the only 'stable'
> alternative.
> > >
> > > 2) Druid SQL. First introduced in Druid 0.10.0. It isn't feature
> complete
> > > yet (multi-value dimensions, datasketches, etc, remain unsupported) but
> > the
> > > API and behavior have been generally stable. No major issues around
> > memory
> > > / performance / etc regressions relative to native Druid queries are
> > > outstanding. IMO, it is well on its way to becoming a first class way
> to
> > > query Druid, and it is a good time to remove the 'experimental' banner.
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@druid.apache.org
> > For additional commands, e-mail: dev-help@druid.apache.org
> >
> >
>
Re: Experimental feature 'graduation' in 0.14
Posted by Dylan Wylie <dy...@apache.org>.
Can attest on our clusters KIS has performed well enough to be considered
non-experimental.
As part of its promotion, can we consider "officially" deprecating
Tranquility? Its status is a little uncertain post-apache and if there's
consensus on deprecating it it'd be good opportunity to collate what work
needs done to get there.
On Tue, 15 Jan 2019 at 11:31, 邱明明 <cs...@gmail.com> wrote:
> +1.
>
> Gian Merlino <gi...@apache.org> 于2019年1月15日周二 上午6:18写道：
> >
> > I'd like to propose graduating a couple of features out of 'experimental'
> > status in 0.14. Both are popular features (judging by mailing list &
> github
> > issue/PR activity). Both have been around for a while and have attained a
> > good level of quality and stability of API & behavior. I believe removing
> > the 'experimental' banner from these features would more accurately
> reflect
> > reality, and be a good signal to the user community.
> >
> > 1) Kafka indexing service. First introduced in Druid 0.9.1, it went
> through
> > a major protocol change in Druid 0.12.0 that added incremental
> publishing,
> > & 'mixing' of data from different partitions. Subjectively, quality
> appears
> > to be getting more solid, based on frequency of bug reports and also
> based
> > on our own experiences running this in production. Finally- I believe it
> is
> > already much more robust than Tranquility, the only 'stable' alternative.
> >
> > 2) Druid SQL. First introduced in Druid 0.10.0. It isn't feature complete
> > yet (multi-value dimensions, datasketches, etc, remain unsupported) but
> the
> > API and behavior have been generally stable. No major issues around
> memory
> > / performance / etc regressions relative to native Druid queries are
> > outstanding. IMO, it is well on its way to becoming a first class way to
> > query Druid, and it is a good time to remove the 'experimental' banner.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@druid.apache.org
> For additional commands, e-mail: dev-help@druid.apache.org
>
>
Re: Experimental feature 'graduation' in 0.14
Posted by 邱明明 <cs...@gmail.com>.
+1.
Gian Merlino <gi...@apache.org> 于2019年1月15日周二 上午6:18写道：
>
> I'd like to propose graduating a couple of features out of 'experimental'
> status in 0.14. Both are popular features (judging by mailing list & github
> issue/PR activity). Both have been around for a while and have attained a
> good level of quality and stability of API & behavior. I believe removing
> the 'experimental' banner from these features would more accurately reflect
> reality, and be a good signal to the user community.
>
> 1) Kafka indexing service. First introduced in Druid 0.9.1, it went through
> a major protocol change in Druid 0.12.0 that added incremental publishing,
> & 'mixing' of data from different partitions. Subjectively, quality appears
> to be getting more solid, based on frequency of bug reports and also based
> on our own experiences running this in production. Finally- I believe it is
> already much more robust than Tranquility, the only 'stable' alternative.
>
> 2) Druid SQL. First introduced in Druid 0.10.0. It isn't feature complete
> yet (multi-value dimensions, datasketches, etc, remain unsupported) but the
> API and behavior have been generally stable. No major issues around memory
> / performance / etc regressions relative to native Druid queries are
> outstanding. IMO, it is well on its way to becoming a first class way to
> query Druid, and it is a good time to remove the 'experimental' banner.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@druid.apache.org
For additional commands, e-mail: dev-help@druid.apache.org