Plans for next Tranquility release

[Stig Rohde Døssing <st...@gmail.com>]

Hi Druid devs,

We're using Tranquility in the Storm project to allow users to write to
Druid from Storm (
https://github.com/apache/storm/tree/master/external/storm-druid), and
we've noticed that the library hasn't had a release for a few years.

Looking at the Druid mailing list and Tranquility issue tracker, it's not
obvious to me whether Tranquility will be developed going forward (
https://lists.apache.org/thread.html/ea60ba5294de5eb39761c8fd1c9a8854d2ffe4a212394078e0e67f54@%3Cdev.druid.apache.org%3E,
https://github.com/druid-io/tranquility/pull/233), or whether it is
compatible with the latest Druid releases.

Are there plans to release a new version of Tranquility any time in the
near future? Is Tranquility compatible with the newest Druid releases? Does
it make sense for Storm to continue to support integration via the
Tranquility library, or should we go another way (e.g. direct users to
ingest via Kafka instead)?

Thanks

Re: Plans for next Tranquility release

[Stig Rohde Døssing <st...@gmail.com>]

Hi Gian,

Thanks, that is very helpful. We might try to direct people to use the
Kafka ingest path as well.

2018-05-08 21:05 GMT+02:00 Gian Merlino <gi...@apache.org>:

> Hi Stig,
>
> Tranquility seems to be in the "if it ain't broke don't fix it" stage of
> its lifecycle. It still works great with the latest Druid releases.
> Although, nobody has seemed to be too interested in taking over its
> maintenance and adding new features. At Imply we do still support it for
> our users, and if it stopped working with a new Druid release then at this
> point we would be motivated to patch it.
>
> However, most of the community effort into streaming ingestion work in the
> past couple of years has gone into Kafka indexing (
> http://druid.io/docs/latest/development/extensions-core/
> kafka-ingestion.html).
> It has a number of benefits over Tranquility, namely that it is capable of
> exactly once ingestion, can read late data, and doesn't require external
> processes (although I can see how the latter one is flipped in the Storm
> case). It has also generally been much more active dev-wise than
> Tranquility.
>
> What I usually do is direct people to the Kafka indexing method and stress
> its benefits, and then if they still want to use Tranquillity (maybe they
> just really don't like Kafka) then they can go for it. It still does make
> sense if you don't need delivery guarantees and you don't want to run
> Kafka.
>
> On Tue, May 8, 2018 at 9:50 AM, Stig Rohde Døssing <stigdoessing@gmail.com
> >
> wrote:
>
> > Hi Druid devs,
> >
> > We're using Tranquility in the Storm project to allow users to write to
> > Druid from Storm (
> > https://github.com/apache/storm/tree/master/external/storm-druid), and
> > we've noticed that the library hasn't had a release for a few years.
> >
> > Looking at the Druid mailing list and Tranquility issue tracker, it's not
> > obvious to me whether Tranquility will be developed going forward (
> > https://lists.apache.org/thread.html/ea60ba5294de5eb39761c8fd1c9a88
> > 54d2ffe4a212394078e0e67f54@%3Cdev.druid.apache.org%3E,
> > https://github.com/druid-io/tranquility/pull/233), or whether it is
> > compatible with the latest Druid releases.
> >
> > Are there plans to release a new version of Tranquility any time in the
> > near future? Is Tranquility compatible with the newest Druid releases?
> Does
> > it make sense for Storm to continue to support integration via the
> > Tranquility library, or should we go another way (e.g. direct users to
> > ingest via Kafka instead)?
> >
> > Thanks
> >
>

Re: Plans for next Tranquility release

[Dyana Rose <dy...@salecycle.com>]

That's good to know about the delivery promises. We're already set up to
deal with that, but it's definitely something I didn't get a sense of from
the docs.

I'm looking to get to terms with the code base. And it looks like I'll need
to get back in touch with Scala

Thanks,
Dyana

On 9 May 2018 at 17:51, Gian Merlino <gi...@apache.org> wrote:

> It is definitely true that there are some use cases where Tranquility makes
> sense. It just hasn't attracted a dev community in the same way that Druid
> itself has (Druid has a ton of active contributors and the contributor base
> is growing, while Tranquillity has only had a few over its lifetime, and
> those folks -- one of them is me -- are no longer actively working on it).
> If anyone steps up as interested in maintaining it then the project will
> spring to life once again. That could be you! If you are interested, please
> dive in and we will totally support you.
>
> Fwiw, Tranquility is not quite at-least-once; if you lose all your Druid
> realtime tasks, then some data will be dropped. It's neither at-most-once
> nor at-least-once since it can double up data but it can also drop data.
>
> On Wed, May 9, 2018 at 1:33 AM, Dyana Rose <dy...@salecycle.com>
> wrote:
>
> > > It still does make sense if you don't need delivery guarantees and you
> > don't want to run Kafka.
> >
> > To add a data point, as some talk about Tranquility recently has had me
> > wondering if the project is going to be abandoned, this is our situation.
> > Druid is just an endpoint of a longer at-least-once pipeline.
> >
> > We distribute to Druid through embedded Tranquility as one endpoint of a
> > data-distribution hub. I would rather not have to add Kafka to our stack
> > just to stream data into Druid.
> >
> > The listed benefits of Kafka:
> > exactly-once (we already are at-least-once)
> > no external service (already have one running anyway, and adding kafka
> > would then be a new external service)
> > reading late data (we take care of this in the de-dup/data cleansing
> > process, a process which is necessary even with kafka)
> >
> > I have to assume we're not the only people with an at-least-once system,
> > where the benefits of adding Kafka look to be low.
> >
> > Dyana
> >
> > On 8 May 2018 at 20:05, Gian Merlino <gi...@apache.org> wrote:
> >
> > > Hi Stig,
> > >
> > > Tranquility seems to be in the "if it ain't broke don't fix it" stage
> of
> > > its lifecycle. It still works great with the latest Druid releases.
> > > Although, nobody has seemed to be too interested in taking over its
> > > maintenance and adding new features. At Imply we do still support it
> for
> > > our users, and if it stopped working with a new Druid release then at
> > this
> > > point we would be motivated to patch it.
> > >
> > > However, most of the community effort into streaming ingestion work in
> > the
> > > past couple of years has gone into Kafka indexing (
> > > http://druid.io/docs/latest/development/extensions-core/
> > > kafka-ingestion.html).
> > > It has a number of benefits over Tranquility, namely that it is capable
> > of
> > > exactly once ingestion, can read late data, and doesn't require
> external
> > > processes (although I can see how the latter one is flipped in the
> Storm
> > > case). It has also generally been much more active dev-wise than
> > > Tranquility.
> > >
> > > What I usually do is direct people to the Kafka indexing method and
> > stress
> > > its benefits, and then if they still want to use Tranquillity (maybe
> they
> > > just really don't like Kafka) then they can go for it. It still does
> make
> > > sense if you don't need delivery guarantees and you don't want to run
> > > Kafka.
> > >
> > > On Tue, May 8, 2018 at 9:50 AM, Stig Rohde Døssing <
> > stigdoessing@gmail.com
> > > >
> > > wrote:
> > >
> > > > Hi Druid devs,
> > > >
> > > > We're using Tranquility in the Storm project to allow users to write
> to
> > > > Druid from Storm (
> > > > https://github.com/apache/storm/tree/master/external/storm-druid),
> and
> > > > we've noticed that the library hasn't had a release for a few years.
> > > >
> > > > Looking at the Druid mailing list and Tranquility issue tracker, it's
> > not
> > > > obvious to me whether Tranquility will be developed going forward (
> > > > https://lists.apache.org/thread.html/ea60ba5294de5eb39761c8fd1c9a88
> > > > 54d2ffe4a212394078e0e67f54@%3Cdev.druid.apache.org%3E,
> > > > https://github.com/druid-io/tranquility/pull/233), or whether it is
> > > > compatible with the latest Druid releases.
> > > >
> > > > Are there plans to release a new version of Tranquility any time in
> the
> > > > near future? Is Tranquility compatible with the newest Druid
> releases?
> > > Does
> > > > it make sense for Storm to continue to support integration via the
> > > > Tranquility library, or should we go another way (e.g. direct users
> to
> > > > ingest via Kafka instead)?
> > > >
> > > > Thanks
> > > >
> > >
> >
> >
> >
> > --
> >
> > Dyana Rose
> > Software Engineer
> >
> >
> > W: www.salecycle.com <http://www.salecycle.com/>
> > [image: Marketing Permissions Service]
> > <https://t.xink.io/Tracking/Index/9LwBAKNtAAAwphkA0>
> >
>



--

Re: Plans for next Tranquility release

[Gian Merlino <gi...@apache.org>]

It is definitely true that there are some use cases where Tranquility makes
sense. It just hasn't attracted a dev community in the same way that Druid
itself has (Druid has a ton of active contributors and the contributor base
is growing, while Tranquillity has only had a few over its lifetime, and
those folks -- one of them is me -- are no longer actively working on it).
If anyone steps up as interested in maintaining it then the project will
spring to life once again. That could be you! If you are interested, please
dive in and we will totally support you.

Fwiw, Tranquility is not quite at-least-once; if you lose all your Druid
realtime tasks, then some data will be dropped. It's neither at-most-once
nor at-least-once since it can double up data but it can also drop data.

On Wed, May 9, 2018 at 1:33 AM, Dyana Rose <dy...@salecycle.com> wrote:

> > It still does make sense if you don't need delivery guarantees and you
> don't want to run Kafka.
>
> To add a data point, as some talk about Tranquility recently has had me
> wondering if the project is going to be abandoned, this is our situation.
> Druid is just an endpoint of a longer at-least-once pipeline.
>
> We distribute to Druid through embedded Tranquility as one endpoint of a
> data-distribution hub. I would rather not have to add Kafka to our stack
> just to stream data into Druid.
>
> The listed benefits of Kafka:
> exactly-once (we already are at-least-once)
> no external service (already have one running anyway, and adding kafka
> would then be a new external service)
> reading late data (we take care of this in the de-dup/data cleansing
> process, a process which is necessary even with kafka)
>
> I have to assume we're not the only people with an at-least-once system,
> where the benefits of adding Kafka look to be low.
>
> Dyana
>
> On 8 May 2018 at 20:05, Gian Merlino <gi...@apache.org> wrote:
>
> > Hi Stig,
> >
> > Tranquility seems to be in the "if it ain't broke don't fix it" stage of
> > its lifecycle. It still works great with the latest Druid releases.
> > Although, nobody has seemed to be too interested in taking over its
> > maintenance and adding new features. At Imply we do still support it for
> > our users, and if it stopped working with a new Druid release then at
> this
> > point we would be motivated to patch it.
> >
> > However, most of the community effort into streaming ingestion work in
> the
> > past couple of years has gone into Kafka indexing (
> > http://druid.io/docs/latest/development/extensions-core/
> > kafka-ingestion.html).
> > It has a number of benefits over Tranquility, namely that it is capable
> of
> > exactly once ingestion, can read late data, and doesn't require external
> > processes (although I can see how the latter one is flipped in the Storm
> > case). It has also generally been much more active dev-wise than
> > Tranquility.
> >
> > What I usually do is direct people to the Kafka indexing method and
> stress
> > its benefits, and then if they still want to use Tranquillity (maybe they
> > just really don't like Kafka) then they can go for it. It still does make
> > sense if you don't need delivery guarantees and you don't want to run
> > Kafka.
> >
> > On Tue, May 8, 2018 at 9:50 AM, Stig Rohde Døssing <
> stigdoessing@gmail.com
> > >
> > wrote:
> >
> > > Hi Druid devs,
> > >
> > > We're using Tranquility in the Storm project to allow users to write to
> > > Druid from Storm (
> > > https://github.com/apache/storm/tree/master/external/storm-druid), and
> > > we've noticed that the library hasn't had a release for a few years.
> > >
> > > Looking at the Druid mailing list and Tranquility issue tracker, it's
> not
> > > obvious to me whether Tranquility will be developed going forward (
> > > https://lists.apache.org/thread.html/ea60ba5294de5eb39761c8fd1c9a88
> > > 54d2ffe4a212394078e0e67f54@%3Cdev.druid.apache.org%3E,
> > > https://github.com/druid-io/tranquility/pull/233), or whether it is
> > > compatible with the latest Druid releases.
> > >
> > > Are there plans to release a new version of Tranquility any time in the
> > > near future? Is Tranquility compatible with the newest Druid releases?
> > Does
> > > it make sense for Storm to continue to support integration via the
> > > Tranquility library, or should we go another way (e.g. direct users to
> > > ingest via Kafka instead)?
> > >
> > > Thanks
> > >
> >
>
>
>
> --
>
> Dyana Rose
> Software Engineer
>
>
> W: www.salecycle.com <http://www.salecycle.com/>
> [image: Marketing Permissions Service]
> <https://t.xink.io/Tracking/Index/9LwBAKNtAAAwphkA0>
>

Re: Plans for next Tranquility release

[Dyana Rose <dy...@salecycle.com>]

> It still does make sense if you don't need delivery guarantees and you
don't want to run Kafka.

To add a data point, as some talk about Tranquility recently has had me
wondering if the project is going to be abandoned, this is our situation.
Druid is just an endpoint of a longer at-least-once pipeline.

We distribute to Druid through embedded Tranquility as one endpoint of a
data-distribution hub. I would rather not have to add Kafka to our stack
just to stream data into Druid.

The listed benefits of Kafka:
exactly-once (we already are at-least-once)
no external service (already have one running anyway, and adding kafka
would then be a new external service)
reading late data (we take care of this in the de-dup/data cleansing
process, a process which is necessary even with kafka)

I have to assume we're not the only people with an at-least-once system,
where the benefits of adding Kafka look to be low.

Dyana

On 8 May 2018 at 20:05, Gian Merlino <gi...@apache.org> wrote:

> Hi Stig,
>
> Tranquility seems to be in the "if it ain't broke don't fix it" stage of
> its lifecycle. It still works great with the latest Druid releases.
> Although, nobody has seemed to be too interested in taking over its
> maintenance and adding new features. At Imply we do still support it for
> our users, and if it stopped working with a new Druid release then at this
> point we would be motivated to patch it.
>
> However, most of the community effort into streaming ingestion work in the
> past couple of years has gone into Kafka indexing (
> http://druid.io/docs/latest/development/extensions-core/
> kafka-ingestion.html).
> It has a number of benefits over Tranquility, namely that it is capable of
> exactly once ingestion, can read late data, and doesn't require external
> processes (although I can see how the latter one is flipped in the Storm
> case). It has also generally been much more active dev-wise than
> Tranquility.
>
> What I usually do is direct people to the Kafka indexing method and stress
> its benefits, and then if they still want to use Tranquillity (maybe they
> just really don't like Kafka) then they can go for it. It still does make
> sense if you don't need delivery guarantees and you don't want to run
> Kafka.
>
> On Tue, May 8, 2018 at 9:50 AM, Stig Rohde Døssing <stigdoessing@gmail.com
> >
> wrote:
>
> > Hi Druid devs,
> >
> > We're using Tranquility in the Storm project to allow users to write to
> > Druid from Storm (
> > https://github.com/apache/storm/tree/master/external/storm-druid), and
> > we've noticed that the library hasn't had a release for a few years.
> >
> > Looking at the Druid mailing list and Tranquility issue tracker, it's not
> > obvious to me whether Tranquility will be developed going forward (
> > https://lists.apache.org/thread.html/ea60ba5294de5eb39761c8fd1c9a88
> > 54d2ffe4a212394078e0e67f54@%3Cdev.druid.apache.org%3E,
> > https://github.com/druid-io/tranquility/pull/233), or whether it is
> > compatible with the latest Druid releases.
> >
> > Are there plans to release a new version of Tranquility any time in the
> > near future? Is Tranquility compatible with the newest Druid releases?
> Does
> > it make sense for Storm to continue to support integration via the
> > Tranquility library, or should we go another way (e.g. direct users to
> > ingest via Kafka instead)?
> >
> > Thanks
> >
>



-- 

Dyana Rose
Software Engineer


W: www.salecycle.com <http://www.salecycle.com/>
[image: Marketing Permissions Service]
<https://t.xink.io/Tracking/Index/9LwBAKNtAAAwphkA0>

Re: Plans for next Tranquility release

[Gian Merlino <gi...@apache.org>]

Hi Stig,

Tranquility seems to be in the "if it ain't broke don't fix it" stage of
its lifecycle. It still works great with the latest Druid releases.
Although, nobody has seemed to be too interested in taking over its
maintenance and adding new features. At Imply we do still support it for
our users, and if it stopped working with a new Druid release then at this
point we would be motivated to patch it.

However, most of the community effort into streaming ingestion work in the
past couple of years has gone into Kafka indexing (
http://druid.io/docs/latest/development/extensions-core/kafka-ingestion.html).
It has a number of benefits over Tranquility, namely that it is capable of
exactly once ingestion, can read late data, and doesn't require external
processes (although I can see how the latter one is flipped in the Storm
case). It has also generally been much more active dev-wise than
Tranquility.

What I usually do is direct people to the Kafka indexing method and stress
its benefits, and then if they still want to use Tranquillity (maybe they
just really don't like Kafka) then they can go for it. It still does make
sense if you don't need delivery guarantees and you don't want to run Kafka.

On Tue, May 8, 2018 at 9:50 AM, Stig Rohde Døssing <st...@gmail.com>
wrote:

> Hi Druid devs,
>
> We're using Tranquility in the Storm project to allow users to write to
> Druid from Storm (
> https://github.com/apache/storm/tree/master/external/storm-druid), and
> we've noticed that the library hasn't had a release for a few years.
>
> Looking at the Druid mailing list and Tranquility issue tracker, it's not
> obvious to me whether Tranquility will be developed going forward (
> https://lists.apache.org/thread.html/ea60ba5294de5eb39761c8fd1c9a88
> 54d2ffe4a212394078e0e67f54@%3Cdev.druid.apache.org%3E,
> https://github.com/druid-io/tranquility/pull/233), or whether it is
> compatible with the latest Druid releases.
>
> Are there plans to release a new version of Tranquility any time in the
> near future? Is Tranquility compatible with the newest Druid releases? Does
> it make sense for Storm to continue to support integration via the
> Tranquility library, or should we go another way (e.g. direct users to
> ingest via Kafka instead)?
>
> Thanks
>