You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@samza.apache.org by jeremy p <at...@gmail.com> on 2015/10/21 17:15:15 UTC
How to do aggregation in Samza?
Hey all,
So, I'm wanting to do aggregate operations in Samza. Counts, averages,
grouping, things of that nature. Basically, the kinds of aggregate
operations you can do in SQL. What's the best way to do this in Samza?
Are there any libraries for this?
I noticed a few projects are currently in development. Have any of them
been used in a production environment? Are any of them being actively
developed?
>From combing through the mailing list archives, I can see that LinkedIn
plans on adding SQL and aggregation to Samza. Any idea what the timeline
is for this?
--Jeremy
Re: How to do aggregation in Samza?
Posted by Thomas Bernhardt <be...@yahoo.com.INVALID>.
Take a look at CEP engines for JVMs, i.e. Drools, Esper, SiddhiBest regards,Tom
From: jeremy p <at...@gmail.com>
To: dev <de...@samza.apache.org>
Sent: Wednesday, October 21, 2015 8:23 PM
Subject: Re: How to do aggregation in Samza?
Wow, Calcite looks really really really cool! It's too bad this feature
isn't further along. My current project calls for aggregation, and we
don't have the resources to roll our own aggregation engine. Perhaps we'll
go with Spark Streaming or Storm for now, and then switch back to Samza
once streaming aggregation has been fully-implemented.
On Wed, Oct 21, 2015 at 5:44 PM, Selina Tech <sw...@gmail.com> wrote:
> Hi, All:
>
> I have same question also. Previously, I thought we just use java code to
> implement, Counts, averages ... in Samza.
> Is anyone kenw any Java libraries?
>
> Sincerely,
> Selina
>
> On Wed, Oct 21, 2015 at 8:15 AM, jeremy p <at...@gmail.com>
> wrote:
>
> > Hey all,
> >
> > So, I'm wanting to do aggregate operations in Samza. Counts, averages,
> > grouping, things of that nature. Basically, the kinds of aggregate
> > operations you can do in SQL. What's the best way to do this in Samza?
> > Are there any libraries for this?
> >
> > I noticed a few projects are currently in development. Have any of them
> > been used in a production environment? Are any of them being actively
> > developed?
> >
> > From combing through the mailing list archives, I can see that LinkedIn
> > plans on adding SQL and aggregation to Samza. Any idea what the timeline
> > is for this?
> >
> > --Jeremy
> >
>
Re: How to do aggregation in Samza?
Posted by jeremy p <at...@gmail.com>.
Wow, Calcite looks really really really cool! It's too bad this feature
isn't further along. My current project calls for aggregation, and we
don't have the resources to roll our own aggregation engine. Perhaps we'll
go with Spark Streaming or Storm for now, and then switch back to Samza
once streaming aggregation has been fully-implemented.
On Wed, Oct 21, 2015 at 5:44 PM, Selina Tech <sw...@gmail.com> wrote:
> Hi, All:
>
> I have same question also. Previously, I thought we just use java code to
> implement, Counts, averages ... in Samza.
> Is anyone kenw any Java libraries?
>
> Sincerely,
> Selina
>
> On Wed, Oct 21, 2015 at 8:15 AM, jeremy p <at...@gmail.com>
> wrote:
>
> > Hey all,
> >
> > So, I'm wanting to do aggregate operations in Samza. Counts, averages,
> > grouping, things of that nature. Basically, the kinds of aggregate
> > operations you can do in SQL. What's the best way to do this in Samza?
> > Are there any libraries for this?
> >
> > I noticed a few projects are currently in development. Have any of them
> > been used in a production environment? Are any of them being actively
> > developed?
> >
> > From combing through the mailing list archives, I can see that LinkedIn
> > plans on adding SQL and aggregation to Samza. Any idea what the timeline
> > is for this?
> >
> > --Jeremy
> >
>
Re: How to do aggregation in Samza?
Posted by Selina Tech <sw...@gmail.com>.
Hi, All:
I have same question also. Previously, I thought we just use java code to
implement, Counts, averages ... in Samza.
Is anyone kenw any Java libraries?
Sincerely,
Selina
On Wed, Oct 21, 2015 at 8:15 AM, jeremy p <at...@gmail.com>
wrote:
> Hey all,
>
> So, I'm wanting to do aggregate operations in Samza. Counts, averages,
> grouping, things of that nature. Basically, the kinds of aggregate
> operations you can do in SQL. What's the best way to do this in Samza?
> Are there any libraries for this?
>
> I noticed a few projects are currently in development. Have any of them
> been used in a production environment? Are any of them being actively
> developed?
>
> From combing through the mailing list archives, I can see that LinkedIn
> plans on adding SQL and aggregation to Samza. Any idea what the timeline
> is for this?
>
> --Jeremy
>
Re: How to do aggregation in Samza?
Posted by Julian Hyde <jh...@apache.org>.
I am helping with the SQL support. I don’t know timelines but I wanted to chime in on the different aggregate operations.
There are several ways to aggregate streams: tumbling, hopping, sliding windows. For example, if you want to periodically emit totals that collapse many rows into one total, you want a tumbling window. Read through [1] and find out which kind of window you need for your application. Ultimately Samza will support each of these kinds of windows, via SQL and via its API.
Julian
[1] https://calcite.incubator.apache.org/docs/stream.html
> On Oct 21, 2015, at 8:15 AM, jeremy p <at...@gmail.com> wrote:
>
> Hey all,
>
> So, I'm wanting to do aggregate operations in Samza. Counts, averages,
> grouping, things of that nature. Basically, the kinds of aggregate
> operations you can do in SQL. What's the best way to do this in Samza?
> Are there any libraries for this?
>
> I noticed a few projects are currently in development. Have any of them
> been used in a production environment? Are any of them being actively
> developed?
>
> From combing through the mailing list archives, I can see that LinkedIn
> plans on adding SQL and aggregation to Samza. Any idea what the timeline
> is for this?
>
> --Jeremy