anyone working on stream processing?

[pi song <pi...@gmail.com>]

Dear everyone,

I've just recognized this file  "TimeWindowSpec.java" in Pig. Is anyone
currently working on stream processing?

Cheers,
Pi

Re: anyone working on stream processing?

[pi song <pi...@gmail.com>]

As incremental update mechanism that utilizes sliding window concept, I
think it is fine.

But for real-time stream processing, I can foresee a lot of workarounds to
make it work using batch-oriented map/reduce.

Pi


On 3/18/08, Chris Olston <ol...@yahoo-inc.com> wrote:
>
> As Utkarsh said, this is a road we started down a while back, but did
> not continue.
>
> If people have use-cases for stream processing in the pig/hadoop
> context, I'd like to hear about it.
>
> -Chris
>
>
> On Mar 17, 2008, at 1:54 PM, pi song wrote:
>
> > Dear everyone,
> >
> > I've just recognized this file  "TimeWindowSpec.java" in Pig. Is
> > anyone
> > currently working on stream processing?
> >
> > Cheers,
> > Pi
>
> --
> Christopher Olston, Ph.D.
> Sr. Research Scientist
> Yahoo! Research
>
>
>

Re: anyone working on stream processing?

[Chris Olston <ol...@yahoo-inc.com>]

As Utkarsh said, this is a road we started down a while back, but did  
not continue.

If people have use-cases for stream processing in the pig/hadoop  
context, I'd like to hear about it.

-Chris


On Mar 17, 2008, at 1:54 PM, pi song wrote:

> Dear everyone,
>
> I've just recognized this file  "TimeWindowSpec.java" in Pig. Is  
> anyone
> currently working on stream processing?
>
> Cheers,
> Pi

--
Christopher Olston, Ph.D.
Sr. Research Scientist
Yahoo! Research

Re: anyone working on stream processing?

[Benjamin Reed <br...@yahoo-inc.com>]

You are correct the TimeWindowSpec was some work we were doing on incremental 
processing. We haven't been working on it for a while.

ben

On Monday 17 March 2008 14:08:29 pi song wrote:
> I feel that this "streaming" in the source file is not in the sense of
> using scripts to run map/reduce
> but rather "sliding window stream processing"  (See
> http://www.cs.cornell.edu/~mirek/papers/sigmod2003.pdf)
>
> Pi
>
> On Tue, Mar 18, 2008 at 7:57 AM, Olga Natkovich <ol...@yahoo-inc.com> wrote:
> > We are working on support for streaming in pig similar to hadoop
> > streaming. Not sure if that's what you are asking. See details in
> > http://wiki.apache.org/pig/PigStreamingFunctionalSpec.
> >
> > Olga
> >
> > > -----Original Message-----
> > > From: pi song [mailto:pi.songs@gmail.com]
> > > Sent: Monday, March 17, 2008 1:54 PM
> > > To: pig-dev@incubator.apache.org
> > > Subject: anyone working on stream processing?
> > >
> > > Dear everyone,
> > >
> > > I've just recognized this file  "TimeWindowSpec.java" in Pig.
> > > Is anyone currently working on stream processing?
> > >
> > > Cheers,
> > > Pi

RE: anyone working on stream processing?

[Utkarsh Srivastava <ut...@yahoo-inc.com>]

We were working on putting stream processing in Pig, but that project
has been inactive for a long time.

Olga, TimeWindowSpec and TupleWindowSpec can be removed. (I thought they
had been removed when we moved to apache).

Utkarsh

> -----Original Message-----
> From: pi song [mailto:pi.songs@gmail.com]
> Sent: Monday, March 17, 2008 1:08 PM
> To: pig-dev@incubator.apache.org
> Subject: Re: anyone working on stream processing?
> 
> I feel that this "streaming" in the source file is not in the sense of
> using
> scripts to run map/reduce
> but rather "sliding window stream processing"  (See
> http://www.cs.cornell.edu/~mirek/papers/sigmod2003.pdf)
> 
> Pi
> 
> On Tue, Mar 18, 2008 at 7:57 AM, Olga Natkovich <ol...@yahoo-inc.com>
> wrote:
> 
> > We are working on support for streaming in pig similar to hadoop
> > streaming. Not sure if that's what you are asking. See details in
> > http://wiki.apache.org/pig/PigStreamingFunctionalSpec.
> >
> > Olga
> >
> > > -----Original Message-----
> > > From: pi song [mailto:pi.songs@gmail.com]
> > > Sent: Monday, March 17, 2008 1:54 PM
> > > To: pig-dev@incubator.apache.org
> > > Subject: anyone working on stream processing?
> > >
> > > Dear everyone,
> > >
> > > I've just recognized this file  "TimeWindowSpec.java" in Pig.
> > > Is anyone currently working on stream processing?
> > >
> > > Cheers,
> > > Pi
> > >
> >

Re: anyone working on stream processing?

[pi song <pi...@gmail.com>]

I feel that this "streaming" in the source file is not in the sense of using
scripts to run map/reduce
but rather "sliding window stream processing"  (See
http://www.cs.cornell.edu/~mirek/papers/sigmod2003.pdf)

Pi

On Tue, Mar 18, 2008 at 7:57 AM, Olga Natkovich <ol...@yahoo-inc.com> wrote:

> We are working on support for streaming in pig similar to hadoop
> streaming. Not sure if that's what you are asking. See details in
> http://wiki.apache.org/pig/PigStreamingFunctionalSpec.
>
> Olga
>
> > -----Original Message-----
> > From: pi song [mailto:pi.songs@gmail.com]
> > Sent: Monday, March 17, 2008 1:54 PM
> > To: pig-dev@incubator.apache.org
> > Subject: anyone working on stream processing?
> >
> > Dear everyone,
> >
> > I've just recognized this file  "TimeWindowSpec.java" in Pig.
> > Is anyone currently working on stream processing?
> >
> > Cheers,
> > Pi
> >
>

RE: anyone working on stream processing?

[Olga Natkovich <ol...@yahoo-inc.com>]

We are working on support for streaming in pig similar to hadoop
streaming. Not sure if that's what you are asking. See details in
http://wiki.apache.org/pig/PigStreamingFunctionalSpec.

Olga 

> -----Original Message-----
> From: pi song [mailto:pi.songs@gmail.com] 
> Sent: Monday, March 17, 2008 1:54 PM
> To: pig-dev@incubator.apache.org
> Subject: anyone working on stream processing?
> 
> Dear everyone,
> 
> I've just recognized this file  "TimeWindowSpec.java" in Pig. 
> Is anyone currently working on stream processing?
> 
> Cheers,
> Pi
>