You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@samza.apache.org by "Yi Pan (Data Infrastructure)" <yi...@linkedin.com> on 2015/05/01 19:05:30 UTC
Review Request 33749: WIP: SAMZA-650 window store implementation
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33749/
-----------------------------------------------------------
Review request for samza, Chris Riccomini, Guozhang Wang, Milinda Pathirage, Navina Ramesh, and Naveen Somasundaram.
Bugs: SAMZA-650
https://issues.apache.org/jira/browse/SAMZA-650
Repository: samza
Description
-------
WIP: SAMZA-650 window store implementation
First patch to implemement window store and message store.
- There are changes made in window operator and join operators to illustrate the usage of window store and message store. However, the implementation of those classes is not completed.
- There is a Calcite dependency failure reported in SAMZA-659. Not fixed yet.
- Unit tests are still WIP and will be updated later
Diffs
-----
build.gradle 97de3a28f6379e3862eec845da87587b1d4f742e
gradle/dependency-versions.gradle ee6dfc411b7ab90b187df79f109884127953862e
samza-sql-core/src/main/java/org/apache/samza/sql/api/data/Relation.java PRE-CREATION
samza-sql-core/src/main/java/org/apache/samza/sql/api/data/Stream.java PRE-CREATION
samza-sql-core/src/main/java/org/apache/samza/sql/api/data/Tuple.java PRE-CREATION
samza-sql-core/src/main/java/org/apache/samza/sql/api/operators/Operator.java PRE-CREATION
samza-sql-core/src/main/java/org/apache/samza/sql/api/operators/SqlOperatorFactory.java PRE-CREATION
samza-sql-core/src/main/java/org/apache/samza/sql/api/operators/spec/OperatorSpec.java PRE-CREATION
samza-sql-core/src/main/java/org/apache/samza/sql/data/IncomingMessageTuple.java PRE-CREATION
samza-sql-core/src/main/java/org/apache/samza/sql/exception/OperatorException.java PRE-CREATION
samza-sql-core/src/main/java/org/apache/samza/sql/operators/factory/SimpleOperator.java PRE-CREATION
samza-sql-core/src/main/java/org/apache/samza/sql/operators/factory/SimpleOperatorFactoryImpl.java PRE-CREATION
samza-sql-core/src/main/java/org/apache/samza/sql/operators/factory/SimpleOperatorSpec.java PRE-CREATION
samza-sql-core/src/main/java/org/apache/samza/sql/operators/join/StreamStreamJoin.java PRE-CREATION
samza-sql-core/src/main/java/org/apache/samza/sql/operators/join/StreamStreamJoinSpec.java PRE-CREATION
samza-sql-core/src/main/java/org/apache/samza/sql/operators/partition/PartitionSpec.java PRE-CREATION
samza-sql-core/src/main/java/org/apache/samza/sql/operators/relation/JoinSpec.java PRE-CREATION
samza-sql-core/src/main/java/org/apache/samza/sql/operators/stream/InsertStreamSpec.java PRE-CREATION
samza-sql-core/src/main/java/org/apache/samza/sql/operators/window/BoundedTimeWindow.java PRE-CREATION
samza-sql-core/src/main/java/org/apache/samza/sql/operators/window/FullStateTimeWindow.java PRE-CREATION
samza-sql-core/src/main/java/org/apache/samza/sql/operators/window/FullStateTimeWindowAutoOp.java PRE-CREATION
samza-sql-core/src/main/java/org/apache/samza/sql/operators/window/FullStateTimeWindowOp.java PRE-CREATION
samza-sql-core/src/main/java/org/apache/samza/sql/operators/window/FullStateWindowOp.java PRE-CREATION
samza-sql-core/src/main/java/org/apache/samza/sql/operators/window/RetentionPolicy.java PRE-CREATION
samza-sql-core/src/main/java/org/apache/samza/sql/operators/window/WindowOp.java PRE-CREATION
samza-sql-core/src/main/java/org/apache/samza/sql/operators/window/WindowOpSpec.java PRE-CREATION
samza-sql-core/src/main/java/org/apache/samza/sql/operators/window/WindowSpec.java PRE-CREATION
samza-sql-core/src/main/java/org/apache/samza/sql/operators/window/WindowState.java PRE-CREATION
samza-sql-core/src/main/java/org/apache/samza/sql/window/storage/FilteredMessageIterator.java PRE-CREATION
samza-sql-core/src/main/java/org/apache/samza/sql/window/storage/HashPrefixedMessageStore.java PRE-CREATION
samza-sql-core/src/main/java/org/apache/samza/sql/window/storage/MessageStore.java PRE-CREATION
samza-sql-core/src/main/java/org/apache/samza/sql/window/storage/OffsetKey.java PRE-CREATION
samza-sql-core/src/main/java/org/apache/samza/sql/window/storage/PrefixedKey.java PRE-CREATION
samza-sql-core/src/main/java/org/apache/samza/sql/window/storage/Range.java PRE-CREATION
samza-sql-core/src/main/java/org/apache/samza/sql/window/storage/TimeAndOffsetKey.java PRE-CREATION
samza-sql-core/src/main/java/org/apache/samza/sql/window/storage/TimeKey.java PRE-CREATION
samza-sql-core/src/main/java/org/apache/samza/sql/window/storage/WindowKey.java PRE-CREATION
samza-sql-core/src/main/java/org/apache/samza/sql/window/storage/WindowOutputStream.java PRE-CREATION
samza-sql-core/src/main/java/org/apache/samza/sql/window/storage/WindowState.java PRE-CREATION
samza-sql-core/src/main/java/org/apache/samza/system/sql/LongOffset.java PRE-CREATION
samza-sql-core/src/main/java/org/apache/samza/system/sql/Offset.java PRE-CREATION
samza-sql-core/src/main/java/org/apache/samza/task/sql/OperatorMessageCollector.java PRE-CREATION
samza-sql-core/src/main/java/org/apache/samza/task/sql/StoreMessageCollector.java PRE-CREATION
samza-sql-core/src/test/java/org/apache/samza/task/sql/RandomOperatorTask.java PRE-CREATION
samza-sql-core/src/test/java/org/apache/samza/task/sql/StreamSqlTask.java PRE-CREATION
Diff: https://reviews.apache.org/r/33749/diff/
Testing
-------
./gradlew clean :samza-sql-core:build
Thanks,
Yi Pan (Data Infrastructure)
Re: Review Request 33749: WIP: SAMZA-650 window store implementation
Posted by "Yi Pan (Data Infrastructure)" <yi...@linkedin.com>.
> On May 6, 2015, 2:11 p.m., Milinda Pathirage wrote:
> > samza-sql-core/src/main/java/org/apache/samza/sql/window/storage/FilteredMessageIterator.java, line 54
> > <https://reviews.apache.org/r/33749/diff/2/?file=948526#file948526line54>
> >
> > I'm not sure whether this is 100% correct. For example lets take a situation where only 1 element is left and that element doesn't match given filters. In this situation hasNext will return true. But next will return null. So this doesn't exactly adhere to iterator semantics we know.
Sure. I was debating on whether the slight semantic deviation is OK or not. Good point and I will change in the next update. Thanks!
- Yi
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33749/#review82673
-----------------------------------------------------------
On May 4, 2015, 6:58 a.m., Yi Pan (Data Infrastructure) wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/33749/
> -----------------------------------------------------------
>
> (Updated May 4, 2015, 6:58 a.m.)
>
>
> Review request for samza, Chris Riccomini, Guozhang Wang, Milinda Pathirage, Navina Ramesh, and Naveen Somasundaram.
>
>
> Bugs: SAMZA-650
> https://issues.apache.org/jira/browse/SAMZA-650
>
>
> Repository: samza
>
>
> Description
> -------
>
> WIP: SAMZA-650 window store implementation
>
> First patch to implemement window store and message store.
>
> - Added window store initial implementation
> - Added MessageStore initial implementation as well
> - Completed skeleton APIs in window operators to illustrate the use cases of window store and message store
> - Added a StreamTask test case w/ a single window operator usage
> - Unit tests are still WIP and will be updated later
>
>
> Diffs
> -----
>
> samza-sql-core/src/main/java/org/apache/samza/sql/api/data/Relation.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/api/data/Stream.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/api/data/Table.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/api/data/Tuple.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/api/operators/Operator.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/api/operators/SqlOperatorFactory.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/api/operators/spec/OperatorSpec.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/data/IncomingMessageTuple.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/exception/OperatorException.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/operators/factory/SimpleOperator.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/operators/factory/SimpleOperatorFactoryImpl.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/operators/factory/SimpleOperatorSpec.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/operators/join/StreamStreamJoinSpec.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/operators/join/StreamStreamJoiner.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/operators/partition/PartitionSpec.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/operators/relation/JoinSpec.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/operators/stream/InsertStreamSpec.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/operators/window/BoundedTimeWindow.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/operators/window/FullStateTimeWindow.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/operators/window/FullStateTimeWindowAutoOp.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/operators/window/FullStateTimeWindowOp.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/operators/window/FullStateWindowOp.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/operators/window/WindowOp.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/operators/window/WindowOpSpec.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/operators/window/WindowSpec.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/operators/window/WindowState.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/window/storage/FilteredMessageIterator.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/window/storage/HashPrefixedMessageStore.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/window/storage/MessageStore.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/window/storage/MessageStoreSpec.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/window/storage/OffsetKey.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/window/storage/OrderedStoreKey.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/window/storage/PrefixedKey.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/window/storage/Range.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/window/storage/TimeAndOffsetKey.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/window/storage/TimeKey.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/window/storage/WindowOutputStream.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/window/storage/WindowState.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/window/storage/WindowStore.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/system/sql/LongOffset.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/system/sql/Offset.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/task/sql/OperatorMessageCollector.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/task/sql/StoreMessageCollector.java PRE-CREATION
> samza-sql-core/src/test/java/org/apache/samza/task/sql/RandomOperatorTask.java PRE-CREATION
> samza-sql-core/src/test/java/org/apache/samza/task/sql/RandomWindowOperatorTask.java PRE-CREATION
> samza-sql-core/src/test/java/org/apache/samza/task/sql/StreamSqlTask.java PRE-CREATION
>
> Diff: https://reviews.apache.org/r/33749/diff/
>
>
> Testing
> -------
>
> ./gradlew clean :samza-sql-core:build
>
>
> Thanks,
>
> Yi Pan (Data Infrastructure)
>
>
Re: Review Request 33749: WIP: SAMZA-650 window store implementation
Posted by Milinda Pathirage <mi...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33749/#review82673
-----------------------------------------------------------
samza-sql-core/src/main/java/org/apache/samza/sql/window/storage/FilteredMessageIterator.java
<https://reviews.apache.org/r/33749/#comment133460>
I'm not sure whether this is 100% correct. For example lets take a situation where only 1 element is left and that element doesn't match given filters. In this situation hasNext will return true. But next will return null. So this doesn't exactly adhere to iterator semantics we know.
- Milinda Pathirage
On May 4, 2015, 6:58 a.m., Yi Pan (Data Infrastructure) wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/33749/
> -----------------------------------------------------------
>
> (Updated May 4, 2015, 6:58 a.m.)
>
>
> Review request for samza, Chris Riccomini, Guozhang Wang, Milinda Pathirage, Navina Ramesh, and Naveen Somasundaram.
>
>
> Bugs: SAMZA-650
> https://issues.apache.org/jira/browse/SAMZA-650
>
>
> Repository: samza
>
>
> Description
> -------
>
> WIP: SAMZA-650 window store implementation
>
> First patch to implemement window store and message store.
>
> - Added window store initial implementation
> - Added MessageStore initial implementation as well
> - Completed skeleton APIs in window operators to illustrate the use cases of window store and message store
> - Added a StreamTask test case w/ a single window operator usage
> - Unit tests are still WIP and will be updated later
>
>
> Diffs
> -----
>
> samza-sql-core/src/main/java/org/apache/samza/sql/api/data/Relation.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/api/data/Stream.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/api/data/Table.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/api/data/Tuple.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/api/operators/Operator.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/api/operators/SqlOperatorFactory.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/api/operators/spec/OperatorSpec.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/data/IncomingMessageTuple.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/exception/OperatorException.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/operators/factory/SimpleOperator.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/operators/factory/SimpleOperatorFactoryImpl.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/operators/factory/SimpleOperatorSpec.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/operators/join/StreamStreamJoinSpec.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/operators/join/StreamStreamJoiner.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/operators/partition/PartitionSpec.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/operators/relation/JoinSpec.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/operators/stream/InsertStreamSpec.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/operators/window/BoundedTimeWindow.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/operators/window/FullStateTimeWindow.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/operators/window/FullStateTimeWindowAutoOp.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/operators/window/FullStateTimeWindowOp.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/operators/window/FullStateWindowOp.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/operators/window/WindowOp.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/operators/window/WindowOpSpec.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/operators/window/WindowSpec.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/operators/window/WindowState.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/window/storage/FilteredMessageIterator.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/window/storage/HashPrefixedMessageStore.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/window/storage/MessageStore.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/window/storage/MessageStoreSpec.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/window/storage/OffsetKey.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/window/storage/OrderedStoreKey.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/window/storage/PrefixedKey.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/window/storage/Range.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/window/storage/TimeAndOffsetKey.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/window/storage/TimeKey.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/window/storage/WindowOutputStream.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/window/storage/WindowState.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/window/storage/WindowStore.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/system/sql/LongOffset.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/system/sql/Offset.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/task/sql/OperatorMessageCollector.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/task/sql/StoreMessageCollector.java PRE-CREATION
> samza-sql-core/src/test/java/org/apache/samza/task/sql/RandomOperatorTask.java PRE-CREATION
> samza-sql-core/src/test/java/org/apache/samza/task/sql/RandomWindowOperatorTask.java PRE-CREATION
> samza-sql-core/src/test/java/org/apache/samza/task/sql/StreamSqlTask.java PRE-CREATION
>
> Diff: https://reviews.apache.org/r/33749/diff/
>
>
> Testing
> -------
>
> ./gradlew clean :samza-sql-core:build
>
>
> Thanks,
>
> Yi Pan (Data Infrastructure)
>
>
Re: Review Request 33749: WIP: SAMZA-650 window store implementation
Posted by "Yi Pan (Data Infrastructure)" <yi...@linkedin.com>.
> On May 6, 2015, 2:16 p.m., Milinda Pathirage wrote:
> > samza-sql-core/src/main/java/org/apache/samza/sql/window/storage/MessageStore.java, line 40
> > <https://reviews.apache.org/r/33749/diff/1-2/?file=947212#file947212line40>
> >
> > Invalid parameter in doc comment.
Thanks! Will fix.
- Yi
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33749/#review82674
-----------------------------------------------------------
On May 4, 2015, 6:58 a.m., Yi Pan (Data Infrastructure) wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/33749/
> -----------------------------------------------------------
>
> (Updated May 4, 2015, 6:58 a.m.)
>
>
> Review request for samza, Chris Riccomini, Guozhang Wang, Milinda Pathirage, Navina Ramesh, and Naveen Somasundaram.
>
>
> Bugs: SAMZA-650
> https://issues.apache.org/jira/browse/SAMZA-650
>
>
> Repository: samza
>
>
> Description
> -------
>
> WIP: SAMZA-650 window store implementation
>
> First patch to implemement window store and message store.
>
> - Added window store initial implementation
> - Added MessageStore initial implementation as well
> - Completed skeleton APIs in window operators to illustrate the use cases of window store and message store
> - Added a StreamTask test case w/ a single window operator usage
> - Unit tests are still WIP and will be updated later
>
>
> Diffs
> -----
>
> samza-sql-core/src/main/java/org/apache/samza/sql/api/data/Relation.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/api/data/Stream.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/api/data/Table.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/api/data/Tuple.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/api/operators/Operator.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/api/operators/SqlOperatorFactory.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/api/operators/spec/OperatorSpec.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/data/IncomingMessageTuple.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/exception/OperatorException.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/operators/factory/SimpleOperator.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/operators/factory/SimpleOperatorFactoryImpl.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/operators/factory/SimpleOperatorSpec.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/operators/join/StreamStreamJoinSpec.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/operators/join/StreamStreamJoiner.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/operators/partition/PartitionSpec.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/operators/relation/JoinSpec.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/operators/stream/InsertStreamSpec.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/operators/window/BoundedTimeWindow.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/operators/window/FullStateTimeWindow.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/operators/window/FullStateTimeWindowAutoOp.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/operators/window/FullStateTimeWindowOp.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/operators/window/FullStateWindowOp.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/operators/window/WindowOp.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/operators/window/WindowOpSpec.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/operators/window/WindowSpec.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/operators/window/WindowState.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/window/storage/FilteredMessageIterator.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/window/storage/HashPrefixedMessageStore.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/window/storage/MessageStore.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/window/storage/MessageStoreSpec.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/window/storage/OffsetKey.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/window/storage/OrderedStoreKey.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/window/storage/PrefixedKey.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/window/storage/Range.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/window/storage/TimeAndOffsetKey.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/window/storage/TimeKey.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/window/storage/WindowOutputStream.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/window/storage/WindowState.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/window/storage/WindowStore.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/system/sql/LongOffset.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/system/sql/Offset.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/task/sql/OperatorMessageCollector.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/task/sql/StoreMessageCollector.java PRE-CREATION
> samza-sql-core/src/test/java/org/apache/samza/task/sql/RandomOperatorTask.java PRE-CREATION
> samza-sql-core/src/test/java/org/apache/samza/task/sql/RandomWindowOperatorTask.java PRE-CREATION
> samza-sql-core/src/test/java/org/apache/samza/task/sql/StreamSqlTask.java PRE-CREATION
>
> Diff: https://reviews.apache.org/r/33749/diff/
>
>
> Testing
> -------
>
> ./gradlew clean :samza-sql-core:build
>
>
> Thanks,
>
> Yi Pan (Data Infrastructure)
>
>
Re: Review Request 33749: WIP: SAMZA-650 window store implementation
Posted by Milinda Pathirage <mi...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33749/#review82674
-----------------------------------------------------------
samza-sql-core/src/main/java/org/apache/samza/sql/window/storage/MessageStore.java
<https://reviews.apache.org/r/33749/#comment133461>
Invalid parameter in doc comment.
- Milinda Pathirage
On May 4, 2015, 6:58 a.m., Yi Pan (Data Infrastructure) wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/33749/
> -----------------------------------------------------------
>
> (Updated May 4, 2015, 6:58 a.m.)
>
>
> Review request for samza, Chris Riccomini, Guozhang Wang, Milinda Pathirage, Navina Ramesh, and Naveen Somasundaram.
>
>
> Bugs: SAMZA-650
> https://issues.apache.org/jira/browse/SAMZA-650
>
>
> Repository: samza
>
>
> Description
> -------
>
> WIP: SAMZA-650 window store implementation
>
> First patch to implemement window store and message store.
>
> - Added window store initial implementation
> - Added MessageStore initial implementation as well
> - Completed skeleton APIs in window operators to illustrate the use cases of window store and message store
> - Added a StreamTask test case w/ a single window operator usage
> - Unit tests are still WIP and will be updated later
>
>
> Diffs
> -----
>
> samza-sql-core/src/main/java/org/apache/samza/sql/api/data/Relation.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/api/data/Stream.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/api/data/Table.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/api/data/Tuple.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/api/operators/Operator.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/api/operators/SqlOperatorFactory.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/api/operators/spec/OperatorSpec.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/data/IncomingMessageTuple.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/exception/OperatorException.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/operators/factory/SimpleOperator.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/operators/factory/SimpleOperatorFactoryImpl.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/operators/factory/SimpleOperatorSpec.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/operators/join/StreamStreamJoinSpec.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/operators/join/StreamStreamJoiner.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/operators/partition/PartitionSpec.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/operators/relation/JoinSpec.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/operators/stream/InsertStreamSpec.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/operators/window/BoundedTimeWindow.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/operators/window/FullStateTimeWindow.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/operators/window/FullStateTimeWindowAutoOp.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/operators/window/FullStateTimeWindowOp.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/operators/window/FullStateWindowOp.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/operators/window/WindowOp.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/operators/window/WindowOpSpec.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/operators/window/WindowSpec.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/operators/window/WindowState.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/window/storage/FilteredMessageIterator.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/window/storage/HashPrefixedMessageStore.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/window/storage/MessageStore.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/window/storage/MessageStoreSpec.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/window/storage/OffsetKey.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/window/storage/OrderedStoreKey.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/window/storage/PrefixedKey.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/window/storage/Range.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/window/storage/TimeAndOffsetKey.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/window/storage/TimeKey.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/window/storage/WindowOutputStream.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/window/storage/WindowState.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/window/storage/WindowStore.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/system/sql/LongOffset.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/system/sql/Offset.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/task/sql/OperatorMessageCollector.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/task/sql/StoreMessageCollector.java PRE-CREATION
> samza-sql-core/src/test/java/org/apache/samza/task/sql/RandomOperatorTask.java PRE-CREATION
> samza-sql-core/src/test/java/org/apache/samza/task/sql/RandomWindowOperatorTask.java PRE-CREATION
> samza-sql-core/src/test/java/org/apache/samza/task/sql/StreamSqlTask.java PRE-CREATION
>
> Diff: https://reviews.apache.org/r/33749/diff/
>
>
> Testing
> -------
>
> ./gradlew clean :samza-sql-core:build
>
>
> Thanks,
>
> Yi Pan (Data Infrastructure)
>
>
Re: Review Request 33749: WIP: SAMZA-650 window store implementation
Posted by "Yi Pan (Data Infrastructure)" <yi...@linkedin.com>.
> On May 7, 2015, 2:22 p.m., Milinda Pathirage wrote:
> > samza-sql-core/src/main/java/org/apache/samza/sql/operators/join/StreamStreamJoiner.java, line 47
> > <https://reviews.apache.org/r/33749/diff/2/?file=948513#file948513line47>
> >
> > I think that stream-to-stream joining is not practical for possibly infinite streams. We may need to define some constraints or some other restrictions to make this practical.
Hi, Milinda, yes. Infinite stream-to-stream join should not be allowed. My thought is that the parser/planner should invalidate the unbounded join via inspecting the join conditions. Consider the each input stream is "ordered on field X in A" and "ordered on field Y in B" if join condition does not put a bounded range according to field X in A and field Y in B, the validation should fail.
- Yi
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33749/#review82821
-----------------------------------------------------------
On May 4, 2015, 6:58 a.m., Yi Pan (Data Infrastructure) wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/33749/
> -----------------------------------------------------------
>
> (Updated May 4, 2015, 6:58 a.m.)
>
>
> Review request for samza, Chris Riccomini, Guozhang Wang, Milinda Pathirage, Navina Ramesh, and Naveen Somasundaram.
>
>
> Bugs: SAMZA-650
> https://issues.apache.org/jira/browse/SAMZA-650
>
>
> Repository: samza
>
>
> Description
> -------
>
> WIP: SAMZA-650 window store implementation
>
> First patch to implemement window store and message store.
>
> - Added window store initial implementation
> - Added MessageStore initial implementation as well
> - Completed skeleton APIs in window operators to illustrate the use cases of window store and message store
> - Added a StreamTask test case w/ a single window operator usage
> - Unit tests are still WIP and will be updated later
>
>
> Diffs
> -----
>
> samza-sql-core/src/main/java/org/apache/samza/sql/api/data/Relation.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/api/data/Stream.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/api/data/Table.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/api/data/Tuple.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/api/operators/Operator.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/api/operators/SqlOperatorFactory.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/api/operators/spec/OperatorSpec.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/data/IncomingMessageTuple.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/exception/OperatorException.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/operators/factory/SimpleOperator.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/operators/factory/SimpleOperatorFactoryImpl.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/operators/factory/SimpleOperatorSpec.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/operators/join/StreamStreamJoinSpec.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/operators/join/StreamStreamJoiner.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/operators/partition/PartitionSpec.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/operators/relation/JoinSpec.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/operators/stream/InsertStreamSpec.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/operators/window/BoundedTimeWindow.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/operators/window/FullStateTimeWindow.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/operators/window/FullStateTimeWindowAutoOp.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/operators/window/FullStateTimeWindowOp.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/operators/window/FullStateWindowOp.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/operators/window/WindowOp.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/operators/window/WindowOpSpec.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/operators/window/WindowSpec.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/operators/window/WindowState.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/window/storage/FilteredMessageIterator.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/window/storage/HashPrefixedMessageStore.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/window/storage/MessageStore.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/window/storage/MessageStoreSpec.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/window/storage/OffsetKey.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/window/storage/OrderedStoreKey.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/window/storage/PrefixedKey.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/window/storage/Range.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/window/storage/TimeAndOffsetKey.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/window/storage/TimeKey.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/window/storage/WindowOutputStream.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/window/storage/WindowState.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/window/storage/WindowStore.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/system/sql/LongOffset.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/system/sql/Offset.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/task/sql/OperatorMessageCollector.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/task/sql/StoreMessageCollector.java PRE-CREATION
> samza-sql-core/src/test/java/org/apache/samza/task/sql/RandomOperatorTask.java PRE-CREATION
> samza-sql-core/src/test/java/org/apache/samza/task/sql/RandomWindowOperatorTask.java PRE-CREATION
> samza-sql-core/src/test/java/org/apache/samza/task/sql/StreamSqlTask.java PRE-CREATION
>
> Diff: https://reviews.apache.org/r/33749/diff/
>
>
> Testing
> -------
>
> ./gradlew clean :samza-sql-core:build
>
>
> Thanks,
>
> Yi Pan (Data Infrastructure)
>
>
Re: Review Request 33749: WIP: SAMZA-650 window store implementation
Posted by Milinda Pathirage <mi...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33749/#review82821
-----------------------------------------------------------
samza-sql-core/src/main/java/org/apache/samza/sql/operators/join/StreamStreamJoiner.java
<https://reviews.apache.org/r/33749/#comment133651>
I think that stream-to-stream joining is not practical for possibly infinite streams. We may need to define some constraints or some other restrictions to make this practical.
- Milinda Pathirage
On May 4, 2015, 6:58 a.m., Yi Pan (Data Infrastructure) wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/33749/
> -----------------------------------------------------------
>
> (Updated May 4, 2015, 6:58 a.m.)
>
>
> Review request for samza, Chris Riccomini, Guozhang Wang, Milinda Pathirage, Navina Ramesh, and Naveen Somasundaram.
>
>
> Bugs: SAMZA-650
> https://issues.apache.org/jira/browse/SAMZA-650
>
>
> Repository: samza
>
>
> Description
> -------
>
> WIP: SAMZA-650 window store implementation
>
> First patch to implemement window store and message store.
>
> - Added window store initial implementation
> - Added MessageStore initial implementation as well
> - Completed skeleton APIs in window operators to illustrate the use cases of window store and message store
> - Added a StreamTask test case w/ a single window operator usage
> - Unit tests are still WIP and will be updated later
>
>
> Diffs
> -----
>
> samza-sql-core/src/main/java/org/apache/samza/sql/api/data/Relation.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/api/data/Stream.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/api/data/Table.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/api/data/Tuple.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/api/operators/Operator.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/api/operators/SqlOperatorFactory.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/api/operators/spec/OperatorSpec.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/data/IncomingMessageTuple.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/exception/OperatorException.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/operators/factory/SimpleOperator.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/operators/factory/SimpleOperatorFactoryImpl.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/operators/factory/SimpleOperatorSpec.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/operators/join/StreamStreamJoinSpec.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/operators/join/StreamStreamJoiner.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/operators/partition/PartitionSpec.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/operators/relation/JoinSpec.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/operators/stream/InsertStreamSpec.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/operators/window/BoundedTimeWindow.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/operators/window/FullStateTimeWindow.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/operators/window/FullStateTimeWindowAutoOp.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/operators/window/FullStateTimeWindowOp.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/operators/window/FullStateWindowOp.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/operators/window/WindowOp.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/operators/window/WindowOpSpec.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/operators/window/WindowSpec.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/operators/window/WindowState.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/window/storage/FilteredMessageIterator.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/window/storage/HashPrefixedMessageStore.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/window/storage/MessageStore.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/window/storage/MessageStoreSpec.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/window/storage/OffsetKey.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/window/storage/OrderedStoreKey.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/window/storage/PrefixedKey.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/window/storage/Range.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/window/storage/TimeAndOffsetKey.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/window/storage/TimeKey.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/window/storage/WindowOutputStream.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/window/storage/WindowState.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/window/storage/WindowStore.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/system/sql/LongOffset.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/system/sql/Offset.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/task/sql/OperatorMessageCollector.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/task/sql/StoreMessageCollector.java PRE-CREATION
> samza-sql-core/src/test/java/org/apache/samza/task/sql/RandomOperatorTask.java PRE-CREATION
> samza-sql-core/src/test/java/org/apache/samza/task/sql/RandomWindowOperatorTask.java PRE-CREATION
> samza-sql-core/src/test/java/org/apache/samza/task/sql/StreamSqlTask.java PRE-CREATION
>
> Diff: https://reviews.apache.org/r/33749/diff/
>
>
> Testing
> -------
>
> ./gradlew clean :samza-sql-core:build
>
>
> Thanks,
>
> Yi Pan (Data Infrastructure)
>
>
Re: Review Request 33749: WIP: SAMZA-650 window store implementation
Posted by "Yi Pan (Data Infrastructure)" <yi...@linkedin.com>.
> On May 7, 2015, 2:35 p.m., Milinda Pathirage wrote:
> > samza-sql-core/src/main/java/org/apache/samza/sql/operators/window/FullStateTimeWindowAutoOp.java, line 30
> > <https://reviews.apache.org/r/33749/diff/2/?file=948519#file948519line30>
> >
> > Hi Yi, What is automated operator in this context?
It is evolving now and will be removed. The original thought is trying to make two overlayed APIs on top of core set of window APIs: a) used by OperatorRouter to automatically execute the connected operators (i.e. the DAG for the query); b) a more low-level API that allows programmer to control when to add message and when to retrieve/flush results. After an internal team discussion, we are going to remove b). Instead, we will add support for user callback functions in the Operator to be invoked right before the process() is called and right before the result is collected and sent.
> On May 7, 2015, 2:35 p.m., Milinda Pathirage wrote:
> > samza-sql-core/src/main/java/org/apache/samza/sql/operators/window/FullStateTimeWindowOp.java, line 76
> > <https://reviews.apache.org/r/33749/diff/2/?file=948520#file948520line76>
> >
> > When will this input disabling happens for a window operator?
Refer to the design doc, the input to a window operator may be disabled if the downstream operator can not accept the output of the window operator (e.g. if the joiner needs to wait for the output from another stream to be available before accepting this window's output). It should not happen in normal case, but may occur if we optimize the message store recovery with lazy recovery or the two streams are skewed w/ large offsets that is larger than the retention size.
- Yi
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33749/#review82824
-----------------------------------------------------------
On May 4, 2015, 6:58 a.m., Yi Pan (Data Infrastructure) wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/33749/
> -----------------------------------------------------------
>
> (Updated May 4, 2015, 6:58 a.m.)
>
>
> Review request for samza, Chris Riccomini, Guozhang Wang, Milinda Pathirage, Navina Ramesh, and Naveen Somasundaram.
>
>
> Bugs: SAMZA-650
> https://issues.apache.org/jira/browse/SAMZA-650
>
>
> Repository: samza
>
>
> Description
> -------
>
> WIP: SAMZA-650 window store implementation
>
> First patch to implemement window store and message store.
>
> - Added window store initial implementation
> - Added MessageStore initial implementation as well
> - Completed skeleton APIs in window operators to illustrate the use cases of window store and message store
> - Added a StreamTask test case w/ a single window operator usage
> - Unit tests are still WIP and will be updated later
>
>
> Diffs
> -----
>
> samza-sql-core/src/main/java/org/apache/samza/sql/api/data/Relation.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/api/data/Stream.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/api/data/Table.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/api/data/Tuple.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/api/operators/Operator.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/api/operators/SqlOperatorFactory.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/api/operators/spec/OperatorSpec.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/data/IncomingMessageTuple.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/exception/OperatorException.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/operators/factory/SimpleOperator.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/operators/factory/SimpleOperatorFactoryImpl.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/operators/factory/SimpleOperatorSpec.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/operators/join/StreamStreamJoinSpec.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/operators/join/StreamStreamJoiner.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/operators/partition/PartitionSpec.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/operators/relation/JoinSpec.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/operators/stream/InsertStreamSpec.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/operators/window/BoundedTimeWindow.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/operators/window/FullStateTimeWindow.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/operators/window/FullStateTimeWindowAutoOp.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/operators/window/FullStateTimeWindowOp.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/operators/window/FullStateWindowOp.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/operators/window/WindowOp.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/operators/window/WindowOpSpec.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/operators/window/WindowSpec.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/operators/window/WindowState.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/window/storage/FilteredMessageIterator.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/window/storage/HashPrefixedMessageStore.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/window/storage/MessageStore.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/window/storage/MessageStoreSpec.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/window/storage/OffsetKey.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/window/storage/OrderedStoreKey.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/window/storage/PrefixedKey.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/window/storage/Range.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/window/storage/TimeAndOffsetKey.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/window/storage/TimeKey.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/window/storage/WindowOutputStream.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/window/storage/WindowState.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/window/storage/WindowStore.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/system/sql/LongOffset.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/system/sql/Offset.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/task/sql/OperatorMessageCollector.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/task/sql/StoreMessageCollector.java PRE-CREATION
> samza-sql-core/src/test/java/org/apache/samza/task/sql/RandomOperatorTask.java PRE-CREATION
> samza-sql-core/src/test/java/org/apache/samza/task/sql/RandomWindowOperatorTask.java PRE-CREATION
> samza-sql-core/src/test/java/org/apache/samza/task/sql/StreamSqlTask.java PRE-CREATION
>
> Diff: https://reviews.apache.org/r/33749/diff/
>
>
> Testing
> -------
>
> ./gradlew clean :samza-sql-core:build
>
>
> Thanks,
>
> Yi Pan (Data Infrastructure)
>
>
Re: Review Request 33749: WIP: SAMZA-650 window store implementation
Posted by Milinda Pathirage <mi...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33749/#review82824
-----------------------------------------------------------
samza-sql-core/src/main/java/org/apache/samza/sql/operators/window/FullStateTimeWindowAutoOp.java
<https://reviews.apache.org/r/33749/#comment133654>
Hi Yi, What is automated operator in this context?
samza-sql-core/src/main/java/org/apache/samza/sql/operators/window/FullStateTimeWindowOp.java
<https://reviews.apache.org/r/33749/#comment133655>
When will this input disabling happens for a window operator?
- Milinda Pathirage
On May 4, 2015, 6:58 a.m., Yi Pan (Data Infrastructure) wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/33749/
> -----------------------------------------------------------
>
> (Updated May 4, 2015, 6:58 a.m.)
>
>
> Review request for samza, Chris Riccomini, Guozhang Wang, Milinda Pathirage, Navina Ramesh, and Naveen Somasundaram.
>
>
> Bugs: SAMZA-650
> https://issues.apache.org/jira/browse/SAMZA-650
>
>
> Repository: samza
>
>
> Description
> -------
>
> WIP: SAMZA-650 window store implementation
>
> First patch to implemement window store and message store.
>
> - Added window store initial implementation
> - Added MessageStore initial implementation as well
> - Completed skeleton APIs in window operators to illustrate the use cases of window store and message store
> - Added a StreamTask test case w/ a single window operator usage
> - Unit tests are still WIP and will be updated later
>
>
> Diffs
> -----
>
> samza-sql-core/src/main/java/org/apache/samza/sql/api/data/Relation.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/api/data/Stream.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/api/data/Table.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/api/data/Tuple.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/api/operators/Operator.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/api/operators/SqlOperatorFactory.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/api/operators/spec/OperatorSpec.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/data/IncomingMessageTuple.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/exception/OperatorException.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/operators/factory/SimpleOperator.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/operators/factory/SimpleOperatorFactoryImpl.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/operators/factory/SimpleOperatorSpec.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/operators/join/StreamStreamJoinSpec.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/operators/join/StreamStreamJoiner.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/operators/partition/PartitionSpec.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/operators/relation/JoinSpec.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/operators/stream/InsertStreamSpec.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/operators/window/BoundedTimeWindow.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/operators/window/FullStateTimeWindow.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/operators/window/FullStateTimeWindowAutoOp.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/operators/window/FullStateTimeWindowOp.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/operators/window/FullStateWindowOp.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/operators/window/WindowOp.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/operators/window/WindowOpSpec.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/operators/window/WindowSpec.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/operators/window/WindowState.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/window/storage/FilteredMessageIterator.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/window/storage/HashPrefixedMessageStore.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/window/storage/MessageStore.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/window/storage/MessageStoreSpec.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/window/storage/OffsetKey.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/window/storage/OrderedStoreKey.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/window/storage/PrefixedKey.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/window/storage/Range.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/window/storage/TimeAndOffsetKey.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/window/storage/TimeKey.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/window/storage/WindowOutputStream.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/window/storage/WindowState.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/sql/window/storage/WindowStore.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/system/sql/LongOffset.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/system/sql/Offset.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/task/sql/OperatorMessageCollector.java PRE-CREATION
> samza-sql-core/src/main/java/org/apache/samza/task/sql/StoreMessageCollector.java PRE-CREATION
> samza-sql-core/src/test/java/org/apache/samza/task/sql/RandomOperatorTask.java PRE-CREATION
> samza-sql-core/src/test/java/org/apache/samza/task/sql/RandomWindowOperatorTask.java PRE-CREATION
> samza-sql-core/src/test/java/org/apache/samza/task/sql/StreamSqlTask.java PRE-CREATION
>
> Diff: https://reviews.apache.org/r/33749/diff/
>
>
> Testing
> -------
>
> ./gradlew clean :samza-sql-core:build
>
>
> Thanks,
>
> Yi Pan (Data Infrastructure)
>
>
Re: Review Request 33749: WIP: SAMZA-650 window store implementation
Posted by "Yi Pan (Data Infrastructure)" <yi...@linkedin.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33749/
-----------------------------------------------------------
(Updated May 4, 2015, 6:58 a.m.)
Review request for samza, Chris Riccomini, Guozhang Wang, Milinda Pathirage, Navina Ramesh, and Naveen Somasundaram.
Bugs: SAMZA-650
https://issues.apache.org/jira/browse/SAMZA-650
Repository: samza
Description (updated)
-------
WIP: SAMZA-650 window store implementation
First patch to implemement window store and message store.
- Added window store initial implementation
- Added MessageStore initial implementation as well
- Completed skeleton APIs in window operators to illustrate the use cases of window store and message store
- Added a StreamTask test case w/ a single window operator usage
- Unit tests are still WIP and will be updated later
Diffs (updated)
-----
samza-sql-core/src/main/java/org/apache/samza/sql/api/data/Relation.java PRE-CREATION
samza-sql-core/src/main/java/org/apache/samza/sql/api/data/Stream.java PRE-CREATION
samza-sql-core/src/main/java/org/apache/samza/sql/api/data/Table.java PRE-CREATION
samza-sql-core/src/main/java/org/apache/samza/sql/api/data/Tuple.java PRE-CREATION
samza-sql-core/src/main/java/org/apache/samza/sql/api/operators/Operator.java PRE-CREATION
samza-sql-core/src/main/java/org/apache/samza/sql/api/operators/SqlOperatorFactory.java PRE-CREATION
samza-sql-core/src/main/java/org/apache/samza/sql/api/operators/spec/OperatorSpec.java PRE-CREATION
samza-sql-core/src/main/java/org/apache/samza/sql/data/IncomingMessageTuple.java PRE-CREATION
samza-sql-core/src/main/java/org/apache/samza/sql/exception/OperatorException.java PRE-CREATION
samza-sql-core/src/main/java/org/apache/samza/sql/operators/factory/SimpleOperator.java PRE-CREATION
samza-sql-core/src/main/java/org/apache/samza/sql/operators/factory/SimpleOperatorFactoryImpl.java PRE-CREATION
samza-sql-core/src/main/java/org/apache/samza/sql/operators/factory/SimpleOperatorSpec.java PRE-CREATION
samza-sql-core/src/main/java/org/apache/samza/sql/operators/join/StreamStreamJoinSpec.java PRE-CREATION
samza-sql-core/src/main/java/org/apache/samza/sql/operators/join/StreamStreamJoiner.java PRE-CREATION
samza-sql-core/src/main/java/org/apache/samza/sql/operators/partition/PartitionSpec.java PRE-CREATION
samza-sql-core/src/main/java/org/apache/samza/sql/operators/relation/JoinSpec.java PRE-CREATION
samza-sql-core/src/main/java/org/apache/samza/sql/operators/stream/InsertStreamSpec.java PRE-CREATION
samza-sql-core/src/main/java/org/apache/samza/sql/operators/window/BoundedTimeWindow.java PRE-CREATION
samza-sql-core/src/main/java/org/apache/samza/sql/operators/window/FullStateTimeWindow.java PRE-CREATION
samza-sql-core/src/main/java/org/apache/samza/sql/operators/window/FullStateTimeWindowAutoOp.java PRE-CREATION
samza-sql-core/src/main/java/org/apache/samza/sql/operators/window/FullStateTimeWindowOp.java PRE-CREATION
samza-sql-core/src/main/java/org/apache/samza/sql/operators/window/FullStateWindowOp.java PRE-CREATION
samza-sql-core/src/main/java/org/apache/samza/sql/operators/window/WindowOp.java PRE-CREATION
samza-sql-core/src/main/java/org/apache/samza/sql/operators/window/WindowOpSpec.java PRE-CREATION
samza-sql-core/src/main/java/org/apache/samza/sql/operators/window/WindowSpec.java PRE-CREATION
samza-sql-core/src/main/java/org/apache/samza/sql/operators/window/WindowState.java PRE-CREATION
samza-sql-core/src/main/java/org/apache/samza/sql/window/storage/FilteredMessageIterator.java PRE-CREATION
samza-sql-core/src/main/java/org/apache/samza/sql/window/storage/HashPrefixedMessageStore.java PRE-CREATION
samza-sql-core/src/main/java/org/apache/samza/sql/window/storage/MessageStore.java PRE-CREATION
samza-sql-core/src/main/java/org/apache/samza/sql/window/storage/MessageStoreSpec.java PRE-CREATION
samza-sql-core/src/main/java/org/apache/samza/sql/window/storage/OffsetKey.java PRE-CREATION
samza-sql-core/src/main/java/org/apache/samza/sql/window/storage/OrderedStoreKey.java PRE-CREATION
samza-sql-core/src/main/java/org/apache/samza/sql/window/storage/PrefixedKey.java PRE-CREATION
samza-sql-core/src/main/java/org/apache/samza/sql/window/storage/Range.java PRE-CREATION
samza-sql-core/src/main/java/org/apache/samza/sql/window/storage/TimeAndOffsetKey.java PRE-CREATION
samza-sql-core/src/main/java/org/apache/samza/sql/window/storage/TimeKey.java PRE-CREATION
samza-sql-core/src/main/java/org/apache/samza/sql/window/storage/WindowOutputStream.java PRE-CREATION
samza-sql-core/src/main/java/org/apache/samza/sql/window/storage/WindowState.java PRE-CREATION
samza-sql-core/src/main/java/org/apache/samza/sql/window/storage/WindowStore.java PRE-CREATION
samza-sql-core/src/main/java/org/apache/samza/system/sql/LongOffset.java PRE-CREATION
samza-sql-core/src/main/java/org/apache/samza/system/sql/Offset.java PRE-CREATION
samza-sql-core/src/main/java/org/apache/samza/task/sql/OperatorMessageCollector.java PRE-CREATION
samza-sql-core/src/main/java/org/apache/samza/task/sql/StoreMessageCollector.java PRE-CREATION
samza-sql-core/src/test/java/org/apache/samza/task/sql/RandomOperatorTask.java PRE-CREATION
samza-sql-core/src/test/java/org/apache/samza/task/sql/RandomWindowOperatorTask.java PRE-CREATION
samza-sql-core/src/test/java/org/apache/samza/task/sql/StreamSqlTask.java PRE-CREATION
Diff: https://reviews.apache.org/r/33749/diff/
Testing
-------
./gradlew clean :samza-sql-core:build
Thanks,
Yi Pan (Data Infrastructure)