You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@samza.apache.org by "Yi Pan (Data Infrastructure)" <yi...@linkedin.com> on 2015/05/28 01:24:30 UTC
Re: Review Request 34664: SAMZA-650 window store implementation
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/34664/
-----------------------------------------------------------
(Updated May 27, 2015, 11:24 p.m.)
Review request for samza, Yan Fang, Chinmay Soman, Chris Riccomini, Guozhang Wang, Milinda Pathirage, Navina Ramesh, and Naveen Somasundaram.
Summary (updated)
-----------------
SAMZA-650 window store implementation
Bugs: SAMZA-650
https://issues.apache.org/jira/browse/SAMZA-650
Repository: samza
Description
-------
WIP: update operator API to allow callbacks and allow a single API to trigger OperatorRouter execution w/ user callbacks
Conflicts:
samza-sql-core/src/main/java/org/apache/samza/sql/api/data/Tuple.java
samza-sql-core/src/main/java/org/apache/samza/sql/api/operators/Operator.java
samza-sql-core/src/main/java/org/apache/samza/sql/data/IncomingMessageTuple.java
samza-sql-core/src/main/java/org/apache/samza/sql/operators/factory/SimpleOperator.java
samza-sql-core/src/main/java/org/apache/samza/sql/operators/factory/SimpleOperatorFactoryImpl.java
samza-sql-core/src/main/java/org/apache/samza/sql/operators/join/StreamStreamJoin.java
samza-sql-core/src/main/java/org/apache/samza/sql/operators/join/StreamStreamJoinSpec.java
samza-sql-core/src/main/java/org/apache/samza/sql/operators/relation/JoinSpec.java
samza-sql-core/src/main/java/org/apache/samza/sql/operators/stream/InsertStreamSpec.java
samza-sql-core/src/main/java/org/apache/samza/sql/operators/window/FullStateTimeWindowOp.java
samza-sql-core/src/main/java/org/apache/samza/sql/operators/window/FullStateWindowOp.java
samza-sql-core/src/main/java/org/apache/samza/sql/operators/window/WindowOp.java
samza-sql-core/src/main/java/org/apache/samza/sql/operators/window/WindowOpSpec.java
samza-sql-core/src/main/java/org/apache/samza/sql/operators/window/WindowState.java
samza-sql-core/src/main/java/org/apache/samza/sql/window/storage/HashPrefixedMessageStore.java
samza-sql-core/src/main/java/org/apache/samza/task/sql/OperatorMessageCollector.java
samza-sql-core/src/test/java/org/apache/samza/task/sql/RandomOperatorTask.java
samza-sql-core/src/test/java/org/apache/samza/task/sql/RandomWindowOperatorTask.java
samza-sql-core/src/test/java/org/apache/samza/task/sql/StreamSqlTask.java
WIP: updated operator API and use case in test tasks
Conflicts:
samza-sql-core/src/main/java/org/apache/samza/sql/api/operators/Operator.java
samza-sql-core/src/main/java/org/apache/samza/sql/api/operators/SimpleOperator.java
samza-sql-core/src/main/java/org/apache/samza/sql/operators/factory/SimpleOperatorFactoryImpl.java
samza-sql-core/src/main/java/org/apache/samza/sql/operators/window/FullStateTimeWindowOp.java
samza-sql-core/src/main/java/org/apache/samza/sql/operators/window/FullStateWindowOp.java
samza-sql-core/src/main/java/org/apache/samza/sql/operators/window/WindowOp.java
samza-sql-core/src/main/java/org/apache/samza/sql/window/storage/MessageStore.java
samza-sql-core/src/main/java/org/apache/samza/sql/window/storage/MessageStoreSpec.java
samza-sql-core/src/main/java/org/apache/samza/sql/window/storage/WindowState.java
samza-sql-core/src/main/java/org/apache/samza/sql/window/storage/WindowStore.java
samza-sql-core/src/main/java/org/apache/samza/task/sql/StoreMessageCollector.java
samza-sql-core/src/test/java/org/apache/samza/task/sql/StreamSqlTask.java
SAMZA-552: Updated operator API w/o callbacks
SAMZA-552: updated Operator API w/o callbacks
SAMZA-552: use OperatorCallback to allow implementation of callbacks w/o inheriting and creating many sub-classes from operators
SAMZA-552 update the operator API
SAMZA-552: operator builder API update
Squashed commit of the following:
commit fad81106901e494d3950eeaafaeefef482ac0125
Author: Yi Pan (Data Infrastructure) <yi...@linkedin.com>
Date: Mon May 25 23:40:00 2015 -0700
SAMZA-650 window message store and window store implementation
commit 58c2eeebf4bb0975f70aeba733379e1104f3a7de
Author: Yi Pan (Data Infrastructure) <yi...@linkedin.com>
Date: Fri May 22 00:25:13 2015 -0700
WIP: window store implementation
commit 917e1b599622c2d46ad9a6c63e52dcded893eb8e
Merge: 174eeb0 1183e9f
Author: Yi Pan (Data Infrastructure) <yi...@linkedin.com>
Date: Fri May 22 00:18:35 2015 -0700
Merge branch 'samza-window-op' into samza-650-v2
Conflicts:
samza-sql-core/src/main/java/org/apache/samza/sql/api/data/EntityName.java
samza-sql-core/src/main/java/org/apache/samza/sql/api/data/Relation.java
samza-sql-core/src/main/java/org/apache/samza/sql/api/data/Stream.java
samza-sql-core/src/main/java/org/apache/samza/sql/api/data/Table.java
samza-sql-core/src/main/java/org/apache/samza/sql/api/data/Tuple.java
samza-sql-core/src/main/java/org/apache/samza/sql/api/operators/Operator.java
samza-sql-core/src/main/java/org/apache/samza/sql/api/operators/OperatorCallback.java
samza-sql-core/src/main/java/org/apache/samza/sql/api/operators/OperatorRouter.java
samza-sql-core/src/main/java/org/apache/samza/sql/api/operators/OperatorSink.java
samza-sql-core/src/main/java/org/apache/samza/sql/api/operators/SimpleOperator.java
samza-sql-core/src/main/java/org/apache/samza/sql/api/operators/SqlOperatorFactory.java
samza-sql-core/src/main/java/org/apache/samza/sql/data/IncomingMessageTuple.java
samza-sql-core/src/main/java/org/apache/samza/sql/operators/factory/SimpleOperatorFactoryImpl.java
samza-sql-core/src/main/java/org/apache/samza/sql/operators/join/StreamStreamJoin.java
samza-sql-core/src/main/java/org/apache/samza/sql/operators/join/StreamStreamJoinSpec.java
samza-sql-core/src/main/java/org/apache/samza/sql/operators/partition/PartitionOp.java
samza-sql-core/src/main/java/org/apache/samza/sql/operators/partition/PartitionSpec.java
samza-sql-core/src/main/java/org/apache/samza/sql/operators/stream/InsertStreamSpec.java
samza-sql-core/src/main/java/org/apache/samza/sql/operators/window/BoundedTimeWindow.java
samza-sql-core/src/main/java/org/apache/samza/sql/operators/window/WindowSpec.java
samza-sql-core/src/main/java/org/apache/samza/sql/window/storage/OrderedStoreKey.java
samza-sql-core/src/main/java/org/apache/samza/task/sql/RouterMessageCollector.java
samza-sql-core/src/main/java/org/apache/samza/task/sql/SimpleMessageCollector.java
samza-sql-core/src/test/java/org/apache/samza/task/sql/RandomWindowOperatorTask.java
samza-sql-core/src/test/java/org/apache/samza/task/sql/StreamSqlTask.java
samza-sql-core/src/test/java/org/apache/samza/task/sql/UserCallbacksSqlTask.java
commit 1183e9fdce56eb2ffa7fbdf9bbe9df92b8229252
Author: Yi Pan (Data Infrastructure) <yi...@linkedin.com>
Date: Thu May 7 17:44:28 2015 -0700
WIP: updated operator API and use case in test tasks
commit 8ada0ccb5be708fe2f98d701387b910f88feb3ff
Author: Yi Pan (Data Infrastructure) <yi...@linkedin.com>
Date: Wed May 6 22:35:08 2015 -0700
WIP: update operator API to allow callbacks and allow a single API to trigger OperatorRouter execution w/ user callbacks
commit ee00b610be39b380acb9cd09e41949df2d36b7c3
Author: Yi Pan (Data Infrastructure) <yi...@linkedin.com>
Date: Fri May 1 14:50:40 2015 -0700
SAMZA-650: adding updateOutputs(), getResult(), and flush() in window operator. Adding example code as use case for window operator.
commit 3c73d81e45c26e1f1a7d6cc43f5a61dd33d3f76d
Author: Yi Pan (Data Infrastructure) <yi...@linkedin.com>
Date: Sun Apr 26 12:54:36 2015 -0700
WIP: SAMZA-650 window store implementation
Diffs
-----
samza-sql-core/src/main/java/org/apache/samza/sql/api/data/EntityName.java PRE-CREATION
samza-sql-core/src/main/java/org/apache/samza/sql/api/data/Table.java PRE-CREATION
samza-sql-core/src/main/java/org/apache/samza/sql/api/operators/Operator.java PRE-CREATION
samza-sql-core/src/main/java/org/apache/samza/sql/api/operators/OperatorCallback.java PRE-CREATION
samza-sql-core/src/main/java/org/apache/samza/sql/api/operators/OperatorRouter.java PRE-CREATION
samza-sql-core/src/main/java/org/apache/samza/sql/api/operators/OperatorSink.java PRE-CREATION
samza-sql-core/src/main/java/org/apache/samza/sql/api/operators/SimpleOperator.java PRE-CREATION
samza-sql-core/src/main/java/org/apache/samza/sql/data/IncomingMessageTuple.java PRE-CREATION
samza-sql-core/src/main/java/org/apache/samza/sql/exception/OperatorException.java PRE-CREATION
samza-sql-core/src/main/java/org/apache/samza/sql/operators/OperatorTopology.java PRE-CREATION
samza-sql-core/src/main/java/org/apache/samza/sql/operators/SimpleOperatorImpl.java PRE-CREATION
samza-sql-core/src/main/java/org/apache/samza/sql/operators/SimpleRouter.java PRE-CREATION
samza-sql-core/src/main/java/org/apache/samza/sql/operators/factory/NoopOperatorCallback.java PRE-CREATION
samza-sql-core/src/main/java/org/apache/samza/sql/operators/factory/SimpleOperatorFactoryImpl.java PRE-CREATION
samza-sql-core/src/main/java/org/apache/samza/sql/operators/factory/SimpleOperatorImpl.java PRE-CREATION
samza-sql-core/src/main/java/org/apache/samza/sql/operators/factory/SimpleOperatorSpec.java PRE-CREATION
samza-sql-core/src/main/java/org/apache/samza/sql/operators/factory/SimpleRouter.java PRE-CREATION
samza-sql-core/src/main/java/org/apache/samza/sql/operators/factory/TopologyBuilder.java PRE-CREATION
samza-sql-core/src/main/java/org/apache/samza/sql/operators/join/StreamStreamJoin.java PRE-CREATION
samza-sql-core/src/main/java/org/apache/samza/sql/operators/join/StreamStreamJoinSpec.java PRE-CREATION
samza-sql-core/src/main/java/org/apache/samza/sql/operators/partition/PartitionOp.java PRE-CREATION
samza-sql-core/src/main/java/org/apache/samza/sql/operators/partition/PartitionSpec.java PRE-CREATION
samza-sql-core/src/main/java/org/apache/samza/sql/operators/window/BoundedTimeWindow.java PRE-CREATION
samza-sql-core/src/main/java/org/apache/samza/sql/operators/window/FullStateTimeWindowOp.java PRE-CREATION
samza-sql-core/src/main/java/org/apache/samza/sql/operators/window/FullStateWindow.java PRE-CREATION
samza-sql-core/src/main/java/org/apache/samza/sql/operators/window/FullStateWindowOp.java PRE-CREATION
samza-sql-core/src/main/java/org/apache/samza/sql/operators/window/RetentionPolicy.java PRE-CREATION
samza-sql-core/src/main/java/org/apache/samza/sql/operators/window/WindowOp.java PRE-CREATION
samza-sql-core/src/main/java/org/apache/samza/sql/operators/window/WindowOpSpec.java PRE-CREATION
samza-sql-core/src/main/java/org/apache/samza/sql/operators/window/WindowSpec.java PRE-CREATION
samza-sql-core/src/main/java/org/apache/samza/sql/operators/window/WindowState.java PRE-CREATION
samza-sql-core/src/main/java/org/apache/samza/sql/window/storage/FilteredMessageIterator.java PRE-CREATION
samza-sql-core/src/main/java/org/apache/samza/sql/window/storage/HashPrefixedMessageStore.java PRE-CREATION
samza-sql-core/src/main/java/org/apache/samza/sql/window/storage/MessageStore.java PRE-CREATION
samza-sql-core/src/main/java/org/apache/samza/sql/window/storage/MessageStoreSpec.java PRE-CREATION
samza-sql-core/src/main/java/org/apache/samza/sql/window/storage/OffsetKey.java PRE-CREATION
samza-sql-core/src/main/java/org/apache/samza/sql/window/storage/OrderedStoreKey.java PRE-CREATION
samza-sql-core/src/main/java/org/apache/samza/sql/window/storage/PrefixedKey.java PRE-CREATION
samza-sql-core/src/main/java/org/apache/samza/sql/window/storage/Range.java PRE-CREATION
samza-sql-core/src/main/java/org/apache/samza/sql/window/storage/TimeAndOffsetKey.java PRE-CREATION
samza-sql-core/src/main/java/org/apache/samza/sql/window/storage/TimeKey.java PRE-CREATION
samza-sql-core/src/main/java/org/apache/samza/sql/window/storage/WindowOutputStream.java PRE-CREATION
samza-sql-core/src/main/java/org/apache/samza/sql/window/storage/WindowState.java PRE-CREATION
samza-sql-core/src/main/java/org/apache/samza/sql/window/storage/WindowStore.java PRE-CREATION
samza-sql-core/src/main/java/org/apache/samza/task/sql/SimpleMessageCollector.java PRE-CREATION
samza-sql-core/src/test/java/org/apache/samza/task/sql/RandomWindowOperatorTask.java PRE-CREATION
samza-sql-core/src/test/java/org/apache/samza/task/sql/StreamSqlTask.java PRE-CREATION
samza-sql-core/src/test/java/org/apache/samza/task/sql/UserCallbacksSqlTask.java PRE-CREATION
Diff: https://reviews.apache.org/r/34664/diff/
Testing
-------
Thanks,
Yi Pan (Data Infrastructure)