You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@samza.apache.org by Milinda Pathirage <mi...@apache.org> on 2015/02/05 17:07:27 UTC
Review Request 30667: Samza Calcite Integration Prototype (SAMZA-483)
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30667/
-----------------------------------------------------------
Review request for samza, Chris Riccomini, Julian Hyde, and Yi Pan (Data Infrastructure).
Repository: samza
Description
-------
# Samza Calcite Integration Prototype
- Supports simple stream query planning based on Calcite's 'chi' branch (https://github.com/julianhyde/incubator-calcite/tree/chi) with STREAM support.
- No query planning customizations specific to Samza yet.
- Calcite integration related code can be found under package 'org.apache.samza.sql.compiler'
## Next Steps
- Implement a compiler which converts query plan generated by Calcite to a set of Samza Jobs
- Integrate the Calcite query optimizer
Diffs
-----
build.gradle 4d0b44f
gradle/dependency-versions.gradle 12b324d
samza-sql/README PRE-CREATION
samza-sql/src/main/java/org/apache/samza/sql/api/data/Data.java PRE-CREATION
samza-sql/src/main/java/org/apache/samza/sql/api/data/EntityName.java PRE-CREATION
samza-sql/src/main/java/org/apache/samza/sql/api/data/Relation.java PRE-CREATION
samza-sql/src/main/java/org/apache/samza/sql/api/data/Schema.java PRE-CREATION
samza-sql/src/main/java/org/apache/samza/sql/api/data/Tuple.java PRE-CREATION
samza-sql/src/main/java/org/apache/samza/sql/api/operators/Operator.java PRE-CREATION
samza-sql/src/main/java/org/apache/samza/sql/api/operators/RelationOperator.java PRE-CREATION
samza-sql/src/main/java/org/apache/samza/sql/api/operators/SqlOperatorFactory.java PRE-CREATION
samza-sql/src/main/java/org/apache/samza/sql/api/operators/TupleOperator.java PRE-CREATION
samza-sql/src/main/java/org/apache/samza/sql/api/operators/spec/OperatorSpec.java PRE-CREATION
samza-sql/src/main/java/org/apache/samza/sql/api/router/OperatorRouter.java PRE-CREATION
samza-sql/src/main/java/org/apache/samza/sql/compiler/SamzaCalciteConnection.java PRE-CREATION
samza-sql/src/main/java/org/apache/samza/sql/compiler/SamzaPreparingStmt.java PRE-CREATION
samza-sql/src/main/java/org/apache/samza/sql/compiler/SamzaQueryPreparator.java PRE-CREATION
samza-sql/src/main/java/org/apache/samza/sql/compiler/SamzaSqlValidator.java PRE-CREATION
samza-sql/src/main/java/org/apache/samza/sql/data/IncomingMessageTuple.java PRE-CREATION
samza-sql/src/main/java/org/apache/samza/sql/data/avro/AvroData.java PRE-CREATION
samza-sql/src/main/java/org/apache/samza/sql/data/avro/AvroSchema.java PRE-CREATION
samza-sql/src/main/java/org/apache/samza/sql/data/serializers/SqlStringSerde.java PRE-CREATION
samza-sql/src/main/java/org/apache/samza/sql/data/serializers/SqlStringSerdeFactory.java PRE-CREATION
samza-sql/src/main/java/org/apache/samza/sql/data/string/StringData.java PRE-CREATION
samza-sql/src/main/java/org/apache/samza/sql/data/string/StringSchema.java PRE-CREATION
samza-sql/src/main/java/org/apache/samza/sql/operators/factory/SimpleOperator.java PRE-CREATION
samza-sql/src/main/java/org/apache/samza/sql/operators/factory/SimpleOperatorFactoryImpl.java PRE-CREATION
samza-sql/src/main/java/org/apache/samza/sql/operators/factory/SimpleOperatorSpec.java PRE-CREATION
samza-sql/src/main/java/org/apache/samza/sql/operators/partition/PartitionOp.java PRE-CREATION
samza-sql/src/main/java/org/apache/samza/sql/operators/partition/PartitionSpec.java PRE-CREATION
samza-sql/src/main/java/org/apache/samza/sql/operators/relation/Join.java PRE-CREATION
samza-sql/src/main/java/org/apache/samza/sql/operators/relation/JoinSpec.java PRE-CREATION
samza-sql/src/main/java/org/apache/samza/sql/operators/stream/InsertStream.java PRE-CREATION
samza-sql/src/main/java/org/apache/samza/sql/operators/stream/InsertStreamSpec.java PRE-CREATION
samza-sql/src/main/java/org/apache/samza/sql/operators/window/BoundedTimeWindow.java PRE-CREATION
samza-sql/src/main/java/org/apache/samza/sql/operators/window/WindowSpec.java PRE-CREATION
samza-sql/src/main/java/org/apache/samza/sql/operators/window/WindowState.java PRE-CREATION
samza-sql/src/main/java/org/apache/samza/sql/router/SimpleRouter.java PRE-CREATION
samza-sql/src/main/java/org/apache/samza/sql/task/StoreMessageCollector.java PRE-CREATION
samza-sql/src/main/java/org/apache/samza/task/sql/OperatorMessageCollector.java PRE-CREATION
samza-sql/src/main/java/org/apache/samza/task/sql/SqlMessageCollector.java PRE-CREATION
samza-sql/src/main/java/org/apache/samza/task/sql/StoreMessageCollector.java PRE-CREATION
samza-sql/src/test/java/org/apache/samza/sql/compiler/PrepareQueryTest.java PRE-CREATION
samza-sql/src/test/java/org/apache/samza/sql/compiler/SamzaStreamTableFactory.java PRE-CREATION
samza-sql/src/test/java/org/apache/samza/sql/task/RandomOperatorTask.java PRE-CREATION
samza-sql/src/test/java/org/apache/samza/task/sql/RandomOperatorTask.java PRE-CREATION
samza-sql/src/test/java/org/apache/samza/task/sql/StreamSqlTask.java PRE-CREATION
settings.gradle bb07a3b
Diff: https://reviews.apache.org/r/30667/diff/
Testing
-------
Thanks,
Milinda Pathirage
Re: Review Request 30667: Samza Calcite Integration Prototype
(SAMZA-483)
Posted by Milinda Pathirage <mi...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30667/
-----------------------------------------------------------
(Updated Feb. 12, 2015, 3:47 a.m.)
Review request for samza, Chris Riccomini, Julian Hyde, and Yi Pan (Data Infrastructure).
Changes
-------
* Calcite based query planner code moved to 'org.apache.samza.sql.planner' package.
* Removed the schema factory prototype. It will be added later with schema-repo based implementaiton.
Repository: samza
Description
-------
# Samza Calcite Integration Prototype
- Supports simple stream query planning based on Calcite's 'chi' branch (https://github.com/julianhyde/incubator-calcite/tree/chi) with STREAM support.
- No query planning customizations specific to Samza yet.
- Calcite integration related code can be found under package 'org.apache.samza.sql.compiler'
## Next Steps
- Implement a compiler which converts query plan generated by Calcite to a set of Samza Jobs
- Integrate the Calcite query optimizer
Diffs (updated)
-----
build.gradle b49c313
samza-sql/src/main/java/org/apache/samza/sql/planner/QueryPlanner.java PRE-CREATION
samza-sql/src/main/java/org/apache/samza/sql/planner/SamzaCalciteConnection.java PRE-CREATION
samza-sql/src/main/java/org/apache/samza/sql/planner/SamzaQueryPreparingStatement.java PRE-CREATION
samza-sql/src/main/java/org/apache/samza/sql/planner/SamzaSqlValidator.java PRE-CREATION
samza-sql/src/test/java/org/apache/samza/sql/planner/QueryPlannerTest.java PRE-CREATION
samza-sql/src/test/java/org/apache/samza/sql/planner/SamzaStreamTableFactory.java PRE-CREATION
Diff: https://reviews.apache.org/r/30667/diff/
Testing (updated)
-------
./gradlew build was successful. But following errors were there in the console.
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "MockSystemConsumer-2"
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "MockSystemConsumer-7"
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "MockSystemConsumer-6"
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "MockSystemConsumer-0"
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "MockSystemConsumer-9"
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "MockSystemConsumer-10"
Thanks,
Milinda Pathirage
Re: Review Request 30667: Samza Calcite Integration Prototype
(SAMZA-483)
Posted by Milinda Pathirage <mi...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30667/
-----------------------------------------------------------
(Updated Feb. 6, 2015, 9:24 p.m.)
Review request for samza, Chris Riccomini, Julian Hyde, and Yi Pan (Data Infrastructure).
Changes
-------
Added **StreamFactory** (implementation of Calcite **TableFactory** interface) which introduces StreamScan to Calcite plan for streams. For example, for the query '**select stream * from orders where quantity > 5**' on **orders** stream, query planner will generate a plan like below:
```
LogicalDelta
LogicalProject(id=[$0], product=[$1], quantity=[$2])
LogicalFilter(condition=[>($2, 5)])
StreamScan(table=[[STREAMS, ORDERS]], fields=[[0, 1, 2]])
```
Repository: samza
Description
-------
# Samza Calcite Integration Prototype
- Supports simple stream query planning based on Calcite's 'chi' branch (https://github.com/julianhyde/incubator-calcite/tree/chi) with STREAM support.
- No query planning customizations specific to Samza yet.
- Calcite integration related code can be found under package 'org.apache.samza.sql.compiler'
## Next Steps
- Implement a compiler which converts query plan generated by Calcite to a set of Samza Jobs
- Integrate the Calcite query optimizer
Diffs (updated)
-----
build.gradle 4d0b44f
gradle/dependency-versions.gradle 12b324d
samza-sql/README PRE-CREATION
samza-sql/src/main/java/org/apache/samza/sql/api/data/Data.java PRE-CREATION
samza-sql/src/main/java/org/apache/samza/sql/api/data/EntityName.java PRE-CREATION
samza-sql/src/main/java/org/apache/samza/sql/api/data/Relation.java PRE-CREATION
samza-sql/src/main/java/org/apache/samza/sql/api/data/Schema.java PRE-CREATION
samza-sql/src/main/java/org/apache/samza/sql/api/data/Tuple.java PRE-CREATION
samza-sql/src/main/java/org/apache/samza/sql/api/operators/Operator.java PRE-CREATION
samza-sql/src/main/java/org/apache/samza/sql/api/operators/RelationOperator.java PRE-CREATION
samza-sql/src/main/java/org/apache/samza/sql/api/operators/SqlOperatorFactory.java PRE-CREATION
samza-sql/src/main/java/org/apache/samza/sql/api/operators/TupleOperator.java PRE-CREATION
samza-sql/src/main/java/org/apache/samza/sql/api/operators/spec/OperatorSpec.java PRE-CREATION
samza-sql/src/main/java/org/apache/samza/sql/api/router/OperatorRouter.java PRE-CREATION
samza-sql/src/main/java/org/apache/samza/sql/compiler/SamzaCalciteConnection.java PRE-CREATION
samza-sql/src/main/java/org/apache/samza/sql/compiler/SamzaPreparingStmt.java PRE-CREATION
samza-sql/src/main/java/org/apache/samza/sql/compiler/SamzaQueryPreparator.java PRE-CREATION
samza-sql/src/main/java/org/apache/samza/sql/compiler/SamzaSqlValidator.java PRE-CREATION
samza-sql/src/main/java/org/apache/samza/sql/data/IncomingMessageTuple.java PRE-CREATION
samza-sql/src/main/java/org/apache/samza/sql/data/avro/AvroData.java PRE-CREATION
samza-sql/src/main/java/org/apache/samza/sql/data/avro/AvroSchema.java PRE-CREATION
samza-sql/src/main/java/org/apache/samza/sql/data/serializers/SqlStringSerde.java PRE-CREATION
samza-sql/src/main/java/org/apache/samza/sql/data/serializers/SqlStringSerdeFactory.java PRE-CREATION
samza-sql/src/main/java/org/apache/samza/sql/data/string/StringData.java PRE-CREATION
samza-sql/src/main/java/org/apache/samza/sql/data/string/StringSchema.java PRE-CREATION
samza-sql/src/main/java/org/apache/samza/sql/metadata/SamzaSchema.java PRE-CREATION
samza-sql/src/main/java/org/apache/samza/sql/metadata/Stream.java PRE-CREATION
samza-sql/src/main/java/org/apache/samza/sql/metadata/StreamFactory.java PRE-CREATION
samza-sql/src/main/java/org/apache/samza/sql/metadata/StreamFieldType.java PRE-CREATION
samza-sql/src/main/java/org/apache/samza/sql/metadata/StreamScan.java PRE-CREATION
samza-sql/src/main/java/org/apache/samza/sql/metadata/Utils.java PRE-CREATION
samza-sql/src/main/java/org/apache/samza/sql/operators/factory/SimpleOperator.java PRE-CREATION
samza-sql/src/main/java/org/apache/samza/sql/operators/factory/SimpleOperatorFactoryImpl.java PRE-CREATION
samza-sql/src/main/java/org/apache/samza/sql/operators/factory/SimpleOperatorSpec.java PRE-CREATION
samza-sql/src/main/java/org/apache/samza/sql/operators/partition/PartitionOp.java PRE-CREATION
samza-sql/src/main/java/org/apache/samza/sql/operators/partition/PartitionSpec.java PRE-CREATION
samza-sql/src/main/java/org/apache/samza/sql/operators/relation/Join.java PRE-CREATION
samza-sql/src/main/java/org/apache/samza/sql/operators/relation/JoinSpec.java PRE-CREATION
samza-sql/src/main/java/org/apache/samza/sql/operators/stream/InsertStream.java PRE-CREATION
samza-sql/src/main/java/org/apache/samza/sql/operators/stream/InsertStreamSpec.java PRE-CREATION
samza-sql/src/main/java/org/apache/samza/sql/operators/window/BoundedTimeWindow.java PRE-CREATION
samza-sql/src/main/java/org/apache/samza/sql/operators/window/WindowSpec.java PRE-CREATION
samza-sql/src/main/java/org/apache/samza/sql/operators/window/WindowState.java PRE-CREATION
samza-sql/src/main/java/org/apache/samza/sql/router/SimpleRouter.java PRE-CREATION
samza-sql/src/main/java/org/apache/samza/sql/task/StoreMessageCollector.java PRE-CREATION
samza-sql/src/main/java/org/apache/samza/task/sql/OperatorMessageCollector.java PRE-CREATION
samza-sql/src/main/java/org/apache/samza/task/sql/SqlMessageCollector.java PRE-CREATION
samza-sql/src/main/java/org/apache/samza/task/sql/StoreMessageCollector.java PRE-CREATION
samza-sql/src/test/java/org/apache/samza/sql/compiler/PrepareQueryTest.java PRE-CREATION
samza-sql/src/test/java/org/apache/samza/sql/compiler/SamzaStreamTableFactory.java PRE-CREATION
samza-sql/src/test/java/org/apache/samza/sql/compiler/StreamFactoryTest.java PRE-CREATION
samza-sql/src/test/java/org/apache/samza/sql/task/RandomOperatorTask.java PRE-CREATION
samza-sql/src/test/java/org/apache/samza/task/sql/RandomOperatorTask.java PRE-CREATION
samza-sql/src/test/java/org/apache/samza/task/sql/StreamSqlTask.java PRE-CREATION
settings.gradle bb07a3b
Diff: https://reviews.apache.org/r/30667/diff/
Testing
-------
Thanks,
Milinda Pathirage
Re: Review Request 30667: Samza Calcite Integration Prototype
(SAMZA-483)
Posted by Julian Hyde <ju...@hydromatic.net>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30667/#review71295
-----------------------------------------------------------
samza-sql/src/main/java/org/apache/samza/sql/api/data/Data.java
<https://reviews.apache.org/r/30667/#comment116963>
I don't know how byte-arrays are handled elsewhere in Samza but consider making this return a ByteString (directly analogous to String but contains bytes rather than chars). It is immutable so you don't have to worry about sharing values.
- Julian Hyde
On Feb. 5, 2015, 4:07 p.m., Milinda Pathirage wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/30667/
> -----------------------------------------------------------
>
> (Updated Feb. 5, 2015, 4:07 p.m.)
>
>
> Review request for samza, Chris Riccomini, Julian Hyde, and Yi Pan (Data Infrastructure).
>
>
> Repository: samza
>
>
> Description
> -------
>
> # Samza Calcite Integration Prototype
>
> - Supports simple stream query planning based on Calcite's 'chi' branch (https://github.com/julianhyde/incubator-calcite/tree/chi) with STREAM support.
> - No query planning customizations specific to Samza yet.
> - Calcite integration related code can be found under package 'org.apache.samza.sql.compiler'
>
> ## Next Steps
>
> - Implement a compiler which converts query plan generated by Calcite to a set of Samza Jobs
> - Integrate the Calcite query optimizer
>
>
> Diffs
> -----
>
> build.gradle 4d0b44f
> gradle/dependency-versions.gradle 12b324d
> samza-sql/README PRE-CREATION
> samza-sql/src/main/java/org/apache/samza/sql/api/data/Data.java PRE-CREATION
> samza-sql/src/main/java/org/apache/samza/sql/api/data/EntityName.java PRE-CREATION
> samza-sql/src/main/java/org/apache/samza/sql/api/data/Relation.java PRE-CREATION
> samza-sql/src/main/java/org/apache/samza/sql/api/data/Schema.java PRE-CREATION
> samza-sql/src/main/java/org/apache/samza/sql/api/data/Tuple.java PRE-CREATION
> samza-sql/src/main/java/org/apache/samza/sql/api/operators/Operator.java PRE-CREATION
> samza-sql/src/main/java/org/apache/samza/sql/api/operators/RelationOperator.java PRE-CREATION
> samza-sql/src/main/java/org/apache/samza/sql/api/operators/SqlOperatorFactory.java PRE-CREATION
> samza-sql/src/main/java/org/apache/samza/sql/api/operators/TupleOperator.java PRE-CREATION
> samza-sql/src/main/java/org/apache/samza/sql/api/operators/spec/OperatorSpec.java PRE-CREATION
> samza-sql/src/main/java/org/apache/samza/sql/api/router/OperatorRouter.java PRE-CREATION
> samza-sql/src/main/java/org/apache/samza/sql/compiler/SamzaCalciteConnection.java PRE-CREATION
> samza-sql/src/main/java/org/apache/samza/sql/compiler/SamzaPreparingStmt.java PRE-CREATION
> samza-sql/src/main/java/org/apache/samza/sql/compiler/SamzaQueryPreparator.java PRE-CREATION
> samza-sql/src/main/java/org/apache/samza/sql/compiler/SamzaSqlValidator.java PRE-CREATION
> samza-sql/src/main/java/org/apache/samza/sql/data/IncomingMessageTuple.java PRE-CREATION
> samza-sql/src/main/java/org/apache/samza/sql/data/avro/AvroData.java PRE-CREATION
> samza-sql/src/main/java/org/apache/samza/sql/data/avro/AvroSchema.java PRE-CREATION
> samza-sql/src/main/java/org/apache/samza/sql/data/serializers/SqlStringSerde.java PRE-CREATION
> samza-sql/src/main/java/org/apache/samza/sql/data/serializers/SqlStringSerdeFactory.java PRE-CREATION
> samza-sql/src/main/java/org/apache/samza/sql/data/string/StringData.java PRE-CREATION
> samza-sql/src/main/java/org/apache/samza/sql/data/string/StringSchema.java PRE-CREATION
> samza-sql/src/main/java/org/apache/samza/sql/operators/factory/SimpleOperator.java PRE-CREATION
> samza-sql/src/main/java/org/apache/samza/sql/operators/factory/SimpleOperatorFactoryImpl.java PRE-CREATION
> samza-sql/src/main/java/org/apache/samza/sql/operators/factory/SimpleOperatorSpec.java PRE-CREATION
> samza-sql/src/main/java/org/apache/samza/sql/operators/partition/PartitionOp.java PRE-CREATION
> samza-sql/src/main/java/org/apache/samza/sql/operators/partition/PartitionSpec.java PRE-CREATION
> samza-sql/src/main/java/org/apache/samza/sql/operators/relation/Join.java PRE-CREATION
> samza-sql/src/main/java/org/apache/samza/sql/operators/relation/JoinSpec.java PRE-CREATION
> samza-sql/src/main/java/org/apache/samza/sql/operators/stream/InsertStream.java PRE-CREATION
> samza-sql/src/main/java/org/apache/samza/sql/operators/stream/InsertStreamSpec.java PRE-CREATION
> samza-sql/src/main/java/org/apache/samza/sql/operators/window/BoundedTimeWindow.java PRE-CREATION
> samza-sql/src/main/java/org/apache/samza/sql/operators/window/WindowSpec.java PRE-CREATION
> samza-sql/src/main/java/org/apache/samza/sql/operators/window/WindowState.java PRE-CREATION
> samza-sql/src/main/java/org/apache/samza/sql/router/SimpleRouter.java PRE-CREATION
> samza-sql/src/main/java/org/apache/samza/sql/task/StoreMessageCollector.java PRE-CREATION
> samza-sql/src/main/java/org/apache/samza/task/sql/OperatorMessageCollector.java PRE-CREATION
> samza-sql/src/main/java/org/apache/samza/task/sql/SqlMessageCollector.java PRE-CREATION
> samza-sql/src/main/java/org/apache/samza/task/sql/StoreMessageCollector.java PRE-CREATION
> samza-sql/src/test/java/org/apache/samza/sql/compiler/PrepareQueryTest.java PRE-CREATION
> samza-sql/src/test/java/org/apache/samza/sql/compiler/SamzaStreamTableFactory.java PRE-CREATION
> samza-sql/src/test/java/org/apache/samza/sql/task/RandomOperatorTask.java PRE-CREATION
> samza-sql/src/test/java/org/apache/samza/task/sql/RandomOperatorTask.java PRE-CREATION
> samza-sql/src/test/java/org/apache/samza/task/sql/StreamSqlTask.java PRE-CREATION
> settings.gradle bb07a3b
>
> Diff: https://reviews.apache.org/r/30667/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Milinda Pathirage
>
>