You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by GitBox <gi...@apache.org> on 2021/12/28 02:41:46 UTC
[GitHub] [incubator-doris] morningman opened a new issue #7502: [Discussion] Doris Roadmap 2022
morningman opened a new issue #7502:
URL: https://github.com/apache/incubator-doris/issues/7502
[English]()
以下是 Doris 社区 2022 年的总体规划。
规划包含待展开的、已展开的、已完成但需要持续优化的代码功能、文档、社区建设等各方面。
> 该规划目前处于讨论阶段，如果你对规划中或规划之外的任何方面有意见或建议，欢迎留言，或发送邮件至 dev@doris.apache.org.
> 我们会逐步对规划中的各个方向建立相关issue或jira以进行详细描述和跟踪进展。同时也欢迎有意愿贡献的开发者直接建立issue并与之关联（留言即可）
> 规划标有（Good First Issue）的方向都是较为独立的模块，比较适合作为新手任务或初次接触Doris的开发者。如果对相关方向感兴趣，请在 dev@doris.apache.org 邮件组中或本issue下与我们联系，我们会进行详细的指导、帮助和讨论。
> 标有（Q1）的方向是目前在一季度将完成的工作。其他方向我们也会逐步更新排期和进展。
> 标有（Done & Optimizing）的方向是目前已完成，但需要进行持续优化的方向。如易用性提升、功能补充和文档补充等。
> 我们鼓励开发者在 dev 邮件组中进行任何讨论，订阅邮件组请参考 [如何订阅](http://doris.incubator.apache.org/master/zh-CN/community/subscribe-mail-list.html#_1-%E5%8F%91%E9%80%81%E8%AE%A2%E9%98%85%E9%82%AE%E4%BB%B6)。
## 功能
- [ ] 全新的查询优化器
- [ ] 可扩展的全新查询优化器框架
- [ ] 统计信息收集和利用
- [ ] TPC-DS 查询计划优化
- [ ] Push-Based 查询执行引擎
- [ ] Pipeline 执行引擎
- [ ] 算子并发控制和资源控制
- [ ] 复合数据类型支持
- [ ] Array（Q1）
- [ ] Map
- [ ] Struct
- [ ] 半结构化数据支持
提供 Schemaless 语义以快速对半结构化数据进行分析。
- [ ] Json
- [ ] 冷热数据分离（Q1）
以分区粒度，支持冷数据存储到对象存储，并提供远程访问能力和本地 Cache 加速能力。
- [ ] 多表物化视图
Doris 当前的 "物化视图" 更偏向 "物化索引" 的概念。后续 Doris 将实现真正的物化视图（Materialized View）以支持单表、多表视图的全量、增量构建。
- [ ] 数据更新
提供类似 Kudu 的数据更新支持。
- [ ] 更多函数支持
- [ ] WindowFunnel
- [ ] UDF
支持新的 UDF 框架已解决现有 C++ 框架编写难度高、隔离性差、兼容性差等问题。
- [ ] UDF
- [ ] UDAF
- [ ] UDTF
- [ ] 导入格式支持（Good First Issue）
- [ ] 支持带转义的 csv 数据（common csv）
- [ ] 支持带 header 的 csv 数据（skip header）
## 性能优化
- [ ] 向量化执行引擎（Q1）
- [ ] 查询层向量化
- [ ] 存储层向量化
- [ ] 向量化函数补充
- [ ] 查询层存储层算子统一
- [ ] 导入向量化
- [ ] Json 解析优化（Good First Issue）
- [ ] 引入 [simdjson](https://github.com/simdjson/simdjson) 替换当前的 rapidjson
## 稳定性和可观测性
- [ ] 内存管理（Q1）
通过全局+线程+任务级别内存管理，解决内存预估不准、OOM等问题，并提升内存可观测性。
- [ ] IO 管理
通过全局 IO 管理提供细粒度的 IO 限速、优先级调度等功能。
- [ ] Tracing
引入 OpenTelemetry 增强系统内部状态的可观测性并统一监控数据格式。
## 测试
- [ ] 单元测试
- [ ] FE
完善 FE 单测框架以支持对功能进行多节点模拟测试。
- [ ] BE
提供测试框架以简化 BE 对复杂单元测试（如数据构建）的编写难度。
- [ ] 回归测试
提供 Case 采集或提交框架，用于完善和积累回归测试集。
- [ ] Benchmark 测试
提供 Benchmark 测试框架以确保新增代码不会对性能造成影响。
- [ ] 混沌测试
接入 ChaosMesh 混沌测试，提升系统在异常情况下的正确性和稳定性。
## 功能优化
- [ ] Z-Order Index
- [ ] 支持行存查询引擎 Z-Order （Done & Optimizing）
- [ ] 支持向量化引擎 Z-Order
- [ ] Agg/Uniq Key 模型支持 Z-Order
- [ ] Schema Change
- [ ] Lateral View
- [ ] 支持 bitmap、string、json_array 的展开（Done & Optimizing）
- [ ] 支持 Array 类型的展开
- [ ] Table Function
## 部署和运维
- [ ] Doris Manager（Q1）
提供可视化的 Doris 部署、监控和运维界面。简化 Doris 部署、扩容、升级、任务管理、状态查看等操作。
- [ ] Doris on K8S（Good First Issue）
提供基于 Helm Charts 的 K8S 部署方案。
## 周边生态
- [ ] 外部数据源联邦查询能力
- [ ] Hive （Done & Optimizing）
- [ ] Iceberg （Q1）
- [ ] Hudi
- [ ] 数据集成
- [ ] Routine Load 支持 Canal 格式解析（Q1）
- [ ] Flink Connector（Done & Optimizing）
- [ ] Spark Connector（Done & Optimizing）
- [ ] SeaTunnel Sink（Done & Optimizing）
- [ ] SeaTunnel Source
- [ ] DataX (Addax)（Done & Optimizing）
- [ ] 编译工具
- [ ] 支持 Clang 编译（Q1）
## 社区
- [ ] Doris 官网重构（Q1）
重构 Doris 官网，提供最佳实践、社区进展、博文、FAQ 等更多内容。
- [ ] Doris 文档编写（Good First Issue）
**非代码贡献和代码贡献同等重要**，社区非常欢迎开发者对项目文档进行改进、校对。
- [ ] 重新梳理 Doris 官方文档内容，增强文档的可阅读性、可操作性和引导性。
- [ ] 英文文档的翻译和校对。
- [ ] Github Action（Good First Issue）
引入更多的 Github Action 来帮助完善代码库的管理。包括但不限于 PR 的自动回复、打标签等等。如果你有好的 Action 推荐，欢迎留言。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] morningman edited a comment on issue #7502: Doris Roadmap 2022
Posted by GitBox <gi...@apache.org>.
morningman edited a comment on issue #7502:
URL: https://github.com/apache/incubator-doris/issues/7502#issuecomment-1001839293
The following is the Roadmap for the Doris community in 2022.
The plan includes all aspects of code features, documentation, community building, etc. that are to be developed, have already been developed, and have been completed but require ongoing optimization.
> The plan is currently under discussion, so if you have comments or suggestions on any aspect of the plan or beyond, please feel free to leave a comment or send an email to dev@doris.apache.org.
> We will gradually create issues or jira for each direction of the plan to describe and track the progress in detail. Developers who wish to contribute are also welcome to create issues directly and associate with them (just leave a comment)
> The directions marked (**Good First Issue**) in the plan are more independent modules, which are more suitable for newbie tasks or developers who are new to Doris. If you are interested in the relevant direction, please contact us at dev@doris.apache.org or under this issue, and we will provide detailed guidance, help and discussion.
> The directions marked with (**Q1**) are the current work to be completed in the first quarter of 2022. We will update the schedule and progress of other directions gradually.
> The marked (**Done & Optimizing**) directions are the directions that are currently completed but need continuous optimization. Such as ease of use, feature additions, and documentation additions.
> We encourage developers to discuss anything in the dev mailing list, to subscribe to the mailing list please refer to [How to subscribe](http://doris.incubator.apache.org/master/en/community/subscribe-mail-list.html).
## Features
- [ ] #7571
- [ ] Extensible new query optimizer framework
- [ ] Statistical information collection and utilization
- [ ] Standard test set support and performance enhancements
+ TPC-DS feature pass rate 100%
+ TPC-H performance enhancements
- [ ] #7572
- [ ] Pipeline execution engine
- [ ] Algorithm Concurrency Control and Resource Control
- [ ] #7573
- [ ] #7570
- [ ] Map
- [ ] Struct
- [ ] #7574
Provides Schemaless semantics for fast analysis of semi-structured data.
- [ ] Json
- [ ] #7575 (Q1)
Supports cold data storage to object storage at partition granularity with remote access capabilities and local Cache acceleration.
- [ ] #7503
Doris' current "materialized view" is more of a "materialized index" concept. Doris will later implement a true Materialized View to support full and incremental construction of single and multi-table views.
- [ ] #7576
Provide Kudu-like data update support.
- [ ] #7577
- [ ] WindowFunnel
- [ ] #7578
Support for the new UDF framework has solved the problems of high writing difficulty, poor isolation, and poor compatibility with existing C++ frameworks.
- [ ] UDF
- [ ] UDAF
- [ ] UDTF
- [ ] #7579 (**Good First Issue**)
- [ ] #7552
- [ ] #7650
## Performance Optimization
- [ ] #7580 (Q1)
- [ ] Query layer vectorization
- [ ] Storage level vectorization
- [ ] Vectorization function supplementation
- [ ] Query layer storage layer arithmetic unification
- [ ] Import Vectorization
- [ ] Json Parsing Optimization (**Good First Issue**)
- [ ] #7551
## Stability and Observability
- [ ] #7553 (Q1)
Solve the problems of inaccurate memory prediction and OOM, and improve memory observability by global + thread + task level memory management.
- [ ] #7581
Provides fine-grained IO speed limit, priority scheduling, etc. through global IO management.
- [ ] #7582
Introduces OpenTelemetry to enhance system internal state observability and unify monitoring data format.
## Testing
- [ ] #7583
- [ ] FE
Refine the FE single test framework to support multi-node simulation testing of features.
- [ ] BE
Provide testing framework to simplify the difficulty of writing complex unit tests (e.g. data builds) for BE.
- [ ] #7584
Provide Case collection or submission framework for refining and accumulating regression test sets.
- [ ] #7585
Provide a Benchmark testing framework to ensure that adding new code does not impact performance.
- [ ] #7586
Implement ChaosMesh chaos testing to improve the correctness and stability of the system in case of anomalies.
## Functional Optimization
- [ ] #7149
- [ ] #7149
- [ ] Support vectorization engine Z-Order
- [ ] Agg/Uniq Key model support for Z-Order
- [ ] Schema Change
- [ ] Lateral View
- [ ] Support bitmap, string, json_array expansion (Done & Optimizing)
- [ ] Array type expansion support
- [ ] Table Function
## Deployment and Maintenance
- [ ] #7591 (Q1)
Provides a visual interface for Doris deployment, monitoring, and operations maintenance. Simplifies Doris deployment, scaling, upgrades, task management, status checking, and other operations.
- [ ] #7587 (Good First Issue)
Provides a Helm Charts-based K8S deployment solution.
## Peripheral Ecology
- [ ] #7588
- [ ] #6568 (Done & Optimizing)
- [ ] #7389 (Q1)
- [ ] Hudi
- [ ] Parquet File Format Support
Generate and read Parquet files directly to facilitate Doris data to be read directly by external systems.
- [ ] Data Integration
- [ ] Routine Load support for Canal format parsing (Q1)
- [ ] Flink Connector (Done & Optimizing)
- [ ] Spark Connector (Done & Optimizing)
- [ ] SeaTunnel Sink (Done & Optimizing)
- [ ] SeaTunnel Source
- [ ] DataX (Addax) (Done & Optimizing)
- [ ] Compilation Tools
- [ ] #7590 (Q1)
## Community
- [ ] #7589 (Q1)
Refactored Doris official website to provide best practices, community progress, blog posts, FAQ, and more.
- [ ] Doris Documentation (**Good First Issue**)
**Non-code contributions are as important as code contributions**, and the community is very open to developers improving and proofreading the project documentation.
- [ ] #6336
- [ ] Reorganization of the official Doris documentation to improve readability, operability, and guidance.
- [ ] Translation and proofreading of the English documentation.
- [ ] Github Action (**Good First Issue**)
Introduced more Github Actions to help improve the management of the code base. This includes but is not limited to PR autoresponders, tagging, etc. If you have a good Action to recommend, please leave a comment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] morningman edited a comment on issue #7502: [Discussion] Doris Roadmap 2022
Posted by GitBox <gi...@apache.org>.
morningman edited a comment on issue #7502:
URL: https://github.com/apache/incubator-doris/issues/7502#issuecomment-1001839293
The following is the Roadmap for the Doris community in 2022.
The plan includes all aspects of code features, documentation, community building, etc. that are to be developed, have already been developed, and have been completed but require ongoing optimization.
> The plan is currently under discussion, so if you have comments or suggestions on any aspect of the plan or beyond, please feel free to leave a comment or send an email to dev@doris.apache.org.
> We will gradually create issues or jira for each direction of the plan to describe and track the progress in detail. Developers who wish to contribute are also welcome to create issues directly and associate with them (just leave a comment)
> The directions marked (**Good First Issue**) in the plan are more independent modules, which are more suitable for newbie tasks or developers who are new to Doris. If you are interested in the relevant direction, please contact us at dev@doris.apache.org or under this issue, and we will provide detailed guidance, help and discussion.
> The directions marked with (**Q1**) are the current work to be completed in the first quarter of 2022. We will update the schedule and progress of other directions gradually.
> The marked (**Done & Optimizing**) directions are the directions that are currently completed but need continuous optimization. Such as ease of use, feature additions, and documentation additions.
> We encourage developers to discuss anything in the dev mailing list, to subscribe to the mailing list please refer to [How to subscribe](http://doris.incubator.apache.org/master/en/community/subscribe-mail-list.html).
## Features
- [ ] #7571
- [ ] Extensible new query optimizer framework
- [ ] Statistical information collection and utilization
- [ ] Standard test set support and performance enhancements
+ TPC-DS feature pass rate 100%
+ TPC-H performance enhancements
- [ ] #7572
- [ ] Pipeline execution engine
- [ ] Algorithm Concurrency Control and Resource Control
- [ ] #7573
- [ ] #7570
- [ ] Map
- [ ] Struct
- [ ] #7574
Provides Schemaless semantics for fast analysis of semi-structured data.
- [ ] Json
- [ ] #7575 (Q1)
Supports cold data storage to object storage at partition granularity with remote access capabilities and local Cache acceleration.
- [ ] #7503
Doris' current "materialized view" is more of a "materialized index" concept. Doris will later implement a true Materialized View to support full and incremental construction of single and multi-table views.
- [ ] #7576
Provide Kudu-like data update support.
- [ ] #7577
- [ ] WindowFunnel
- [ ] #7578
Support for the new UDF framework has solved the problems of high writing difficulty, poor isolation, and poor compatibility with existing C++ frameworks.
- [ ] UDF
- [ ] UDAF
- [ ] UDTF
- [ ] #7579 (**Good First Issue**)
- [ ] #7552
## Performance Optimization
- [ ] #7580 (Q1)
- [ ] Query layer vectorization
- [ ] Storage level vectorization
- [ ] Vectorization function supplementation
- [ ] Query layer storage layer arithmetic unification
- [ ] Import Vectorization
- [ ] Json Parsing Optimization (**Good First Issue**)
- [ ] #7551
## Stability and Observability
- [ ] #7553 (Q1)
Solve the problems of inaccurate memory prediction and OOM, and improve memory observability by global + thread + task level memory management.
- [ ] #7581
Provides fine-grained IO speed limit, priority scheduling, etc. through global IO management.
- [ ] #7582
Introduces OpenTelemetry to enhance system internal state observability and unify monitoring data format.
## Testing
- [ ] #7583
- [ ] FE
Refine the FE single test framework to support multi-node simulation testing of features.
- [ ] BE
Provide testing framework to simplify the difficulty of writing complex unit tests (e.g. data builds) for BE.
- [ ] #7584
Provide Case collection or submission framework for refining and accumulating regression test sets.
- [ ] #7585
Provide a Benchmark testing framework to ensure that adding new code does not impact performance.
- [ ] #7586
Implement ChaosMesh chaos testing to improve the correctness and stability of the system in case of anomalies.
## Functional Optimization
- [ ] #7149
- [ ] #7149
- [ ] Support vectorization engine Z-Order
- [ ] Agg/Uniq Key model support for Z-Order
- [ ] Schema Change
- [ ] Lateral View
- [ ] Support bitmap, string, json_array expansion (Done & Optimizing)
- [ ] Array type expansion support
- [ ] Table Function
## Deployment and Maintenance
- [ ] Doris Manager (Q1)
Provides a visual interface for Doris deployment, monitoring, and operations maintenance. Simplifies Doris deployment, scaling, upgrades, task management, status checking, and other operations.
- [ ] #7587 (Good First Issue)
Provides a Helm Charts-based K8S deployment solution.
## Peripheral Ecology
- [ ] #7588
- [ ] #6568 (Done & Optimizing)
- [ ] #7389 (Q1)
- [ ] Hudi
- [ ] Parquet File Format Support
Generate and read Parquet files directly to facilitate Doris data to be read directly by external systems.
- [ ] Data Integration
- [ ] Routine Load support for Canal format parsing (Q1)
- [ ] Flink Connector (Done & Optimizing)
- [ ] Spark Connector (Done & Optimizing)
- [ ] SeaTunnel Sink (Done & Optimizing)
- [ ] SeaTunnel Source
- [ ] DataX (Addax) (Done & Optimizing)
- [ ] Compilation Tools
- [ ] #7590 (Q1)
## Community
- [ ] #7589 (Q1)
Refactored Doris official website to provide best practices, community progress, blog posts, FAQ, and more.
- [ ] Doris Documentation (**Good First Issue**)
**Non-code contributions are as important as code contributions**, and the community is very open to developers improving and proofreading the project documentation.
- [ ] #6336
- [ ] Reorganization of the official Doris documentation to improve readability, operability, and guidance.
- [ ] Translation and proofreading of the English documentation.
- [ ] Github Action (Good First Issue)
Introduced more Github Actions to help improve the management of the code base. This includes but is not limited to PR autoresponders, tagging, etc. If you have a good Action to recommend, please leave a comment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] morningman edited a comment on issue #7502: Doris Roadmap 2022
Posted by GitBox <gi...@apache.org>.
morningman edited a comment on issue #7502:
URL: https://github.com/apache/incubator-doris/issues/7502#issuecomment-1001839293
The following is the Roadmap for the Doris community in 2022.
The plan includes all aspects of code features, documentation, community building, etc. that are to be developed, have already been developed, and have been completed but require ongoing optimization.
> The plan is currently under discussion, so if you have comments or suggestions on any aspect of the plan or beyond, please feel free to leave a comment or send an email to dev@doris.apache.org.
> We will gradually create issues or jira for each direction of the plan to describe and track the progress in detail. Developers who wish to contribute are also welcome to create issues directly and associate with them (just leave a comment)
> The directions marked (**Good First Issue**) in the plan are more independent modules, which are more suitable for newbie tasks or developers who are new to Doris. If you are interested in the relevant direction, please contact us at dev@doris.apache.org or under this issue, and we will provide detailed guidance, help and discussion.
> The directions marked with (**Q1**) are the current work to be completed in the first quarter of 2022. We will update the schedule and progress of other directions gradually.
> The marked (**Done & Optimizing**) directions are the directions that are currently completed but need continuous optimization. Such as ease of use, feature additions, and documentation additions.
> We encourage developers to discuss anything in the dev mailing list, to subscribe to the mailing list please refer to [How to subscribe](http://doris.incubator.apache.org/master/en/community/subscribe-mail-list.html).
## Features
- [ ] #7571
- [ ] Extensible new query optimizer framework
- [ ] Statistical information collection and utilization
- [ ] Standard test set support and performance enhancements
+ TPC-DS feature pass rate 100%
+ TPC-H performance enhancements
- [ ] #7572
- [ ] Pipeline execution engine
- [ ] Algorithm Concurrency Control and Resource Control
- [ ] #7573
- [ ] #7570
- [ ] Map
- [ ] Struct
- [ ] #7574
Provides Schemaless semantics for fast analysis of semi-structured data.
- [ ] Json
- [ ] #7575 (Q1)
Supports cold data storage to object storage at partition granularity with remote access capabilities and local Cache acceleration.
- [ ] #7503
Doris' current "materialized view" is more of a "materialized index" concept. Doris will later implement a true Materialized View to support full and incremental construction of single and multi-table views.
- [ ] #7576
Provide Kudu-like data update support.
- [ ] #7577
- [ ] WindowFunnel
- [ ] #7578
Support for the new UDF framework has solved the problems of high writing difficulty, poor isolation, and poor compatibility with existing C++ frameworks.
- [ ] UDF
- [ ] UDAF
- [ ] UDTF
- [ ] #7579 (**Good First Issue**)
- [ ] #7552
- [ ] #7650
- [ ] Add more resource limits
- [ ] #7129
- [ ] More builtin function support
- [ ] #7678
## Performance Optimization
- [ ] #7580 (Q1)
- [ ] Query layer vectorization
- [ ] Storage level vectorization
- [ ] Vectorization function supplementation
- [ ] Query layer storage layer arithmetic unification
- [ ] Import Vectorization
- [ ] Json Parsing Optimization (**Good First Issue**)
- [ ] #7551
- [ ] #7743
Optimize the performance of compaction task. And try to refactor the compaction logic. For example, only one replica do the compaction and sync to other replicas.
## Stability and Observability
- [ ] #7553 (Q1)
Solve the problems of inaccurate memory prediction and OOM, and improve memory observability by global + thread + task level memory management.
- [ ] #7581
Provides fine-grained IO speed limit, priority scheduling, etc. through global IO management.
- [ ] #7582
Introduces OpenTelemetry to enhance system internal state observability and unify monitoring data format.
## Testing
- [ ] #7583
- [ ] FE
Refine the FE single test framework to support multi-node simulation testing of features.
- [ ] BE
Provide testing framework to simplify the difficulty of writing complex unit tests (e.g. data builds) for BE.
- [ ] #7584
Provide Case collection or submission framework for refining and accumulating regression test sets.
- [ ] #7585
Provide a Benchmark testing framework to ensure that adding new code does not impact performance.
- [ ] #7586
Implement ChaosMesh chaos testing to improve the correctness and stability of the system in case of anomalies.
## Functional Optimization
- [ ] #7149
- [ ] #7149
- [ ] Support vectorization engine Z-Order
- [ ] Agg/Uniq Key model support for Z-Order
- [ ] Schema Change
- [ ] Lateral View
- [ ] Support bitmap, string, json_array expansion (Done & Optimizing)
- [ ] Array type expansion support
- [ ] Table Function
- [ ] Other features
- [ ] #7680
- [ ] CreateTableAsStmt support decimal
## Deployment and Maintenance
- [ ] #7591 (Q1)
Provides a visual interface for Doris deployment, monitoring, and operations maintenance. Simplifies Doris deployment, scaling, upgrades, task management, status checking, and other operations.
- [ ] #7587 (Good First Issue)
Provides a Helm Charts-based K8S deployment solution.
## Peripheral Ecology
- [ ] #7588
- [ ] #6568 (Done & Optimizing)
- [ ] #7389 (Q1)
- [ ] Hudi
- [ ] Parquet File Format Support
Generate and read Parquet files directly to facilitate Doris data to be read directly by external systems.
- [ ] Data Integration
- [ ] Routine Load support for Canal format parsing (Q1)
- [ ] Flink Connector (Done & Optimizing)
- [ ] Spark Connector (Done & Optimizing)
- [ ] SeaTunnel Sink (Done & Optimizing)
- [ ] SeaTunnel Source
- [ ] DataX (Addax) (Done & Optimizing)
- [ ] #7781
- [ ] Compilation Tools
- [ ] #7590 (Q1)
## Community
- [ ] #7589 (Q1)
Refactored Doris official website to provide best practices, community progress, blog posts, FAQ, and more.
- [ ] Doris Documentation (**Good First Issue**)
**Non-code contributions are as important as code contributions**, and the community is very open to developers improving and proofreading the project documentation.
- [ ] #6336
- [ ] Reorganization of the official Doris documentation to improve readability, operability, and guidance.
- [ ] Translation and proofreading of the English documentation.
- [ ] Github Action (**Good First Issue**)
Introduced more Github Actions to help improve the management of the code base. This includes but is not limited to PR autoresponders, tagging, etc. If you have a good Action to recommend, please leave a comment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] jackwener commented on issue #7502: Doris Roadmap 2022
Posted by GitBox <gi...@apache.org>.
jackwener commented on issue #7502:
URL: https://github.com/apache/incubator-doris/issues/7502#issuecomment-1004580582
这是 cockroach 的 [实践](https://github.com/cockroachdb/cockroach/blob/master/docs/RFCS/README.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] morningman edited a comment on issue #7502: Doris Roadmap 2022
Posted by GitBox <gi...@apache.org>.
morningman edited a comment on issue #7502:
URL: https://github.com/apache/incubator-doris/issues/7502#issuecomment-1001839293
The following is the Roadmap for the Doris community in 2022.
The plan includes all aspects of code features, documentation, community building, etc. that are to be developed, have already been developed, and have been completed but require ongoing optimization.
> The plan is currently under discussion, so if you have comments or suggestions on any aspect of the plan or beyond, please feel free to leave a comment or send an email to dev@doris.apache.org.
> We will gradually create issues or jira for each direction of the plan to describe and track the progress in detail. Developers who wish to contribute are also welcome to create issues directly and associate with them (just leave a comment)
> The directions marked (**Good First Issue**) in the plan are more independent modules, which are more suitable for newbie tasks or developers who are new to Doris. If you are interested in the relevant direction, please contact us at dev@doris.apache.org or under this issue, and we will provide detailed guidance, help and discussion.
> The directions marked with (**Q1**) are the current work to be completed in the first quarter of 2022. We will update the schedule and progress of other directions gradually.
> The marked (**Done & Optimizing**) directions are the directions that are currently completed but need continuous optimization. Such as ease of use, feature additions, and documentation additions.
> We encourage developers to discuss anything in the dev mailing list, to subscribe to the mailing list please refer to [How to subscribe](http://doris.incubator.apache.org/master/en/community/subscribe-mail-list.html).
## Features
- [ ] #7571
- [ ] Extensible new query optimizer framework
- [ ] #6370
- [ ] Standard test set support and performance enhancements
+ TPC-DS feature pass rate 100%
+ TPC-H performance enhancements
- [ ] #7572
- [ ] Pipeline execution engine
- [ ] Algorithm Concurrency Control and Resource Control
- [ ] #7573
- [ ] #7570
- [ ] Map
- [ ] Struct
- [ ] #7574
Provides Schemaless semantics for fast analysis of semi-structured data.
- [ ] Json
- [ ] #7575 (Q1)
Supports cold data storage to object storage at partition granularity with remote access capabilities and local Cache acceleration.
- [ ] #7503
Doris' current "materialized view" is more of a "materialized index" concept. Doris will later implement a true Materialized View to support full and incremental construction of single and multi-table views.
- [ ] #7576
Provide Kudu-like data update support.
- [ ] #7577
- [ ] WindowFunnel #8299
- [ ] #7578
Support for the new UDF framework has solved the problems of high writing difficulty, poor isolation, and poor compatibility with existing C++ frameworks.
- [x] UDF #7519
- [ ] UDAF #8312
- [ ] UDTF
- [ ] Java UDF #8389
- [ ] #7579 (**Good First Issue**)
- [ ] #7552
- [ ] #7650
- [ ] Add more resource limits
- [ ] #7129
- [ ] More builtin function support
- [ ] #7678
## Performance Optimization
- [ ] #7580 (Q1)
- [ ] Query layer vectorization
- [ ] Storage level vectorization
- [ ] Vectorization function supplementation
- [ ] Query layer storage layer arithmetic unification
- [ ] Import Vectorization
- [ ] Json Parsing Optimization (**Good First Issue**)
- [ ] #7551
- [ ] #7743
Optimize the performance of compaction task. And try to refactor the compaction logic. For example, only one replica do the compaction and sync to other replicas.
## Stability and Observability
- [ ] #7553 (Q1)
Solve the problems of inaccurate memory prediction and OOM, and improve memory observability by global + thread + task level memory management.
- [ ] #7581
Provides fine-grained IO speed limit, priority scheduling, etc. through global IO management.
- [ ] #7582
Introduces OpenTelemetry to enhance system internal state observability and unify monitoring data format.
## Testing
- [ ] #7583
- [ ] FE
Refine the FE single test framework to support multi-node simulation testing of features.
- [ ] BE
Provide testing framework to simplify the difficulty of writing complex unit tests (e.g. data builds) for BE.
- [ ] #7584
Provide Case collection or submission framework for refining and accumulating regression test sets.
- [ ] #7585
Provide a Benchmark testing framework to ensure that adding new code does not impact performance.
- [ ] #7586
Implement ChaosMesh chaos testing to improve the correctness and stability of the system in case of anomalies.
## Functional Optimization
- [ ] #7149
- [ ] #7149
- [ ] Support vectorization engine Z-Order
- [ ] Agg/Uniq Key model support for Z-Order
- [ ] Schema Change
- [ ] Lateral View
- [ ] Support bitmap, string, json_array expansion (Done & Optimizing)
- [ ] Array type expansion support
- [ ] Table Function
- [ ] Other features
- [ ] #7680
- [ ] CreateTableAsStmt support decimal
## Deployment and Maintenance
- [ ] #7591 (Q1)
Provides a visual interface for Doris deployment, monitoring, and operations maintenance. Simplifies Doris deployment, scaling, upgrades, task management, status checking, and other operations.
- [ ] #7587 (Good First Issue)
Provides a Helm Charts-based K8S deployment solution.
## Peripheral Ecology
- [ ] #7588
- [ ] #6568 (Done & Optimizing)
- [ ] #7389 (Q1)
- [ ] Hudi
- [ ] Parquet File Format Support
Generate and read Parquet files directly to facilitate Doris data to be read directly by external systems.
- [ ] Data Integration
- [ ] Routine Load support for Canal format parsing (Q1)
- [ ] Flink Connector (Done & Optimizing)
- [ ] Spark Connector (Done & Optimizing)
- [ ] SeaTunnel Sink (Done & Optimizing)
- [ ] SeaTunnel Source
- [ ] DataX (Addax) (Done & Optimizing)
- [ ] #7781
- [ ] Compilation Tools
- [ ] #7590 (Q1)
## Community
- [ ] #7589 (Q1)
Refactored Doris official website to provide best practices, community progress, blog posts, FAQ, and more.
- [ ] Doris Documentation (**Good First Issue**)
**Non-code contributions are as important as code contributions**, and the community is very open to developers improving and proofreading the project documentation.
- [ ] #6336
- [ ] Reorganization of the official Doris documentation to improve readability, operability, and guidance.
- [ ] Translation and proofreading of the English documentation.
- [ ] Github Action (**Good First Issue**)
Introduced more Github Actions to help improve the management of the code base. This includes but is not limited to PR autoresponders, tagging, etc. If you have a good Action to recommend, please leave a comment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] hf200012 edited a comment on issue #7502: [Discussion] Doris Roadmap 2022
Posted by GitBox <gi...@apache.org>.
hf200012 edited a comment on issue #7502:
URL: https://github.com/apache/incubator-doris/issues/7502#issuecomment-1001856012
Doris Manager：
1.Follow-up Doris Manager upgrade
2.User UI interaction improvement
3.Doris Manager supports Doris automated upgrade
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] yiguolei commented on issue #7502: Doris Roadmap 2022
Posted by GitBox <gi...@apache.org>.
yiguolei commented on issue #7502:
URL: https://github.com/apache/incubator-doris/issues/7502#issuecomment-1015197025
@i7xh
Two example:
1. Currently doris‘s concurrency control is based on tablet, one tablet ---> exec fragment， there is only one thread to deal with the data at query engine. If use pushed engine, could expand the computing thread num at run time.
2. In pull engine, if one fragment contains 3 or more node, like scan--> filter--> agg there is only one node is executing, but in push based engine, node executing could be async, for example scan and agg could execute at same time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] hf200012 commented on issue #7502: Doris Roadmap 2022
Posted by GitBox <gi...@apache.org>.
hf200012 commented on issue #7502:
URL: https://github.com/apache/incubator-doris/issues/7502#issuecomment-1015048834
#7678 max_by, min_by aggregate function support
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] 924060929 commented on issue #7502: [Discussion] Doris Roadmap 2022
Posted by GitBox <gi...@apache.org>.
924060929 commented on issue #7502:
URL: https://github.com/apache/incubator-doris/issues/7502#issuecomment-1001898872
我们公司已经有一个回归测试框架。大体是用groovy的dsl去完成测试sql、stream load、安装tpch等功能，大概使用方式如下图。
后续可以提给社区
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] morningman edited a comment on issue #7502: Doris Roadmap 2022
Posted by GitBox <gi...@apache.org>.
morningman edited a comment on issue #7502:
URL: https://github.com/apache/incubator-doris/issues/7502#issuecomment-1001839293
The following is the Roadmap for the Doris community in 2022.
The plan includes all aspects of code features, documentation, community building, etc. that are to be developed, have already been developed, and have been completed but require ongoing optimization.
> The plan is currently under discussion, so if you have comments or suggestions on any aspect of the plan or beyond, please feel free to leave a comment or send an email to dev@doris.apache.org.
> We will gradually create issues or jira for each direction of the plan to describe and track the progress in detail. Developers who wish to contribute are also welcome to create issues directly and associate with them (just leave a comment)
> The directions marked (**Good First Issue**) in the plan are more independent modules, which are more suitable for newbie tasks or developers who are new to Doris. If you are interested in the relevant direction, please contact us at dev@doris.apache.org or under this issue, and we will provide detailed guidance, help and discussion.
> The directions marked with (**Q1**) are the current work to be completed in the first quarter of 2022. We will update the schedule and progress of other directions gradually.
> The marked (**Done & Optimizing**) directions are the directions that are currently completed but need continuous optimization. Such as ease of use, feature additions, and documentation additions.
> We encourage developers to discuss anything in the dev mailing list, to subscribe to the mailing list please refer to [How to subscribe](http://doris.incubator.apache.org/master/en/community/subscribe-mail-list.html).
## Features
- [ ] #7571
- [ ] Extensible new query optimizer framework
- [ ] #6370
- [ ] Standard test set support and performance enhancements
+ TPC-DS feature pass rate 100%
+ TPC-H performance enhancements
- [ ] #7572
- [ ] Pipeline execution engine
- [ ] Algorithm Concurrency Control and Resource Control
- [ ] #7573
- [ ] #7570
- [ ] Map
- [ ] Struct
- [ ] #7574
Provides Schemaless semantics for fast analysis of semi-structured data.
- [ ] Json
- [ ] #7575 (Q1)
Supports cold data storage to object storage at partition granularity with remote access capabilities and local Cache acceleration.
- [ ] #7503
Doris' current "materialized view" is more of a "materialized index" concept. Doris will later implement a true Materialized View to support full and incremental construction of single and multi-table views.
- [ ] #7576
Provide Kudu-like data update support.
- [ ] #7577
- [ ] WindowFunnel #8299
- [ ] #7578
Support for the new UDF framework has solved the problems of high writing difficulty, poor isolation, and poor compatibility with existing C++ frameworks.
- [x] UDF #7519
- [ ] UDAF
- [ ] UDTF
- [ ] #7579 (**Good First Issue**)
- [ ] #7552
- [ ] #7650
- [ ] Add more resource limits
- [ ] #7129
- [ ] More builtin function support
- [ ] #7678
## Performance Optimization
- [ ] #7580 (Q1)
- [ ] Query layer vectorization
- [ ] Storage level vectorization
- [ ] Vectorization function supplementation
- [ ] Query layer storage layer arithmetic unification
- [ ] Import Vectorization
- [ ] Json Parsing Optimization (**Good First Issue**)
- [ ] #7551
- [ ] #7743
Optimize the performance of compaction task. And try to refactor the compaction logic. For example, only one replica do the compaction and sync to other replicas.
## Stability and Observability
- [ ] #7553 (Q1)
Solve the problems of inaccurate memory prediction and OOM, and improve memory observability by global + thread + task level memory management.
- [ ] #7581
Provides fine-grained IO speed limit, priority scheduling, etc. through global IO management.
- [ ] #7582
Introduces OpenTelemetry to enhance system internal state observability and unify monitoring data format.
## Testing
- [ ] #7583
- [ ] FE
Refine the FE single test framework to support multi-node simulation testing of features.
- [ ] BE
Provide testing framework to simplify the difficulty of writing complex unit tests (e.g. data builds) for BE.
- [ ] #7584
Provide Case collection or submission framework for refining and accumulating regression test sets.
- [ ] #7585
Provide a Benchmark testing framework to ensure that adding new code does not impact performance.
- [ ] #7586
Implement ChaosMesh chaos testing to improve the correctness and stability of the system in case of anomalies.
## Functional Optimization
- [ ] #7149
- [ ] #7149
- [ ] Support vectorization engine Z-Order
- [ ] Agg/Uniq Key model support for Z-Order
- [ ] Schema Change
- [ ] Lateral View
- [ ] Support bitmap, string, json_array expansion (Done & Optimizing)
- [ ] Array type expansion support
- [ ] Table Function
- [ ] Other features
- [ ] #7680
- [ ] CreateTableAsStmt support decimal
## Deployment and Maintenance
- [ ] #7591 (Q1)
Provides a visual interface for Doris deployment, monitoring, and operations maintenance. Simplifies Doris deployment, scaling, upgrades, task management, status checking, and other operations.
- [ ] #7587 (Good First Issue)
Provides a Helm Charts-based K8S deployment solution.
## Peripheral Ecology
- [ ] #7588
- [ ] #6568 (Done & Optimizing)
- [ ] #7389 (Q1)
- [ ] Hudi
- [ ] Parquet File Format Support
Generate and read Parquet files directly to facilitate Doris data to be read directly by external systems.
- [ ] Data Integration
- [ ] Routine Load support for Canal format parsing (Q1)
- [ ] Flink Connector (Done & Optimizing)
- [ ] Spark Connector (Done & Optimizing)
- [ ] SeaTunnel Sink (Done & Optimizing)
- [ ] SeaTunnel Source
- [ ] DataX (Addax) (Done & Optimizing)
- [ ] #7781
- [ ] Compilation Tools
- [ ] #7590 (Q1)
## Community
- [ ] #7589 (Q1)
Refactored Doris official website to provide best practices, community progress, blog posts, FAQ, and more.
- [ ] Doris Documentation (**Good First Issue**)
**Non-code contributions are as important as code contributions**, and the community is very open to developers improving and proofreading the project documentation.
- [ ] #6336
- [ ] Reorganization of the official Doris documentation to improve readability, operability, and guidance.
- [ ] Translation and proofreading of the English documentation.
- [ ] Github Action (**Good First Issue**)
Introduced more Github Actions to help improve the management of the code base. This includes but is not limited to PR autoresponders, tagging, etc. If you have a good Action to recommend, please leave a comment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] morningman edited a comment on issue #7502: Doris Roadmap 2022
Posted by GitBox <gi...@apache.org>.
morningman edited a comment on issue #7502:
URL: https://github.com/apache/incubator-doris/issues/7502#issuecomment-1001839293
The following is the Roadmap for the Doris community in 2022.
The plan includes all aspects of code features, documentation, community building, etc. that are to be developed, have already been developed, and have been completed but require ongoing optimization.
> The plan is currently under discussion, so if you have comments or suggestions on any aspect of the plan or beyond, please feel free to leave a comment or send an email to dev@doris.apache.org.
> We will gradually create issues or jira for each direction of the plan to describe and track the progress in detail. Developers who wish to contribute are also welcome to create issues directly and associate with them (just leave a comment)
> The directions marked (**Good First Issue**) in the plan are more independent modules, which are more suitable for newbie tasks or developers who are new to Doris. If you are interested in the relevant direction, please contact us at dev@doris.apache.org or under this issue, and we will provide detailed guidance, help and discussion.
> The directions marked with (**Q1**) are the current work to be completed in the first quarter of 2022. We will update the schedule and progress of other directions gradually.
> The marked (**Done & Optimizing**) directions are the directions that are currently completed but need continuous optimization. Such as ease of use, feature additions, and documentation additions.
> We encourage developers to discuss anything in the dev mailing list, to subscribe to the mailing list please refer to [How to subscribe](http://doris.incubator.apache.org/master/en/community/subscribe-mail-list.html).
## Features
- [ ] #7571
- [ ] Extensible new query optimizer framework
- [ ] #6370
- [ ] Standard test set support and performance enhancements
+ TPC-DS feature pass rate 100%
+ TPC-H performance enhancements
- [ ] #7572
- [ ] Pipeline execution engine
- [ ] Algorithm Concurrency Control and Resource Control
- [ ] #7573
- [ ] #7570
- [ ] Map
- [ ] Struct
- [ ] #7574
Provides Schemaless semantics for fast analysis of semi-structured data.
- [ ] Json
- [ ] #7575 (Q1)
Supports cold data storage to object storage at partition granularity with remote access capabilities and local Cache acceleration.
- [ ] #7503
Doris' current "materialized view" is more of a "materialized index" concept. Doris will later implement a true Materialized View to support full and incremental construction of single and multi-table views.
- [ ] #7576
Provide Kudu-like data update support.
- [ ] #7577
- [ ] WindowFunnel
- [ ] #7578
Support for the new UDF framework has solved the problems of high writing difficulty, poor isolation, and poor compatibility with existing C++ frameworks.
- [x] UDF #7519
- [ ] UDAF
- [ ] UDTF
- [ ] #7579 (**Good First Issue**)
- [ ] #7552
- [ ] #7650
- [ ] Add more resource limits
- [ ] #7129
- [ ] More builtin function support
- [ ] #7678
## Performance Optimization
- [ ] #7580 (Q1)
- [ ] Query layer vectorization
- [ ] Storage level vectorization
- [ ] Vectorization function supplementation
- [ ] Query layer storage layer arithmetic unification
- [ ] Import Vectorization
- [ ] Json Parsing Optimization (**Good First Issue**)
- [ ] #7551
- [ ] #7743
Optimize the performance of compaction task. And try to refactor the compaction logic. For example, only one replica do the compaction and sync to other replicas.
## Stability and Observability
- [ ] #7553 (Q1)
Solve the problems of inaccurate memory prediction and OOM, and improve memory observability by global + thread + task level memory management.
- [ ] #7581
Provides fine-grained IO speed limit, priority scheduling, etc. through global IO management.
- [ ] #7582
Introduces OpenTelemetry to enhance system internal state observability and unify monitoring data format.
## Testing
- [ ] #7583
- [ ] FE
Refine the FE single test framework to support multi-node simulation testing of features.
- [ ] BE
Provide testing framework to simplify the difficulty of writing complex unit tests (e.g. data builds) for BE.
- [ ] #7584
Provide Case collection or submission framework for refining and accumulating regression test sets.
- [ ] #7585
Provide a Benchmark testing framework to ensure that adding new code does not impact performance.
- [ ] #7586
Implement ChaosMesh chaos testing to improve the correctness and stability of the system in case of anomalies.
## Functional Optimization
- [ ] #7149
- [ ] #7149
- [ ] Support vectorization engine Z-Order
- [ ] Agg/Uniq Key model support for Z-Order
- [ ] Schema Change
- [ ] Lateral View
- [ ] Support bitmap, string, json_array expansion (Done & Optimizing)
- [ ] Array type expansion support
- [ ] Table Function
- [ ] Other features
- [ ] #7680
- [ ] CreateTableAsStmt support decimal
## Deployment and Maintenance
- [ ] #7591 (Q1)
Provides a visual interface for Doris deployment, monitoring, and operations maintenance. Simplifies Doris deployment, scaling, upgrades, task management, status checking, and other operations.
- [ ] #7587 (Good First Issue)
Provides a Helm Charts-based K8S deployment solution.
## Peripheral Ecology
- [ ] #7588
- [ ] #6568 (Done & Optimizing)
- [ ] #7389 (Q1)
- [ ] Hudi
- [ ] Parquet File Format Support
Generate and read Parquet files directly to facilitate Doris data to be read directly by external systems.
- [ ] Data Integration
- [ ] Routine Load support for Canal format parsing (Q1)
- [ ] Flink Connector (Done & Optimizing)
- [ ] Spark Connector (Done & Optimizing)
- [ ] SeaTunnel Sink (Done & Optimizing)
- [ ] SeaTunnel Source
- [ ] DataX (Addax) (Done & Optimizing)
- [ ] #7781
- [ ] Compilation Tools
- [ ] #7590 (Q1)
## Community
- [ ] #7589 (Q1)
Refactored Doris official website to provide best practices, community progress, blog posts, FAQ, and more.
- [ ] Doris Documentation (**Good First Issue**)
**Non-code contributions are as important as code contributions**, and the community is very open to developers improving and proofreading the project documentation.
- [ ] #6336
- [ ] Reorganization of the official Doris documentation to improve readability, operability, and guidance.
- [ ] Translation and proofreading of the English documentation.
- [ ] Github Action (**Good First Issue**)
Introduced more Github Actions to help improve the management of the code base. This includes but is not limited to PR autoresponders, tagging, etc. If you have a good Action to recommend, please leave a comment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] yangzhg commented on issue #7502: [Discussion] Doris Roadmap 2022
Posted by GitBox <gi...@apache.org>.
yangzhg commented on issue #7502:
URL: https://github.com/apache/incubator-doris/issues/7502#issuecomment-1001845011
支持parquet 文件存储格式也应该加进去吧
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] i7xh commented on issue #7502: Doris Roadmap 2022
Posted by GitBox <gi...@apache.org>.
i7xh commented on issue #7502:
URL: https://github.com/apache/incubator-doris/issues/7502#issuecomment-1016385262
<img width="524" alt="企业微信截图_a18eb192-6b6a-4615-a6a5-eeae4d0430d2" src="https://user-images.githubusercontent.com/635169/150123300-5dc4c6d5-71ad-4a5f-a0d8-ba272d2ee316.png">
look forward imperatively
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] jackwener commented on issue #7502: [Discussion] Doris Roadmap 2022
Posted by GitBox <gi...@apache.org>.
jackwener commented on issue #7502:
URL: https://github.com/apache/incubator-doris/issues/7502#issuecomment-1003732305
既然后续有这么多内容，关于社区部分建一个 RFC 目录挺有必要的，大型的 PR 的 design doc 放进去，一方面是为了社区新人的快速融入，另外也减小PR review的压力
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] Henry2SS commented on issue #7502: Doris Roadmap 2022
Posted by GitBox <gi...@apache.org>.
Henry2SS commented on issue #7502:
URL: https://github.com/apache/incubator-doris/issues/7502#issuecomment-1006594182
> > What about supporting AVRO format in LOAD function?
>
> #7650
Thx for opening an issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] morningman edited a comment on issue #7502: Doris Roadmap 2022
Posted by GitBox <gi...@apache.org>.
morningman edited a comment on issue #7502:
URL: https://github.com/apache/incubator-doris/issues/7502#issuecomment-1001839293
The following is the Roadmap for the Doris community in 2022.
The plan includes all aspects of code features, documentation, community building, etc. that are to be developed, have already been developed, and have been completed but require ongoing optimization.
> The plan is currently under discussion, so if you have comments or suggestions on any aspect of the plan or beyond, please feel free to leave a comment or send an email to dev@doris.apache.org.
> We will gradually create issues or jira for each direction of the plan to describe and track the progress in detail. Developers who wish to contribute are also welcome to create issues directly and associate with them (just leave a comment)
> The directions marked (**Good First Issue**) in the plan are more independent modules, which are more suitable for newbie tasks or developers who are new to Doris. If you are interested in the relevant direction, please contact us at dev@doris.apache.org or under this issue, and we will provide detailed guidance, help and discussion.
> The directions marked with (**Q1**) are the current work to be completed in the first quarter of 2022. We will update the schedule and progress of other directions gradually.
> The marked (**Done & Optimizing**) directions are the directions that are currently completed but need continuous optimization. Such as ease of use, feature additions, and documentation additions.
> We encourage developers to discuss anything in the dev mailing list, to subscribe to the mailing list please refer to [How to subscribe](http://doris.incubator.apache.org/master/en/community/subscribe-mail-list.html).
## Features
- [ ] #7571
- [ ] Extensible new query optimizer framework
- [ ] Statistical information collection and utilization
- [ ] Standard test set support and performance enhancements
+ TPC-DS feature pass rate 100%
+ TPC-H performance enhancements
- [ ] #7572
- [ ] Pipeline execution engine
- [ ] Algorithm Concurrency Control and Resource Control
- [ ] #7573
- [ ] #7570
- [ ] Map
- [ ] Struct
- [ ] #7574
Provides Schemaless semantics for fast analysis of semi-structured data.
- [ ] Json
- [ ] #7575 (Q1)
Supports cold data storage to object storage at partition granularity with remote access capabilities and local Cache acceleration.
- [ ] #7503
Doris' current "materialized view" is more of a "materialized index" concept. Doris will later implement a true Materialized View to support full and incremental construction of single and multi-table views.
- [ ] #7576
Provide Kudu-like data update support.
- [ ] #7577
- [ ] WindowFunnel
- [ ] #7578
Support for the new UDF framework has solved the problems of high writing difficulty, poor isolation, and poor compatibility with existing C++ frameworks.
- [ ] UDF
- [ ] UDAF
- [ ] UDTF
- [ ] #7579 (**Good First Issue**)
- [ ] #7552
- [ ] #7650
- [ ] Add more resource limits
- [ ] #7129
## Performance Optimization
- [ ] #7580 (Q1)
- [ ] Query layer vectorization
- [ ] Storage level vectorization
- [ ] Vectorization function supplementation
- [ ] Query layer storage layer arithmetic unification
- [ ] Import Vectorization
- [ ] Json Parsing Optimization (**Good First Issue**)
- [ ] #7551
- [ ] #7743
Optimize the performance of compaction task. And try to refactor the compaction logic. For example, only one replica do the compaction and sync to other replicas.
## Stability and Observability
- [ ] #7553 (Q1)
Solve the problems of inaccurate memory prediction and OOM, and improve memory observability by global + thread + task level memory management.
- [ ] #7581
Provides fine-grained IO speed limit, priority scheduling, etc. through global IO management.
- [ ] #7582
Introduces OpenTelemetry to enhance system internal state observability and unify monitoring data format.
## Testing
- [ ] #7583
- [ ] FE
Refine the FE single test framework to support multi-node simulation testing of features.
- [ ] BE
Provide testing framework to simplify the difficulty of writing complex unit tests (e.g. data builds) for BE.
- [ ] #7584
Provide Case collection or submission framework for refining and accumulating regression test sets.
- [ ] #7585
Provide a Benchmark testing framework to ensure that adding new code does not impact performance.
- [ ] #7586
Implement ChaosMesh chaos testing to improve the correctness and stability of the system in case of anomalies.
## Functional Optimization
- [ ] #7149
- [ ] #7149
- [ ] Support vectorization engine Z-Order
- [ ] Agg/Uniq Key model support for Z-Order
- [ ] Schema Change
- [ ] Lateral View
- [ ] Support bitmap, string, json_array expansion (Done & Optimizing)
- [ ] Array type expansion support
- [ ] Table Function
## Deployment and Maintenance
- [ ] #7591 (Q1)
Provides a visual interface for Doris deployment, monitoring, and operations maintenance. Simplifies Doris deployment, scaling, upgrades, task management, status checking, and other operations.
- [ ] #7587 (Good First Issue)
Provides a Helm Charts-based K8S deployment solution.
## Peripheral Ecology
- [ ] #7588
- [ ] #6568 (Done & Optimizing)
- [ ] #7389 (Q1)
- [ ] Hudi
- [ ] Parquet File Format Support
Generate and read Parquet files directly to facilitate Doris data to be read directly by external systems.
- [ ] Data Integration
- [ ] Routine Load support for Canal format parsing (Q1)
- [ ] Flink Connector (Done & Optimizing)
- [ ] Spark Connector (Done & Optimizing)
- [ ] SeaTunnel Sink (Done & Optimizing)
- [ ] SeaTunnel Source
- [ ] DataX (Addax) (Done & Optimizing)
- [ ] Compilation Tools
- [ ] #7590 (Q1)
## Community
- [ ] #7589 (Q1)
Refactored Doris official website to provide best practices, community progress, blog posts, FAQ, and more.
- [ ] Doris Documentation (**Good First Issue**)
**Non-code contributions are as important as code contributions**, and the community is very open to developers improving and proofreading the project documentation.
- [ ] #6336
- [ ] Reorganization of the official Doris documentation to improve readability, operability, and guidance.
- [ ] Translation and proofreading of the English documentation.
- [ ] Github Action (**Good First Issue**)
Introduced more Github Actions to help improve the management of the code base. This includes but is not limited to PR autoresponders, tagging, etc. If you have a good Action to recommend, please leave a comment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] morningman edited a comment on issue #7502: Doris Roadmap 2022
Posted by GitBox <gi...@apache.org>.
morningman edited a comment on issue #7502:
URL: https://github.com/apache/incubator-doris/issues/7502#issuecomment-1001839293
The following is the Roadmap for the Doris community in 2022.
The plan includes all aspects of code features, documentation, community building, etc. that are to be developed, have already been developed, and have been completed but require ongoing optimization.
> The plan is currently under discussion, so if you have comments or suggestions on any aspect of the plan or beyond, please feel free to leave a comment or send an email to dev@doris.apache.org.
> We will gradually create issues or jira for each direction of the plan to describe and track the progress in detail. Developers who wish to contribute are also welcome to create issues directly and associate with them (just leave a comment)
> The directions marked (**Good First Issue**) in the plan are more independent modules, which are more suitable for newbie tasks or developers who are new to Doris. If you are interested in the relevant direction, please contact us at dev@doris.apache.org or under this issue, and we will provide detailed guidance, help and discussion.
> The directions marked with (**Q1**) are the current work to be completed in the first quarter of 2022. We will update the schedule and progress of other directions gradually.
> The marked (**Done & Optimizing**) directions are the directions that are currently completed but need continuous optimization. Such as ease of use, feature additions, and documentation additions.
> We encourage developers to discuss anything in the dev mailing list, to subscribe to the mailing list please refer to [How to subscribe](http://doris.incubator.apache.org/master/en/community/subscribe-mail-list.html).
## Features
- [ ] #7571
- [ ] Extensible new query optimizer framework
- [ ] Statistical information collection and utilization
- [ ] Standard test set support and performance enhancements
+ TPC-DS feature pass rate 100%
+ TPC-H performance enhancements
- [ ] #7572
- [ ] Pipeline execution engine
- [ ] Algorithm Concurrency Control and Resource Control
- [ ] #7573
- [ ] #7570
- [ ] Map
- [ ] Struct
- [ ] #7574
Provides Schemaless semantics for fast analysis of semi-structured data.
- [ ] Json
- [ ] #7575 (Q1)
Supports cold data storage to object storage at partition granularity with remote access capabilities and local Cache acceleration.
- [ ] #7503
Doris' current "materialized view" is more of a "materialized index" concept. Doris will later implement a true Materialized View to support full and incremental construction of single and multi-table views.
- [ ] #7576
Provide Kudu-like data update support.
- [ ] #7577
- [ ] WindowFunnel
- [ ] #7578
Support for the new UDF framework has solved the problems of high writing difficulty, poor isolation, and poor compatibility with existing C++ frameworks.
- [ ] UDF
- [ ] UDAF
- [ ] UDTF
- [ ] #7579 (**Good First Issue**)
- [ ] #7552
- [ ] #7650
## Performance Optimization
- [ ] #7580 (Q1)
- [ ] Query layer vectorization
- [ ] Storage level vectorization
- [ ] Vectorization function supplementation
- [ ] Query layer storage layer arithmetic unification
- [ ] Import Vectorization
- [ ] Json Parsing Optimization (**Good First Issue**)
- [ ] #7551
- [ ] #7743
Optimize the performance of compaction task. And try to refactor the compaction logic. For example, only one replica do the compaction and sync to other replicas.
## Stability and Observability
- [ ] #7553 (Q1)
Solve the problems of inaccurate memory prediction and OOM, and improve memory observability by global + thread + task level memory management.
- [ ] #7581
Provides fine-grained IO speed limit, priority scheduling, etc. through global IO management.
- [ ] #7582
Introduces OpenTelemetry to enhance system internal state observability and unify monitoring data format.
## Testing
- [ ] #7583
- [ ] FE
Refine the FE single test framework to support multi-node simulation testing of features.
- [ ] BE
Provide testing framework to simplify the difficulty of writing complex unit tests (e.g. data builds) for BE.
- [ ] #7584
Provide Case collection or submission framework for refining and accumulating regression test sets.
- [ ] #7585
Provide a Benchmark testing framework to ensure that adding new code does not impact performance.
- [ ] #7586
Implement ChaosMesh chaos testing to improve the correctness and stability of the system in case of anomalies.
## Functional Optimization
- [ ] #7149
- [ ] #7149
- [ ] Support vectorization engine Z-Order
- [ ] Agg/Uniq Key model support for Z-Order
- [ ] Schema Change
- [ ] Lateral View
- [ ] Support bitmap, string, json_array expansion (Done & Optimizing)
- [ ] Array type expansion support
- [ ] Table Function
## Deployment and Maintenance
- [ ] #7591 (Q1)
Provides a visual interface for Doris deployment, monitoring, and operations maintenance. Simplifies Doris deployment, scaling, upgrades, task management, status checking, and other operations.
- [ ] #7587 (Good First Issue)
Provides a Helm Charts-based K8S deployment solution.
## Peripheral Ecology
- [ ] #7588
- [ ] #6568 (Done & Optimizing)
- [ ] #7389 (Q1)
- [ ] Hudi
- [ ] Parquet File Format Support
Generate and read Parquet files directly to facilitate Doris data to be read directly by external systems.
- [ ] Data Integration
- [ ] Routine Load support for Canal format parsing (Q1)
- [ ] Flink Connector (Done & Optimizing)
- [ ] Spark Connector (Done & Optimizing)
- [ ] SeaTunnel Sink (Done & Optimizing)
- [ ] SeaTunnel Source
- [ ] DataX (Addax) (Done & Optimizing)
- [ ] Compilation Tools
- [ ] #7590 (Q1)
## Community
- [ ] #7589 (Q1)
Refactored Doris official website to provide best practices, community progress, blog posts, FAQ, and more.
- [ ] Doris Documentation (**Good First Issue**)
**Non-code contributions are as important as code contributions**, and the community is very open to developers improving and proofreading the project documentation.
- [ ] #6336
- [ ] Reorganization of the official Doris documentation to improve readability, operability, and guidance.
- [ ] Translation and proofreading of the English documentation.
- [ ] Github Action (**Good First Issue**)
Introduced more Github Actions to help improve the management of the code base. This includes but is not limited to PR autoresponders, tagging, etc. If you have a good Action to recommend, please leave a comment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] morningman commented on issue #7502: Doris Roadmap 2022
Posted by GitBox <gi...@apache.org>.
morningman commented on issue #7502:
URL: https://github.com/apache/incubator-doris/issues/7502#issuecomment-1004517609
> 既然后续有这么多内容，关于社区部分建一个 RFC 目录挺有必要的，大型的 PR 的 design doc 放进去，一方面是为了社区新人的快速融入，另外也减小PR review的压力
好主意，你是否有一些RFC 模板可供参考？
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] i7xh edited a comment on issue #7502: Doris Roadmap 2022
Posted by GitBox <gi...@apache.org>.
i7xh edited a comment on issue #7502:
URL: https://github.com/apache/incubator-doris/issues/7502#issuecomment-1015191906
Why Doris need push based query execution engine?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] lordk911 commented on issue #7502: Doris Roadmap 2022
Posted by GitBox <gi...@apache.org>.
lordk911 commented on issue #7502:
URL: https://github.com/apache/incubator-doris/issues/7502#issuecomment-1061367317
want to know when will vectorized query engine could be released.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] hf200012 commented on issue #7502: Doris Roadmap 2022
Posted by GitBox <gi...@apache.org>.
hf200012 commented on issue #7502:
URL: https://github.com/apache/incubator-doris/issues/7502#issuecomment-1015047945
#7680 Data export function supports exporting to db, kafka, etc.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] yiguolei commented on issue #7502: [Discussion] Doris Roadmap 2022
Posted by GitBox <gi...@apache.org>.
yiguolei commented on issue #7502:
URL: https://github.com/apache/incubator-doris/issues/7502#issuecomment-1001841703
For regression test and performance test, we could follow clickhouse's test method. If it is allowed, I could do this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] morningman commented on issue #7502: Doris Roadmap 2022
Posted by GitBox <gi...@apache.org>.
morningman commented on issue #7502:
URL: https://github.com/apache/incubator-doris/issues/7502#issuecomment-1006698059
> > > What about supporting AVRO format in LOAD function?
> >
> >
> > #7650
>
> Thx for opening an issue.
Add to the roadmap
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] caiconghui commented on issue #7502: Doris Roadmap 2022
Posted by GitBox <gi...@apache.org>.
caiconghui commented on issue #7502:
URL: https://github.com/apache/incubator-doris/issues/7502#issuecomment-1006591968
> What about supporting AVRO format in LOAD function?
#7650
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] yiguolei commented on issue #7502: Doris Roadmap 2022
Posted by GitBox <gi...@apache.org>.
yiguolei commented on issue #7502:
URL: https://github.com/apache/incubator-doris/issues/7502#issuecomment-1015182922
Could use vectorized method to optimize load process??
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] morningman edited a comment on issue #7502: Doris Roadmap 2022
Posted by GitBox <gi...@apache.org>.
morningman edited a comment on issue #7502:
URL: https://github.com/apache/incubator-doris/issues/7502#issuecomment-1001839293
The following is the Roadmap for the Doris community in 2022.
The plan includes all aspects of code features, documentation, community building, etc. that are to be developed, have already been developed, and have been completed but require ongoing optimization.
> The plan is currently under discussion, so if you have comments or suggestions on any aspect of the plan or beyond, please feel free to leave a comment or send an email to dev@doris.apache.org.
> We will gradually create issues or jira for each direction of the plan to describe and track the progress in detail. Developers who wish to contribute are also welcome to create issues directly and associate with them (just leave a comment)
> The directions marked (**Good First Issue**) in the plan are more independent modules, which are more suitable for newbie tasks or developers who are new to Doris. If you are interested in the relevant direction, please contact us at dev@doris.apache.org or under this issue, and we will provide detailed guidance, help and discussion.
> The directions marked with (**Q1**) are the current work to be completed in the first quarter of 2022. We will update the schedule and progress of other directions gradually.
> The marked (**Done & Optimizing**) directions are the directions that are currently completed but need continuous optimization. Such as ease of use, feature additions, and documentation additions.
> We encourage developers to discuss anything in the dev mailing list, to subscribe to the mailing list please refer to [How to subscribe](http://doris.incubator.apache.org/master/en/community/subscribe-mail-list.html).
## Features
- [ ] #7571
- [ ] Extensible new query optimizer framework
- [ ] #6370
- [ ] Standard test set support and performance enhancements
+ TPC-DS feature pass rate 100%
+ TPC-H performance enhancements
- [ ] #7572
- [ ] Pipeline execution engine
- [ ] Algorithm Concurrency Control and Resource Control
- [ ] #7573
- [ ] #7570
- [ ] Map
- [ ] Struct
- [ ] #7574
Provides Schemaless semantics for fast analysis of semi-structured data.
- [ ] Json
- [ ] #7575 (Q1)
Supports cold data storage to object storage at partition granularity with remote access capabilities and local Cache acceleration.
- [ ] #7503
Doris' current "materialized view" is more of a "materialized index" concept. Doris will later implement a true Materialized View to support full and incremental construction of single and multi-table views.
- [ ] #7576
Provide Kudu-like data update support.
- [ ] #7577
- [ ] WindowFunnel
- [ ] #7578
Support for the new UDF framework has solved the problems of high writing difficulty, poor isolation, and poor compatibility with existing C++ frameworks.
- [ ] UDF
- [ ] UDAF
- [ ] UDTF
- [ ] #7579 (**Good First Issue**)
- [ ] #7552
- [ ] #7650
- [ ] Add more resource limits
- [ ] #7129
- [ ] More builtin function support
- [ ] #7678
## Performance Optimization
- [ ] #7580 (Q1)
- [ ] Query layer vectorization
- [ ] Storage level vectorization
- [ ] Vectorization function supplementation
- [ ] Query layer storage layer arithmetic unification
- [ ] Import Vectorization
- [ ] Json Parsing Optimization (**Good First Issue**)
- [ ] #7551
- [ ] #7743
Optimize the performance of compaction task. And try to refactor the compaction logic. For example, only one replica do the compaction and sync to other replicas.
## Stability and Observability
- [ ] #7553 (Q1)
Solve the problems of inaccurate memory prediction and OOM, and improve memory observability by global + thread + task level memory management.
- [ ] #7581
Provides fine-grained IO speed limit, priority scheduling, etc. through global IO management.
- [ ] #7582
Introduces OpenTelemetry to enhance system internal state observability and unify monitoring data format.
## Testing
- [ ] #7583
- [ ] FE
Refine the FE single test framework to support multi-node simulation testing of features.
- [ ] BE
Provide testing framework to simplify the difficulty of writing complex unit tests (e.g. data builds) for BE.
- [ ] #7584
Provide Case collection or submission framework for refining and accumulating regression test sets.
- [ ] #7585
Provide a Benchmark testing framework to ensure that adding new code does not impact performance.
- [ ] #7586
Implement ChaosMesh chaos testing to improve the correctness and stability of the system in case of anomalies.
## Functional Optimization
- [ ] #7149
- [ ] #7149
- [ ] Support vectorization engine Z-Order
- [ ] Agg/Uniq Key model support for Z-Order
- [ ] Schema Change
- [ ] Lateral View
- [ ] Support bitmap, string, json_array expansion (Done & Optimizing)
- [ ] Array type expansion support
- [ ] Table Function
- [ ] Other features
- [ ] #7680
- [ ] CreateTableAsStmt support decimal
## Deployment and Maintenance
- [ ] #7591 (Q1)
Provides a visual interface for Doris deployment, monitoring, and operations maintenance. Simplifies Doris deployment, scaling, upgrades, task management, status checking, and other operations.
- [ ] #7587 (Good First Issue)
Provides a Helm Charts-based K8S deployment solution.
## Peripheral Ecology
- [ ] #7588
- [ ] #6568 (Done & Optimizing)
- [ ] #7389 (Q1)
- [ ] Hudi
- [ ] Parquet File Format Support
Generate and read Parquet files directly to facilitate Doris data to be read directly by external systems.
- [ ] Data Integration
- [ ] Routine Load support for Canal format parsing (Q1)
- [ ] Flink Connector (Done & Optimizing)
- [ ] Spark Connector (Done & Optimizing)
- [ ] SeaTunnel Sink (Done & Optimizing)
- [ ] SeaTunnel Source
- [ ] DataX (Addax) (Done & Optimizing)
- [ ] #7781
- [ ] Compilation Tools
- [ ] #7590 (Q1)
## Community
- [ ] #7589 (Q1)
Refactored Doris official website to provide best practices, community progress, blog posts, FAQ, and more.
- [ ] Doris Documentation (**Good First Issue**)
**Non-code contributions are as important as code contributions**, and the community is very open to developers improving and proofreading the project documentation.
- [ ] #6336
- [ ] Reorganization of the official Doris documentation to improve readability, operability, and guidance.
- [ ] Translation and proofreading of the English documentation.
- [ ] Github Action (**Good First Issue**)
Introduced more Github Actions to help improve the management of the code base. This includes but is not limited to PR autoresponders, tagging, etc. If you have a good Action to recommend, please leave a comment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] wangshuo128 commented on issue #7502: [Discussion] Doris Roadmap 2022
Posted by GitBox <gi...@apache.org>.
wangshuo128 commented on issue #7502:
URL: https://github.com/apache/incubator-doris/issues/7502#issuecomment-1001845516
希望考虑跨版本升级功能。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] hf200012 commented on issue #7502: [Discussion] Doris Roadmap 2022
Posted by GitBox <gi...@apache.org>.
hf200012 commented on issue #7502:
URL: https://github.com/apache/incubator-doris/issues/7502#issuecomment-1001856012
外部数据源联邦查询能力：
1.kudu support
Doris Manager：
1.Follow-up Doris Manager upgrade
2.User UI interaction improvement
3.Doris Manager supports Doris automated upgrade
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] morningman edited a comment on issue #7502: [Discussion] Doris Roadmap 2022
Posted by GitBox <gi...@apache.org>.
morningman edited a comment on issue #7502:
URL: https://github.com/apache/incubator-doris/issues/7502#issuecomment-1001839293
The following is the Roadmap for the Doris community in 2022.
The plan includes all aspects of code features, documentation, community building, etc. that are to be developed, have already been developed, and have been completed but require ongoing optimization.
> The plan is currently under discussion, so if you have comments or suggestions on any aspect of the plan or beyond, please feel free to leave a comment or send an email to dev@doris.apache.org.
> We will gradually create issues or jira for each direction of the plan to describe and track the progress in detail. Developers who wish to contribute are also welcome to create issues directly and associate with them (just leave a comment)
> The directions marked (**Good First Issue**) in the plan are more independent modules, which are more suitable for newbie tasks or developers who are new to Doris. If you are interested in the relevant direction, please contact us at dev@doris.apache.org or under this issue, and we will provide detailed guidance, help and discussion.
> The directions marked with (**Q1**) are the current work to be completed in the first quarter of 2022. We will update the schedule and progress of other directions gradually.
> The marked (**Done & Optimizing**) directions are the directions that are currently completed but need continuous optimization. Such as ease of use, feature additions, and documentation additions.
> We encourage developers to discuss anything in the dev mailing list, to subscribe to the mailing list please refer to [How to subscribe](http://doris.incubator.apache.org/master/en/community/subscribe-mail-list.html).
## Features
- [ ] #7571
- [ ] Extensible new query optimizer framework
- [ ] Statistical information collection and utilization
- [ ] Standard test set support and performance enhancements
+ TPC-DS feature pass rate 100%
+ TPC-H performance enhancements
- [ ] #7572
- [ ] Pipeline execution engine
- [ ] Algorithm Concurrency Control and Resource Control
- [ ] #7573
- [ ] #7570
- [ ] Map
- [ ] Struct
- [ ] #7574
Provides Schemaless semantics for fast analysis of semi-structured data.
- [ ] Json
- [ ] #7575 (Q1)
Supports cold data storage to object storage at partition granularity with remote access capabilities and local Cache acceleration.
- [ ] #7503
Doris' current "materialized view" is more of a "materialized index" concept. Doris will later implement a true Materialized View to support full and incremental construction of single and multi-table views.
- [ ] #7576
Provide Kudu-like data update support.
- [ ] #7577
- [ ] WindowFunnel
- [ ] #7578
Support for the new UDF framework has solved the problems of high writing difficulty, poor isolation, and poor compatibility with existing C++ frameworks.
- [ ] UDF
- [ ] UDAF
- [ ] UDTF
- [ ] #7579 (**Good First Issue**)
- [ ] #7552
## Performance Optimization
- [ ] #7580 (Q1)
- [ ] Query layer vectorization
- [ ] Storage level vectorization
- [ ] Vectorization function supplementation
- [ ] Query layer storage layer arithmetic unification
- [ ] Import Vectorization
- [ ] Json Parsing Optimization (**Good First Issue**)
- [ ] #7551
## Stability and Observability
- [ ] #7553 (Q1)
Solve the problems of inaccurate memory prediction and OOM, and improve memory observability by global + thread + task level memory management.
- [ ] #7581
Provides fine-grained IO speed limit, priority scheduling, etc. through global IO management.
- [ ] #7582
Introduces OpenTelemetry to enhance system internal state observability and unify monitoring data format.
## Testing
- [ ] #7583
- [ ] FE
Refine the FE single test framework to support multi-node simulation testing of features.
- [ ] BE
Provide testing framework to simplify the difficulty of writing complex unit tests (e.g. data builds) for BE.
- [ ] #7584
Provide Case collection or submission framework for refining and accumulating regression test sets.
- [ ] #7585
Provide a Benchmark testing framework to ensure that adding new code does not impact performance.
- [ ] #7586
Implement ChaosMesh chaos testing to improve the correctness and stability of the system in case of anomalies.
## Functional Optimization
- [ ] #7149
- [ ] #7149
- [ ] Support vectorization engine Z-Order
- [ ] Agg/Uniq Key model support for Z-Order
- [ ] Schema Change
- [ ] Lateral View
- [ ] Support bitmap, string, json_array expansion (Done & Optimizing)
- [ ] Array type expansion support
- [ ] Table Function
## Deployment and Maintenance
- [ ] #7591 (Q1)
Provides a visual interface for Doris deployment, monitoring, and operations maintenance. Simplifies Doris deployment, scaling, upgrades, task management, status checking, and other operations.
- [ ] #7587 (Good First Issue)
Provides a Helm Charts-based K8S deployment solution.
## Peripheral Ecology
- [ ] #7588
- [ ] #6568 (Done & Optimizing)
- [ ] #7389 (Q1)
- [ ] Hudi
- [ ] Parquet File Format Support
Generate and read Parquet files directly to facilitate Doris data to be read directly by external systems.
- [ ] Data Integration
- [ ] Routine Load support for Canal format parsing (Q1)
- [ ] Flink Connector (Done & Optimizing)
- [ ] Spark Connector (Done & Optimizing)
- [ ] SeaTunnel Sink (Done & Optimizing)
- [ ] SeaTunnel Source
- [ ] DataX (Addax) (Done & Optimizing)
- [ ] Compilation Tools
- [ ] #7590 (Q1)
## Community
- [ ] #7589 (Q1)
Refactored Doris official website to provide best practices, community progress, blog posts, FAQ, and more.
- [ ] Doris Documentation (**Good First Issue**)
**Non-code contributions are as important as code contributions**, and the community is very open to developers improving and proofreading the project documentation.
- [ ] #6336
- [ ] Reorganization of the official Doris documentation to improve readability, operability, and guidance.
- [ ] Translation and proofreading of the English documentation.
- [ ] Github Action (**Good First Issue**)
Introduced more Github Actions to help improve the management of the code base. This includes but is not limited to PR autoresponders, tagging, etc. If you have a good Action to recommend, please leave a comment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] zbtzbtzbt edited a comment on issue #7502: [Discussion] Doris Roadmap 2022
Posted by GitBox <gi...@apache.org>.
zbtzbtzbt edited a comment on issue #7502:
URL: https://github.com/apache/incubator-doris/issues/7502#issuecomment-1001854489
Looking forward to push based pipeline engine @morningman-cmy @yiguolei
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] morningman edited a comment on issue #7502: Doris Roadmap 2022
Posted by GitBox <gi...@apache.org>.
morningman edited a comment on issue #7502:
URL: https://github.com/apache/incubator-doris/issues/7502#issuecomment-1001839293
The following is the Roadmap for the Doris community in 2022.
The plan includes all aspects of code features, documentation, community building, etc. that are to be developed, have already been developed, and have been completed but require ongoing optimization.
> The plan is currently under discussion, so if you have comments or suggestions on any aspect of the plan or beyond, please feel free to leave a comment or send an email to dev@doris.apache.org.
> We will gradually create issues or jira for each direction of the plan to describe and track the progress in detail. Developers who wish to contribute are also welcome to create issues directly and associate with them (just leave a comment)
> The directions marked (**Good First Issue**) in the plan are more independent modules, which are more suitable for newbie tasks or developers who are new to Doris. If you are interested in the relevant direction, please contact us at dev@doris.apache.org or under this issue, and we will provide detailed guidance, help and discussion.
> The directions marked with (**Q1**) are the current work to be completed in the first quarter of 2022. We will update the schedule and progress of other directions gradually.
> The marked (**Done & Optimizing**) directions are the directions that are currently completed but need continuous optimization. Such as ease of use, feature additions, and documentation additions.
> We encourage developers to discuss anything in the dev mailing list, to subscribe to the mailing list please refer to [How to subscribe](http://doris.incubator.apache.org/master/en/community/subscribe-mail-list.html).
## Features
- [ ] #7571
- [ ] Extensible new query optimizer framework
- [ ] #6370
- [ ] Standard test set support and performance enhancements
+ TPC-DS feature pass rate 100%
+ TPC-H performance enhancements
- [ ] #7572
- [ ] Pipeline execution engine
- [ ] Algorithm Concurrency Control and Resource Control
- [ ] #7573
- [ ] #7570
- [ ] Map
- [ ] Struct
- [ ] #7574
Provides Schemaless semantics for fast analysis of semi-structured data.
- [ ] Json
- [ ] #7575 (Q1)
Supports cold data storage to object storage at partition granularity with remote access capabilities and local Cache acceleration.
- [ ] #7503
Doris' current "materialized view" is more of a "materialized index" concept. Doris will later implement a true Materialized View to support full and incremental construction of single and multi-table views.
- [ ] #7576
Provide Kudu-like data update support.
- [ ] #7577
- [ ] WindowFunnel #8299
- [ ] #7578
Support for the new UDF framework has solved the problems of high writing difficulty, poor isolation, and poor compatibility with existing C++ frameworks.
- [x] UDF #7519
- [ ] UDAF #8312
- [ ] UDTF
- [ ] #7579 (**Good First Issue**)
- [ ] #7552
- [ ] #7650
- [ ] Add more resource limits
- [ ] #7129
- [ ] More builtin function support
- [ ] #7678
## Performance Optimization
- [ ] #7580 (Q1)
- [ ] Query layer vectorization
- [ ] Storage level vectorization
- [ ] Vectorization function supplementation
- [ ] Query layer storage layer arithmetic unification
- [ ] Import Vectorization
- [ ] Json Parsing Optimization (**Good First Issue**)
- [ ] #7551
- [ ] #7743
Optimize the performance of compaction task. And try to refactor the compaction logic. For example, only one replica do the compaction and sync to other replicas.
## Stability and Observability
- [ ] #7553 (Q1)
Solve the problems of inaccurate memory prediction and OOM, and improve memory observability by global + thread + task level memory management.
- [ ] #7581
Provides fine-grained IO speed limit, priority scheduling, etc. through global IO management.
- [ ] #7582
Introduces OpenTelemetry to enhance system internal state observability and unify monitoring data format.
## Testing
- [ ] #7583
- [ ] FE
Refine the FE single test framework to support multi-node simulation testing of features.
- [ ] BE
Provide testing framework to simplify the difficulty of writing complex unit tests (e.g. data builds) for BE.
- [ ] #7584
Provide Case collection or submission framework for refining and accumulating regression test sets.
- [ ] #7585
Provide a Benchmark testing framework to ensure that adding new code does not impact performance.
- [ ] #7586
Implement ChaosMesh chaos testing to improve the correctness and stability of the system in case of anomalies.
## Functional Optimization
- [ ] #7149
- [ ] #7149
- [ ] Support vectorization engine Z-Order
- [ ] Agg/Uniq Key model support for Z-Order
- [ ] Schema Change
- [ ] Lateral View
- [ ] Support bitmap, string, json_array expansion (Done & Optimizing)
- [ ] Array type expansion support
- [ ] Table Function
- [ ] Other features
- [ ] #7680
- [ ] CreateTableAsStmt support decimal
## Deployment and Maintenance
- [ ] #7591 (Q1)
Provides a visual interface for Doris deployment, monitoring, and operations maintenance. Simplifies Doris deployment, scaling, upgrades, task management, status checking, and other operations.
- [ ] #7587 (Good First Issue)
Provides a Helm Charts-based K8S deployment solution.
## Peripheral Ecology
- [ ] #7588
- [ ] #6568 (Done & Optimizing)
- [ ] #7389 (Q1)
- [ ] Hudi
- [ ] Parquet File Format Support
Generate and read Parquet files directly to facilitate Doris data to be read directly by external systems.
- [ ] Data Integration
- [ ] Routine Load support for Canal format parsing (Q1)
- [ ] Flink Connector (Done & Optimizing)
- [ ] Spark Connector (Done & Optimizing)
- [ ] SeaTunnel Sink (Done & Optimizing)
- [ ] SeaTunnel Source
- [ ] DataX (Addax) (Done & Optimizing)
- [ ] #7781
- [ ] Compilation Tools
- [ ] #7590 (Q1)
## Community
- [ ] #7589 (Q1)
Refactored Doris official website to provide best practices, community progress, blog posts, FAQ, and more.
- [ ] Doris Documentation (**Good First Issue**)
**Non-code contributions are as important as code contributions**, and the community is very open to developers improving and proofreading the project documentation.
- [ ] #6336
- [ ] Reorganization of the official Doris documentation to improve readability, operability, and guidance.
- [ ] Translation and proofreading of the English documentation.
- [ ] Github Action (**Good First Issue**)
Introduced more Github Actions to help improve the management of the code base. This includes but is not limited to PR autoresponders, tagging, etc. If you have a good Action to recommend, please leave a comment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] i7xh commented on issue #7502: Doris Roadmap 2022
Posted by GitBox <gi...@apache.org>.
i7xh commented on issue #7502:
URL: https://github.com/apache/incubator-doris/issues/7502#issuecomment-1016379365
> Provides Schemaless semantics for fast analysis of semi-structured data
> Json Parsing Optimization
There are a lot of user cases schema-less or semi-structured that lead to support json optimization especially，mainly with decouple change of the schema?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] i7xh commented on issue #7502: Doris Roadmap 2022
Posted by GitBox <gi...@apache.org>.
i7xh commented on issue #7502:
URL: https://github.com/apache/incubator-doris/issues/7502#issuecomment-1015191906
Why Doris need push based query engine?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] morningman edited a comment on issue #7502: Doris Roadmap 2022
Posted by GitBox <gi...@apache.org>.
morningman edited a comment on issue #7502:
URL: https://github.com/apache/incubator-doris/issues/7502#issuecomment-1001839293
The following is the Roadmap for the Doris community in 2022.
The plan includes all aspects of code features, documentation, community building, etc. that are to be developed, have already been developed, and have been completed but require ongoing optimization.
> The plan is currently under discussion, so if you have comments or suggestions on any aspect of the plan or beyond, please feel free to leave a comment or send an email to dev@doris.apache.org.
> We will gradually create issues or jira for each direction of the plan to describe and track the progress in detail. Developers who wish to contribute are also welcome to create issues directly and associate with them (just leave a comment)
> The directions marked (**Good First Issue**) in the plan are more independent modules, which are more suitable for newbie tasks or developers who are new to Doris. If you are interested in the relevant direction, please contact us at dev@doris.apache.org or under this issue, and we will provide detailed guidance, help and discussion.
> The directions marked with (**Q1**) are the current work to be completed in the first quarter of 2022. We will update the schedule and progress of other directions gradually.
> The marked (**Done & Optimizing**) directions are the directions that are currently completed but need continuous optimization. Such as ease of use, feature additions, and documentation additions.
> We encourage developers to discuss anything in the dev mailing list, to subscribe to the mailing list please refer to [How to subscribe](http://doris.incubator.apache.org/master/en/community/subscribe-mail-list.html).
## Features
- [ ] #7571
- [ ] Extensible new query optimizer framework
- [ ] #6370
- [ ] Standard test set support and performance enhancements
+ TPC-DS feature pass rate 100%
+ TPC-H performance enhancements
- [ ] #7572
- [ ] Pipeline execution engine
- [ ] Algorithm Concurrency Control and Resource Control
- [ ] #7573
- [ ] #7570
- [ ] Map
- [ ] Struct
- [ ] #7574
Provides Schemaless semantics for fast analysis of semi-structured data.
- [ ] Json
- [ ] #7575 (Q1)
Supports cold data storage to object storage at partition granularity with remote access capabilities and local Cache acceleration.
- [ ] #7503
Doris' current "materialized view" is more of a "materialized index" concept. Doris will later implement a true Materialized View to support full and incremental construction of single and multi-table views.
- [ ] #7576
Provide Kudu-like data update support.
- [ ] #7577
- [ ] WindowFunnel
- [ ] #7578
Support for the new UDF framework has solved the problems of high writing difficulty, poor isolation, and poor compatibility with existing C++ frameworks.
- [ ] UDF
- [ ] UDAF
- [ ] UDTF
- [ ] #7579 (**Good First Issue**)
- [ ] #7552
- [ ] #7650
- [ ] Add more resource limits
- [ ] #7129
- [ ] More builtin function support
- [ ] #7678
## Performance Optimization
- [ ] #7580 (Q1)
- [ ] Query layer vectorization
- [ ] Storage level vectorization
- [ ] Vectorization function supplementation
- [ ] Query layer storage layer arithmetic unification
- [ ] Import Vectorization
- [ ] Json Parsing Optimization (**Good First Issue**)
- [ ] #7551
- [ ] #7743
Optimize the performance of compaction task. And try to refactor the compaction logic. For example, only one replica do the compaction and sync to other replicas.
## Stability and Observability
- [ ] #7553 (Q1)
Solve the problems of inaccurate memory prediction and OOM, and improve memory observability by global + thread + task level memory management.
- [ ] #7581
Provides fine-grained IO speed limit, priority scheduling, etc. through global IO management.
- [ ] #7582
Introduces OpenTelemetry to enhance system internal state observability and unify monitoring data format.
## Testing
- [ ] #7583
- [ ] FE
Refine the FE single test framework to support multi-node simulation testing of features.
- [ ] BE
Provide testing framework to simplify the difficulty of writing complex unit tests (e.g. data builds) for BE.
- [ ] #7584
Provide Case collection or submission framework for refining and accumulating regression test sets.
- [ ] #7585
Provide a Benchmark testing framework to ensure that adding new code does not impact performance.
- [ ] #7586
Implement ChaosMesh chaos testing to improve the correctness and stability of the system in case of anomalies.
## Functional Optimization
- [ ] #7149
- [ ] #7149
- [ ] Support vectorization engine Z-Order
- [ ] Agg/Uniq Key model support for Z-Order
- [ ] Schema Change
- [ ] Lateral View
- [ ] Support bitmap, string, json_array expansion (Done & Optimizing)
- [ ] Array type expansion support
- [ ] Table Function
- [ ] Other features
- [ ] #7680
- [ ] CreateTableAsStmt support decimal
## Deployment and Maintenance
- [ ] #7591 (Q1)
Provides a visual interface for Doris deployment, monitoring, and operations maintenance. Simplifies Doris deployment, scaling, upgrades, task management, status checking, and other operations.
- [ ] #7587 (Good First Issue)
Provides a Helm Charts-based K8S deployment solution.
## Peripheral Ecology
- [ ] #7588
- [ ] #6568 (Done & Optimizing)
- [ ] #7389 (Q1)
- [ ] Hudi
- [ ] Parquet File Format Support
Generate and read Parquet files directly to facilitate Doris data to be read directly by external systems.
- [ ] Data Integration
- [ ] Routine Load support for Canal format parsing (Q1)
- [ ] Flink Connector (Done & Optimizing)
- [ ] Spark Connector (Done & Optimizing)
- [ ] SeaTunnel Sink (Done & Optimizing)
- [ ] SeaTunnel Source
- [ ] DataX (Addax) (Done & Optimizing)
- [ ] #7781
- [ ] Compilation Tools
- [ ] #7590 (Q1)
## Community
- [ ] #7589 (Q1)
Refactored Doris official website to provide best practices, community progress, blog posts, FAQ, and more.
- [ ] Doris Documentation (**Good First Issue**)
**Non-code contributions are as important as code contributions**, and the community is very open to developers improving and proofreading the project documentation.
- [ ] #6336
- [ ] Reorganization of the official Doris documentation to improve readability, operability, and guidance.
- [ ] Translation and proofreading of the English documentation.
- [ ] Github Action (**Good First Issue**)
Introduced more Github Actions to help improve the management of the code base. This includes but is not limited to PR autoresponders, tagging, etc. If you have a good Action to recommend, please leave a comment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] Henry2SS commented on issue #7502: [Discussion] Doris Roadmap 2022
Posted by GitBox <gi...@apache.org>.
Henry2SS commented on issue #7502:
URL: https://github.com/apache/incubator-doris/issues/7502#issuecomment-1001847083
What about supporting AVRO format in LOAD function?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] morningman commented on issue #7502: [Discussion] Doris Roadmap 2022
Posted by GitBox <gi...@apache.org>.
morningman commented on issue #7502:
URL: https://github.com/apache/incubator-doris/issues/7502#issuecomment-1001839293
For English Version(To be added)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] zbtzbtzbt commented on issue #7502: [Discussion] Doris Roadmap 2022
Posted by GitBox <gi...@apache.org>.
zbtzbtzbt commented on issue #7502:
URL: https://github.com/apache/incubator-doris/issues/7502#issuecomment-1001854489
Looking forward to push based pipeline engine @yiguolei yi
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] morningman commented on issue #7502: Doris Roadmap 2022
Posted by GitBox <gi...@apache.org>.
morningman commented on issue #7502:
URL: https://github.com/apache/incubator-doris/issues/7502#issuecomment-1015074507
> support decimal data type for create table as select statement. (detailMessage = Unsupported type 'DECIMAL(9,0)' in create table as select statement)
Added to the roadmap
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] huligong1234 edited a comment on issue #7502: Doris Roadmap 2022
Posted by GitBox <gi...@apache.org>.
huligong1234 edited a comment on issue #7502:
URL: https://github.com/apache/incubator-doris/issues/7502#issuecomment-1015068122
support decimal data type for create table as select statement. (detailMessage = Unsupported type 'DECIMAL(9,0)' in create table as select statement)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] morningman commented on issue #7502: Doris Roadmap 2022
Posted by GitBox <gi...@apache.org>.
morningman commented on issue #7502:
URL: https://github.com/apache/incubator-doris/issues/7502#issuecomment-1015074419
> #7680 Data export function supports exporting to db, kafka, etc.
> #7678 max_by, min_by aggregate function support
Added to the Roadmap
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] morningman edited a comment on issue #7502: Doris Roadmap 2022
Posted by GitBox <gi...@apache.org>.
morningman edited a comment on issue #7502:
URL: https://github.com/apache/incubator-doris/issues/7502#issuecomment-1001839293
The following is the Roadmap for the Doris community in 2022.
The plan includes all aspects of code features, documentation, community building, etc. that are to be developed, have already been developed, and have been completed but require ongoing optimization.
> The plan is currently under discussion, so if you have comments or suggestions on any aspect of the plan or beyond, please feel free to leave a comment or send an email to dev@doris.apache.org.
> We will gradually create issues or jira for each direction of the plan to describe and track the progress in detail. Developers who wish to contribute are also welcome to create issues directly and associate with them (just leave a comment)
> The directions marked (**Good First Issue**) in the plan are more independent modules, which are more suitable for newbie tasks or developers who are new to Doris. If you are interested in the relevant direction, please contact us at dev@doris.apache.org or under this issue, and we will provide detailed guidance, help and discussion.
> The directions marked with (**Q1**) are the current work to be completed in the first quarter of 2022. We will update the schedule and progress of other directions gradually.
> The marked (**Done & Optimizing**) directions are the directions that are currently completed but need continuous optimization. Such as ease of use, feature additions, and documentation additions.
> We encourage developers to discuss anything in the dev mailing list, to subscribe to the mailing list please refer to [How to subscribe](http://doris.incubator.apache.org/master/en/community/subscribe-mail-list.html).
## Features
- [ ] #7571
- [ ] Extensible new query optimizer framework
- [ ] Statistical information collection and utilization
- [ ] Standard test set support and performance enhancements
+ TPC-DS feature pass rate 100%
+ TPC-H performance enhancements
- [ ] #7572
- [ ] Pipeline execution engine
- [ ] Algorithm Concurrency Control and Resource Control
- [ ] #7573
- [ ] #7570
- [ ] Map
- [ ] Struct
- [ ] #7574
Provides Schemaless semantics for fast analysis of semi-structured data.
- [ ] Json
- [ ] #7575 (Q1)
Supports cold data storage to object storage at partition granularity with remote access capabilities and local Cache acceleration.
- [ ] #7503
Doris' current "materialized view" is more of a "materialized index" concept. Doris will later implement a true Materialized View to support full and incremental construction of single and multi-table views.
- [ ] #7576
Provide Kudu-like data update support.
- [ ] #7577
- [ ] WindowFunnel
- [ ] #7578
Support for the new UDF framework has solved the problems of high writing difficulty, poor isolation, and poor compatibility with existing C++ frameworks.
- [ ] UDF
- [ ] UDAF
- [ ] UDTF
- [ ] #7579 (**Good First Issue**)
- [ ] #7552
- [ ] #7650
- [ ] Add more resource limits
- [ ] #7129
## Performance Optimization
- [ ] #7580 (Q1)
- [ ] Query layer vectorization
- [ ] Storage level vectorization
- [ ] Vectorization function supplementation
- [ ] Query layer storage layer arithmetic unification
- [ ] Import Vectorization
- [ ] Json Parsing Optimization (**Good First Issue**)
- [ ] #7551
- [ ] #7743
Optimize the performance of compaction task. And try to refactor the compaction logic. For example, only one replica do the compaction and sync to other replicas.
## Stability and Observability
- [ ] #7553 (Q1)
Solve the problems of inaccurate memory prediction and OOM, and improve memory observability by global + thread + task level memory management.
- [ ] #7581
Provides fine-grained IO speed limit, priority scheduling, etc. through global IO management.
- [ ] #7582
Introduces OpenTelemetry to enhance system internal state observability and unify monitoring data format.
## Testing
- [ ] #7583
- [ ] FE
Refine the FE single test framework to support multi-node simulation testing of features.
- [ ] BE
Provide testing framework to simplify the difficulty of writing complex unit tests (e.g. data builds) for BE.
- [ ] #7584
Provide Case collection or submission framework for refining and accumulating regression test sets.
- [ ] #7585
Provide a Benchmark testing framework to ensure that adding new code does not impact performance.
- [ ] #7586
Implement ChaosMesh chaos testing to improve the correctness and stability of the system in case of anomalies.
## Functional Optimization
- [ ] #7149
- [ ] #7149
- [ ] Support vectorization engine Z-Order
- [ ] Agg/Uniq Key model support for Z-Order
- [ ] Schema Change
- [ ] Lateral View
- [ ] Support bitmap, string, json_array expansion (Done & Optimizing)
- [ ] Array type expansion support
- [ ] Table Function
## Deployment and Maintenance
- [ ] #7591 (Q1)
Provides a visual interface for Doris deployment, monitoring, and operations maintenance. Simplifies Doris deployment, scaling, upgrades, task management, status checking, and other operations.
- [ ] #7587 (Good First Issue)
Provides a Helm Charts-based K8S deployment solution.
## Peripheral Ecology
- [ ] #7588
- [ ] #6568 (Done & Optimizing)
- [ ] #7389 (Q1)
- [ ] Hudi
- [ ] Parquet File Format Support
Generate and read Parquet files directly to facilitate Doris data to be read directly by external systems.
- [ ] Data Integration
- [ ] Routine Load support for Canal format parsing (Q1)
- [ ] Flink Connector (Done & Optimizing)
- [ ] Spark Connector (Done & Optimizing)
- [ ] SeaTunnel Sink (Done & Optimizing)
- [ ] SeaTunnel Source
- [ ] DataX (Addax) (Done & Optimizing)
- [ ] #7781
- [ ] Compilation Tools
- [ ] #7590 (Q1)
## Community
- [ ] #7589 (Q1)
Refactored Doris official website to provide best practices, community progress, blog posts, FAQ, and more.
- [ ] Doris Documentation (**Good First Issue**)
**Non-code contributions are as important as code contributions**, and the community is very open to developers improving and proofreading the project documentation.
- [ ] #6336
- [ ] Reorganization of the official Doris documentation to improve readability, operability, and guidance.
- [ ] Translation and proofreading of the English documentation.
- [ ] Github Action (**Good First Issue**)
Introduced more Github Actions to help improve the management of the code base. This includes but is not limited to PR autoresponders, tagging, etc. If you have a good Action to recommend, please leave a comment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] huligong1234 commented on issue #7502: Doris Roadmap 2022
Posted by GitBox <gi...@apache.org>.
huligong1234 commented on issue #7502:
URL: https://github.com/apache/incubator-doris/issues/7502#issuecomment-1015068122
looking forward to support decimal data type for create table as select statement. (detailMessage = Unsupported type 'DECIMAL(9,0)' in create table as select statement)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] morningman edited a comment on issue #7502: Doris Roadmap 2022
Posted by GitBox <gi...@apache.org>.
morningman edited a comment on issue #7502:
URL: https://github.com/apache/incubator-doris/issues/7502#issuecomment-1015074419
> #7680 Data export function supports exporting to db, kafka, etc.
> #7678 max_by, min_by aggregate function support
Added to the Roadmap
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] EmmyMiao87 commented on issue #7502: [Discussion] Doris Roadmap 2022
Posted by GitBox <gi...@apache.org>.
EmmyMiao87 commented on issue #7502:
URL: https://github.com/apache/incubator-doris/issues/7502#issuecomment-1001843669
Could you please open an email to discuss Roadmap 2022 of Doris ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] yiguolei commented on issue #7502: [Discussion] Doris Roadmap 2022
Posted by GitBox <gi...@apache.org>.
yiguolei commented on issue #7502:
URL: https://github.com/apache/incubator-doris/issues/7502#issuecomment-1001842543
Clang compile is already on process, see https://github.com/apache/incubator-doris/pull/7451
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org