You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "csun5285 (via GitHub)" <gi...@apache.org> on 2026/01/04 08:03:50 UTC
[PR] [improve](variant) add varaint doc mode [doris-website]
csun5285 opened a new pull request, #3253:
URL: https://github.com/apache/doris-website/pull/3253
## Versions
- [x] dev
- [ ] 4.x
- [ ] 3.x
- [ ] 2.1
## Languages
- [x] Chinese
- [x] English
## Docs Checklist
- [ ] Checked by AI
- [ ] Test Cases Built
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](variant) add varaint doc mode [doris-website]
Posted by "eldenmoon (via GitHub)" <gi...@apache.org>.
eldenmoon commented on code in PR #3253:
URL: https://github.com/apache/doris-website/pull/3253#discussion_r2660170030
##########
i18n/zh-CN/docusaurus-plugin-content-docs/current/sql-manual/basic-element/sql-data-types/semi-structured/VARIANT.md:
##########
@@ -428,16 +452,40 @@ CREATE TABLE example_table (
'path_2' : STRING,
properties(
'variant_max_subcolumns_count' = '2048',
- 'variant_enable_typed_paths_to_sparse' = 'true'
+ 'variant_enable_typed_paths_to_sparse' = 'true',
+ 'variant_sparse_hash_shard_count' = '64'
)
>
);
```
<table>
-<tr><td>属性<br/></td><td>描述<br/></td></tr>
+<tr><td>稀疏列属性<br/></td><td>描述<br/></td></tr>
Review Comment:
把sparseSharding也一起补充一下
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](variant) add varaint doc mode [doris-website]
Posted by "eldenmoon (via GitHub)" <gi...@apache.org>.
eldenmoon merged PR #3253:
URL: https://github.com/apache/doris-website/pull/3253
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](variant) add varaint doc mode [doris-website]
Posted by "eldenmoon (via GitHub)" <gi...@apache.org>.
eldenmoon commented on code in PR #3253:
URL: https://github.com/apache/doris-website/pull/3253#discussion_r2660159096
##########
i18n/zh-CN/docusaurus-plugin-content-docs/current/sql-manual/basic-element/sql-data-types/semi-structured/VARIANT.md:
##########
@@ -391,10 +391,34 @@ SELECT * FROM tbl WHERE v['str'] MATCH 'Doris';
| `VARCHAR` | ✔ | ✔ |
| `JSON` | ✔ | ✔ |
+## 宽列
+
+当导入的数据中包含了大量的不同的 JSON key 时,VARAINT 的子列就会变多,达到一定程度就会出现性能下降问题。为了解决列数过多的问题,VARIANT 提供了两种机制。
+
+1. 稀疏列机制:将所有的子列按照稀疏度排序,不稀疏的子列物化为独立子列,剩余的所有的稀疏的子列共同存放到稀疏列中。例如:设置 VARAINT 的最大子列数为 2048,导入的数据一万行中包含 5000个不同的 JSON key,在写入到磁盘文件上时,会根据每个 JSON key 实际存在的行数排序,行数最多的前2048列会物化成单独子列,剩余的2952列被存放到稀疏列中。
+ - 稀疏列机制引入了 sharding 机制,例如指定 sharding count 为 10,在上述的例子中,磁盘上会存储10个稀疏列,平均每个稀疏列中有 295 个子列。
+
+2. DOC编码机制:讲所有的子列物化为独立子列,并且将原始的 json 作为一个"存储字段"保存下来,用于查询整个 JSON。这个机制和Elasticsearch中的 stored fields 作用类似,在Elasticsearch中,可以把整个 DOC保存下来,用于取回整 DOC。
Review Comment:
缺少对doc编码优劣的描述, 参考下CK的文章和文档
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](variant) add varaint doc mode [doris-website]
Posted by "eldenmoon (via GitHub)" <gi...@apache.org>.
eldenmoon commented on code in PR #3253:
URL: https://github.com/apache/doris-website/pull/3253#discussion_r2660159530
##########
i18n/zh-CN/docusaurus-plugin-content-docs/current/sql-manual/basic-element/sql-data-types/semi-structured/VARIANT.md:
##########
@@ -391,10 +391,34 @@ SELECT * FROM tbl WHERE v['str'] MATCH 'Doris';
| `VARCHAR` | ✔ | ✔ |
| `JSON` | ✔ | ✔ |
+## 宽列
+
+当导入的数据中包含了大量的不同的 JSON key 时,VARAINT 的子列就会变多,达到一定程度就会出现性能下降问题。为了解决列数过多的问题,VARIANT 提供了两种机制。
+
+1. 稀疏列机制:将所有的子列按照稀疏度排序,不稀疏的子列物化为独立子列,剩余的所有的稀疏的子列共同存放到稀疏列中。例如:设置 VARAINT 的最大子列数为 2048,导入的数据一万行中包含 5000个不同的 JSON key,在写入到磁盘文件上时,会根据每个 JSON key 实际存在的行数排序,行数最多的前2048列会物化成单独子列,剩余的2952列被存放到稀疏列中。
+ - 稀疏列机制引入了 sharding 机制,例如指定 sharding count 为 10,在上述的例子中,磁盘上会存储10个稀疏列,平均每个稀疏列中有 295 个子列。
+
+2. DOC编码机制:讲所有的子列物化为独立子列,并且将原始的 json 作为一个"存储字段"保存下来,用于查询整个 JSON。这个机制和Elasticsearch中的 stored fields 作用类似,在Elasticsearch中,可以把整个 DOC保存下来,用于取回整 DOC。
Review Comment:
最好有一个最佳实践,来展示这些配置怎么处理
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](variant) add varaint doc mode [doris-website]
Posted by "eldenmoon (via GitHub)" <gi...@apache.org>.
eldenmoon commented on code in PR #3253:
URL: https://github.com/apache/doris-website/pull/3253#discussion_r2660156294
##########
i18n/zh-CN/docusaurus-plugin-content-docs/current/sql-manual/basic-element/sql-data-types/semi-structured/VARIANT.md:
##########
@@ -391,10 +391,34 @@ SELECT * FROM tbl WHERE v['str'] MATCH 'Doris';
| `VARCHAR` | ✔ | ✔ |
| `JSON` | ✔ | ✔ |
+## 宽列
+
+当导入的数据中包含了大量的不同的 JSON key 时,VARAINT 的子列就会变多,达到一定程度就会出现性能下降问题。为了解决列数过多的问题,VARIANT 提供了两种机制。
+
+1. 稀疏列机制:将所有的子列按照稀疏度排序,不稀疏的子列物化为独立子列,剩余的所有的稀疏的子列共同存放到稀疏列中。例如:设置 VARAINT 的最大子列数为 2048,导入的数据一万行中包含 5000个不同的 JSON key,在写入到磁盘文件上时,会根据每个 JSON key 实际存在的行数排序,行数最多的前2048列会物化成单独子列,剩余的2952列被存放到稀疏列中。
+ - 稀疏列机制引入了 sharding 机制,例如指定 sharding count 为 10,在上述的例子中,磁盘上会存储10个稀疏列,平均每个稀疏列中有 295 个子列。
+
+2. DOC编码机制:讲所有的子列物化为独立子列,并且将原始的 json 作为一个"存储字段"保存下来,用于查询整个 JSON。这个机制和Elasticsearch中的 stored fields 作用类似,在Elasticsearch中,可以把整个 DOC保存下来,用于取回整 DOC。
Review Comment:
typo 『讲』-> 将
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](variant) add varaint doc mode [doris-website]
Posted by "eldenmoon (via GitHub)" <gi...@apache.org>.
eldenmoon commented on code in PR #3253:
URL: https://github.com/apache/doris-website/pull/3253#discussion_r2660171418
##########
i18n/zh-CN/docusaurus-plugin-content-docs/current/sql-manual/basic-element/sql-data-types/semi-structured/VARIANT.md:
##########
@@ -428,16 +452,40 @@ CREATE TABLE example_table (
'path_2' : STRING,
properties(
'variant_max_subcolumns_count' = '2048',
- 'variant_enable_typed_paths_to_sparse' = 'true'
+ 'variant_enable_typed_paths_to_sparse' = 'true',
+ 'variant_sparse_hash_shard_count' = '64'
)
>
);
```
<table>
-<tr><td>属性<br/></td><td>描述<br/></td></tr>
+<tr><td>稀疏列属性<br/></td><td>描述<br/></td></tr>
<tr><td>`variant_max_subcolumns_count`<br/></td><td>控制 Path 物化列数的上限;超过后新增路径可能存放于共享数据结构。默认 0 表示不限,建议设置为 2048;不推荐超过 10000。<br/></td></tr>
<tr><td>`variant_enable_typed_paths_to_sparse`<br/></td><td>默认指定了 Path 类型后,该 Path 一定会被提取(不计入 `variant_max_subcolumns_count`)。设置为 `true` 后也会计入阈值,可能被收敛到共享结构。<br/></td></tr>
+<tr><td>`variant_sparse_hash_shard_count`<br/></td><td>控制稀疏列的分片数量。将稀疏子列分散存储到多个稀疏列中,以以提升查询性能。默认值为 1,建议根据稀疏子列数量适当调整。<br/></td></tr>
+</table>
+
+```sql
+CREATE TABLE example_table (
+ id INT,
+ data_variant VARIANT<
+ 'path_1' : INT,
+ 'path_2' : STRING,
+ properties(
+ 'variant_enable_doc_mode' = 'true',
+ 'variant_doc_materialization_min_rows' = '10000',
+ 'variant_doc_hash_shard_count' = '64'
+ )
+ >
+);
+```
+
+<table>
+<tr><td>DOC 编码属性<br/></td><td>描述<br/></td></tr>
+<tr><td>`variant_enable_doc_mode`<br/></td><td>是否启用 DOC 编码模式。设置为 `true` 时,原始 JSON 会作为存储字段保存,用于快速返回整个 JSON 文档。启用后将无法使用稀疏列机制。默认值为 `false`。<br/></td></tr>
+<tr><td>`variant_doc_materialization_min_rows`<br/></td><td>DOC 编码模式下触发子列物化的最小行数阈值。当写入行数低于该值时,仅存储原始 JSON;当文件合并后行数达到该阈值时,才将子列物化为独立子列。用于减少小批量写入时的开销。<br/></td></tr>
+<tr><td>`variant_doc_hash_shard_count`<br/></td><td>控制 DOC 编码的分片数量。原始 JSON 会被拆散存储到指定数量的列中,查询整个 JSON 时再组装这些分片。默认值为 64,可根据 JSON 大小和并发需求调整。<br/></td></tr>
Review Comment:
可以参考下ck怎么描述的
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improve](variant) add varaint doc mode [doris-website]
Posted by "csun5285 (via GitHub)" <gi...@apache.org>.
csun5285 commented on PR #3253:
URL: https://github.com/apache/doris-website/pull/3253#issuecomment-3707846068
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org