You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "zzzxl1993 (via GitHub)" <gi...@apache.org> on 2024/09/23 03:08:08 UTC
[PR] [opt](inverted index) optimize the space usage of the inverted index dictionary file and position information [doris]
zzzxl1993 opened a new pull request, #41114:
URL: https://github.com/apache/doris/pull/41114
## Proposed changes
Issue Number: close #xxx
<!--Describe your changes.-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](inverted index) optimize the space usage of the inverted index dictionary file and position information [doris]
Posted by "zzzxl1993 (via GitHub)" <gi...@apache.org>.
zzzxl1993 commented on PR #41114:
URL: https://github.com/apache/doris/pull/41114#issuecomment-2370558501
@Unalian
1. Earlier versions of Doris only applied the PFOR compression algorithm to the inverted lists of the inverted index. We plan to use the PFOR algorithm for position information and Zstd compression for dictionary information, which will further reduce the space usage of Doris indexes.
2. Doris only utilizes column storage and inverted indexes, resulting in less space usage compared to Elasticsearch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](inverted index) optimize the space usage of the inverted index dictionary file and position information [doris]
Posted by "Unalian (via GitHub)" <gi...@apache.org>.
Unalian commented on PR #41114:
URL: https://github.com/apache/doris/pull/41114#issuecomment-2370379422
Has Doris's inverted index been compressed before? Why is it smaller than the inverted index of Lucene/Clucene? Thank you for your ideas.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](inverted index) optimize the space usage of the inverted index dictionary file and position information [doris]
Posted by "zzzxl1993 (via GitHub)" <gi...@apache.org>.
zzzxl1993 commented on PR #41114:
URL: https://github.com/apache/doris/pull/41114#issuecomment-2370811511
@Unalian
1. Did you use the official Clucene for your tests?
2. Doris has implemented some optimizations and modifications to Clucene. You can refer to this codebase: https://github.com/apache/doris-thirdparty/tree/clucene and the related pull request: https://github.com/apache/doris-thirdparty/pull/238.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](inverted index) optimize the space usage of the inverted index dictionary file and position information [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #41114:
URL: https://github.com/apache/doris/pull/41114#issuecomment-2367156652
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR)
Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website).
See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](inverted index) optimize the space usage of the inverted index dictionary file and position information [doris]
Posted by "Unalian (via GitHub)" <gi...@apache.org>.
Unalian commented on PR #41114:
URL: https://github.com/apache/doris/pull/41114#issuecomment-2371017217
> @Unalian
>
> 1. Did you use the official Clucene for your tests?
> 2. Doris has implemented some optimizations and modifications to Clucene. You can refer to this codebase: https://github.com/apache/doris-thirdparty/tree/clucene and the related pull request: [[opt](inverted index) optimize the space usage of the inverted index dictionary file and position information doris-thirdparty#238](https://github.com/apache/doris-thirdparty/pull/238).
1. I used this version: git://clucene.git.sourceforge.net/gitroot/clucene/clucene; And I set simple analyzer; Set writer config: SOTRE_NO; INDEX_NONORMS; INDEX_TOKENIZED; Add I set the config to make sure there is only one segment.
2. Thank you! I am reading the code here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](inverted index) optimize the space usage of the inverted index dictionary file and position information [doris]
Posted by "Unalian (via GitHub)" <gi...@apache.org>.
Unalian commented on PR #41114:
URL: https://github.com/apache/doris/pull/41114#issuecomment-2370747500
> @Unalian
>
> 1. Earlier versions of Doris only applied the PFOR compression algorithm to the inverted lists of the inverted index. We plan to use the PFOR algorithm for position information and Zstd compression for dictionary information, which will further reduce the space usage of Doris indexes.
> 2. Doris only utilizes column storage and inverted indexes, resulting in less space usage compared to Elasticsearch.
Thank you for answering!
1. I see PFOR compression on .frq files from codes. However, .frq (and as your plan, .prx file )files are always tiny compared with .tis(see from clucene). Does PFOR compression algorithm on .frq/.prx have a big impact on the inverted index size?
<img width="263" alt="image" src="https://github.com/user-attachments/assets/b0408c10-20d0-4e02-ac65-94cd30dd6938">
Or is there a big difference between the inverted index files' sizes generated by Clucene and that generated by Doris?
2. If we ignore all the data and only consider the size of the inverted index, I use Lucene to build an inverted index for the same data, and Doris uses less space. This makes me curious.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](inverted index) Optimize the compression of inverted index position information [doris]
Posted by "zzzxl1993 (via GitHub)" <gi...@apache.org>.
zzzxl1993 closed pull request #41114: [opt](inverted index) Optimize the compression of inverted index position information
URL: https://github.com/apache/doris/pull/41114
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org