You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "zzzxl1993 (via GitHub)" <gi...@apache.org> on 2024/10/12 08:37:31 UTC
[PR] [opt](inverted index) Optimize the compression of inverted index position information [doris-thirdparty]
zzzxl1993 opened a new pull request, #242:
URL: https://github.com/apache/doris-thirdparty/pull/242
(no comment)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](inverted index) Optimize the compression of inverted index position information [doris-thirdparty]
Posted by "zzzxl1993 (via GitHub)" <gi...@apache.org>.
zzzxl1993 commented on PR #242:
URL: https://github.com/apache/doris-thirdparty/pull/242#issuecomment-2408459618
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@doris.apache.org
For additional commands, e-mail: dev-help@doris.apache.org
Re: [PR] [opt](inverted index) Optimize the compression of inverted index position information [doris-thirdparty]
Posted by "zzzxl1993 (via GitHub)" <gi...@apache.org>.
zzzxl1993 commented on PR #242:
URL: https://github.com/apache/doris-thirdparty/pull/242#issuecomment-2419371701
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@doris.apache.org
For additional commands, e-mail: dev-help@doris.apache.org
Re: [PR] [opt](inverted index) Optimize the compression of inverted index position information [doris-thirdparty]
Posted by "zzzxl1993 (via GitHub)" <gi...@apache.org>.
zzzxl1993 commented on PR #242:
URL: https://github.com/apache/doris-thirdparty/pull/242#issuecomment-2410476697
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@doris.apache.org
For additional commands, e-mail: dev-help@doris.apache.org
Re: [PR] [opt](inverted index) Optimize the compression of inverted index position information [doris-thirdparty]
Posted by "airborne12 (via GitHub)" <gi...@apache.org>.
airborne12 commented on code in PR #242:
URL: https://github.com/apache/doris-thirdparty/pull/242#discussion_r1797694674
##########
src/core/CLucene/index/_SegmentHeader.h:
##########
@@ -86,6 +87,62 @@ class TermDocsBuffer {
IndexVersion indexVersion_ = IndexVersion::kV0;
};
+class TermPostingsBuffer {
Review Comment:
why we need this buffer? index input already has buffer inside.
##########
src/core/CLucene/index/IndexWriter.cpp:
##########
@@ -1308,20 +1309,36 @@ void IndexWriter::indexCompaction(std::vector<lucene::store::Directory *> &src_d
}
assert(readers.size() == numIndices);
- // check hasProx
+ // check hasProx, indexVersion
bool hasProx = false;
+ IndexVersion indexVersion = IndexVersion::kV1;
{
if (!readers.empty()) {
+ auto release_readers = [this]() {
Review Comment:
why we release reader here?
##########
src/core/CLucene/util/PFORUtil.h:
##########
@@ -18,9 +18,22 @@
#include <cstddef>
#include <cstdint>
+#include <vector>
+
+#include "CLucene.h"
+#include "CLucene/store/IndexOutput.h"
+
+CL_NS_USE(store)
size_t P4DEC(unsigned char *__restrict in, size_t n, uint32_t *__restrict out);
size_t P4NZDEC(unsigned char *__restrict in, size_t n, uint32_t *__restrict out);
size_t P4ENC(uint32_t *__restrict in, size_t n, unsigned char *__restrict out);
size_t P4NZENC(uint32_t *__restrict in, size_t n, unsigned char *__restrict out);
+class PforUtil {
+public:
+ static constexpr size_t blockSize = 128;
Review Comment:
why change block size to 128 from 512?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@doris.apache.org
For additional commands, e-mail: dev-help@doris.apache.org
Re: [PR] [opt](inverted index) Optimize the compression of inverted index position information [doris-thirdparty]
Posted by "airborne12 (via GitHub)" <gi...@apache.org>.
airborne12 merged PR #242:
URL: https://github.com/apache/doris-thirdparty/pull/242
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [opt](inverted index) Optimize the compression of inverted index position information [doris-thirdparty]
Posted by "zzzxl1993 (via GitHub)" <gi...@apache.org>.
zzzxl1993 commented on code in PR #242:
URL: https://github.com/apache/doris-thirdparty/pull/242#discussion_r1799008204
##########
src/core/CLucene/util/PFORUtil.h:
##########
@@ -18,9 +18,22 @@
#include <cstddef>
#include <cstdint>
+#include <vector>
+
+#include "CLucene.h"
+#include "CLucene/store/IndexOutput.h"
+
+CL_NS_USE(store)
size_t P4DEC(unsigned char *__restrict in, size_t n, uint32_t *__restrict out);
size_t P4NZDEC(unsigned char *__restrict in, size_t n, uint32_t *__restrict out);
size_t P4ENC(uint32_t *__restrict in, size_t n, unsigned char *__restrict out);
size_t P4NZENC(uint32_t *__restrict in, size_t n, unsigned char *__restrict out);
+class PforUtil {
+public:
+ static constexpr size_t blockSize = 128;
Review Comment:
blockSize is only used in position compression
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@doris.apache.org
For additional commands, e-mail: dev-help@doris.apache.org
Re: [PR] [opt](inverted index) Optimize the compression of inverted index position information [doris-thirdparty]
Posted by "zzzxl1993 (via GitHub)" <gi...@apache.org>.
zzzxl1993 commented on code in PR #242:
URL: https://github.com/apache/doris-thirdparty/pull/242#discussion_r1799010998
##########
src/core/CLucene/index/IndexWriter.cpp:
##########
@@ -1308,20 +1309,36 @@ void IndexWriter::indexCompaction(std::vector<lucene::store::Directory *> &src_d
}
assert(readers.size() == numIndices);
- // check hasProx
+ // check hasProx, indexVersion
bool hasProx = false;
+ IndexVersion indexVersion = IndexVersion::kV1;
{
if (!readers.empty()) {
+ auto release_readers = [this]() {
Review Comment:
Release memory in the try->catch->finally code
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@doris.apache.org
For additional commands, e-mail: dev-help@doris.apache.org