You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "itschrispeck (via GitHub)" <gi...@apache.org> on 2023/08/15 19:27:54 UTC
[GitHub] [pinot] itschrispeck opened a new pull request, #11354: add additional lucene index configs
itschrispeck opened a new pull request, #11354:
URL: https://github.com/apache/pinot/pull/11354
Adds two configs for Lucene backed text index, both can be used to optimize for index writing time
1. luceneUseCompoundFile: [compound file format](https://lucene.apache.org/core/8_1_1/core/org/apache/lucene/index/IndexWriterConfig.html#setUseCompoundFile-boolean-)
2. luceneMaxBufferSizeMB: [max buffer size](https://lucene.apache.org/core/8_1_1/core/org/apache/lucene/index/IndexWriterConfig.html#setRAMBufferSizeMB-double-)
Internal testing showed tuning these can improve realtime -> offline segment build time by ~30%. Some sample settings are shown below, collected by creating identical (minus lucene configs) tables in the same cluster. Build time is visualized via ingestion delay metrics:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] codecov-commenter commented on pull request #11354: add additional lucene index configs
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #11354:
URL: https://github.com/apache/pinot/pull/11354#issuecomment-1679501533
## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/11354?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
> Merging [#11354](https://app.codecov.io/gh/apache/pinot/pull/11354?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) (de3b65b) into [master](https://app.codecov.io/gh/apache/pinot/commit/80d3d48d16dc9ef968fe56bf0885a57eb9b220e9?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) (80d3d48) will **decrease** coverage by `53.84%`.
> Report is 2 commits behind head on master.
> The diff coverage is `0.00%`.
```diff
@@ Coverage Diff @@
## master #11354 +/- ##
=============================================
- Coverage 68.48% 14.64% -53.84%
+ Complexity 6535 494 -6041
=============================================
Files 2233 2159 -74
Lines 119983 116851 -3132
Branches 18193 17751 -442
=============================================
- Hits 82167 17114 -65053
- Misses 31977 98323 +66346
+ Partials 5839 1414 -4425
```
| Flag | Coverage ฮ | |
|---|---|---|
| integration1 | `?` | |
| integration2 | `?` | |
| java-11 | `?` | |
| java-17 | `?` | |
| java-20 | `14.64% <0.00%> (-51.22%)` | :arrow_down: |
| temurin | `14.64% <0.00%> (-53.84%)` | :arrow_down: |
| unittests1 | `?` | |
| unittests2 | `14.64% <0.00%> (-0.03%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files Changed](https://app.codecov.io/gh/apache/pinot/pull/11354?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage ฮ | |
|---|---|---|
| [...me/impl/invertedindex/RealtimeLuceneTextIndex.java](https://app.codecov.io/gh/apache/pinot/pull/11354?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9yZWFsdGltZS9pbXBsL2ludmVydGVkaW5kZXgvUmVhbHRpbWVMdWNlbmVUZXh0SW5kZXguamF2YQ==) | `0.00% <0.00%> (-66.67%)` | :arrow_down: |
| [...ment/creator/impl/text/LuceneTextIndexCreator.java](https://app.codecov.io/gh/apache/pinot/pull/11354?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2NyZWF0b3IvaW1wbC90ZXh0L0x1Y2VuZVRleHRJbmRleENyZWF0b3IuamF2YQ==) | `0.00% <0.00%> (-80.77%)` | :arrow_down: |
| [...cal/segment/index/text/TextIndexConfigBuilder.java](https://app.codecov.io/gh/apache/pinot/pull/11354?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L3RleHQvVGV4dEluZGV4Q29uZmlnQnVpbGRlci5qYXZh) | `0.00% <0.00%> (-90.00%)` | :arrow_down: |
| [...egment/local/segment/index/text/TextIndexType.java](https://app.codecov.io/gh/apache/pinot/pull/11354?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L3RleHQvVGV4dEluZGV4VHlwZS5qYXZh) | `0.00% <0.00%> (-80.00%)` | :arrow_down: |
| [...pache/pinot/segment/spi/index/TextIndexConfig.java](https://app.codecov.io/gh/apache/pinot/pull/11354?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3Qtc2VnbWVudC1zcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3Bpbm90L3NlZ21lbnQvc3BpL2luZGV4L1RleHRJbmRleENvbmZpZy5qYXZh) | `0.00% <0.00%> (-65.31%)` | :arrow_down: |
| [...org/apache/pinot/spi/config/table/FieldConfig.java](https://app.codecov.io/gh/apache/pinot/pull/11354?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3Qtc3BpL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zcGkvY29uZmlnL3RhYmxlL0ZpZWxkQ29uZmlnLmphdmE=) | `0.00% <รธ> (-82.82%)` | :arrow_down: |
... and [1706 files with indirect coverage changes](https://app.codecov.io/gh/apache/pinot/pull/11354/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
:mega: Weโre building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang commented on pull request #11354: add additional lucene index configs
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on PR #11354:
URL: https://github.com/apache/pinot/pull/11354#issuecomment-1684669436
Can you also help update the Pinot doc (this [page](https://docs.pinot.apache.org/basics/indexing/text-search-support)) for the new configs?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang merged pull request #11354: add additional lucene index configs
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang merged PR #11354:
URL: https://github.com/apache/pinot/pull/11354
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang commented on a diff in pull request #11354: add additional lucene index configs
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on code in PR #11354:
URL: https://github.com/apache/pinot/pull/11354#discussion_r1298132140
##########
pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/index/TextIndexConfig.java:
##########
@@ -99,6 +123,8 @@ public static abstract class AbstractBuilder {
protected boolean _useANDForMultiTermQueries = true;
protected List<String> _stopWordsInclude = new ArrayList<>();
protected List<String> _stopWordsExclude = new ArrayList<>();
+ protected boolean _luceneUseCompoundFile = true;
Review Comment:
```suggestion
protected boolean _luceneUseCompoundFile = LUCENE_INDEX_DEFAULT_USE_COMPOUND_FILE;
```
##########
pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/index/TextIndexConfig.java:
##########
@@ -32,15 +32,20 @@
public class TextIndexConfig extends IndexConfig {
- public static final TextIndexConfig DISABLED = new TextIndexConfig(true, null, null, false, false,
- Collections.emptyList(), Collections.emptyList());
+ private static final int LUCENE_INDEX_DEFAULT_MAX_BUFFER_SIZE_MB = 500;
Review Comment:
Let's also add a constant for:
```
private static final int LUCENE_INDEX_DEFAULT_USE_COMPOUND_FILE = true;
```
##########
pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/index/TextIndexConfig.java:
##########
@@ -50,14 +55,19 @@ public TextIndexConfig(
@JsonProperty("queryCache") boolean enableQueryCache,
@JsonProperty("useANDForMultiTermQueries") boolean useANDForMultiTermQueries,
@JsonProperty("stopWordsInclude") List<String> stopWordsInclude,
- @JsonProperty("stopWordsExclude") List<String> stopWordsExclude) {
+ @JsonProperty("stopWordsExclude") List<String> stopWordsExclude,
+ @JsonProperty("luceneUseCompoundFile") Boolean luceneUseCompoundFile,
+ @JsonProperty("luceneMaxBufferSizeMB") Integer luceneMaxBufferSizeMB) {
super(disabled);
_fstType = fstType;
_rawValueForTextIndex = rawValueForTextIndex;
_enableQueryCache = enableQueryCache;
_useANDForMultiTermQueries = useANDForMultiTermQueries;
_stopWordsInclude = stopWordsInclude;
_stopWordsExclude = stopWordsExclude;
+ _luceneUseCompoundFile = luceneUseCompoundFile == null || luceneUseCompoundFile; // default to useCompoundFile
Review Comment:
```suggestion
_luceneUseCompoundFile = luceneUseCompoundFile == null ? LUCENE_INDEX_DEFAULT_USE_COMPOUND_FILE : luceneUseCompoundFile;
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] itschrispeck commented on a diff in pull request #11354: add additional lucene index configs
Posted by "itschrispeck (via GitHub)" <gi...@apache.org>.
itschrispeck commented on code in PR #11354:
URL: https://github.com/apache/pinot/pull/11354#discussion_r1298837945
##########
pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/index/TextIndexConfig.java:
##########
@@ -32,15 +32,20 @@
public class TextIndexConfig extends IndexConfig {
- public static final TextIndexConfig DISABLED = new TextIndexConfig(true, null, null, false, false,
- Collections.emptyList(), Collections.emptyList());
+ private static final int LUCENE_INDEX_DEFAULT_MAX_BUFFER_SIZE_MB = 500;
Review Comment:
๐ thanks for the review!
##########
pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/index/TextIndexConfig.java:
##########
@@ -32,15 +32,20 @@
public class TextIndexConfig extends IndexConfig {
- public static final TextIndexConfig DISABLED = new TextIndexConfig(true, null, null, false, false,
- Collections.emptyList(), Collections.emptyList());
+ private static final int LUCENE_INDEX_DEFAULT_MAX_BUFFER_SIZE_MB = 500;
Review Comment:
๐ thanks for the review!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org