You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@lucene.apache.org by GitBox <gi...@apache.org> on 2020/07/06 13:25:40 UTC
[GitHub] [lucene-solr] mayya-sharipova opened a new pull request #1653: Stored fields
mayya-sharipova opened a new pull request #1653:
URL: https://github.com/apache/lucene-solr/pull/1653
Store schema separately for stored fields
Currently, stored fields for a document are stored in the following
format:
MetadataForField1, DataForField1 | MetadataForField2, DataForField2 ....
This patch changes the format to:
MetadataField1, MetadataField2...|DataField1, DataField2, ...
As metadata are combined together, we hope that this will improve
compression.
Co-authored-by: Colin Goodheart-Smithe <co...@users.noreply.github.com>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene-solr] mayya-sharipova edited a comment on pull request #1653: Stored fields
Posted by GitBox <gi...@apache.org>.
mayya-sharipova edited a comment on pull request #1653:
URL: https://github.com/apache/lucene-solr/pull/1653#issuecomment-654242849
## Benchmarking
### Index size
We have 3 stored fields:
1. id
2. body
3. date
CompressionMode = Fast
Executing command: `find . -name '*.fdt' -exec du -ch {} +`
| Dataset | Size of *.fdt files trunk| Size of *.fdt files patch|
| ------------- | -----:|-----:|
| wikimedium 1M | 606M | 606M |
| wikimedium 10M | 5.5G | 5.5G|
|wikimediumall 33M | 16G | 16G|
The patch doesn't seem to affect the size of the index.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene-solr] mayya-sharipova edited a comment on pull request #1653: Stored fields
Posted by GitBox <gi...@apache.org>.
mayya-sharipova edited a comment on pull request #1653:
URL: https://github.com/apache/lucene-solr/pull/1653#issuecomment-654242849
## Benchmarking
### Index size
We have 3 stored fields:
1. id
2. body
3. date
Executing command: `find . -name '*.fdt' -exec du -ch {} +`
| Dataset | Size of *.fdt files trunk| Size of *.fdt files patch|
| ------------- | -----:|-----:|
| wikimedium 1M | 606M | 606M |
| wikimedium 10M | 5.5G | 5.5G|
|wikimediumall 3326M | | |
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene-solr] mayya-sharipova commented on pull request #1653: Stored fields
Posted by GitBox <gi...@apache.org>.
mayya-sharipova commented on pull request #1653:
URL: https://github.com/apache/lucene-solr/pull/1653#issuecomment-654242849
## Index size
We have 3 stored fields:
1. id
2. body
3. date
Executing command: `find . -name '*.fdt' -exec du -ch {} +`
| Dataset | Size of *.fdt files trunk| Size of *.fdt files patch|
| ------------- | -----:|-----:|
| wikimedium 1M | 606M | 606M |
| wikimedium 10M | 5.5G | 5.5G|
|wikimediumall 3326M | | |
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene-solr] mayya-sharipova edited a comment on pull request #1653: Stored fields
Posted by GitBox <gi...@apache.org>.
mayya-sharipova edited a comment on pull request #1653:
URL: https://github.com/apache/lucene-solr/pull/1653#issuecomment-654242849
## Benchmarking
### Index size
We have 3 stored fields:
1. id
2. body
3. date
Executing command: `find . -name '*.fdt' -exec du -ch {} +`
| Dataset | Size of *.fdt files trunk| Size of *.fdt files patch|
| ------------- | -----:|-----:|
| wikimedium 1M | 606M | 606M |
| wikimedium 10M | 5.5G | 5.5G|
|wikimediumall 33M | 16G | 16G|
The patch doesn't seem to affect the size of the index.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org