You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@lucene.apache.org by "balaramsharma (via GitHub)" <gi...@apache.org> on 2025/01/02 04:22:19 UTC

[I] Inquiry regarding storage reduction by omitting NORMS during indexing [lucene]

balaramsharma opened a new issue, #14093:
URL: https://github.com/apache/lucene/issues/14093

   ### Description
   
   Dear Developers,
   
   I learned that **omitting norms during indexing for a field saves a byte per document in Lucene**. Reference: [https://lucidworks.com/post/scaling-lucene-and-solr/](url) . However, during my testing, I observed varying results in the overall size of the Lucene index (collection of documents) when disabling norms for string fields during indexing.
   
   Here are the configuration details for reference:
   
   **Lucene Version: 5.3.1**
   **Java Version: OpenJDK 17.0.8.1**
   **Indexer Configuration:**
   index.merge_factor: 10
   index.partition_max_doc: 5,000,000
   indexer.commit_interval_sec: 60
   indexer.commit_max_doc: 100,000
   **Merge Policy:** LogByteSizeMergePolicy
   **Test Results:**
   Please take a look at the attached image.
   <img width="887" alt="Screenshot 2025-01-02 at 10 05 48" src="https://github.com/user-attachments/assets/1b6729b9-fb9e-43d6-ad2f-ade0b00dad1a" />
   
   Could you please provide insights or clarify whether this behavior aligns with the expected impact on index size? Additionally, could you explain why the size reduction appears to be unpredictable?
   
   Thank you for your assistance!
   
   
   ### Gradle command to reproduce
   
   _No response_


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [I] Inquiry regarding storage reduction by omitting NORMS during indexing [lucene]

Posted by "jpountz (via GitHub)" <gi...@apache.org>.
jpountz closed issue #14093: Inquiry regarding storage reduction by omitting NORMS during indexing
URL: https://github.com/apache/lucene/issues/14093


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org


Re: [I] Inquiry regarding storage reduction by omitting NORMS during indexing [lucene]

Posted by "jpountz (via GitHub)" <gi...@apache.org>.
jpountz commented on issue #14093:
URL: https://github.com/apache/lucene/issues/14093#issuecomment-2572481305

   I'm closing in favor of the discussion on the dev@lucene.a.o mailing-list.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org