You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-dev@lucene.apache.org by "Siddharth Gargate (JIRA)" <ji...@apache.org> on 2009/05/07 13:19:30 UTC
[jira] Created: (SOLR-1150) OutofMemoryError on enabling
highlighting
OutofMemoryError on enabling highlighting
-----------------------------------------
Key: SOLR-1150
URL: https://issues.apache.org/jira/browse/SOLR-1150
Project: Solr
Issue Type: Bug
Components: highlighter
Affects Versions: 1.4
Reporter: Siddharth Gargate
Fix For: 1.4
Please refer to following mail thread
http://markmail.org/message/5nhkm5h3ongqlput
I am testing with 2MB document size and just 500 documents. Indexing is working fine even with 128MB heap size. But on searching Solr throws OOM error. This issue is observed only when we enable highlighting. While indexing I am storing 1 MB text. While searching Solr reads all the 500 documents in the memory. It also reads the complete 1 MB stored field in the memory for all 500 documents. Due to this 500 docs * 1 MB * 2 (2 bytes per char) = 1000 MB memory is required for searching.
This memory usage can be reduced by reading one document at a time.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (SOLR-1150) OutofMemoryError on enabling
highlighting
Posted by "Yonik Seeley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-1150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12719599#action_12719599 ]
Yonik Seeley commented on SOLR-1150:
------------------------------------
It's trying to read the loop iterator (i.e. 0-9) ;-)
> OutofMemoryError on enabling highlighting
> -----------------------------------------
>
> Key: SOLR-1150
> URL: https://issues.apache.org/jira/browse/SOLR-1150
> Project: Solr
> Issue Type: Improvement
> Components: highlighter
> Affects Versions: 1.4
> Reporter: Siddharth Gargate
> Assignee: Mark Miller
> Fix For: 1.4
>
> Attachments: SOLR-1150.patch
>
>
> Please refer to following mail thread
> http://markmail.org/message/5nhkm5h3ongqlput
> I am testing with 2MB document size and just 500 documents. Indexing is working fine even with 128MB heap size. But on searching Solr throws OOM error. This issue is observed only when we enable highlighting. While indexing I am storing 1 MB text. While searching Solr reads all the 500 documents in the memory. It also reads the complete 1 MB stored field in the memory for all 500 documents. Due to this 500 docs * 1 MB * 2 (2 bytes per char) = 1000 MB memory is required for searching.
> This memory usage can be reduced by reading one document at a time.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (SOLR-1150) OutofMemoryError on enabling
highlighting
Posted by "Mark Miller (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-1150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12719598#action_12719598 ]
Mark Miller commented on SOLR-1150:
-----------------------------------
There is a problem with distrib and highlighting as well.
> OutofMemoryError on enabling highlighting
> -----------------------------------------
>
> Key: SOLR-1150
> URL: https://issues.apache.org/jira/browse/SOLR-1150
> Project: Solr
> Issue Type: Improvement
> Components: highlighter
> Affects Versions: 1.4
> Reporter: Siddharth Gargate
> Assignee: Mark Miller
> Fix For: 1.4
>
> Attachments: SOLR-1150.patch
>
>
> Please refer to following mail thread
> http://markmail.org/message/5nhkm5h3ongqlput
> I am testing with 2MB document size and just 500 documents. Indexing is working fine even with 128MB heap size. But on searching Solr throws OOM error. This issue is observed only when we enable highlighting. While indexing I am storing 1 MB text. While searching Solr reads all the 500 documents in the memory. It also reads the complete 1 MB stored field in the memory for all 500 documents. Due to this 500 docs * 1 MB * 2 (2 bytes per char) = 1000 MB memory is required for searching.
> This memory usage can be reduced by reading one document at a time.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (SOLR-1150) OutofMemoryError on enabling
highlighting
Posted by "Mark Miller (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-1150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mark Miller updated SOLR-1150:
------------------------------
Attachment: SOLR-1150.patch
> OutofMemoryError on enabling highlighting
> -----------------------------------------
>
> Key: SOLR-1150
> URL: https://issues.apache.org/jira/browse/SOLR-1150
> Project: Solr
> Issue Type: Improvement
> Components: highlighter
> Affects Versions: 1.4
> Reporter: Siddharth Gargate
> Assignee: Mark Miller
> Fix For: 1.4
>
> Attachments: SOLR-1150.patch, SOLR-1150.patch
>
>
> Please refer to following mail thread
> http://markmail.org/message/5nhkm5h3ongqlput
> I am testing with 2MB document size and just 500 documents. Indexing is working fine even with 128MB heap size. But on searching Solr throws OOM error. This issue is observed only when we enable highlighting. While indexing I am storing 1 MB text. While searching Solr reads all the 500 documents in the memory. It also reads the complete 1 MB stored field in the memory for all 500 documents. Due to this 500 docs * 1 MB * 2 (2 bytes per char) = 1000 MB memory is required for searching.
> This memory usage can be reduced by reading one document at a time.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (SOLR-1150) OutofMemoryError on enabling
highlighting
Posted by "Siddharth Gargate (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-1150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Siddharth Gargate updated SOLR-1150:
------------------------------------
Attachment: SOLR-1150.patch
I have attached the patch file with previous suggested code changes..
> OutofMemoryError on enabling highlighting
> -----------------------------------------
>
> Key: SOLR-1150
> URL: https://issues.apache.org/jira/browse/SOLR-1150
> Project: Solr
> Issue Type: Improvement
> Components: highlighter
> Affects Versions: 1.4
> Reporter: Siddharth Gargate
> Fix For: 1.4
>
> Attachments: SOLR-1150.patch
>
>
> Please refer to following mail thread
> http://markmail.org/message/5nhkm5h3ongqlput
> I am testing with 2MB document size and just 500 documents. Indexing is working fine even with 128MB heap size. But on searching Solr throws OOM error. This issue is observed only when we enable highlighting. While indexing I am storing 1 MB text. While searching Solr reads all the 500 documents in the memory. It also reads the complete 1 MB stored field in the memory for all 500 documents. Due to this 500 docs * 1 MB * 2 (2 bytes per char) = 1000 MB memory is required for searching.
> This memory usage can be reduced by reading one document at a time.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (SOLR-1150) OutofMemoryError on enabling
highlighting
Posted by "Mark Miller (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-1150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12711582#action_12711582 ]
Mark Miller commented on SOLR-1150:
-----------------------------------
Patch looks right - my only concern would be a performance impact, but that does look unlikely. There is likely to be some benefit in loading them all at once, but I can't imagine one at a time is much of a loss.
> OutofMemoryError on enabling highlighting
> -----------------------------------------
>
> Key: SOLR-1150
> URL: https://issues.apache.org/jira/browse/SOLR-1150
> Project: Solr
> Issue Type: Improvement
> Components: highlighter
> Affects Versions: 1.4
> Reporter: Siddharth Gargate
> Fix For: 1.4
>
> Attachments: SOLR-1150.patch
>
>
> Please refer to following mail thread
> http://markmail.org/message/5nhkm5h3ongqlput
> I am testing with 2MB document size and just 500 documents. Indexing is working fine even with 128MB heap size. But on searching Solr throws OOM error. This issue is observed only when we enable highlighting. While indexing I am storing 1 MB text. While searching Solr reads all the 500 documents in the memory. It also reads the complete 1 MB stored field in the memory for all 500 documents. Due to this 500 docs * 1 MB * 2 (2 bytes per char) = 1000 MB memory is required for searching.
> This memory usage can be reduced by reading one document at a time.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (SOLR-1150) OutofMemoryError on enabling
highlighting
Posted by "Mark Miller (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-1150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mark Miller reassigned SOLR-1150:
---------------------------------
Assignee: Mark Miller
> OutofMemoryError on enabling highlighting
> -----------------------------------------
>
> Key: SOLR-1150
> URL: https://issues.apache.org/jira/browse/SOLR-1150
> Project: Solr
> Issue Type: Improvement
> Components: highlighter
> Affects Versions: 1.4
> Reporter: Siddharth Gargate
> Assignee: Mark Miller
> Fix For: 1.4
>
> Attachments: SOLR-1150.patch
>
>
> Please refer to following mail thread
> http://markmail.org/message/5nhkm5h3ongqlput
> I am testing with 2MB document size and just 500 documents. Indexing is working fine even with 128MB heap size. But on searching Solr throws OOM error. This issue is observed only when we enable highlighting. While indexing I am storing 1 MB text. While searching Solr reads all the 500 documents in the memory. It also reads the complete 1 MB stored field in the memory for all 500 documents. Due to this 500 docs * 1 MB * 2 (2 bytes per char) = 1000 MB memory is required for searching.
> This memory usage can be reduced by reading one document at a time.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (SOLR-1150) OutofMemoryError on enabling
highlighting
Posted by "Mark Miller (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-1150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12719601#action_12719601 ]
Mark Miller commented on SOLR-1150:
-----------------------------------
ah, thanks for the spot yonik.
Ill switch it to use doc ids instead and see how things go :)
> OutofMemoryError on enabling highlighting
> -----------------------------------------
>
> Key: SOLR-1150
> URL: https://issues.apache.org/jira/browse/SOLR-1150
> Project: Solr
> Issue Type: Improvement
> Components: highlighter
> Affects Versions: 1.4
> Reporter: Siddharth Gargate
> Assignee: Mark Miller
> Fix For: 1.4
>
> Attachments: SOLR-1150.patch
>
>
> Please refer to following mail thread
> http://markmail.org/message/5nhkm5h3ongqlput
> I am testing with 2MB document size and just 500 documents. Indexing is working fine even with 128MB heap size. But on searching Solr throws OOM error. This issue is observed only when we enable highlighting. While indexing I am storing 1 MB text. While searching Solr reads all the 500 documents in the memory. It also reads the complete 1 MB stored field in the memory for all 500 documents. Due to this 500 docs * 1 MB * 2 (2 bytes per char) = 1000 MB memory is required for searching.
> This memory usage can be reduced by reading one document at a time.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (SOLR-1150) OutofMemoryError on enabling
highlighting
Posted by "Siddharth Gargate (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-1150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12711952#action_12711952 ]
Siddharth Gargate commented on SOLR-1150:
-----------------------------------------
Thanks Mark.
SolrIndexSearcher.readDocs method internally reads one doc at a time. So there shouldn't be any performance loss.
> OutofMemoryError on enabling highlighting
> -----------------------------------------
>
> Key: SOLR-1150
> URL: https://issues.apache.org/jira/browse/SOLR-1150
> Project: Solr
> Issue Type: Improvement
> Components: highlighter
> Affects Versions: 1.4
> Reporter: Siddharth Gargate
> Fix For: 1.4
>
> Attachments: SOLR-1150.patch
>
>
> Please refer to following mail thread
> http://markmail.org/message/5nhkm5h3ongqlput
> I am testing with 2MB document size and just 500 documents. Indexing is working fine even with 128MB heap size. But on searching Solr throws OOM error. This issue is observed only when we enable highlighting. While indexing I am storing 1 MB text. While searching Solr reads all the 500 documents in the memory. It also reads the complete 1 MB stored field in the memory for all 500 documents. Due to this 500 docs * 1 MB * 2 (2 bytes per char) = 1000 MB memory is required for searching.
> This memory usage can be reduced by reading one document at a time.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (SOLR-1150) OutofMemoryError on enabling
highlighting
Posted by "Siddharth Gargate (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-1150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Siddharth Gargate updated SOLR-1150:
------------------------------------
Issue Type: Improvement (was: Bug)
> OutofMemoryError on enabling highlighting
> -----------------------------------------
>
> Key: SOLR-1150
> URL: https://issues.apache.org/jira/browse/SOLR-1150
> Project: Solr
> Issue Type: Improvement
> Components: highlighter
> Affects Versions: 1.4
> Reporter: Siddharth Gargate
> Fix For: 1.4
>
>
> Please refer to following mail thread
> http://markmail.org/message/5nhkm5h3ongqlput
> I am testing with 2MB document size and just 500 documents. Indexing is working fine even with 128MB heap size. But on searching Solr throws OOM error. This issue is observed only when we enable highlighting. While indexing I am storing 1 MB text. While searching Solr reads all the 500 documents in the memory. It also reads the complete 1 MB stored field in the memory for all 500 documents. Due to this 500 docs * 1 MB * 2 (2 bytes per char) = 1000 MB memory is required for searching.
> This memory usage can be reduced by reading one document at a time.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (SOLR-1150) OutofMemoryError on enabling
highlighting
Posted by "Mark Miller (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-1150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12719597#action_12719597 ]
Mark Miller commented on SOLR-1150:
-----------------------------------
Odd - change looks like it wouldnt affect this, but somehow the highlighter test fails as it attempts to access a deleted doc. Not quite sure what is up yet.
> OutofMemoryError on enabling highlighting
> -----------------------------------------
>
> Key: SOLR-1150
> URL: https://issues.apache.org/jira/browse/SOLR-1150
> Project: Solr
> Issue Type: Improvement
> Components: highlighter
> Affects Versions: 1.4
> Reporter: Siddharth Gargate
> Assignee: Mark Miller
> Fix For: 1.4
>
> Attachments: SOLR-1150.patch
>
>
> Please refer to following mail thread
> http://markmail.org/message/5nhkm5h3ongqlput
> I am testing with 2MB document size and just 500 documents. Indexing is working fine even with 128MB heap size. But on searching Solr throws OOM error. This issue is observed only when we enable highlighting. While indexing I am storing 1 MB text. While searching Solr reads all the 500 documents in the memory. It also reads the complete 1 MB stored field in the memory for all 500 documents. Due to this 500 docs * 1 MB * 2 (2 bytes per char) = 1000 MB memory is required for searching.
> This memory usage can be reduced by reading one document at a time.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (SOLR-1150) OutofMemoryError on enabling
highlighting
Posted by "Mark Miller (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-1150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mark Miller resolved SOLR-1150.
-------------------------------
Resolution: Fixed
Thanks Siddharth!
> OutofMemoryError on enabling highlighting
> -----------------------------------------
>
> Key: SOLR-1150
> URL: https://issues.apache.org/jira/browse/SOLR-1150
> Project: Solr
> Issue Type: Improvement
> Components: highlighter
> Affects Versions: 1.4
> Reporter: Siddharth Gargate
> Assignee: Mark Miller
> Fix For: 1.4
>
> Attachments: SOLR-1150.patch, SOLR-1150.patch
>
>
> Please refer to following mail thread
> http://markmail.org/message/5nhkm5h3ongqlput
> I am testing with 2MB document size and just 500 documents. Indexing is working fine even with 128MB heap size. But on searching Solr throws OOM error. This issue is observed only when we enable highlighting. While indexing I am storing 1 MB text. While searching Solr reads all the 500 documents in the memory. It also reads the complete 1 MB stored field in the memory for all 500 documents. Due to this 500 docs * 1 MB * 2 (2 bytes per char) = 1000 MB memory is required for searching.
> This memory usage can be reduced by reading one document at a time.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.