You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Alexander Kanarsky (JIRA)" <ji...@apache.org> on 2010/08/03 04:48:16 UTC
[jira] Created: (LUCENE-2584) Concurrency issues in
SegmentInfo.files() could lead to ConcurrentModificationException
Concurrency issues in SegmentInfo.files() could lead to ConcurrentModificationException
---------------------------------------------------------------------------------------
Key: LUCENE-2584
URL: https://issues.apache.org/jira/browse/LUCENE-2584
Project: Lucene - Java
Issue Type: Bug
Affects Versions: 3.0.2, 3.0.1, 3.0, 2.9.3, 2.9.2, 2.9.1, 2.9
Reporter: Alexander Kanarsky
Priority: Minor
The multi-threaded call of the files() in SegmentInfo could lead to the ConcurrentModificationException if one thread is not finished additions to the ArrayList (files) yet while the other thread already obtained it as cached (see below). This is a rare exception, but it would be nice to fix. I see the code is no longer problematic in the trunk (and others ported from flex_1458), looks it was fixed while implementing post 3.x features. The fix to 3.x and 2.9.x branches could be the same - create the files set first and populate it, and then assign to the member variable at the end of the method. This will resolve the issue. I could prepare the patch for 2.9.4 and 3.x, if needed.
--
INFO: [19] webapp= path=/replication params={command=fetchindex&wt=javabin} status=0 QTime=1
Jul 30, 2010 9:13:05 AM org.apache.solr.core.SolrCore execute
INFO: [19] webapp= path=/replication params={command=details&wt=javabin} status=0 QTime=24
Jul 30, 2010 9:13:05 AM org.apache.solr.handler.ReplicationHandler doFetch
SEVERE: SnapPull failed
java.util.ConcurrentModificationException
at java.util.AbstractList$Itr.checkForComodification(AbstractList.java:372)
at java.util.AbstractList$Itr.next(AbstractList.java:343)
at java.util.AbstractCollection.addAll(AbstractCollection.java:305)
at org.apache.lucene.index.SegmentInfos.files(SegmentInfos.java:826)
at org.apache.lucene.index.DirectoryReader$ReaderCommit.<init>(DirectoryReader.java:916)
at org.apache.lucene.index.DirectoryReader.getIndexCommit(DirectoryReader.java:856)
at org.apache.solr.search.SolrIndexReader.getIndexCommit(SolrIndexReader.java:454)
at org.apache.solr.handler.SnapPuller.fetchLatestIndex(SnapPuller.java:261)
at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:264)
at org.apache.solr.handler.ReplicationHandler$1.run(ReplicationHandler.java:146)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Updated: (LUCENE-2584) Concurrency issues in
SegmentInfo.files() could lead to ConcurrentModificationException
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Michael McCandless updated LUCENE-2584:
---------------------------------------
Fix Version/s: 3.1
4.0
> Concurrency issues in SegmentInfo.files() could lead to ConcurrentModificationException
> ---------------------------------------------------------------------------------------
>
> Key: LUCENE-2584
> URL: https://issues.apache.org/jira/browse/LUCENE-2584
> Project: Lucene - Java
> Issue Type: Bug
> Affects Versions: 2.9, 2.9.1, 2.9.2, 2.9.3, 3.0, 3.0.1, 3.0.2
> Reporter: Alexander Kanarsky
> Priority: Minor
> Fix For: 3.1, 4.0
>
>
> The multi-threaded call of the files() in SegmentInfo could lead to the ConcurrentModificationException if one thread is not finished additions to the ArrayList (files) yet while the other thread already obtained it as cached (see below). This is a rare exception, but it would be nice to fix. I see the code is no longer problematic in the trunk (and others ported from flex_1458), looks it was fixed while implementing post 3.x features. The fix to 3.x and 2.9.x branches could be the same - create the files set first and populate it, and then assign to the member variable at the end of the method. This will resolve the issue. I could prepare the patch for 2.9.4 and 3.x, if needed.
> --
> INFO: [19] webapp= path=/replication params={command=fetchindex&wt=javabin} status=0 QTime=1
> Jul 30, 2010 9:13:05 AM org.apache.solr.core.SolrCore execute
> INFO: [19] webapp= path=/replication params={command=details&wt=javabin} status=0 QTime=24
> Jul 30, 2010 9:13:05 AM org.apache.solr.handler.ReplicationHandler doFetch
> SEVERE: SnapPull failed
> java.util.ConcurrentModificationException
> at java.util.AbstractList$Itr.checkForComodification(AbstractList.java:372)
> at java.util.AbstractList$Itr.next(AbstractList.java:343)
> at java.util.AbstractCollection.addAll(AbstractCollection.java:305)
> at org.apache.lucene.index.SegmentInfos.files(SegmentInfos.java:826)
> at org.apache.lucene.index.DirectoryReader$ReaderCommit.<init>(DirectoryReader.java:916)
> at org.apache.lucene.index.DirectoryReader.getIndexCommit(DirectoryReader.java:856)
> at org.apache.solr.search.SolrIndexReader.getIndexCommit(SolrIndexReader.java:454)
> at org.apache.solr.handler.SnapPuller.fetchLatestIndex(SnapPuller.java:261)
> at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:264)
> at org.apache.solr.handler.ReplicationHandler$1.run(ReplicationHandler.java:146)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Updated: (LUCENE-2584) Concurrency issues in
SegmentInfo.files() could lead to ConcurrentModificationException
Posted by "Alexander Kanarsky (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alexander Kanarsky updated LUCENE-2584:
---------------------------------------
Attachment: LUCENE-2584-branch_3x.patch
LUCENE-2584-lucene-2_9.patch
LUCENE-2584-lucene-3_0.patch
Mike, the patches are attached. One note, I noticed (in flex_1458/trunk) that you are using the HashSet to collect the file names and then convert it to ArrayList; while this is a good idea to guarantee the uniqueness of the file names, it also could mask any code that accidentally adds the same file more than once (something that I would prefer to notice rather than mask). So I used an assertion to ensure both the uniqueness of the names and correctness of the code that adds names. Also, with assertions disabled, this approach adds no additional performance overhead at all. But if you'd like to use the HashSet to collect the file names, let me know so I will redo the patches.
> Concurrency issues in SegmentInfo.files() could lead to ConcurrentModificationException
> ---------------------------------------------------------------------------------------
>
> Key: LUCENE-2584
> URL: https://issues.apache.org/jira/browse/LUCENE-2584
> Project: Lucene - Java
> Issue Type: Bug
> Affects Versions: 2.9, 2.9.1, 2.9.2, 2.9.3, 3.0, 3.0.1, 3.0.2
> Reporter: Alexander Kanarsky
> Priority: Minor
> Fix For: 3.1, 4.0
>
> Attachments: LUCENE-2584-branch_3x.patch, LUCENE-2584-lucene-2_9.patch, LUCENE-2584-lucene-3_0.patch
>
>
> The multi-threaded call of the files() in SegmentInfo could lead to the ConcurrentModificationException if one thread is not finished additions to the ArrayList (files) yet while the other thread already obtained it as cached (see below). This is a rare exception, but it would be nice to fix. I see the code is no longer problematic in the trunk (and others ported from flex_1458), looks it was fixed while implementing post 3.x features. The fix to 3.x and 2.9.x branches could be the same - create the files set first and populate it, and then assign to the member variable at the end of the method. This will resolve the issue. I could prepare the patch for 2.9.4 and 3.x, if needed.
> --
> INFO: [19] webapp= path=/replication params={command=fetchindex&wt=javabin} status=0 QTime=1
> Jul 30, 2010 9:13:05 AM org.apache.solr.core.SolrCore execute
> INFO: [19] webapp= path=/replication params={command=details&wt=javabin} status=0 QTime=24
> Jul 30, 2010 9:13:05 AM org.apache.solr.handler.ReplicationHandler doFetch
> SEVERE: SnapPull failed
> java.util.ConcurrentModificationException
> at java.util.AbstractList$Itr.checkForComodification(AbstractList.java:372)
> at java.util.AbstractList$Itr.next(AbstractList.java:343)
> at java.util.AbstractCollection.addAll(AbstractCollection.java:305)
> at org.apache.lucene.index.SegmentInfos.files(SegmentInfos.java:826)
> at org.apache.lucene.index.DirectoryReader$ReaderCommit.<init>(DirectoryReader.java:916)
> at org.apache.lucene.index.DirectoryReader.getIndexCommit(DirectoryReader.java:856)
> at org.apache.solr.search.SolrIndexReader.getIndexCommit(SolrIndexReader.java:454)
> at org.apache.solr.handler.SnapPuller.fetchLatestIndex(SnapPuller.java:261)
> at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:264)
> at org.apache.solr.handler.ReplicationHandler$1.run(ReplicationHandler.java:146)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2584) Concurrency issues in
SegmentInfo.files() could lead to ConcurrentModificationException
Posted by "Alexander Kanarsky (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12895119#action_12895119 ]
Alexander Kanarsky commented on LUCENE-2584:
--------------------------------------------
OK, I'll create patches tomorrow.
> Concurrency issues in SegmentInfo.files() could lead to ConcurrentModificationException
> ---------------------------------------------------------------------------------------
>
> Key: LUCENE-2584
> URL: https://issues.apache.org/jira/browse/LUCENE-2584
> Project: Lucene - Java
> Issue Type: Bug
> Affects Versions: 2.9, 2.9.1, 2.9.2, 2.9.3, 3.0, 3.0.1, 3.0.2
> Reporter: Alexander Kanarsky
> Priority: Minor
> Fix For: 3.1, 4.0
>
>
> The multi-threaded call of the files() in SegmentInfo could lead to the ConcurrentModificationException if one thread is not finished additions to the ArrayList (files) yet while the other thread already obtained it as cached (see below). This is a rare exception, but it would be nice to fix. I see the code is no longer problematic in the trunk (and others ported from flex_1458), looks it was fixed while implementing post 3.x features. The fix to 3.x and 2.9.x branches could be the same - create the files set first and populate it, and then assign to the member variable at the end of the method. This will resolve the issue. I could prepare the patch for 2.9.4 and 3.x, if needed.
> --
> INFO: [19] webapp= path=/replication params={command=fetchindex&wt=javabin} status=0 QTime=1
> Jul 30, 2010 9:13:05 AM org.apache.solr.core.SolrCore execute
> INFO: [19] webapp= path=/replication params={command=details&wt=javabin} status=0 QTime=24
> Jul 30, 2010 9:13:05 AM org.apache.solr.handler.ReplicationHandler doFetch
> SEVERE: SnapPull failed
> java.util.ConcurrentModificationException
> at java.util.AbstractList$Itr.checkForComodification(AbstractList.java:372)
> at java.util.AbstractList$Itr.next(AbstractList.java:343)
> at java.util.AbstractCollection.addAll(AbstractCollection.java:305)
> at org.apache.lucene.index.SegmentInfos.files(SegmentInfos.java:826)
> at org.apache.lucene.index.DirectoryReader$ReaderCommit.<init>(DirectoryReader.java:916)
> at org.apache.lucene.index.DirectoryReader.getIndexCommit(DirectoryReader.java:856)
> at org.apache.solr.search.SolrIndexReader.getIndexCommit(SolrIndexReader.java:454)
> at org.apache.solr.handler.SnapPuller.fetchLatestIndex(SnapPuller.java:261)
> at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:264)
> at org.apache.solr.handler.ReplicationHandler$1.run(ReplicationHandler.java:146)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Updated: (LUCENE-2584) Concurrency issues in
SegmentInfo.files() could lead to ConcurrentModificationException
Posted by "Robert Muir (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Muir updated LUCENE-2584:
--------------------------------
Component/s: Index
> Concurrency issues in SegmentInfo.files() could lead to ConcurrentModificationException
> ---------------------------------------------------------------------------------------
>
> Key: LUCENE-2584
> URL: https://issues.apache.org/jira/browse/LUCENE-2584
> Project: Lucene - Java
> Issue Type: Bug
> Components: Index
> Affects Versions: 2.9, 2.9.1, 2.9.2, 2.9.3, 3.0, 3.0.1, 3.0.2
> Reporter: Alexander Kanarsky
> Priority: Minor
> Fix For: 3.1, 4.0
>
> Attachments: LUCENE-2584-branch_3x.patch, LUCENE-2584-lucene-2_9.patch, LUCENE-2584-lucene-3_0.patch
>
>
> The multi-threaded call of the files() in SegmentInfo could lead to the ConcurrentModificationException if one thread is not finished additions to the ArrayList (files) yet while the other thread already obtained it as cached (see below). This is a rare exception, but it would be nice to fix. I see the code is no longer problematic in the trunk (and others ported from flex_1458), looks it was fixed while implementing post 3.x features. The fix to 3.x and 2.9.x branches could be the same - create the files set first and populate it, and then assign to the member variable at the end of the method. This will resolve the issue. I could prepare the patch for 2.9.4 and 3.x, if needed.
> --
> INFO: [19] webapp= path=/replication params={command=fetchindex&wt=javabin} status=0 QTime=1
> Jul 30, 2010 9:13:05 AM org.apache.solr.core.SolrCore execute
> INFO: [19] webapp= path=/replication params={command=details&wt=javabin} status=0 QTime=24
> Jul 30, 2010 9:13:05 AM org.apache.solr.handler.ReplicationHandler doFetch
> SEVERE: SnapPull failed
> java.util.ConcurrentModificationException
> at java.util.AbstractList$Itr.checkForComodification(AbstractList.java:372)
> at java.util.AbstractList$Itr.next(AbstractList.java:343)
> at java.util.AbstractCollection.addAll(AbstractCollection.java:305)
> at org.apache.lucene.index.SegmentInfos.files(SegmentInfos.java:826)
> at org.apache.lucene.index.DirectoryReader$ReaderCommit.<init>(DirectoryReader.java:916)
> at org.apache.lucene.index.DirectoryReader.getIndexCommit(DirectoryReader.java:856)
> at org.apache.solr.search.SolrIndexReader.getIndexCommit(SolrIndexReader.java:454)
> at org.apache.solr.handler.SnapPuller.fetchLatestIndex(SnapPuller.java:261)
> at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:264)
> at org.apache.solr.handler.ReplicationHandler$1.run(ReplicationHandler.java:146)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2584) Concurrency issues in
SegmentInfo.files() could lead to ConcurrentModificationException
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12894860#action_12894860 ]
Michael McCandless commented on LUCENE-2584:
--------------------------------------------
Ahh great catch! Yes please create a patch? Thanks!
> Concurrency issues in SegmentInfo.files() could lead to ConcurrentModificationException
> ---------------------------------------------------------------------------------------
>
> Key: LUCENE-2584
> URL: https://issues.apache.org/jira/browse/LUCENE-2584
> Project: Lucene - Java
> Issue Type: Bug
> Affects Versions: 2.9, 2.9.1, 2.9.2, 2.9.3, 3.0, 3.0.1, 3.0.2
> Reporter: Alexander Kanarsky
> Priority: Minor
>
> The multi-threaded call of the files() in SegmentInfo could lead to the ConcurrentModificationException if one thread is not finished additions to the ArrayList (files) yet while the other thread already obtained it as cached (see below). This is a rare exception, but it would be nice to fix. I see the code is no longer problematic in the trunk (and others ported from flex_1458), looks it was fixed while implementing post 3.x features. The fix to 3.x and 2.9.x branches could be the same - create the files set first and populate it, and then assign to the member variable at the end of the method. This will resolve the issue. I could prepare the patch for 2.9.4 and 3.x, if needed.
> --
> INFO: [19] webapp= path=/replication params={command=fetchindex&wt=javabin} status=0 QTime=1
> Jul 30, 2010 9:13:05 AM org.apache.solr.core.SolrCore execute
> INFO: [19] webapp= path=/replication params={command=details&wt=javabin} status=0 QTime=24
> Jul 30, 2010 9:13:05 AM org.apache.solr.handler.ReplicationHandler doFetch
> SEVERE: SnapPull failed
> java.util.ConcurrentModificationException
> at java.util.AbstractList$Itr.checkForComodification(AbstractList.java:372)
> at java.util.AbstractList$Itr.next(AbstractList.java:343)
> at java.util.AbstractCollection.addAll(AbstractCollection.java:305)
> at org.apache.lucene.index.SegmentInfos.files(SegmentInfos.java:826)
> at org.apache.lucene.index.DirectoryReader$ReaderCommit.<init>(DirectoryReader.java:916)
> at org.apache.lucene.index.DirectoryReader.getIndexCommit(DirectoryReader.java:856)
> at org.apache.solr.search.SolrIndexReader.getIndexCommit(SolrIndexReader.java:454)
> at org.apache.solr.handler.SnapPuller.fetchLatestIndex(SnapPuller.java:261)
> at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:264)
> at org.apache.solr.handler.ReplicationHandler$1.run(ReplicationHandler.java:146)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org