You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "ASF subversion and git services (JIRA)" <ji...@apache.org> on 2019/04/10 07:30:00 UTC
[jira] [Commented] (LUCENE-8754) SegmentInfo#toString can cause
ConcurrentModificationException
[ https://issues.apache.org/jira/browse/LUCENE-8754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16814148#comment-16814148 ]
ASF subversion and git services commented on LUCENE-8754:
---------------------------------------------------------
Commit a9503d2e81eb039bf06e03cef257c43ba34e5906 in lucene-solr's branch refs/heads/master from Simon Willnauer
[ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=a9503d2 ]
LUCENE-8754: Prevent ConcurrentModificationException in SegmentInfo (#637)
In order to prevent ConcurrentModificationException this change makes
an unmodifiable copy on write for all maps in SegmentInfo. MergePolicies
can access these maps without synchronization and cause exceptions if
it's modified in the merge thread.
> SegmentInfo#toString can cause ConcurrentModificationException
> --------------------------------------------------------------
>
> Key: LUCENE-8754
> URL: https://issues.apache.org/jira/browse/LUCENE-8754
> Project: Lucene - Core
> Issue Type: Improvement
> Reporter: Simon Willnauer
> Priority: Major
> Time Spent: 3h
> Remaining Estimate: 0h
>
> A recent change increased the likelihood for this issue to show up but it can already happen before since we are using the attributes map in the StoredFieldsFormat for quite some time. I found this issue due to a test failure on our CI:
> {noformat}
> 13:11:56 [junit4] Suite: org.apache.lucene.index.TestIndexSorting
> 13:11:56 [junit4] 2> apr 05, 2019 8:11:53 AM com.carrotsearch.randomizedtesting.RandomizedRunner$QueueUncaughtExceptionsHandler uncaughtException
> 13:11:56 [junit4] 2> WARNING: Uncaught exception in thread: Thread[Thread-507,5,TGRP-TestIndexSorting]
> 13:11:56 [junit4] 2> java.util.ConcurrentModificationException
> 13:11:56 [junit4] 2> at __randomizedtesting.SeedInfo.seed([7C25B308F180203B]:0)
> 13:11:56 [junit4] 2> at java.util.HashMap$HashIterator.nextNode(HashMap.java:1442)
> 13:11:56 [junit4] 2> at java.util.HashMap$EntryIterator.next(HashMap.java:1476)
> 13:11:56 [junit4] 2> at java.util.HashMap$EntryIterator.next(HashMap.java:1474)
> 13:11:56 [junit4] 2> at java.util.AbstractMap.toString(AbstractMap.java:554)
> 13:11:56 [junit4] 2> at org.apache.lucene.index.SegmentInfo.toString(SegmentInfo.java:222)
> 13:11:56 [junit4] 2> at org.apache.lucene.index.SegmentCommitInfo.toString(SegmentCommitInfo.java:345)
> 13:11:56 [junit4] 2> at org.apache.lucene.index.SegmentCommitInfo.toString(SegmentCommitInfo.java:364)
> 13:11:56 [junit4] 2> at java.lang.String.valueOf(String.java:2994)
> 13:11:56 [junit4] 2> at java.lang.StringBuilder.append(StringBuilder.java:131)
> 13:11:56 [junit4] 2> at java.util.AbstractMap.toString(AbstractMap.java:557)
> 13:11:56 [junit4] 2> at java.util.Collections$UnmodifiableMap.toString(Collections.java:1493)
> 13:11:56 [junit4] 2> at java.lang.String.valueOf(String.java:2994)
> 13:11:56 [junit4] 2> at java.lang.StringBuilder.append(StringBuilder.java:131)
> 13:11:56 [junit4] 2> at org.apache.lucene.index.TieredMergePolicy.findForcedMerges(TieredMergePolicy.java:628)
> 13:11:56 [junit4] 2> at org.apache.lucene.index.IndexWriter.updatePendingMerges(IndexWriter.java:2181)
> 13:11:56 [junit4] 2> at org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:2154)
> 13:11:56 [junit4] 2> at org.apache.lucene.index.IndexWriter.forceMerge(IndexWriter.java:1988)
> 13:11:56 [junit4] 2> at org.apache.lucene.index.IndexWriter.forceMerge(IndexWriter.java:1939)
> 13:11:56 [junit4] 2> at org.apache.lucene.index.TestIndexSorting$UpdateRunnable.run(TestIndexSorting.java:1851)
> 13:11:56 [junit4] 2> at java.lang.Thread.run(Thread.java:748)
> 13:11:56 [junit4] 2>
> 13:11:56 [junit4] 2> NOTE: reproduce with: ant test -Dtestcase=TestIndexSorting -Dtests.method=testConcurrentUpdates -Dtests.seed=7C25B308F180203B -Dtests.slow=true -Dtest
> {noformat}
> The issue is that we update the attributes map (also we similarly do the same for diagnostics but it's not necessarily causing the issue since the diagnostics map is never modified) during the merge process but access it in the merge policy when looking at running merges and there we call toString on SegmentCommitInfo which happens without any synchronization. This is technically unsafe publication but IW is a mess along those lines and real fixes would require significant changes.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org