You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucenenet.apache.org by GitBox <gi...@apache.org> on 2020/08/11 18:21:47 UTC
[GitHub] [lucenenet] NightOwl888 edited a comment on pull request #325: Fix for LineFileDocs Bottleneck/Performance Improvements
NightOwl888 edited a comment on pull request #325:
URL: https://github.com/apache/lucenenet/pull/325#issuecomment-672154841
I confirmed that the error exists in the master branch, and I got it to occur on `beta00006` on the second try with a `[Repeat(5000)]` attribute. It does not occur in `beta00005` running 100,000 times.
I also noticed that if the repeat attribute is changed to 10,000 times it will cause an `OutOfMemoryException`.
I also saw this error occur from the same test:
```
Test Name: TestMultiThreadedSnapshotting
Test FullName: Lucene.Net.Tests._I-J.Lucene.Net.Index.TestSnapshotDeletionPolicy.TestMultiThreadedSnapshotting
Test Source: F:\Projects\lucenenet\src\Lucene.Net.Tests\Index\TestSnapshotDeletionPolicy.cs : line 353
Test Outcome: Failed
Test Duration: 0:00:00
Test Name: TestMultiThreadedSnapshotting
Test Outcome: Failed
Result StackTrace:
at J2N.Threading.ThreadJob.Join()
at Lucene.Net.Index.TestSnapshotDeletionPolicy.TestMultiThreadedSnapshotting() in F:\Projects\lucenenet\src\Lucene.Net.Tests\Index\TestSnapshotDeletionPolicy.cs:line 375
--DebugAssertException
Result Message:
System.Exception : Method <method> failed with 'full flush buffer should be empty: System.Collections.Generic.List`1[Lucene.Net.Index.DocumentsWriterPerThread]
', and was translated to Microsoft.VisualStudio.TestPlatform.TestHost.DebugAssertException to avoid terminating the process hosting the test.
Data:
OriginalMessage: System.Exception: Method <method> failed with 'full flush buffer should be empty: System.Collections.Generic.List`1[Lucene.Net.Index.DocumentsWriterPerThread]
', and was translated to Microsoft.VisualStudio.TestPlatform.TestHost.DebugAssertException to avoid terminating the process hosting the test.
---> Microsoft.VisualStudio.TestPlatform.TestHost.DebugAssertException: Method <method> failed with 'full flush buffer should be empty: System.Collections.Generic.List`1[Lucene.Net.Index.DocumentsWriterPerThread]
', and was translated to Microsoft.VisualStudio.TestPlatform.TestHost.DebugAssertException to avoid terminating the process hosting the test.
--- End of inner exception stack trace ---
at Lucene.Net.Index.TestSnapshotDeletionPolicy.ThreadAnonymousInnerClassHelper2.Run() in F:\Projects\lucenenet\src\Lucene.Net.Tests\Index\TestSnapshotDeletionPolicy.cs:line 420
at J2N.Threading.ThreadJob.SafeRun(ThreadStart start)
----> Microsoft.VisualStudio.TestPlatform.TestHost.DebugAssertException : Method <method> failed with 'full flush buffer should be empty: System.Collections.Generic.List`1[Lucene.Net.Index.DocumentsWriterPerThread]
', and was translated to Microsoft.VisualStudio.TestPlatform.TestHost.DebugAssertException to avoid terminating the process hosting the test.
Result StandardOutput:
Culture: ru-KZ
Time Zone: (UTC+04:00) Astrakhan, Ulyanovsk
Default Codec: CheapBastard (CheapBastardCodec)
Default Similarity: RandomSimilarityProvider(queryNorm=False,coord=crazy): {}
Nightly: False
Weekly: False
Slow: True
Awaits Fix: False
Directory: random
Verbose: True
Random Multiplier: 1
IFD 1 [11.08.2020 18:10:18; NonParallelWorker]: init: current segments file is ""; deletionPolicy=Lucene.Net.Index.SnapshotDeletionPolicy
IFD 1 [11.08.2020 18:10:18; NonParallelWorker]: now checkpoint "" [0 segments ; isCommit = False]
IFD 1 [11.08.2020 18:10:18; NonParallelWorker]: 1 msec to checkpoint
IW 1 [11.08.2020 18:10:18; NonParallelWorker]: init: create=True
IW 1 [11.08.2020 18:10:18; NonParallelWorker]:
dir=MockDirectoryWrapper(RAMDirectory@548c1d lockFactory=Lucene.Net.Store.SingleInstanceLockFactory)
index=
version=4.8.0
matchVersion=LUCENE_48
analyzer=MockAnalyzer
ramBufferSizeMB=16
maxBufferedDocs=493
maxBufferedDeleteTerms=-1
mergedSegmentWarmer=
readerTermsIndexDivisor=2
termIndexInterval=58
delPolicy=SnapshotDeletionPolicy
commit=null
openMode=CREATE_OR_APPEND
similarity=RandomSimilarityProvider
mergeScheduler=ConcurrentMergeScheduler: maxThreadCount=1, maxMergeCount=2, mergeThreadPriority=-1
default WRITE_LOCK_TIMEOUT=1000
writeLockTimeout=1000
codec=CheapBastard
infoStream=ThreadNameFixingPrintStreamInfoStream
mergePolicy=[LogDocMergePolicy: minMergeSize=1000, mergeFactor=13, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=True, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8796093022207,999, noCFSRatio=1]
indexerThreadPool=Lucene.Net.Index.ThreadAffinityDocumentsWriterThreadPool
readerPooling=True
perThreadHardLimitMB=1945
useCompoundFile=True
checkIntegrityAtMerge=False
IW 1 [11.08.2020 18:10:18; t6]: commit: start
IW 1 [11.08.2020 18:10:18; t5]: commit: start
IW 1 [11.08.2020 18:10:18; t2]: commit: start
IW 1 [11.08.2020 18:10:18; t6]: commit: enter lock
IW 1 [11.08.2020 18:10:18; t1]: commit: start
IW 1 [11.08.2020 18:10:18; t3]: commit: start
IW 1 [11.08.2020 18:10:18; t6]: commit: now prepare
IW 1 [11.08.2020 18:10:18; t9]: commit: start
IW 1 [11.08.2020 18:10:18; t7]: commit: start
IW 1 [11.08.2020 18:10:18; t8]: commit: start
IW 1 [11.08.2020 18:10:18; t0]: commit: start
IW 1 [11.08.2020 18:10:18; t4]: commit: start
IW 1 [11.08.2020 18:10:18; t6]: prepareCommit: flush
IW 1 [11.08.2020 18:10:18; t6]: index before flush
DW 1 [11.08.2020 18:10:18; t6]: startFullFlush
DW 1 [11.08.2020 18:10:18; t6]: anyChanges? numDocsInRam=10 deletes=False hasTickets:False pendingChangesInFullFlush: False
DWFC 1 [11.08.2020 18:10:18; t6]: addFlushableState DocumentsWriterPerThread [pendingDeletes=gen=0, segment=_0, aborting=False, numDocsInRAM=2, deleteQueue=DWDQ: [ generation: 0 ]]
DWFC 1 [11.08.2020 18:10:18; t6]: addFlushableState DocumentsWriterPerThread [pendingDeletes=gen=0, segment=_1, aborting=False, numDocsInRAM=2, deleteQueue=DWDQ: [ generation: 0 ]]
DWFC 1 [11.08.2020 18:10:18; t6]: addFlushableState DocumentsWriterPerThread [pendingDeletes=gen=0, segment=_6, aborting=False, numDocsInRAM=1, deleteQueue=DWDQ: [ generation: 0 ]]
DWFC 1 [11.08.2020 18:10:18; t6]: addFlushableState DocumentsWriterPerThread [pendingDeletes=gen=0, segment=_2, aborting=False, numDocsInRAM=1, deleteQueue=DWDQ: [ generation: 0 ]]
DWFC 1 [11.08.2020 18:10:18; t6]: addFlushableState DocumentsWriterPerThread [pendingDeletes=gen=0, segment=_3, aborting=False, numDocsInRAM=1, deleteQueue=DWDQ: [ generation: 0 ]]
DWFC 1 [11.08.2020 18:10:18; t6]: addFlushableState DocumentsWriterPerThread [pendingDeletes=gen=0, segment=_4, aborting=False, numDocsInRAM=1, deleteQueue=DWDQ: [ generation: 0 ]]
DWFC 1 [11.08.2020 18:10:18; t6]: addFlushableState DocumentsWriterPerThread [pendingDeletes=gen=0, segment=_5, aborting=False, numDocsInRAM=1, deleteQueue=DWDQ: [ generation: 0 ]]
DWFC 1 [11.08.2020 18:10:18; t6]: addFlushableState Docume< Truncated >
```
I have opened #326 to put a higher priority on this task - we need to get the "asserts" working in Release mode to ensure that our test suite is testing everything that it is supposed to so more issues like this don't creep in without us being aware of them.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org