You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@solr.apache.org by "Houston Putman (Jira)" <ji...@apache.org> on 2021/09/22 14:36:00 UTC
[jira] [Commented] (SOLR-15639) S3 tests are needlessly slow.
[ https://issues.apache.org/jira/browse/SOLR-15639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17418633#comment-17418633 ]
Houston Putman commented on SOLR-15639:
---------------------------------------
Here is the breakdown of the speed of the S3 tests:
{quote}The slowest tests (exceeding 500 ms) during this run:
58.79s S3IncrementalBackupTest.testBackupIncremental (:solr:contrib:s3-repository)
11.34s S3IncrementalBackupTest.testSimple (:solr:contrib:s3-repository)
10.27s S3IncrementalBackupTest.testRestoreToOriginalCollection (:solr:contrib:s3-repository)
5.65s S3BackupRepositoryTest.testCopyFiles (:solr:contrib:s3-repository)
3.96s S3OutputStreamTest.testWriteLargeBuffer (:solr:contrib:s3-repository)
2.81s S3PathsTest.testDeleteBatching (:solr:contrib:s3-repository)
1.97s S3OutputStreamTest.testFlushLargeBuffer (:solr:contrib:s3-repository)
1.54s S3PathsTest.testDeleteDirectoryMultipleLevels (:solr:contrib:s3-repository)
1.28s S3OutputStreamTest.testFlushSmallBuffer (:solr:contrib:s3-repository)
1.25s S3ReadWriteTest.testBasicWriteRead (:solr:contrib:s3-repository)
The slowest suites (exceeding 1s) during this run:
88.75s S3IncrementalBackupTest (:solr:contrib:s3-repository)
13.85s S3BackupRepositoryTest (:solr:contrib:s3-repository)
12.96s S3OutputStreamTest (:solr:contrib:s3-repository)
10.52s S3PathsTest (:solr:contrib:s3-repository)
7.13s S3ReadWriteTest (:solr:contrib:s3-repository)
1.93s S3IndexInputTest (:solr:contrib:s3-repository)
{quote}
The S3IncrementalBackupTest is a specific case of the {{AbstractIncrementalBackupTest}} , so if we speed this up, it will make GCS and other IncrementalBackupTests faster at the same time
Two other tests - {{S3BackupRepositoryTest.testCopyFiles}} and {{S3OutputStreamTest.testWriteLargeBuffer}} are slow because they are writing a large amount of data, but these can probably be sped up.
The rest are slow, seemingly only because S3Mock itself is really slow. There is no other reason that a single write and read would take 1.25 seconds ( {{S3ReadWriteTest.testBasicWriteRead}} ). If we can figure out how to make S3Mock fast, or use something else that is faster, then I think that will fix the speed of the tests in the contrib module.
Maybe we can look into another S3Mock that is completely in-memory (we have no need to go to disk): https://github.com/findify/s3mock
> S3 tests are needlessly slow.
> -----------------------------
>
> Key: SOLR-15639
> URL: https://issues.apache.org/jira/browse/SOLR-15639
> Project: Solr
> Issue Type: Test
> Security Level: Public(Default Security Level. Issues are Public)
> Components: contrib - S3 Repository, Tests
> Reporter: Mark Robert Miller
> Assignee: Mark Robert Miller
> Priority: Major
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@solr.apache.org
For additional commands, e-mail: issues-help@solr.apache.org