You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@lucene.apache.org by "cbuescher (via GitHub)" <gi...@apache.org> on 2023/04/25 13:33:36 UTC
[GitHub] [lucene] cbuescher opened a new pull request, #12241: Add ordering of files in compound files
cbuescher opened a new pull request, #12241:
URL: https://github.com/apache/lucene/pull/12241
Today there is no specific ordering of how files are written to a compound file.
The current order is determined by iterating over the set of file names in
SegmentInfo, which is unspecific. This PR proposes to change to an order based
on file size. Colocating data from files that are smaller (typically metadata
files like terms index, field info etc...) but accessed often can help when
parts of these files are help in cache.
In our particular case, the motivation is coming from reading larger compound
files from a remote object store through a caching layer that keeps chunks of
the file in pages. Keeping small files together can help improve the efficiency
of the cache because data that is read often (like metadata) is kept together.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] romseygeek commented on a diff in pull request #12241: Add ordering of files in compound files
Posted by "romseygeek (via GitHub)" <gi...@apache.org>.
romseygeek commented on code in PR #12241:
URL: https://github.com/apache/lucene/pull/12241#discussion_r1177646123
##########
lucene/core/src/test/org/apache/lucene/codecs/lucene90/TestLucene90CompoundFormat.java:
##########
@@ -27,4 +36,58 @@ public class TestLucene90CompoundFormat extends BaseCompoundFormatTestCase {
protected Codec getCodec() {
return codec;
}
+
+ public void testFileLengthOrdering() throws IOException {
+ Directory dir = newDirectory();
+ // Setup the test segment
+ String segment = "_123";
+ int chunk = 1024; // internal buffer size used by the stream
+ SegmentInfo si = newSegmentInfo(dir, segment);
+ byte[] segId = si.getId();
+ List<String> files = new ArrayList<>();
+ int randomFileSize = random().nextInt(0, chunk);
+ for (int i = 0; i < 10; i++) {
+ String filename = segment + "." + i;
+ createRandomFile(dir, filename, randomFileSize, segId);
+ // increase the next files size by a random amount
+ randomFileSize += random().nextInt(1, 100);
+ files.add(filename);
+ }
+ si.setFiles(files);
Review Comment:
Can we explicitly shuffle the `files` list to make it clear that things are in a random order within the segment info? And maybe add a comment to the point that it's held internally as a set so there's no defined ordering in any case?
##########
lucene/core/src/java/org/apache/lucene/codecs/lucene90/Lucene90CompoundFormat.java:
##########
@@ -102,11 +103,40 @@ public void write(Directory dir, SegmentInfo si, IOContext context) throws IOExc
}
}
+ private static class SizedFile {
+ String name;
Review Comment:
Let's make these final?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] cbuescher commented on pull request #12241: Add ordering of files in compound files
Posted by "cbuescher (via GitHub)" <gi...@apache.org>.
cbuescher commented on PR #12241:
URL: https://github.com/apache/lucene/pull/12241#issuecomment-1521944721
@romseygeek fyi
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] mikemccand commented on pull request #12241: Add ordering of files in compound files
Posted by "mikemccand (via GitHub)" <gi...@apache.org>.
mikemccand commented on PR #12241:
URL: https://github.com/apache/lucene/pull/12241#issuecomment-1521829039
+1, cool idea!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] cbuescher commented on a diff in pull request #12241: Add ordering of files in compound files
Posted by "cbuescher (via GitHub)" <gi...@apache.org>.
cbuescher commented on code in PR #12241:
URL: https://github.com/apache/lucene/pull/12241#discussion_r1177689578
##########
lucene/core/src/java/org/apache/lucene/codecs/lucene90/Lucene90CompoundFormat.java:
##########
@@ -102,11 +103,40 @@ public void write(Directory dir, SegmentInfo si, IOContext context) throws IOExc
}
}
+ private static class SizedFile {
+ String name;
Review Comment:
done
##########
lucene/core/src/test/org/apache/lucene/codecs/lucene90/TestLucene90CompoundFormat.java:
##########
@@ -27,4 +36,58 @@ public class TestLucene90CompoundFormat extends BaseCompoundFormatTestCase {
protected Codec getCodec() {
return codec;
}
+
+ public void testFileLengthOrdering() throws IOException {
+ Directory dir = newDirectory();
+ // Setup the test segment
+ String segment = "_123";
+ int chunk = 1024; // internal buffer size used by the stream
+ SegmentInfo si = newSegmentInfo(dir, segment);
+ byte[] segId = si.getId();
+ List<String> files = new ArrayList<>();
+ int randomFileSize = random().nextInt(0, chunk);
+ for (int i = 0; i < 10; i++) {
+ String filename = segment + "." + i;
+ createRandomFile(dir, filename, randomFileSize, segId);
+ // increase the next files size by a random amount
+ randomFileSize += random().nextInt(1, 100);
+ files.add(filename);
+ }
+ si.setFiles(files);
Review Comment:
I added something for that
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] romseygeek merged pull request #12241: Add ordering of files in compound files
Posted by "romseygeek (via GitHub)" <gi...@apache.org>.
romseygeek merged PR #12241:
URL: https://github.com/apache/lucene/pull/12241
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org