You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@lucene.apache.org by mi...@apache.org on 2023/11/06 13:26:17 UTC
(lucene) branch branch_9x updated (7ee8b0af3b7 -> b0a1b52e148)
This is an automated email from the ASF dual-hosted git repository.
mikemccand pushed a change to branch branch_9x
in repository https://gitbox.apache.org/repos/asf/lucene.git
from 7ee8b0af3b7 Use growNoCopy for SortingStoredFieldsConsumer#NO_COMPRESSION (#12733)
new 97f84a58708 Clean up ByteBlockPool (#12506)
new b0a1b52e148 switch to TestUtil for the random int since Random#nextInt(int, int) is not available in Java 11
The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
lucene/CHANGES.txt | 5 +
.../org/apache/lucene/index/ByteSlicePool.java | 141 +++++++++++
.../org/apache/lucene/index/ByteSliceReader.java | 10 +-
.../org/apache/lucene/index/TermsHashPerField.java | 97 +------
.../java/org/apache/lucene/util/ByteBlockPool.java | 102 +++++---
.../org/apache/lucene/util/BytesRefBlockPool.java | 6 +-
.../org/apache/lucene/index/TestByteSlicePool.java | 279 +++++++++++++++++++++
.../apache/lucene/index/TestByteSliceReader.java | 5 +-
.../apache/lucene/index/TestTermsHashPerField.java | 34 ---
9 files changed, 509 insertions(+), 170 deletions(-)
create mode 100644 lucene/core/src/java/org/apache/lucene/index/ByteSlicePool.java
create mode 100644 lucene/core/src/test/org/apache/lucene/index/TestByteSlicePool.java
(lucene) 01/02: Clean up ByteBlockPool (#12506)
Posted by mi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
mikemccand pushed a commit to branch branch_9x
in repository https://gitbox.apache.org/repos/asf/lucene.git
commit 97f84a58708a13a8b18792d50cf072690845d38b
Author: Stefan Vodita <41...@users.noreply.github.com>
AuthorDate: Mon Nov 6 13:12:16 2023 +0000
Clean up ByteBlockPool (#12506)
* Clean up ByteBlockPool
1. Move slice functionality to a separate class, ByteSlicePool.
2. Add exception case if the requested slice size is larger than the
block size.
3. Make pool buffers private instead of the comment asking not to modify it.
4. Consolidate setBytesRef methods with int offsets.
5. Simplify ramBytesUsed.
6. Update and expand comments and javadoc.
* Revert to long offsets in ByteBlockPool; Clean-up
* Remove slice functionality from TermsHashPerField
* First working test
* [Broken] Test interleaving slices
* Fixed randomized test
* Remove redundant tests
* Tidy
* Add CHANGES
* Move ByteSlicePool to oal.index
---
lucene/CHANGES.txt | 5 +
.../org/apache/lucene/index/ByteSlicePool.java | 141 +++++++++++
.../org/apache/lucene/index/ByteSliceReader.java | 10 +-
.../org/apache/lucene/index/TermsHashPerField.java | 97 +------
.../java/org/apache/lucene/util/ByteBlockPool.java | 102 +++++---
.../org/apache/lucene/util/BytesRefBlockPool.java | 6 +-
.../org/apache/lucene/index/TestByteSlicePool.java | 279 +++++++++++++++++++++
.../apache/lucene/index/TestByteSliceReader.java | 5 +-
.../apache/lucene/index/TestTermsHashPerField.java | 34 ---
9 files changed, 509 insertions(+), 170 deletions(-)
diff --git a/lucene/CHANGES.txt b/lucene/CHANGES.txt
index 08b07b0e801..72eaf9e0d0d 100644
--- a/lucene/CHANGES.txt
+++ b/lucene/CHANGES.txt
@@ -170,6 +170,11 @@ Other
* GITHUB#12625: Refactor ByteBlockPool so it is just a "shift/mask big array". (Ignacio Vera)
+* GITHUB#6675: Various improvements related to ByteBlockPool. Slice functionality on top of ByteBlockPool moved to its
+ own class, ByteSlicePool. ByteBlockPool's array of buffers is made private. There are new exceptions for buffer index
+ overflows and slices that are too large. Some bits of code are simplified. Documentation is updated and expanded.
+ (Stefan Vodita)
+
======================== Lucene 9.8.0 =======================
API Changes
diff --git a/lucene/core/src/java/org/apache/lucene/index/ByteSlicePool.java b/lucene/core/src/java/org/apache/lucene/index/ByteSlicePool.java
new file mode 100644
index 00000000000..4e73bb460f7
--- /dev/null
+++ b/lucene/core/src/java/org/apache/lucene/index/ByteSlicePool.java
@@ -0,0 +1,141 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.lucene.index;
+
+import org.apache.lucene.util.BitUtil;
+import org.apache.lucene.util.ByteBlockPool;
+
+/**
+ * Class that Posting and PostingVector use to write interleaved byte streams into shared fixed-size
+ * byte[] arrays. The idea is to allocate slices of increasing lengths. For example, the first slice
+ * is 5 bytes, the next slice is 14, etc. We start by writing our bytes into the first 5 bytes. When
+ * we hit the end of the slice, we allocate the next slice and then write the address of the new
+ * slice into the last 4 bytes of the previous slice (the "forwarding address").
+ *
+ * <p>Each slice is filled with 0's initially, and we mark the end with a non-zero byte. This way
+ * the methods that are writing into the slice don't need to record its length and instead allocate
+ * a new slice once they hit a non-zero byte.
+ *
+ * @lucene.internal
+ */
+final class ByteSlicePool {
+ /**
+ * The underlying structure consists of fixed-size blocks. We overlay variable-length slices on
+ * top. Each slice is contiguous in memory, i.e. it does not straddle multiple blocks.
+ */
+ public final ByteBlockPool pool;
+
+ /**
+ * An array holding the level sizes for byte slices. The first slice is 5 bytes, the second is 14,
+ * and so on.
+ */
+ public static final int[] LEVEL_SIZE_ARRAY = {5, 14, 20, 30, 40, 40, 80, 80, 120, 200};
+
+ /**
+ * An array holding indexes for the {@link #LEVEL_SIZE_ARRAY}, to quickly navigate to the next
+ * slice level. These are encoded on 4 bits in the slice, so the values in this array should be
+ * less than 16.
+ *
+ * <p>{@code NEXT_LEVEL_ARRAY[x] == x + 1}, except for the last element, where {@code
+ * NEXT_LEVEL_ARRAY[x] == x}, pointing at the maximum slice size.
+ */
+ public static final int[] NEXT_LEVEL_ARRAY = {1, 2, 3, 4, 5, 6, 7, 8, 9, 9};
+
+ /** The first level size for new slices. */
+ public static final int FIRST_LEVEL_SIZE = LEVEL_SIZE_ARRAY[0];
+
+ public ByteSlicePool(ByteBlockPool pool) {
+ this.pool = pool;
+ }
+
+ /**
+ * Allocates a new slice with the given size and level 0.
+ *
+ * @return the position where the slice starts
+ */
+ public int newSlice(final int size) {
+ if (size > ByteBlockPool.BYTE_BLOCK_SIZE) {
+ throw new IllegalArgumentException(
+ "Slice size "
+ + size
+ + " should be less than the block size "
+ + ByteBlockPool.BYTE_BLOCK_SIZE);
+ }
+
+ if (pool.byteUpto > ByteBlockPool.BYTE_BLOCK_SIZE - size) {
+ pool.nextBuffer();
+ }
+ final int upto = pool.byteUpto;
+ pool.byteUpto += size;
+ pool.buffer[pool.byteUpto - 1] = 16; // This codifies level 0.
+ return upto;
+ }
+
+ /**
+ * Creates a new byte slice in continuation of the provided slice and return its offset into the
+ * pool.
+ *
+ * @param slice the current slice
+ * @param upto the offset into the current slice, which is expected to point to the last byte of
+ * the slice
+ * @return the new slice's offset in the pool
+ */
+ public int allocSlice(final byte[] slice, final int upto) {
+ return allocKnownSizeSlice(slice, upto) >> 8;
+ }
+
+ /**
+ * Create a new byte slice in continuation of the provided slice and return its length and offset
+ * into the pool.
+ *
+ * @param slice the current slice
+ * @param upto the offset into the current slice, which is expected to point to the last byte of
+ * the slice
+ * @return the new slice's length on the lower 8 bits and the offset into the pool on the other 24
+ * bits
+ */
+ public int allocKnownSizeSlice(final byte[] slice, final int upto) {
+ final int level = slice[upto] & 15; // The last 4 bits codify the level.
+ final int newLevel = NEXT_LEVEL_ARRAY[level];
+ final int newSize = LEVEL_SIZE_ARRAY[newLevel];
+
+ // Maybe allocate another block
+ if (pool.byteUpto > ByteBlockPool.BYTE_BLOCK_SIZE - newSize) {
+ pool.nextBuffer();
+ }
+
+ final int newUpto = pool.byteUpto;
+ final int offset = newUpto + pool.byteOffset;
+ pool.byteUpto += newSize;
+
+ // Copy forward the past 3 bytes (which we are about to overwrite with the forwarding address).
+ // We actually copy 4 bytes at once since VarHandles make it cheap.
+ int past3Bytes = ((int) BitUtil.VH_LE_INT.get(slice, upto - 3)) & 0xFFFFFF;
+ // Ensure we're not changing the content of `buffer` by setting 4 bytes instead of 3. This
+ // should never happen since the next `newSize` bytes must be equal to 0.
+ assert pool.buffer[newUpto + 3] == 0;
+ BitUtil.VH_LE_INT.set(pool.buffer, newUpto, past3Bytes);
+
+ // Write forwarding address at end of last slice:
+ BitUtil.VH_LE_INT.set(slice, upto - 3, offset);
+
+ // Write new level:
+ pool.buffer[pool.byteUpto - 1] = (byte) (16 | newLevel);
+
+ return ((newUpto + 3) << 8) | (newSize - 3);
+ }
+}
diff --git a/lucene/core/src/java/org/apache/lucene/index/ByteSliceReader.java b/lucene/core/src/java/org/apache/lucene/index/ByteSliceReader.java
index 6b20251d74f..118aaf04c2b 100644
--- a/lucene/core/src/java/org/apache/lucene/index/ByteSliceReader.java
+++ b/lucene/core/src/java/org/apache/lucene/index/ByteSliceReader.java
@@ -50,10 +50,10 @@ final class ByteSliceReader extends DataInput {
level = 0;
bufferUpto = startIndex / ByteBlockPool.BYTE_BLOCK_SIZE;
bufferOffset = bufferUpto * ByteBlockPool.BYTE_BLOCK_SIZE;
- buffer = pool.buffers[bufferUpto];
+ buffer = pool.getBuffer(bufferUpto);
upto = startIndex & ByteBlockPool.BYTE_BLOCK_MASK;
- final int firstSize = TermsHashPerField.LEVEL_SIZE_ARRAY[0];
+ final int firstSize = ByteSlicePool.LEVEL_SIZE_ARRAY[0];
if (startIndex + firstSize >= endIndex) {
// There is only this one slice to read
@@ -97,13 +97,13 @@ final class ByteSliceReader extends DataInput {
// Skip to our next slice
final int nextIndex = (int) BitUtil.VH_LE_INT.get(buffer, limit);
- level = TermsHashPerField.NEXT_LEVEL_ARRAY[level];
- final int newSize = TermsHashPerField.LEVEL_SIZE_ARRAY[level];
+ level = ByteSlicePool.NEXT_LEVEL_ARRAY[level];
+ final int newSize = ByteSlicePool.LEVEL_SIZE_ARRAY[level];
bufferUpto = nextIndex / ByteBlockPool.BYTE_BLOCK_SIZE;
bufferOffset = bufferUpto * ByteBlockPool.BYTE_BLOCK_SIZE;
- buffer = pool.buffers[bufferUpto];
+ buffer = pool.getBuffer(bufferUpto);
upto = nextIndex & ByteBlockPool.BYTE_BLOCK_MASK;
if (nextIndex + newSize >= endIndex) {
diff --git a/lucene/core/src/java/org/apache/lucene/index/TermsHashPerField.java b/lucene/core/src/java/org/apache/lucene/index/TermsHashPerField.java
index 0ef4c39ddec..2662ec8f2de 100644
--- a/lucene/core/src/java/org/apache/lucene/index/TermsHashPerField.java
+++ b/lucene/core/src/java/org/apache/lucene/index/TermsHashPerField.java
@@ -17,7 +17,6 @@
package org.apache.lucene.index;
import java.io.IOException;
-import org.apache.lucene.util.BitUtil;
import org.apache.lucene.util.ByteBlockPool;
import org.apache.lucene.util.BytesRef;
import org.apache.lucene.util.BytesRefHash;
@@ -39,6 +38,7 @@ abstract class TermsHashPerField implements Comparable<TermsHashPerField> {
private final TermsHashPerField nextPerField;
private final IntBlockPool intPool;
final ByteBlockPool bytePool;
+ private final ByteSlicePool slicePool;
// for each term we store an integer per stream that points into the bytePool above
// the address is updated once data is written to the stream to point to the next free offset
// in the terms stream. The start address for the stream is stored in
@@ -73,6 +73,7 @@ abstract class TermsHashPerField implements Comparable<TermsHashPerField> {
IndexOptions indexOptions) {
this.intPool = intPool;
this.bytePool = bytePool;
+ this.slicePool = new ByteSlicePool(bytePool);
this.streamCount = streamCount;
this.fieldName = fieldName;
this.nextPerField = nextPerField;
@@ -98,7 +99,7 @@ abstract class TermsHashPerField implements Comparable<TermsHashPerField> {
final int offsetInAddressBuffer = streamStartOffset & IntBlockPool.INT_BLOCK_MASK;
reader.init(
bytePool,
- postingsArray.byteStarts[termID] + stream * FIRST_LEVEL_SIZE,
+ postingsArray.byteStarts[termID] + stream * ByteSlicePool.FIRST_LEVEL_SIZE,
streamAddressBuffer[offsetInAddressBuffer + stream]);
}
@@ -143,8 +144,8 @@ abstract class TermsHashPerField implements Comparable<TermsHashPerField> {
/**
* Called when we first encounter a new term. We must allocate slies to store the postings (vInt
- * compressed doc/freq/prox), and also the int pointers to where (in our ByteBlockPool storage)
- * the postings for this term begin.
+ * compressed doc/freq/prox), and also the int pointers to where (in our {@link ByteBlockPool}
+ * storage) the postings for this term begin.
*/
private void initStreamSlices(int termID, int docID) throws IOException {
// Init stream slices
@@ -154,7 +155,8 @@ abstract class TermsHashPerField implements Comparable<TermsHashPerField> {
intPool.nextBuffer();
}
- if (ByteBlockPool.BYTE_BLOCK_SIZE - bytePool.byteUpto < (2 * streamCount) * FIRST_LEVEL_SIZE) {
+ if (ByteBlockPool.BYTE_BLOCK_SIZE - bytePool.byteUpto
+ < (2 * streamCount) * ByteSlicePool.FIRST_LEVEL_SIZE) {
// can we fit at least one byte per stream in the current buffer, if not allocate a new one
bytePool.nextBuffer();
}
@@ -168,7 +170,7 @@ abstract class TermsHashPerField implements Comparable<TermsHashPerField> {
for (int i = 0; i < streamCount; i++) {
// initialize each stream with a slice we start with ByteBlockPool.FIRST_LEVEL_SIZE)
// and grow as we need more space. see ByteBlockPool.LEVEL_SIZE_ARRAY
- final int upto = newSlice(bytePool, FIRST_LEVEL_SIZE, 0);
+ final int upto = slicePool.newSlice(ByteSlicePool.FIRST_LEVEL_SIZE);
termStreamAddressBuffer[streamAddressOffset + i] = upto + bytePool.byteOffset;
}
postingsArray.byteStarts[termID] = termStreamAddressBuffer[streamAddressOffset];
@@ -216,12 +218,12 @@ abstract class TermsHashPerField implements Comparable<TermsHashPerField> {
final void writeByte(int stream, byte b) {
int streamAddress = streamAddressOffset + stream;
int upto = termStreamAddressBuffer[streamAddress];
- byte[] bytes = bytePool.buffers[upto >> ByteBlockPool.BYTE_BLOCK_SHIFT];
+ byte[] bytes = bytePool.getBuffer(upto >> ByteBlockPool.BYTE_BLOCK_SHIFT);
assert bytes != null;
int offset = upto & ByteBlockPool.BYTE_BLOCK_MASK;
if (bytes[offset] != 0) {
// End of slice; allocate a new one
- offset = allocSlice(bytePool, bytes, offset);
+ offset = slicePool.allocSlice(bytes, offset);
bytes = bytePool.buffer;
termStreamAddressBuffer[streamAddress] = offset + bytePool.byteOffset;
}
@@ -233,7 +235,7 @@ abstract class TermsHashPerField implements Comparable<TermsHashPerField> {
final int end = offset + len;
int streamAddress = streamAddressOffset + stream;
int upto = termStreamAddressBuffer[streamAddress];
- byte[] slice = bytePool.buffers[upto >> ByteBlockPool.BYTE_BLOCK_SHIFT];
+ byte[] slice = bytePool.getBuffer(upto >> ByteBlockPool.BYTE_BLOCK_SHIFT);
assert slice != null;
int sliceOffset = upto & ByteBlockPool.BYTE_BLOCK_MASK;
@@ -243,7 +245,7 @@ abstract class TermsHashPerField implements Comparable<TermsHashPerField> {
}
while (offset < end) {
- int offsetAndLength = allocKnownSizeSlice(bytePool, slice, sliceOffset);
+ int offsetAndLength = slicePool.allocKnownSizeSlice(slice, sliceOffset);
sliceOffset = offsetAndLength >> 8;
int sliceLength = offsetAndLength & 0xff;
slice = bytePool.buffer;
@@ -255,81 +257,6 @@ abstract class TermsHashPerField implements Comparable<TermsHashPerField> {
}
}
- // Size of each slice. These arrays should be at most 16
- // elements (index is encoded with 4 bits). First array
- // is just a compact way to encode X+1 with a max. Second
- // array is the length of each slice, ie first slice is 5
- // bytes, next slice is 14 bytes, etc.
-
- /**
- * An array holding the offset into the {@link #LEVEL_SIZE_ARRAY} to quickly navigate to the next
- * slice level.
- */
- static final int[] NEXT_LEVEL_ARRAY = {1, 2, 3, 4, 5, 6, 7, 8, 9, 9};
-
- /** An array holding the level sizes for byte slices. */
- static final int[] LEVEL_SIZE_ARRAY = {5, 14, 20, 30, 40, 40, 80, 80, 120, 200};
-
- /** The first level size for new slices */
- static final int FIRST_LEVEL_SIZE = LEVEL_SIZE_ARRAY[0];
-
- /**
- * Allocates a new slice with the given size. As each slice is filled with 0's initially, we mark
- * the end with a non-zero byte. This way we don't need to record its length and instead allocate
- * new slice once they hit a non-zero byte.
- */
- // pkg private for access by tests
- static int newSlice(ByteBlockPool bytePool, final int size, final int level) {
- assert LEVEL_SIZE_ARRAY[level] == size;
- // Maybe allocate another block
- if (bytePool.byteUpto > ByteBlockPool.BYTE_BLOCK_SIZE - size) {
- bytePool.nextBuffer();
- }
- final int upto = bytePool.byteUpto;
- bytePool.byteUpto += size;
- bytePool.buffer[bytePool.byteUpto - 1] = (byte) (16 | level);
- return upto;
- }
-
- /**
- * Creates a new byte slice with the given starting size and returns the slices offset in the
- * pool.
- */
- // pkg private for access by tests
- static int allocSlice(ByteBlockPool bytePool, final byte[] slice, final int upto) {
- return allocKnownSizeSlice(bytePool, slice, upto) >> 8;
- }
-
- /**
- * Create a new byte slice with the given starting size return the slice offset in the pool and
- * length. The lower 8 bits of the returned int represent the length of the slice, and the upper
- * 24 bits represent the offset.
- */
- // pkg private for access by tests
- static int allocKnownSizeSlice(ByteBlockPool bytePool, final byte[] slice, final int upto) {
- // The idea is to allocate the next slice and then write the address of the new slice
- // into the last 4 bytes of the previous slice (the "forwarding address").
- final int level = slice[upto] & 15;
- final int newLevel = NEXT_LEVEL_ARRAY[level];
- final int newSize = LEVEL_SIZE_ARRAY[newLevel];
-
- final int newUpto = newSlice(bytePool, newSize, newLevel);
- final int offset = newUpto + bytePool.byteOffset;
-
- // Copy forward the past 3 bytes (which we are about to overwrite with the forwarding address).
- // We actually copy 4 bytes at once since VarHandles make it cheap.
- final int past3Bytes = ((int) BitUtil.VH_LE_INT.get(slice, upto - 3)) & 0xFFFFFF;
- // Ensure we're not changing the content of `buffer` by setting 4 bytes instead of 3. This
- // should never happen since the next `newSize` bytes must be equal to 0.
- assert bytePool.buffer[newUpto + 3] == 0;
- BitUtil.VH_LE_INT.set(bytePool.buffer, newUpto, past3Bytes);
-
- // Write forwarding address at end of last slice:
- BitUtil.VH_LE_INT.set(slice, upto - 3, offset);
-
- return ((newUpto + 3) << 8) | (newSize - 3);
- }
-
final void writeVInt(int stream, int i) {
assert stream < streamCount;
while ((i & ~0x7F) != 0) {
diff --git a/lucene/core/src/java/org/apache/lucene/util/ByteBlockPool.java b/lucene/core/src/java/org/apache/lucene/util/ByteBlockPool.java
index d2d5fc00d8c..9fab36f56a4 100644
--- a/lucene/core/src/java/org/apache/lucene/util/ByteBlockPool.java
+++ b/lucene/core/src/java/org/apache/lucene/util/ByteBlockPool.java
@@ -19,12 +19,13 @@ package org.apache.lucene.util;
import static org.apache.lucene.util.RamUsageEstimator.NUM_BYTES_OBJECT_REF;
import java.util.Arrays;
-import java.util.List;
/**
- * Represents a logical byte[] as a series of blocks. You can write into it by using append and read
- * using the offset position (random access). The buffers can be reset to reuse the allocated
- * buffers.
+ * This class enables the allocation of fixed-size buffers and their management as part of a buffer
+ * array. Allocation is done through the use of an {@link Allocator} which can be customized, e.g.
+ * to allow recycling old buffers. There are methods for writing ({@link #append(BytesRef)} and
+ * reading from the buffers (e.g. {@link #readBytes(long, byte[], int, int)}, which handle
+ * read/write operations across buffer boundaries.
*
* @lucene.internal
*/
@@ -32,8 +33,25 @@ public final class ByteBlockPool implements Accountable {
private static final long BASE_RAM_BYTES =
RamUsageEstimator.shallowSizeOfInstance(ByteBlockPool.class);
+ /**
+ * Use this to find the index of the buffer containing a byte, given an offset to that byte.
+ *
+ * <p>bufferUpto = globalOffset >> BYTE_BLOCK_SHIFT
+ *
+ * <p>bufferUpto = globalOffset / BYTE_BLOCK_SIZE
+ */
public static final int BYTE_BLOCK_SHIFT = 15;
+
+ /** The size of each buffer in the pool. */
public static final int BYTE_BLOCK_SIZE = 1 << BYTE_BLOCK_SHIFT;
+
+ /**
+ * Use this to find the position of a global offset in a particular buffer.
+ *
+ * <p>positionInCurrentBuffer = globalOffset & BYTE_BLOCK_MASK
+ *
+ * <p>positionInCurrentBuffer = globalOffset % BYTE_BLOCK_SIZE
+ */
public static final int BYTE_BLOCK_MASK = BYTE_BLOCK_SIZE - 1;
/** Abstract class for allocating and freeing byte blocks. */
@@ -46,11 +64,6 @@ public final class ByteBlockPool implements Accountable {
public abstract void recycleByteBlocks(byte[][] blocks, int start, int end);
- public void recycleByteBlocks(List<byte[]> blocks) {
- final byte[][] b = blocks.toArray(new byte[blocks.size()][]);
- recycleByteBlocks(b, 0, b.length);
- }
-
public byte[] getByteBlock() {
return new byte[blockSize];
}
@@ -90,24 +103,24 @@ public final class ByteBlockPool implements Accountable {
}
}
}
- ;
- /**
- * array of buffers currently used in the pool. Buffers are allocated if needed don't modify this
- * outside of this class.
- */
- public byte[][] buffers = new byte[10][];
+ /** Array of buffers currently used in the pool. Buffers are allocated if needed. */
+ private byte[][] buffers = new byte[10][];
/** index into the buffers array pointing to the current buffer used as the head */
private int bufferUpto = -1; // Which buffer we are upto
- /** Where we are in head buffer */
+ /** Where we are in the head buffer. */
public int byteUpto = BYTE_BLOCK_SIZE;
- /** Current head buffer */
+ /** Current head buffer. */
public byte[] buffer;
- /** Current head offset */
+ /**
+ * Offset from the start of the first buffer to the start of the current buffer, which is
+ * bufferUpto * BYTE_BLOCK_SIZE. The buffer pool maintains this offset because it is the first to
+ * overflow if there are too many allocated blocks.
+ */
public int byteOffset = -BYTE_BLOCK_SIZE;
private final Allocator allocator;
@@ -117,21 +130,22 @@ public final class ByteBlockPool implements Accountable {
}
/**
- * Resets the pool to its initial state reusing the first buffer and fills all buffers with <code>
- * 0</code> bytes before they reused or passed to {@link Allocator#recycleByteBlocks(byte[][],
- * int, int)}. Calling {@link ByteBlockPool#nextBuffer()} is not needed after reset.
+ * Resets the pool to its initial state, reusing the first buffer and filling all buffers with
+ * {@code 0} bytes before they are reused or passed to {@link
+ * Allocator#recycleByteBlocks(byte[][], int, int)}. Calling {@link ByteBlockPool#nextBuffer()} is
+ * not needed after reset.
*/
public void reset() {
reset(true, true);
}
/**
- * Expert: Resets the pool to its initial state reusing the first buffer. Calling {@link
+ * Expert: Resets the pool to its initial state, while reusing the first buffer. Calling {@link
* ByteBlockPool#nextBuffer()} is not needed after reset.
*
- * @param zeroFillBuffers if <code>true</code> the buffers are filled with <code>0</code>. This
- * should be set to <code>true</code> if this pool is used with slices.
- * @param reuseFirst if <code>true</code> the first buffer will be reused and calling {@link
+ * @param zeroFillBuffers if {@code true} the buffers are filled with {@code 0}. This should be
+ * set to {@code true} if this pool is used with slices.
+ * @param reuseFirst if {@code true} the first buffer will be reused and calling {@link
* ByteBlockPool#nextBuffer()} is not needed after reset iff the block pool was used before
* ie. {@link ByteBlockPool#nextBuffer()} was called before.
*/
@@ -170,42 +184,42 @@ public final class ByteBlockPool implements Accountable {
}
/**
- * Advances the pool to its next buffer. This method should be called once after the constructor
- * to initialize the pool. In contrast to the constructor a {@link ByteBlockPool#reset()} call
- * will advance the pool to its first buffer immediately.
+ * Allocates a new buffer and advances the pool to it. This method should be called once after the
+ * constructor to initialize the pool. In contrast to the constructor, a {@link
+ * ByteBlockPool#reset()} call will advance the pool to its first buffer immediately.
*/
public void nextBuffer() {
if (1 + bufferUpto == buffers.length) {
+ // The buffer array is full - expand it
byte[][] newBuffers =
new byte[ArrayUtil.oversize(buffers.length + 1, NUM_BYTES_OBJECT_REF)][];
System.arraycopy(buffers, 0, newBuffers, 0, buffers.length);
buffers = newBuffers;
}
+ // Allocate new buffer and advance the pool to it
buffer = buffers[1 + bufferUpto] = allocator.getByteBlock();
bufferUpto++;
-
byteUpto = 0;
byteOffset = Math.addExact(byteOffset, BYTE_BLOCK_SIZE);
}
/**
- * Fill the provided {@link BytesRef} with the bytes at the specified offset/length slice. This
- * will avoid copying the bytes, if the slice fits into a single block; otherwise, it uses the
- * provided {@link BytesRefBuilder} to copy bytes over.
+ * Fill the provided {@link BytesRef} with the bytes at the specified offset and length. This will
+ * avoid copying the bytes if the slice fits into a single block; otherwise, it uses the provided
+ * {@link BytesRefBuilder} to copy bytes over.
*/
void setBytesRef(BytesRefBuilder builder, BytesRef result, long offset, int length) {
result.length = length;
- int bufferIndex = (int) (offset >> BYTE_BLOCK_SHIFT);
+ int bufferIndex = Math.toIntExact(offset >> BYTE_BLOCK_SHIFT);
byte[] buffer = buffers[bufferIndex];
int pos = (int) (offset & BYTE_BLOCK_MASK);
if (pos + length <= BYTE_BLOCK_SIZE) {
- // common case where the slice lives in a single block: just reference the buffer directly
- // without copying
+ // Common case: The slice lives in a single block. Reference the buffer directly.
result.bytes = buffer;
result.offset = pos;
} else {
- // uncommon case: the slice spans at least 2 blocks, so we must copy the bytes:
+ // Uncommon case: The slice spans at least 2 blocks, so we must copy the bytes.
builder.grow(length);
result.bytes = builder.get().bytes;
result.offset = 0;
@@ -244,7 +258,7 @@ public final class ByteBlockPool implements Accountable {
*/
public void readBytes(final long offset, final byte[] bytes, int bytesOffset, int bytesLength) {
int bytesLeft = bytesLength;
- int bufferIndex = (int) (offset >> BYTE_BLOCK_SHIFT);
+ int bufferIndex = Math.toIntExact(offset >> BYTE_BLOCK_SHIFT);
int pos = (int) (offset & BYTE_BLOCK_MASK);
while (bytesLeft > 0) {
byte[] buffer = buffers[bufferIndex++];
@@ -259,14 +273,20 @@ public final class ByteBlockPool implements Accountable {
@Override
public long ramBytesUsed() {
long size = BASE_RAM_BYTES;
- size += RamUsageEstimator.sizeOfObject(buffer);
size += RamUsageEstimator.shallowSizeOf(buffers);
for (byte[] buf : buffers) {
- if (buf == buffer) {
- continue;
- }
size += RamUsageEstimator.sizeOfObject(buf);
}
return size;
}
+
+ /** the current position (in absolute value) of this byte pool */
+ public long getPosition() {
+ return bufferUpto * allocator.blockSize + byteUpto;
+ }
+
+ /** Retrieve the buffer at the specified index from the buffer pool. */
+ public byte[] getBuffer(int bufferIndex) {
+ return buffers[bufferIndex];
+ }
}
diff --git a/lucene/core/src/java/org/apache/lucene/util/BytesRefBlockPool.java b/lucene/core/src/java/org/apache/lucene/util/BytesRefBlockPool.java
index c15972438dd..57194a400ef 100644
--- a/lucene/core/src/java/org/apache/lucene/util/BytesRefBlockPool.java
+++ b/lucene/core/src/java/org/apache/lucene/util/BytesRefBlockPool.java
@@ -60,7 +60,7 @@ public class BytesRefBlockPool implements Accountable {
* @see #fillBytesRef(BytesRef, int)
*/
public void fillBytesRef(BytesRef term, int start) {
- final byte[] bytes = term.bytes = byteBlockPool.buffers[start >> BYTE_BLOCK_SHIFT];
+ final byte[] bytes = term.bytes = byteBlockPool.getBuffer(start >> BYTE_BLOCK_SHIFT);
int pos = start & BYTE_BLOCK_MASK;
if ((bytes[pos] & 0x80) == 0) {
// length is 1 byte
@@ -126,7 +126,7 @@ public class BytesRefBlockPool implements Accountable {
*/
int hash(int start) {
final int offset = start & BYTE_BLOCK_MASK;
- final byte[] bytes = byteBlockPool.buffers[start >> BYTE_BLOCK_SHIFT];
+ final byte[] bytes = byteBlockPool.getBuffer(start >> BYTE_BLOCK_SHIFT);
final int len;
int pos;
if ((bytes[offset] & 0x80) == 0) {
@@ -153,7 +153,7 @@ public class BytesRefBlockPool implements Accountable {
* It just saves the work of filling the BytesRef.
*/
boolean equals(int start, BytesRef b) {
- final byte[] bytes = byteBlockPool.buffers[start >> BYTE_BLOCK_SHIFT];
+ final byte[] bytes = byteBlockPool.getBuffer(start >> BYTE_BLOCK_SHIFT);
int pos = start & BYTE_BLOCK_MASK;
final int length;
final int offset;
diff --git a/lucene/core/src/test/org/apache/lucene/index/TestByteSlicePool.java b/lucene/core/src/test/org/apache/lucene/index/TestByteSlicePool.java
new file mode 100644
index 00000000000..30cc11aa31c
--- /dev/null
+++ b/lucene/core/src/test/org/apache/lucene/index/TestByteSlicePool.java
@@ -0,0 +1,279 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.lucene.index;
+
+import org.apache.lucene.tests.util.LuceneTestCase;
+import org.apache.lucene.tests.util.TestUtil;
+import org.apache.lucene.util.BitUtil;
+import org.apache.lucene.util.ByteBlockPool;
+import org.apache.lucene.util.Counter;
+
+public class TestByteSlicePool extends LuceneTestCase {
+ public void testAllocKnownSizeSlice() {
+ Counter bytesUsed = Counter.newCounter();
+ ByteBlockPool blockPool =
+ new ByteBlockPool(new ByteBlockPool.DirectTrackingAllocator(bytesUsed));
+ blockPool.nextBuffer();
+ ByteSlicePool slicePool = new ByteSlicePool(blockPool);
+ for (int i = 0; i < 100; i++) {
+ int size;
+ if (random().nextBoolean()) {
+ size = TestUtil.nextInt(random(), 100, 1000);
+ } else {
+ size = TestUtil.nextInt(random(), 50000, 100000);
+ }
+ byte[] randomData = new byte[size];
+ random().nextBytes(randomData);
+
+ int upto = slicePool.newSlice(ByteSlicePool.FIRST_LEVEL_SIZE);
+
+ for (int offset = 0; offset < size; ) {
+ if ((blockPool.buffer[upto] & 16) == 0) {
+ blockPool.buffer[upto++] = randomData[offset++];
+ } else {
+ int offsetAndLength = slicePool.allocKnownSizeSlice(blockPool.buffer, upto);
+ int sliceLength = offsetAndLength & 0xff;
+ upto = offsetAndLength >> 8;
+ assertNotEquals(0, blockPool.buffer[upto + sliceLength - 1]);
+ assertEquals(0, blockPool.buffer[upto]);
+ int writeLength = Math.min(sliceLength - 1, size - offset);
+ System.arraycopy(randomData, offset, blockPool.buffer, upto, writeLength);
+ offset += writeLength;
+ upto += writeLength;
+ }
+ }
+ }
+ }
+
+ public void testAllocLargeSlice() {
+ ByteBlockPool blockPool = new ByteBlockPool(new ByteBlockPool.DirectAllocator());
+ ByteSlicePool slicePool = new ByteSlicePool(blockPool);
+
+ assertEquals(0, slicePool.newSlice(ByteBlockPool.BYTE_BLOCK_SIZE));
+ assertArrayEquals(blockPool.buffer, blockPool.getBuffer(0));
+
+ blockPool.nextBuffer();
+ assertThrows(
+ IllegalArgumentException.class,
+ () -> slicePool.newSlice(ByteBlockPool.BYTE_BLOCK_SIZE + 1));
+ }
+
+ /** Create a random byte array and write it to a {@link ByteSlicePool} one slice at a time. */
+ static class SliceWriter {
+ boolean hasStarted = false;
+
+ ByteBlockPool blockPool;
+ ByteSlicePool slicePool;
+
+ int size;
+ byte[] randomData;
+ int dataOffset;
+
+ byte[] slice;
+ int sliceLength;
+ int sliceOffset;
+
+ int firstSliceOffset;
+ byte[] firstSlice;
+
+ SliceWriter(ByteSlicePool slicePool) {
+ this.slicePool = slicePool;
+ this.blockPool = slicePool.pool;
+
+ if (random().nextBoolean()) {
+ // size < ByteBlockPool.BYTE_BLOCK_SIZE
+ size = TestUtil.nextInt(random(), 100, 1000);
+ } else {
+ // size > ByteBlockPool.BYTE_BLOCK_SIZE
+ size = TestUtil.nextInt(random(), 50000, 100000);
+ }
+ randomData = new byte[size];
+ random().nextBytes(randomData);
+ }
+
+ /**
+ * Write the next slice of data.
+ *
+ * @return true if we wrote a slice and false if we're out of data to write
+ */
+ boolean writeSlice() {
+ // The first slice is special
+ if (hasStarted == false) {
+ dataOffset = 0; // Offset into the data buffer
+ sliceLength = ByteSlicePool.FIRST_LEVEL_SIZE;
+ sliceOffset = slicePool.newSlice(sliceLength);
+ firstSliceOffset = sliceOffset; // We will need this later
+ firstSlice = blockPool.buffer; // We will need this later
+ slice = firstSlice;
+ int writeLength = Math.min(size, sliceLength - 1);
+ System.arraycopy(randomData, dataOffset, blockPool.buffer, sliceOffset, writeLength);
+ dataOffset += writeLength;
+
+ hasStarted = true;
+ return true;
+ }
+ // Have we written everything?
+ if (dataOffset == size) {
+ return false;
+ }
+ // No, write more
+ int offsetAndLength = slicePool.allocKnownSizeSlice(slice, sliceOffset + sliceLength - 1);
+ slice = blockPool.buffer;
+ sliceLength = offsetAndLength & 0xff;
+ sliceOffset = offsetAndLength >> 8;
+ int writeLength = Math.min(size - dataOffset, sliceLength - 1);
+ System.arraycopy(randomData, dataOffset, slice, sliceOffset, writeLength);
+ dataOffset += writeLength;
+ return true;
+ }
+ }
+
+ /** Read a sequence of slices into a byte array. */
+ static class SliceReader {
+ boolean hasStarted = false;
+
+ ByteBlockPool blockPool;
+ ByteSlicePool slicePool;
+
+ int size;
+ byte[] readData;
+ int dataOffset;
+
+ int sliceLength;
+ int sliceOffset;
+
+ byte[] slice;
+ int sliceSizeIdx;
+
+ SliceReader(ByteSlicePool slicePool, int size, int firstSliceOffset, byte[] firstSlice) {
+ this.slicePool = slicePool;
+ this.blockPool = slicePool.pool;
+ this.size = size;
+ this.sliceOffset = firstSliceOffset;
+ this.slice = firstSlice;
+ readData = new byte[size];
+ }
+
+ /**
+ * Read the next slice of data.
+ *
+ * @return true if we read a slice and false if we'd already read the entire sequence of slices
+ */
+ boolean readSlice() {
+ // The first slice is special
+ if (hasStarted == false) {
+ dataOffset = 0;
+ // Index into LEVEL_SIZE_ARRAY, allowing us to find the size of the current slice
+ sliceSizeIdx = 0;
+ // 4 bytes are for the offset to the next slice, we can't use them for data
+ sliceLength = ByteSlicePool.LEVEL_SIZE_ARRAY[sliceSizeIdx] - 4;
+ int readLength;
+ if (dataOffset + sliceLength + 3 >= size) {
+ // We are reading the last slice, there is no more offset, just a byte for the level
+ readLength = size - dataOffset;
+ } else {
+ readLength = sliceLength;
+ }
+ System.arraycopy(slice, sliceOffset, readData, dataOffset, readLength);
+ dataOffset += readLength;
+ sliceSizeIdx = Math.min(sliceSizeIdx + 1, ByteSlicePool.LEVEL_SIZE_ARRAY.length - 1);
+
+ hasStarted = true;
+ return true;
+ }
+ // Have we read everything?
+ if (dataOffset == size) {
+ return false;
+ }
+ // No, read more
+ int globalSliceOffset = (int) BitUtil.VH_LE_INT.get(slice, sliceOffset + sliceLength);
+ slice = blockPool.getBuffer(globalSliceOffset / ByteBlockPool.BYTE_BLOCK_SIZE);
+ sliceOffset = globalSliceOffset % ByteBlockPool.BYTE_BLOCK_SIZE;
+ sliceLength = ByteSlicePool.LEVEL_SIZE_ARRAY[sliceSizeIdx] - 4;
+ int readLength;
+ if (dataOffset + sliceLength + 3 >= size) {
+ // We are reading the last slice, there is no more offset, just a byte for the level
+ readLength = size - dataOffset;
+ } else {
+ readLength = sliceLength;
+ }
+ System.arraycopy(slice, sliceOffset, readData, dataOffset, readLength);
+ dataOffset += readLength;
+ sliceSizeIdx = Math.min(sliceSizeIdx + 1, ByteSlicePool.LEVEL_SIZE_ARRAY.length - 1);
+ return true;
+ }
+ }
+
+ /**
+ * Run multiple slice writers, creating interleaved slices. Read the slices afterwards and check
+ * that we read back the same data we wrote.
+ */
+ public void testRandomInterleavedSlices() {
+ ByteBlockPool blockPool = new ByteBlockPool(new ByteBlockPool.DirectAllocator());
+ ByteSlicePool slicePool = new ByteSlicePool(blockPool);
+
+ int n = random().nextInt(2, 4); // 2 or 3 writers and readers
+ SliceWriter[] sliceWriters = new SliceWriter[n];
+ SliceReader[] sliceReaders = new SliceReader[n];
+
+ // Init slice writers
+ for (int i = 0; i < n; i++) {
+ sliceWriters[i] = new SliceWriter(slicePool);
+ }
+
+ // Write slices
+ while (true) {
+ int i = random().nextInt(n);
+ boolean succeeded = sliceWriters[i].writeSlice();
+ if (succeeded == false) {
+ for (int j = 0; j < n; j++) {
+ while (sliceWriters[j].writeSlice())
+ ;
+ }
+ break;
+ }
+ }
+
+ // Init slice readers
+ for (int i = 0; i < n; i++) {
+ sliceReaders[i] =
+ new SliceReader(
+ slicePool,
+ sliceWriters[i].size,
+ sliceWriters[i].firstSliceOffset,
+ sliceWriters[i].firstSlice);
+ }
+
+ // Read slices
+ while (true) {
+ int i = random().nextInt(n);
+ boolean succeeded = sliceReaders[i].readSlice();
+ if (succeeded == false) {
+ for (int j = 0; j < n; j++) {
+ while (sliceReaders[j].readSlice())
+ ;
+ }
+ break;
+ }
+ }
+
+ // Compare written data with read data
+ for (int i = 0; i < n; i++) {
+ assertArrayEquals(sliceWriters[i].randomData, sliceReaders[i].readData);
+ }
+ }
+}
diff --git a/lucene/core/src/test/org/apache/lucene/index/TestByteSliceReader.java b/lucene/core/src/test/org/apache/lucene/index/TestByteSliceReader.java
index 403bc41a031..b36108f0e83 100644
--- a/lucene/core/src/test/org/apache/lucene/index/TestByteSliceReader.java
+++ b/lucene/core/src/test/org/apache/lucene/index/TestByteSliceReader.java
@@ -36,11 +36,12 @@ public class TestByteSliceReader extends LuceneTestCase {
BLOCK_POOL = new ByteBlockPool(new ByteBlockPool.DirectAllocator());
BLOCK_POOL.nextBuffer();
+ ByteSlicePool slicePool = new ByteSlicePool(BLOCK_POOL);
byte[] buffer = BLOCK_POOL.buffer;
- int upto = TermsHashPerField.newSlice(BLOCK_POOL, TermsHashPerField.FIRST_LEVEL_SIZE, 0);
+ int upto = slicePool.newSlice(ByteSlicePool.FIRST_LEVEL_SIZE);
for (byte randomByte : RANDOM_DATA) {
if ((buffer[upto] & 16) != 0) {
- upto = TermsHashPerField.allocSlice(BLOCK_POOL, buffer, upto);
+ upto = slicePool.allocSlice(buffer, upto);
buffer = BLOCK_POOL.buffer;
}
buffer[upto++] = randomByte;
diff --git a/lucene/core/src/test/org/apache/lucene/index/TestTermsHashPerField.java b/lucene/core/src/test/org/apache/lucene/index/TestTermsHashPerField.java
index e3e107cf6c7..9c9d39f7092 100644
--- a/lucene/core/src/test/org/apache/lucene/index/TestTermsHashPerField.java
+++ b/lucene/core/src/test/org/apache/lucene/index/TestTermsHashPerField.java
@@ -323,38 +323,4 @@ public class TestTermsHashPerField extends LuceneTestCase {
}
}
}
-
- public void testAllocKnowSizeSlice() {
- Counter bytesUsed = Counter.newCounter();
- ByteBlockPool pool = new ByteBlockPool(new ByteBlockPool.DirectTrackingAllocator(bytesUsed));
- pool.nextBuffer();
- for (int i = 0; i < 100; i++) {
- int size;
- if (random().nextBoolean()) {
- size = TestUtil.nextInt(random(), 100, 1000);
- } else {
- size = TestUtil.nextInt(random(), 50000, 100000);
- }
- byte[] randomData = new byte[size];
- random().nextBytes(randomData);
-
- int upto = TermsHashPerField.newSlice(pool, TermsHashPerField.FIRST_LEVEL_SIZE, 0);
-
- for (int offset = 0; offset < size; ) {
- if ((pool.buffer[upto] & 16) == 0) {
- pool.buffer[upto++] = randomData[offset++];
- } else {
- int offsetAndLength = TermsHashPerField.allocKnownSizeSlice(pool, pool.buffer, upto);
- int sliceLength = offsetAndLength & 0xff;
- upto = offsetAndLength >> 8;
- assertNotEquals(0, pool.buffer[upto + sliceLength - 1]);
- assertEquals(0, pool.buffer[upto]);
- int writeLength = Math.min(sliceLength - 1, size - offset);
- System.arraycopy(randomData, offset, pool.buffer, upto, writeLength);
- offset += writeLength;
- upto += writeLength;
- }
- }
- }
- }
}
(lucene) 02/02: switch to TestUtil for the random int since Random#nextInt(int, int) is not available in Java 11
Posted by mi...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
mikemccand pushed a commit to branch branch_9x
in repository https://gitbox.apache.org/repos/asf/lucene.git
commit b0a1b52e1485919f2a3c0d12b121974567bf2d93
Author: Mike McCandless <mi...@apache.org>
AuthorDate: Mon Nov 6 08:26:06 2023 -0500
switch to TestUtil for the random int since Random#nextInt(int, int) is not available in Java 11
---
lucene/core/src/test/org/apache/lucene/index/TestByteSlicePool.java | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/lucene/core/src/test/org/apache/lucene/index/TestByteSlicePool.java b/lucene/core/src/test/org/apache/lucene/index/TestByteSlicePool.java
index 30cc11aa31c..4e7593960da 100644
--- a/lucene/core/src/test/org/apache/lucene/index/TestByteSlicePool.java
+++ b/lucene/core/src/test/org/apache/lucene/index/TestByteSlicePool.java
@@ -226,7 +226,7 @@ public class TestByteSlicePool extends LuceneTestCase {
ByteBlockPool blockPool = new ByteBlockPool(new ByteBlockPool.DirectAllocator());
ByteSlicePool slicePool = new ByteSlicePool(blockPool);
- int n = random().nextInt(2, 4); // 2 or 3 writers and readers
+ int n = TestUtil.nextInt(random(), 2, 3); // 2 or 3 writers and readers
SliceWriter[] sliceWriters = new SliceWriter[n];
SliceReader[] sliceReaders = new SliceReader[n];