You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Lukas Zapletal <lu...@qcm.cz> on 2005/09/06 13:15:02 UTC
Re: Uneffective writeBytes and readBytes [FIX]
And here comes the fixes:
OutputStream:
/**
* Writes an array of bytes.
*
* @param b
* the bytes to write
* @param length
* the number of bytes to write
* @see InputStream#readBytes(byte[],int,int)
*/
public final void writeBytes(byte[] b, int length)
throws IOException {
// for (int i = 0; i < length; i++) writeByte(b[i]);
if (bufferPosition > 0) // flush buffer
flush();
if (length < BUFFER_SIZE) {
flushBuffer(b, length);
} else {
int pos = 0;
int size;
while (pos < length) {
if (length - pos < BUFFER_SIZE) {
size = length - pos;
} else {
size = BUFFER_SIZE;
}
System.arraycopy(b, pos,
buffer, 0, size); pos += size;
flushBuffer(buffer, size);
bufferStart += size;
}
}
}
InputStream:
public final void readBytes(byte[] b, int offset, int len)
throws IOException {
// if (len < BUFFER_SIZE) { // not required
// for (int i = 0; i < len; i++)
// // read byte-by-byte
// b[i + offset] = (byte) readByte();
// } else { // read all-at-once
long start = getFilePointer();
seekInternal(start);
readInternal(b, offset, len);
bufferStart = start + len;
bufferPosition = 0;
bufferLength = 0;
// }
}
You can try to test it in the SVN version in BufferedIndexInput and
BufferedIndexOutput, there are these methods too.
There is significant time improvement for writing and slight for
reading. I also recommend set the buffer to 8 or 16 kilobytes.
--
S pozdravem / Best regards
Lukas Zapletal
sefredaktor / editor-in-chief
LinuxEXPRES - opravdovy linuxovy magazin
www.LinuxEXPRES.cz
tel.:+420 777 003 843
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: Delaying buffer allocation in BufferedIndexInput
Posted by Doug Cutting <cu...@apache.org>.
Paul Elschot wrote:
> I tried delaying the buffer allocation in BufferedIndexInput by
> using this clone() method:
>
> public Object clone() {
> BufferedIndexInput clone = (BufferedIndexInput)super.clone();
> clone.buffer = null;
> clone.bufferLength = 0;
> clone.bufferPosition = 0;
> clone.bufferStart = getFilePointer();
> return clone;
> }
>
> As far as I can see, this delaying should work, but it doesn't and
> I have no idea why.
At a glance it looks good to me, so I am also baffled.
> I noticed that RAMIndexInput extends BufferedIndexInput.
> It has all data in buffers already, so why is there another
> layer of buffering?
No good reason: it's historical.
To avoid this either: (a) the BufferedIndexInput API would need to be
modified to permit subclasses to supply the buffer; or (b)
RAMInputStream could subclass IndexInput directly, using its own
buffers. The latter would probably be simpler.
It would be interesting to see if this made things much faster...
Doug
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Delaying buffer allocation in BufferedIndexInput
Posted by Paul Elschot <pa...@xs4all.nl>.
On Friday 09 September 2005 00:34, Doug Cutting wrote:
> Paul Elschot wrote:
> > I suppose one of these cases are when many terms are used in a query.
> > Would it be easily possible to make the buffer size for a term iterator
> > depend on the numbers of documents to be iterated?
> > Many terms only occur in a few documents, so this could be a
> > nice win on total buffer size for the many terms case.
>
> This would not be too difficult.
>
> Look in SegmentTermDocs.java. The buffer may be allocated when the
> parent's stream is first cloned, but clone() won't allocate a buffer if
> the source hasn't had a buffer allocated yet, and nothing should perform
> i/o directly on the parent's freqStream, so in practice a buffer should
> not be allocated until the first read is performed on the clone.
I tried delaying the buffer allocation in BufferedIndexInput by
using this clone() method:
public Object clone() {
BufferedIndexInput clone = (BufferedIndexInput)super.clone();
clone.buffer = null;
clone.bufferLength = 0;
clone.bufferPosition = 0;
clone.bufferStart = getFilePointer();
return clone;
}
With this all term document iterators seem to be empty, no
query in the test cases gives any results, for example TestDemo
and TestBoolean2.
As far as I can see, this delaying should work, but it doesn't and
I have no idea why.
> So one could add an BufferedIndexInput.setBufferSize() method and then
> call it in SegmentTermDocs.seek(TermInfo), when the df is known and a
> buffer has not yet been allocated.
Indeed that looks easy enough. Now, if I could only delay
the buffer allocation...
I noticed that RAMIndexInput extends BufferedIndexInput.
It has all data in buffers already, so why is there another
layer of buffering?
Regards,
Paul Elschot
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: Uneffective writeBytes and readBytes [FIX]
Posted by Doug Cutting <cu...@apache.org>.
Paul Elschot wrote:
> I suppose one of these cases are when many terms are used in a query.
> Would it be easily possible to make the buffer size for a term iterator
> depend on the numbers of documents to be iterated?
> Many terms only occur in a few documents, so this could be a
> nice win on total buffer size for the many terms case.
This would not be too difficult.
Look in SegmentTermDocs.java. The buffer may be allocated when the
parent's stream is first cloned, but clone() won't allocate a buffer if
the source hasn't had a buffer allocated yet, and nothing should perform
i/o directly on the parent's freqStream, so in practice a buffer should
not be allocated until the first read is performed on the clone.
So one could add an BufferedIndexInput.setBufferSize() method and then
call it in SegmentTermDocs.seek(TermInfo), when the df is known and a
buffer has not yet been allocated.
Doug
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: Uneffective writeBytes and readBytes [FIX]
Posted by Paul Elschot <pa...@xs4all.nl>.
On Thursday 08 September 2005 23:17, Doug Cutting wrote:
> I don't in general disagree with this sort of optimization, but I think
> a good fix is a bit more complicated than what you posted.
>
> Lukas Zapletal wrote:
> > And here comes the fixes:
> >
...
>
> > There is significant time improvement for writing and slight for
> > reading. I also recommend set the buffer to 8 or 16 kilobytes.
>
> In certain cases Lucene allocates many stream buffers. Making these
> larger can thus greatly increase the amount of memory used. Also, the
> filesystem should optimize sequential reads so that the primary
> improvement seen with a larger buffer size is fewer system calls. In my
> experience, a buffer of 1k or so is usually large enough so that the
> system call overheads are minimal.
>
I suppose one of these cases are when many terms are used in a query.
Would it be easily possible to make the buffer size for a term iterator
depend on the numbers of documents to be iterated?
Many terms only occur in a few documents, so this could be a
nice win on total buffer size for the many terms case.
Regards,
Paul Elschot
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: Uneffective writeBytes [FIXED PATCH]
Posted by Lukas Zapletal <lu...@qcm.cz>.
> I don't in general disagree with this sort of optimization, but I think
> a good fix is a bit more complicated than what you posted.
Hi Doug, I had some time to redo my fixes...
> This forces a flush() each time a byte array of any size is written.
> That could be much slower when lots of small byte arrays are written,
> since flush() invokes a system call. What would be best is, if there is
Yes, I made a better version in an exact way you have described above.
The patch replaces writeBytes method with the whole new implementation:
public void writeBytes(byte[] b, int length) throws IOException {
int bytesLeft = BUFFER_SIZE - bufferPosition;
// is there enough space in the buffer?
if (bytesLeft >= length) {
// we add the data to the end of the buffer
System.arraycopy(b, 0, buffer, bufferPosition, length);
bufferPosition += length;
// if the buffer is full, flush it
if (BUFFER_SIZE - bufferPosition == 0)
flush();
} else {
// we fill/flush the buffer (until the input is written)
int pos = 0; // position in the input data
int pieceLength;
while (pos < length) {
pieceLength = (length - pos < bytesLeft) ?
length - pos : bytesLeft;
System.arraycopy(b, pos, buffer,
bufferPosition, pieceLength);
pos += pieceLength;
bufferPosition += pieceLength;
// if the buffer is full, flush it
bytesLeft = BUFFER_SIZE - bufferPosition;
if (bytesLeft == 0) {
flush();
bytesLeft = BUFFER_SIZE;
}
}
}
}
This is much smarter and it is not slow for lots of one-byte writes. I
agree with the assession of my "fix" of readBytes:
> Again, this could be much slower when lots of small arrays are written,
> since each call forces seek and read system calls. However this could
> be optimized for the case where the desired data resides entirely in the
> current buffer to use System.arraycopy.
So I tried to optimize the reads from buffer, but the method was so
complicated I dropped this thought. Readbytes is fast enough.
I have modified the StoreTest, now you can test the *byte or *bytes
methods (3rd parameter). Because of creating and testing the buffers,
there is not speed improvement here but when you write from some
prepared buffer (e.g. from a file), the speed increases.
--
S pozdravem / Best regards
Lukas Zapletal
sefredaktor / editor-in-chief
LinuxEXPRES - opravdovy linuxovy magazin
www.LinuxEXPRES.cz
tel.:+420 777 003 843
Re: Uneffective writeBytes and readBytes [FIX]
Posted by Lukas Zapletal <lu...@qcm.cz>.
Dne Thu, 08 Sep 2005 14:17:57 -0700
Doug Cutting <cu...@apache.org> napsal(a):
> This forces a flush() each time a byte array of any size is written.
Oh I didnt scroll down the message, I have read this today.
I see, I thought Lucene uses writeByte instead. I will reimplement the
patch, sure.
--
S pozdravem / Best regards
Lukas Zapletal
sefredaktor / editor-in-chief
LinuxEXPRES - opravdovy linuxovy magazin
www.LinuxEXPRES.cz
tel.:+420 777 003 843
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: Uneffective writeBytes and readBytes [FIX]
Posted by Doug Cutting <cu...@apache.org>.
I don't in general disagree with this sort of optimization, but I think
a good fix is a bit more complicated than what you posted.
Lukas Zapletal wrote:
> And here comes the fixes:
>
> OutputStream:
>
> /**
> * Writes an array of bytes.
> *
> * @param b
> * the bytes to write
> * @param length
> * the number of bytes to write
> * @see InputStream#readBytes(byte[],int,int)
> */
> public final void writeBytes(byte[] b, int length)
> throws IOException {
>
> // for (int i = 0; i < length; i++) writeByte(b[i]);
>
> if (bufferPosition > 0) // flush buffer
> flush();
>
> if (length < BUFFER_SIZE) {
> flushBuffer(b, length);
> } else {
> int pos = 0;
> int size;
> while (pos < length) {
> if (length - pos < BUFFER_SIZE) {
> size = length - pos;
> } else {
> size = BUFFER_SIZE;
> }
> System.arraycopy(b, pos,
> buffer, 0, size); pos += size;
> flushBuffer(buffer, size);
> bufferStart += size;
> }
> }
> }
This forces a flush() each time a byte array of any size is written.
That could be much slower when lots of small byte arrays are written,
since flush() invokes a system call. What would be best is, if there is
room in the buffer, to simply use System.arraycopy to append the new
data to the buffer, with no flush. If the new data is larger than a
buffer, then the buffer should be flushed and the new data written
directly, without ever copying it into the buffer. If the new data is
smaller than a buffer but larger than the room available in the current
buffer, then it should be used to fill the current buffer, that should
be flushed, then the remainder should be copied to the buffer. Does
that sound right?
> InputStream:
>
> public final void readBytes(byte[] b, int offset, int len)
> throws IOException {
> // if (len < BUFFER_SIZE) { // not required
> // for (int i = 0; i < len; i++)
> // // read byte-by-byte
> // b[i + offset] = (byte) readByte();
> // } else { // read all-at-once
> long start = getFilePointer();
> seekInternal(start);
> readInternal(b, offset, len);
>
> bufferStart = start + len;
> bufferPosition = 0;
> bufferLength = 0;
> // }
> }
Again, this could be much slower when lots of small arrays are written,
since each call forces seek and read system calls. However this could
be optimized for the case where the desired data resides entirely in the
current buffer to use System.arraycopy.
> There is significant time improvement for writing and slight for
> reading. I also recommend set the buffer to 8 or 16 kilobytes.
In certain cases Lucene allocates many stream buffers. Making these
larger can thus greatly increase the amount of memory used. Also, the
filesystem should optimize sequential reads so that the primary
improvement seen with a larger buffer size is fewer system calls. In my
experience, a buffer of 1k or so is usually large enough so that the
system call overheads are minimal.
Doug
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: Uneffective writeBytes and readBytes [FIX]
Posted by Maik Schreiber <bl...@blizzy.de>.
> did anyone tested the patches? Any results?
Perhaps you should try and create a real patch (unified diff comes to mind),
and post that one here.
--
Maik Schreiber * http://www.blizzy.de
GPG public key: http://pgp.mit.edu:11371/pks/lookup?op=get&search=0x1F11D713
Key fingerprint: CF19 AFCE 6E3D 5443 9599 18B5 5640 1F11 D713
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: Uneffective writeBytes and readBytes [FIX]
Posted by Lukas Zapletal <lu...@qcm.cz>.
Dne Tue, 6 Sep 2005 13:15:02 +0200
Lukas Zapletal <lu...@qcm.cz> napsal(a):
> And here comes the fixes:
Hi there,
did anyone tested the patches? Any results?
--
S pozdravem / Best regards
Lukas Zapletal
sefredaktor / editor-in-chief
LinuxEXPRES - opravdovy linuxovy magazin
www.LinuxEXPRES.cz
tel.:+420 777 003 843
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: Uneffective writeBytes and readBytes [FIX]
Posted by Lukas Zapletal <lu...@qcm.cz>.
Dne Thu, 8 Sep 2005 05:45:59 -0700 (PDT)
Otis Gospodnetic <ot...@yahoo.com> napsal(a):
> Did you measure, or at least observe the performance improvement this
> change made?
Oh yes!
In the indexing of one 200MB HTML archive, there was no measurable
performance gain, BUT when calling method writeBytes in one sequence
(e.g. in my URLDirectory implementation - in the Bugzilla already - that
loads the index from a ZIP file), there is about 100-150% gain.
I have cleaned the code and fixed two mistakes. If are you interested,
please let ke know and I will create an unified patch against current
version from repository.
--
S pozdravem / Best regards
Lukas Zapletal
sefredaktor / editor-in-chief
LinuxEXPRES - opravdovy linuxovy magazin
www.LinuxEXPRES.cz
tel.:+420 777 003 843
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
Re: Uneffective writeBytes and readBytes [FIX]
Posted by Otis Gospodnetic <ot...@yahoo.com>.
Hi Lukas,
Did you measure, or at least observe the performance improvement this
change made?
The changes look fine to me, although there are comments next to code
for reading/writing 1 byte at a time (the non-buffered part), so Doug
obviously wanted it that way, and I don't know why...
Otis
--- Lukas Zapletal <lu...@qcm.cz> wrote:
> And here comes the fixes:
>
> OutputStream:
>
> /**
> * Writes an array of bytes.
> *
> * @param b
> * the bytes to write
> * @param length
> * the number of bytes to write
> * @see InputStream#readBytes(byte[],int,int)
> */
> public final void writeBytes(byte[] b, int length)
> throws IOException {
>
> // for (int i = 0; i < length; i++) writeByte(b[i]);
>
> if (bufferPosition > 0) // flush buffer
> flush();
>
> if (length < BUFFER_SIZE) {
> flushBuffer(b, length);
> } else {
> int pos = 0;
> int size;
> while (pos < length) {
> if (length - pos < BUFFER_SIZE) {
> size = length - pos;
> } else {
> size = BUFFER_SIZE;
> }
> System.arraycopy(b, pos,
> buffer, 0, size); pos += size;
> flushBuffer(buffer, size);
> bufferStart += size;
> }
> }
> }
>
> InputStream:
>
> public final void readBytes(byte[] b, int offset, int len)
> throws IOException {
> // if (len < BUFFER_SIZE) { // not required
> // for (int i = 0; i < len; i++)
> // // read byte-by-byte
> // b[i + offset] = (byte) readByte();
> // } else { // read all-at-once
> long start = getFilePointer();
> seekInternal(start);
> readInternal(b, offset, len);
>
> bufferStart = start + len;
> bufferPosition = 0;
> bufferLength = 0;
> // }
> }
>
> You can try to test it in the SVN version in BufferedIndexInput and
> BufferedIndexOutput, there are these methods too.
>
> There is significant time improvement for writing and slight for
> reading. I also recommend set the buffer to 8 or 16 kilobytes.
>
> --
> S pozdravem / Best regards
> Lukas Zapletal
> sefredaktor / editor-in-chief
> LinuxEXPRES - opravdovy linuxovy magazin
> www.LinuxEXPRES.cz
> tel.:+420 777 003 843
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org