You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@lucene.apache.org by "ChrisHegarty (via GitHub)" <gi...@apache.org> on 2023/08/01 15:57:56 UTC
[GitHub] [lucene] ChrisHegarty opened a new issue, #12482: Examine the potential of loading vector data directly from the memory segment
ChrisHegarty opened a new issue, #12482:
URL: https://github.com/apache/lucene/issues/12482
### Description
Just a crazy idea! Thanks @jpountz ;-)
The Panama Vector API supports for loading directly from a memory segment. If we could do this, then for vector similarity purposes we could load the underlying vector data directly from the MemorySegment (rather than the current copy from the segment into a float[] ).
A quick experiment, if I did it correctly, and only with float dotProduct, shows approx a 2x performance improvement. ( the experiment builds upon @rmuir 's vectorbench, which we used for the original work that integrated the Panama Vector API )
```
Linux, AVX 512
Benchmark (size) Mode Cnt Score Error Units
FloatDotProductBenchmark.dotProductCopyFromArray 1024 thrpt 5 9.714 ± 0.002 ops/us
FloatDotProductBenchmark.dotProductFromMemorySegment 1024 thrpt 5 18.111 ± 0.008 ops/us
```
https://github.com/ChrisHegarty/vectorbench/blob/5380ebf01df77e1c56d9f392a391df780c19d35e/src/main/java/testing/FloatDotProductBenchmark.java
The benchmark method, `dotProductCopyFromArray`, is what we have in Lucene today, but I added a copy from the memory segment into the two float[]'s to simulate `MemorySegmentIndexInput::readFloats`.
The benchmark method, `dotProductFromMemorySegment`, has the exact same shape of vector operations, just that the values are loaded directly from the memory segment using `FloatVector::fromMemorySegment`. A single segment holds both vectors. I suspect we'd want similar in Lucene too is this were ever to proceed.
In practice I don't see any straightforward way that we could approach this, and there is the complication of using preview / Incubating JDK APIs. That aside, I think that the general idea is sound - avoid the extra intermediate copy to the float[].
I filed this issue mostly to raise awareness of `fromMemorySegment`, and also to provoke a discussion and get thoughts from others. Is it worth trying to pursue? Maybe just crudely hack something and get rough luceneutil bench results to see if / how much it could affect things? `MemorySegmentIndexInput` could expose a way to return XXVector's from a given position? Something else?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] ChrisHegarty commented on issue #12482: Examine the potential of loading vector data directly from the memory segment
Posted by "ChrisHegarty (via GitHub)" <gi...@apache.org>.
ChrisHegarty commented on issue #12482:
URL: https://github.com/apache/lucene/issues/12482#issuecomment-1660636619
Note: there is memory segment -> byte buffer interop, which we could use for `MappedByteBufferIndexInputProvider`, so we could do something there too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] uschindler commented on issue #12482: Examine the potential of loading vector data directly from the memory segment
Posted by "uschindler (via GitHub)" <gi...@apache.org>.
uschindler commented on issue #12482:
URL: https://github.com/apache/lucene/issues/12482#issuecomment-1660657815
I have something like this on the agenda but it's currently impossible with non released API and Java 11.
There is another solution: ByteBuffer is still there and you can create vector as view on ByteBuffer. So we could add some DataInput method which returns a byte buffer view on a segment (if the vector is on buffer boundary the copy would of course take place).
Please take in mind that I don't trust the benchmark fully. There are more complications not tested here: you have to make a slice first and so on.
If someone uses another indexinput the default impl would use readBytes or readFloat and wrap it with a heap buffer.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] uschindler commented on issue #12482: Examine the potential of loading vector data directly from the memory segment
Posted by "uschindler (via GitHub)" <gi...@apache.org>.
uschindler commented on issue #12482:
URL: https://github.com/apache/lucene/issues/12482#issuecomment-1660738848
Hi,
> > The problem with older panama vector apis (around java 17) was that there was no real byte/float buffer interop available. Implementations at that time still copied to heap.
>
> Ah ok. I think I'm still missing something. We don't want a ByteBuffer or MemorySegment per vector, right? This would be too much garbage. We just want to return a VectorXX loaded with the values from a given position and length.
It creates garbage in all cases. It does not matter if it returns a slice of a MemorySegment or a Buffer. Both of them will be killed by escape analysis. I see no problem with this. We can't fully remove all instances, because vectors are non mutable. So the DataInput must return a new instance anyways. It does not matter if it is a FloatBuffer or FloatVector.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] ChrisHegarty commented on issue #12482: Examine the potential of loading vector data directly from the memory segment
Posted by "ChrisHegarty (via GitHub)" <gi...@apache.org>.
ChrisHegarty commented on issue #12482:
URL: https://github.com/apache/lucene/issues/12482#issuecomment-1660728819
Thanks for engaging :-)
> * boundary conditions: if the file is ginormous, we don't just map it to a single memory segment but multiple segments i think, how can any new api prevent this complexity from working its way into the vector formulas (which are already complicated enough, this would be too much)
Yeah, as @uschindler said, depending on how this was structured we likely fallback to a copy. I don't expect this to be too common as the default is 16GB. But of course it has to work correctly.
> * alignment issues: afaik, with heap arrays there is some amount of alignment by the jvm, but with mapped files this may not be the case unless we do extra work. do we care? what is the penalty (on common arm/x86 cpus)?
You are spot on, alignment does affect things. As it already does - when reading from an unaligned location in the segment to a float[]. I quickly updated the micro benchmark to read from an offset of one byte in the map file. The readFloats / memory segment copy hurts a lot! While the vector load fromMemorySegment hurts a lot less. This by itself deserves some investigation.
```
Benchmark (size) Mode Cnt Score Error Units
FloatDotProductBenchmark.dotProductCopyFromArray 1024 thrpt 5 0.697 ± 0.002 ops/us
FloatDotProductBenchmark.dotProductFromMemorySegment 1024 thrpt 5 15.864 ± 0.005 ops/us
```
> Please note: the idea for this issue was already discussed in the original mmapdir issues. I proposed this already there.
Ok. So it is not new, and not crazy. Cool!
>The problem with older panama vector apis (around java 17) was that there was no real byte/float buffer interop available. Implementations at that time still copied to heap.
Ah ok. I think I'm still missing something. We don't want a ByteBuffer or MemorySegment per vector, right? This would be too much garbage. We just want to return a VectorXX loaded with the values from a given position and length.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] uschindler commented on issue #12482: Examine the potential of loading vector data directly from the memory segment
Posted by "uschindler (via GitHub)" <gi...@apache.org>.
uschindler commented on issue #12482:
URL: https://github.com/apache/lucene/issues/12482#issuecomment-1660707500
Please note: the idea for this issue was already discussed in the original mmapdir issues. I proposed this already there.
The problem with older panama vector apis (around java 17) was that there was no real byte/float buffer interop available.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] ChrisHegarty commented on issue #12482: Examine the potential of loading vector data directly from the memory segment
Posted by "ChrisHegarty (via GitHub)" <gi...@apache.org>.
ChrisHegarty commented on issue #12482:
URL: https://github.com/apache/lucene/issues/12482#issuecomment-1661909114
While the benchmark is just a micro, it can offer some insights, especially into the aligned / unaligned aspect of this. I updated it with aligned / unaligned variants, [here][1]. TLDR there is little difference between aligned and unaligned.
```
Mac m1
Benchmark (size) Mode Cnt Score Error Units
FloatDotProductBenchmark.dotProductCopyFromArrayAligned 1024 thrpt 5 5.592 ± 0.241 ops/us
FloatDotProductBenchmark.dotProductCopyFromArrayUnaligned 1024 thrpt 5 4.871 ± 0.059 ops/us
FloatDotProductBenchmark.dotProductFromMemorySegmentAligned 1024 thrpt 5 6.792 ± 0.119 ops/us
FloatDotProductBenchmark.dotProductFromMemorySegmentUnaligned 1024 thrpt 5 6.789 ± 0.178 ops/us
```
```
Linux rocket lake
Benchmark (size) Mode Cnt Score Error Units
FloatDotProductBenchmark.dotProductCopyFromArrayAligned 1024 thrpt 5 10.513 ± 0.035 ops/us
FloatDotProductBenchmark.dotProductCopyFromArrayUnaligned 1024 thrpt 5 10.389 ± 0.005 ops/us
FloatDotProductBenchmark.dotProductFromMemorySegmentAligned 1024 thrpt 5 15.765 ± 0.022 ops/us
FloatDotProductBenchmark.dotProductFromMemorySegmentUnaligned 1024 thrpt 5 14.877 ± 0.025 ops/us
```
[1]: https://github.com/chrishegarty/vectorbench/tree/fromMemorySegment
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org
[GitHub] [lucene] rmuir commented on issue #12482: Examine the potential of loading vector data directly from the memory segment
Posted by "rmuir (via GitHub)" <gi...@apache.org>.
rmuir commented on issue #12482:
URL: https://github.com/apache/lucene/issues/12482#issuecomment-1660636776
makes sense, a few concerns of mine:
* boundary conditions: if the file is ginormous, we don't just map it to a single memory segment but multiple segments i think, how can any new api prevent this complexity from working its way into the vector formulas (which are already complicated enough, this would be too much)
* alignment issues: afaik, with heap arrays there is some amount of alignment by the jvm, but with mapped files this may not be the case unless we do extra work. do we care? what is the penalty (on common arm/x86 cpus)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@lucene.apache.org
For additional commands, e-mail: issues-help@lucene.apache.org