You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Renaud Delbru (JIRA)" <ji...@apache.org> on 2010/01/18 01:18:54 UTC
[jira] Created: (LUCENE-2222) FixedIntBlockIndexInput.Reader does
not initialise 'pending' int array
FixedIntBlockIndexInput.Reader does not initialise 'pending' int array
----------------------------------------------------------------------
Key: LUCENE-2222
URL: https://issues.apache.org/jira/browse/LUCENE-2222
Project: Lucene - Java
Issue Type: Bug
Components: Index
Affects Versions: Flex Branch
Reporter: Renaud Delbru
Fix For: Flex Branch
The FixedIntBlockIndexInput.Reader.pending int array is not initialised. As a consequence, the FixedIntBlockIndexInput.Reader#next() method returns always 0.
A call to FixedIntBlockIndexInput.Reader#blockReader.readBlock() during the Reader initialisation may solve the issue (to be tested).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2222) FixedIntBlockIndexInput.Reader does
not initialise 'pending' int array
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12801775#action_12801775 ]
Michael McCandless commented on LUCENE-2222:
--------------------------------------------
OK I'll commit shortly.
{quote}
For the moment, I first try to use FrameOfRef, and compare it with some simpler encoding methods such as VInt using the Codec interface. I would like to see if the BlockReader and Reader interface do not add too much overhead compared to a simple index input based on vint, and therefore loose the speed benefits we got on decompression.
{quote}
OK -- sounds interesting -- please report back!
bq. P.S.: Sorry for the first patches, my eclipse is configured to format automatically the code (add final, this., etc.) and therefore my patches are generally including all these modifications. Is it a problem for you ?
Actually I don't mind these additions...
> FixedIntBlockIndexInput.Reader does not initialise 'pending' int array
> ----------------------------------------------------------------------
>
> Key: LUCENE-2222
> URL: https://issues.apache.org/jira/browse/LUCENE-2222
> Project: Lucene - Java
> Issue Type: Bug
> Components: Index
> Affects Versions: Flex Branch
> Reporter: Renaud Delbru
> Priority: Minor
> Fix For: Flex Branch
>
> Attachments: LUCENE-2222.patch, LUCENE-2222.patch, LUCENE-2222.patch
>
>
> The FixedIntBlockIndexInput.Reader.pending int array is not initialised. As a consequence, the FixedIntBlockIndexInput.Reader#next() method returns always 0.
> A call to FixedIntBlockIndexInput.Reader#blockReader.readBlock() during the Reader initialisation may solve the issue (to be tested).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Updated: (LUCENE-2222) FixedIntBlockIndexInput.Reader does
not initialise 'pending' int array
Posted by "Renaud Delbru (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2222?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Renaud Delbru updated LUCENE-2222:
----------------------------------
Attachment: LUCENE-2222.patch
Fixed patch
> FixedIntBlockIndexInput.Reader does not initialise 'pending' int array
> ----------------------------------------------------------------------
>
> Key: LUCENE-2222
> URL: https://issues.apache.org/jira/browse/LUCENE-2222
> Project: Lucene - Java
> Issue Type: Bug
> Components: Index
> Affects Versions: Flex Branch
> Reporter: Renaud Delbru
> Priority: Minor
> Fix For: Flex Branch
>
> Attachments: LUCENE-2222.patch, LUCENE-2222.patch
>
>
> The FixedIntBlockIndexInput.Reader.pending int array is not initialised. As a consequence, the FixedIntBlockIndexInput.Reader#next() method returns always 0.
> A call to FixedIntBlockIndexInput.Reader#blockReader.readBlock() during the Reader initialisation may solve the issue (to be tested).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Updated: (LUCENE-2222) FixedIntBlockIndexInput.Reader does
not initialise 'pending' int array
Posted by "Renaud Delbru (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2222?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Renaud Delbru updated LUCENE-2222:
----------------------------------
Priority: Minor (was: Major)
> FixedIntBlockIndexInput.Reader does not initialise 'pending' int array
> ----------------------------------------------------------------------
>
> Key: LUCENE-2222
> URL: https://issues.apache.org/jira/browse/LUCENE-2222
> Project: Lucene - Java
> Issue Type: Bug
> Components: Index
> Affects Versions: Flex Branch
> Reporter: Renaud Delbru
> Priority: Minor
> Fix For: Flex Branch
>
>
> The FixedIntBlockIndexInput.Reader.pending int array is not initialised. As a consequence, the FixedIntBlockIndexInput.Reader#next() method returns always 0.
> A call to FixedIntBlockIndexInput.Reader#blockReader.readBlock() during the Reader initialisation may solve the issue (to be tested).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Updated: (LUCENE-2222) FixedIntBlockIndexInput.Reader does
not initialise 'pending' int array
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2222?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Michael McCandless updated LUCENE-2222:
---------------------------------------
Attachment: LUCENE-2222.patch
It's great that you're working with the intblock codec, Renaud!
Are you using PForDelta? Or some other means of encoding?
I think, instead of reading a block in the ctor for Reader, we should set upto = blockSize? Ie, the file may have nothing it it, so we should defer reading the block until the first read. I added 2 unit tests showing the issues. Attached patch.
> FixedIntBlockIndexInput.Reader does not initialise 'pending' int array
> ----------------------------------------------------------------------
>
> Key: LUCENE-2222
> URL: https://issues.apache.org/jira/browse/LUCENE-2222
> Project: Lucene - Java
> Issue Type: Bug
> Components: Index
> Affects Versions: Flex Branch
> Reporter: Renaud Delbru
> Priority: Minor
> Fix For: Flex Branch
>
> Attachments: LUCENE-2222.patch, LUCENE-2222.patch, LUCENE-2222.patch
>
>
> The FixedIntBlockIndexInput.Reader.pending int array is not initialised. As a consequence, the FixedIntBlockIndexInput.Reader#next() method returns always 0.
> A call to FixedIntBlockIndexInput.Reader#blockReader.readBlock() during the Reader initialisation may solve the issue (to be tested).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2222) FixedIntBlockIndexInput.Reader does
not initialise 'pending' int array
Posted by "Renaud Delbru (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12801770#action_12801770 ]
Renaud Delbru commented on LUCENE-2222:
---------------------------------------
For the moment, I first try to use FrameOfRef, and compare it with some simpler encoding methods such as VInt using the Codec interface. I would like to see if the BlockReader and Reader interface do not add too much overhead compared to a simple index input based on vint, and therefore loose the speed benefits we got on decompression.
I agree with your modifications, it is more clean. I have applied the patch, and run the tests, all seems to run without problems.
P.S.: I had some problems (errors on CHANGES.txt and FixedIntBlockIndexInput.java) when applying the patch (within eclipse). I had to do it manually, Don't know what is the cause.
P.S.: Sorry for the first patches, my eclipse is configured to format automatically the code (add final, this., etc.) and therefore my patches are generally including all these modifications. Is it a problem for you ?
> FixedIntBlockIndexInput.Reader does not initialise 'pending' int array
> ----------------------------------------------------------------------
>
> Key: LUCENE-2222
> URL: https://issues.apache.org/jira/browse/LUCENE-2222
> Project: Lucene - Java
> Issue Type: Bug
> Components: Index
> Affects Versions: Flex Branch
> Reporter: Renaud Delbru
> Priority: Minor
> Fix For: Flex Branch
>
> Attachments: LUCENE-2222.patch, LUCENE-2222.patch, LUCENE-2222.patch
>
>
> The FixedIntBlockIndexInput.Reader.pending int array is not initialised. As a consequence, the FixedIntBlockIndexInput.Reader#next() method returns always 0.
> A call to FixedIntBlockIndexInput.Reader#blockReader.readBlock() during the Reader initialisation may solve the issue (to be tested).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2222) FixedIntBlockIndexInput.Reader does
not initialise 'pending' int array
Posted by "Paul Elschot (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12801895#action_12801895 ]
Paul Elschot commented on LUCENE-2222:
--------------------------------------
ForDecompress.decodeAnyFrame() is pretty slow, it is mainly there as a fallback and I have also used it for testing initially.
For decoding blocks just shorter than 32, a few Simple9 blocks will be faster than vByte.
Simple16 (4 bits for the cases, just as Simple9) would be good. I remember looking for a link to an implementation of Simple16 in one of the articles on Simple9, but the linked page had gone, so I just went ahead myself and started on Simple9.
One can also use 32 or 64 cases (5 or 6 bits for the cases) and allow a variable number of ints (or bytes) in such SimpleNN variants. Such variants nicely hit the middle ground between Simple9 and PFOR in block length and decoding speed. Perhaps we'd better continue the discussion on such methods at LUCENE-2189.
When there are sufficient methods available an issue could be opened for a general variable length block encoding/decoding that uses the best of such methods depending on the block length. This makes sense because in lucene indexes the block length is always known.
> FixedIntBlockIndexInput.Reader does not initialise 'pending' int array
> ----------------------------------------------------------------------
>
> Key: LUCENE-2222
> URL: https://issues.apache.org/jira/browse/LUCENE-2222
> Project: Lucene - Java
> Issue Type: Bug
> Components: Index
> Affects Versions: Flex Branch
> Reporter: Renaud Delbru
> Priority: Minor
> Fix For: Flex Branch
>
> Attachments: LUCENE-2222.patch, LUCENE-2222.patch, LUCENE-2222.patch
>
>
> The FixedIntBlockIndexInput.Reader.pending int array is not initialised. As a consequence, the FixedIntBlockIndexInput.Reader#next() method returns always 0.
> A call to FixedIntBlockIndexInput.Reader#blockReader.readBlock() during the Reader initialisation may solve the issue (to be tested).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2222) FixedIntBlockIndexInput.Reader does
not initialise 'pending' int array
Posted by "Paul Elschot (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12801864#action_12801864 ]
Paul Elschot commented on LUCENE-2222:
--------------------------------------
{quote}
I have noticed also another problem with the block index I/O and PFOR I/O. The fixed int block index can be configured with any block size, but PFOR requires at least a block size of 32 (and even, I think it requires a block size which is a product of 32), otherwise the decompression do not work correctly (the inputSize in decompressFrame is based on frameOfRef.unComprSize). There should be a block size checking in the PFOR index I/O. Should I open a new issue ?
{quote}
I have not looked at the flex branch yet, but the PFOR implementation at LUCENE-1410 has the possibility to encode/decode any block length. To decode blocks that are not a multiple of 32 in size it will fall back from the faster decoding routine (the generated code) to the general (and slow) decoding routine at ForDecompress.decodeAnyFrame().
Lately I've been tinkering with Simple9 (LUCENE-2189) which is simpler than PFOR but has the problem that the input block size is not known beforehand. As far as I can see now, we might end up with a general pair of block encoding/decoding routines that uses PFOR for longer blocks, Simple9 (or a variant of that) for the remainder and vByte for whatever is left.
> FixedIntBlockIndexInput.Reader does not initialise 'pending' int array
> ----------------------------------------------------------------------
>
> Key: LUCENE-2222
> URL: https://issues.apache.org/jira/browse/LUCENE-2222
> Project: Lucene - Java
> Issue Type: Bug
> Components: Index
> Affects Versions: Flex Branch
> Reporter: Renaud Delbru
> Priority: Minor
> Fix For: Flex Branch
>
> Attachments: LUCENE-2222.patch, LUCENE-2222.patch, LUCENE-2222.patch
>
>
> The FixedIntBlockIndexInput.Reader.pending int array is not initialised. As a consequence, the FixedIntBlockIndexInput.Reader#next() method returns always 0.
> A call to FixedIntBlockIndexInput.Reader#blockReader.readBlock() during the Reader initialisation may solve the issue (to be tested).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Issue Comment Edited: (LUCENE-2222)
FixedIntBlockIndexInput.Reader does not initialise 'pending' int array
Posted by "Paul Elschot (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12801895#action_12801895 ]
Paul Elschot edited comment on LUCENE-2222 at 1/18/10 6:45 PM:
---------------------------------------------------------------
ForDecompress.decodeAnyFrame() is pretty slow, it is mainly there as a fallback and I have also used it for testing initially.
For decoding blocks just shorter than 32, a few Simple9 blocks will be faster than ForDecompress.decodeAnyFrame().
Simple16 (4 bits for the cases, just as Simple9) would be good. I remember looking for a link to an implementation of Simple16 in one of the articles on Simple9, but the linked page had gone, so I just went ahead myself and started on Simple9.
One can also use 32 or 64 cases (5 or 6 bits for the cases) and allow a variable number of ints (or bytes) in such SimpleNN variants. Such variants nicely hit the middle ground between Simple9 and PFOR in block length and decoding speed. Perhaps we'd better continue the discussion on such methods at LUCENE-2189.
When there are sufficient methods available an issue could be opened for a general variable length block encoding/decoding that uses the best of such methods depending on the block length. This makes sense because in lucene indexes the block length is always known.
was (Author: paul.elschot@xs4all.nl):
ForDecompress.decodeAnyFrame() is pretty slow, it is mainly there as a fallback and I have also used it for testing initially.
For decoding blocks just shorter than 32, a few Simple9 blocks will be faster than vByte.
Simple16 (4 bits for the cases, just as Simple9) would be good. I remember looking for a link to an implementation of Simple16 in one of the articles on Simple9, but the linked page had gone, so I just went ahead myself and started on Simple9.
One can also use 32 or 64 cases (5 or 6 bits for the cases) and allow a variable number of ints (or bytes) in such SimpleNN variants. Such variants nicely hit the middle ground between Simple9 and PFOR in block length and decoding speed. Perhaps we'd better continue the discussion on such methods at LUCENE-2189.
When there are sufficient methods available an issue could be opened for a general variable length block encoding/decoding that uses the best of such methods depending on the block length. This makes sense because in lucene indexes the block length is always known.
> FixedIntBlockIndexInput.Reader does not initialise 'pending' int array
> ----------------------------------------------------------------------
>
> Key: LUCENE-2222
> URL: https://issues.apache.org/jira/browse/LUCENE-2222
> Project: Lucene - Java
> Issue Type: Bug
> Components: Index
> Affects Versions: Flex Branch
> Reporter: Renaud Delbru
> Priority: Minor
> Fix For: Flex Branch
>
> Attachments: LUCENE-2222.patch, LUCENE-2222.patch, LUCENE-2222.patch
>
>
> The FixedIntBlockIndexInput.Reader.pending int array is not initialised. As a consequence, the FixedIntBlockIndexInput.Reader#next() method returns always 0.
> A call to FixedIntBlockIndexInput.Reader#blockReader.readBlock() during the Reader initialisation may solve the issue (to be tested).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Resolved: (LUCENE-2222) FixedIntBlockIndexInput.Reader does
not initialise 'pending' int array
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2222?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Michael McCandless resolved LUCENE-2222.
----------------------------------------
Resolution: Fixed
> FixedIntBlockIndexInput.Reader does not initialise 'pending' int array
> ----------------------------------------------------------------------
>
> Key: LUCENE-2222
> URL: https://issues.apache.org/jira/browse/LUCENE-2222
> Project: Lucene - Java
> Issue Type: Bug
> Components: Index
> Affects Versions: Flex Branch
> Reporter: Renaud Delbru
> Priority: Minor
> Fix For: Flex Branch
>
> Attachments: LUCENE-2222.patch, LUCENE-2222.patch, LUCENE-2222.patch
>
>
> The FixedIntBlockIndexInput.Reader.pending int array is not initialised. As a consequence, the FixedIntBlockIndexInput.Reader#next() method returns always 0.
> A call to FixedIntBlockIndexInput.Reader#blockReader.readBlock() during the Reader initialisation may solve the issue (to be tested).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2222) FixedIntBlockIndexInput.Reader does
not initialise 'pending' int array
Posted by "Renaud Delbru (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12801811#action_12801811 ]
Renaud Delbru commented on LUCENE-2222:
---------------------------------------
Yes, it is something that should be tested in the PFOR codec.
> FixedIntBlockIndexInput.Reader does not initialise 'pending' int array
> ----------------------------------------------------------------------
>
> Key: LUCENE-2222
> URL: https://issues.apache.org/jira/browse/LUCENE-2222
> Project: Lucene - Java
> Issue Type: Bug
> Components: Index
> Affects Versions: Flex Branch
> Reporter: Renaud Delbru
> Priority: Minor
> Fix For: Flex Branch
>
> Attachments: LUCENE-2222.patch, LUCENE-2222.patch, LUCENE-2222.patch
>
>
> The FixedIntBlockIndexInput.Reader.pending int array is not initialised. As a consequence, the FixedIntBlockIndexInput.Reader#next() method returns always 0.
> A call to FixedIntBlockIndexInput.Reader#blockReader.readBlock() during the Reader initialisation may solve the issue (to be tested).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2222) FixedIntBlockIndexInput.Reader does
not initialise 'pending' int array
Posted by "Renaud Delbru (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12801874#action_12801874 ]
Renaud Delbru commented on LUCENE-2222:
---------------------------------------
{quote}
I have not looked at the flex branch yet, but the PFOR implementation at LUCENE-1410 has the possibility to encode/decode any block length. To decode blocks that are not a multiple of 32 in size it will fall back from the faster decoding routine (the generated code) to the general (and slow) decoding routine at ForDecompress.decodeAnyFrame().
{quote}
Ok, I see. I deactivated this functionalities in my FrameOfReference implementation. My mistake.
{quote}
Lately I've been tinkering with Simple9 (LUCENE-2189) which is simpler than PFOR but has the problem that the input block size is not known beforehand. As far as I can see now, we might end up with a general pair of block encoding/decoding routines that uses PFOR for longer blocks, Simple9 (or a variant of that) for the remainder and vByte for whatever is left.
{quote}
I have an implementation of Simple16 (which normally wastes less bits than Simple 9). It is based on some code I found on the Web (a problem is that I don't find anymore its origin. If I remember, it was from an academic project in c) that I translated, fixed and optimised. I can share it if you're interested.
> FixedIntBlockIndexInput.Reader does not initialise 'pending' int array
> ----------------------------------------------------------------------
>
> Key: LUCENE-2222
> URL: https://issues.apache.org/jira/browse/LUCENE-2222
> Project: Lucene - Java
> Issue Type: Bug
> Components: Index
> Affects Versions: Flex Branch
> Reporter: Renaud Delbru
> Priority: Minor
> Fix For: Flex Branch
>
> Attachments: LUCENE-2222.patch, LUCENE-2222.patch, LUCENE-2222.patch
>
>
> The FixedIntBlockIndexInput.Reader.pending int array is not initialised. As a consequence, the FixedIntBlockIndexInput.Reader#next() method returns always 0.
> A call to FixedIntBlockIndexInput.Reader#blockReader.readBlock() during the Reader initialisation may solve the issue (to be tested).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2222) FixedIntBlockIndexInput.Reader does
not initialise 'pending' int array
Posted by "Renaud Delbru (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12801782#action_12801782 ]
Renaud Delbru commented on LUCENE-2222:
---------------------------------------
I have noticed also another problem with the block index I/O and PFOR I/O. The fixed int block index can be configured with any block size, but PFOR requires at least a block size of 32 (and even, I think it requires a block size which is a product of 32), otherwise the decompression do not work correctly (the inputSize in decompressFrame is based on frameOfRef.unComprSize). There should be a block size checking in the PFOR index I/O. Should I open a new issue ?
> FixedIntBlockIndexInput.Reader does not initialise 'pending' int array
> ----------------------------------------------------------------------
>
> Key: LUCENE-2222
> URL: https://issues.apache.org/jira/browse/LUCENE-2222
> Project: Lucene - Java
> Issue Type: Bug
> Components: Index
> Affects Versions: Flex Branch
> Reporter: Renaud Delbru
> Priority: Minor
> Fix For: Flex Branch
>
> Attachments: LUCENE-2222.patch, LUCENE-2222.patch, LUCENE-2222.patch
>
>
> The FixedIntBlockIndexInput.Reader.pending int array is not initialised. As a consequence, the FixedIntBlockIndexInput.Reader#next() method returns always 0.
> A call to FixedIntBlockIndexInput.Reader#blockReader.readBlock() during the Reader initialisation may solve the issue (to be tested).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-2222) FixedIntBlockIndexInput.Reader does
not initialise 'pending' int array
Posted by "Michael McCandless (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12801800#action_12801800 ]
Michael McCandless commented on LUCENE-2222:
--------------------------------------------
This is a check that should be added to the PFOR codec (currently lives as a patch on LUCENE-1410), right?
> FixedIntBlockIndexInput.Reader does not initialise 'pending' int array
> ----------------------------------------------------------------------
>
> Key: LUCENE-2222
> URL: https://issues.apache.org/jira/browse/LUCENE-2222
> Project: Lucene - Java
> Issue Type: Bug
> Components: Index
> Affects Versions: Flex Branch
> Reporter: Renaud Delbru
> Priority: Minor
> Fix For: Flex Branch
>
> Attachments: LUCENE-2222.patch, LUCENE-2222.patch, LUCENE-2222.patch
>
>
> The FixedIntBlockIndexInput.Reader.pending int array is not initialised. As a consequence, the FixedIntBlockIndexInput.Reader#next() method returns always 0.
> A call to FixedIntBlockIndexInput.Reader#blockReader.readBlock() during the Reader initialisation may solve the issue (to be tested).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Updated: (LUCENE-2222) FixedIntBlockIndexInput.Reader does
not initialise 'pending' int array
Posted by "Renaud Delbru (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2222?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Renaud Delbru updated LUCENE-2222:
----------------------------------
Attachment: LUCENE-2222.patch
Simple patch that adds a call to blockReader.readBlock() in the Reader initialisation
> FixedIntBlockIndexInput.Reader does not initialise 'pending' int array
> ----------------------------------------------------------------------
>
> Key: LUCENE-2222
> URL: https://issues.apache.org/jira/browse/LUCENE-2222
> Project: Lucene - Java
> Issue Type: Bug
> Components: Index
> Affects Versions: Flex Branch
> Reporter: Renaud Delbru
> Priority: Minor
> Fix For: Flex Branch
>
> Attachments: LUCENE-2222.patch
>
>
> The FixedIntBlockIndexInput.Reader.pending int array is not initialised. As a consequence, the FixedIntBlockIndexInput.Reader#next() method returns always 0.
> A call to FixedIntBlockIndexInput.Reader#blockReader.readBlock() during the Reader initialisation may solve the issue (to be tested).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Issue Comment Edited: (LUCENE-2222)
FixedIntBlockIndexInput.Reader does not initialise 'pending' int array
Posted by "Paul Elschot (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/LUCENE-2222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12801895#action_12801895 ]
Paul Elschot edited comment on LUCENE-2222 at 1/18/10 6:49 PM:
---------------------------------------------------------------
ForDecompress.decodeAnyFrame() is pretty slow, it is mainly there as a fallback and I have also used it for testing initially.
For decoding blocks just shorter than 32, a few Simple9 blocks will be faster than ForDecompress.decodeAnyFrame().
Simple16 (4 bits for the cases, just as Simple9) would be good. I remember looking for a link to an implementation of Simple16 in one of the articles on Simple9, but the linked page had gone, so I just went ahead myself and started on Simple9.
One can also use 32 or 64 cases (5 or 6 bits for the cases) and use a variable number of ints (or bytes) in the encoded forms of such SimpleNN variants. Such variants nicely hit the middle ground between Simple9 and PFOR in block length and decoding speed. Perhaps we'd better continue the discussion on such methods at LUCENE-2189.
When there are sufficient methods available an issue could be opened for a general variable length block encoding/decoding that uses the best of such methods depending on the block length. This makes sense because in lucene indexes the block length is always known.
was (Author: paul.elschot@xs4all.nl):
ForDecompress.decodeAnyFrame() is pretty slow, it is mainly there as a fallback and I have also used it for testing initially.
For decoding blocks just shorter than 32, a few Simple9 blocks will be faster than ForDecompress.decodeAnyFrame().
Simple16 (4 bits for the cases, just as Simple9) would be good. I remember looking for a link to an implementation of Simple16 in one of the articles on Simple9, but the linked page had gone, so I just went ahead myself and started on Simple9.
One can also use 32 or 64 cases (5 or 6 bits for the cases) and allow a variable number of ints (or bytes) in such SimpleNN variants. Such variants nicely hit the middle ground between Simple9 and PFOR in block length and decoding speed. Perhaps we'd better continue the discussion on such methods at LUCENE-2189.
When there are sufficient methods available an issue could be opened for a general variable length block encoding/decoding that uses the best of such methods depending on the block length. This makes sense because in lucene indexes the block length is always known.
> FixedIntBlockIndexInput.Reader does not initialise 'pending' int array
> ----------------------------------------------------------------------
>
> Key: LUCENE-2222
> URL: https://issues.apache.org/jira/browse/LUCENE-2222
> Project: Lucene - Java
> Issue Type: Bug
> Components: Index
> Affects Versions: Flex Branch
> Reporter: Renaud Delbru
> Priority: Minor
> Fix For: Flex Branch
>
> Attachments: LUCENE-2222.patch, LUCENE-2222.patch, LUCENE-2222.patch
>
>
> The FixedIntBlockIndexInput.Reader.pending int array is not initialised. As a consequence, the FixedIntBlockIndexInput.Reader#next() method returns always 0.
> A call to FixedIntBlockIndexInput.Reader#blockReader.readBlock() during the Reader initialisation may solve the issue (to be tested).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org