You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Martijn van Groningen (Created) (JIRA)" <ji...@apache.org> on 2011/10/07 14:41:29 UTC

[jira] [Created] (LUCENE-3496) Support grouping by IndexDocValues

Support grouping by IndexDocValues
----------------------------------

                 Key: LUCENE-3496
                 URL: https://issues.apache.org/jira/browse/LUCENE-3496
             Project: Lucene - Java
          Issue Type: New Feature
          Components: modules/grouping
            Reporter: Martijn van Groningen


Although IDV is not yet finalized (More particular the SortedSource). I think we already can discuss / investigate implementing grouping by IDV.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


[jira] [Commented] (LUCENE-3496) Support grouping by IndexDocValues

Posted by "Simon Willnauer (Commented) (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/LUCENE-3496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13123201#comment-13123201 ] 

Simon Willnauer commented on LUCENE-3496:
-----------------------------------------

bq.Wow, only a 9% performance penalty for not keeping the values in the JVM heap? That's quite an achievement (and almost hard to believe, unless the base query were really expensive or something). Nice job!

wait! this is in memory - martjin can you test with getDirectSource() instead of getSource()?
                
> Support grouping by IndexDocValues
> ----------------------------------
>
>                 Key: LUCENE-3496
>                 URL: https://issues.apache.org/jira/browse/LUCENE-3496
>             Project: Lucene - Java
>          Issue Type: New Feature
>          Components: modules/grouping
>            Reporter: Martijn van Groningen
>         Attachments: LUCENE-3496.patch, LUCENE-3496.patch
>
>
> Although IDV is not yet finalized (More particular the SortedSource). I think we already can discuss / investigate implementing grouping by IDV.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


[jira] [Updated] (LUCENE-3496) Support grouping by IndexDocValues

Posted by "Martijn van Groningen (Updated) (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/LUCENE-3496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Martijn van Groningen updated LUCENE-3496:
------------------------------------------

    Attachment: LUCENE-3496.patch

Updated patch to work with the trunk.
                
> Support grouping by IndexDocValues
> ----------------------------------
>
>                 Key: LUCENE-3496
>                 URL: https://issues.apache.org/jira/browse/LUCENE-3496
>             Project: Lucene - Java
>          Issue Type: New Feature
>          Components: modules/grouping
>            Reporter: Martijn van Groningen
>         Attachments: LUCENE-3496.patch, LUCENE-3496.patch, LUCENE-3496.patch, LUCENE-3496.patch, LUCENE-3496.patch
>
>
> Although IDV is not yet finalized (More particular the SortedSource). I think we already can discuss / investigate implementing grouping by IDV.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


[jira] [Commented] (LUCENE-3496) Support grouping by IndexDocValues

Posted by "Michael McCandless (Commented) (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/LUCENE-3496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13124982#comment-13124982 ] 

Michael McCandless commented on LUCENE-3496:
--------------------------------------------

It's great that rounding up to 8/16/32/64 bit width resolved the performance problem!

I think we need to make this (RAM vs CPU tradeoff) settable by the app, at indexing time?
                
> Support grouping by IndexDocValues
> ----------------------------------
>
>                 Key: LUCENE-3496
>                 URL: https://issues.apache.org/jira/browse/LUCENE-3496
>             Project: Lucene - Java
>          Issue Type: New Feature
>          Components: modules/grouping
>            Reporter: Martijn van Groningen
>         Attachments: LUCENE-3496.patch, LUCENE-3496.patch, LUCENE-3496.patch
>
>
> Although IDV is not yet finalized (More particular the SortedSource). I think we already can discuss / investigate implementing grouping by IDV.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


[jira] [Commented] (LUCENE-3496) Support grouping by IndexDocValues

Posted by "Simon Willnauer (Commented) (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/LUCENE-3496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13125211#comment-13125211 ] 

Simon Willnauer commented on LUCENE-3496:
-----------------------------------------

bq. I think we need to make this (RAM vs CPU tradeoff) settable by the app, at indexing time?

I think we should do that. Maybe via IWC?


bq. I noticed that with BYTES_VAR_SORTED the memory usage varied a lot during the test execution from around 10MB to 320MB. The memory usage for BYTES_FIXED_SORTED is stable around 10MB.

yeah, the VAR_SORTED variant loads the ords in memory which is wrong though. However, LUCENE-3507 is adressing this issue.

                
> Support grouping by IndexDocValues
> ----------------------------------
>
>                 Key: LUCENE-3496
>                 URL: https://issues.apache.org/jira/browse/LUCENE-3496
>             Project: Lucene - Java
>          Issue Type: New Feature
>          Components: modules/grouping
>            Reporter: Martijn van Groningen
>         Attachments: LUCENE-3496.patch, LUCENE-3496.patch, LUCENE-3496.patch
>
>
> Although IDV is not yet finalized (More particular the SortedSource). I think we already can discuss / investigate implementing grouping by IDV.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


[jira] [Resolved] (LUCENE-3496) Support grouping by IndexDocValues

Posted by "Martijn van Groningen (Resolved) (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/LUCENE-3496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Martijn van Groningen resolved LUCENE-3496.
-------------------------------------------

       Resolution: Fixed
    Lucene Fields: Patch Available  (was: New)

Committed to trunk.
                
> Support grouping by IndexDocValues
> ----------------------------------
>
>                 Key: LUCENE-3496
>                 URL: https://issues.apache.org/jira/browse/LUCENE-3496
>             Project: Lucene - Java
>          Issue Type: New Feature
>          Components: modules/grouping
>            Reporter: Martijn van Groningen
>            Assignee: Martijn van Groningen
>             Fix For: 4.0
>
>         Attachments: LUCENE-3496.patch, LUCENE-3496.patch, LUCENE-3496.patch, LUCENE-3496.patch, LUCENE-3496.patch, LUCENE-3496.patch, LUCENE-3496.patch, LUCENE-3496.patch
>
>
> Although IDV is not yet finalized (More particular the SortedSource). I think we already can discuss / investigate implementing grouping by IDV.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


[jira] [Updated] (LUCENE-3496) Support grouping by IndexDocValues

Posted by "Martijn van Groningen (Updated) (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/LUCENE-3496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Martijn van Groningen updated LUCENE-3496:
------------------------------------------

    Attachment: LUCENE-3496.patch

Updated patch. Added idv implementation for second pass grouping collector. Also the tests have been changed to randomly select an idv based implementation when possible (tests now choose randomly between term, idv and function based implementations).
                
> Support grouping by IndexDocValues
> ----------------------------------
>
>                 Key: LUCENE-3496
>                 URL: https://issues.apache.org/jira/browse/LUCENE-3496
>             Project: Lucene - Java
>          Issue Type: New Feature
>          Components: modules/grouping
>            Reporter: Martijn van Groningen
>         Attachments: LUCENE-3496.patch, LUCENE-3496.patch, LUCENE-3496.patch, LUCENE-3496.patch, LUCENE-3496.patch, LUCENE-3496.patch
>
>
> Although IDV is not yet finalized (More particular the SortedSource). I think we already can discuss / investigate implementing grouping by IDV.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


[jira] [Updated] (LUCENE-3496) Support grouping by IndexDocValues

Posted by "Martijn van Groningen (Updated) (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/LUCENE-3496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Martijn van Groningen updated LUCENE-3496:
------------------------------------------

    Attachment: LUCENE-3496.patch

Updated patch.
* All tests pass now.
* Added dv based impl for AllGroupHeadCollector.

I will commit this soon.
                
> Support grouping by IndexDocValues
> ----------------------------------
>
>                 Key: LUCENE-3496
>                 URL: https://issues.apache.org/jira/browse/LUCENE-3496
>             Project: Lucene - Java
>          Issue Type: New Feature
>          Components: modules/grouping
>            Reporter: Martijn van Groningen
>            Assignee: Martijn van Groningen
>             Fix For: 4.0
>
>         Attachments: LUCENE-3496.patch, LUCENE-3496.patch, LUCENE-3496.patch, LUCENE-3496.patch, LUCENE-3496.patch, LUCENE-3496.patch, LUCENE-3496.patch, LUCENE-3496.patch
>
>
> Although IDV is not yet finalized (More particular the SortedSource). I think we already can discuss / investigate implementing grouping by IDV.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


[jira] [Commented] (LUCENE-3496) Support grouping by IndexDocValues

Posted by "Simon Willnauer (Commented) (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/LUCENE-3496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13125269#comment-13125269 ] 

Simon Willnauer commented on LUCENE-3496:
-----------------------------------------

bq. +1 I will create an issue

FYI - LUCENE-3509
                
> Support grouping by IndexDocValues
> ----------------------------------
>
>                 Key: LUCENE-3496
>                 URL: https://issues.apache.org/jira/browse/LUCENE-3496
>             Project: Lucene - Java
>          Issue Type: New Feature
>          Components: modules/grouping
>            Reporter: Martijn van Groningen
>         Attachments: LUCENE-3496.patch, LUCENE-3496.patch, LUCENE-3496.patch
>
>
> Although IDV is not yet finalized (More particular the SortedSource). I think we already can discuss / investigate implementing grouping by IDV.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


[jira] [Commented] (LUCENE-3496) Support grouping by IndexDocValues

Posted by "Martijn van Groningen (Commented) (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/LUCENE-3496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13142486#comment-13142486 ] 

Martijn van Groningen commented on LUCENE-3496:
-----------------------------------------------

I think this can be committed soon. The fixed size packed ints options for DV will be further improved in LUCENE-3509.
                
> Support grouping by IndexDocValues
> ----------------------------------
>
>                 Key: LUCENE-3496
>                 URL: https://issues.apache.org/jira/browse/LUCENE-3496
>             Project: Lucene - Java
>          Issue Type: New Feature
>          Components: modules/grouping
>            Reporter: Martijn van Groningen
>             Fix For: 4.0
>
>         Attachments: LUCENE-3496.patch, LUCENE-3496.patch, LUCENE-3496.patch, LUCENE-3496.patch, LUCENE-3496.patch, LUCENE-3496.patch, LUCENE-3496.patch
>
>
> Although IDV is not yet finalized (More particular the SortedSource). I think we already can discuss / investigate implementing grouping by IDV.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


[jira] [Issue Comment Edited] (LUCENE-3496) Support grouping by IndexDocValues

Posted by "Martijn van Groningen (Issue Comment Edited) (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/LUCENE-3496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13122740#comment-13122740 ] 

Martijn van Groningen edited comment on LUCENE-3496 at 10/7/11 12:43 PM:
-------------------------------------------------------------------------

Attached initial IDV AbstractAllGroupsCollector implementation.
                
      was (Author: martijn.v.groningen):
    Attached initial IDV implementation of AbstractAllGroupsCollector.
                  
> Support grouping by IndexDocValues
> ----------------------------------
>
>                 Key: LUCENE-3496
>                 URL: https://issues.apache.org/jira/browse/LUCENE-3496
>             Project: Lucene - Java
>          Issue Type: New Feature
>          Components: modules/grouping
>            Reporter: Martijn van Groningen
>         Attachments: LUCENE-3496.patch
>
>
> Although IDV is not yet finalized (More particular the SortedSource). I think we already can discuss / investigate implementing grouping by IDV.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


[jira] [Updated] (LUCENE-3496) Support grouping by IndexDocValues

Posted by "Martijn van Groningen (Updated) (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/LUCENE-3496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Martijn van Groningen updated LUCENE-3496:
------------------------------------------

    Attachment: LUCENE-3496.patch

Attached initial IDV implementation of AbstractAllGroupsCollector.
                
> Support grouping by IndexDocValues
> ----------------------------------
>
>                 Key: LUCENE-3496
>                 URL: https://issues.apache.org/jira/browse/LUCENE-3496
>             Project: Lucene - Java
>          Issue Type: New Feature
>          Components: modules/grouping
>            Reporter: Martijn van Groningen
>         Attachments: LUCENE-3496.patch
>
>
> Although IDV is not yet finalized (More particular the SortedSource). I think we already can discuss / investigate implementing grouping by IDV.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


[jira] [Updated] (LUCENE-3496) Support grouping by IndexDocValues

Posted by "Martijn van Groningen (Updated) (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/LUCENE-3496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Martijn van Groningen updated LUCENE-3496:
------------------------------------------

    Attachment: LUCENE-3496.patch

Updated patch. Moved Simon's changes to force the use of the sorted variants to use fixed size packed ints to LUCENE-3509.
                
> Support grouping by IndexDocValues
> ----------------------------------
>
>                 Key: LUCENE-3496
>                 URL: https://issues.apache.org/jira/browse/LUCENE-3496
>             Project: Lucene - Java
>          Issue Type: New Feature
>          Components: modules/grouping
>            Reporter: Martijn van Groningen
>         Attachments: LUCENE-3496.patch, LUCENE-3496.patch, LUCENE-3496.patch, LUCENE-3496.patch, LUCENE-3496.patch, LUCENE-3496.patch, LUCENE-3496.patch
>
>
> Although IDV is not yet finalized (More particular the SortedSource). I think we already can discuss / investigate implementing grouping by IDV.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


[jira] [Updated] (LUCENE-3496) Support grouping by IndexDocValues

Posted by "Martijn van Groningen (Updated) (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/LUCENE-3496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Martijn van Groningen updated LUCENE-3496:
------------------------------------------

    Fix Version/s: 4.0
         Assignee: Martijn van Groningen
    
> Support grouping by IndexDocValues
> ----------------------------------
>
>                 Key: LUCENE-3496
>                 URL: https://issues.apache.org/jira/browse/LUCENE-3496
>             Project: Lucene - Java
>          Issue Type: New Feature
>          Components: modules/grouping
>            Reporter: Martijn van Groningen
>            Assignee: Martijn van Groningen
>             Fix For: 4.0
>
>         Attachments: LUCENE-3496.patch, LUCENE-3496.patch, LUCENE-3496.patch, LUCENE-3496.patch, LUCENE-3496.patch, LUCENE-3496.patch, LUCENE-3496.patch
>
>
> Although IDV is not yet finalized (More particular the SortedSource). I think we already can discuss / investigate implementing grouping by IDV.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


[jira] [Commented] (LUCENE-3496) Support grouping by IndexDocValues

Posted by "Simon Willnauer (Commented) (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/LUCENE-3496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13125256#comment-13125256 ] 

Simon Willnauer commented on LUCENE-3496:
-----------------------------------------

bq. That seems good? I think it should default to rounding up? Apps that want to tune RAM usage down can then change the default.

+1 I will create an issue
                
> Support grouping by IndexDocValues
> ----------------------------------
>
>                 Key: LUCENE-3496
>                 URL: https://issues.apache.org/jira/browse/LUCENE-3496
>             Project: Lucene - Java
>          Issue Type: New Feature
>          Components: modules/grouping
>            Reporter: Martijn van Groningen
>         Attachments: LUCENE-3496.patch, LUCENE-3496.patch, LUCENE-3496.patch
>
>
> Although IDV is not yet finalized (More particular the SortedSource). I think we already can discuss / investigate implementing grouping by IDV.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


[jira] [Issue Comment Edited] (LUCENE-3496) Support grouping by IndexDocValues

Posted by "Martijn van Groningen (Issue Comment Edited) (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/LUCENE-3496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13123082#comment-13123082 ] 

Martijn van Groningen edited comment on LUCENE-3496 at 10/7/11 7:10 PM:
------------------------------------------------------------------------

Simon, this looks great!
The performance is much better now. An average the search time for grouping by BYTES_VAR_SORTED is 329 ms and BYTES_FIXED_SORTED is 338 ms. The average search time for grouping by indexed values is 310 ms. So grouping by IDV is pretty close to grouping by indexed values!
                
      was (Author: martijn.v.groningen):
    Simon, this looks great!
The performance is much better now. An average the search time for grouping by BYTES_VAR_SORTED is 329 ms and BYTES_FIXED_SORTED is 338 ms. The average search time for grouping by indexed values is 310 ms. So grouping by IDV pretty close to grouping by indexed values!
                  
> Support grouping by IndexDocValues
> ----------------------------------
>
>                 Key: LUCENE-3496
>                 URL: https://issues.apache.org/jira/browse/LUCENE-3496
>             Project: Lucene - Java
>          Issue Type: New Feature
>          Components: modules/grouping
>            Reporter: Martijn van Groningen
>         Attachments: LUCENE-3496.patch, LUCENE-3496.patch
>
>
> Although IDV is not yet finalized (More particular the SortedSource). I think we already can discuss / investigate implementing grouping by IDV.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


[jira] [Commented] (LUCENE-3496) Support grouping by IndexDocValues

Posted by "Martijn van Groningen (Commented) (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/LUCENE-3496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13123763#comment-13123763 ] 

Martijn van Groningen commented on LUCENE-3496:
-----------------------------------------------

getDirectSource() search times:
* BYTES_VAR_SORTED: 455 ms
* BYTES_FIXED_SORTED: 713 ms

I noticed that with BYTES_VAR_SORTED the memory usage varied a lot during the test execution from around 10MB to 320MB. The memory usage for BYTES_FIXED_SORTED is stable around 10MB. 

I'll try to use luceneutil and modify it to benchmark grouping by IDV, so others can run performance tests as well. 
                
> Support grouping by IndexDocValues
> ----------------------------------
>
>                 Key: LUCENE-3496
>                 URL: https://issues.apache.org/jira/browse/LUCENE-3496
>             Project: Lucene - Java
>          Issue Type: New Feature
>          Components: modules/grouping
>            Reporter: Martijn van Groningen
>         Attachments: LUCENE-3496.patch, LUCENE-3496.patch
>
>
> Although IDV is not yet finalized (More particular the SortedSource). I think we already can discuss / investigate implementing grouping by IDV.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


[jira] [Commented] (LUCENE-3496) Support grouping by IndexDocValues

Posted by "Simon Willnauer (Commented) (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/LUCENE-3496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13122751#comment-13122751 ] 

Simon Willnauer commented on LUCENE-3496:
-----------------------------------------

hey martjin, do yo have some benchmark results for IDV showing the impact? I would guess that depending on the type ie. var vs. fixed size you could see some perf hits. We might need to round up the packedint sizes inside IDV for lookup perf and pay the prize for a slightly larger size on disk / memory.
                
> Support grouping by IndexDocValues
> ----------------------------------
>
>                 Key: LUCENE-3496
>                 URL: https://issues.apache.org/jira/browse/LUCENE-3496
>             Project: Lucene - Java
>          Issue Type: New Feature
>          Components: modules/grouping
>            Reporter: Martijn van Groningen
>         Attachments: LUCENE-3496.patch
>
>
> Although IDV is not yet finalized (More particular the SortedSource). I think we already can discuss / investigate implementing grouping by IDV.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


[jira] [Updated] (LUCENE-3496) Support grouping by IndexDocValues

Posted by "Martijn van Groningen (Updated) (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/LUCENE-3496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Martijn van Groningen updated LUCENE-3496:
------------------------------------------

    Attachment: LUCENE-3496.patch

bq. one thing I noticed about the patch is that you consider 0 ords as null values which is not the case in IDV.
I changed that in the new patch.
I also included IDV implementation for AbstractFirstPassGroupingCollector.

I think it is time to include these collectors into the already existing tests. Besides that we need IDV implementations for AbstractSecondPassGroupingCollector and AbstractAllGroupHeadsCollector.
                
> Support grouping by IndexDocValues
> ----------------------------------
>
>                 Key: LUCENE-3496
>                 URL: https://issues.apache.org/jira/browse/LUCENE-3496
>             Project: Lucene - Java
>          Issue Type: New Feature
>          Components: modules/grouping
>            Reporter: Martijn van Groningen
>         Attachments: LUCENE-3496.patch, LUCENE-3496.patch, LUCENE-3496.patch, LUCENE-3496.patch
>
>
> Although IDV is not yet finalized (More particular the SortedSource). I think we already can discuss / investigate implementing grouping by IDV.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


[jira] [Commented] (LUCENE-3496) Support grouping by IndexDocValues

Posted by "Martijn van Groningen (Commented) (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/LUCENE-3496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13126191#comment-13126191 ] 

Martijn van Groningen commented on LUCENE-3496:
-----------------------------------------------

After a svn update (that includes changes for LUCENE-3507) the heap usage for grouping by BYTES_VAR_SORTED IDV and using disk resident values is much more stable! It stays between 5MB and 16MB and the average search time is around 580ms.
                
> Support grouping by IndexDocValues
> ----------------------------------
>
>                 Key: LUCENE-3496
>                 URL: https://issues.apache.org/jira/browse/LUCENE-3496
>             Project: Lucene - Java
>          Issue Type: New Feature
>          Components: modules/grouping
>            Reporter: Martijn van Groningen
>         Attachments: LUCENE-3496.patch, LUCENE-3496.patch, LUCENE-3496.patch, LUCENE-3496.patch
>
>
> Although IDV is not yet finalized (More particular the SortedSource). I think we already can discuss / investigate implementing grouping by IDV.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


[jira] [Commented] (LUCENE-3496) Support grouping by IndexDocValues

Posted by "Martijn van Groningen (Commented) (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/LUCENE-3496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13149689#comment-13149689 ] 

Martijn van Groningen commented on LUCENE-3496:
-----------------------------------------------

I was planning on doing this. I'm almost ready to commit it. I'm only a bit stuck on documents that don't have a value for a group field.

The random grouping tests also add documents with a null value for the group field and an empty string for the group field. This works fine with the term based implementations, but not the DV based implementations (random test fail). Should we not use null as group value if the dv based implementations are used during the test?
                
> Support grouping by IndexDocValues
> ----------------------------------
>
>                 Key: LUCENE-3496
>                 URL: https://issues.apache.org/jira/browse/LUCENE-3496
>             Project: Lucene - Java
>          Issue Type: New Feature
>          Components: modules/grouping
>            Reporter: Martijn van Groningen
>            Assignee: Martijn van Groningen
>             Fix For: 4.0
>
>         Attachments: LUCENE-3496.patch, LUCENE-3496.patch, LUCENE-3496.patch, LUCENE-3496.patch, LUCENE-3496.patch, LUCENE-3496.patch, LUCENE-3496.patch
>
>
> Although IDV is not yet finalized (More particular the SortedSource). I think we already can discuss / investigate implementing grouping by IDV.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


[jira] [Commented] (LUCENE-3496) Support grouping by IndexDocValues

Posted by "Yonik Seeley (Commented) (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/LUCENE-3496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13123147#comment-13123147 ] 

Yonik Seeley commented on LUCENE-3496:
--------------------------------------

Wow, only a 9% performance penalty for not keeping the values in the JVM heap?  That's quite an achievement (and almost hard to believe, unless the base query were really expensive or something).  Nice job!
                
> Support grouping by IndexDocValues
> ----------------------------------
>
>                 Key: LUCENE-3496
>                 URL: https://issues.apache.org/jira/browse/LUCENE-3496
>             Project: Lucene - Java
>          Issue Type: New Feature
>          Components: modules/grouping
>            Reporter: Martijn van Groningen
>         Attachments: LUCENE-3496.patch, LUCENE-3496.patch
>
>
> Although IDV is not yet finalized (More particular the SortedSource). I think we already can discuss / investigate implementing grouping by IDV.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


[jira] [Updated] (LUCENE-3496) Support grouping by IndexDocValues

Posted by "Simon Willnauer (Updated) (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/LUCENE-3496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Simon Willnauer updated LUCENE-3496:
------------------------------------

    Attachment: LUCENE-3496.patch

martjin, here is a patch that forces the sorted variants to use fixed size packed ints. can you check if the perf changes?
                
> Support grouping by IndexDocValues
> ----------------------------------
>
>                 Key: LUCENE-3496
>                 URL: https://issues.apache.org/jira/browse/LUCENE-3496
>             Project: Lucene - Java
>          Issue Type: New Feature
>          Components: modules/grouping
>            Reporter: Martijn van Groningen
>         Attachments: LUCENE-3496.patch, LUCENE-3496.patch
>
>
> Although IDV is not yet finalized (More particular the SortedSource). I think we already can discuss / investigate implementing grouping by IDV.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


[jira] [Commented] (LUCENE-3496) Support grouping by IndexDocValues

Posted by "Michael McCandless (Commented) (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/LUCENE-3496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13125244#comment-13125244 ] 

Michael McCandless commented on LUCENE-3496:
--------------------------------------------

{quote}
bq. I think we need to make this (RAM vs CPU tradeoff) settable by the app, at indexing time?

I think we should do that. Maybe via IWC?
{quote}

That seems good?  I think it should default to rounding up?  Apps that want to tune RAM usage down can then change the default.
                
> Support grouping by IndexDocValues
> ----------------------------------
>
>                 Key: LUCENE-3496
>                 URL: https://issues.apache.org/jira/browse/LUCENE-3496
>             Project: Lucene - Java
>          Issue Type: New Feature
>          Components: modules/grouping
>            Reporter: Martijn van Groningen
>         Attachments: LUCENE-3496.patch, LUCENE-3496.patch, LUCENE-3496.patch
>
>
> Although IDV is not yet finalized (More particular the SortedSource). I think we already can discuss / investigate implementing grouping by IDV.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


[jira] [Issue Comment Edited] (LUCENE-3496) Support grouping by IndexDocValues

Posted by "Simon Willnauer (Issue Comment Edited) (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/LUCENE-3496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13125211#comment-13125211 ] 

Simon Willnauer edited comment on LUCENE-3496 at 10/11/11 5:35 PM:
-------------------------------------------------------------------

bq. I think we need to make this (RAM vs CPU tradeoff) settable by the app, at indexing time?

I think we should do that. Maybe via IWC?


bq. I noticed that with BYTES_VAR_SORTED the memory usage varied a lot during the test execution from around 10MB to 320MB. The memory usage for BYTES_FIXED_SORTED is stable around 10MB.

yeah, the VAR_SORTED variant loads the ords in memory which is wrong though. However, LUCENE-3507 is adressing this issue.

one thing I noticed about the patch is that you consider 0 ords as null values which is not the case in IDV. All documents do have values, there is nothing like missing values. VAR_BYTES will return a 0-length byte and FIXED_BYTES will return a fixed-length 0-byte array. Numeric variants default to 0 and 0.0 receptively.

                
      was (Author: simonw):
    bq. I think we need to make this (RAM vs CPU tradeoff) settable by the app, at indexing time?

I think we should do that. Maybe via IWC?


bq. I noticed that with BYTES_VAR_SORTED the memory usage varied a lot during the test execution from around 10MB to 320MB. The memory usage for BYTES_FIXED_SORTED is stable around 10MB.

yeah, the VAR_SORTED variant loads the ords in memory which is wrong though. However, LUCENE-3507 is adressing this issue.

                  
> Support grouping by IndexDocValues
> ----------------------------------
>
>                 Key: LUCENE-3496
>                 URL: https://issues.apache.org/jira/browse/LUCENE-3496
>             Project: Lucene - Java
>          Issue Type: New Feature
>          Components: modules/grouping
>            Reporter: Martijn van Groningen
>         Attachments: LUCENE-3496.patch, LUCENE-3496.patch, LUCENE-3496.patch
>
>
> Although IDV is not yet finalized (More particular the SortedSource). I think we already can discuss / investigate implementing grouping by IDV.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


[jira] [Commented] (LUCENE-3496) Support grouping by IndexDocValues

Posted by "Simon Willnauer (Commented) (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/LUCENE-3496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13149543#comment-13149543 ] 

Simon Willnauer commented on LUCENE-3496:
-----------------------------------------

Martjin, the last patch looks ok to me. you should go ahead and commit this...
                
> Support grouping by IndexDocValues
> ----------------------------------
>
>                 Key: LUCENE-3496
>                 URL: https://issues.apache.org/jira/browse/LUCENE-3496
>             Project: Lucene - Java
>          Issue Type: New Feature
>          Components: modules/grouping
>            Reporter: Martijn van Groningen
>            Assignee: Martijn van Groningen
>             Fix For: 4.0
>
>         Attachments: LUCENE-3496.patch, LUCENE-3496.patch, LUCENE-3496.patch, LUCENE-3496.patch, LUCENE-3496.patch, LUCENE-3496.patch, LUCENE-3496.patch
>
>
> Although IDV is not yet finalized (More particular the SortedSource). I think we already can discuss / investigate implementing grouping by IDV.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


[jira] [Updated] (LUCENE-3496) Support grouping by IndexDocValues

Posted by "Martijn van Groningen (Updated) (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/LUCENE-3496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Martijn van Groningen updated LUCENE-3496:
------------------------------------------

    Attachment: LUCENE-3496.patch

Minor update to IndexDocValuesAllGroupsCollector to use disk resident IDV.
                
> Support grouping by IndexDocValues
> ----------------------------------
>
>                 Key: LUCENE-3496
>                 URL: https://issues.apache.org/jira/browse/LUCENE-3496
>             Project: Lucene - Java
>          Issue Type: New Feature
>          Components: modules/grouping
>            Reporter: Martijn van Groningen
>         Attachments: LUCENE-3496.patch, LUCENE-3496.patch, LUCENE-3496.patch
>
>
> Although IDV is not yet finalized (More particular the SortedSource). I think we already can discuss / investigate implementing grouping by IDV.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


[jira] [Commented] (LUCENE-3496) Support grouping by IndexDocValues

Posted by "Martijn van Groningen (Commented) (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/LUCENE-3496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13123082#comment-13123082 ] 

Martijn van Groningen commented on LUCENE-3496:
-----------------------------------------------

Simon, this looks great!
The performance is much better now. An average the search time for grouping by BYTES_VAR_SORTED is 329 ms and BYTES_FIXED_SORTED is 338 ms. The average search time for grouping by indexed values is 310 ms. So grouping by IDV pretty close to grouping by indexed values!
                
> Support grouping by IndexDocValues
> ----------------------------------
>
>                 Key: LUCENE-3496
>                 URL: https://issues.apache.org/jira/browse/LUCENE-3496
>             Project: Lucene - Java
>          Issue Type: New Feature
>          Components: modules/grouping
>            Reporter: Martijn van Groningen
>         Attachments: LUCENE-3496.patch, LUCENE-3496.patch
>
>
> Although IDV is not yet finalized (More particular the SortedSource). I think we already can discuss / investigate implementing grouping by IDV.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


[jira] [Commented] (LUCENE-3496) Support grouping by IndexDocValues

Posted by "Martijn van Groningen (Commented) (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/LUCENE-3496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13122942#comment-13122942 ] 

Martijn van Groningen commented on LUCENE-3496:
-----------------------------------------------

Up until now I only checked the difference between BYTES_VAR_SORTED and BYTES_FIXED_SORTED. The search time and memory usage seem to be more or less the same. I executed all queries on 30M index and the performance with grouping by IDV is somewhere between 500 ms and 550 ms. The same test with grouping by indexed values have search times somewhere between 300 ms and 330 ms.

bq. We might need to round up the packedint sizes inside IDV for lookup perf and pay the prize for a slightly larger size on disk / memory.
I think we need this. Just like we have with FC#getTermsIndex(). A boolean option that either uses PackedInt or DirectInt. I think the ~200 ms difference is b/c IDV is using PackedInt and FC uses DirectInt.

                
> Support grouping by IndexDocValues
> ----------------------------------
>
>                 Key: LUCENE-3496
>                 URL: https://issues.apache.org/jira/browse/LUCENE-3496
>             Project: Lucene - Java
>          Issue Type: New Feature
>          Components: modules/grouping
>            Reporter: Martijn van Groningen
>         Attachments: LUCENE-3496.patch
>
>
> Although IDV is not yet finalized (More particular the SortedSource). I think we already can discuss / investigate implementing grouping by IDV.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org