You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@commons.apache.org by Michael Thomson <mi...@canb.auug.org.au> on 2007/03/10 02:34:09 UTC

Question About larger than memory multithreaded sort?

Hi,

Not sure if this is the right forum but I thought I would ask as I  am 
sure other people have run into this similar "common" problem.

I have to sort some data that the combined sort keys are larger than memory.

I have written a general routine that  in core sorts what it can  and 
then merge/sorts the resultant sorted files.  It seems to work pretty 
well but  now I am thinking I want to take advantage of all the other 
processors on the executing machines.

This is not a new problem and it seems strange that I cannot seem to 
find any existing implementations?  Am I looking in the wrong place or 
does this not exist?

Any one know where I should be looking?

Thanks in advance
Michael.



---------------------------------------------------------------------
To unsubscribe, e-mail: commons-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: commons-user-help@jakarta.apache.org


Re: Question About larger than memory multithreaded sort?

Posted by Paul J DeCoursey <pa...@decoursey.net>.
How do databases handle this kind of thing?  I know that if it is an 
indexed field the index will be pre-sorted, but how would it manage 
non-indexed fields. I'd say this is the wrong forum, but I'm not sure 
what the right one is.  Where are you getting the data?  could you use 
an embedded Derby database and create a temp table to sort all the data 
and then query that to return the results?

Paul


Michael Thomson wrote:
> Hi,
>
> Not sure if this is the right forum but I thought I would ask as I  am 
> sure other people have run into this similar "common" problem.
>
> I have to sort some data that the combined sort keys are larger than 
> memory.
>
> I have written a general routine that  in core sorts what it can  and 
> then merge/sorts the resultant sorted files.  It seems to work pretty 
> well but  now I am thinking I want to take advantage of all the other 
> processors on the executing machines.
>
> This is not a new problem and it seems strange that I cannot seem to 
> find any existing implementations?  Am I looking in the wrong place or 
> does this not exist?
>
> Any one know where I should be looking?
>
> Thanks in advance
> Michael.
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: commons-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: commons-user-help@jakarta.apache.org
>
>


---------------------------------------------------------------------
To unsubscribe, e-mail: commons-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: commons-user-help@jakarta.apache.org