You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@commons.apache.org by Michael Thomson <mi...@canb.auug.org.au> on 2007/03/10 02:34:09 UTC
Question About larger than memory multithreaded sort?
Hi,
Not sure if this is the right forum but I thought I would ask as I am
sure other people have run into this similar "common" problem.
I have to sort some data that the combined sort keys are larger than memory.
I have written a general routine that in core sorts what it can and
then merge/sorts the resultant sorted files. It seems to work pretty
well but now I am thinking I want to take advantage of all the other
processors on the executing machines.
This is not a new problem and it seems strange that I cannot seem to
find any existing implementations? Am I looking in the wrong place or
does this not exist?
Any one know where I should be looking?
Thanks in advance
Michael.
---------------------------------------------------------------------
To unsubscribe, e-mail: commons-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: commons-user-help@jakarta.apache.org
Re: Question About larger than memory multithreaded sort?
Posted by Paul J DeCoursey <pa...@decoursey.net>.
How do databases handle this kind of thing? I know that if it is an
indexed field the index will be pre-sorted, but how would it manage
non-indexed fields. I'd say this is the wrong forum, but I'm not sure
what the right one is. Where are you getting the data? could you use
an embedded Derby database and create a temp table to sort all the data
and then query that to return the results?
Paul
Michael Thomson wrote:
> Hi,
>
> Not sure if this is the right forum but I thought I would ask as I am
> sure other people have run into this similar "common" problem.
>
> I have to sort some data that the combined sort keys are larger than
> memory.
>
> I have written a general routine that in core sorts what it can and
> then merge/sorts the resultant sorted files. It seems to work pretty
> well but now I am thinking I want to take advantage of all the other
> processors on the executing machines.
>
> This is not a new problem and it seems strange that I cannot seem to
> find any existing implementations? Am I looking in the wrong place or
> does this not exist?
>
> Any one know where I should be looking?
>
> Thanks in advance
> Michael.
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: commons-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: commons-user-help@jakarta.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: commons-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: commons-user-help@jakarta.apache.org