You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@jena.apache.org by "A. Soroka" <aj...@virginia.edu> on 2016/10/28 13:33:42 UTC
another naive question about tdbloader2
I'm still learning about tdbloader2 and have another question about the index phase: is there any reason why the processes for the various index orderings (SPO, GSPO, etc.) couldn't go on in parallel? Or am I missing some switch or setting that already allows that?
---
A. Soroka
The University of Virginia Library
Re: another naive question about tdbloader2
Posted by Andy Seaborne <an...@apache.org>.
On 28/10/16 16:21, A. Soroka wrote:
> Okay, that's pretty much the sensible answer I expected: "YMMV, try some experiments and let's work with hard data."
>
> I need to reread Andy's post [1] about the packing algorithm to make sure I understand what's going on there.
>
> The hardware in question is definitely "$$$"-- maybe even "$$$$". {grin} This is contract work, so before going too much further I need to check with my principals to prioritize this kind of speculative work and we will make a plan accordingly.
>
> I'm not sure immediately how to measure the rates in terms of tuples/sec-- I'm not aware of any such logging from sort itself, but I haven't looked for it. I assume that tdbloader2 can emit messages as it packs tuples in the same way that it emits messages with an average rate periodically in the 'data' phase?
It prints the elapsed time and the number of triples/quads
(unfortunately that comes from a java step - you have to do the
calculation).
It isn't as chatty as tdbloader because it is using external tools.
Andy
>
> ---
> A. Soroka
> The University of Virginia Library
>
> [1] https://seaborne.blogspot.com/2010/12/repacking-btrees.html
>
>> On Oct 28, 2016, at 11:12 AM, Andy Seaborne <an...@apache.org> wrote:
>>
>>
>>
>> On 28/10/16 15:58, A. Soroka wrote:
>>> That's right-- if I understand it correctly, there are two steps--
>>> POSIX sort to develop the index orderings, and then packing the
>>> actual index files.
>>>
>>> For the POSIX sort step, it's certainly true that more parallelism
>>> than needed would be a bad thing. With Andy's help I just made a
>>> commit that allows a little more control using the common --parallel
>>> flag for sort. But the current ergonomics seem suboptimal. E.g. with
>>> the current settings indexing a 300Mt dataset on a 24-core box with
>>> fast storage, I saw only one core in full use, and very little IO
>>> usage. Aliasing in some parallelism via the sort flag brought several
>>> more cores into play and cut the time spent by two-thirds.
>>
>> What rates are you getting?
>>
>>> I don't
>>> know how normal that is, but for the sort step, my argument is not
>>> that we could find universally better ergonomics, but that we could
>>> bake some flexibility in for those who want to try adjustments on
>>> their particular hardware, including the ability to try running
>>> multiple sorts at one time.
>>
>> That sounds like an interesting experiment to carry out and if successful change the released code.
>>
>>>
>>> For the other step, I don't feel like I understand the index-packing
>>> code well enough yet to form an opinion, which is one reason for the
>>> question. It seems that it could run in parallel without difficulty,
>>> but maybe I don't understand the relationships between the indexes
>>> well enough.
>>
>> Index packing is I/O bound and is sequential. There is little computation going on.
>>
>> Doing two packings in parallel would break up the sequential write sequence so there would be need to be a noticable gain in some way to compensate for the impact.
>>
>> Bus contention when it's a SSD may come into play. The quality/speed of the connection to the SSD is related to how much $$$ the server cost!
>>
>>> Another question then would be: maybe we could split the current
>>> 'index' phase into 'order' and 'pack' phases, again for those who
>>> would like to try tuning each step for their situation?
>>
>> Interesting possibility - needs trying out and bedding down before it goes into the standard release scripts IMO. What works well in one environment may not in another. Lots of options suits some people and not others.
>>
>> Andy
>>
>>> ---
>>> A. Soroka
>>> The University of Virginia Library
>>>
>>>> On Oct 28, 2016, at 10:24 AM, Rob Vesse <rv...@dotnetrdf.org> wrote:
>>>>
>>>> If memory serves those are the phases that use POSIX sort right?
>>>>
>>>> Sort will try and do an in-memory sort as far as possible and fall back to a disk-based merge sort if not. Also we usually configure sort to run in parallel
>>>>
>>>> If you try to process different indexing in parallel you would create a lot of memory and disk contention which would likely slowdown overall performance
>>>>
>>>> For sufficiently large data sets there is also a risk of exhausting disk space during the sort phase and building multiple indexes in parallel would only exacerbate this
>>>>
>>>> Rob
>>>>
>>>> On 28/10/2016 14:33, "A. Soroka" <aj...@virginia.edu> wrote:
>>>>
>>>> I'm still learning about tdbloader2 and have another question about the index phase: is there any reason why the processes for the various index orderings (SPO, GSPO, etc.) couldn't go on in parallel? Or am I missing some switch or setting that already allows that?
>>>>
>>>> ---
>>>> A. Soroka
>>>> The University of Virginia Library
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>
>
Re: another naive question about tdbloader2
Posted by "A. Soroka" <aj...@virginia.edu>.
Okay, that's pretty much the sensible answer I expected: "YMMV, try some experiments and let's work with hard data."
I need to reread Andy's post [1] about the packing algorithm to make sure I understand what's going on there.
The hardware in question is definitely "$$$"-- maybe even "$$$$". {grin} This is contract work, so before going too much further I need to check with my principals to prioritize this kind of speculative work and we will make a plan accordingly.
I'm not sure immediately how to measure the rates in terms of tuples/sec-- I'm not aware of any such logging from sort itself, but I haven't looked for it. I assume that tdbloader2 can emit messages as it packs tuples in the same way that it emits messages with an average rate periodically in the 'data' phase?
---
A. Soroka
The University of Virginia Library
[1] https://seaborne.blogspot.com/2010/12/repacking-btrees.html
> On Oct 28, 2016, at 11:12 AM, Andy Seaborne <an...@apache.org> wrote:
>
>
>
> On 28/10/16 15:58, A. Soroka wrote:
>> That's right-- if I understand it correctly, there are two steps--
>> POSIX sort to develop the index orderings, and then packing the
>> actual index files.
>>
>> For the POSIX sort step, it's certainly true that more parallelism
>> than needed would be a bad thing. With Andy's help I just made a
>> commit that allows a little more control using the common --parallel
>> flag for sort. But the current ergonomics seem suboptimal. E.g. with
>> the current settings indexing a 300Mt dataset on a 24-core box with
>> fast storage, I saw only one core in full use, and very little IO
>> usage. Aliasing in some parallelism via the sort flag brought several
>> more cores into play and cut the time spent by two-thirds.
>
> What rates are you getting?
>
>> I don't
>> know how normal that is, but for the sort step, my argument is not
>> that we could find universally better ergonomics, but that we could
>> bake some flexibility in for those who want to try adjustments on
>> their particular hardware, including the ability to try running
>> multiple sorts at one time.
>
> That sounds like an interesting experiment to carry out and if successful change the released code.
>
>>
>> For the other step, I don't feel like I understand the index-packing
>> code well enough yet to form an opinion, which is one reason for the
>> question. It seems that it could run in parallel without difficulty,
>> but maybe I don't understand the relationships between the indexes
>> well enough.
>
> Index packing is I/O bound and is sequential. There is little computation going on.
>
> Doing two packings in parallel would break up the sequential write sequence so there would be need to be a noticable gain in some way to compensate for the impact.
>
> Bus contention when it's a SSD may come into play. The quality/speed of the connection to the SSD is related to how much $$$ the server cost!
>
>> Another question then would be: maybe we could split the current
>> 'index' phase into 'order' and 'pack' phases, again for those who
>> would like to try tuning each step for their situation?
>
> Interesting possibility - needs trying out and bedding down before it goes into the standard release scripts IMO. What works well in one environment may not in another. Lots of options suits some people and not others.
>
> Andy
>
>> ---
>> A. Soroka
>> The University of Virginia Library
>>
>>> On Oct 28, 2016, at 10:24 AM, Rob Vesse <rv...@dotnetrdf.org> wrote:
>>>
>>> If memory serves those are the phases that use POSIX sort right?
>>>
>>> Sort will try and do an in-memory sort as far as possible and fall back to a disk-based merge sort if not. Also we usually configure sort to run in parallel
>>>
>>> If you try to process different indexing in parallel you would create a lot of memory and disk contention which would likely slowdown overall performance
>>>
>>> For sufficiently large data sets there is also a risk of exhausting disk space during the sort phase and building multiple indexes in parallel would only exacerbate this
>>>
>>> Rob
>>>
>>> On 28/10/2016 14:33, "A. Soroka" <aj...@virginia.edu> wrote:
>>>
>>> I'm still learning about tdbloader2 and have another question about the index phase: is there any reason why the processes for the various index orderings (SPO, GSPO, etc.) couldn't go on in parallel? Or am I missing some switch or setting that already allows that?
>>>
>>> ---
>>> A. Soroka
>>> The University of Virginia Library
>>>
>>>
>>>
>>>
>>>
>>>
>>
Re: another naive question about tdbloader2
Posted by Andy Seaborne <an...@apache.org>.
On 28/10/16 15:58, A. Soroka wrote:
> That's right-- if I understand it correctly, there are two steps--
> POSIX sort to develop the index orderings, and then packing the
> actual index files.
>
> For the POSIX sort step, it's certainly true that more parallelism
> than needed would be a bad thing. With Andy's help I just made a
> commit that allows a little more control using the common --parallel
> flag for sort. But the current ergonomics seem suboptimal. E.g. with
> the current settings indexing a 300Mt dataset on a 24-core box with
> fast storage, I saw only one core in full use, and very little IO
> usage. Aliasing in some parallelism via the sort flag brought several
> more cores into play and cut the time spent by two-thirds.
What rates are you getting?
> I don't
> know how normal that is, but for the sort step, my argument is not
> that we could find universally better ergonomics, but that we could
> bake some flexibility in for those who want to try adjustments on
> their particular hardware, including the ability to try running
> multiple sorts at one time.
That sounds like an interesting experiment to carry out and if
successful change the released code.
>
> For the other step, I don't feel like I understand the index-packing
> code well enough yet to form an opinion, which is one reason for the
> question. It seems that it could run in parallel without difficulty,
> but maybe I don't understand the relationships between the indexes
> well enough.
Index packing is I/O bound and is sequential. There is little
computation going on.
Doing two packings in parallel would break up the sequential write
sequence so there would be need to be a noticable gain in some way to
compensate for the impact.
Bus contention when it's a SSD may come into play. The quality/speed of
the connection to the SSD is related to how much $$$ the server cost!
> Another question then would be: maybe we could split the current
> 'index' phase into 'order' and 'pack' phases, again for those who
> would like to try tuning each step for their situation?
Interesting possibility - needs trying out and bedding down before it
goes into the standard release scripts IMO. What works well in one
environment may not in another. Lots of options suits some people and
not others.
Andy
> ---
> A. Soroka
> The University of Virginia Library
>
>> On Oct 28, 2016, at 10:24 AM, Rob Vesse <rv...@dotnetrdf.org> wrote:
>>
>> If memory serves those are the phases that use POSIX sort right?
>>
>> Sort will try and do an in-memory sort as far as possible and fall back to a disk-based merge sort if not. Also we usually configure sort to run in parallel
>>
>> If you try to process different indexing in parallel you would create a lot of memory and disk contention which would likely slowdown overall performance
>>
>> For sufficiently large data sets there is also a risk of exhausting disk space during the sort phase and building multiple indexes in parallel would only exacerbate this
>>
>> Rob
>>
>> On 28/10/2016 14:33, "A. Soroka" <aj...@virginia.edu> wrote:
>>
>> I'm still learning about tdbloader2 and have another question about the index phase: is there any reason why the processes for the various index orderings (SPO, GSPO, etc.) couldn't go on in parallel? Or am I missing some switch or setting that already allows that?
>>
>> ---
>> A. Soroka
>> The University of Virginia Library
>>
>>
>>
>>
>>
>>
>
Re: another naive question about tdbloader2
Posted by "A. Soroka" <aj...@virginia.edu>.
That's right-- if I understand it correctly, there are two steps-- POSIX sort to develop the index orderings, and then packing the actual index files.
For the POSIX sort step, it's certainly true that more parallelism than needed would be a bad thing. With Andy's help I just made a commit that allows a little more control using the common --parallel flag for sort. But the current ergonomics seem suboptimal. E.g. with the current settings indexing a 300Mt dataset on a 24-core box with fast storage, I saw only one core in full use, and very little IO usage. Aliasing in some parallelism via the sort flag brought several more cores into play and cut the time spent by two-thirds. I don't know how normal that is, but for the sort step, my argument is not that we could find universally better ergonomics, but that we could bake some flexibility in for those who want to try adjustments on their particular hardware, including the ability to try running multiple sorts at one time.
For the other step, I don't feel like I understand the index-packing code well enough yet to form an opinion, which is one reason for the question. It seems that it could run in parallel without difficulty, but maybe I don't understand the relationships between the indexes well enough.
Another question then would be: maybe we could split the current 'index' phase into 'order' and 'pack' phases, again for those who would like to try tuning each step for their situation?
---
A. Soroka
The University of Virginia Library
> On Oct 28, 2016, at 10:24 AM, Rob Vesse <rv...@dotnetrdf.org> wrote:
>
> If memory serves those are the phases that use POSIX sort right?
>
> Sort will try and do an in-memory sort as far as possible and fall back to a disk-based merge sort if not. Also we usually configure sort to run in parallel
>
> If you try to process different indexing in parallel you would create a lot of memory and disk contention which would likely slowdown overall performance
>
> For sufficiently large data sets there is also a risk of exhausting disk space during the sort phase and building multiple indexes in parallel would only exacerbate this
>
> Rob
>
> On 28/10/2016 14:33, "A. Soroka" <aj...@virginia.edu> wrote:
>
> I'm still learning about tdbloader2 and have another question about the index phase: is there any reason why the processes for the various index orderings (SPO, GSPO, etc.) couldn't go on in parallel? Or am I missing some switch or setting that already allows that?
>
> ---
> A. Soroka
> The University of Virginia Library
>
>
>
>
>
>
Re: another naive question about tdbloader2
Posted by Andy Seaborne <an...@apache.org>.
Each index sort could happen in in parallel so that's above and beyond
--parallel to sort(1) which make each sort parallel.
That can also saturate the machine!
As Rob says, the memory and disk contention can kill performance - two
streams of writes in parallel can be slower than one stream followed by
the other.
This is a strong effect when it is to the same rotating disk but seems
to be true as well for SSD to a lesser extent. I guess the bottleneck
is contention for the system bus access and it is optimized for the
common "no contention" case.
A high end server may have multiple independent disk paths to eliminate
this.
It needs quite a lot of data before tdbloader2 overtakes the plain
tdbloader. That point is sensitive to the hardware.
Andy
On 28/10/16 15:24, Rob Vesse wrote:
> If memory serves those are the phases that use POSIX sort right?
>
> Sort will try and do an in-memory sort as far as possible and fall back to a disk-based merge sort if not. Also we usually configure sort to run in parallel
>
> If you try to process different indexing in parallel you would create a lot of memory and disk contention which would likely slowdown overall performance
>
> For sufficiently large data sets there is also a risk of exhausting disk space during the sort phase and building multiple indexes in parallel would only exacerbate this
>
> Rob
>
> On 28/10/2016 14:33, "A. Soroka" <aj...@virginia.edu> wrote:
>
> I'm still learning about tdbloader2 and have another question about the index phase: is there any reason why the processes for the various index orderings (SPO, GSPO, etc.) couldn't go on in parallel? Or am I missing some switch or setting that already allows that?
>
> ---
> A. Soroka
> The University of Virginia Library
>
>
>
>
>
>
Re: another naive question about tdbloader2
Posted by Rob Vesse <rv...@dotnetrdf.org>.
If memory serves those are the phases that use POSIX sort right?
Sort will try and do an in-memory sort as far as possible and fall back to a disk-based merge sort if not. Also we usually configure sort to run in parallel
If you try to process different indexing in parallel you would create a lot of memory and disk contention which would likely slowdown overall performance
For sufficiently large data sets there is also a risk of exhausting disk space during the sort phase and building multiple indexes in parallel would only exacerbate this
Rob
On 28/10/2016 14:33, "A. Soroka" <aj...@virginia.edu> wrote:
I'm still learning about tdbloader2 and have another question about the index phase: is there any reason why the processes for the various index orderings (SPO, GSPO, etc.) couldn't go on in parallel? Or am I missing some switch or setting that already allows that?
---
A. Soroka
The University of Virginia Library