You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@marmotta.apache.org by Mark Breedlove <mb...@dp.la> on 2015/09/18 21:37:27 UTC
Scaling Marmotta's LDP interface
Hello, Marmotta Users,
At the Digital Public Library of America, we have a large Marmotta
triplestore, with which we interact entirely over LDP.
We're looking for some advice about scaling Marmotta's LDP interface past
our current size. In the short term, we are hoping that we can find ways to
tune PostgreSQL to mitigate some problems we have seen; in the long term,
we are open to advice about alternate backends.
A high-level overview of how we interact with our LDP Resources is
documented in [1]. While we have had to do some LDP-specific tuning
(especially introducing a partial index on `triples.context`) for all
processes, we have seen particular trouble in cases where we GET,
transform, then PUT an LDP RDFSource (see: *Enrichment *in the overview
link).
That overview is part of a greater wiki that we've put together to document
our installation and performance-tuning activities [2].
Our biggest problem at the moment is addressing slow updates and inserts
[3], observed when we GET and PUT those RDFSources with two concurrent
mapping or enrichment activities. If we run one of these activities,
GETing, transforming, and PUTing in serial, performance seems to be network
and CPU bound, and is not very bad. But as soon as we run a second mapping
or enrichment, work performed grinds practically to a halt, as described in
[3].
To give you a sense of the scale at which we're operating, we have about
two million LDP-RSs, typically including about 50 triples and a handful of
blank nodes (around 5 to 15). Our `triples` table has about 294M rows now
and takes up 32GB for the table, and 13GB each for its two largest indices.
Our entire Marmotta database takes up about 140GB. We've had some successes
with improving index performance with low cardinality in `triples.context`
[4] and tuning the Amazon EC2 instances that we run on [5][6]. The I/O wait
problem with concurrent LDP operations, however, is the new blocker.
Some supplemental information:
* An overview of the project for which Marmotta is being
used:
https://digitalpubliclibraryofamerica.atlassian.net/wiki/display/TECH/Heidrun
* The application (a Rails engine) that makes all of these LDP requests:
https://github.com/dpla/KriKri
* Our configuration-management project, with details on how some of our
stack is configured: https://github.com/dpla/automation
We'd be grateful for any feedback that you might have that would assist us
with handling large volumes of data over LDP. Thanks for your help!
- Mark Breedlove and Tom Johnson,
Digital Public Library of America (http://dp.la/)
tech@dp.la
[1]
https://digitalpubliclibraryofamerica.atlassian.net/wiki/display/TECH/LDP+Interactions+Overview
[2]
https://digitalpubliclibraryofamerica.atlassian.net/wiki/display/TECH/Marmotta
[3]
https://digitalpubliclibraryofamerica.atlassian.net/wiki/display/TECH/Addressing+slow+updates+and+inserts
[4]
https://digitalpubliclibraryofamerica.atlassian.net/wiki/display/TECH/Index+performance+with+high+context+counts
[5]
https://digitalpubliclibraryofamerica.atlassian.net/wiki/display/TECH/Amazon+EC2+adjustments
[6]
https://digitalpubliclibraryofamerica.atlassian.net/wiki/display/TECH/Using+irqbalance+and+SMP+IRQ+affinity
Re: Scaling Marmotta's LDP interface
Posted by Sergio Fernández <wi...@apache.org>.
Hi Mark,
sorry, I've been with limited connectivity for some days, but let's catch
up this.
On Fri, Oct 2, 2015 at 9:46 PM, Mark Breedlove <mb...@dp.la> wrote:
>
> I just wanted to get back to you with the status of our experimentation. I
> still need to gather some more information by running some more of our
> activities, but the activities that were bogged down last week are running
> now without the I/O wait that we were encountering. I'll report back when
> we've run the full range of our processes. (Including processes that do
> more inserts updates.)
>
That are great news! Performance tuning is very tricky.
BTW, since you are already using AWS,have you considered to give a try
to Amazon RDS for PostgreSQL? I have to admit I've never had the chance to
test Marmotta on it, but it'd be really interesting to compare the
performance in scenarios like yours.
Well, internally each LDP-R uses its own context. Therefore all indexes
>>> that could improve the lookup would help. Try to create also at least cop.
>>> [...]
>>>
>>
>> OK, that's on my to-do list and I'll let you know ... :-)
>>
>
> I've created an index, `idx_triples_cop` on triples (context, object,
> predicate) where deleted = false.
>
> I made note of a query last week that is executed frequently, which now
> uses this index:
>
> SELECT
> id,subject,predicate,object,context,deleted,inferred,creator,createdAt,deletedAt
> FROM triples WHERE deleted = false AND context = $1 AND subject = $2 AND
> predicate = $3 AND object = $4
>
> If I `explain analyze` that query, PG says "Index Scan using
> idx_triples_cop on triples".
>
Exactly, that query is executed very frequently in KiWi, so the index helps.
>
> One query that's showing up frequently in our slow query log is now:
>
> SELECT
> id,subject,predicate,object,context,deleted,inferred,creator,createdAt,deletedAt
> FROM triples WHERE deleted = false AND subject = $1 AND predicate = $2 AND
> context = $3
>
> This is supposed to use a full index scan on idx_triples_spo, according to
> `explain analyze`. However, I think that the index is just so huge that a
> full index scan is what's still causing it to take about 6 seconds in a lot
> of cases.
>
Well, specifically the spc index, that could also be quite big.
We're apparently getting a 98% buffer cache hit rate on idx_triples_spo,
> according to a query on pg_statio_user_indexes. The idx_triples_spo index
> is 8GB in size.
>
Good.
> Another thing that I did was to increase work_mem to 100GB (again, before
> adding the memory), which didn't make any difference at the time, but which
> sounds like a good idea in general (and may be have been allowed to show
> its benefits after the filesystem cache was increased).
>
Well, keep observing how that working memory behaves, and maybe you can
decrease it a bit and safe some memory for other components of your setup.
Since we only have about one million ore:Aggregations in our system now,
> out of 11 million, I'm going to have to see how big the triples table has
> to get before we run into more I/O wait issues as a result of the
> filesystem cache / shared buffers running out. But so far so good, insofar
> as the I/O wait has dropped. I will follow up later with you after we try
> some of our harvest and mapping activities [1].
>
> [1]
> https://digitalpubliclibraryofamerica.atlassian.net/wiki/display/TECH/LDP+Interactions+Overview
>
Based on this experience maybe we can add some of that to the project
documentation, so it will help other users in the future.
Thanks for sharing all this with us!
Cheers,
--
Sergio Fernández
Partner Technology Manager
Redlink GmbH
m: +43 6602747925
e: sergio.fernandez@redlink.co
w: http://redlink.co
Re: Scaling Marmotta's LDP interface
Posted by Mark Breedlove <mb...@dp.la>.
Hi, Sergio,
I just wanted to get back to you with the status of our experimentation. I
still need to gather some more information by running some more of our
activities, but the activities that were bogged down last week are running
now without the I/O wait that we were encountering. I'll report back when
we've run the full range of our processes. (Including processes that do
more inserts updates.)
On Fri, Sep 25, 2015 at 3:30 PM, Mark Breedlove <mb...@dp.la> wrote:
>
>
> On Thu, Sep 24, 2015 at 10:47 AM, Sergio Fernández <wi...@apache.org>
> wrote:
>
>>
>> From my experience Postgres takes as much as it can, but never risking
>> the system. We normally also use 40% as orientative figure. But with such
>> small server I'd try to give it more memory (6GB?) to see how it behaves.
>>
>
> OK. I'll experiment with that, too, including, likely, an upgrade of the
> instance type. It will require some scheduling, and will probably take me
> some number of days to have it done.
>
I've upgraded our EC2 instance to 60GB memory (r3.2xlarge) and increased
shared_buffers to 23.9GB (40%).
This is the only thing that has produced a noticeable improvement of all of
the things I said I'd try; but the difference is great.
I have been running two concurrent enrichment [1] activities after each
tuning step that I'll mention here. I upgraded the server memory and
shared_buffers as a last step, so I have observed the situation before and
after each change. After upgrading the memory and shared_buffers, I/O wait
dropped to less than 1% after an initial period of about 15 minutes for the
operating system's filesystem cache to warm up.
> Because that, another key aspect that dramatically improves the
>> performance on updates is I/O. Are you using SSD? Actual SSD, I mean,
>> virtual SSDs are a completely different story.
>>
>
> We're using SSD-backed provisioned IOPS Amazon EBS volumes and
> EBS-optimized instances. [...] I'm not sure I'd want to go to bare-metal
> hardware yet, just for a database of 1 million ore:Aggregations; seems like
> we might want to revisit the choice of backend first. I could, however,
> resize the volumes again and boost the IOPS. I'll have to schedule a time
> to do that.
>
I've added a volume with the fastest requests / second rating that Amazon
provides (20K IOPS) and have transferred the `triples` table to it. This
actually made no difference, as (before the memory upgrade) we were still
maxing out the volume's requests-per-second with the enrichment activities
mentioned above.
By the way, in addition to the requests / second limit, EBS volumes deliver
a maximum throughput of 320MiB / second, but we don't get close to it. The
most I've seen us attain is 16 to 20 MiB / second. It's always the
requests per second that gets us.
> Could you point me in the right direction for the indices `cop`, `cp`, and
>>> `literals`? We have `idx_triples_cspo` on `triples` from the Marmotta
>>> installation and created one that we called `idx_triples_c` [4]. Are there
>>> optional indices that Marmotta doesn't create when it automatically creates
>>> the tables the first time you run it?
>>>
>>
>> Well, internally each LDP-R uses its own context. Therefore all indexes
>> that could improve the lookup would help. Try to create also at least cop.
>> [...]
>>
>
> OK, that's on my to-do list and I'll let you know ... :-)
>
I've created an index, `idx_triples_cop` on triples (context, object,
predicate) where deleted = false.
I made note of a query last week that is executed frequently, which now
uses this index:
SELECT
id,subject,predicate,object,context,deleted,inferred,creator,createdAt,deletedAt
FROM triples WHERE deleted = false AND context = $1 AND subject = $2 AND
predicate = $3 AND object = $4
If I `explain analyze` that query, PG says "Index Scan using
idx_triples_cop on triples".
One query that's showing up frequently in our slow query log is now:
SELECT
id,subject,predicate,object,context,deleted,inferred,creator,createdAt,deletedAt
FROM triples WHERE deleted = false AND subject = $1 AND predicate = $2 AND
context = $3
This is supposed to use a full index scan on idx_triples_spo, according to
`explain analyze`. However, I think that the index is just so huge that a
full index scan is what's still causing it to take about 6 seconds in a lot
of cases.
We're apparently getting a 98% buffer cache hit rate on idx_triples_spo,
according to a query on pg_statio_user_indexes. The idx_triples_spo index
is 8GB in size.
Another thing that I did was to increase work_mem to 100GB (again, before
adding the memory), which didn't make any difference at the time, but which
sounds like a good idea in general (and may be have been allowed to show
its benefits after the filesystem cache was increased).
Since we only have about one million ore:Aggregations in our system now,
out of 11 million, I'm going to have to see how big the triples table has
to get before we run into more I/O wait issues as a result of the
filesystem cache / shared buffers running out. But so far so good, insofar
as the I/O wait has dropped. I will follow up later with you after we try
some of our harvest and mapping activities [1].
Thanks for your help!
-Mark
[1]
https://digitalpubliclibraryofamerica.atlassian.net/wiki/display/TECH/LDP+Interactions+Overview
Re: Scaling Marmotta's LDP interface
Posted by Mark Breedlove <mb...@dp.la>.
On Thu, Sep 24, 2015 at 10:47 AM, Sergio Fernández <wi...@apache.org>
wrote:
>
> From my experience Postgres takes as much as it can, but never risking the
> system. We normally also use 40% as orientative figure. But with such small
> server I'd try to give it more memory (6GB?) to see how it behaves.
>
OK. I'll experiment with that, too, including, likely, an upgrade of the
instance type. It will require some scheduling, and will probably take me
some number of days to have it done.
> Because that, another key aspect that dramatically improves the
> performance on updates is I/O. Are you using SSD? Actual SSD, I mean,
> virtual SSDs are a completely different story.
>
We're using SSD-backed provisioned IOPS Amazon EBS volumes[1] and
EBS-optimized instances. That would mean, of course, that they are
virtualized. They're rated currently at 12,000 IOPS (Amazon's own metric
that corresponds to actual read/write operations in ways that can be
difficult to calculate).
We've raised the bandwidth rating of the volumes once already and suppose
that we can try that some more, though one encounters a similar problem to
increasing the shared buffers: figuring out how much is the right amount,
projecting for the growing size of the database and transactions per
second. (How far do we have to go?) The problem of one activity achieving
satisfactory performance with low transactions per second and two
concurrent ones displaying many multiples the amount is what interests me
the most at the moment.
I'm not sure I'd want to go to bare-metal hardware yet, just for a database
of 1 million ore:Aggregations; seems like we might want to revisit the
choice of backend first. I could, however, resize the volumes again and
boost the IOPS. I'll have to schedule a time to do that.
Could you point me in the right direction for the indices `cop`, `cp`, and
>> `literals`? We have `idx_triples_cspo` on `triples` from the Marmotta
>> installation and created one that we called `idx_triples_c` [4]. Are there
>> optional indices that Marmotta doesn't create when it automatically creates
>> the tables the first time you run it?
>>
>
> Well, internally each LDP-R uses its own context. Therefore all indexes
> that could improve the lookup would help. Try to create also at least cop.
> [...]
>
OK, that's on my to-do list and I'll let you know ... :-)
Re: Scaling Marmotta's LDP interface
Posted by Sergio Fernández <wi...@apache.org>.
Hi Mark,
On Thu, Sep 24, 2015 at 12:53 AM, Mark Breedlove <mb...@dp.la> wrote:
>
> * max_connections: default is 100, 1000 is at least recommended, but goes
>> together with the available memory (see next)
>>
>
> Our Marmotta tends to use just 10 connections, and we have max_connections
> set to 150 for the server, which contains a few other databases in addition
> to Marmotta's that have very light usage. The only time we've hit the
> max_connections limit was during a period when locks were not being
> released. During a normal mapping or enrichment activity, even with the
> problems I've mentioned, we seldom go above 10 connections for the Marmotta
> database, so I'm hesitant to raise the potential memory usage of the
> system, which would result in paging. I'm focused at the moment on stopping
> the problem of the locks being held for so long.
>
I do agree with that. We should find the actual source of those locks.
* shared_buffers: default is just 32MB, as much as you give the more
>> postres would not need to touch disk for any operation (we have
>> installations with 16GB behaving really well)
>>
>
> We have this set to 40% of our real memory -- 3072MB on a 7.5GB server.
> The 40% ratio is based on a statement in the postgresql.org wiki[2],
> which states "given the way PostgreSQL also relies on the operating system
> cache, it's unlikely you'll find using more than 40% of RAM to work better
> than a smaller amount." Would you agree with the basic ratio, or would you
> suggest raising it?
>
> The table for triples alone is 32GB, so I could see adding more memory and
> increasing shared_buffers, but I'm not sure if I'd ever be able to add
> enough to cover the expanding size of the table.
>
>From my experience Postgres takes as much as it can, but never risking the
system. We normally also use 40% as orientative figure. But with such small
server I'd try to give it more memory (6GB?) to see how it behaves.
Because that, another key aspect that dramatically improves the performance
on updates is I/O. Are you using SSD? Actual SSD, I mean, virtual SSDs are
a completely different story.
> * work_mem: default is 4MB, you can dramatically increase it (128 or 256
>> MB) to improve each each transaction handling
>>
> * maintenance_work_mem: is less critical, but useful on maintenance tasks
>> you periodically should run (see below)
>>
>
> work_mem is 38MB at the moment. I'll try increasing it and will let you
> know what I encounter.
>
That should have an impact on handling concurrent transaction. Let's see...
* If you have resources, create all the indexes (cspo, cop, cp, literals,
>> etc) you may need to improve performance
>>
>
> Could you point me in the right direction for the indices `cop`, `cp`, and
> `literals`? We have `idx_triples_cspo` on `triples` from the Marmotta
> installation and created one that we called `idx_triples_c` [4]. Are there
> optional indices that Marmotta doesn't create when it automatically creates
> the tables the first time you run it?
>
Well, internally each LDP-R uses its own context. Therefore all indexes
that could improve the lookup would help. Try to create also at least cop.
In the end you can tune your indexes based on your needs and available
resources. Marmotta does not create all possible indexes by default, but
you can do it if you consider it.
* If you do not use versioning, cleanup periodically (nightly) deleted
>> triples: DELETE FROM triples where deleted = true;
>>
>
> Thanks for clarifying that those rows can be deleted. We've been assuming
> that we want to use versioning in the future, and will need to talk among
> ourselves about the implications of foregoing versioning. I believe that
> the main reason for using the KiWi, and thus PostgreSQL, was to have the
> possibility of versioning.
>
Then discuss it. If you do not need it, cleaning delete triples also helps.
Thanks again. Does the information referenced above and elsewhere on our
> wiki clarify how we have things tuned? Our biggest problem at the moment
> is the slowness of UPDATE statements when we run concurrent LDP requests,
> detailed in [3]. We see locks piling up, but need to ascertain whether
> that's a cause or a symptom of something else. The thing that's most
> notable is how a single batch of LDP requests (in the course of one of our
> mapping or enrichment runs) works very efficiently, but two concurrent
> batches of LDP requests have severely degraded performance. I just want to
> make sure first that nothing in our database tuning jumps out as a culprit
> before suggesting changes to the code, yours our ours.
>
Thanks, having more hints about your setup help quite a lot. I'd say that,
based on my experience with virtual disks, the I/O could be the source of
the issues. So either you move to a more performance disk or we find
strategies to improve the in-memory work.
Sorry for not being so helpful. Performance tuning is very tricky.
Cheers,
--
Sergio Fernández
Partner Technology Manager
Redlink GmbH
m: +43 6602747925
e: sergio.fernandez@redlink.co
w: http://redlink.co
Re: Scaling Marmotta's LDP interface
Posted by Mark Breedlove <mb...@dp.la>.
On Tue, Sep 22, 2015 at 2:22 AM, Sergio Fernández <wi...@apache.org> wrote:
>
> take into account that by default PostgreSQL is very very conservative.
> Have you customized it?
>
Thanks for your help with this!
Yes, we've done quite a bit of tuning, as described in [1], [3], and [4].
* max_connections: default is 100, 1000 is at least recommended, but goes
> together with the available memory (see next)
>
Our Marmotta tends to use just 10 connections, and we have max_connections
set to 150 for the server, which contains a few other databases in addition
to Marmotta's that have very light usage. The only time we've hit the
max_connections limit was during a period when locks were not being
released. During a normal mapping or enrichment activity, even with the
problems I've mentioned, we seldom go above 10 connections for the Marmotta
database, so I'm hesitant to raise the potential memory usage of the
system, which would result in paging. I'm focused at the moment on stopping
the problem of the locks being held for so long.
> * shared_buffers: default is just 32MB, as much as you give the more
> postres would not need to touch disk for any operation (we have
> installations with 16GB behaving really well)
>
We have this set to 40% of our real memory -- 3072MB on a 7.5GB server.
The 40% ratio is based on a statement in the postgresql.org wiki[2], which
states "given the way PostgreSQL also relies on the operating system cache,
it's unlikely you'll find using more than 40% of RAM to work better than a
smaller amount." Would you agree with the basic ratio, or would you
suggest raising it?
The table for triples alone is 32GB, so I could see adding more memory and
increasing shared_buffers, but I'm not sure if I'd ever be able to add
enough to cover the expanding size of the table.
> * work_mem: default is 4MB, you can dramatically increase it (128 or 256
> MB) to improve each each transaction handling
>
* maintenance_work_mem: is less critical, but useful on maintenance tasks
> you periodically should run (see below)
>
work_mem is 38MB at the moment. I'll try increasing it and will let you
know what I encounter. maintenance_work_mem is 100MB.
>
> * checkpoint_segments: by default is just 3, but a much larger value
> improves transaction handling
>
checkpoint_segments is 128, and checkpoint_timeout is 20min[3].
> * think about your vacuum strategy
>
We've adjusted our autovacuuming per [1].
> * If you have resources, create all the indexes (cspo, cop, cp, literals,
> etc) you may need to improve performance
>
Could you point me in the right direction for the indices `cop`, `cp`, and
`literals`? We have `idx_triples_cspo` on `triples` from the Marmotta
installation and created one that we called `idx_triples_c` [4]. Are there
optional indices that Marmotta doesn't create when it automatically creates
the tables the first time you run it?
* If you do not use versioning, cleanup periodically (nightly) deleted
> triples: DELETE FROM triples where deleted = true;
>
Thanks for clarifying that those rows can be deleted. We've been assuming
that we want to use versioning in the future, and will need to talk among
ourselves about the implications of foregoing versioning. I believe that
the main reason for using the KiWi, and thus PostgreSQL, was to have the
possibility of versioning.
> Hope that helps. Try to implement some of those suggestion in your system
> and tell us how they behave and where you still need more help, and maybe
> code patching.
>
Thanks again. Does the information referenced above and elsewhere on our
wiki clarify how we have things tuned? Our biggest problem at the moment
is the slowness of UPDATE statements when we run concurrent LDP requests,
detailed in [3]. We see locks piling up, but need to ascertain whether
that's a cause or a symptom of something else. The thing that's most
notable is how a single batch of LDP requests (in the course of one of our
mapping or enrichment runs) works very efficiently, but two concurrent
batches of LDP requests have severely degraded performance. I just want to
make sure first that nothing in our database tuning jumps out as a culprit
before suggesting changes to the code, yours our ours.
References:
[1]
https://digitalpubliclibraryofamerica.atlassian.net/wiki/pages/viewpage.action?pageId=26804351
[2] https://wiki.postgresql.org/wiki/Tuning_Your_PostgreSQL_Server
[3]
https://digitalpubliclibraryofamerica.atlassian.net/wiki/display/TECH/Addressing+slow+updates+and+inserts
[4]
https://digitalpubliclibraryofamerica.atlassian.net/wiki/display/TECH/Index+performance+with+high+context+counts
Re: Scaling Marmotta's LDP interface
Posted by Sergio Fernández <wi...@apache.org>.
Hi Mark and Tom,
take into account that by default PostgreSQL is very very conservative.
Have you customized it?
In the official wiki you can find some useful information:
https://wiki.postgresql.org/wiki/Tuning_Your_PostgreSQL_Server
We typically modify the following settings in deployments environments, but
some may need to be adapted depending on the concrete scenario and
available resources:
* max_connections: default is 100, 1000 is at least recommended, but goes
together with the available memory (see next)
* shared_buffers: default is just 32MB, as much as you give the more
postres would not need to touch disk for any operation (we have
installations with 16GB behaving really well)
* work_mem: default is 4MB, you can dramatically increase it (128 or 256
MB) to improve each each transaction handling
* maintenance_work_mem: is less critical, but useful on maintenance tasks
you periodically should run (see below)
* checkpoint_segments: by default is just 3, but a much larger value
improves transaction handling
* think about your vacuum strategy
I'm not an expert on performance tuning, but I'm pretty sure that with some
time your sysadmin would manage to find the right settings for your
installation.
In addition to the general Posgres stuff, for Marmotta there are few more
things that are critical:
* If you have resources, create all the indexes (cspo, cop, cp, literals,
etc) you may need to improve performance
* If you do not use versioning, cleanup periodically (nightly) deleted
triples: DELETE FROM triples where deleted = true;
Hope that helps. Try to implement some of those suggestion in your system
and tell us how they behave and where you still need more help, and maybe
code patching.
Cheers,
On Fri, Sep 18, 2015 at 9:37 PM, Mark Breedlove <mb...@dp.la> wrote:
>
> Hello, Marmotta Users,
>
> At the Digital Public Library of America, we have a large Marmotta
> triplestore, with which we interact entirely over LDP.
>
> We're looking for some advice about scaling Marmotta's LDP interface past
> our current size. In the short term, we are hoping that we can find ways to
> tune PostgreSQL to mitigate some problems we have seen; in the long term,
> we are open to advice about alternate backends.
>
> A high-level overview of how we interact with our LDP Resources is
> documented in [1]. While we have had to do some LDP-specific tuning
> (especially introducing a partial index on `triples.context`) for all
> processes, we have seen particular trouble in cases where we GET,
> transform, then PUT an LDP RDFSource (see: *Enrichment *in the overview
> link).
>
> That overview is part of a greater wiki that we've put together to
> document our installation and performance-tuning activities [2].
>
> Our biggest problem at the moment is addressing slow updates and inserts
> [3], observed when we GET and PUT those RDFSources with two concurrent
> mapping or enrichment activities. If we run one of these activities,
> GETing, transforming, and PUTing in serial, performance seems to be network
> and CPU bound, and is not very bad. But as soon as we run a second mapping
> or enrichment, work performed grinds practically to a halt, as described in
> [3].
>
> To give you a sense of the scale at which we're operating, we have about
> two million LDP-RSs, typically including about 50 triples and a handful of
> blank nodes (around 5 to 15). Our `triples` table has about 294M rows now
> and takes up 32GB for the table, and 13GB each for its two largest indices.
> Our entire Marmotta database takes up about 140GB. We've had some successes
> with improving index performance with low cardinality in `triples.context`
> [4] and tuning the Amazon EC2 instances that we run on [5][6]. The I/O wait
> problem with concurrent LDP operations, however, is the new blocker.
>
> Some supplemental information:
>
> * An overview of the project for which Marmotta is being
> used:
> https://digitalpubliclibraryofamerica.atlassian.net/wiki/display/TECH/Heidrun
>
> * The application (a Rails engine) that makes all of these LDP requests:
> https://github.com/dpla/KriKri
>
> * Our configuration-management project, with details on how some of our
> stack is configured: https://github.com/dpla/automation
>
> We'd be grateful for any feedback that you might have that would assist us
> with handling large volumes of data over LDP. Thanks for your help!
>
> - Mark Breedlove and Tom Johnson,
> Digital Public Library of America (http://dp.la/)
> tech@dp.la
>
>
> [1]
> https://digitalpubliclibraryofamerica.atlassian.net/wiki/display/TECH/LDP+Interactions+Overview
> [2]
> https://digitalpubliclibraryofamerica.atlassian.net/wiki/display/TECH/Marmotta
> [3]
> https://digitalpubliclibraryofamerica.atlassian.net/wiki/display/TECH/Addressing+slow+updates+and+inserts
> [4]
> https://digitalpubliclibraryofamerica.atlassian.net/wiki/display/TECH/Index+performance+with+high+context+counts
> [5]
> https://digitalpubliclibraryofamerica.atlassian.net/wiki/display/TECH/Amazon+EC2+adjustments
> [6]
> https://digitalpubliclibraryofamerica.atlassian.net/wiki/display/TECH/Using+irqbalance+and+SMP+IRQ+affinity
>
--
Sergio Fernández
Partner Technology Manager
Redlink GmbH
m: +43 6602747925
e: sergio.fernandez@redlink.co
w: http://redlink.co
Re: Scaling Marmotta's LDP interface
Posted by "Mark A. Matienzo" <ma...@gmail.com>.
Hi Raffaele,
I'm writing on behalf of Mark and Tom, who are currently in transit to a
conference. To be clear, we're not immediately looking to use a different
backend, although we may consider that at a later date. As Mark and Tom
noted above, we're having some serious performance issues in our production
environment, which runs Postgres, and this is where we're seeking advice on
improving that performance.
Our immediate concern is the notably poor performance of the LDP
implementation when we have more than one job interacting with it. The
Marmotta documentation strongly recommends PostgreSQL in several places
[A], [B], [C]. In particular, the wiki page on performance tuning [C] seems
to suggest that an "adequately-tuned" Postgres server will give us
reasonable performance.
Thanks again, and please feel free to follow up with us if you have
additional questions.
[A] http://marmotta.apache.org/configuration.html
[B] http://marmotta.apache.org/kiwi/triplestore.html
[C] http://wiki.apache.org/marmotta/PerformanceTuning
Best,
Mark A. Matienzo, Director of Technology
Digital Public Library of America | http://dp.la
mark@dp.la
On Sat, Sep 19, 2015 at 12:27 PM, Raffaele Palmieri <
raffaele.palmieri@gmail.com> wrote:
> Hello Mark and Tom, in the past some big data backends were implemented
> but not distributed in official release, such as Big Data Triple Store, now
> Blazegraph [1], Titan [2] or Accumulo Graph [3]. Personally I tested this
> last integration option, and it worked fine, although I didn't any
> benchmark.
> To build Marmotta with a different backend, you need build it with the
> corresponding profile in maven configurator and set the specific properties.
> Hoping that I've given you useful information,
> regards,
> Raffaele.
>
> [1] https://www.blazegraph.com/
> [2] http://thinkaurelius.github.io/titan/
> [3] https://github.com/JHUAPL/AccumuloGraph
>
>
>
>
> 2015-09-18 21:37 GMT+02:00 Mark Breedlove <mb...@dp.la>:
>
>>
>> Hello, Marmotta Users,
>>
>> At the Digital Public Library of America, we have a large Marmotta
>> triplestore, with which we interact entirely over LDP.
>>
>> We're looking for some advice about scaling Marmotta's LDP interface past
>> our current size. In the short term, we are hoping that we can find ways to
>> tune PostgreSQL to mitigate some problems we have seen; in the long term,
>> we are open to advice about alternate backends.
>>
>> A high-level overview of how we interact with our LDP Resources is
>> documented in [1]. While we have had to do some LDP-specific tuning
>> (especially introducing a partial index on `triples.context`) for all
>> processes, we have seen particular trouble in cases where we GET,
>> transform, then PUT an LDP RDFSource (see: *Enrichment *in the overview
>> link).
>>
>> That overview is part of a greater wiki that we've put together to
>> document our installation and performance-tuning activities [2].
>>
>> Our biggest problem at the moment is addressing slow updates and inserts
>> [3], observed when we GET and PUT those RDFSources with two concurrent
>> mapping or enrichment activities. If we run one of these activities,
>> GETing, transforming, and PUTing in serial, performance seems to be network
>> and CPU bound, and is not very bad. But as soon as we run a second mapping
>> or enrichment, work performed grinds practically to a halt, as described in
>> [3].
>>
>> To give you a sense of the scale at which we're operating, we have about
>> two million LDP-RSs, typically including about 50 triples and a handful of
>> blank nodes (around 5 to 15). Our `triples` table has about 294M rows now
>> and takes up 32GB for the table, and 13GB each for its two largest indices.
>> Our entire Marmotta database takes up about 140GB. We've had some successes
>> with improving index performance with low cardinality in `triples.context`
>> [4] and tuning the Amazon EC2 instances that we run on [5][6]. The I/O wait
>> problem with concurrent LDP operations, however, is the new blocker.
>>
>> Some supplemental information:
>>
>> * An overview of the project for which Marmotta is being
>> used:
>> https://digitalpubliclibraryofamerica.atlassian.net/wiki/display/TECH/Heidrun
>>
>> * The application (a Rails engine) that makes all of these LDP requests:
>> https://github.com/dpla/KriKri
>>
>> * Our configuration-management project, with details on how some of our
>> stack is configured: https://github.com/dpla/automation
>>
>> We'd be grateful for any feedback that you might have that would assist
>> us with handling large volumes of data over LDP. Thanks for your help!
>>
>> - Mark Breedlove and Tom Johnson,
>> Digital Public Library of America (http://dp.la/)
>> tech@dp.la
>>
>>
>> [1]
>> https://digitalpubliclibraryofamerica.atlassian.net/wiki/display/TECH/LDP+Interactions+Overview
>> [2]
>> https://digitalpubliclibraryofamerica.atlassian.net/wiki/display/TECH/Marmotta
>> [3]
>> https://digitalpubliclibraryofamerica.atlassian.net/wiki/display/TECH/Addressing+slow+updates+and+inserts
>> [4]
>> https://digitalpubliclibraryofamerica.atlassian.net/wiki/display/TECH/Index+performance+with+high+context+counts
>> [5]
>> https://digitalpubliclibraryofamerica.atlassian.net/wiki/display/TECH/Amazon+EC2+adjustments
>> [6]
>> https://digitalpubliclibraryofamerica.atlassian.net/wiki/display/TECH/Using+irqbalance+and+SMP+IRQ+affinity
>>
>
>
Re: Scaling Marmotta's LDP interface
Posted by Raffaele Palmieri <ra...@gmail.com>.
Hello Mark and Tom, in the past some big data backends were implemented but
not distributed in official release, such as Big Data Triple Store, now
Blazegraph [1], Titan [2] or Accumulo Graph [3]. Personally I tested this
last integration option, and it worked fine, although I didn't any
benchmark.
To build Marmotta with a different backend, you need build it with the
corresponding profile in maven configurator and set the specific properties.
Hoping that I've given you useful information,
regards,
Raffaele.
[1] https://www.blazegraph.com/
[2] http://thinkaurelius.github.io/titan/
[3] https://github.com/JHUAPL/AccumuloGraph
2015-09-18 21:37 GMT+02:00 Mark Breedlove <mb...@dp.la>:
>
> Hello, Marmotta Users,
>
> At the Digital Public Library of America, we have a large Marmotta
> triplestore, with which we interact entirely over LDP.
>
> We're looking for some advice about scaling Marmotta's LDP interface past
> our current size. In the short term, we are hoping that we can find ways to
> tune PostgreSQL to mitigate some problems we have seen; in the long term,
> we are open to advice about alternate backends.
>
> A high-level overview of how we interact with our LDP Resources is
> documented in [1]. While we have had to do some LDP-specific tuning
> (especially introducing a partial index on `triples.context`) for all
> processes, we have seen particular trouble in cases where we GET,
> transform, then PUT an LDP RDFSource (see: *Enrichment *in the overview
> link).
>
> That overview is part of a greater wiki that we've put together to
> document our installation and performance-tuning activities [2].
>
> Our biggest problem at the moment is addressing slow updates and inserts
> [3], observed when we GET and PUT those RDFSources with two concurrent
> mapping or enrichment activities. If we run one of these activities,
> GETing, transforming, and PUTing in serial, performance seems to be network
> and CPU bound, and is not very bad. But as soon as we run a second mapping
> or enrichment, work performed grinds practically to a halt, as described in
> [3].
>
> To give you a sense of the scale at which we're operating, we have about
> two million LDP-RSs, typically including about 50 triples and a handful of
> blank nodes (around 5 to 15). Our `triples` table has about 294M rows now
> and takes up 32GB for the table, and 13GB each for its two largest indices.
> Our entire Marmotta database takes up about 140GB. We've had some successes
> with improving index performance with low cardinality in `triples.context`
> [4] and tuning the Amazon EC2 instances that we run on [5][6]. The I/O wait
> problem with concurrent LDP operations, however, is the new blocker.
>
> Some supplemental information:
>
> * An overview of the project for which Marmotta is being
> used:
> https://digitalpubliclibraryofamerica.atlassian.net/wiki/display/TECH/Heidrun
>
> * The application (a Rails engine) that makes all of these LDP requests:
> https://github.com/dpla/KriKri
>
> * Our configuration-management project, with details on how some of our
> stack is configured: https://github.com/dpla/automation
>
> We'd be grateful for any feedback that you might have that would assist us
> with handling large volumes of data over LDP. Thanks for your help!
>
> - Mark Breedlove and Tom Johnson,
> Digital Public Library of America (http://dp.la/)
> tech@dp.la
>
>
> [1]
> https://digitalpubliclibraryofamerica.atlassian.net/wiki/display/TECH/LDP+Interactions+Overview
> [2]
> https://digitalpubliclibraryofamerica.atlassian.net/wiki/display/TECH/Marmotta
> [3]
> https://digitalpubliclibraryofamerica.atlassian.net/wiki/display/TECH/Addressing+slow+updates+and+inserts
> [4]
> https://digitalpubliclibraryofamerica.atlassian.net/wiki/display/TECH/Index+performance+with+high+context+counts
> [5]
> https://digitalpubliclibraryofamerica.atlassian.net/wiki/display/TECH/Amazon+EC2+adjustments
> [6]
> https://digitalpubliclibraryofamerica.atlassian.net/wiki/display/TECH/Using+irqbalance+and+SMP+IRQ+affinity
>