You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@jena.apache.org by Matt Whitby <ma...@gmail.com> on 2021/07/23 17:32:18 UTC
Scalability
A little bit of a vague question, and perhaps a silly one.
How well does Jena scale? Would it tap out after a given number of triples?
Do people sometimes split very large datasets over different instances and
just query across the different servers?
Thanks all.
--
Matt
Southend. Essex, England
Guff follows....
Me: http://www.about.me/matt.whitby
Photography: http://www.whitbyphoto.com
Travels: http://www.whitbyadventures.com
Music: http://www.last.fm/user/MattWhitby
<http://www.last.fm/user/MattWhitby/%3C/a%3E>
Reading: https://www.goodreads.com/user_challenges/19398505
Development: https://www.hackerrank.com/matt_whitby
Re: Scalability
Posted by Paul Tyson <ph...@sbcglobal.net>.
On 7/23/21 12:32 PM, Matt Whitby wrote:
> A little bit of a vague question, and perhaps a silly one.
>
> How well does Jena scale? Would it tap out after a given number of triples?
There are way too many variables to give a simple answer. I curate a
dataset of 1 billion triples that is refreshed nightly, using a very old
version of jena. I assume the performance has not regressed in the past
6 years. The dataset supports a web application driven by fairly complex
sparql queries, in what might be called a rudimentary "linked data"
paradigm.
Regards,
--Paul
>
> Do people sometimes split very large datasets over different instances and
> just query across the different servers?
>
> Thanks all.
>
>
Re: Scalability
Posted by Matt Whitby <ma...@gmail.com>.
I wouldn't expect the graph to be very 'wide' on our dataset.
On Sat, 24 Jul 2021 at 11:52, Adrian Gschwend <ml...@netlabs.org> wrote:
> On 24.07.21 11:35, Matt Whitby wrote:
>
> > I did say it was somewhat of a 'length of a piece of string' type
> > question. Having just dealt with relational databases for 25 years I
> know
> > pretty much nothing about triplestores. Realistically I don't see us
> > storing over 250m triples and if you say the whole Wikidata dump works ok
> > (and that's sitting at 12,878,628,927) then I'm pretty sure we'll be
> okay.
>
> 250M is not a problem per se, it only depends on how "graphy" your
> queries will be. Standard stuff should be fine, for very complex joins
> Jena runs into troubles at some point.
>
> regards
>
> Adrian
>
--
Matt
Southend. Essex, England
Guff follows....
Me: http://www.about.me/matt.whitby
Photography: http://www.whitbyphoto.com
Travels: http://www.whitbyadventures.com
Music: http://www.last.fm/user/MattWhitby
<http://www.last.fm/user/MattWhitby/%3C/a%3E>
Reading: https://www.goodreads.com/user_challenges/19398505
Development: https://www.hackerrank.com/matt_whitby
Re: Scalability
Posted by Adrian Gschwend <ml...@netlabs.org>.
On 24.07.21 11:35, Matt Whitby wrote:
> I did say it was somewhat of a 'length of a piece of string' type
> question. Having just dealt with relational databases for 25 years I know
> pretty much nothing about triplestores. Realistically I don't see us
> storing over 250m triples and if you say the whole Wikidata dump works ok
> (and that's sitting at 12,878,628,927) then I'm pretty sure we'll be okay.
250M is not a problem per se, it only depends on how "graphy" your
queries will be. Standard stuff should be fine, for very complex joins
Jena runs into troubles at some point.
regards
Adrian
Re: Scalability
Posted by Matt Whitby <ma...@gmail.com>.
I did say it was somewhat of a 'length of a piece of string' type
question. Having just dealt with relational databases for 25 years I know
pretty much nothing about triplestores. Realistically I don't see us
storing over 250m triples and if you say the whole Wikidata dump works ok
(and that's sitting at 12,878,628,927) then I'm pretty sure we'll be okay.
;)
On Sat, 24 Jul 2021 at 09:33, Lorenz Buehmann <
buehmann@informatik.uni-leipzig.de> wrote:
> "Scale" in terms of what? I mean, people loaded full Wikidata dump [1],
> so that works for sure on a single (though powerful) machine and indeed
> takes some (lots of) time. Query response time does indeed depend on the
> query size and complexity as well as the data size. But it's impossible
> to give any numbers here. You could have a look at benchmarks for some
> estimates or even better, just try it and load your dataset.
>
> There is no dedicated cluster version of Jena, but some people managed
> to somehow make some extension (dubbed Mosaic) on top
>
>
> [1] http://wiki.bitplan.com/index.php/Get_your_own_copy_of_WikiData
>
> On 23.07.21 19:32, Matt Whitby wrote:
> > A little bit of a vague question, and perhaps a silly one.
> >
> > How well does Jena scale? Would it tap out after a given number of
> triples?
> >
> > Do people sometimes split very large datasets over different instances
> and
> > just query across the different servers?
> >
> > Thanks all.
> >
> >
>
--
Matt
Southend. Essex, England
Guff follows....
Me: http://www.about.me/matt.whitby
Photography: http://www.whitbyphoto.com
Travels: http://www.whitbyadventures.com
Music: http://www.last.fm/user/MattWhitby
<http://www.last.fm/user/MattWhitby/%3C/a%3E>
Reading: https://www.goodreads.com/user_challenges/19398505
Development: https://www.hackerrank.com/matt_whitby
Re: Scalability
Posted by Lorenz Buehmann <bu...@informatik.uni-leipzig.de>.
"Scale" in terms of what? I mean, people loaded full Wikidata dump [1],
so that works for sure on a single (though powerful) machine and indeed
takes some (lots of) time. Query response time does indeed depend on the
query size and complexity as well as the data size. But it's impossible
to give any numbers here. You could have a look at benchmarks for some
estimates or even better, just try it and load your dataset.
There is no dedicated cluster version of Jena, but some people managed
to somehow make some extension (dubbed Mosaic) on top
[1] http://wiki.bitplan.com/index.php/Get_your_own_copy_of_WikiData
On 23.07.21 19:32, Matt Whitby wrote:
> A little bit of a vague question, and perhaps a silly one.
>
> How well does Jena scale? Would it tap out after a given number of triples?
>
> Do people sometimes split very large datasets over different instances and
> just query across the different servers?
>
> Thanks all.
>
>