You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@jena.apache.org by Marco Franke <fm...@biba.uni-bremen.de> on 2021/01/25 07:02:33 UTC
[Apache Fuseki] Limits of Apache Fuseki Triple Store
Dear developers,
we use the Apache Fuseki Triple Store to store sensory data. Up to now, it works pretty well.
The challenge is that we have to import a lot of data in the upcoming month.
At the moment, we have stored already ca. 500 000 triples and everything is fine.
My question is whether there is a limitation of the amount of triples to be stored and queried?
My second question is how to download the data again.
I know there is the ..../get endpoint for each endpoint. Does this endpoint works for huge datasets, too?
Thanks in Advance,
Kind regards,
Marco Franke
[cid:image001.jpg@01D6F2F0.6E664FC0]
i.A. Dipl. -Inf. Marco Franke
Wissenschaftlicher Mitarbeiter in 2.2
BIBA - Bremer Institut für Produktion und Logistik GmbH
Informations- und kommunikationstechnische Anwendungen in der Produktion
Prof. Dr.-Ing. Klaus-Dieter Thoben
Raum 1490
Tel.: +49 (0)421 218-50089
Fax: +49 (0)421 218-50007
fma@biba.uni-bremen.de<ma...@biba.uni-bremen.de>
[In-2C-28px-R]<https://www.linkedin.com/company/biba.uni-bremen.de>[apple-touch-icon-180x180]<https://www.researchgate.net/institution/BIBA-Bremer_Institut_fuer_Produktion_und_Logistik> [FB-f-Logo__blue_29] <https://www.facebook.com/BIBA.Produktion.Logistik> [YouTube-social-icon_red_48px] <http://www.youtube.com/channel/UCieF5Uq5Qix9XZAZuYhTQhg>
BIBA - Bremer Institut für Produktion und Logistik GmbH
Postanschrift: Postfach P.O.B. 33 05 60 · 28335 Bremen / Germany
Geschäftssitz: Hochschulring 20 · 28359 Bremen / Germany
USt-ID: DE814890109 · Amtsgericht Bremen HRB 24505 HB
Tel: +49 (0)421/218-50000 · Fax: +49 (0)421/218-50031
E-Mail: info@biba.uni-bremen.de<ma...@biba.uni-bremen.de> · Internet: www.biba.uni-bremen.de<http://www.biba.uni-bremen.de/>
Geschäftsführer: Prof. Dr.-Ing. M. Freitag, Prof. Dr.-Ing. K.-D. Thoben, O. Simon
Re: [Apache Fuseki] Limits of Apache Fuseki Triple Store
Posted by Andy Seaborne <an...@apache.org>.
On 25/01/2021 07:02, Marco Franke wrote:
> Dear developers,
>
> we use the Apache Fuseki Triple Store to store sensory data. Up to now,
> it works pretty well.
>
> The challenge is that we have to import a lot of data in the upcoming month.
>
> At the moment, we have stored already ca. 500 000 triples and everything
> is fine.
In memory, 10's of millions compact and in-memory isn't something that
is specially optimized but the bounds are RAM not design. e.g. the
default heap size (-Xmx in the scripts) for Fuseki is quite low.
See Pauls' reply for when using TDB.
> My question is whether there is a limitation of the amount of triples to
> be stored and queried?
Internal design limitations are much higher.
> My second question is how to download the data again.
>
> I know there is the …./get endpoint for each endpoint. Does this
> endpoint works for huge datasets, too?
Yes - it streams and it is transaction so it reads consistent snapshot
and does not require all the data in memory at once.
See also the /$/backup operation which writes a quads file in the
background to the local disk (so it does not go over the network and
does not require the client to hang around).
Andy
>
> Thanks in Advance,
>
> Kind regards,
>
> Marco Franke
>
> __
>
> *i.A. Dipl. -Inf. Marco Franke***
>
> Wissenschaftlicher Mitarbeiter in 2.2
>
> *BIBA**- Bremer Institut für Produktion und Logistik GmbH*
>
> Informations- und kommunikationstechnische Anwendungen in der Produktion
>
> Prof. Dr.-Ing. Klaus-Dieter Thoben
>
> Raum 1490
> Tel.: +49 (0)421 218-50089
>
> Fax: +49 (0)421 218-50007
>
> fma@biba.uni-bremen.de <ma...@biba.uni-bremen.de>
>
> In-2C-28px-R
> <https://www.linkedin.com/company/biba.uni-bremen.de>apple-touch-icon-180x180
> <https://www.researchgate.net/institution/BIBA-Bremer_Institut_fuer_Produktion_und_Logistik>FB-f-Logo__blue_29
> <https://www.facebook.com/BIBA.Produktion.Logistik>YouTube-social-icon_red_48px
> <http://www.youtube.com/channel/UCieF5Uq5Qix9XZAZuYhTQhg>
>
> BIBA – Bremer Institut für Produktion und Logistik GmbH
>
> Postanschrift: Postfach P.O.B. 33 05 60 · 28335 Bremen / Germany
>
> Geschäftssitz: Hochschulring 20 · 28359 Bremen / Germany
>
> USt-ID: DE814890109 · Amtsgericht Bremen HRB 24505 HB
>
> Tel: +49 (0)421/218-50000 · Fax: +49 (0)421/218-50031
>
> E-Mail: info@biba.uni-bremen.de <ma...@biba.uni-bremen.de>
> · Internet: www.biba.uni-bremen.de <http://www.biba.uni-bremen.de/>
>
> Geschäftsführer: Prof. Dr.-Ing. M. Freitag, Prof. Dr.-Ing. K.-D. Thoben,
> O. Simon
>
Re: [Apache Fuseki] Limits of Apache Fuseki Triple Store
Posted by Paul Tyson <ph...@sbcglobal.net>.
Regarding first question: I maintain a dataset of 1 billion triples in a very old fuseki tdb version.
I would like to know if anyone’s working in trillion triple range , perhaps with RDF/HDT.
Regards,
—Paul
> On Jan 25, 2021, at 01:02, Marco Franke <fm...@biba.uni-bremen.de> wrote:
>
>
> Dear developers,
>
> we use the Apache Fuseki Triple Store to store sensory data. Up to now, it works pretty well.
>
> The challenge is that we have to import a lot of data in the upcoming month.
> At the moment, we have stored already ca. 500 000 triples and everything is fine.
>
> My question is whether there is a limitation of the amount of triples to be stored and queried?
>
> My second question is how to download the data again.
> I know there is the …./get endpoint for each endpoint. Does this endpoint works for huge datasets, too?
>
> Thanks in Advance,
>
> Kind regards,
>
> Marco Franke
>
>
>
> i.A. Dipl. -Inf. Marco Franke
> Wissenschaftlicher Mitarbeiter in 2.2
>
> BIBA - Bremer Institut für Produktion und Logistik GmbH
>
> Informations- und kommunikationstechnische Anwendungen in der Produktion
> Prof. Dr.-Ing. Klaus-Dieter Thoben
>
> Raum 1490
> Tel.: +49 (0)421 218-50089
> Fax: +49 (0)421 218-50007
> fma@biba.uni-bremen.de
>
>
>
> BIBA – Bremer Institut für Produktion und Logistik GmbH
> Postanschrift: Postfach P.O.B. 33 05 60 · 28335 Bremen / Germany
> Geschäftssitz: Hochschulring 20 · 28359 Bremen / Germany
> USt-ID: DE814890109 · Amtsgericht Bremen HRB 24505 HB
> Tel: +49 (0)421/218-50000 · Fax: +49 (0)421/218-50031
> E-Mail: info@biba.uni-bremen.de · Internet: www.biba.uni-bremen.de
> Geschäftsführer: Prof. Dr.-Ing. M. Freitag, Prof. Dr.-Ing. K.-D. Thoben, O. Simon
>
>