RYA ingest performance question

[Adam Sotona <ad...@gmail.com>]

Hi,
I've installed Hadoop Hortonworks 2.3 distribution on a small AWS cluster(3 data nodes), enabled Accumulo, compiled RYA from develop branch, built a small custom Java application performing a simple ingest of 10M triples of the Berlin Benchmark dataset (by simple adding the NT file stream to the RepositoryConnection).

And now I'm running into troubles where the ingest seems to be very slow (just about 10 triples per second), resulting into estimation of 11 days just to ingest it. 

Is there any trick on the Accumulo side or RYA side or should I just use different Java code to ingest the data?

Thank you for your help,
Adam Sotona

Sent from my BlackBerry 10 smartphone.

Re: RYA ingest performance question

[so...@seznam.cz]

Hi,
After short profiling I found the bottleneck in default setting to flush each added triple individually.
Now the performance seems to be closer to my expectations.
However any performance hints are still welcome.

Thanks,
Adam Sotona


Sent from my BlackBerry 10 smartphone.
  Original Message  
From: Adam Sotona
Sent: středa, 16. března 2016 10:06
To: dev@rya.incubator.apache.org
Reply To: dev@rya.incubator.apache.org
Subject: RYA ingest performance question

Hi,
I've installed Hadoop Hortonworks 2.3 distribution on a small AWS cluster(3 data nodes), enabled Accumulo, compiled RYA from develop branch, built a small custom Java application performing a simple ingest of 10M triples of the Berlin Benchmark dataset (by simple adding the NT file stream to the RepositoryConnection).

And now I'm running into troubles where the ingest seems to be very slow (just about 10 triples per second), resulting into estimation of 11 days just to ingest it. 

Is there any trick on the Accumulo side or RYA side or should I just use different Java code to ingest the data?

Thank you for your help,
Adam Sotona

Sent from my BlackBerry 10 smartphone.