You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@jena.apache.org by "rvesse (via GitHub)" <gi...@apache.org> on 2024/04/08 14:43:57 UTC
[I] RDFParserBuilder.toDatasetGraph() is much slower than calling RDFParserBuilder.parse(DatasetGraph) [jena]
rvesse opened a new issue, #2400:
URL: https://github.com/apache/jena/issues/2400
### Version
5.0.0
### What happened?
Was refactoring some code that uses `RDFParserBuilder` and noticed it was running much slower than previously, on comparing the old vs new code noticed the difference was I'd starting using `toDatasetGraph()` instead of creating my own `DatasetGraph` and then calling `parse()`
For example consider the following test cases:
```java
@Test
public void parse_to_dataset_01() {
RDFParserBuilder builder = RDFParserBuilder.create().lang(Lang.NQUADS).fromString(testLarge);
DatasetGraph dsg = builder.toDatasetGraph();
assertEquals(dsg.stream().count(), LARGE_SIZE);
}
@Test
public void parse_to_dataset_02() {
RDFParserBuilder builder = RDFParserBuilder.create().lang(Lang.NQUADS).fromString(testLarge);
DatasetGraph dsg = DatasetGraphFactory.create();
builder.parse(dsg);
assertEquals(dsg.stream().count(), LARGE_SIZE);
}
@Test
public void parse_to_dataset_03() {
RDFParserBuilder builder = RDFParserBuilder.create().lang(Lang.NQUADS).fromString(testLarge);
DatasetGraph dsg = DatasetGraphFactory.createTxnMem();
dsg.executeWrite(() -> builder.parse(dsg));
assertEquals(dsg.stream().count(), LARGE_SIZE);
}
```
Where `testLarge` is simply a generated string containing a sufficient number of quads to illustrate the performance difference (10,000 quads is a good minimum, 100k shows more obvious difference)
With 100k quads the first test takes ~1s, the 2nd ~200ms and the 3rd ~500ms
As the tests demonstrate the difference seems to be down to the use of transactions. Calling `toDatasetGraph()` creates a transactional dataset but then doesn't use transactions on it meaning all the writes from parsing are treated as individual commits AFAICT. Adding an explicit transaction as in the 3rd test yields a 2x speed up.
In my code the use of a transactional dataset was actually unnecessary so calling `parse()` with my desired `DatasetGraph` implementation was the solution.
Possible solutions:
1. Have Jena use the fresh transactional dataset in a write transaction inside `RDFParser.toDatasetGraph()` to reduce the performance hit
2. Have `StreamRDFLib.dataset()` automatically start and stop a transaction in
3. Document the `toDatasetGraph()` behaviour and its potential pitfalls more clearly
2 is my preferred solution as it offers improved parsing performance without impacting potential other uses of the more general `StreamRDFLib.dataset()` API
### Relevant output and stacktrace
_No response_
### Are you interested in making a pull request?
Yes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@jena.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@jena.apache.org
For additional commands, e-mail: issues-help@jena.apache.org
Re: [I] RDFParserBuilder.toDatasetGraph() is much slower than calling RDFParserBuilder.parse(DatasetGraph) [jena]
Posted by "afs (via GitHub)" <gi...@apache.org>.
afs commented on issue #2400:
URL: https://github.com/apache/jena/issues/2400#issuecomment-2043375384
JIT costs?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@jena.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@jena.apache.org
For additional commands, e-mail: issues-help@jena.apache.org
Re: [I] RDFParserBuilder.toDatasetGraph() is much slower than calling RDFParserBuilder.parse(DatasetGraph) [jena]
Posted by "rvesse (via GitHub)" <gi...@apache.org>.
rvesse closed issue #2400: RDFParserBuilder.toDatasetGraph() is much slower than calling RDFParserBuilder.parse(DatasetGraph)
URL: https://github.com/apache/jena/issues/2400
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@jena.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@jena.apache.org
For additional commands, e-mail: issues-help@jena.apache.org
Re: [I] RDFParserBuilder.toDatasetGraph() is much slower than calling RDFParserBuilder.parse(DatasetGraph) [jena]
Posted by "afs (via GitHub)" <gi...@apache.org>.
afs commented on issue #2400:
URL: https://github.com/apache/jena/issues/2400#issuecomment-2045361221
This is what I get:
`t` is with transaction, otherwise without.
Same data text creation code.
One round of 5 each: (dataset created outside of the timing)
```
1: 0.677s
2: 0.463s
3: 0.372s
4: 0.442s
5: 0.335s
t1: 0.255s
t2: 0.249s
t3: 0.328s
t4: 0.302s
t5: 0.237s
```
reversed:
```
t1: 0.587s
t2: 0.331s
t3: 0.356s
t4: 0.258s
t5: 0.353s
1: 0.427s
2: 0.322s
3: 0.321s
4: 0.327s
5: 0.335s
```
and after a few more rounds, `5: 0.315s` and `t5: 0.237s`. So some benefit from a transaction and no harm but specific times by contract
TIM datasets are general and robust (they provide MR+SW and are always thread safe with an "autocommit".
If the app really wants speed, use `DatasetGraphFactory.create` as in `parse-to-dataset_02`-- `5: 0.083s` and `t5: 0.083s`. The transaction support is a lock and, being uncontended, it is cheap.
Speed up comes from class loading happening once and JIT-optimizing the parser
```java
public static void run(String label, Creator<DatasetGraph> maker, String text) {
DatasetGraph dsg = maker.create();
long millis = Timer.time(()->{
RDFParserBuilder builder = RDFParserBuilder.create().lang(Lang.NQUADS).fromString(text);
builder.parse(dsg);
});
System.out.println(label+ ": "+Timer.timeStr(millis)+"s");
}
public static void runTxn(String label, Creator<DatasetGraph> maker, String text) {
DatasetGraph dsg = maker.create();
long millis = Timer.time(()->{
dsg.executeWrite(()->{
RDFParserBuilder builder = RDFParserBuilder.create().lang(Lang.NQUADS).fromString(text);
builder.parse(dsg);
});
});
System.out.println(label+": "+Timer.timeStr(millis)+"s");
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@jena.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@jena.apache.org
For additional commands, e-mail: issues-help@jena.apache.org
Re: [I] RDFParserBuilder.toDatasetGraph() is much slower than calling RDFParserBuilder.parse(DatasetGraph) [jena]
Posted by "rvesse (via GitHub)" <gi...@apache.org>.
rvesse commented on issue #2400:
URL: https://github.com/apache/jena/issues/2400#issuecomment-2044559519
> JIT costs?
Possibly, this might be a good area to write some JMH benchmarks around
The difference in performance is fairly minimal (though noticeable) in these test cases but I first found this issue because I refactored some $dayjob code and performance went from 15ms to 15 seconds so in worst case it can be dramatic difference
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@jena.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@jena.apache.org
For additional commands, e-mail: issues-help@jena.apache.org