You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@jena.apache.org by "mpagni12 (via GitHub)" <gi...@apache.org> on 2025/03/04 14:04:35 UTC
[I] timeout parameter is not always respected by fuseki server [jena]
mpagni12 opened a new issue, #3044:
URL: https://github.com/apache/jena/issues/3044
### Version
5.3.0
### What happened?
Using the Python _requests_ client, I am running sequentially a list of diverse SPARQL queries on a fuseki endpoint at localhost:3030, using POST request and a timeout parameter set to 10 s. For the few first lengthy query in the list, the timeout works as expected, i.e. the request terminates after approx 10 s and with a 503 status code. However at some point the timeout of subsequent queries is ignored, i.e. the queries executing correctly and completely.
As I suspect that this problem might be linked to the fuseki HTTP server, I have not investigated the content of the queries that are not timing out. Should I?
Thanks in advance,
Marco
### Relevant output and stacktrace
```shell
```
### Are you interested in making a pull request?
None
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@jena.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@jena.apache.org
For additional commands, e-mail: issues-help@jena.apache.org
Re: [I] timeout parameter is not always respected by fuseki server [jena]
Posted by "afs (via GitHub)" <gi...@apache.org>.
afs closed issue #3044: timeout parameter is not always respected by fuseki server
URL: https://github.com/apache/jena/issues/3044
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@jena.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@jena.apache.org
For additional commands, e-mail: issues-help@jena.apache.org
Re: [I] timeout parameter is not always respected by fuseki server [jena]
Posted by "Aklakan (via GitHub)" <gi...@apache.org>.
Aklakan commented on issue #3044:
URL: https://github.com/apache/jena/issues/3044#issuecomment-2702065894
Perhaps the reason is not the sequence path, but a join that scans all of the data without producing results. So this pattern should be unrelated to the locking issue.
It hangs with this stack trace:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@jena.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@jena.apache.org
For additional commands, e-mail: issues-help@jena.apache.org
Re: [I] timeout parameter is not always respected by fuseki server [jena]
Posted by "afs (via GitHub)" <gi...@apache.org>.
afs commented on issue #3044:
URL: https://github.com/apache/jena/issues/3044#issuecomment-2701436941
I looked for `timeout=` and matched queries to their response code: e.g. `[34] 503 Service Unavailable (10.879 s)` or `[26] 200 OK (7.066 s)`
I didn't find any 200's over 10 seconds.
I didn't find any 503's under 10 seconds.
I did find one outlier `[20] 503 Service Unavailable (377.566 s)`. It is like the query from #3021 without the suggested workaround but has extra work added. The `OPTIONAL` look like they have a partial cross-product in them.
The timeout is tested every so often - it isn't an interrupt - so the query is in a busy CPU loop (probably fixable - caveat polling the timeout too much is a bit expensive).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@jena.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@jena.apache.org
For additional commands, e-mail: issues-help@jena.apache.org
Re: [I] timeout parameter is not always respected by fuseki server [jena]
Posted by "afs (via GitHub)" <gi...@apache.org>.
afs commented on issue #3044:
URL: https://github.com/apache/jena/issues/3044#issuecomment-2702017658
A sequence path should be expanded to triple patterns.
```sparql
{ ?x rdfs:label / rdfs:label ?z }
```
```sparql
{
?x rdfs:label ??P1 .
??P1 rdfs:label ?z .
}
```
Looking at the log, the timeout isn't getting forgotten - it is delayed (presumably by being busy in some tight loop).
The execution plan for the `[20]` does not have a hash join in it. It's `conditional` and `sequence` but I think the OPTIONALs can massive fan-out and should be hash left joins.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@jena.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@jena.apache.org
For additional commands, e-mail: issues-help@jena.apache.org
Re: [I] timeout parameter is not always respected by fuseki server [jena]
Posted by "mpagni12 (via GitHub)" <gi...@apache.org>.
mpagni12 commented on issue #3044:
URL: https://github.com/apache/jena/issues/3044#issuecomment-2701364083
I rebuild a database and rerun the same test using 5.0.0: all lengthy queries timeout after 10 s :-)
This strongly suggests that the bug was introduce later.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@jena.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@jena.apache.org
For additional commands, e-mail: issues-help@jena.apache.org
Re: [I] timeout parameter is not always respected by fuseki server [jena]
Posted by "osma (via GitHub)" <gi...@apache.org>.
osma commented on issue #3044:
URL: https://github.com/apache/jena/issues/3044#issuecomment-2720484943
I've been investigating a similar problem - probably the same one - where Fuseki 5.3.0 seems to ignore the timeout on certain inefficient queries. In the worst case that I saw, a single query was allowed to run for hours, even with the timeout set to 1 second (1000 ms). If you run a public or semi-public SPARQL endpoint, such queries are almost bound to happen eventually, and it leads to a DoS-style situation especially if such long-running queries start piling up. Each of them will fully utilize a single CPU core.
I only noticed this issue now, since it wasn't created yet when I started my investigation. I'm adding some details and information about my case in the hope that it's helpful, even though the main issue seems already known and a potential fix is available in PR #3047. (I will test the fix as well and provide comments)
I'm using a TDB2 dataset `tdb2-yso` where the file [yso-skos.ttl](https://github.com/NatLibFi/Finto-data/blob/master/vocabularies/yso/yso-skos.ttl) has been loaded into the graph `http://www.yso.fi/onto/yso/`
Here is a relatively simple (but stupid, I know) query that triggers the problem:
```
PREFIX skos: <http://www.w3.org/2004/02/skos/core#>
SELECT *
WHERE {
GRAPH <http://www.yso.fi/onto/yso/> {
?uri ?p ?o .
OPTIONAL {
?x skos:member ?o .
FILTER NOT EXISTS {
?x skos:member ?other .
FILTER NOT EXISTS {
?other skos:broader ?uri
}
}
}
} VALUES (?uri) {
(<http://www.yso.fi/onto/yso/p24009>)
}
}
```
Here is my Fuseki configuration that sets a timeout of 2 seconds for the service (just to demonstrate that it won't help either) and 1 second for the dataset:
```
@prefix : <http://base/#> .
@prefix tdb2: <http://jena.apache.org/2016/tdb#> .
@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .
@prefix ja: <http://jena.hpl.hp.com/2005/11/Assembler#> .
@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> .
@prefix fuseki: <http://jena.apache.org/fuseki#> .
[] rdf:type fuseki:Server ;
ja:context [ ja:cxtName "arq:queryTimeout" ; ja:cxtValue "2000" ] .
:tdb2 a fuseki:Service ;
rdfs:label "TDB2" ;
fuseki:dataset :dataset ;
fuseki:name "ds" ;
fuseki:serviceQuery "query" , "sparql" ;
fuseki:serviceReadGraphStore "get" ;
fuseki:serviceReadWriteGraphStore "data" ;
fuseki:serviceUpdate "update" ;
fuseki:serviceUpload "upload" .
:dataset
a tdb2:DatasetTDB2 ;
ja:context [ ja:cxtName "arq:queryTimeout" ; ja:cxtValue "1000" ] ;
tdb2:location "tdb2-yso" .
```
Here is what happens with Fuseki 5.3.0 (and the current SNAPSHOT from `main`) when I run the query:
```
10:53:06 INFO Server :: Apache Jena Fuseki 5.3.0
10:53:06 INFO Config :: Fuseki Base = /home/oisuomin/proj/test-fuseki5-timeout/run
10:53:06 INFO Config :: Database: /ds
10:53:06 INFO Config :: UI Base = fuseki-server.jar
10:53:06 INFO Server :: Configuration file: fuseki-config.ttl
10:53:06 INFO Server :: Path = /ds
10:53:06 INFO Server :: Memory: 4.0 GiB
10:53:06 INFO Server :: Java: 21.0.6
10:53:06 INFO Server :: OS: Linux 6.11.0-19-generic amd64
10:53:06 INFO Server :: PID: 480591
10:53:06 INFO Shiro :: Shiro configuration: file:/home/oisuomin/proj/test-fuseki5-timeout/run/shiro.ini
10:53:06 INFO Server :: Start Fuseki (http=3030)
10:53:11 INFO Fuseki :: [3] POST http://localhost:3030/ds/
10:53:11 INFO Fuseki :: [3] Query = PREFIX skos: <http://www.w3.org/2004/02/skos/core#> SELECT * WHERE { GRAPH <http://www.yso.fi/onto/yso/> { ?uri ?p ?o . OPTIONAL { ?x skos:member ?o . FILTER NOT EXISTS { ?x skos:member ?other . FILTER NOT EXISTS { ?other skos:broader ?uri } } } } VALUES (?uri) { (<http://www.yso.fi/onto/yso/p24009>) } }
10:53:22 INFO Fuseki :: [3] 503 Service Unavailable (10.769 s)
```
So the query runs for 10+ seconds before being cancelled. The 1 second timeout doesn't cut it off at 1 second as it should.
I tested earlier Jena Fuseki releases as well. 5.2.0, 5.1.0 and 5.0.0 all had the same problem. The last release where the timeout setting still worked was 4.10.0:
```
10:52:00 INFO Server :: Apache Jena Fuseki 4.10.0
10:52:00 INFO Config :: FUSEKI_HOME=/home/oisuomin/sw/apache-jena-fuseki-4.10.0
10:52:00 INFO Config :: FUSEKI_BASE=/home/oisuomin/proj/test-fuseki5-timeout/run
10:52:00 INFO Config :: Shiro file: file:///home/oisuomin/proj/test-fuseki5-timeout/run/shiro.ini
10:52:01 INFO Server :: Path = /ds
10:52:01 INFO Server :: Memory: 4.0 GiB
10:52:01 INFO Server :: Java: 21.0.6
10:52:01 INFO Server :: OS: Linux 6.11.0-19-generic amd64
10:52:01 INFO Server :: PID: 480212
10:52:01 INFO Server :: Started 2025/03/13 10:52:01 EET on port 3030
10:52:03 INFO Fuseki :: [1] POST http://localhost:3030/ds/
10:52:03 INFO Fuseki :: [1] Query = PREFIX skos: <http://www.w3.org/2004/02/skos/core#> SELECT * WHERE { GRAPH <http://www.yso.fi/onto/yso/> { ?uri ?p ?o . OPTIONAL { ?x skos:member ?o . FILTER NOT EXISTS { ?x skos:member ?other . FILTER NOT EXISTS { ?other skos:broader ?uri } } } } VALUES (?uri) { (<http://www.yso.fi/onto/yso/p24009>) } }
10:52:04 INFO Fuseki :: [1] 503 Service Unavailable (1.020 s)
```
I'll provide comments on the PR separately in a moment (it seems to work).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@jena.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@jena.apache.org
For additional commands, e-mail: issues-help@jena.apache.org
Re: [I] timeout parameter is not always respected by fuseki server [jena]
Posted by "mpagni12 (via GitHub)" <gi...@apache.org>.
mpagni12 commented on issue #3044:
URL: https://github.com/apache/jena/issues/3044#issuecomment-2700829105
Here is the log: [fuseki.log.gz](https://github.com/user-attachments/files/19089167/fuseki.log.gz)
Storage is TDB with no inference.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@jena.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@jena.apache.org
For additional commands, e-mail: issues-help@jena.apache.org
Re: [I] timeout parameter is not always respected by fuseki server [jena]
Posted by "mpagni12 (via GitHub)" <gi...@apache.org>.
mpagni12 commented on issue #3044:
URL: https://github.com/apache/jena/issues/3044#issuecomment-2700830306
> I quite recently fixed several cases where timeouts were not working properly. Without your query load its hard to tell whether this is related to something I did not fix - or perhaps to something that I accidentally broke.
>
> Are update requests involved? Do the timeouts work properly for your query load with jena 5.0.0?
I am using 5.3.0, should I downgrade my version for testing?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@jena.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@jena.apache.org
For additional commands, e-mail: issues-help@jena.apache.org
Re: [I] timeout parameter is not always respected by fuseki server [jena]
Posted by "Aklakan (via GitHub)" <gi...@apache.org>.
Aklakan commented on issue #3044:
URL: https://github.com/apache/jena/issues/3044#issuecomment-2698777198
I quite recently fixed several cases where timeouts were not working properly. Without your query load its hard to tell whether this is related to something I did not fix - or perhaps something that I accidentally broke.
Are update requests involved?
Do the timeouts work properly for your query load with jena 5.0.0?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@jena.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@jena.apache.org
For additional commands, e-mail: issues-help@jena.apache.org
Re: [I] timeout parameter is not always respected by fuseki server [jena]
Posted by "Aklakan (via GitHub)" <gi...@apache.org>.
Aklakan commented on issue #3044:
URL: https://github.com/apache/jena/issues/3044#issuecomment-2700935993
> I am using 5.3.0, should I downgrade my version for testing?
That would be very helpful in order to determine whether the issue existed for a while or was introduced recently.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@jena.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@jena.apache.org
For additional commands, e-mail: issues-help@jena.apache.org
Re: [I] timeout parameter is not always respected by fuseki server [jena]
Posted by "Aklakan (via GitHub)" <gi...@apache.org>.
Aklakan commented on issue #3044:
URL: https://github.com/apache/jena/issues/3044#issuecomment-2702118002
The problem with eager iterators in the snippet below is, that if `getPlan().iterator()` fails, then there is no resource that can be closed -> likely resource leak.
https://github.com/apache/jena/blob/6a86b5d1869c575142391e3e8bcc8f00f1a75132/jena-arq/src/main/java/org/apache/jena/sparql/exec/QueryExecDataset.java#L472-L475
IMO the clean solution would be to change AbstractIterHashJoin to be lazy, then the lock would only be held shortly for the iterator construction, and then the timeout can kick in normally.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@jena.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@jena.apache.org
For additional commands, e-mail: issues-help@jena.apache.org
Re: [I] timeout parameter is not always respected by fuseki server [jena]
Posted by "afs (via GitHub)" <gi...@apache.org>.
afs commented on issue #3044:
URL: https://github.com/apache/jena/issues/3044#issuecomment-2698246619
@mpagni12 -- theer isn't enough to recreate the situation from the description.
Could you provide the fuseki server log file for the relevant first request (timeouts) and later (non-timeouts). If that can be "verbose" (server "`-v`") that would be best.
What is the storage? TDB2? Any inference?
> I have not investigated the content of the queries that are not timing out. Should I?
It does seem to need know what the queries are.
Note: if the server has just started, first requests are slower. Java starts optimizing code and the file cache becomes more active over time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@jena.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@jena.apache.org
For additional commands, e-mail: issues-help@jena.apache.org
Re: [I] timeout parameter is not always respected by fuseki server [jena]
Posted by "Aklakan (via GitHub)" <gi...@apache.org>.
Aklakan commented on issue #3044:
URL: https://github.com/apache/jena/issues/3044#issuecomment-2701831446
So I can reproduce timeouts being ignored on property paths, such as this one on a dbpedia dataset (lots of labels):
```sparql
PREFIX rdfs: <http://www.w3.org/2000/01/rdf-schema#>
SELECT ?x { ?x rdfs:label/rdfs:label ?z }
```
```bash
time curl http://localhost:3030/ds --data-urlencode 'query=PREFIX rdfs: <http://www.w3.org/2000/01/rdf-schema#> SELECT ?x { ?x rdfs:label/rdfs:label ?z }' --data-urlencode 'timeout=1'
```
I have not analyzed property paths further.
I can also reproduce a similar (perhaps the same) problem with this query:
```sparql
PREFIX rdfs: <http://www.w3.org/2000/01/rdf-schema#>
SELECT ?x {
?x rdfs:label ?o
{ ?x rdfs:label ?z }
UNION
{ BIND("hi" AS ?s) }
}
```
The problem is synchronization:
The timeout fires and tries to abort, but doesn't get the lock:
https://github.com/apache/jena/blob/6a86b5d1869c575142391e3e8bcc8f00f1a75132/jena-arq/src/main/java/org/apache/jena/sparql/exec/QueryExecDataset.java#L364-L367
The lock is acquired here:
https://github.com/apache/jena/blob/6a86b5d1869c575142391e3e8bcc8f00f1a75132/jena-arq/src/main/java/org/apache/jena/sparql/exec/QueryExecDataset.java#L438-L445
And while the lock is held `QueryExecDataset.startQueryIteratorActual` executes a hash join eagerly - so the timeout does not get through:
https://github.com/apache/jena/blob/6a86b5d1869c575142391e3e8bcc8f00f1a75132/jena-arq/src/main/java/org/apache/jena/sparql/exec/QueryExecDataset.java#L472-L475
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@jena.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@jena.apache.org
For additional commands, e-mail: issues-help@jena.apache.org