You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@jena.apache.org by GitBox <gi...@apache.org> on 2022/08/12 15:34:43 UTC
[GitHub] [jena] Aklakan opened a new issue, #1477: Improved AsyncParser API
Aklakan opened a new issue, #1477:
URL: https://github.com/apache/jena/issues/1477
### Version
4.6.0-SNAPSHOT
### Feature
Issues:
- Closing the iterator's returned by AsyncParser does not abort the parsing process. In fact, repeatedly abandoning iterators will cause parsing threads to silently pile up.
- AsyncParser's default chunk size of 100K tuples introduces a long delay unsuitable for content probing
- The EltStreamRDF is private. As mentioned in [https://issues.apache.org/jira/browse/JENA-2309] those events would useful in an hadoop/spark setting to scan for prefixes, thereby stopping the parser once only data is seen anymore.
I am about to create a PR with the following improvements:
- Changed AsyncParser API to return IteratorCloseables whose close() method actually cancels parsing.
- Added public E**v**tStreamRDF interface from parsing events with the existing private E**l**tStreamRDF as the internal data object. The naming is up for discussion :)
- Added a Builder that gives control over chunk and queue sizes: `AsyncParserNew.Builder.of(in, Lang.TRIG, null).setChunkSize(100).asyncParseQuads();`. The builder also has `asyncParseIterator` which returns `IteratorCloseable<EvtStreamRDF>`.
- If a parser fails then all remaining parsers are still started with a destination in 'interrupted state' in order for them to close their resources.
### Are you interested in contributing a solution yourself?
Yes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@jena.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@jena.apache.org
For additional commands, e-mail: issues-help@jena.apache.org
[GitHub] [jena] afs commented on issue #1477: Improved AsyncParser API
Posted by GitBox <gi...@apache.org>.
afs commented on issue #1477:
URL: https://github.com/apache/jena/issues/1477#issuecomment-1251383492
> > And it's not the only unstable test. There is a small set of tests that fail often in short bursts of a few days.
>
> Are you referring to test cases for the AsyncParser or other ones?
No - other tests - query timeout and also TDB related ones. A number of factors including random huge pauses and I/O or virtualised I/O effects.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@jena.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@jena.apache.org
For additional commands, e-mail: issues-help@jena.apache.org
[GitHub] [jena] Aklakan commented on issue #1477: Improved AsyncParser API
Posted by GitBox <gi...@apache.org>.
Aklakan commented on issue #1477:
URL: https://github.com/apache/jena/issues/1477#issuecomment-1250946972
It's a bug in the test; the configured limit does not consider the farthest possible read ahead until the queue blocks because of becoming full. The justification for 60K bytes is as follows:
- Chunk size is set to 100, one triple in the test is always exactly 50 bytes - so one chunk corresponds to 5000bytes.
- The queue size is 10 - so it can hold 50K, after that it blocks.
- However, we consume 10 triples which removes one chunk, which gets filled up again - at this point we consumed 55K.
- Then we abort, which clears the queue and schedules abort on the next chunk. When it arrives 60K have been consumed.
So the read ahead in the test is `(queueSize + 2) * (chunkSize * tripleByteSize)`. If the parser used buffering then a bit more could be read. But e.g. setting the limit in the test to e.g. 70.000 should fix it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@jena.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@jena.apache.org
For additional commands, e-mail: issues-help@jena.apache.org
[GitHub] [jena] afs closed issue #1477: Improved AsyncParser API
Posted by GitBox <gi...@apache.org>.
afs closed issue #1477: Improved AsyncParser API
URL: https://github.com/apache/jena/issues/1477
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@jena.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@jena.apache.org
For additional commands, e-mail: issues-help@jena.apache.org
[GitHub] [jena] afs commented on issue #1477: Improved AsyncParser API
Posted by GitBox <gi...@apache.org>.
afs commented on issue #1477:
URL: https://github.com/apache/jena/issues/1477#issuecomment-1250766495
I tried again at what is often a quieter time (2022/09/19, 08:00 UTC, 10:00 CET). Sometimes test fail because of the general load on the VM hosts causing extreme bursty-ness of processing, or the I/O patterns are different in these virtualized environments.
But.
MacOS still failing (different number of bytes read this time 20000 not 60000)
https://github.com/apache/jena/actions/runs/3080754379/jobs/4978478502
Windows: (still 60000)
https://github.com/apache/jena/actions/runs/3080780062/jobs/4978535429
If this test is testing for an expected, normal behaviour, but isn't guaranteed behaviour under all conditions (which would not be the first time Jena has encountered this), and it's impractical to make the test robust, then let's remove it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@jena.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@jena.apache.org
For additional commands, e-mail: issues-help@jena.apache.org
[GitHub] [jena] Aklakan commented on issue #1477: Improved AsyncParser API
Posted by GitBox <gi...@apache.org>.
Aklakan commented on issue #1477:
URL: https://github.com/apache/jena/issues/1477#issuecomment-1251523038
I updated the tests. I just started running the windows and mac tests on my github account and I am now waiting for the results.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@jena.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@jena.apache.org
For additional commands, e-mail: issues-help@jena.apache.org
[GitHub] [jena] afs closed issue #1477: Improved AsyncParser API
Posted by GitBox <gi...@apache.org>.
afs closed issue #1477: Improved AsyncParser API
URL: https://github.com/apache/jena/issues/1477
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@jena.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@jena.apache.org
For additional commands, e-mail: issues-help@jena.apache.org
[GitHub] [jena] afs commented on issue #1477: Improved AsyncParser API
Posted by GitBox <gi...@apache.org>.
afs commented on issue #1477:
URL: https://github.com/apache/jena/issues/1477#issuecomment-1251386807
> There are a couple of test cases that spawn many (100) threads and subsequently test that all have terminated. This might be a problem in github CI pipelines which AFAIK only use 2 cores - it would be possible to decrease the number e.g. to 20 or 10.
Yes - probably a good idea to reduce the number. Many threads should not be a problem though they may go away in a very patchy manner. But if the test is good when using 10-20, it makes the suite a little bit more general.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@jena.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@jena.apache.org
For additional commands, e-mail: issues-help@jena.apache.org
[GitHub] [jena] afs commented on issue #1477: Improved AsyncParser API
Posted by GitBox <gi...@apache.org>.
afs commented on issue #1477:
URL: https://github.com/apache/jena/issues/1477#issuecomment-1250376590
After merging PR #1478, 2 of the github build actions failed:
MacOS: https://github.com/apache/jena/actions/runs/3077784931
Windows: https://github.com/apache/jena/actions/runs/3077784687
```
[INFO] Running org.apache.jena.riot.system.TS_RiotSystem
Error: Tests run: 342, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 1.662 s <<< FAILURE! - in org.apache.jena.riot.system.TS_RiotSystem
Error: lowLatencyParse(org.apache.jena.riot.system.TestAsyncParser) Time elapsed: 0.25 s <<< FAILURE!
java.lang.AssertionError: Too many bytes consumed from input stream (60000)
at org.apache.jena.riot.system.TestAsyncParser.lowLatencyParse(TestAsyncParser.java:188)
```
The Linux build passed.
https://github.com/apache/jena/actions/runs/3077784184
The github actions are part of the code base and can be run on forked repos.
Building in these different environments (GH actions, ASF Jenkins) can be challenging. They are virtualized and extremely long pauses can occur (I've seen pauses of over 30 seconds due to the virtual machine simply not running for a while).
Can the test be made more robust?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@jena.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@jena.apache.org
For additional commands, e-mail: issues-help@jena.apache.org
[GitHub] [jena] afs commented on issue #1477: Improved AsyncParser API
Posted by GitBox <gi...@apache.org>.
afs commented on issue #1477:
URL: https://github.com/apache/jena/issues/1477#issuecomment-1250845301
Welcome to the demanding (hostile) world of CI servers 😀🔥
We can disable it if it starts to become a blockage. It's not ATM - Linux passes so it's not a code main error.
And it's not the only unstable test. There is a small set of tests that fail often in short bursts of a few days.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@jena.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@jena.apache.org
For additional commands, e-mail: issues-help@jena.apache.org
[GitHub] [jena] Aklakan commented on issue #1477: Improved AsyncParser API
Posted by GitBox <gi...@apache.org>.
Aklakan commented on issue #1477:
URL: https://github.com/apache/jena/issues/1477#issuecomment-1250810828
I will look into it today what goes wrong there; the test itself should check for excessive read ahead when abort is requested; maybe it fails because of different internal buffer sizes on different os but not sure yet.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@jena.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@jena.apache.org
For additional commands, e-mail: issues-help@jena.apache.org