You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@stormcrawler.apache.org by "dpol1 (via GitHub)" <gi...@apache.org> on 2026/06/04 08:18:05 UTC

[PR] Fix thread safety, resource leaks, and error handling across core and external modules [stormcrawler]

dpol1 opened a new pull request, #1928:
URL: https://github.com/apache/stormcrawler/pull/1928

   ## Summary  
     
   This PR addresses a set of code quality, correctness, and robustness issues found across the `core` and `external` modules.  
     
   ### Thread Safety  
   - Replace non-thread-safe `SimpleDateFormat` with `DateTimeFormatter` in `FileResponse` and `DateUtils.parseDate()` in `CookieConverter`  
   - Replace shared mutable `Matcher` in `RefreshTag` with a `Pattern` constant and per-call `Matcher` instances  
     
   ### Resource Leaks  
   - Use try-with-resources for `FileInputStream` in `FileResponse` and `InputStream` in `RegexURLFilterBase`  
   - Add `cleanup()` to `DebugParseFilter` to close the output stream  
   - Ensure Playwright tracing is always stopped via a `finally` block in `HttpProtocol`, even when an exception is thrown  
     
   ### Error Handling & Robustness  
   - Wrap `Long.parseLong()` / `Integer.parseInt()` in `FetcherBolt` with try-catch to handle invalid metadata values gracefully  
   - Add null check for classpath resource stream in `RegexURLFilterBase`  
   - Fix duplicate `setExpiryDate()` call and add null guard in `CookieConverter`  
   - Fix misleading error message in `CloudSearchUtils` ("must be score" → "must NOT be score")  
     
   ### Logging & Code Cleanup  
   - Replace all `e.printStackTrace()` calls with proper SLF4J `LOG.error()`  
   - Replace `URLEncoder.encode(url, "UTF-8")` (unnecessary checked exception) with the `StandardCharsets.UTF_8` overload  
   - Replace manual `MessageDigest` boilerplate with `DigestUtils.sha512Hex()`  
   - Use actual document charset in `JsRenderingDetector` instead of hardcoded UTF-8  
     
   ### No behavior changes intended  
   All changes are backward-compatible. The crawl logic and output remain the same.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@stormcrawler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Fix thread safety, resource leaks, and error handling across core and external modules [stormcrawler]

Posted by "dpol1 (via GitHub)" <gi...@apache.org>.
dpol1 commented on PR #1928:
URL: https://github.com/apache/stormcrawler/pull/1928#issuecomment-4620488642

   Sure, sorry about that — got a bit carried away. Will split it into three PRs: thread-safety, resource leaks, and fault tolerance/cleanup.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@stormcrawler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Fix thread safety, resource leaks, and error handling across core and external modules [stormcrawler]

Posted by "jnioche (via GitHub)" <gi...@apache.org>.
jnioche commented on PR #1928:
URL: https://github.com/apache/stormcrawler/pull/1928#issuecomment-4620447348

   Thanks @dpol1. Having a PR affecting loads of different files and covering such different aspects does not make it easy to review. Instead of a large dump, having one PR per type of issue (e.g. Resource Leaks) means that changes can be reviewed quicker, merged in isolation and also reviewed in parallel.
   Any chance you could break it down a bit?
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@stormcrawler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] Fix thread safety, resource leaks, and error handling across core and external modules [stormcrawler]

Posted by "dpol1 (via GitHub)" <gi...@apache.org>.
dpol1 closed pull request #1928: Fix thread safety, resource leaks, and error handling across core and external modules  
URL: https://github.com/apache/stormcrawler/pull/1928


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscribe@stormcrawler.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org