You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@stormcrawler.apache.org by "rzo1 (via GitHub)" <gi...@apache.org> on 2026/09/18 19:51:18 UTC
[PR] #2091 Protocols: reject metadata keys arriving in a response header [stormcrawler]
rzo1 opened a new pull request, #2171:
URL: https://github.com/apache/stormcrawler/pull/2171
The protocols copy every response header into the response metadata, and a header name is not restricted to anything, so a fetched server could write the keys the crawler uses to record its own fetch. `WARCRecordFormat` promotes a record from `resource` to `response` whenever `_response.headers_` is non-blank and embeds the block verbatim, and `WARCRequestRecordFormat` writes `_request.headers_` as what the crawler supposedly sent — so the archive could be made to contain a request and response of the server's choosing, with its own `WARC-IP-Address` and `WARC-Date`. Values under `metrics.*` reached the fetcher metric registry the same way, which is the other half of #2099.
`ProtocolResponse.isReservedMetadataKey` names the keys the crawler writes itself, and the okhttp and Playwright protocols skip a header which matches.
Two details worth a look in review:
- In okhttp the keys `HTTPHeadersInterceptor` passes through the response headers stay readable, but only while that interceptor is installed, which is what `http.store.headers` does. Without that condition the change would break `http.store.headers: true`.
- The interceptor sets the cipher suite only when there is a handshake, so over `http://` a value from the server survived even with headers stored. It now drops that header in the else branch.
This also removes an unguarded base64 decode: a value under `_request.headers_` which was not valid base64 threw `IllegalArgumentException` out of `getProtocolOutput` and failed the fetch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@stormcrawler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] #2091 Protocols: reject metadata keys arriving in a response header [stormcrawler]
Posted by "rzo1 (via GitHub)" <gi...@apache.org>.
rzo1 merged PR #2171:
URL: https://github.com/apache/stormcrawler/pull/2171
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@stormcrawler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] #2091 Protocols: reject metadata keys arriving in a response header [stormcrawler]
Posted by "rzo1 (via GitHub)" <gi...@apache.org>.
rzo1 commented on code in PR #2171:
URL: https://github.com/apache/stormcrawler/pull/2171#discussion_r4053779130
##########
core/src/main/java/org/apache/stormcrawler/protocol/ProtocolResponse.java:
##########
@@ -78,6 +82,36 @@ public class ProtocolResponse {
*/
public static final String PROTOCOL_MD_PREFIX_PARAM = "protocol.md.prefix";
+ /** Prefix of the metadata keys holding the timings a protocol reports about a fetch. */
+ public static final String METRICS_PREFIX = "metrics.";
+
+ /** Lowercased: metadata keys are normalised that way, and so are header names. */
+ private static final Set<String> RESERVED_METADATA_KEYS =
+ Stream.of(
+ REQUEST_HEADERS_KEY,
+ RESPONSE_HEADERS_KEY,
+ RESPONSE_IP_KEY,
+ REQUEST_TIME_KEY,
+ PROTOCOL_VERSIONS_KEY,
+ CIPHER_SUITE_KEY,
+ TRIMMED_RESPONSE_KEY,
+ TRIMMED_RESPONSE_REASON_KEY,
+ REDIRECTED_TO_KEY)
Review Comment:
Yes, reserved `playwright.protocol.start` and `playwright.protocol.end` as well, with a test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@stormcrawler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] #2091 Protocols: reject metadata keys arriving in a response header [stormcrawler]
Posted by "dpol1 (via GitHub)" <gi...@apache.org>.
dpol1 commented on code in PR #2171:
URL: https://github.com/apache/stormcrawler/pull/2171#discussion_r4052759323
##########
core/src/main/java/org/apache/stormcrawler/protocol/ProtocolResponse.java:
##########
@@ -78,6 +82,36 @@ public class ProtocolResponse {
*/
public static final String PROTOCOL_MD_PREFIX_PARAM = "protocol.md.prefix";
+ /** Prefix of the metadata keys holding the timings a protocol reports about a fetch. */
+ public static final String METRICS_PREFIX = "metrics.";
+
+ /** Lowercased: metadata keys are normalised that way, and so are header names. */
+ private static final Set<String> RESERVED_METADATA_KEYS =
+ Stream.of(
+ REQUEST_HEADERS_KEY,
+ RESPONSE_HEADERS_KEY,
+ RESPONSE_IP_KEY,
+ REQUEST_TIME_KEY,
+ PROTOCOL_VERSIONS_KEY,
+ CIPHER_SUITE_KEY,
+ TRIMMED_RESPONSE_KEY,
+ TRIMMED_RESPONSE_REASON_KEY,
+ REDIRECTED_TO_KEY)
Review Comment:
`playwright.protocol.end` is another key a protocol writes and the js rendering guards read through `skipIfMetadataPresent`: with an empty `protocol.md.prefix`, or the key configured with it, a server can set it through okhttp. worth reserving too?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@stormcrawler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org