You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@stormcrawler.apache.org by "rzo1 (via GitHub)" <gi...@apache.org> on 2026/08/29 11:28:11 UTC
[I] WARC digest algorithm is hard coded to SHA-1 with no way to choose another [stormcrawler]
rzo1 opened a new issue, #2106:
URL: https://github.com/apache/stormcrawler/issues/2106
## What happens
`WARCRecordFormat.getDigestSha1()` computes a SHA-1 over the bytes and returns it as `sha1:<base32>`. That value is used for `WARC-Payload-Digest` and `WARC-Block-Digest` in response, request, resource and metadata records. There is no configuration key for the algorithm, so an operator who wants SHA-256 digests in their archives, which WARC 1.1 allows, cannot have them.
## Where
`external/warc/src/main/java/org/apache/stormcrawler/warc/WARCRecordFormat.java:105-112`, used at `:402`, `:404`, `:409`, and reused by `MetadataRecordFormat.java:106` and `WARCRequestRecordFormat.java:85`.
```java
public static String getDigestSha1(byte[] bytes) {
return "sha1:" + base32.encodeAsString(DigestUtils.sha1(bytes));
}
```
## Why it matters
`sha1:` base32 is the convention across the WARC ecosystem, and the field identifies content for deduplication rather than authenticating it, so this is not urgent. It is still a fixed choice in a place where the bytes being digested come from the network, and operators whose own policy rules out SHA-1, or whose downstream tooling wants SHA-256, have no option other than rewriting the records. Making the algorithm a setting costs little and removes the argument.
## Reproduction
Save as `external/warc/src/test/java/org/apache/stormcrawler/warc/WARCDigestAlgorithmTest.java`.
```java
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to you under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.stormcrawler.warc;
import static org.junit.jupiter.api.Assertions.assertTrue;
import static org.mockito.Mockito.mock;
import static org.mockito.Mockito.when;
import java.nio.charset.StandardCharsets;
import java.util.List;
import org.apache.storm.tuple.Tuple;
import org.apache.stormcrawler.Metadata;
import org.junit.jupiter.api.Test;
/**
* Documents the current behaviour: the digest algorithm is hard coded to SHA-1 in
* WARCRecordFormat.getDigestSha1 and there is no configuration key to pick another one. Once the
* algorithm is configurable, this test should be replaced by one that sets the key and asserts the
* prefix changes.
*/
class WARCDigestAlgorithmTest {
@Test
void digestPrefixIsAlwaysSha1() {
String digest = WARCRecordFormat.getDigestSha1("some crawled bytes".getBytes(StandardCharsets.UTF_8));
System.out.println("digest: " + digest);
assertTrue(digest.startsWith("sha1:"), "digest algorithm is fixed to sha1");
}
@Test
void metadataRecordBlockDigestIsAlwaysSha1() {
Metadata metadata = new Metadata();
metadata.addValue("source", "a source");
Tuple tuple = mock(Tuple.class);
when(tuple.getStringByField("url")).thenReturn("https://www.example.org/");
when(tuple.getValueByField("metadata")).thenReturn(metadata);
MetadataRecordFormat format = new MetadataRecordFormat(List.of("source"));
String warcString = new String(format.format(tuple), StandardCharsets.UTF_8);
System.out.println(warcString);
assertTrue(
warcString.contains("WARC-Block-Digest: sha1:"),
"no configuration can change the algorithm used here");
}
}
```
Run it:
```
mvn -pl external/warc test -Dtest=WARCDigestAlgorithmTest
```
It passes on main and pins the current behaviour, since there is no setting to assert against yet. Replace it after the fix with one that sets the key and checks the prefix changes.
```
digest: sha1:B7VTSBZ5WRVHOA4OQPESX36RFAJF2URB
...
Content-Type: application/warc-fields
WARC-Block-Digest: sha1:2O53BWA6NLGDTA4665VJAM6IJBA4C5DK
source: a source
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0
```
## Suggested fix
Add a configuration key, for example `warc.digest.algorithm`, accepting `sha1` and `sha256`, and thread it through `WARCRecordFormat` so `MetadataRecordFormat` and `WARCRequestRecordFormat` pick it up too. Replace the static `getDigestSha1` helpers with an instance method that emits the matching `sha1:` or `sha256:` prefix, keeping the static ones for compatibility if anything outside the module calls them. Keep `sha1` as the default: CDX indexes and revisit tooling downstream assume `sha1:` base32, and changing the default would break them. Decide the default separately from adding the option.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@stormcrawler.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] WARC digest algorithm is hard coded to SHA-1 with no way to choose another [stormcrawler]
Posted by "rzo1 (via GitHub)" <gi...@apache.org>.
rzo1 closed issue #2106: WARC digest algorithm is hard coded to SHA-1 with no way to choose another
URL: https://github.com/apache/stormcrawler/issues/2106
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@stormcrawler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] WARC digest algorithm is hard coded to SHA-1 with no way to choose another [stormcrawler]
Posted by "abhinav-phi (via GitHub)" <gi...@apache.org>.
abhinav-phi commented on issue #2106:
URL: https://github.com/apache/stormcrawler/issues/2106#issuecomment-5462628615
Hi! I'd like to take this up.
Plan: I'll make the WARC digest algorithm configurable (instead of hardcoding SHA-1), default to the current behavior for compatibility, document the new option, and add a test.
Could you please assign this to me? Thanks!
/assign
take
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@stormcrawler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] WARC digest algorithm is hard coded to SHA-1 with no way to choose another [stormcrawler]
Posted by "rzo1 (via GitHub)" <gi...@apache.org>.
rzo1 commented on issue #2106:
URL: https://github.com/apache/stormcrawler/issues/2106#issuecomment-5462673176
Please also check https://github.com/apache/stormcrawler/pull/2034 if this is duplicated there.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@stormcrawler.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org