[jira] [Commented] (NUTCH-3209) Replace SHA-1 URL fingerprints in dump tools with SHA-256

["ASF GitHub Bot (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/NUTCH-3209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18109748#comment-18109748 ] 

ASF GitHub Bot commented on NUTCH-3209:
---------------------------------------

lewismc opened a new pull request, #964:
URL: https://github.com/apache/nutch/pull/964

   PR for [NUTCH-3209](https://issues.apache.org/jira/browse/NUTCH-3209)
   
   I addressed some minor formatting and Javadoc whilst in these files but made sure not to overdo it.
   
   New unit test `testCreateFileNameFromUrl`.
   
   Thanks for any review.




> Replace SHA-1 URL fingerprints in dump tools with SHA-256
> ---------------------------------------------------------
>
>                 Key: NUTCH-3209
>                 URL: https://issues.apache.org/jira/browse/NUTCH-3209
>             Project: Nutch
>          Issue Type: Improvement
>          Components: dumpers
>    Affects Versions: 1.23
>            Reporter: Lewis John McGibbney
>            Assignee: Lewis John McGibbney
>            Priority: Minor
>             Fix For: 1.24
>
>
> SonarCloud flags two uses of DigestUtils.sha1Hex (java:S4790, "Weak hashing algorithms should not be used") on master:
> - org.apache.nutch.util.DumpFileUtil#createFileNameFromUrl
> - org.apache.nutch.tools.CommonCrawlDataDumper (reverse-key construction)
> Both hash a URL to name dump directories / Common Crawl reverse-key values. This is fingerprinting for filesystem and JSON layout, not password storage or cryptographic integrity of crawl data.
> Replace SHA-1 with SHA-256 (DigestUtils.sha256Hex), which satisfies S4790 for non-sensitive fingerprinting. commons-codec is already a dependency.
> Compatibility: dump path leaves and reverse-key strings will change (40-char hex -> 64-char hex). Re-running a dump into an existing output tree will not collocate files with older SHA-1 directories. CrawlDb signatures (MD5Signature / db.signature.class) are unchanged.
> Out of scope: Hadoop MD5Hash usage (DumpFileUtil.getUrlMD5, MD5Signature, TextMD5Signature, TextProfileSignature) is not reported by this Sonar rule and would be a separate, more compatibility-sensitive change.
> Test: extend DumpFileUtilTest to cover createFileNameFromUrl.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)