[I] POST /records accepts invalid UTF-8 and stores altered text under the original digest [sourcelume-registry]

["ANierbeck (via GitHub) via issues" <is...@sourcelume.apache.org>]

ANierbeck opened a new issue, #32:
URL: https://github.com/apache/sourcelume-registry/issues/32

   ## Problem
   
   `POST /records` decodes the request body with `new String(received, StandardCharsets.UTF_8)` for the stored text (`IngestResource.java:167`) but computes the digest over the original bytes (`setSha256(sha256Hex(received))`, line 208). For valid UTF-8 the two agree. For a body that is not valid UTF-8, the decoder silently substitutes U+FFFD, so:
   
   - the stored text is no longer what was submitted,
   - the `sha256` still describes the submitted bytes,
   - the record is accepted (`VALIDATED` if the rest is fine).
   
   ## Why it matters
   
   The attestation thread settled on signing the stored bytes (detached signature). With #30 (`GET /records/{id}/document`) the registry returns the stored text with the `sha256` as `ETag`. For an input with invalid UTF-8 the returned document does not hash to the `ETag`, and a signature made over the posted bytes does not verify against it. Nothing in the status view or the response tells the consumer why.
   
   Not introduced by #30 — the POST path behaves this way on `main` today; the document endpoint is where it becomes visible.
   
   ## Reproduction
   
   Submit an otherwise valid record whose `name` contains the bytes `0xFF 0xFE`, then fetch it back (via `/document` from #30) and compare `shasum -a 256` of the posted file and the downloaded one. Checked against Atlas 2.5.0 (dev-support stack): POST answers `VALIDATED` with the digest of the posted bytes, the downloaded document hashes differently.
   
   Valid UTF-8 (umlauts, emoji, CRLF line endings) round-trips byte for byte; only invalid input is affected.
   
   ## Proposal
   
   Refuse such input at POST time: decode with a strict `CharsetDecoder` (`CodingErrorAction.REPORT`) and answer 400 when the body is not valid UTF-8. RFC 8259 requires UTF-8 for JSON exchanged between systems, and the registry already answers 400 for unparseable bodies. That makes "the stored text is the submission" an invariant.
   
   A test that posts an otherwise valid record with two invalid bytes should expect the 400; a BOM at the start of the body deserves a test case next to it (not checked yet).
   
   Alternatives: compute the digest over the bytes of the stored text (the `ETag` stays consistent, but no longer equals the digest of what was sent), or store the bytes encoded and decode on the way out (needs a typedef change).
   
   Related: #29, #30, dev@ thread "[DISCUSS]: Initiating development of the Sourcelume Attestation library".
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@sourcelume.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@sourcelume.apache.org
For additional commands, e-mail: issues-help@sourcelume.apache.org

Re: [I] POST /records accepts invalid UTF-8 and stores altered text under the original digest [sourcelume-registry]

["jgoodyear (via GitHub) via issues" <is...@sourcelume.apache.org>]

jgoodyear commented on issue #32:
URL: https://github.com/apache/sourcelume-registry/issues/32#issuecomment-6543137683

   +1 
   
   I lean toward strict input requirements here - it'll simplify things.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@sourcelume.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@sourcelume.apache.org
For additional commands, e-mail: issues-help@sourcelume.apache.org