You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hop.apache.org by "vbhanuchander-lang (via GitHub)" <gi...@apache.org> on 2026/08/01 15:26:40 UTC

Re: [I] [Bug]: Hop serializes numeric values into scientific notation, causing precision loss in downstream databases (hop)

vbhanuchander-lang commented on issue #6111:
URL: https://github.com/apache/hop/issues/6111#issuecomment-5152073689

   I've been digging into this and believe I've found the exact mechanism, so sharing it here. I've opened #7744 with a fix.
   
   The formatting layer isn't the culprit — both of Hop's default masks render the value correctly:
   
   ```
   DecimalFormat("####0.0#########")             -> 55487400.0   (default NUMBER mask)
   DecimalFormat("######0.0###################") -> 55487400.0   (default BIGNUMBER mask)
   ```
   
   It comes from the Number to BigNumber conversion instead. `BigDecimal.valueOf(double)` is specified as `new BigDecimal(Double.toString(val))`, and `Double.toString(55487400.0)` returns `"5.54874E7"`. That parses into an unscaled value of 554874 with a **scale of -2**, and it is that negative scale which makes `BigDecimal.toString()` emit `5.54874E+7`:
   
   ```java
   BigDecimal bd = BigDecimal.valueOf(55487400.0);
   bd.scale();          // -2
   bd.unscaledValue();  // 554874
   bd.toString();       // "5.54874E+7"
   bd.toPlainString();  // "55487400"
   ```
   
   The `+` sign is a useful giveaway that this is the BigDecimal path rather than a bare `Double.toString()`, which would give `5.54874E7` without it.
   
   In `ValueMetaBase` this happens in `getBigNumber()` for `TYPE_NUMBER` (all three storage-type branches) and in `convertStringToBigNumber()`. The negative scale then leaks out through every consumer that serializes with `toString()`:
   
   - JDBC drivers that inline statement parameters — the Table Output case reported here
   - `writeBigNumber()`, used by the binary row serialization
   - `getDataXml()`, used by the XML data serialization
   
   One clarification on the downstream drift: at the Java level `Double.parseDouble("5.54874E+7") == Double.parseDouble("55487400")` is `true`, so the `55487400.00000001` you observed is coming from ClickHouse's own string-to-Float64 parsing rather than from information lost in the string Hop emits. That is consistent with what you found about ClickHouse PR #52791 affecting string-to-float conversion while INSERT statements behaved as before. Hop still should not be emitting scientific notation here, so a Hop-side fix is warranted regardless — it just means the fix removes the trigger rather than the rounding itself.
   
   The fix in #7744 routes those conversions through a helper that rescales to zero when the scale would be negative. Rescaling upwards from a negative scale is exact, so no rounding occurs and the numeric value is unchanged; values that already convert to a non-negative scale keep their exact previous scale. The full `core` module suite passes (1039 tests).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@hop.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org