You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by GitBox <gi...@apache.org> on 2021/03/11 21:57:04 UTC
[GitHub] [druid] ziggythehamster opened a new issue #10983: SQL JSON interface should have a context option to write decimals as strings to avoid unintended float coercion
ziggythehamster opened a new issue #10983:
URL: https://github.com/apache/druid/issues/10983
### Description
Currently, the SQL JSON interface will return decimals as e.g., `{"foo": 1.2345}`. Client libraries which don't know to handle this as a decimal value instead of a float value would then coerce this to a float, and potentially cause a loss of precision. This appears to be a relatively simple change to allow the client to control the Jackson JsonGenerator.Feature WRITE_BIGDECIMAL_AS_PLAIN, but I'm not 100% sure that decimals are BigDecimal inside Druid. The other option, to make all numbers strings, is not appropriate because you wouldn't want to have to convert whole numbers or floats from strings back into their native types.
### Motivation
- I'm working on a Ruby client library
- Wrapping all decimal calculations in `CAST(x AS VARCHAR)` is boilerplate
- The Avatica API already handles this correctly by passing type information, but the Avatica API is not documented with enough detail that it would be feasible for a non-Calcite developer to write an Avatica client in an unsupported language. FFI + the Go version might be an option, but that seems like a whole lot more work than dealing with this simple JSON API.
- This API does not pass type information, which means that an approach of converting anything that has a decimal point into a precision-preserving format (BigDecimal in Ruby) wastes resources unnecessarily. A generic client library could ship a decimal string to a user without loss of precision or performance, and then the user, who knows the data types, could convert as needed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] ziggythehamster commented on issue #10983: SQL JSON interface should have a context option to write decimals as strings to avoid unintended float coercion
Posted by GitBox <gi...@apache.org>.
ziggythehamster commented on issue #10983:
URL: https://github.com/apache/druid/issues/10983#issuecomment-805547456
That's a good point that I hadn't remembered when I wrote this bug.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] JulianJaffePinterest commented on issue #10983: SQL JSON interface should have a context option to write decimals as strings to avoid unintended float coercion
Posted by GitBox <gi...@apache.org>.
JulianJaffePinterest commented on issue #10983:
URL: https://github.com/apache/druid/issues/10983#issuecomment-800892414
Druid doesn't operate over fixed-point decimal numbers (see e.g. [the Druid SQL type docs](http://druid.apache.org/docs/latest/querying/sql.html#data-types) or the [dimension spec supported data types](http://druid.apache.org/docs/latest/ingestion/index.html#dimension-objects). I haven't poked through the Calcite code recently; it's possible that doing something like `SELECT CAST(1 AS DECIMAL)/CAST(4.7 AS DECIMAL)` would use fixed-point math if Calcite reduces the constant expression without passing to Druid for evaluation. Anything that compiles into a Druid query under the hood will use floating-point arithmetic.
If you're just building a client and want to include decimals for completeness, there's not much else you can do beyond note the limitation in your documentation. If you're trying to perform fixed-point calculations over your data in Druid and you don't need arbitrary precision, only precision up to some number of decimal places (for example, currency operations), you can scale your data at ingestion and then use integer division when computing your final result.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] JulianJaffePinterest edited a comment on issue #10983: SQL JSON interface should have a context option to write decimals as strings to avoid unintended float coercion
Posted by GitBox <gi...@apache.org>.
JulianJaffePinterest edited a comment on issue #10983:
URL: https://github.com/apache/druid/issues/10983#issuecomment-800892414
Druid doesn't operate over fixed-point decimal numbers (see e.g. [the Druid SQL type docs](http://druid.apache.org/docs/latest/querying/sql.html#data-types) or the [dimension spec supported data types](http://druid.apache.org/docs/latest/ingestion/index.html#dimension-objects)). I haven't poked through the Calcite code recently; it's possible that doing something like `SELECT CAST(1 AS DECIMAL)/CAST(4.7 AS DECIMAL)` would use fixed-point math if Calcite reduces the constant expression without passing to Druid for evaluation. Anything that compiles into a Druid query under the hood will use floating-point arithmetic.
If you're just building a client and want to include decimals for completeness, there's not much else you can do beyond note the limitation in your documentation. If you're trying to perform fixed-point calculations over your data in Druid and you don't need arbitrary precision, only precision up to some number of decimal places (for example, currency operations), you can scale your data at ingestion and then use integer division when computing your final result.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org