You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@olingo.apache.org by "Svante von Erichsen (JIRA)" <ji...@apache.org> on 2017/03/02 10:18:45 UTC
[jira] [Comment Edited] (OLINGO-1087) Precision in EdmDecimal not
calculated correctly
[ https://issues.apache.org/jira/browse/OLINGO-1087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15891980#comment-15891980 ]
Svante von Erichsen edited comment on OLINGO-1087 at 3/2/17 10:18 AM:
----------------------------------------------------------------------
OK, I think I get it.
First, any mathematical notion of mantissa and exponent seems misleading for this (otherwise my first idea could seem to be right).
Instead, only look at the printed representation in radix 10, omitting all leading zeroes before the decimal point and all trailing zeroes after it.
If scale is a nonnegative integer, it is the maximum count of digits after the decimal point in that printed representation.
If scale is "variable" or omitted, precision is the maximum count of digits in that printed representation. If scale is a nonnegative integer, then precision minus scale is the maximum count of digits before the decimal point.
Examples:
0.0302 fits in Precision=4, Scale=4, but not in Precision=3, Scale=4.
320 fits in Precision=3, Scale=0, but not in Precision=3, Scale=1
was (Author: serichsen):
OK, I think I get it.
First, any mathematical notion of mantissa and exponent seems misleading for this (otherwise my first idea could seem to be right).
Instead, only look at the printed representation in radix 10, omitting all leading zeroes before the decimal point and all trailing zeroes after it.
If scale is a nonnegative integer, it is the maximum count of digits after the decimal point in that printed representation.
If scale is "variable" or omitted, precision is the count of digits in that printed representation. If scale is a nonnegative integer, then precision minus scale is the maximum count of digits before the decimal point.
Examples:
0.0302 fits in Precision=4, Scale=4, but not in Precision=3, Scale=4.
320 fits in Precision=3, Scale=0, but not in Precision=3, Scale=1
> Precision in EdmDecimal not calculated correctly
> ------------------------------------------------
>
> Key: OLINGO-1087
> URL: https://issues.apache.org/jira/browse/OLINGO-1087
> Project: Olingo
> Issue Type: Bug
> Components: odata2-core
> Affects Versions: V2 2.0.8
> Reporter: Svante von Erichsen
> Attachments: 0001-OLINGO-1087-Fix-calculation-of-precision.patch
>
>
> The "precision" found in a read number is currently at least the number of decimals (omitting trailing zeroes). That seems to be wrong, or at least not consistent with at least some server side view of the matter, for numbers smaller than 0.1 (the number shown has 35 decimals, but precision 34):
> org.apache.olingo.odata2.api.edm.EdmSimpleTypeException: The metadata constraints '[Precision=34]' do not match the literal '0.08102359900000000170194880411145277'.
> at org.apache.olingo.odata2.core.edm.EdmDecimal.internalValueOfString(EdmDecimal.java:107)
> at org.apache.olingo.odata2.core.edm.AbstractSimpleType.valueOfString(AbstractSimpleType.java:91)
> at org.apache.olingo.odata2.core.ep.consumer.JsonPropertyConsumer.readSimpleProperty(JsonPropertyConsumer.java:236)
> at org.apache.olingo.odata2.core.ep.consumer.JsonPropertyConsumer.readPropertyValue(JsonPropertyConsumer.java:169)
> ... 69 common frames omitted
> Wrapped by: org.apache.olingo.odata2.api.ep.EntityProviderException: An exception of type 'EdmSimpleTypeException' occurred.
> at org.apache.olingo.odata2.core.ep.consumer.JsonPropertyConsumer.readPropertyValue(JsonPropertyConsumer.java:171)
> at org.apache.olingo.odata2.core.ep.consumer.JsonEntryConsumer.handleName(JsonEntryConsumer.java:172)
> at org.apache.olingo.odata2.core.ep.consumer.JsonEntryConsumer.readEntryContent(JsonEntryConsumer.java:130)
> at org.apache.olingo.odata2.core.ep.consumer.JsonEntryConsumer.readFeedEntry(JsonEntryConsumer.java:117)
> at org.apache.olingo.odata2.core.ep.consumer.JsonFeedConsumer.readArrayContent(JsonFeedConsumer.java:153)
> at org.apache.olingo.odata2.core.ep.consumer.JsonFeedConsumer.handleName(JsonFeedConsumer.java:122)
> at org.apache.olingo.odata2.core.ep.consumer.JsonFeedConsumer.readFeedContent(JsonFeedConsumer.java:111)
> at org.apache.olingo.odata2.core.ep.consumer.JsonFeedConsumer.readFeed(JsonFeedConsumer.java:96)
> at org.apache.olingo.odata2.core.ep.consumer.JsonFeedConsumer.readFeedStandalone(JsonFeedConsumer.java:63)
> at org.apache.olingo.odata2.core.ep.consumer.JsonEntityConsumer.readDeltaFeed(JsonEntityConsumer.java:95)
> at org.apache.olingo.odata2.core.ep.consumer.JsonEntityConsumer.readFeed(JsonEntityConsumer.java:81)
> at org.apache.olingo.odata2.core.ep.JsonEntityProvider.readFeed(JsonEntityProvider.java:309)
> at org.apache.olingo.odata2.core.ep.ProviderFacadeImpl.readFeed(ProviderFacadeImpl.java:165)
> at org.apache.olingo.odata2.api.ep.EntityProvider.readFeed(EntityProvider.java:676)
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)