You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@pinot.apache.org by GitBox <gi...@apache.org> on 2021/03/31 23:40:30 UTC

[GitHub] [incubator-pinot] mr-agrwal opened a new issue #5829: Pinot Query Console loosing Long granularity in results.

mr-agrwal opened a new issue #5829:
URL: https://github.com/apache/incubator-pinot/issues/5829


   I am querying a table using the Pinot Query Console with id = 21400000008726127. 
   The response returns with id = 21400000008726130. 
   The same if I do through a broker, it is working fine.
   Also, the broker is printing the query correctly in both cases. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [incubator-pinot] Jackie-Jiang closed issue #5829: Pinot Query Console loosing Long granularity in results.

Posted by GitBox <gi...@apache.org>.

Jackie-Jiang closed issue #5829:
URL: https://github.com/apache/incubator-pinot/issues/5829


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [incubator-pinot] Jackie-Jiang commented on issue #5829: Pinot Query Console loosing Long granularity in results.

Posted by GitBox <gi...@apache.org>.

Jackie-Jiang commented on issue #5829:
URL: https://github.com/apache/incubator-pinot/issues/5829#issuecomment-820834802


   After more offline discussion, to prevent breaking the client code, we decide to limit the changes to the UI to handle large numbers.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [incubator-pinot] Jackie-Jiang commented on issue #5829: Pinot Query Console loosing Long granularity in results.

Posted by GitBox <gi...@apache.org>.

Jackie-Jiang commented on issue #5829:
URL: https://github.com/apache/incubator-pinot/issues/5829#issuecomment-812773280


   @mcvsubbu @yupeng9 @siddharthteotia @fx19880617 Do you think it could cause backward incompatible issue on client side if we return values as string as mentioned above?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [incubator-pinot] mayankshriv commented on issue #5829: Pinot Query Console loosing Long granularity in results.

Posted by GitBox <gi...@apache.org>.

mayankshriv commented on issue #5829:
URL: https://github.com/apache/incubator-pinot/issues/5829#issuecomment-812143648


   If we go route1, then we might want to measure overhead with a large
   response. I have seen orders of magnitude difference in latency seen by the
   broker vs something on top of the broker (eg query console, python script,
   etc) which end up doing these kinds of transformations per value.
   
   Cheers,
   Mayank
   
   On Thu, Apr 1, 2021 at 11:58 AM Gaurav Joshi ***@***.***>
   wrote:
   
   > I believe there are a couple of ways to tackle this.
   >
   > *1.* Given that long precision is lost when UI requests query result and
   > that the result is display-only (meaning it is not further used in any
   > business logic or for any processing) the server can simply send all the
   > data as string values. However, this will require changes in the server as
   > well as minor changes in the UI. In case the data is required to be used
   > for some processing, it can be converted into appropriate type using
   > columnDataTypes - which is when we will run into this issue, again.
   > (Please correct me in case I am wrong, given some gaps in my Pinot UI
   > domain knowledge.)
   >
   > *2.* We can limit the changes to the UI. One solution I can think of is
   > using json-bigint or similar library that supports converting large long
   > values to JavaScript BigInt type and thus retain precision. This will
   > require disabling native JSON parsing in axios with transformResponse and
   > use the chosen JSON parsing library instead.
   >
   > @shahsank3t <https://github.com/shahsank3t>
   >
   > —
   > You are receiving this because you are subscribed to this thread.
   > Reply to this email directly, view it on GitHub
   > <https://github.com/apache/incubator-pinot/issues/5829#issuecomment-812107499>,
   > or unsubscribe
   > <https://github.com/notifications/unsubscribe-auth/ADAIBTIM62BSEACXL3JIL7TTGS66XANCNFSM4PX3R2MQ>
   > .
   >
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [incubator-pinot] Jackie-Jiang commented on issue #5829: Pinot Query Console loosing Long granularity in results.

Posted by GitBox <gi...@apache.org>.

Jackie-Jiang commented on issue #5829:
URL: https://github.com/apache/incubator-pinot/issues/5829#issuecomment-811533898


   Reopen this issue as we should show long values correctly in Pinot Query Console


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [incubator-pinot] shahsank3t commented on issue #5829: Pinot Query Console loosing Long granularity in results.

Posted by GitBox <gi...@apache.org>.

shahsank3t commented on issue #5829:
URL: https://github.com/apache/incubator-pinot/issues/5829#issuecomment-872952070


   PR to fix this issue: https://github.com/apache/incubator-pinot/pull/7120


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [incubator-pinot] kishoreg commented on issue #5829: Pinot Query Console loosing Long granularity in results.

Posted by GitBox <gi...@apache.org>.

kishoreg commented on issue #5829:
URL: https://github.com/apache/incubator-pinot/issues/5829#issuecomment-673886031


   @mr-agrwal is this still an issue?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [incubator-pinot] mr-agrwal commented on issue #5829: Pinot Query Console loosing Long granularity in results.

Posted by GitBox <gi...@apache.org>.

mr-agrwal commented on issue #5829:
URL: https://github.com/apache/incubator-pinot/issues/5829#issuecomment-670653462


   Query is `select * from table where id = 21400000008726127 limit 10`
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [incubator-pinot] mr-agrwal commented on issue #5829: Pinot Query Console loosing Long granularity in results.

Posted by GitBox <gi...@apache.org>.

mr-agrwal commented on issue #5829:
URL: https://github.com/apache/incubator-pinot/issues/5829#issuecomment-675407962


   @kishoreg Closing this, as this is an issue with javascript parsing long values. This is further discussed in detail in https://www.techempower.com/blog/2016/07/05/mangling-json-numbers/


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [incubator-pinot] kishoreg commented on issue #5829: Pinot Query Console loosing Long granularity in results.

Posted by GitBox <gi...@apache.org>.

kishoreg commented on issue #5829:
URL: https://github.com/apache/incubator-pinot/issues/5829#issuecomment-670645164


   whats the query, can you share the screen shots?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [incubator-pinot] mr-agrwal closed issue #5829: Pinot Query Console loosing Long granularity in results.

Posted by GitBox <gi...@apache.org>.

mr-agrwal closed issue #5829:
URL: https://github.com/apache/incubator-pinot/issues/5829


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [incubator-pinot] Jackie-Jiang commented on issue #5829: Pinot Query Console loosing Long granularity in results.

Posted by GitBox <gi...@apache.org>.

Jackie-Jiang commented on issue #5829:
URL: https://github.com/apache/incubator-pinot/issues/5829#issuecomment-812237787


   For approach 1, my concern is more about changing the format in the json response, which might break the service parsing it. But approach 1 seems the better solution because if we only change our UI to handle it, other clients reading the result can also run into the same issue.
   
   For approach 1, there are 2 options:
   1. Return all values as string (with double quotes)
   2. Only convert large values to string
   
   IMO, option 1 has consistent format of the result (always have values double quoted), but will change the format of the current results; option 2 could return some values with double quotes and some without, but won't change the format for most of the results.
   
   Thoughts on this?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [incubator-pinot] Jackie-Jiang commented on issue #5829: Pinot Query Console loosing Long granularity in results.

Posted by GitBox <gi...@apache.org>.

Jackie-Jiang commented on issue #5829:
URL: https://github.com/apache/incubator-pinot/issues/5829#issuecomment-811534840


   UI result (lose precision):
   ![Screen Shot 2021-03-31 at 3 55 46 PM](https://user-images.githubusercontent.com/17555551/113224244-df301f80-923f-11eb-99d3-6c5094f64a5f.png)
   
   Controller rest API result (accurate):
   ```
   jackie@MacBook-Pro incubator-pinot % curl -H "Content-Type: application/json" -X POST -d '{"sql":"select 9223372036854775807 from mytable"}' http://localhost:18998/sql
   {"resultTable":{"dataSchema":{"columnDataTypes":["LONG"],"columnNames":["9223372036854775807"]},"rows":[[9223372036854775807]]},"exceptions":[],"numServersQueried":0,"numServersResponded":0,"numSegmentsQueried":0,"numSegmentsProcessed":0,"numSegmentsMatched":0,"numConsumingSegmentsQueried":0,"numDocsScanned":0,"numEntriesScannedInFilter":0,"numEntriesScannedPostFilter":0,"numGroupsLimitReached":false,"totalDocs":0,"timeUsedMs":0,"segmentStatistics":[],"traceInfo":{},"minConsumingFreshnessTimeMs":0}
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [incubator-pinot] joshigaurava commented on issue #5829: Pinot Query Console loosing Long granularity in results.

Posted by GitBox <gi...@apache.org>.

joshigaurava commented on issue #5829:
URL: https://github.com/apache/incubator-pinot/issues/5829#issuecomment-812107499

I believe there are a couple of ways to tackle this.

**1.** Given that long precision is lost when UI requests query result and that the result is display-only (meaning it is not further used in any business logic or for any processing) the server can simply send all the data as string values. However, this will require changes in the server as well as minor changes in the UI. In case the data is required to be used for some processing, it can be converted into appropriate type using `columnDataTypes` - which is when we will run into this issue, again. (Please correct me in case I am wrong, given some gaps in my Pinot UI domain knowledge.)

**2.** We can limit the changes to the UI. One solution I can think of is using json-bigint or similar library that supports converting large long values to JavaScript BigInt type and thus retain precision. This will require disabling native JSON parsing in axios with `transformResponse` and use the chosen JSON parsing library instead.

@shahsank3t

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [incubator-pinot] Jackie-Jiang commented on issue #5829: Pinot Query Console loosing Long granularity in results.

Posted by GitBox <gi...@apache.org>.

Jackie-Jiang commented on issue #5829:
URL: https://github.com/apache/incubator-pinot/issues/5829#issuecomment-813748893


   Decide to go with option 2 (only convert large values to string) as it won't have any impact to most use cases (values larger than 2^53 or smaller than -2^53 are very uncommon): #6745 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org