You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "jvenant (via GitHub)" <gi...@apache.org> on 2023/04/27 21:21:17 UTC
[GitHub] [pinot] jvenant opened a new issue, #10697: double Nan as default value can massivly expand off heap mutable dictionary
jvenant opened a new issue, #10697:
URL: https://github.com/apache/pinot/issues/10697
Hi,
It seems that using NaN as a default value for Double in schemas can lead to much more calls to the BaseOffHeapMutableDictionary.expand that required.
And because a multiplication factor is apply at each call. It can quickly try to allocate a huge amount of off heap memory
I suspect a hazardous conversion of NaN Double to null somewhere that prevent the DoubleOffHeapMutableDictionary.compare to achieve its goal correctly
I wasn't able to get much more details, but maybe be it will be enough to ring a bell to someone knowing this part of Pinot.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] double NaN as default value can massivly expand off heap mutable dictionary [pinot]
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on issue #10697:
URL: https://github.com/apache/pinot/issues/10697#issuecomment-1762709119
To simplify the handling, as a short term solution, I'd propose:
- Not allowing `NaN` and replace it with default null value when ingested
- Replacing `-0.0` with `0.0`
We may consider following PostgreSQL behavior as long term solution
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] jvenant commented on issue #10697: double Nan as default value can massivly expand off heap mutable dictionary
Posted by "jvenant (via GitHub)" <gi...@apache.org>.
jvenant commented on issue #10697:
URL: https://github.com/apache/pinot/issues/10697#issuecomment-1527101420
Maybe a quick fix would be to disable NaN.
But from my standpoint, to be able to use NaN would be a better solution as it is not semantically equivalent to Infinities, null or zero. It is a very common value used in mathematical and finance domains.
The only other place where we have issue with NaN so far is in the where filters. We have to do something like : `cast(num_col as STRING) = 'NaN'`.
`num_col = 'NaN'`, `num_col != 'NaN'` run, but doesn't filter the records correctly
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang commented on issue #10697: double NaN as default value can massivly expand off heap mutable dictionary
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on issue #10697:
URL: https://github.com/apache/pinot/issues/10697#issuecomment-1728675910
I feel we should just reject `NaN` for now. Dictionary binary search might also fail.
To reject it, we can add a check in `FieldSpec.getDefaultNullValue()`, and in the record reader, replace `NaN` with the default value. @jvenant Do you want to help contribute this?
cc @snleee
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] double NaN as default value can massivly expand off heap mutable dictionary [pinot]
Posted by "snleee (via GitHub)" <gi...@apache.org>.
snleee commented on issue #10697:
URL: https://github.com/apache/pinot/issues/10697#issuecomment-1762547917
MySQL: not allow `NaN`
Postgres: allow `NaN`
official documentation states, `In order to allow numeric values to be sorted and used in tree-based indexes, PostgreSQL treats NaN values as equal, and greater than all non-NaN values.`
Some similar issue with `-0.0` and `0.0` (This would also mess up the binary search/sorting)
https://jonisalonen.com/2013/positive-and-negative-zeros-and-mysql/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] double NaN as default value can massivly expand off heap mutable dictionary [pinot]
Posted by "hpvd (via GitHub)" <gi...@apache.org>.
hpvd commented on issue #10697:
URL: https://github.com/apache/pinot/issues/10697#issuecomment-2170915489
@Jackie-Jiang since now also the second bugfix is merged: can this issue be closed or do we keep it open till `PostgreSQL behavior` is implemented?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] jvenant commented on issue #10697: double NaN as default value can massivly expand off heap mutable dictionary
Posted by "jvenant (via GitHub)" <gi...@apache.org>.
jvenant commented on issue #10697:
URL: https://github.com/apache/pinot/issues/10697#issuecomment-1728195024
I think this bug can be very dangerous. Maybe waiting for a good NaN management, we could reject schemas in case on NaN default...?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] jvenant commented on issue #10697: double NaN as default value can massivly expand off heap mutable dictionary
Posted by "jvenant (via GitHub)" <gi...@apache.org>.
jvenant commented on issue #10697:
URL: https://github.com/apache/pinot/issues/10697#issuecomment-1736110874
Not sure the PR actually fix the issue. It's more a protection than a fix
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] double NaN as default value can massivly expand off heap mutable dictionary [pinot]
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang closed issue #10697: double NaN as default value can massivly expand off heap mutable dictionary
URL: https://github.com/apache/pinot/issues/10697
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang closed issue #10697: double NaN as default value can massivly expand off heap mutable dictionary
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang closed issue #10697: double NaN as default value can massivly expand off heap mutable dictionary
URL: https://github.com/apache/pinot/issues/10697
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang commented on issue #10697: double Nan as default value can massivly expand off heap mutable dictionary
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on issue #10697:
URL: https://github.com/apache/pinot/issues/10697#issuecomment-1526632574
Good catch. The problem is from `DoubleOffHeapMutableDictionary.equalsValueAt()` which will always return `false` when input `value` is `NaN`.
We can fix it by comparing the `Double.doubleToLongBits()` (bytes level same), but I'm not sure if allowing `NaN` can cause other problems (it might break the binary search as well). Another option is to prevent `NaN` to be ingested (not allowing it as default value, and also replace it with default value when seeing it in the input data)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang commented on issue #10697: double NaN as default value can massivly expand off heap mutable dictionary
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on issue #10697:
URL: https://github.com/apache/pinot/issues/10697#issuecomment-1736119952
It is auto-closed by merging #11661. Reopened it since we haven't addressed the issue completely
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] double NaN as default value can massivly expand off heap mutable dictionary [pinot]
Posted by "snleee (via GitHub)" <gi...@apache.org>.
snleee commented on issue #10697:
URL: https://github.com/apache/pinot/issues/10697#issuecomment-1769008775
@Jackie-Jiang sounds good to me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org