You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by GitBox <gi...@apache.org> on 2022/04/13 14:30:11 UTC
[GitHub] [druid] lingguang opened a new issue, #12434: GroupBy query with DST timezone hangs in Historical server
lingguang opened a new issue, #12434:
URL: https://github.com/apache/druid/issues/12434
GroupBy query with DST timezone hangs in Historical server.
### Affected Version
At lease from 0.20.0 to master
### Description
We found some native groupby queries hung in historical server and caused query timeout.
All these query threads took up all merge buffers, then we had to kill the historical server for restarting.
So we did some debugging and found the cause:
The PeriodGranularity may not calculate the bucketStart correctly when works with timeZone that DST is used.
Here's what in our groupby query:
"granularity": {
"type": "period",
"period": "P7D",
"timeZone": "America/Denver",
"origin": "2022-03-27T02:35:00.000-06:00"
},
"intervals": [
"2022-03-06T02:35:00.000-07:00/2022-03-06T03:45:00.000-07:00"
]
Check this code:
org.apache.druid.java.util.common.granularity.PeriodGranularity.truncate(long)
Left: current, Right: possible fix
Check the output from our test program(attached):
[TestPeriod.zip](https://github.com/apache/druid/files/8482207/TestPeriod.zip)
origin 2022-03-27T02:35:00.000-06:00
segment min 2022-03-06T02:40:00.000-07:00
bucketStart 2022-03-06T03:35:00.000-07:00
-21D(expected) 2022-03-06T02:35:00.000-07:00
-14D 2022-03-13T03:35:00.000-06:00
-14D-7D 2022-03-06T03:35:00.000-07:00
DST Transition 2022-03-13T03:00:00.000-06:00
Due to the DST Transition, the result from origin -21D is not the same with -14D then -7D. It should have the same issue with period Week/Month.
In our case, it causes the granulizer selected nothing for the bucketInterval in the following code:
VectorGroupByEngine$VectorGroupByEngineIterator.initNewDelegate()
Finally causing a infinite loop in (the selected while loop):
VectorGroupByEngine$VectorGroupByEngineIterator.hasNext()
The query thread will take 100% CPU and never exit, after all merge buffers have been taken by these threads, no more groupby queries can be executed on this historical server. We had to kill this process for restarting.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [I] GroupBy query with DST timezone hangs in Historical server (druid)
Posted by "lingguang (via GitHub)" <gi...@apache.org>.
lingguang commented on issue #12434:
URL: https://github.com/apache/druid/issues/12434#issuecomment-1731601771
No, that's not a violation of the bucket size. Instead it is why it is called P7D not PT168H (7*24H). If you need keep the same time duration you should use PT168H. The P (period) part (Y/M/D/W) should be based on the calendar.
The lib org.joda.time.chrono.ISOChronology which used in PeriodGranularity class follows the ISO8601 standard and works as expected.
So the code here just has a minor defect, -14D-7D introduced problem because of the time gap of DST transition, -21D should be OK.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [I] GroupBy query with DST timezone hangs in Historical server (druid)
Posted by "LakshSingla (via GitHub)" <gi...@apache.org>.
LakshSingla commented on issue #12434:
URL: https://github.com/apache/druid/issues/12434#issuecomment-1705669037
In any case, the user shouldn't face this error, therefore we should check the reason why this is happening and take some measures. The solution you proposed seems fine since it doesn't take into account the diff that we are maintaining though I would like to know if there's a bug with the library/our code/ or some discrepancy due to entering timezone and the time lag in the same period to identify the root cause.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [I] GroupBy query with DST timezone hangs in Historical server (druid)
Posted by "gargvishesh (via GitHub)" <gi...@apache.org>.
gargvishesh commented on issue #12434:
URL: https://github.com/apache/druid/issues/12434#issuecomment-1731451161
If we consider `2022-03-06T02:35:00.000-07:00` as `-21D` value for `2022-03-27T02:35:00.000-06:00`, then effectively the time duration is `20D23H` -- since 1 hour was lost in DST transition. This leads to violation of the bucket size of `7D`. To account for the lost hour is the reason the right value should be `2022-03-06T01:35:00.000-07:00`.
Another way to arrive at the proposed value would be to convert the GMT time of `2022-03-06T08:35:00.000Z` to America/Denver time as that would automatically ensure correct bucket size while incorporating DST transition for the timezone.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [I] GroupBy query with DST timezone hangs in Historical server (druid)
Posted by "cryptoe (via GitHub)" <gi...@apache.org>.
cryptoe closed issue #12434: GroupBy query with DST timezone hangs in Historical server
URL: https://github.com/apache/druid/issues/12434
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [I] GroupBy query with DST timezone hangs in Historical server (druid)
Posted by "lingguang (via GitHub)" <gi...@apache.org>.
lingguang commented on issue #12434:
URL: https://github.com/apache/druid/issues/12434#issuecomment-1731118767
If you try my test program by changing the timezone to GMT:
DateTimeZone tz = DateTimeZone.forID("GMT");
origin 2022-03-27T08:35:00.000Z
-21D(expected) 2022-03-06T08:35:00.000Z
-14D 2022-03-13T08:35:00.000Z
-14D-7D 2022-03-06T08:35:00.000Z
both -21D and -14D-7D will get the same correct result. So the problem only exists for the timezone with DST transition.
It is expected to maintain the time bucket according to the user specified timezone, and it may have different results for different timezones.
So in this case:
if the timezone is GMT, the result should be 2022-03-06T08:35:00.000Z
if the timezone is America/Denver, the result should be 2022-03-06T02:35:00.000-07:00
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [I] GroupBy query with DST timezone hangs in Historical server (druid)
Posted by "lingguang (via GitHub)" <gi...@apache.org>.
lingguang commented on issue #12434:
URL: https://github.com/apache/druid/issues/12434#issuecomment-1706190365
"period": "P7D"
origin 2022-03-27T02:35:00.000-06:00
-21D(expected) 2022-03-06T02:35:00.000-07:00
-14D 2022-03-13T03:35:00.000-06:00
-14D-7D 2022-03-06T03:35:00.000-07:00
DST Transition 2022-03-13T03:00:00.000-06:00
We are using the timezone (America/Denver), and there is a DST transition on 2022-03-13 from -7:00 to -6:00.
-14D happened to fall into the DST transition gap, so there was no 2:35 local time on that day, and the nearest 03:35 was returned. Then when -7D again, it returned 3:35 local time on 2022-03-06.
With -21D, it returned 2:35 local tome on 2022-03-06, which is expected.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [I] GroupBy query with DST timezone hangs in Historical server (druid)
Posted by "LakshSingla (via GitHub)" <gi...@apache.org>.
LakshSingla commented on issue #12434:
URL: https://github.com/apache/druid/issues/12434#issuecomment-1705667854
Thanks for the detailed explanation of the issue @lingguang. One thing that I am still confused from the description is that why did the -14D-7D transition didn't take into account the DST. Seems like that there is some funny business going on where the DST isn't considered if we operate on the date time before hand 🤔, since the the second part of this transition should have taken into account the the DST change.
Also, why is the granularity specifying both the timezone (America/Denver) and the offset (-6:00) simultaneously? Is there any advantage in doing so?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [I] GroupBy query with DST timezone hangs in Historical server (druid)
Posted by "gargvishesh (via GitHub)" <gi...@apache.org>.
gargvishesh commented on issue #12434:
URL: https://github.com/apache/druid/issues/12434#issuecomment-1730947411
@lingguang I'm working to put in the fix for this.
In the example you posted, if we convert the values to GMT times, none of the outputs among `-14D-7D` and `-21D` seem correct.
GMT Times:
origin 2022-03-27T08:35:00.000Z
-21D(expected) 2022-03-06T09:35:00.000Z
-14D 2022-03-13T09:35:00.000Z
-14D-7D 2022-03-06T10:35:00.000Z
To maintain correct `7D` bucket sizes, the correct `-21D` GMT time should rather be `2022-03-06T08:35:00.000Z`, which converted to `America/Denver` timezone -- taking into account DST transition -- becomes `2022-03-06T01:35:00.000-07:00`. Do you agree with this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org