You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by "GWphua (via GitHub)" <gi...@apache.org> on 2025/09/04 04:02:16 UTC
[PR] Optimize interval deserialization (druid)
GWphua opened a new pull request, #18477:
URL: https://github.com/apache/druid/pull/18477
### Description
<!-- Describe the goal of this PR, what problem are you fixing. If there is a corresponding issue (referenced above), it's not necessary to repeat the description here, however, you may choose to keep one summary sentence. -->
<!-- Describe your patch: what did you change in code? How did you fix the problem? -->
<!-- If there are several relatively logically separate changes in this PR, create a mini-section for each of them. For example: -->
This PR introduces a faster deserialization strategy when dealing with Intervals.
#### Motivation
Many Druid JSON payloads represent intervals as "YYYY-MM-DD'T'HH:mm:ss.SSSZ/YYYY-MM-DD'T'HH:mm:ss.SSSZ". The general parser behind Intervals.of(String) is flexible but performs extra work (branching, multiple format probes, allocations). This shows up as a hot spot in flame graphs, especially in streaming paths like JsonParserIterator.
<img width="1920" height="958" alt="image" src="https://github.com/user-attachments/assets/fdb6fca4-f7a9-45bb-8f6c-bbd4ac901876" />
#### Faster deserialization
Added a fast-path deserializer for `Interval` that works with the common interval string representations, and falls back to `Intervals.of(String)` for non-matching inputs.
We minimize work done per parse by:
- Avoid checking if String can be converted into a `ReadableInterval` / `ReadWriteableInterval`.
- Precompiled `DateTimeFormatter` replaces the flexible parser, with fixed chronology.
- Avoid building 2 `DateTime` objects per parse.
#### Benchmarks
Added a benchmark to test performance of old (legacy) and new (optimized) implementation on 2 cases:
1. The expected interval format
2. The interval format which causes a fallback to the old implementation. (Suffix: Fallback)
```
Benchmark (numValues) Mode Cnt Score Error Units
JodaIntervalDeserialization.deserializeLegacy 20000 avgt 30 21.466 ± 0.191 ms/op
JodaIntervalDeserialization.deserializeLegacyFallback 20000 avgt 30 13.953 ± 0.249 ms/op
JodaIntervalDeserialization.deserializeOptimized 20000 avgt 30 11.921 ± 0.082 ms/op
JodaIntervalDeserialization.deserializeOptimizedFallback 20000 avgt 30 30.500 ± 0.304 ms/op
```
Benchmark shows an estimated 44% speed-up when working with the interval of format "YYYY-MM-DD'T'HH:mm:ss.SSSZ/YYYY-MM-DD'T'HH:mm:ss.SSSZ".
However, for other formats, the fallback causes the performance to drop.
#### Release note
Speed up deserialization time of Intervals by >40%
<hr>
##### Key changed/added classes in this PR
* `processing/src/main/java/org/apache/druid/java/util/common/Intervals.java`
* `processing/src/main/java/org/apache/druid/jackson/JodaStuff.java`
* `processing/src/test/java/org/apache/druid/java/util/common/IntervalsTest.java`
* `benchmarks/src/test/java/org/apache/druid/benchmark/JodaIntervalDeserialization.java`
<hr>
<!-- Check the items by putting "x" in the brackets for the done things. Not all of these items apply to every PR. Remove the items which are not done or not relevant to the PR. None of the items from the checklist below are strictly necessary, but it would be very helpful if you at least self-review the PR. -->
This PR has:
- [x] been self-reviewed.
- [x] a release note entry in the PR description.
- [x] added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
- [x] added or updated version, license, or notice information in [licenses.yaml](https://github.com/apache/druid/blob/master/dev/license.md)
- [x] added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
- [x] added unit tests or modified existing tests to cover new code paths, ensuring the threshold for [code coverage](https://github.com/apache/druid/blob/master/dev/code-review/code-coverage.md) is met.
- [x] been tested in a test Druid cluster.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Optimize interval deserialization (druid)
Posted by "kfaraz (via GitHub)" <gi...@apache.org>.
kfaraz commented on code in PR #18477:
URL: https://github.com/apache/druid/pull/18477#discussion_r2320811376
##########
processing/src/main/java/org/apache/druid/java/util/common/Intervals.java:
##########
@@ -53,6 +57,46 @@ public static Interval of(String format, Object... formatArgs)
return of(StringUtils.format(format, formatArgs));
}
+ /**
+ * @return Null if cannot parse with optimized strategy, else return the Interval.
Review Comment:
For what kind of interval strings does this method return null?
Is there a simpler way to short circuit in those cases rather than trying the parse unnecessarily?\
Maybe a `canDeserializeOptimally()` method which checks it upfront? (Perhaps just check if there is `P` in there 😛 )
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Optimize interval deserialization (druid)
Posted by "GWphua (via GitHub)" <gi...@apache.org>.
GWphua commented on code in PR #18477:
URL: https://github.com/apache/druid/pull/18477#discussion_r2328603280
##########
processing/src/main/java/org/apache/druid/java/util/common/Intervals.java:
##########
@@ -25,13 +25,17 @@
import org.joda.time.DateTime;
import org.joda.time.Interval;
import org.joda.time.chrono.ISOChronology;
+import org.joda.time.format.DateTimeFormatter;
+import org.joda.time.format.ISODateTimeFormat;
import javax.annotation.Nullable;
public final class Intervals
{
public static final Interval ETERNITY = utc(JodaUtils.MIN_INSTANT, JodaUtils.MAX_INSTANT);
public static final ImmutableList<Interval> ONLY_ETERNITY = ImmutableList.of(ETERNITY);
+ private static final DateTimeFormatter FAST_ISO_UTC_FORMATTER =
Review Comment:
I did a benchmark for the formatters. Seems `DateTimes.ISO_DATE_TIME` is a bit slower than the current optimization because it tries to create `DateTime` unnecessary objects. `Intervals#utc` can work with `long` to create `Interval` directly, and hence performs faster.
On the topic of using other data formats, seems like we can look for `DateTimes.ISO_DATE_OPTIONAL_TIME` to improve the deserialization, but the improvement is quite small.
```
Benchmark (numValues) Mode Cnt Score Error Units
JodaIntervalDeserializationBenchmark.deserializeLegacy 20000 avgt 50 20.858 ± 0.035 ms/op
JodaIntervalDeserializationBenchmark.deserializeOptimized 20000 avgt 50 11.940 ± 0.009 ms/op
JodaIntervalDeserializationBenchmark.deserializeDateTime 20000 avgt 50 12.524 ± 0.025 ms/op
JodaIntervalDeserializationBenchmark.deserializeDateTimeOffset 20000 avgt 50 23.505 ± 0.050 ms/op
JodaIntervalDeserializationBenchmark.deserializeOptionalDateTime 20000 avgt 50 20.663 ± 0.111 ms/op
JodaIntervalDeserializationBenchmark.deserializeLegacyFallback 20000 avgt 50 19.035 ± 0.032 ms/op
JodaIntervalDeserializationBenchmark.deserializeOptimizedFallback 20000 avgt 50 33.782 ± 0.105 ms/op
JodaIntervalDeserializationBenchmark.deserializeDateTimeFallback 20000 avgt 50 36.642 ± 0.065 ms/op
JodaIntervalDeserializationBenchmark.deserializeDateTimeOffsetFallback 20000 avgt 50 22.067 ± 0.046 ms/op
JodaIntervalDeserializationBenchmark.deserializeOptionalDateTimeFallback 20000 avgt 50 18.861 ± 0.012 ms/op
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Optimize interval deserialization (druid)
Posted by "GWphua (via GitHub)" <gi...@apache.org>.
GWphua commented on code in PR #18477:
URL: https://github.com/apache/druid/pull/18477#discussion_r2328603280
##########
processing/src/main/java/org/apache/druid/java/util/common/Intervals.java:
##########
@@ -25,13 +25,17 @@
import org.joda.time.DateTime;
import org.joda.time.Interval;
import org.joda.time.chrono.ISOChronology;
+import org.joda.time.format.DateTimeFormatter;
+import org.joda.time.format.ISODateTimeFormat;
import javax.annotation.Nullable;
public final class Intervals
{
public static final Interval ETERNITY = utc(JodaUtils.MIN_INSTANT, JodaUtils.MAX_INSTANT);
public static final ImmutableList<Interval> ONLY_ETERNITY = ImmutableList.of(ETERNITY);
+ private static final DateTimeFormatter FAST_ISO_UTC_FORMATTER =
Review Comment:
Hi @kfaraz
Got quite a bit of time to kill, so I did a benchmark for the formatters...
Seems `DateTimes.ISO_DATE_TIME` is a bit slower than the current optimization because it tries to create `DateTime` unnecessary objects. `Intervals#utc` can work with `long` to create `Interval` directly, and hence performs faster.
On the topic of using other data formats, seems like we can look for `DateTimes.ISO_DATE_OPTIONAL_TIME` to improve the deserialization, but the improvement is quite small.
```
Benchmark (numValues) Mode Cnt Score Error Units
JodaIntervalDeserializationBenchmark.deserializeLegacy 20000 avgt 50 20.858 ± 0.035 ms/op
JodaIntervalDeserializationBenchmark.deserializeOptimized 20000 avgt 50 11.940 ± 0.009 ms/op
JodaIntervalDeserializationBenchmark.deserializeDateTime 20000 avgt 50 12.524 ± 0.025 ms/op
JodaIntervalDeserializationBenchmark.deserializeDateTimeOffset 20000 avgt 50 23.505 ± 0.050 ms/op
JodaIntervalDeserializationBenchmark.deserializeOptionalDateTime 20000 avgt 50 20.663 ± 0.111 ms/op
JodaIntervalDeserializationBenchmark.deserializeLegacyFallback 20000 avgt 50 19.035 ± 0.032 ms/op
JodaIntervalDeserializationBenchmark.deserializeOptimizedFallback 20000 avgt 50 33.782 ± 0.105 ms/op
JodaIntervalDeserializationBenchmark.deserializeDateTimeFallback 20000 avgt 50 36.642 ± 0.065 ms/op
JodaIntervalDeserializationBenchmark.deserializeDateTimeOffsetFallback 20000 avgt 50 22.067 ± 0.046 ms/op
JodaIntervalDeserializationBenchmark.deserializeOptionalDateTimeFallback 20000 avgt 50 18.861 ± 0.012 ms/op
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Optimize interval deserialization (druid)
Posted by "kfaraz (via GitHub)" <gi...@apache.org>.
kfaraz commented on code in PR #18477:
URL: https://github.com/apache/druid/pull/18477#discussion_r2321013789
##########
processing/src/main/java/org/apache/druid/java/util/common/Intervals.java:
##########
@@ -53,6 +57,46 @@ public static Interval of(String format, Object... formatArgs)
return of(StringUtils.format(format, formatArgs));
}
+ /**
+ * @return Null if cannot parse with optimized strategy, else return the Interval.
Review Comment:
Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Optimize interval deserialization (druid)
Posted by "GWphua (via GitHub)" <gi...@apache.org>.
GWphua commented on PR #18477:
URL: https://github.com/apache/druid/pull/18477#issuecomment-3257506148
@kfaraz I'm having trouble trying to use segmentId's `tryParse()`, since it deals with data of this shape: `"datasource_2015-01-02T00:00:00.000Z_2015-01-03T00:00:00.000Z_ver_0_1"`, but our Interval representation is `"2015-01-02T00:00:00.000Z/2015-01-03T00:00:00.000Z"`. This means we will require some manipulation of the String that's going into DataSegment. lmk if im missing anything!
I can change DataSegment's `interval` input to `String` type, and use `Intervals.fromString()` on it to give the performance boost, since that's the goal of this PR -- to optimize the deserialization time for `DataSegment`. If we figure out that most of Druid handles interval string representation of the compatible type format, we can consider using the optimized deserialization strategy instead.
I have previously used `Intervals.fromString()` to be the default deserializer in `JodaStuff`. Now will revert that change, and use deserialization for only DataSegments.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Optimize interval deserialization (druid)
Posted by "GWphua (via GitHub)" <gi...@apache.org>.
GWphua commented on PR #18477:
URL: https://github.com/apache/druid/pull/18477#issuecomment-3258971968
> @GWphua , what does the flamegraph look like after the change in this PR?
Hi @kfaraz, unfortunately I cannot get a flamegraph this week due to internal code freezes, that's why I am presenting performances via benchmarks as of now. I will also be on a break for a while, so you can expect to hear back from me regarding non-trivial code changes after i return -- probably early October.
Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Optimize interval deserialization (druid)
Posted by "kfaraz (via GitHub)" <gi...@apache.org>.
kfaraz commented on code in PR #18477:
URL: https://github.com/apache/druid/pull/18477#discussion_r2321015803
##########
processing/src/main/java/org/apache/druid/java/util/common/Intervals.java:
##########
@@ -53,6 +57,46 @@ public static Interval of(String format, Object... formatArgs)
return of(StringUtils.format(format, formatArgs));
}
+ /**
+ * @return Null if cannot parse with optimized strategy, else return the Interval.
Review Comment:
> I have listed all cases i can think of under the test file.
Please also list out these cases briefly in the javadoc of the new `Intervals.fromString()` method
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Optimize interval deserialization (druid)
Posted by "kfaraz (via GitHub)" <gi...@apache.org>.
kfaraz commented on code in PR #18477:
URL: https://github.com/apache/druid/pull/18477#discussion_r2321039287
##########
processing/src/test/java/org/apache/druid/java/util/common/IntervalsTest.java:
##########
@@ -79,6 +79,44 @@ public void testFindOverlappingInterval()
);
}
+ @Test
+ public void testValidIntervalStrings()
+ {
+ final String[] intervalStringRepresentations = new String[]{
+ // Tests that use does not fallback to Intervals.of()
+ // Zulu with millis
+ "2022-01-01T00:00:00.000Z/2022-01-02T00:00:00.000Z",
+ "2021-03-14T12:34:56.789Z/2021-03-15T12:34:56.789Z",
+
+ // Offset with colon
+ "2022-01-01T00:00:00.000+05:30/2022-01-01T01:00:00.000+05:30",
+ "2022-01-01T07:00:00.000-07:00/2022-01-01T08:00:00.000-07:00",
+
+ // Basic offset without colon
+ "2022-01-01T00:00:00.000+0530/2022-01-01T01:00:00.000+0530",
+
+ // Tests that fallback to Intervals.of()
+ // Zulu without millis
+ "2022-01-01T00:00:00Z/2022-01-02T00:00:00Z",
+ // Date-only
+ "2022-01-01/2022-01-02",
+ // start/period
+ "2022-01-01T00:00:00.000Z/P1D",
+ "2022-01-01T12:00:00Z/PT6H",
+ "2022-01-01T00:00:00Z/P2DT3H4M5S",
+ // period/end
+ "P1D/2022-01-02T00:00:00.000Z",
+ "PT6H/2022-01-01T18:00:00Z",
+ "P2DT3H4M5S/2022-01-03T03:04:05Z"
Review Comment:
I feel like these cases could also be deserialized in an optimal manner if we only know which category it falls into.
e.g. if we could quickly identify that the string has a `period/end` configuration, we could parse both parts separately.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Optimize interval deserialization (druid)
Posted by "kfaraz (via GitHub)" <gi...@apache.org>.
kfaraz commented on PR #18477:
URL: https://github.com/apache/druid/pull/18477#issuecomment-3254480919
> My opinion is that to create a new class named as SegmentInterval which inherits from current Interval class and then create a dedicated deserialization for this one to handle the segment interval format.
Ah, that would have been nice but unfortunately `joda.Interval`is a final class.
But we can still do this by passing in a `String` interval to `DataSegment` constructor
and then using `SegmentId.tryParse()` as it already uses the improved algorithm.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Optimize interval deserialization (druid)
Posted by "GWphua (via GitHub)" <gi...@apache.org>.
GWphua commented on code in PR #18477:
URL: https://github.com/apache/druid/pull/18477#discussion_r2321391094
##########
processing/src/main/java/org/apache/druid/java/util/common/Intervals.java:
##########
@@ -53,6 +57,46 @@ public static Interval of(String format, Object... formatArgs)
return of(StringUtils.format(format, formatArgs));
}
+ /**
+ * @return Null if cannot parse with optimized strategy, else return the Interval.
Review Comment:
Hi @kfaraz an update on `canDeserializeOptimally()`, I added a check to see if the interval contains a `Period`, and conduct a fail-fast. The results look much better.
```
Benchmark (numValues) Mode Cnt Score Error Units
JodaIntervalDeserializationBenchmark.deserializeLegacy 20000 avgt 5 21.115 ± 0.302 ms/op
JodaIntervalDeserializationBenchmark.deserializeLegacyFallback 20000 avgt 5 13.831 ± 0.373 ms/op
JodaIntervalDeserializationBenchmark.deserializeOptimized 20000 avgt 5 11.909 ± 0.343 ms/op
JodaIntervalDeserializationBenchmark.deserializeOptimizedFallback 20000 avgt 5 14.014 ± 0.477 ms/op
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Optimize interval deserialization (druid)
Posted by "GWphua (via GitHub)" <gi...@apache.org>.
GWphua commented on code in PR #18477:
URL: https://github.com/apache/druid/pull/18477#discussion_r2321411252
##########
processing/src/test/java/org/apache/druid/java/util/common/IntervalsTest.java:
##########
@@ -79,6 +79,44 @@ public void testFindOverlappingInterval()
);
}
+ @Test
+ public void testValidIntervalStrings()
+ {
+ final String[] intervalStringRepresentations = new String[]{
+ // Tests that use does not fallback to Intervals.of()
+ // Zulu with millis
+ "2022-01-01T00:00:00.000Z/2022-01-02T00:00:00.000Z",
+ "2021-03-14T12:34:56.789Z/2021-03-15T12:34:56.789Z",
+
+ // Offset with colon
+ "2022-01-01T00:00:00.000+05:30/2022-01-01T01:00:00.000+05:30",
+ "2022-01-01T07:00:00.000-07:00/2022-01-01T08:00:00.000-07:00",
+
+ // Basic offset without colon
+ "2022-01-01T00:00:00.000+0530/2022-01-01T01:00:00.000+0530",
+
+ // Tests that fallback to Intervals.of()
+ // Zulu without millis
+ "2022-01-01T00:00:00Z/2022-01-02T00:00:00Z",
+ // Date-only
+ "2022-01-01/2022-01-02",
+ // start/period
+ "2022-01-01T00:00:00.000Z/P1D",
+ "2022-01-01T12:00:00Z/PT6H",
+ "2022-01-01T00:00:00Z/P2DT3H4M5S",
+ // period/end
+ "P1D/2022-01-02T00:00:00.000Z",
+ "PT6H/2022-01-01T18:00:00Z",
+ "P2DT3H4M5S/2022-01-03T03:04:05Z"
Review Comment:
Hi @kfaraz
I attempted to make `tryOptimizedIntervalDeserialization` more inclusive (e.g. Work on Date-only, and Without Millis case), but this requires changing the `FAST_ISO_UTC_FORMATTER` DateTimeFormatter to a more flexible formatter. This causes performance to resemble the un-optimized versions.
These 2 formatters allow formats like `"2022-01-01/2022-01-02"` or `"2022-01-01T00:00:00Z/2022-01-02T00:00:00Z"`, but the performance is lacking...
```
private static final DateTimeFormatter FAST_ISO_UTC_FORMATTER =
ISODateTimeFormat.dateOptionalTimeParser()
.withOffsetParsed()
.withChronology(ISOChronology.getInstanceUTC());
Benchmark (numValues) Mode Cnt Score Error Units
JodaIntervalDeserializationBenchmark.deserializeLegacy 20000 avgt 2 21.243 ms/op
JodaIntervalDeserializationBenchmark.deserializeLegacyFallback 20000 avgt 2 15.217 ms/op
JodaIntervalDeserializationBenchmark.deserializeOptimized 20000 avgt 2 20.605 ms/op
JodaIntervalDeserializationBenchmark.deserializeOptimizedFallback 20000 avgt 2 31.051 ms/op
```
```
private static final DateTimeFormatter FAST_ISO_UTC_FORMATTER =
ISODateTimeFormat.dateTimeParser().withChronology(ISOChronology.getInstanceUTC());
Benchmark (numValues) Mode Cnt Score Error Units
JodaIntervalDeserializationBenchmark.deserializeLegacy 20000 avgt 3 21.502 ± 0.995 ms/op
JodaIntervalDeserializationBenchmark.deserializeLegacyFallback 20000 avgt 3 15.365 ± 1.227 ms/op
JodaIntervalDeserializationBenchmark.deserializeOptimized 20000 avgt 3 21.239 ± 1.543 ms/op
JodaIntervalDeserializationBenchmark.deserializeOptimizedFallback 20000 avgt 3 31.526 ± 3.083 ms/op
```
I think we can simply take advantage of knowing how our data will look like, and optimize the parse process for the most frequently-used data format.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Optimize interval deserialization (druid)
Posted by "GWphua (via GitHub)" <gi...@apache.org>.
GWphua commented on code in PR #18477:
URL: https://github.com/apache/druid/pull/18477#discussion_r2321423422
##########
processing/src/test/java/org/apache/druid/java/util/common/IntervalsTest.java:
##########
@@ -79,6 +79,44 @@ public void testFindOverlappingInterval()
);
}
+ @Test
+ public void testValidIntervalStrings()
+ {
+ final String[] intervalStringRepresentations = new String[]{
+ // Tests that use does not fallback to Intervals.of()
+ // Zulu with millis
+ "2022-01-01T00:00:00.000Z/2022-01-02T00:00:00.000Z",
+ "2021-03-14T12:34:56.789Z/2021-03-15T12:34:56.789Z",
+
+ // Offset with colon
+ "2022-01-01T00:00:00.000+05:30/2022-01-01T01:00:00.000+05:30",
+ "2022-01-01T07:00:00.000-07:00/2022-01-01T08:00:00.000-07:00",
+
+ // Basic offset without colon
+ "2022-01-01T00:00:00.000+0530/2022-01-01T01:00:00.000+0530",
+
+ // Tests that fallback to Intervals.of()
+ // Zulu without millis
+ "2022-01-01T00:00:00Z/2022-01-02T00:00:00Z",
+ // Date-only
+ "2022-01-01/2022-01-02",
+ // start/period
+ "2022-01-01T00:00:00.000Z/P1D",
+ "2022-01-01T12:00:00Z/PT6H",
+ "2022-01-01T00:00:00Z/P2DT3H4M5S",
+ // period/end
+ "P1D/2022-01-02T00:00:00.000Z",
+ "PT6H/2022-01-01T18:00:00Z",
+ "P2DT3H4M5S/2022-01-03T03:04:05Z"
Review Comment:
I think it is possible to check manually, but I feel this method feels abit too hacky, we might miss some DateTime cases:
For start/end string, we check
1. If is Date-only, append T00:00:000Z
2. If does not have millisecond part, append turn T00:00:00Z --> T00:00:00:000Z, and throw it into the formatter.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Optimize interval deserialization (druid)
Posted by "kfaraz (via GitHub)" <gi...@apache.org>.
kfaraz commented on code in PR #18477:
URL: https://github.com/apache/druid/pull/18477#discussion_r2320816342
##########
processing/src/main/java/org/apache/druid/java/util/common/Intervals.java:
##########
@@ -53,6 +57,46 @@ public static Interval of(String format, Object... formatArgs)
return of(StringUtils.format(format, formatArgs));
}
+ /**
+ * @return Null if cannot parse with optimized strategy, else return the Interval.
Review Comment:
Otherwise, the new deserialize becomes much slower than the legacy deserialize for fallback cases (13 vs 30).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Optimize interval deserialization (druid)
Posted by "GWphua (via GitHub)" <gi...@apache.org>.
GWphua commented on code in PR #18477:
URL: https://github.com/apache/druid/pull/18477#discussion_r2320967879
##########
processing/src/main/java/org/apache/druid/java/util/common/Intervals.java:
##########
@@ -53,6 +57,46 @@ public static Interval of(String format, Object... formatArgs)
return of(StringUtils.format(format, formatArgs));
}
+ /**
+ * @return Null if cannot parse with optimized strategy, else return the Interval.
Review Comment:
Hi @kfaraz, thanks for the quick review. I have incorporated your suggestions.
I have listed all cases i can think of under the test file.
```
// Zulu without millis
"2022-01-01T00:00:00Z/2022-01-02T00:00:00Z",
// Date-only
"2022-01-01/2022-01-02",
// start/period
"2022-01-01T12:00:00Z/PT6H",
// period/end
"P2DT3H4M5S/2022-01-01T00:00:00Z"
```
I will work more into short-circuit solutions. Will update this PR soon!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Optimize interval deserialization (druid)
Posted by "kfaraz (via GitHub)" <gi...@apache.org>.
kfaraz commented on PR #18477:
URL: https://github.com/apache/druid/pull/18477#issuecomment-3257554095
> since it deals with data of this shape: "datasource_2015-01-02T00:00:00.000Z_2015-01-03T00:00:00.000Z_ver_0_1", but our Interval representation is "2015-01-02T00:00:00.000Z/2015-01-03T00:00:00.000Z".
Thanks for checking, @GWphua !
We would need to add some new constructor to `SegmentId` to make it work but we might as well
just use the new method you have already added in this PR.
> I can change DataSegment's interval input to String type, and use Intervals.fromString() on it to give the performance boost, since that's the goal of this PR -- to optimize the deserialization time for DataSegment.
I have previously used Intervals.fromString() to be the default deserializer in JodaStuff. Now will revert that change, and use deserialization for only DataSegments.
Yeah, this makes sense. Let's proceed with this.
Please make this JsonCreator constructor private and add a separate constructor so that all the call sites need not change.
In the javadoc of `Intervals.fromString()`, please indicate that this method is to be used for
segment intervals only (at least for the time being).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Optimize interval deserialization (druid)
Posted by "GWphua (via GitHub)" <gi...@apache.org>.
GWphua commented on code in PR #18477:
URL: https://github.com/apache/druid/pull/18477#discussion_r2328558173
##########
processing/src/main/java/org/apache/druid/java/util/common/Intervals.java:
##########
@@ -53,6 +57,74 @@ public static Interval of(String format, Object... formatArgs)
return of(StringUtils.format(format, formatArgs));
}
+ /**
+ * A performance-optimized method for parsing a Joda-Time {@link Interval} from a string.
+ * This method is significantly faster than the standard {@link Intervals#of(String)} for the following
+ * group of offsets:
+ * <ol>
+ * <li>"2022-01-01T00:00:00.000Z/2022-01-02T00:00:00.000Z"</li>
+ * <li>"2022-01-01T00:00:00.000+05:30/2022-01-01T01:00:00.000+05:30"</li>
+ * <li>"2022-01-01T00:00:00.000+0530/2022-01-01T01:00:00.000+0530"</li>
+ * </ol>
+ * <p>
+ * If the input string does not match the format, it will fall back to the more flexible but
+ * slower {@link Intervals#of(String)} parser. If you are dealing with any Intervals format examples below,
+ * consider using {@link Intervals#of(String)} instead:
+ * <ol>
+ * <li>"2022-01-01T00:00:00Z/2022-01-02T00:00:00Z" (without millis)</li>
+ * <li>"2022-01-01/2022-01-02" (Date only)</li>
+ * <li>"2022-01-01T12:00:00.000Z/PT6H" (Periods in start / end)</li>
+ * </ol>
+ *
+ * Currently, this method is only used in {@link org.apache.druid.timeline.SegmentId}.
+ * Should you find additional places where the Interval format is guaranteed to be compatible with this method,
+ * feel free to open a PR.
Review Comment:
Hi @FrankChen021
While this method is only used in `DataSegment` now, should another place find this deserialization useful, we may need to move the method back to `Intervals` to avoid DRY.
Furthermore, I feel that this we should place a method that deals with interval parsing in `Intervals`. What do you think?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Optimize interval deserialization (druid)
Posted by "GWphua (via GitHub)" <gi...@apache.org>.
GWphua commented on code in PR #18477:
URL: https://github.com/apache/druid/pull/18477#discussion_r2328603280
##########
processing/src/main/java/org/apache/druid/java/util/common/Intervals.java:
##########
@@ -25,13 +25,17 @@
import org.joda.time.DateTime;
import org.joda.time.Interval;
import org.joda.time.chrono.ISOChronology;
+import org.joda.time.format.DateTimeFormatter;
+import org.joda.time.format.ISODateTimeFormat;
import javax.annotation.Nullable;
public final class Intervals
{
public static final Interval ETERNITY = utc(JodaUtils.MIN_INSTANT, JodaUtils.MAX_INSTANT);
public static final ImmutableList<Interval> ONLY_ETERNITY = ImmutableList.of(ETERNITY);
+ private static final DateTimeFormatter FAST_ISO_UTC_FORMATTER =
Review Comment:
Hi @kfaraz
Got quite a bit of time to kill, so I did a benchmark for the formatters...
Seems `DateTimes.ISO_DATE_TIME` is a bit slower than the current optimization because `ISO_DATE_TIME` tries to create `DateTime` unnecessary objects. The current implementation creates an `Interval` directly using `long` via`Intervals#utc` and hence performs faster.
On the topic of using other data formats, seems like we can look for `DateTimes.ISO_DATE_OPTIONAL_TIME` to improve the deserialization, but the improvement is quite small.
```
Benchmark (numValues) Mode Cnt Score Error Units
JodaIntervalDeserializationBenchmark.deserializeLegacy 20000 avgt 50 20.858 ± 0.035 ms/op
JodaIntervalDeserializationBenchmark.deserializeOptimized 20000 avgt 50 11.940 ± 0.009 ms/op
JodaIntervalDeserializationBenchmark.deserializeDateTime 20000 avgt 50 12.524 ± 0.025 ms/op
JodaIntervalDeserializationBenchmark.deserializeDateTimeOffset 20000 avgt 50 23.505 ± 0.050 ms/op
JodaIntervalDeserializationBenchmark.deserializeOptionalDateTime 20000 avgt 50 20.663 ± 0.111 ms/op
JodaIntervalDeserializationBenchmark.deserializeLegacyFallback 20000 avgt 50 19.035 ± 0.032 ms/op
JodaIntervalDeserializationBenchmark.deserializeOptimizedFallback 20000 avgt 50 33.782 ± 0.105 ms/op
JodaIntervalDeserializationBenchmark.deserializeDateTimeFallback 20000 avgt 50 36.642 ± 0.065 ms/op
JodaIntervalDeserializationBenchmark.deserializeDateTimeOffsetFallback 20000 avgt 50 22.067 ± 0.046 ms/op
JodaIntervalDeserializationBenchmark.deserializeOptionalDateTimeFallback 20000 avgt 50 18.861 ± 0.012 ms/op
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Optimize interval deserialization (druid)
Posted by "GWphua (via GitHub)" <gi...@apache.org>.
GWphua commented on code in PR #18477:
URL: https://github.com/apache/druid/pull/18477#discussion_r2321525496
##########
processing/src/test/java/org/apache/druid/java/util/common/IntervalsTest.java:
##########
@@ -79,6 +79,44 @@ public void testFindOverlappingInterval()
);
}
+ @Test
+ public void testValidIntervalStrings()
+ {
+ final String[] intervalStringRepresentations = new String[]{
+ // Tests that use does not fallback to Intervals.of()
+ // Zulu with millis
+ "2022-01-01T00:00:00.000Z/2022-01-02T00:00:00.000Z",
+ "2021-03-14T12:34:56.789Z/2021-03-15T12:34:56.789Z",
+
+ // Offset with colon
+ "2022-01-01T00:00:00.000+05:30/2022-01-01T01:00:00.000+05:30",
+ "2022-01-01T07:00:00.000-07:00/2022-01-01T08:00:00.000-07:00",
+
+ // Basic offset without colon
+ "2022-01-01T00:00:00.000+0530/2022-01-01T01:00:00.000+0530",
+
+ // Tests that fallback to Intervals.of()
+ // Zulu without millis
+ "2022-01-01T00:00:00Z/2022-01-02T00:00:00Z",
+ // Date-only
+ "2022-01-01/2022-01-02",
+ // start/period
+ "2022-01-01T00:00:00.000Z/P1D",
+ "2022-01-01T12:00:00Z/PT6H",
+ "2022-01-01T00:00:00Z/P2DT3H4M5S",
+ // period/end
+ "P1D/2022-01-02T00:00:00.000Z",
+ "PT6H/2022-01-01T18:00:00Z",
+ "P2DT3H4M5S/2022-01-03T03:04:05Z"
Review Comment:
> if we could quickly identify that the string has a period/end configuration, we could parse both parts separately.
Regarding this, I have this solution in mind:
1. Suppose start is a timestamp, end is a period string.
2. Parse the period string to a long.
3. `endMillis` = period in long + `startMillis`
This method does not work all the time though. We may not get the correct number for leap years, or even months. We will need to calculate start/end period with awareness of these semantics, which is more complex.
I feel we can stick with knowing that we are dealing with the most common data format (`"YYYY-MM-DD'T'HH:mm:ss.SSSZ/YYYY-MM-DD'T'HH:mm:ss.SSSZ"`) for this method, and use `Interval.of(String)` for other formats?
##########
processing/src/test/java/org/apache/druid/java/util/common/IntervalsTest.java:
##########
@@ -79,6 +79,44 @@ public void testFindOverlappingInterval()
);
}
+ @Test
+ public void testValidIntervalStrings()
+ {
+ final String[] intervalStringRepresentations = new String[]{
+ // Tests that use does not fallback to Intervals.of()
+ // Zulu with millis
+ "2022-01-01T00:00:00.000Z/2022-01-02T00:00:00.000Z",
+ "2021-03-14T12:34:56.789Z/2021-03-15T12:34:56.789Z",
+
+ // Offset with colon
+ "2022-01-01T00:00:00.000+05:30/2022-01-01T01:00:00.000+05:30",
+ "2022-01-01T07:00:00.000-07:00/2022-01-01T08:00:00.000-07:00",
+
+ // Basic offset without colon
+ "2022-01-01T00:00:00.000+0530/2022-01-01T01:00:00.000+0530",
+
+ // Tests that fallback to Intervals.of()
+ // Zulu without millis
+ "2022-01-01T00:00:00Z/2022-01-02T00:00:00Z",
+ // Date-only
+ "2022-01-01/2022-01-02",
+ // start/period
+ "2022-01-01T00:00:00.000Z/P1D",
+ "2022-01-01T12:00:00Z/PT6H",
+ "2022-01-01T00:00:00Z/P2DT3H4M5S",
+ // period/end
+ "P1D/2022-01-02T00:00:00.000Z",
+ "PT6H/2022-01-01T18:00:00Z",
+ "P2DT3H4M5S/2022-01-03T03:04:05Z"
Review Comment:
> if we could quickly identify that the string has a period/end configuration, we could parse both parts separately.
Regarding this, I have this solution in mind:
1. Suppose start is a timestamp, end is a period string.
2. Parse the period string to a long.
3. `endMillis` = period in long + `startMillis`
This method does not work all the time though. We may not get the correct number for leap years, or even months. We will need to calculate start/end period with awareness of these semantics, which is more complex.
I feel we can stick with knowing that we are dealing with the most common data format (`"YYYY-MM-DD'T'HH:mm:ss.SSSZ/YYYY-MM-DD'T'HH:mm:ss.SSSZ"`) for this method, and use `Interval.of(String)` for other formats?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Optimize interval deserialization (druid)
Posted by "kfaraz (via GitHub)" <gi...@apache.org>.
kfaraz commented on code in PR #18477:
URL: https://github.com/apache/druid/pull/18477#discussion_r2320816714
##########
benchmarks/src/test/java/org/apache/druid/benchmark/JodaIntervalDeserialization.java:
##########
@@ -0,0 +1,152 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.benchmark;
+
+import com.fasterxml.jackson.core.JsonParser;
+import com.fasterxml.jackson.databind.DeserializationContext;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.databind.deser.std.StdDeserializer;
+import com.fasterxml.jackson.databind.module.SimpleModule;
+import org.apache.druid.jackson.DefaultObjectMapper;
+import org.apache.druid.java.util.common.Intervals;
+import org.joda.time.Interval;
+import org.openjdk.jmh.annotations.Benchmark;
+import org.openjdk.jmh.annotations.BenchmarkMode;
+import org.openjdk.jmh.annotations.Fork;
+import org.openjdk.jmh.annotations.Measurement;
+import org.openjdk.jmh.annotations.Mode;
+import org.openjdk.jmh.annotations.OutputTimeUnit;
+import org.openjdk.jmh.annotations.Param;
+import org.openjdk.jmh.annotations.Scope;
+import org.openjdk.jmh.annotations.Setup;
+import org.openjdk.jmh.annotations.State;
+import org.openjdk.jmh.annotations.Warmup;
+import org.openjdk.jmh.infra.Blackhole;
+import org.openjdk.jmh.runner.Runner;
+import org.openjdk.jmh.runner.RunnerException;
+import org.openjdk.jmh.runner.options.Options;
+import org.openjdk.jmh.runner.options.OptionsBuilder;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.TimeUnit;
+
+@State(Scope.Benchmark)
+@Fork(value = 1)
+@Warmup(iterations = 3)
+@Measurement(iterations = 5)
+@BenchmarkMode(Mode.AverageTime)
+@OutputTimeUnit(TimeUnit.MILLISECONDS)
+public class JodaIntervalDeserialization
Review Comment:
```suggestion
public class JodaIntervalDeserializationBenchmark
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Optimize interval deserialization (druid)
Posted by "GWphua (via GitHub)" <gi...@apache.org>.
GWphua commented on code in PR #18477:
URL: https://github.com/apache/druid/pull/18477#discussion_r2321555536
##########
processing/src/main/java/org/apache/druid/java/util/common/Intervals.java:
##########
@@ -53,6 +57,46 @@ public static Interval of(String format, Object... formatArgs)
return of(StringUtils.format(format, formatArgs));
}
+ /**
+ * @return Null if cannot parse with optimized strategy, else return the Interval.
Review Comment:
I have updated the cases on the Javadocs of `Intervals.fromString()`, please take a look again. 😄
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Optimize interval deserialization (druid)
Posted by "GWphua (via GitHub)" <gi...@apache.org>.
GWphua commented on code in PR #18477:
URL: https://github.com/apache/druid/pull/18477#discussion_r2320967879
##########
processing/src/main/java/org/apache/druid/java/util/common/Intervals.java:
##########
@@ -53,6 +57,46 @@ public static Interval of(String format, Object... formatArgs)
return of(StringUtils.format(format, formatArgs));
}
+ /**
+ * @return Null if cannot parse with optimized strategy, else return the Interval.
Review Comment:
Hi @kfaraz, thanks for the quick review. I have incorporated your suggestions.
Regarding the short circuit, I agree we can add a short-circuit for working with Periods , though it is hard to find an exhaustive way to cover all fallback cases.
I have listed all cases i can think of under the test file. Here's a TLDR of what's considered fallback.
```
// Zulu without millis
"2022-01-01T00:00:00Z/2022-01-02T00:00:00Z",
// Date-only
"2022-01-01/2022-01-02",
// start/period
"2022-01-01T12:00:00Z/PT6H",
// period/end
"P2DT3H4M5S/2022-01-01T00:00:00Z"
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Optimize interval deserialization (druid)
Posted by "GWphua (via GitHub)" <gi...@apache.org>.
GWphua commented on code in PR #18477:
URL: https://github.com/apache/druid/pull/18477#discussion_r2321411252
##########
processing/src/test/java/org/apache/druid/java/util/common/IntervalsTest.java:
##########
@@ -79,6 +79,44 @@ public void testFindOverlappingInterval()
);
}
+ @Test
+ public void testValidIntervalStrings()
+ {
+ final String[] intervalStringRepresentations = new String[]{
+ // Tests that use does not fallback to Intervals.of()
+ // Zulu with millis
+ "2022-01-01T00:00:00.000Z/2022-01-02T00:00:00.000Z",
+ "2021-03-14T12:34:56.789Z/2021-03-15T12:34:56.789Z",
+
+ // Offset with colon
+ "2022-01-01T00:00:00.000+05:30/2022-01-01T01:00:00.000+05:30",
+ "2022-01-01T07:00:00.000-07:00/2022-01-01T08:00:00.000-07:00",
+
+ // Basic offset without colon
+ "2022-01-01T00:00:00.000+0530/2022-01-01T01:00:00.000+0530",
+
+ // Tests that fallback to Intervals.of()
+ // Zulu without millis
+ "2022-01-01T00:00:00Z/2022-01-02T00:00:00Z",
+ // Date-only
+ "2022-01-01/2022-01-02",
+ // start/period
+ "2022-01-01T00:00:00.000Z/P1D",
+ "2022-01-01T12:00:00Z/PT6H",
+ "2022-01-01T00:00:00Z/P2DT3H4M5S",
+ // period/end
+ "P1D/2022-01-02T00:00:00.000Z",
+ "PT6H/2022-01-01T18:00:00Z",
+ "P2DT3H4M5S/2022-01-03T03:04:05Z"
Review Comment:
Hi @kfaraz
I attempted to make `tryOptimizedIntervalDeserialization` more inclusive (e.g. Work on Date-only, and Without Millis case), but this requires changing the `FAST_ISO_UTC_FORMATTER` DateTimeFormatter to a more flexible formatter. This causes performance to resemble the un-optimized versions.
These 2 formatters allow formats like `"2022-01-01/2022-01-02"` or `"2022-01-01T00:00:00Z/2022-01-02T00:00:00Z"`, but the performance is lacking...
```
private static final DateTimeFormatter FAST_ISO_UTC_FORMATTER =
ISODateTimeFormat.dateOptionalTimeParser()
.withOffsetParsed()
.withChronology(ISOChronology.getInstanceUTC());
Benchmark (numValues) Mode Cnt Score Error Units
JodaIntervalDeserializationBenchmark.deserializeLegacy 20000 avgt 2 21.243 ms/op
JodaIntervalDeserializationBenchmark.deserializeLegacyFallback 20000 avgt 2 15.217 ms/op
JodaIntervalDeserializationBenchmark.deserializeOptimized 20000 avgt 2 20.605 ms/op
JodaIntervalDeserializationBenchmark.deserializeOptimizedFallback 20000 avgt 2 31.051 ms/op
```
```
private static final DateTimeFormatter FAST_ISO_UTC_FORMATTER =
ISODateTimeFormat.dateTimeParser().withChronology(ISOChronology.getInstanceUTC());
Benchmark (numValues) Mode Cnt Score Error Units
JodaIntervalDeserializationBenchmark.deserializeLegacy 20000 avgt 3 21.502 ± 0.995 ms/op
JodaIntervalDeserializationBenchmark.deserializeLegacyFallback 20000 avgt 3 15.365 ± 1.227 ms/op
JodaIntervalDeserializationBenchmark.deserializeOptimized 20000 avgt 3 21.239 ± 1.543 ms/op
JodaIntervalDeserializationBenchmark.deserializeOptimizedFallback 20000 avgt 3 31.526 ± 3.083 ms/op
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Optimize interval deserialization (druid)
Posted by "GWphua (via GitHub)" <gi...@apache.org>.
GWphua commented on code in PR #18477:
URL: https://github.com/apache/druid/pull/18477#discussion_r2321525496
##########
processing/src/test/java/org/apache/druid/java/util/common/IntervalsTest.java:
##########
@@ -79,6 +79,44 @@ public void testFindOverlappingInterval()
);
}
+ @Test
+ public void testValidIntervalStrings()
+ {
+ final String[] intervalStringRepresentations = new String[]{
+ // Tests that use does not fallback to Intervals.of()
+ // Zulu with millis
+ "2022-01-01T00:00:00.000Z/2022-01-02T00:00:00.000Z",
+ "2021-03-14T12:34:56.789Z/2021-03-15T12:34:56.789Z",
+
+ // Offset with colon
+ "2022-01-01T00:00:00.000+05:30/2022-01-01T01:00:00.000+05:30",
+ "2022-01-01T07:00:00.000-07:00/2022-01-01T08:00:00.000-07:00",
+
+ // Basic offset without colon
+ "2022-01-01T00:00:00.000+0530/2022-01-01T01:00:00.000+0530",
+
+ // Tests that fallback to Intervals.of()
+ // Zulu without millis
+ "2022-01-01T00:00:00Z/2022-01-02T00:00:00Z",
+ // Date-only
+ "2022-01-01/2022-01-02",
+ // start/period
+ "2022-01-01T00:00:00.000Z/P1D",
+ "2022-01-01T12:00:00Z/PT6H",
+ "2022-01-01T00:00:00Z/P2DT3H4M5S",
+ // period/end
+ "P1D/2022-01-02T00:00:00.000Z",
+ "PT6H/2022-01-01T18:00:00Z",
+ "P2DT3H4M5S/2022-01-03T03:04:05Z"
Review Comment:
> if we could quickly identify that the string has a period/end configuration, we could parse both parts separately.
Regarding this, I have this solution in mind:
1. Suppose start is a timestamp, end is a period string.
2. Parse the period string to a long.
3. `endMillis` = period in long + `startMillis`
This method does not work all the time though. We may not get the correct number for leap years, or even months. We will need to calculate start/end period with awareness of these semantics, which is more complex. I feel we can stick with knowing that we are dealing with the most common data format (`"YYYY-MM-DD'T'HH:mm:ss.SSSZ/YYYY-MM-DD'T'HH:mm:ss.SSSZ"`) for this method, and use `Interval.of(String)` for other formats?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Optimize interval deserialization (druid)
Posted by "kfaraz (via GitHub)" <gi...@apache.org>.
kfaraz commented on code in PR #18477:
URL: https://github.com/apache/druid/pull/18477#discussion_r2325372696
##########
processing/src/main/java/org/apache/druid/java/util/common/Intervals.java:
##########
@@ -53,6 +57,74 @@ public static Interval of(String format, Object... formatArgs)
return of(StringUtils.format(format, formatArgs));
}
+ /**
+ * A performance-optimized method for parsing a Joda-Time {@link Interval} from a string.
+ * This method is significantly faster than the standard {@link Intervals#of(String)} for the following
+ * group of offsets:
+ * <ol>
+ * <li>"2022-01-01T00:00:00.000Z/2022-01-02T00:00:00.000Z"</li>
+ * <li>"2022-01-01T00:00:00.000+05:30/2022-01-01T01:00:00.000+05:30"</li>
+ * <li>"2022-01-01T00:00:00.000+0530/2022-01-01T01:00:00.000+0530"</li>
+ * </ol>
+ * <p>
+ * If the input string does not match the format, it will fall back to the more flexible but
+ * slower {@link Intervals#of(String)} parser. If you are dealing with any Intervals format examples below,
+ * consider using {@link Intervals#of(String)} instead:
+ * <ol>
+ * <li>"2022-01-01T00:00:00Z/2022-01-02T00:00:00Z" (without millis)</li>
+ * <li>"2022-01-01/2022-01-02" (Date only)</li>
+ * <li>"2022-01-01T12:00:00.000Z/PT6H" (Periods in start / end)</li>
+ * </ol>
+ *
+ * Currently, this method is only used in {@link org.apache.druid.timeline.SegmentId}.
+ * Should you find additional places where the Interval format is guaranteed to be compatible with this method,
+ * feel free to open a PR.
Review Comment:
Nit: We can leave out these lines.
##########
processing/src/main/java/org/apache/druid/java/util/common/Intervals.java:
##########
@@ -25,13 +25,17 @@
import org.joda.time.DateTime;
import org.joda.time.Interval;
import org.joda.time.chrono.ISOChronology;
+import org.joda.time.format.DateTimeFormatter;
+import org.joda.time.format.ISODateTimeFormat;
import javax.annotation.Nullable;
public final class Intervals
{
public static final Interval ETERNITY = utc(JodaUtils.MIN_INSTANT, JodaUtils.MAX_INSTANT);
public static final ImmutableList<Interval> ONLY_ETERNITY = ImmutableList.of(ETERNITY);
+ private static final DateTimeFormatter FAST_ISO_UTC_FORMATTER =
Review Comment:
Maybe also check out the other two constant formatters in that class. See if they can help improve performance of the new method for other date formats. But this can be done in a follow up PR.
##########
processing/src/main/java/org/apache/druid/java/util/common/Intervals.java:
##########
@@ -25,13 +25,17 @@
import org.joda.time.DateTime;
import org.joda.time.Interval;
import org.joda.time.chrono.ISOChronology;
+import org.joda.time.format.DateTimeFormatter;
+import org.joda.time.format.ISODateTimeFormat;
import javax.annotation.Nullable;
public final class Intervals
{
public static final Interval ETERNITY = utc(JodaUtils.MIN_INSTANT, JodaUtils.MAX_INSTANT);
public static final ImmutableList<Interval> ONLY_ETERNITY = ImmutableList.of(ETERNITY);
+ private static final DateTimeFormatter FAST_ISO_UTC_FORMATTER =
Review Comment:
I think you can use `DateTimes.ISO_DATE_TIME` instead of this since it uses the same formatter internally.
##########
processing/src/main/java/org/apache/druid/java/util/common/Intervals.java:
##########
@@ -53,6 +57,74 @@ public static Interval of(String format, Object... formatArgs)
return of(StringUtils.format(format, formatArgs));
}
+ /**
+ * A performance-optimized method for parsing a Joda-Time {@link Interval} from a string.
+ * This method is significantly faster than the standard {@link Intervals#of(String)} for the following
+ * group of offsets:
+ * <ol>
+ * <li>"2022-01-01T00:00:00.000Z/2022-01-02T00:00:00.000Z"</li>
+ * <li>"2022-01-01T00:00:00.000+05:30/2022-01-01T01:00:00.000+05:30"</li>
+ * <li>"2022-01-01T00:00:00.000+0530/2022-01-01T01:00:00.000+0530"</li>
+ * </ol>
+ * <p>
+ * If the input string does not match the format, it will fall back to the more flexible but
+ * slower {@link Intervals#of(String)} parser. If you are dealing with any Intervals format examples below,
+ * consider using {@link Intervals#of(String)} instead:
+ * <ol>
+ * <li>"2022-01-01T00:00:00Z/2022-01-02T00:00:00Z" (without millis)</li>
+ * <li>"2022-01-01/2022-01-02" (Date only)</li>
+ * <li>"2022-01-01T12:00:00.000Z/PT6H" (Periods in start / end)</li>
+ * </ol>
+ *
+ * Currently, this method is only used in {@link org.apache.druid.timeline.SegmentId}.
+ * Should you find additional places where the Interval format is guaranteed to be compatible with this method,
+ * feel free to open a PR.
+ */
+ public static Interval fromString(String string)
+ {
+ Interval interval = null;
+ if (canDeserializeIntervalOptimallyFromString(string)) {
+ interval = tryOptimizedIntervalDeserialization(string);
+ }
+
+ if (interval == null) {
+ return Intervals.of(string);
+ } else {
+ return interval;
+ }
Review Comment:
Nit:
```suggestion
return interval == null ? Intervals.of(string) : interval;
```
##########
processing/src/main/java/org/apache/druid/timeline/DataSegment.java:
##########
@@ -196,6 +197,37 @@ public DataSegment(
@JsonProperty("size") long size,
@JacksonInject PruneSpecsHolder pruneSpecsHolder
)
+ {
+ this(
Review Comment:
Please add a comment here mentioning that we take `interval` input as String so that we can deserialize it optimally.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Optimize interval deserialization (druid)
Posted by "GWphua (via GitHub)" <gi...@apache.org>.
GWphua commented on PR #18477:
URL: https://github.com/apache/druid/pull/18477#issuecomment-3409727800
@kfaraz
Here's the flame graph after the changes:
<img width="3456" height="1916" alt="image" src="https://github.com/user-attachments/assets/743ededa-d043-4fbc-b390-d8a9487d48bf" />
Now, the initialization of Interval is roughly equal to initialization of DataSegment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Optimize interval deserialization (druid)
Posted by "GWphua (via GitHub)" <gi...@apache.org>.
GWphua commented on code in PR #18477:
URL: https://github.com/apache/druid/pull/18477#discussion_r2320967879
##########
processing/src/main/java/org/apache/druid/java/util/common/Intervals.java:
##########
@@ -53,6 +57,46 @@ public static Interval of(String format, Object... formatArgs)
return of(StringUtils.format(format, formatArgs));
}
+ /**
+ * @return Null if cannot parse with optimized strategy, else return the Interval.
Review Comment:
Hi @kfaraz, thanks for the quick review. I have incorporated your suggestions.
Regarding the short circuit, I agree we can add a short-circuit, though it is hard to find an exhaustive way to test.
I have listed all cases i can think of under the test file. Here's a TLDR of what's considered fallback.
```
// Zulu without millis
"2022-01-01T00:00:00Z/2022-01-02T00:00:00Z",
// Date-only
"2022-01-01/2022-01-02",
// start/period
"2022-01-01T12:00:00Z/PT6H",
// period/end
"P2DT3H4M5S/2022-01-01T00:00:00Z"
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Optimize interval deserialization (druid)
Posted by "GWphua (via GitHub)" <gi...@apache.org>.
GWphua commented on code in PR #18477:
URL: https://github.com/apache/druid/pull/18477#discussion_r2328603280
##########
processing/src/main/java/org/apache/druid/java/util/common/Intervals.java:
##########
@@ -25,13 +25,17 @@
import org.joda.time.DateTime;
import org.joda.time.Interval;
import org.joda.time.chrono.ISOChronology;
+import org.joda.time.format.DateTimeFormatter;
+import org.joda.time.format.ISODateTimeFormat;
import javax.annotation.Nullable;
public final class Intervals
{
public static final Interval ETERNITY = utc(JodaUtils.MIN_INSTANT, JodaUtils.MAX_INSTANT);
public static final ImmutableList<Interval> ONLY_ETERNITY = ImmutableList.of(ETERNITY);
+ private static final DateTimeFormatter FAST_ISO_UTC_FORMATTER =
Review Comment:
Hi @kfaraz
Got quite a bit of time to kill, so I did a benchmark for the formatters...
Seems `DateTimes.ISO_DATE_TIME` is a bit slower than the current optimization because `ISO_DATE_TIME` tries to create 2 unnecessary `DateTime` objects, before creating an `Interval` object. The current implementation creates an `Interval` directly using `long` via`Intervals#utc` and hence performs faster.
On the topic of using other data formats, seems like we can look for `DateTimes.ISO_DATE_OPTIONAL_TIME` to improve the deserialization, but the improvement is quite small.
```
Benchmark (numValues) Mode Cnt Score Error Units
JodaIntervalDeserializationBenchmark.deserializeLegacy 20000 avgt 50 20.858 ± 0.035 ms/op
JodaIntervalDeserializationBenchmark.deserializeOptimized 20000 avgt 50 11.940 ± 0.009 ms/op
JodaIntervalDeserializationBenchmark.deserializeDateTime 20000 avgt 50 12.524 ± 0.025 ms/op
JodaIntervalDeserializationBenchmark.deserializeDateTimeOffset 20000 avgt 50 23.505 ± 0.050 ms/op
JodaIntervalDeserializationBenchmark.deserializeOptionalDateTime 20000 avgt 50 20.663 ± 0.111 ms/op
JodaIntervalDeserializationBenchmark.deserializeLegacyFallback 20000 avgt 50 19.035 ± 0.032 ms/op
JodaIntervalDeserializationBenchmark.deserializeOptimizedFallback 20000 avgt 50 33.782 ± 0.105 ms/op
JodaIntervalDeserializationBenchmark.deserializeDateTimeFallback 20000 avgt 50 36.642 ± 0.065 ms/op
JodaIntervalDeserializationBenchmark.deserializeDateTimeOffsetFallback 20000 avgt 50 22.067 ± 0.046 ms/op
JodaIntervalDeserializationBenchmark.deserializeOptionalDateTimeFallback 20000 avgt 50 18.861 ± 0.012 ms/op
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Optimize interval deserialization (druid)
Posted by "GWphua (via GitHub)" <gi...@apache.org>.
GWphua commented on code in PR #18477:
URL: https://github.com/apache/druid/pull/18477#discussion_r2328558173
##########
processing/src/main/java/org/apache/druid/java/util/common/Intervals.java:
##########
@@ -53,6 +57,74 @@ public static Interval of(String format, Object... formatArgs)
return of(StringUtils.format(format, formatArgs));
}
+ /**
+ * A performance-optimized method for parsing a Joda-Time {@link Interval} from a string.
+ * This method is significantly faster than the standard {@link Intervals#of(String)} for the following
+ * group of offsets:
+ * <ol>
+ * <li>"2022-01-01T00:00:00.000Z/2022-01-02T00:00:00.000Z"</li>
+ * <li>"2022-01-01T00:00:00.000+05:30/2022-01-01T01:00:00.000+05:30"</li>
+ * <li>"2022-01-01T00:00:00.000+0530/2022-01-01T01:00:00.000+0530"</li>
+ * </ol>
+ * <p>
+ * If the input string does not match the format, it will fall back to the more flexible but
+ * slower {@link Intervals#of(String)} parser. If you are dealing with any Intervals format examples below,
+ * consider using {@link Intervals#of(String)} instead:
+ * <ol>
+ * <li>"2022-01-01T00:00:00Z/2022-01-02T00:00:00Z" (without millis)</li>
+ * <li>"2022-01-01/2022-01-02" (Date only)</li>
+ * <li>"2022-01-01T12:00:00.000Z/PT6H" (Periods in start / end)</li>
+ * </ol>
+ *
+ * Currently, this method is only used in {@link org.apache.druid.timeline.SegmentId}.
+ * Should you find additional places where the Interval format is guaranteed to be compatible with this method,
+ * feel free to open a PR.
Review Comment:
Hi @FrankChen021
While this method is only used in `DataSegment` now, should another place find this deserialization useful, we may need to move the method back to `Intervals`.
Furthermore, I feel that this we should place a method that deals with interval parsing in `Intervals`. What do you think?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Optimize interval deserialization (druid)
Posted by "kfaraz (via GitHub)" <gi...@apache.org>.
kfaraz merged PR #18477:
URL: https://github.com/apache/druid/pull/18477
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Optimize interval deserialization (druid)
Posted by "FrankChen021 (via GitHub)" <gi...@apache.org>.
FrankChen021 commented on code in PR #18477:
URL: https://github.com/apache/druid/pull/18477#discussion_r2326702918
##########
processing/src/main/java/org/apache/druid/java/util/common/Intervals.java:
##########
@@ -53,6 +57,74 @@ public static Interval of(String format, Object... formatArgs)
return of(StringUtils.format(format, formatArgs));
}
+ /**
+ * A performance-optimized method for parsing a Joda-Time {@link Interval} from a string.
+ * This method is significantly faster than the standard {@link Intervals#of(String)} for the following
+ * group of offsets:
+ * <ol>
+ * <li>"2022-01-01T00:00:00.000Z/2022-01-02T00:00:00.000Z"</li>
+ * <li>"2022-01-01T00:00:00.000+05:30/2022-01-01T01:00:00.000+05:30"</li>
+ * <li>"2022-01-01T00:00:00.000+0530/2022-01-01T01:00:00.000+0530"</li>
+ * </ol>
+ * <p>
+ * If the input string does not match the format, it will fall back to the more flexible but
+ * slower {@link Intervals#of(String)} parser. If you are dealing with any Intervals format examples below,
+ * consider using {@link Intervals#of(String)} instead:
+ * <ol>
+ * <li>"2022-01-01T00:00:00Z/2022-01-02T00:00:00Z" (without millis)</li>
+ * <li>"2022-01-01/2022-01-02" (Date only)</li>
+ * <li>"2022-01-01T12:00:00.000Z/PT6H" (Periods in start / end)</li>
+ * </ol>
+ *
+ * Currently, this method is only used in {@link org.apache.druid.timeline.SegmentId}.
+ * Should you find additional places where the Interval format is guaranteed to be compatible with this method,
+ * feel free to open a PR.
Review Comment:
Since this method is an optimization and used by the DataSegment only, my opinion is to move this method from the Interval class to the DataSegment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Optimize interval deserialization (druid)
Posted by "kfaraz (via GitHub)" <gi...@apache.org>.
kfaraz commented on PR #18477:
URL: https://github.com/apache/druid/pull/18477#issuecomment-3260246924
> Hi @kfaraz, unfortunately I cannot get a flamegraph this week due to internal code freezes, that's why I am presenting performances via benchmarks as of now. I will also be on a break for a while, so you can expect to hear back from me regarding non-trivial code changes after i return -- probably early October.
Sure, @GWphua , thanks for the update! 🙂
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Optimize interval deserialization (druid)
Posted by "kfaraz (via GitHub)" <gi...@apache.org>.
kfaraz commented on code in PR #18477:
URL: https://github.com/apache/druid/pull/18477#discussion_r2328680357
##########
processing/src/main/java/org/apache/druid/java/util/common/Intervals.java:
##########
@@ -25,13 +25,17 @@
import org.joda.time.DateTime;
import org.joda.time.Interval;
import org.joda.time.chrono.ISOChronology;
+import org.joda.time.format.DateTimeFormatter;
+import org.joda.time.format.ISODateTimeFormat;
import javax.annotation.Nullable;
public final class Intervals
{
public static final Interval ETERNITY = utc(JodaUtils.MIN_INSTANT, JodaUtils.MAX_INSTANT);
public static final ImmutableList<Interval> ONLY_ETERNITY = ImmutableList.of(ETERNITY);
+ private static final DateTimeFormatter FAST_ISO_UTC_FORMATTER =
Review Comment:
Thanks a lot for trying these out, @GWphua ! 🙂
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Optimize interval deserialization (druid)
Posted by "GWphua (via GitHub)" <gi...@apache.org>.
GWphua commented on PR #18477:
URL: https://github.com/apache/druid/pull/18477#issuecomment-3253036874
Hi @kfaraz
Since the benchmark samples involving period show a similar result now, I have replaced the benchmark with new fallback samples, all of which uses non-period in start/end (e.g. without millis specified)
When calling deserialization with the optimized method, we should try our best to not use intervals of this format, else risking the below performance:
1. `"2022-01-01T00:00:00Z/2022-01-02T00:00:00Z"`
2. `"2022-01-01/2022-01-02"`
Benchmark results are as shown:
```
Benchmark (numValues) Mode Cnt Score Error Units
JodaIntervalDeserializationBenchmark.deserializeLegacyFallback 20000 avgt 19.085 ms/op
JodaIntervalDeserializationBenchmark.deserializeOptimizedFallback 20000 avgt 33.561 ms/op
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Optimize interval deserialization (druid)
Posted by "kfaraz (via GitHub)" <gi...@apache.org>.
kfaraz commented on code in PR #18477:
URL: https://github.com/apache/druid/pull/18477#discussion_r2320811376
##########
processing/src/main/java/org/apache/druid/java/util/common/Intervals.java:
##########
@@ -53,6 +57,46 @@ public static Interval of(String format, Object... formatArgs)
return of(StringUtils.format(format, formatArgs));
}
+ /**
+ * @return Null if cannot parse with optimized strategy, else return the Interval.
Review Comment:
For what kind of interval strings does this method return null?
Is there a simpler way to short circuit in those cases rather than trying the parse unnecessarily?\
Maybe a `canDeserializeOptimally()` method which checks it upfront?
##########
processing/src/main/java/org/apache/druid/java/util/common/Intervals.java:
##########
@@ -53,6 +57,46 @@ public static Interval of(String format, Object... formatArgs)
return of(StringUtils.format(format, formatArgs));
}
+ /**
+ * @return Null if cannot parse with optimized strategy, else return the Interval.
Review Comment:
```suggestion
* @return null if cannot parse with optimized strategy, else return the Interval.
```
##########
processing/src/main/java/org/apache/druid/java/util/common/Intervals.java:
##########
@@ -53,6 +57,46 @@ public static Interval of(String format, Object... formatArgs)
return of(StringUtils.format(format, formatArgs));
}
+ /**
+ * @return Null if cannot parse with optimized strategy, else return the Interval.
+ */
+ @Nullable
+ private static Interval tryOptimizedIntervalDeserialization(final String intervalText)
+ {
+ final int slashIndex = intervalText.indexOf('/');
+ if (slashIndex <= 0 || slashIndex >= intervalText.length() - 1) {
+ return null;
+ }
+ try {
+ final String startStr = intervalText.substring(0, slashIndex);
+ final long startMillis = FAST_ISO_UTC_FORMATTER.parseMillis(startStr);
+
+ final String endStr = intervalText.substring(slashIndex + 1);
+ final long endMillis = FAST_ISO_UTC_FORMATTER.parseMillis(endStr);
+ return Intervals.utc(startMillis, endMillis);
+ }
+ catch (IllegalArgumentException e) {
+ return null;
+ }
+ }
+
+ /**
+ * Parses a Joda-Time {@link Interval} from its string representation.
+ *
+ * Tries a fast path for ISO-8601 UTC, slash-delimited intervals (e.g.
+ * "2022-09-16T00:00:00.000Z/2022-09-17T00:00:00.000Z"); otherwise falls back to the
+ * general interval parser. The returned interval uses ISO chronology in UTC.
+ */
+ public static Interval deserialize(String string)
Review Comment:
```suggestion
public static Interval fromString(String string)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [PR] Optimize interval deserialization (druid)
Posted by "FrankChen021 (via GitHub)" <gi...@apache.org>.
FrankChen021 commented on PR #18477:
URL: https://github.com/apache/druid/pull/18477#issuecomment-3254294682
```
// Zulu without millis
"2022-01-01T00:00:00Z/2022-01-02T00:00:00Z",
// Date-only
"2022-01-01/2022-01-02",
// start/period
"2022-01-01T12:00:00Z/PT6H",
// period/end
"P2DT3H4M5S/2022-01-01T00:00:00Z"
```
do we have such interval string for a segment ? I think considering all cases should be too complicated.
the problem here is mainly related to the deserialization of interval of a segment because of extra large number of segment during loading.
My opinion is that to create a new class named as `SegmentInterval` which inherits from current `Interval` class and then create a dedicated deserialization for this one to handle the segment interval format.
This would simplify the change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org