You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by "GWphua (via GitHub)" <gi...@apache.org> on 2025/09/04 04:02:16 UTC

[PR] Optimize interval deserialization (druid)

GWphua opened a new pull request, #18477:
URL: https://github.com/apache/druid/pull/18477

   ### Description
   
   <!-- Describe the goal of this PR, what problem are you fixing. If there is a corresponding issue (referenced above), it's not necessary to repeat the description here, however, you may choose to keep one summary sentence. -->
   
   <!-- Describe your patch: what did you change in code? How did you fix the problem? -->
   
   <!-- If there are several relatively logically separate changes in this PR, create a mini-section for each of them. For example: -->
   
   This PR introduces a faster deserialization strategy when dealing with Intervals.
   
   #### Motivation
   
   Many Druid JSON payloads represent intervals as "YYYY-MM-DD'T'HH:mm:ss.SSSZ/YYYY-MM-DD'T'HH:mm:ss.SSSZ". The general parser behind Intervals.of(String) is flexible but performs extra work (branching, multiple format probes, allocations). This shows up as a hot spot in flame graphs, especially in streaming paths like JsonParserIterator.
   
   <img width="1920" height="958" alt="image" src="https://github.com/user-attachments/assets/fdb6fca4-f7a9-45bb-8f6c-bbd4ac901876" />
   
   #### Faster deserialization
   Added a fast-path deserializer for `Interval` that works with the common interval string representations, and falls back to `Intervals.of(String)` for non-matching inputs.
   
   We minimize work done per parse by:
   - Avoid checking if String can be converted into a `ReadableInterval` / `ReadWriteableInterval`.
   - Precompiled `DateTimeFormatter` replaces the flexible parser, with fixed chronology.
   - Avoid building 2 `DateTime` objects per parse.
   
   #### Benchmarks
   Added a benchmark to test performance of old (legacy) and new (optimized) implementation on 2 cases:
   1. The expected interval format
   2. The interval format which causes a fallback to the old implementation. (Suffix: Fallback)
   
   ```
   Benchmark                                                 (numValues)  Mode  Cnt   Score   Error  Units
   JodaIntervalDeserialization.deserializeLegacy                   20000  avgt   30  21.466 ± 0.191  ms/op
   JodaIntervalDeserialization.deserializeLegacyFallback           20000  avgt   30  13.953 ± 0.249  ms/op
   JodaIntervalDeserialization.deserializeOptimized                20000  avgt   30  11.921 ± 0.082  ms/op
   JodaIntervalDeserialization.deserializeOptimizedFallback        20000  avgt   30  30.500 ± 0.304  ms/op
   ```
   
   Benchmark shows an estimated 44% speed-up when working with the interval of format "YYYY-MM-DD'T'HH:mm:ss.SSSZ/YYYY-MM-DD'T'HH:mm:ss.SSSZ".
   
   However, for other formats, the fallback causes the performance to drop. 
   
   
   
   #### Release note
   Speed up deserialization time of Intervals by >40%
   
   <hr>
   
   ##### Key changed/added classes in this PR
    * `processing/src/main/java/org/apache/druid/java/util/common/Intervals.java`
    * `processing/src/main/java/org/apache/druid/jackson/JodaStuff.java`
    * `processing/src/test/java/org/apache/druid/java/util/common/IntervalsTest.java`
    * `benchmarks/src/test/java/org/apache/druid/benchmark/JodaIntervalDeserialization.java`
   
   <hr>
   
   <!-- Check the items by putting "x" in the brackets for the done things. Not all of these items apply to every PR. Remove the items which are not done or not relevant to the PR. None of the items from the checklist below are strictly necessary, but it would be very helpful if you at least self-review the PR. -->
   
   This PR has:
   
   - [x] been self-reviewed.
   - [x] a release note entry in the PR description.
   - [x] added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
   - [x] added or updated version, license, or notice information in [licenses.yaml](https://github.com/apache/druid/blob/master/dev/license.md)
   - [x] added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
   - [x] added unit tests or modified existing tests to cover new code paths, ensuring the threshold for [code coverage](https://github.com/apache/druid/blob/master/dev/code-review/code-coverage.md) is met.
   - [x] been tested in a test Druid cluster.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Optimize interval deserialization (druid)

Posted by "kfaraz (via GitHub)" <gi...@apache.org>.
kfaraz commented on code in PR #18477:
URL: https://github.com/apache/druid/pull/18477#discussion_r2320811376


##########
processing/src/main/java/org/apache/druid/java/util/common/Intervals.java:
##########
@@ -53,6 +57,46 @@ public static Interval of(String format, Object... formatArgs)
     return of(StringUtils.format(format, formatArgs));
   }
 
+  /**
+   * @return Null if cannot parse with optimized strategy, else return the Interval.

Review Comment:
   For what kind of interval strings does this method return null?
   
   Is there a simpler way to short circuit in those cases rather than trying the parse unnecessarily?\
   Maybe a `canDeserializeOptimally()` method which checks it upfront? (Perhaps just check if there is `P` in there 😛 )



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Optimize interval deserialization (druid)

Posted by "GWphua (via GitHub)" <gi...@apache.org>.
GWphua commented on code in PR #18477:
URL: https://github.com/apache/druid/pull/18477#discussion_r2328603280


##########
processing/src/main/java/org/apache/druid/java/util/common/Intervals.java:
##########
@@ -25,13 +25,17 @@
 import org.joda.time.DateTime;
 import org.joda.time.Interval;
 import org.joda.time.chrono.ISOChronology;
+import org.joda.time.format.DateTimeFormatter;
+import org.joda.time.format.ISODateTimeFormat;
 
 import javax.annotation.Nullable;
 
 public final class Intervals
 {
   public static final Interval ETERNITY = utc(JodaUtils.MIN_INSTANT, JodaUtils.MAX_INSTANT);
   public static final ImmutableList<Interval> ONLY_ETERNITY = ImmutableList.of(ETERNITY);
+  private static final DateTimeFormatter FAST_ISO_UTC_FORMATTER =

Review Comment:
   I did a benchmark for the formatters. Seems `DateTimes.ISO_DATE_TIME` is a bit slower than the current optimization because it tries to create `DateTime` unnecessary objects. `Intervals#utc` can work with `long` to create `Interval` directly, and hence performs faster.
   
   On the topic of using other data formats, seems like we can look for `DateTimes.ISO_DATE_OPTIONAL_TIME` to improve the deserialization, but the improvement is quite small.
   
   ```
   Benchmark                                                                 (numValues)  Mode  Cnt   Score   Error  Units
   JodaIntervalDeserializationBenchmark.deserializeLegacy                          20000  avgt   50  20.858 ± 0.035  ms/op
   JodaIntervalDeserializationBenchmark.deserializeOptimized                       20000  avgt   50  11.940 ± 0.009  ms/op
   JodaIntervalDeserializationBenchmark.deserializeDateTime                        20000  avgt   50  12.524 ± 0.025  ms/op
   JodaIntervalDeserializationBenchmark.deserializeDateTimeOffset                  20000  avgt   50  23.505 ± 0.050  ms/op
   JodaIntervalDeserializationBenchmark.deserializeOptionalDateTime                20000  avgt   50  20.663 ± 0.111  ms/op
   
   JodaIntervalDeserializationBenchmark.deserializeLegacyFallback                  20000  avgt   50  19.035 ± 0.032  ms/op
   JodaIntervalDeserializationBenchmark.deserializeOptimizedFallback               20000  avgt   50  33.782 ± 0.105  ms/op
   JodaIntervalDeserializationBenchmark.deserializeDateTimeFallback                20000  avgt   50  36.642 ± 0.065  ms/op
   JodaIntervalDeserializationBenchmark.deserializeDateTimeOffsetFallback          20000  avgt   50  22.067 ± 0.046  ms/op
   JodaIntervalDeserializationBenchmark.deserializeOptionalDateTimeFallback        20000  avgt   50  18.861 ± 0.012  ms/op
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Optimize interval deserialization (druid)

Posted by "GWphua (via GitHub)" <gi...@apache.org>.
GWphua commented on code in PR #18477:
URL: https://github.com/apache/druid/pull/18477#discussion_r2328603280


##########
processing/src/main/java/org/apache/druid/java/util/common/Intervals.java:
##########
@@ -25,13 +25,17 @@
 import org.joda.time.DateTime;
 import org.joda.time.Interval;
 import org.joda.time.chrono.ISOChronology;
+import org.joda.time.format.DateTimeFormatter;
+import org.joda.time.format.ISODateTimeFormat;
 
 import javax.annotation.Nullable;
 
 public final class Intervals
 {
   public static final Interval ETERNITY = utc(JodaUtils.MIN_INSTANT, JodaUtils.MAX_INSTANT);
   public static final ImmutableList<Interval> ONLY_ETERNITY = ImmutableList.of(ETERNITY);
+  private static final DateTimeFormatter FAST_ISO_UTC_FORMATTER =

Review Comment:
   Hi @kfaraz 
   
   Got quite a bit of time to kill, so I did a benchmark for the formatters...
   
   Seems `DateTimes.ISO_DATE_TIME` is a bit slower than the current optimization because it tries to create `DateTime` unnecessary objects. `Intervals#utc` can work with `long` to create `Interval` directly, and hence performs faster.
   
   On the topic of using other data formats, seems like we can look for `DateTimes.ISO_DATE_OPTIONAL_TIME` to improve the deserialization, but the improvement is quite small.
   
   ```
   Benchmark                                                                 (numValues)  Mode  Cnt   Score   Error  Units
   JodaIntervalDeserializationBenchmark.deserializeLegacy                          20000  avgt   50  20.858 ± 0.035  ms/op
   JodaIntervalDeserializationBenchmark.deserializeOptimized                       20000  avgt   50  11.940 ± 0.009  ms/op
   JodaIntervalDeserializationBenchmark.deserializeDateTime                        20000  avgt   50  12.524 ± 0.025  ms/op
   JodaIntervalDeserializationBenchmark.deserializeDateTimeOffset                  20000  avgt   50  23.505 ± 0.050  ms/op
   JodaIntervalDeserializationBenchmark.deserializeOptionalDateTime                20000  avgt   50  20.663 ± 0.111  ms/op
   
   JodaIntervalDeserializationBenchmark.deserializeLegacyFallback                  20000  avgt   50  19.035 ± 0.032  ms/op
   JodaIntervalDeserializationBenchmark.deserializeOptimizedFallback               20000  avgt   50  33.782 ± 0.105  ms/op
   JodaIntervalDeserializationBenchmark.deserializeDateTimeFallback                20000  avgt   50  36.642 ± 0.065  ms/op
   JodaIntervalDeserializationBenchmark.deserializeDateTimeOffsetFallback          20000  avgt   50  22.067 ± 0.046  ms/op
   JodaIntervalDeserializationBenchmark.deserializeOptionalDateTimeFallback        20000  avgt   50  18.861 ± 0.012  ms/op
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Optimize interval deserialization (druid)

Posted by "kfaraz (via GitHub)" <gi...@apache.org>.
kfaraz commented on code in PR #18477:
URL: https://github.com/apache/druid/pull/18477#discussion_r2321013789


##########
processing/src/main/java/org/apache/druid/java/util/common/Intervals.java:
##########
@@ -53,6 +57,46 @@ public static Interval of(String format, Object... formatArgs)
     return of(StringUtils.format(format, formatArgs));
   }
 
+  /**
+   * @return Null if cannot parse with optimized strategy, else return the Interval.

Review Comment:
   Thanks!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Optimize interval deserialization (druid)

Posted by "GWphua (via GitHub)" <gi...@apache.org>.
GWphua commented on PR #18477:
URL: https://github.com/apache/druid/pull/18477#issuecomment-3257506148

   @kfaraz I'm having trouble trying to use segmentId's `tryParse()`, since it deals with data of this shape: `"datasource_2015-01-02T00:00:00.000Z_2015-01-03T00:00:00.000Z_ver_0_1"`, but our Interval representation is `"2015-01-02T00:00:00.000Z/2015-01-03T00:00:00.000Z"`. This means we will require some manipulation of the String that's going into DataSegment. lmk if im missing anything!
   
   I can change DataSegment's `interval` input to `String` type, and use `Intervals.fromString()` on it to give the performance boost, since that's the goal of this PR -- to optimize the deserialization time for `DataSegment`. If we figure out that most of Druid handles interval string representation of the compatible type format, we can consider using the optimized deserialization strategy instead. 
   
   I have previously used `Intervals.fromString()` to be the default deserializer in `JodaStuff`. Now will revert that change, and use deserialization for only DataSegments.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Optimize interval deserialization (druid)

Posted by "GWphua (via GitHub)" <gi...@apache.org>.
GWphua commented on PR #18477:
URL: https://github.com/apache/druid/pull/18477#issuecomment-3258971968

   > @GWphua , what does the flamegraph look like after the change in this PR?
   
   Hi @kfaraz, unfortunately I cannot get a flamegraph this week due to internal code freezes, that's why I am presenting performances via benchmarks as of now. I will also be on a break for a while, so you can expect to hear back from me regarding non-trivial code changes after i return -- probably early October. 
   
   Thanks!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Optimize interval deserialization (druid)

Posted by "kfaraz (via GitHub)" <gi...@apache.org>.
kfaraz commented on code in PR #18477:
URL: https://github.com/apache/druid/pull/18477#discussion_r2321015803


##########
processing/src/main/java/org/apache/druid/java/util/common/Intervals.java:
##########
@@ -53,6 +57,46 @@ public static Interval of(String format, Object... formatArgs)
     return of(StringUtils.format(format, formatArgs));
   }
 
+  /**
+   * @return Null if cannot parse with optimized strategy, else return the Interval.

Review Comment:
   > I have listed all cases i can think of under the test file.
   
   Please also list out these cases briefly in the javadoc of the new `Intervals.fromString()` method



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Optimize interval deserialization (druid)

Posted by "kfaraz (via GitHub)" <gi...@apache.org>.
kfaraz commented on code in PR #18477:
URL: https://github.com/apache/druid/pull/18477#discussion_r2321039287


##########
processing/src/test/java/org/apache/druid/java/util/common/IntervalsTest.java:
##########
@@ -79,6 +79,44 @@ public void testFindOverlappingInterval()
     );
   }
 
+  @Test
+  public void testValidIntervalStrings()
+  {
+    final String[] intervalStringRepresentations = new String[]{
+        // Tests that use does not fallback to Intervals.of()
+        // Zulu with millis
+        "2022-01-01T00:00:00.000Z/2022-01-02T00:00:00.000Z",
+        "2021-03-14T12:34:56.789Z/2021-03-15T12:34:56.789Z",
+
+        // Offset with colon
+        "2022-01-01T00:00:00.000+05:30/2022-01-01T01:00:00.000+05:30",
+        "2022-01-01T07:00:00.000-07:00/2022-01-01T08:00:00.000-07:00",
+
+        // Basic offset without colon
+        "2022-01-01T00:00:00.000+0530/2022-01-01T01:00:00.000+0530",
+
+        // Tests that fallback to Intervals.of()
+        // Zulu without millis
+        "2022-01-01T00:00:00Z/2022-01-02T00:00:00Z",
+        // Date-only
+        "2022-01-01/2022-01-02",
+        // start/period
+        "2022-01-01T00:00:00.000Z/P1D",
+        "2022-01-01T12:00:00Z/PT6H",
+        "2022-01-01T00:00:00Z/P2DT3H4M5S",
+        // period/end
+        "P1D/2022-01-02T00:00:00.000Z",
+        "PT6H/2022-01-01T18:00:00Z",
+        "P2DT3H4M5S/2022-01-03T03:04:05Z"

Review Comment:
   I feel like these cases could also be deserialized in an optimal manner if we only know which category it falls into.
   e.g. if we could quickly identify that the string has a `period/end` configuration, we could parse both parts separately. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Optimize interval deserialization (druid)

Posted by "kfaraz (via GitHub)" <gi...@apache.org>.
kfaraz commented on PR #18477:
URL: https://github.com/apache/druid/pull/18477#issuecomment-3254480919

   > My opinion is that to create a new class named as SegmentInterval which inherits from current Interval class and then create a dedicated deserialization for this one to handle the segment interval format.
   
   Ah, that would have been nice but unfortunately `joda.Interval`is a final class.
   But we can still do this by passing in a `String` interval to `DataSegment` constructor
   and then using `SegmentId.tryParse()` as it already uses the improved algorithm.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Optimize interval deserialization (druid)

Posted by "GWphua (via GitHub)" <gi...@apache.org>.
GWphua commented on code in PR #18477:
URL: https://github.com/apache/druid/pull/18477#discussion_r2321391094


##########
processing/src/main/java/org/apache/druid/java/util/common/Intervals.java:
##########
@@ -53,6 +57,46 @@ public static Interval of(String format, Object... formatArgs)
     return of(StringUtils.format(format, formatArgs));
   }
 
+  /**
+   * @return Null if cannot parse with optimized strategy, else return the Interval.

Review Comment:
   Hi @kfaraz an update on `canDeserializeOptimally()`, I added a check to see if the interval contains a `Period`, and conduct a fail-fast. The results look much better.
   
   ```
   Benchmark                                                          (numValues)  Mode  Cnt   Score   Error  Units
   JodaIntervalDeserializationBenchmark.deserializeLegacy                   20000  avgt    5  21.115 ± 0.302  ms/op
   JodaIntervalDeserializationBenchmark.deserializeLegacyFallback           20000  avgt    5  13.831 ± 0.373  ms/op
   JodaIntervalDeserializationBenchmark.deserializeOptimized                20000  avgt    5  11.909 ± 0.343  ms/op
   JodaIntervalDeserializationBenchmark.deserializeOptimizedFallback        20000  avgt    5  14.014 ± 0.477  ms/op
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Optimize interval deserialization (druid)

Posted by "GWphua (via GitHub)" <gi...@apache.org>.
GWphua commented on code in PR #18477:
URL: https://github.com/apache/druid/pull/18477#discussion_r2321411252


##########
processing/src/test/java/org/apache/druid/java/util/common/IntervalsTest.java:
##########
@@ -79,6 +79,44 @@ public void testFindOverlappingInterval()
     );
   }
 
+  @Test
+  public void testValidIntervalStrings()
+  {
+    final String[] intervalStringRepresentations = new String[]{
+        // Tests that use does not fallback to Intervals.of()
+        // Zulu with millis
+        "2022-01-01T00:00:00.000Z/2022-01-02T00:00:00.000Z",
+        "2021-03-14T12:34:56.789Z/2021-03-15T12:34:56.789Z",
+
+        // Offset with colon
+        "2022-01-01T00:00:00.000+05:30/2022-01-01T01:00:00.000+05:30",
+        "2022-01-01T07:00:00.000-07:00/2022-01-01T08:00:00.000-07:00",
+
+        // Basic offset without colon
+        "2022-01-01T00:00:00.000+0530/2022-01-01T01:00:00.000+0530",
+
+        // Tests that fallback to Intervals.of()
+        // Zulu without millis
+        "2022-01-01T00:00:00Z/2022-01-02T00:00:00Z",
+        // Date-only
+        "2022-01-01/2022-01-02",
+        // start/period
+        "2022-01-01T00:00:00.000Z/P1D",
+        "2022-01-01T12:00:00Z/PT6H",
+        "2022-01-01T00:00:00Z/P2DT3H4M5S",
+        // period/end
+        "P1D/2022-01-02T00:00:00.000Z",
+        "PT6H/2022-01-01T18:00:00Z",
+        "P2DT3H4M5S/2022-01-03T03:04:05Z"

Review Comment:
   Hi @kfaraz 
   
   I attempted to make `tryOptimizedIntervalDeserialization` more inclusive (e.g. Work on Date-only, and Without Millis case), but this requires changing the `FAST_ISO_UTC_FORMATTER` DateTimeFormatter to a more flexible formatter. This causes performance to resemble the un-optimized versions.
   
   These 2 formatters allow formats like `"2022-01-01/2022-01-02"` or `"2022-01-01T00:00:00Z/2022-01-02T00:00:00Z"`, but the performance is lacking...
   
   ```
   private static final DateTimeFormatter FAST_ISO_UTC_FORMATTER =
         ISODateTimeFormat.dateOptionalTimeParser()
                          .withOffsetParsed()
                          .withChronology(ISOChronology.getInstanceUTC());
   
   Benchmark                                                          (numValues)  Mode  Cnt   Score   Error  Units
   JodaIntervalDeserializationBenchmark.deserializeLegacy                   20000  avgt    2  21.243          ms/op
   JodaIntervalDeserializationBenchmark.deserializeLegacyFallback           20000  avgt    2  15.217          ms/op
   JodaIntervalDeserializationBenchmark.deserializeOptimized                20000  avgt    2  20.605          ms/op
   JodaIntervalDeserializationBenchmark.deserializeOptimizedFallback        20000  avgt    2  31.051          ms/op
   ```
   
   ```
   private static final DateTimeFormatter FAST_ISO_UTC_FORMATTER =
         ISODateTimeFormat.dateTimeParser().withChronology(ISOChronology.getInstanceUTC());
   
   
   Benchmark                                                          (numValues)  Mode  Cnt   Score   Error  Units
   JodaIntervalDeserializationBenchmark.deserializeLegacy                   20000  avgt    3  21.502 ± 0.995  ms/op
   JodaIntervalDeserializationBenchmark.deserializeLegacyFallback           20000  avgt    3  15.365 ± 1.227  ms/op
   JodaIntervalDeserializationBenchmark.deserializeOptimized                20000  avgt    3  21.239 ± 1.543  ms/op
   JodaIntervalDeserializationBenchmark.deserializeOptimizedFallback        20000  avgt    3  31.526 ± 3.083  ms/op
   ```
   
   I think we can simply take advantage of knowing how our data will look like, and optimize the parse process for the most frequently-used data format. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Optimize interval deserialization (druid)

Posted by "GWphua (via GitHub)" <gi...@apache.org>.
GWphua commented on code in PR #18477:
URL: https://github.com/apache/druid/pull/18477#discussion_r2321423422


##########
processing/src/test/java/org/apache/druid/java/util/common/IntervalsTest.java:
##########
@@ -79,6 +79,44 @@ public void testFindOverlappingInterval()
     );
   }
 
+  @Test
+  public void testValidIntervalStrings()
+  {
+    final String[] intervalStringRepresentations = new String[]{
+        // Tests that use does not fallback to Intervals.of()
+        // Zulu with millis
+        "2022-01-01T00:00:00.000Z/2022-01-02T00:00:00.000Z",
+        "2021-03-14T12:34:56.789Z/2021-03-15T12:34:56.789Z",
+
+        // Offset with colon
+        "2022-01-01T00:00:00.000+05:30/2022-01-01T01:00:00.000+05:30",
+        "2022-01-01T07:00:00.000-07:00/2022-01-01T08:00:00.000-07:00",
+
+        // Basic offset without colon
+        "2022-01-01T00:00:00.000+0530/2022-01-01T01:00:00.000+0530",
+
+        // Tests that fallback to Intervals.of()
+        // Zulu without millis
+        "2022-01-01T00:00:00Z/2022-01-02T00:00:00Z",
+        // Date-only
+        "2022-01-01/2022-01-02",
+        // start/period
+        "2022-01-01T00:00:00.000Z/P1D",
+        "2022-01-01T12:00:00Z/PT6H",
+        "2022-01-01T00:00:00Z/P2DT3H4M5S",
+        // period/end
+        "P1D/2022-01-02T00:00:00.000Z",
+        "PT6H/2022-01-01T18:00:00Z",
+        "P2DT3H4M5S/2022-01-03T03:04:05Z"

Review Comment:
   I think it is possible to check manually, but I feel this method feels abit too hacky, we might miss some DateTime cases:
   
   For start/end string, we check
   1. If is Date-only, append T00:00:000Z
   2. If does not have millisecond part, append turn T00:00:00Z --> T00:00:00:000Z, and throw it into the formatter.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Optimize interval deserialization (druid)

Posted by "kfaraz (via GitHub)" <gi...@apache.org>.
kfaraz commented on code in PR #18477:
URL: https://github.com/apache/druid/pull/18477#discussion_r2320816342


##########
processing/src/main/java/org/apache/druid/java/util/common/Intervals.java:
##########
@@ -53,6 +57,46 @@ public static Interval of(String format, Object... formatArgs)
     return of(StringUtils.format(format, formatArgs));
   }
 
+  /**
+   * @return Null if cannot parse with optimized strategy, else return the Interval.

Review Comment:
   Otherwise, the new deserialize becomes much slower than the legacy deserialize for fallback cases (13 vs 30).



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Optimize interval deserialization (druid)

Posted by "GWphua (via GitHub)" <gi...@apache.org>.
GWphua commented on code in PR #18477:
URL: https://github.com/apache/druid/pull/18477#discussion_r2320967879


##########
processing/src/main/java/org/apache/druid/java/util/common/Intervals.java:
##########
@@ -53,6 +57,46 @@ public static Interval of(String format, Object... formatArgs)
     return of(StringUtils.format(format, formatArgs));
   }
 
+  /**
+   * @return Null if cannot parse with optimized strategy, else return the Interval.

Review Comment:
   Hi @kfaraz, thanks for the quick review. I have incorporated your suggestions. 
   
   I have listed all cases i can think of under the test file.
   
   ```
   // Zulu without millis
   "2022-01-01T00:00:00Z/2022-01-02T00:00:00Z",
   // Date-only
   "2022-01-01/2022-01-02",
   // start/period
   "2022-01-01T12:00:00Z/PT6H",
   // period/end
   "P2DT3H4M5S/2022-01-01T00:00:00Z"
   ```
   
   I will work more into short-circuit solutions. Will update this PR soon!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Optimize interval deserialization (druid)

Posted by "kfaraz (via GitHub)" <gi...@apache.org>.
kfaraz commented on PR #18477:
URL: https://github.com/apache/druid/pull/18477#issuecomment-3257554095

   > since it deals with data of this shape: "datasource_2015-01-02T00:00:00.000Z_2015-01-03T00:00:00.000Z_ver_0_1", but our Interval representation is "2015-01-02T00:00:00.000Z/2015-01-03T00:00:00.000Z".
   
   Thanks for checking, @GWphua !
   We would need to add some new constructor to `SegmentId` to make it work but we might as well
   just use the new method you have already added in this PR.
   
   > I can change DataSegment's interval input to String type, and use Intervals.fromString() on it to give the performance boost, since that's the goal of this PR -- to optimize the deserialization time for DataSegment.
   I have previously used Intervals.fromString() to be the default deserializer in JodaStuff. Now will revert that change, and use deserialization for only DataSegments.
   
   Yeah, this makes sense. Let's proceed with this.
   Please make this JsonCreator constructor private and add a separate constructor so that all the call sites need not change.
   
   In the javadoc of `Intervals.fromString()`, please indicate that this method is to be used for
   segment intervals only (at least for the time being).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Optimize interval deserialization (druid)

Posted by "GWphua (via GitHub)" <gi...@apache.org>.
GWphua commented on code in PR #18477:
URL: https://github.com/apache/druid/pull/18477#discussion_r2328558173


##########
processing/src/main/java/org/apache/druid/java/util/common/Intervals.java:
##########
@@ -53,6 +57,74 @@ public static Interval of(String format, Object... formatArgs)
     return of(StringUtils.format(format, formatArgs));
   }
 
+  /**
+   * A performance-optimized method for parsing a Joda-Time {@link Interval} from a string.
+   * This method is significantly faster than the standard {@link Intervals#of(String)} for the following
+   * group of offsets:
+   * <ol>
+   *   <li>"2022-01-01T00:00:00.000Z/2022-01-02T00:00:00.000Z"</li>
+   *   <li>"2022-01-01T00:00:00.000+05:30/2022-01-01T01:00:00.000+05:30"</li>
+   *   <li>"2022-01-01T00:00:00.000+0530/2022-01-01T01:00:00.000+0530"</li>
+   * </ol>
+   * <p>
+   * If the input string does not match the format, it will fall back to the more flexible but
+   * slower {@link Intervals#of(String)} parser. If you are dealing with any Intervals format examples below,
+   * consider using {@link Intervals#of(String)} instead:
+   * <ol>
+   *   <li>"2022-01-01T00:00:00Z/2022-01-02T00:00:00Z" (without millis)</li>
+   *   <li>"2022-01-01/2022-01-02" (Date only)</li>
+   *   <li>"2022-01-01T12:00:00.000Z/PT6H" (Periods in start / end)</li>
+   * </ol>
+   *
+   * Currently, this method is only used in {@link org.apache.druid.timeline.SegmentId}.
+   * Should you find additional places where the Interval format is guaranteed to be compatible with this method,
+   * feel free to open a PR.

Review Comment:
   Hi @FrankChen021 
   
   While this method is only used in `DataSegment` now, should another place find this deserialization useful, we may need to move the method back to `Intervals` to avoid DRY.
   
   Furthermore, I feel that this we should place a method that deals with interval parsing in `Intervals`. What do you think?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Optimize interval deserialization (druid)

Posted by "GWphua (via GitHub)" <gi...@apache.org>.
GWphua commented on code in PR #18477:
URL: https://github.com/apache/druid/pull/18477#discussion_r2328603280


##########
processing/src/main/java/org/apache/druid/java/util/common/Intervals.java:
##########
@@ -25,13 +25,17 @@
 import org.joda.time.DateTime;
 import org.joda.time.Interval;
 import org.joda.time.chrono.ISOChronology;
+import org.joda.time.format.DateTimeFormatter;
+import org.joda.time.format.ISODateTimeFormat;
 
 import javax.annotation.Nullable;
 
 public final class Intervals
 {
   public static final Interval ETERNITY = utc(JodaUtils.MIN_INSTANT, JodaUtils.MAX_INSTANT);
   public static final ImmutableList<Interval> ONLY_ETERNITY = ImmutableList.of(ETERNITY);
+  private static final DateTimeFormatter FAST_ISO_UTC_FORMATTER =

Review Comment:
   Hi @kfaraz 
   
   Got quite a bit of time to kill, so I did a benchmark for the formatters...
   
   Seems `DateTimes.ISO_DATE_TIME` is a bit slower than the current optimization because `ISO_DATE_TIME` tries to create `DateTime` unnecessary objects. The current implementation creates an `Interval` directly using `long` via`Intervals#utc` and hence performs faster.
   
   On the topic of using other data formats, seems like we can look for `DateTimes.ISO_DATE_OPTIONAL_TIME` to improve the deserialization, but the improvement is quite small.
   
   ```
   Benchmark                                                                 (numValues)  Mode  Cnt   Score   Error  Units
   JodaIntervalDeserializationBenchmark.deserializeLegacy                          20000  avgt   50  20.858 ± 0.035  ms/op
   JodaIntervalDeserializationBenchmark.deserializeOptimized                       20000  avgt   50  11.940 ± 0.009  ms/op
   JodaIntervalDeserializationBenchmark.deserializeDateTime                        20000  avgt   50  12.524 ± 0.025  ms/op
   JodaIntervalDeserializationBenchmark.deserializeDateTimeOffset                  20000  avgt   50  23.505 ± 0.050  ms/op
   JodaIntervalDeserializationBenchmark.deserializeOptionalDateTime                20000  avgt   50  20.663 ± 0.111  ms/op
   
   JodaIntervalDeserializationBenchmark.deserializeLegacyFallback                  20000  avgt   50  19.035 ± 0.032  ms/op
   JodaIntervalDeserializationBenchmark.deserializeOptimizedFallback               20000  avgt   50  33.782 ± 0.105  ms/op
   JodaIntervalDeserializationBenchmark.deserializeDateTimeFallback                20000  avgt   50  36.642 ± 0.065  ms/op
   JodaIntervalDeserializationBenchmark.deserializeDateTimeOffsetFallback          20000  avgt   50  22.067 ± 0.046  ms/op
   JodaIntervalDeserializationBenchmark.deserializeOptionalDateTimeFallback        20000  avgt   50  18.861 ± 0.012  ms/op
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Optimize interval deserialization (druid)

Posted by "GWphua (via GitHub)" <gi...@apache.org>.
GWphua commented on code in PR #18477:
URL: https://github.com/apache/druid/pull/18477#discussion_r2321525496


##########
processing/src/test/java/org/apache/druid/java/util/common/IntervalsTest.java:
##########
@@ -79,6 +79,44 @@ public void testFindOverlappingInterval()
     );
   }
 
+  @Test
+  public void testValidIntervalStrings()
+  {
+    final String[] intervalStringRepresentations = new String[]{
+        // Tests that use does not fallback to Intervals.of()
+        // Zulu with millis
+        "2022-01-01T00:00:00.000Z/2022-01-02T00:00:00.000Z",
+        "2021-03-14T12:34:56.789Z/2021-03-15T12:34:56.789Z",
+
+        // Offset with colon
+        "2022-01-01T00:00:00.000+05:30/2022-01-01T01:00:00.000+05:30",
+        "2022-01-01T07:00:00.000-07:00/2022-01-01T08:00:00.000-07:00",
+
+        // Basic offset without colon
+        "2022-01-01T00:00:00.000+0530/2022-01-01T01:00:00.000+0530",
+
+        // Tests that fallback to Intervals.of()
+        // Zulu without millis
+        "2022-01-01T00:00:00Z/2022-01-02T00:00:00Z",
+        // Date-only
+        "2022-01-01/2022-01-02",
+        // start/period
+        "2022-01-01T00:00:00.000Z/P1D",
+        "2022-01-01T12:00:00Z/PT6H",
+        "2022-01-01T00:00:00Z/P2DT3H4M5S",
+        // period/end
+        "P1D/2022-01-02T00:00:00.000Z",
+        "PT6H/2022-01-01T18:00:00Z",
+        "P2DT3H4M5S/2022-01-03T03:04:05Z"

Review Comment:
   > if we could quickly identify that the string has a period/end configuration, we could parse both parts separately.
   
   Regarding this, I have this solution in mind:
   1. Suppose start is a timestamp, end is a period string.
   2. Parse the period string to a long. 
   3. `endMillis` = period in long + `startMillis`
   
   This method does not work all the time though. We may not get the correct number for leap years, or even months. We will need to calculate start/end period with awareness of these semantics, which is more complex. 
   I feel we can stick with knowing that we are dealing with the most common data format (`"YYYY-MM-DD'T'HH:mm:ss.SSSZ/YYYY-MM-DD'T'HH:mm:ss.SSSZ"`) for this method, and use `Interval.of(String)` for other formats?



##########
processing/src/test/java/org/apache/druid/java/util/common/IntervalsTest.java:
##########
@@ -79,6 +79,44 @@ public void testFindOverlappingInterval()
     );
   }
 
+  @Test
+  public void testValidIntervalStrings()
+  {
+    final String[] intervalStringRepresentations = new String[]{
+        // Tests that use does not fallback to Intervals.of()
+        // Zulu with millis
+        "2022-01-01T00:00:00.000Z/2022-01-02T00:00:00.000Z",
+        "2021-03-14T12:34:56.789Z/2021-03-15T12:34:56.789Z",
+
+        // Offset with colon
+        "2022-01-01T00:00:00.000+05:30/2022-01-01T01:00:00.000+05:30",
+        "2022-01-01T07:00:00.000-07:00/2022-01-01T08:00:00.000-07:00",
+
+        // Basic offset without colon
+        "2022-01-01T00:00:00.000+0530/2022-01-01T01:00:00.000+0530",
+
+        // Tests that fallback to Intervals.of()
+        // Zulu without millis
+        "2022-01-01T00:00:00Z/2022-01-02T00:00:00Z",
+        // Date-only
+        "2022-01-01/2022-01-02",
+        // start/period
+        "2022-01-01T00:00:00.000Z/P1D",
+        "2022-01-01T12:00:00Z/PT6H",
+        "2022-01-01T00:00:00Z/P2DT3H4M5S",
+        // period/end
+        "P1D/2022-01-02T00:00:00.000Z",
+        "PT6H/2022-01-01T18:00:00Z",
+        "P2DT3H4M5S/2022-01-03T03:04:05Z"

Review Comment:
   > if we could quickly identify that the string has a period/end configuration, we could parse both parts separately.
   
   Regarding this, I have this solution in mind:
   1. Suppose start is a timestamp, end is a period string.
   2. Parse the period string to a long. 
   3. `endMillis` = period in long + `startMillis`
   
   This method does not work all the time though. We may not get the correct number for leap years, or even months. We will need to calculate start/end period with awareness of these semantics, which is more complex. 
   
   I feel we can stick with knowing that we are dealing with the most common data format (`"YYYY-MM-DD'T'HH:mm:ss.SSSZ/YYYY-MM-DD'T'HH:mm:ss.SSSZ"`) for this method, and use `Interval.of(String)` for other formats?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Optimize interval deserialization (druid)

Posted by "kfaraz (via GitHub)" <gi...@apache.org>.
kfaraz commented on code in PR #18477:
URL: https://github.com/apache/druid/pull/18477#discussion_r2320816714


##########
benchmarks/src/test/java/org/apache/druid/benchmark/JodaIntervalDeserialization.java:
##########
@@ -0,0 +1,152 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.benchmark;
+
+import com.fasterxml.jackson.core.JsonParser;
+import com.fasterxml.jackson.databind.DeserializationContext;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.fasterxml.jackson.databind.deser.std.StdDeserializer;
+import com.fasterxml.jackson.databind.module.SimpleModule;
+import org.apache.druid.jackson.DefaultObjectMapper;
+import org.apache.druid.java.util.common.Intervals;
+import org.joda.time.Interval;
+import org.openjdk.jmh.annotations.Benchmark;
+import org.openjdk.jmh.annotations.BenchmarkMode;
+import org.openjdk.jmh.annotations.Fork;
+import org.openjdk.jmh.annotations.Measurement;
+import org.openjdk.jmh.annotations.Mode;
+import org.openjdk.jmh.annotations.OutputTimeUnit;
+import org.openjdk.jmh.annotations.Param;
+import org.openjdk.jmh.annotations.Scope;
+import org.openjdk.jmh.annotations.Setup;
+import org.openjdk.jmh.annotations.State;
+import org.openjdk.jmh.annotations.Warmup;
+import org.openjdk.jmh.infra.Blackhole;
+import org.openjdk.jmh.runner.Runner;
+import org.openjdk.jmh.runner.RunnerException;
+import org.openjdk.jmh.runner.options.Options;
+import org.openjdk.jmh.runner.options.OptionsBuilder;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.TimeUnit;
+
+@State(Scope.Benchmark)
+@Fork(value = 1)
+@Warmup(iterations = 3)
+@Measurement(iterations = 5)
+@BenchmarkMode(Mode.AverageTime)
+@OutputTimeUnit(TimeUnit.MILLISECONDS)
+public class JodaIntervalDeserialization

Review Comment:
   ```suggestion
   public class JodaIntervalDeserializationBenchmark
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Optimize interval deserialization (druid)

Posted by "GWphua (via GitHub)" <gi...@apache.org>.
GWphua commented on code in PR #18477:
URL: https://github.com/apache/druid/pull/18477#discussion_r2321555536


##########
processing/src/main/java/org/apache/druid/java/util/common/Intervals.java:
##########
@@ -53,6 +57,46 @@ public static Interval of(String format, Object... formatArgs)
     return of(StringUtils.format(format, formatArgs));
   }
 
+  /**
+   * @return Null if cannot parse with optimized strategy, else return the Interval.

Review Comment:
   I have updated the cases on the Javadocs of `Intervals.fromString()`, please take a look again. 😄



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Optimize interval deserialization (druid)

Posted by "GWphua (via GitHub)" <gi...@apache.org>.
GWphua commented on code in PR #18477:
URL: https://github.com/apache/druid/pull/18477#discussion_r2320967879


##########
processing/src/main/java/org/apache/druid/java/util/common/Intervals.java:
##########
@@ -53,6 +57,46 @@ public static Interval of(String format, Object... formatArgs)
     return of(StringUtils.format(format, formatArgs));
   }
 
+  /**
+   * @return Null if cannot parse with optimized strategy, else return the Interval.

Review Comment:
   Hi @kfaraz, thanks for the quick review. I have incorporated your suggestions. 
   
   Regarding the short circuit, I agree we can add a short-circuit for working with Periods , though it is hard to find an exhaustive way to cover all fallback cases. 
   
   I have listed all cases i can think of under the test file. Here's a TLDR of what's considered fallback.
   
   ```
   // Zulu without millis
   "2022-01-01T00:00:00Z/2022-01-02T00:00:00Z",
   // Date-only
   "2022-01-01/2022-01-02",
   // start/period
   "2022-01-01T12:00:00Z/PT6H",
   // period/end
   "P2DT3H4M5S/2022-01-01T00:00:00Z"
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Optimize interval deserialization (druid)

Posted by "GWphua (via GitHub)" <gi...@apache.org>.
GWphua commented on code in PR #18477:
URL: https://github.com/apache/druid/pull/18477#discussion_r2321411252


##########
processing/src/test/java/org/apache/druid/java/util/common/IntervalsTest.java:
##########
@@ -79,6 +79,44 @@ public void testFindOverlappingInterval()
     );
   }
 
+  @Test
+  public void testValidIntervalStrings()
+  {
+    final String[] intervalStringRepresentations = new String[]{
+        // Tests that use does not fallback to Intervals.of()
+        // Zulu with millis
+        "2022-01-01T00:00:00.000Z/2022-01-02T00:00:00.000Z",
+        "2021-03-14T12:34:56.789Z/2021-03-15T12:34:56.789Z",
+
+        // Offset with colon
+        "2022-01-01T00:00:00.000+05:30/2022-01-01T01:00:00.000+05:30",
+        "2022-01-01T07:00:00.000-07:00/2022-01-01T08:00:00.000-07:00",
+
+        // Basic offset without colon
+        "2022-01-01T00:00:00.000+0530/2022-01-01T01:00:00.000+0530",
+
+        // Tests that fallback to Intervals.of()
+        // Zulu without millis
+        "2022-01-01T00:00:00Z/2022-01-02T00:00:00Z",
+        // Date-only
+        "2022-01-01/2022-01-02",
+        // start/period
+        "2022-01-01T00:00:00.000Z/P1D",
+        "2022-01-01T12:00:00Z/PT6H",
+        "2022-01-01T00:00:00Z/P2DT3H4M5S",
+        // period/end
+        "P1D/2022-01-02T00:00:00.000Z",
+        "PT6H/2022-01-01T18:00:00Z",
+        "P2DT3H4M5S/2022-01-03T03:04:05Z"

Review Comment:
   Hi @kfaraz 
   
   I attempted to make `tryOptimizedIntervalDeserialization` more inclusive (e.g. Work on Date-only, and Without Millis case), but this requires changing the `FAST_ISO_UTC_FORMATTER` DateTimeFormatter to a more flexible formatter. This causes performance to resemble the un-optimized versions.
   
   These 2 formatters allow formats like `"2022-01-01/2022-01-02"` or `"2022-01-01T00:00:00Z/2022-01-02T00:00:00Z"`, but the performance is lacking...
   
   ```
   private static final DateTimeFormatter FAST_ISO_UTC_FORMATTER =
         ISODateTimeFormat.dateOptionalTimeParser()
                          .withOffsetParsed()
                          .withChronology(ISOChronology.getInstanceUTC());
   
   Benchmark                                                          (numValues)  Mode  Cnt   Score   Error  Units
   JodaIntervalDeserializationBenchmark.deserializeLegacy                   20000  avgt    2  21.243          ms/op
   JodaIntervalDeserializationBenchmark.deserializeLegacyFallback           20000  avgt    2  15.217          ms/op
   JodaIntervalDeserializationBenchmark.deserializeOptimized                20000  avgt    2  20.605          ms/op
   JodaIntervalDeserializationBenchmark.deserializeOptimizedFallback        20000  avgt    2  31.051          ms/op
   ```
   
   ```
   private static final DateTimeFormatter FAST_ISO_UTC_FORMATTER =
         ISODateTimeFormat.dateTimeParser().withChronology(ISOChronology.getInstanceUTC());
   
   
   Benchmark                                                          (numValues)  Mode  Cnt   Score   Error  Units
   JodaIntervalDeserializationBenchmark.deserializeLegacy                   20000  avgt    3  21.502 ± 0.995  ms/op
   JodaIntervalDeserializationBenchmark.deserializeLegacyFallback           20000  avgt    3  15.365 ± 1.227  ms/op
   JodaIntervalDeserializationBenchmark.deserializeOptimized                20000  avgt    3  21.239 ± 1.543  ms/op
   JodaIntervalDeserializationBenchmark.deserializeOptimizedFallback        20000  avgt    3  31.526 ± 3.083  ms/op
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Optimize interval deserialization (druid)

Posted by "GWphua (via GitHub)" <gi...@apache.org>.
GWphua commented on code in PR #18477:
URL: https://github.com/apache/druid/pull/18477#discussion_r2321525496


##########
processing/src/test/java/org/apache/druid/java/util/common/IntervalsTest.java:
##########
@@ -79,6 +79,44 @@ public void testFindOverlappingInterval()
     );
   }
 
+  @Test
+  public void testValidIntervalStrings()
+  {
+    final String[] intervalStringRepresentations = new String[]{
+        // Tests that use does not fallback to Intervals.of()
+        // Zulu with millis
+        "2022-01-01T00:00:00.000Z/2022-01-02T00:00:00.000Z",
+        "2021-03-14T12:34:56.789Z/2021-03-15T12:34:56.789Z",
+
+        // Offset with colon
+        "2022-01-01T00:00:00.000+05:30/2022-01-01T01:00:00.000+05:30",
+        "2022-01-01T07:00:00.000-07:00/2022-01-01T08:00:00.000-07:00",
+
+        // Basic offset without colon
+        "2022-01-01T00:00:00.000+0530/2022-01-01T01:00:00.000+0530",
+
+        // Tests that fallback to Intervals.of()
+        // Zulu without millis
+        "2022-01-01T00:00:00Z/2022-01-02T00:00:00Z",
+        // Date-only
+        "2022-01-01/2022-01-02",
+        // start/period
+        "2022-01-01T00:00:00.000Z/P1D",
+        "2022-01-01T12:00:00Z/PT6H",
+        "2022-01-01T00:00:00Z/P2DT3H4M5S",
+        // period/end
+        "P1D/2022-01-02T00:00:00.000Z",
+        "PT6H/2022-01-01T18:00:00Z",
+        "P2DT3H4M5S/2022-01-03T03:04:05Z"

Review Comment:
   > if we could quickly identify that the string has a period/end configuration, we could parse both parts separately.
   
   Regarding this, I have this solution in mind:
   1. Suppose start is a timestamp, end is a period string.
   2. Parse the period string to a long. 
   3. `endMillis` = period in long + `startMillis`
   
   This method does not work all the time though. We may not get the correct number for leap years, or even months. We will need to calculate start/end period with awareness of these semantics, which is more complex. I feel we can stick with knowing that we are dealing with the most common data format (`"YYYY-MM-DD'T'HH:mm:ss.SSSZ/YYYY-MM-DD'T'HH:mm:ss.SSSZ"`) for this method, and use `Interval.of(String)` for other formats?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Optimize interval deserialization (druid)

Posted by "kfaraz (via GitHub)" <gi...@apache.org>.
kfaraz commented on code in PR #18477:
URL: https://github.com/apache/druid/pull/18477#discussion_r2325372696


##########
processing/src/main/java/org/apache/druid/java/util/common/Intervals.java:
##########
@@ -53,6 +57,74 @@ public static Interval of(String format, Object... formatArgs)
     return of(StringUtils.format(format, formatArgs));
   }
 
+  /**
+   * A performance-optimized method for parsing a Joda-Time {@link Interval} from a string.
+   * This method is significantly faster than the standard {@link Intervals#of(String)} for the following
+   * group of offsets:
+   * <ol>
+   *   <li>"2022-01-01T00:00:00.000Z/2022-01-02T00:00:00.000Z"</li>
+   *   <li>"2022-01-01T00:00:00.000+05:30/2022-01-01T01:00:00.000+05:30"</li>
+   *   <li>"2022-01-01T00:00:00.000+0530/2022-01-01T01:00:00.000+0530"</li>
+   * </ol>
+   * <p>
+   * If the input string does not match the format, it will fall back to the more flexible but
+   * slower {@link Intervals#of(String)} parser. If you are dealing with any Intervals format examples below,
+   * consider using {@link Intervals#of(String)} instead:
+   * <ol>
+   *   <li>"2022-01-01T00:00:00Z/2022-01-02T00:00:00Z" (without millis)</li>
+   *   <li>"2022-01-01/2022-01-02" (Date only)</li>
+   *   <li>"2022-01-01T12:00:00.000Z/PT6H" (Periods in start / end)</li>
+   * </ol>
+   *
+   * Currently, this method is only used in {@link org.apache.druid.timeline.SegmentId}.
+   * Should you find additional places where the Interval format is guaranteed to be compatible with this method,
+   * feel free to open a PR.

Review Comment:
   Nit: We can leave out these lines.



##########
processing/src/main/java/org/apache/druid/java/util/common/Intervals.java:
##########
@@ -25,13 +25,17 @@
 import org.joda.time.DateTime;
 import org.joda.time.Interval;
 import org.joda.time.chrono.ISOChronology;
+import org.joda.time.format.DateTimeFormatter;
+import org.joda.time.format.ISODateTimeFormat;
 
 import javax.annotation.Nullable;
 
 public final class Intervals
 {
   public static final Interval ETERNITY = utc(JodaUtils.MIN_INSTANT, JodaUtils.MAX_INSTANT);
   public static final ImmutableList<Interval> ONLY_ETERNITY = ImmutableList.of(ETERNITY);
+  private static final DateTimeFormatter FAST_ISO_UTC_FORMATTER =

Review Comment:
   Maybe also check out the other two constant formatters in that class. See if they can help improve performance of the new method for other date formats. But this can be done in a follow up PR.



##########
processing/src/main/java/org/apache/druid/java/util/common/Intervals.java:
##########
@@ -25,13 +25,17 @@
 import org.joda.time.DateTime;
 import org.joda.time.Interval;
 import org.joda.time.chrono.ISOChronology;
+import org.joda.time.format.DateTimeFormatter;
+import org.joda.time.format.ISODateTimeFormat;
 
 import javax.annotation.Nullable;
 
 public final class Intervals
 {
   public static final Interval ETERNITY = utc(JodaUtils.MIN_INSTANT, JodaUtils.MAX_INSTANT);
   public static final ImmutableList<Interval> ONLY_ETERNITY = ImmutableList.of(ETERNITY);
+  private static final DateTimeFormatter FAST_ISO_UTC_FORMATTER =

Review Comment:
   I think you can use `DateTimes.ISO_DATE_TIME` instead of this since it uses the same formatter internally.



##########
processing/src/main/java/org/apache/druid/java/util/common/Intervals.java:
##########
@@ -53,6 +57,74 @@ public static Interval of(String format, Object... formatArgs)
     return of(StringUtils.format(format, formatArgs));
   }
 
+  /**
+   * A performance-optimized method for parsing a Joda-Time {@link Interval} from a string.
+   * This method is significantly faster than the standard {@link Intervals#of(String)} for the following
+   * group of offsets:
+   * <ol>
+   *   <li>"2022-01-01T00:00:00.000Z/2022-01-02T00:00:00.000Z"</li>
+   *   <li>"2022-01-01T00:00:00.000+05:30/2022-01-01T01:00:00.000+05:30"</li>
+   *   <li>"2022-01-01T00:00:00.000+0530/2022-01-01T01:00:00.000+0530"</li>
+   * </ol>
+   * <p>
+   * If the input string does not match the format, it will fall back to the more flexible but
+   * slower {@link Intervals#of(String)} parser. If you are dealing with any Intervals format examples below,
+   * consider using {@link Intervals#of(String)} instead:
+   * <ol>
+   *   <li>"2022-01-01T00:00:00Z/2022-01-02T00:00:00Z" (without millis)</li>
+   *   <li>"2022-01-01/2022-01-02" (Date only)</li>
+   *   <li>"2022-01-01T12:00:00.000Z/PT6H" (Periods in start / end)</li>
+   * </ol>
+   *
+   * Currently, this method is only used in {@link org.apache.druid.timeline.SegmentId}.
+   * Should you find additional places where the Interval format is guaranteed to be compatible with this method,
+   * feel free to open a PR.
+   */
+  public static Interval fromString(String string)
+  {
+    Interval interval = null;
+    if (canDeserializeIntervalOptimallyFromString(string)) {
+      interval = tryOptimizedIntervalDeserialization(string);
+    }
+
+    if (interval == null) {
+      return Intervals.of(string);
+    } else {
+      return interval;
+    }

Review Comment:
   Nit:
   ```suggestion
       return interval == null ? Intervals.of(string) : interval;
   ```



##########
processing/src/main/java/org/apache/druid/timeline/DataSegment.java:
##########
@@ -196,6 +197,37 @@ public DataSegment(
       @JsonProperty("size") long size,
       @JacksonInject PruneSpecsHolder pruneSpecsHolder
   )
+  {
+    this(

Review Comment:
   Please add a comment here mentioning that we take `interval` input as String so that we can deserialize it optimally.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Optimize interval deserialization (druid)

Posted by "GWphua (via GitHub)" <gi...@apache.org>.
GWphua commented on PR #18477:
URL: https://github.com/apache/druid/pull/18477#issuecomment-3409727800

   @kfaraz 
   
   Here's the flame graph after the changes:
   <img width="3456" height="1916" alt="image" src="https://github.com/user-attachments/assets/743ededa-d043-4fbc-b390-d8a9487d48bf" />
   
   Now, the initialization of Interval is roughly equal to initialization of DataSegment.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Optimize interval deserialization (druid)

Posted by "GWphua (via GitHub)" <gi...@apache.org>.
GWphua commented on code in PR #18477:
URL: https://github.com/apache/druid/pull/18477#discussion_r2320967879


##########
processing/src/main/java/org/apache/druid/java/util/common/Intervals.java:
##########
@@ -53,6 +57,46 @@ public static Interval of(String format, Object... formatArgs)
     return of(StringUtils.format(format, formatArgs));
   }
 
+  /**
+   * @return Null if cannot parse with optimized strategy, else return the Interval.

Review Comment:
   Hi @kfaraz, thanks for the quick review. I have incorporated your suggestions. 
   
   Regarding the short circuit, I agree we can add a short-circuit, though it is hard to find an exhaustive way to test. 
   
   I have listed all cases i can think of under the test file. Here's a TLDR of what's considered fallback.
   
   ```
   // Zulu without millis
   "2022-01-01T00:00:00Z/2022-01-02T00:00:00Z",
   // Date-only
   "2022-01-01/2022-01-02",
   // start/period
   "2022-01-01T12:00:00Z/PT6H",
   // period/end
   "P2DT3H4M5S/2022-01-01T00:00:00Z"
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Optimize interval deserialization (druid)

Posted by "GWphua (via GitHub)" <gi...@apache.org>.
GWphua commented on code in PR #18477:
URL: https://github.com/apache/druid/pull/18477#discussion_r2328603280


##########
processing/src/main/java/org/apache/druid/java/util/common/Intervals.java:
##########
@@ -25,13 +25,17 @@
 import org.joda.time.DateTime;
 import org.joda.time.Interval;
 import org.joda.time.chrono.ISOChronology;
+import org.joda.time.format.DateTimeFormatter;
+import org.joda.time.format.ISODateTimeFormat;
 
 import javax.annotation.Nullable;
 
 public final class Intervals
 {
   public static final Interval ETERNITY = utc(JodaUtils.MIN_INSTANT, JodaUtils.MAX_INSTANT);
   public static final ImmutableList<Interval> ONLY_ETERNITY = ImmutableList.of(ETERNITY);
+  private static final DateTimeFormatter FAST_ISO_UTC_FORMATTER =

Review Comment:
   Hi @kfaraz 
   
   Got quite a bit of time to kill, so I did a benchmark for the formatters...
   
   Seems `DateTimes.ISO_DATE_TIME` is a bit slower than the current optimization because `ISO_DATE_TIME` tries to create 2 unnecessary `DateTime` objects, before creating an `Interval` object. The current implementation creates an `Interval` directly using `long` via`Intervals#utc` and hence performs faster.
   
   On the topic of using other data formats, seems like we can look for `DateTimes.ISO_DATE_OPTIONAL_TIME` to improve the deserialization, but the improvement is quite small.
   
   ```
   Benchmark                                                                 (numValues)  Mode  Cnt   Score   Error  Units
   JodaIntervalDeserializationBenchmark.deserializeLegacy                          20000  avgt   50  20.858 ± 0.035  ms/op
   JodaIntervalDeserializationBenchmark.deserializeOptimized                       20000  avgt   50  11.940 ± 0.009  ms/op
   JodaIntervalDeserializationBenchmark.deserializeDateTime                        20000  avgt   50  12.524 ± 0.025  ms/op
   JodaIntervalDeserializationBenchmark.deserializeDateTimeOffset                  20000  avgt   50  23.505 ± 0.050  ms/op
   JodaIntervalDeserializationBenchmark.deserializeOptionalDateTime                20000  avgt   50  20.663 ± 0.111  ms/op
   
   JodaIntervalDeserializationBenchmark.deserializeLegacyFallback                  20000  avgt   50  19.035 ± 0.032  ms/op
   JodaIntervalDeserializationBenchmark.deserializeOptimizedFallback               20000  avgt   50  33.782 ± 0.105  ms/op
   JodaIntervalDeserializationBenchmark.deserializeDateTimeFallback                20000  avgt   50  36.642 ± 0.065  ms/op
   JodaIntervalDeserializationBenchmark.deserializeDateTimeOffsetFallback          20000  avgt   50  22.067 ± 0.046  ms/op
   JodaIntervalDeserializationBenchmark.deserializeOptionalDateTimeFallback        20000  avgt   50  18.861 ± 0.012  ms/op
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Optimize interval deserialization (druid)

Posted by "GWphua (via GitHub)" <gi...@apache.org>.
GWphua commented on code in PR #18477:
URL: https://github.com/apache/druid/pull/18477#discussion_r2328558173


##########
processing/src/main/java/org/apache/druid/java/util/common/Intervals.java:
##########
@@ -53,6 +57,74 @@ public static Interval of(String format, Object... formatArgs)
     return of(StringUtils.format(format, formatArgs));
   }
 
+  /**
+   * A performance-optimized method for parsing a Joda-Time {@link Interval} from a string.
+   * This method is significantly faster than the standard {@link Intervals#of(String)} for the following
+   * group of offsets:
+   * <ol>
+   *   <li>"2022-01-01T00:00:00.000Z/2022-01-02T00:00:00.000Z"</li>
+   *   <li>"2022-01-01T00:00:00.000+05:30/2022-01-01T01:00:00.000+05:30"</li>
+   *   <li>"2022-01-01T00:00:00.000+0530/2022-01-01T01:00:00.000+0530"</li>
+   * </ol>
+   * <p>
+   * If the input string does not match the format, it will fall back to the more flexible but
+   * slower {@link Intervals#of(String)} parser. If you are dealing with any Intervals format examples below,
+   * consider using {@link Intervals#of(String)} instead:
+   * <ol>
+   *   <li>"2022-01-01T00:00:00Z/2022-01-02T00:00:00Z" (without millis)</li>
+   *   <li>"2022-01-01/2022-01-02" (Date only)</li>
+   *   <li>"2022-01-01T12:00:00.000Z/PT6H" (Periods in start / end)</li>
+   * </ol>
+   *
+   * Currently, this method is only used in {@link org.apache.druid.timeline.SegmentId}.
+   * Should you find additional places where the Interval format is guaranteed to be compatible with this method,
+   * feel free to open a PR.

Review Comment:
   Hi @FrankChen021 
   
   While this method is only used in `DataSegment` now, should another place find this deserialization useful, we may need to move the method back to `Intervals`.
   
   Furthermore, I feel that this we should place a method that deals with interval parsing in `Intervals`. What do you think?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Optimize interval deserialization (druid)

Posted by "kfaraz (via GitHub)" <gi...@apache.org>.
kfaraz merged PR #18477:
URL: https://github.com/apache/druid/pull/18477


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Optimize interval deserialization (druid)

Posted by "FrankChen021 (via GitHub)" <gi...@apache.org>.
FrankChen021 commented on code in PR #18477:
URL: https://github.com/apache/druid/pull/18477#discussion_r2326702918


##########
processing/src/main/java/org/apache/druid/java/util/common/Intervals.java:
##########
@@ -53,6 +57,74 @@ public static Interval of(String format, Object... formatArgs)
     return of(StringUtils.format(format, formatArgs));
   }
 
+  /**
+   * A performance-optimized method for parsing a Joda-Time {@link Interval} from a string.
+   * This method is significantly faster than the standard {@link Intervals#of(String)} for the following
+   * group of offsets:
+   * <ol>
+   *   <li>"2022-01-01T00:00:00.000Z/2022-01-02T00:00:00.000Z"</li>
+   *   <li>"2022-01-01T00:00:00.000+05:30/2022-01-01T01:00:00.000+05:30"</li>
+   *   <li>"2022-01-01T00:00:00.000+0530/2022-01-01T01:00:00.000+0530"</li>
+   * </ol>
+   * <p>
+   * If the input string does not match the format, it will fall back to the more flexible but
+   * slower {@link Intervals#of(String)} parser. If you are dealing with any Intervals format examples below,
+   * consider using {@link Intervals#of(String)} instead:
+   * <ol>
+   *   <li>"2022-01-01T00:00:00Z/2022-01-02T00:00:00Z" (without millis)</li>
+   *   <li>"2022-01-01/2022-01-02" (Date only)</li>
+   *   <li>"2022-01-01T12:00:00.000Z/PT6H" (Periods in start / end)</li>
+   * </ol>
+   *
+   * Currently, this method is only used in {@link org.apache.druid.timeline.SegmentId}.
+   * Should you find additional places where the Interval format is guaranteed to be compatible with this method,
+   * feel free to open a PR.

Review Comment:
   Since this method is an optimization and used by the DataSegment only, my opinion is to move this method from the Interval class to the DataSegment.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Optimize interval deserialization (druid)

Posted by "kfaraz (via GitHub)" <gi...@apache.org>.
kfaraz commented on PR #18477:
URL: https://github.com/apache/druid/pull/18477#issuecomment-3260246924

   > Hi @kfaraz, unfortunately I cannot get a flamegraph this week due to internal code freezes, that's why I am presenting performances via benchmarks as of now. I will also be on a break for a while, so you can expect to hear back from me regarding non-trivial code changes after i return -- probably early October.
   
   Sure, @GWphua , thanks for the update! 🙂 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Optimize interval deserialization (druid)

Posted by "kfaraz (via GitHub)" <gi...@apache.org>.
kfaraz commented on code in PR #18477:
URL: https://github.com/apache/druid/pull/18477#discussion_r2328680357


##########
processing/src/main/java/org/apache/druid/java/util/common/Intervals.java:
##########
@@ -25,13 +25,17 @@
 import org.joda.time.DateTime;
 import org.joda.time.Interval;
 import org.joda.time.chrono.ISOChronology;
+import org.joda.time.format.DateTimeFormatter;
+import org.joda.time.format.ISODateTimeFormat;
 
 import javax.annotation.Nullable;
 
 public final class Intervals
 {
   public static final Interval ETERNITY = utc(JodaUtils.MIN_INSTANT, JodaUtils.MAX_INSTANT);
   public static final ImmutableList<Interval> ONLY_ETERNITY = ImmutableList.of(ETERNITY);
+  private static final DateTimeFormatter FAST_ISO_UTC_FORMATTER =

Review Comment:
   Thanks a lot for trying these out, @GWphua ! 🙂 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Optimize interval deserialization (druid)

Posted by "GWphua (via GitHub)" <gi...@apache.org>.
GWphua commented on PR #18477:
URL: https://github.com/apache/druid/pull/18477#issuecomment-3253036874

   Hi @kfaraz 
   
   Since the benchmark samples involving period show a similar result now, I have replaced the benchmark with new fallback samples, all of which uses non-period in start/end (e.g. without millis specified)
   
   When calling deserialization with the optimized method, we should try our best to not use intervals of this format, else risking the below performance:
   
   1. `"2022-01-01T00:00:00Z/2022-01-02T00:00:00Z"`
   2. `"2022-01-01/2022-01-02"`
   
   Benchmark results are as shown:
   ```
   Benchmark                                                          (numValues)  Mode  Cnt   Score   Error  Units
   JodaIntervalDeserializationBenchmark.deserializeLegacyFallback           20000  avgt       19.085          ms/op
   JodaIntervalDeserializationBenchmark.deserializeOptimizedFallback        20000  avgt       33.561          ms/op
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Optimize interval deserialization (druid)

Posted by "kfaraz (via GitHub)" <gi...@apache.org>.
kfaraz commented on code in PR #18477:
URL: https://github.com/apache/druid/pull/18477#discussion_r2320811376


##########
processing/src/main/java/org/apache/druid/java/util/common/Intervals.java:
##########
@@ -53,6 +57,46 @@ public static Interval of(String format, Object... formatArgs)
     return of(StringUtils.format(format, formatArgs));
   }
 
+  /**
+   * @return Null if cannot parse with optimized strategy, else return the Interval.

Review Comment:
   For what kind of interval strings does this method return null?
   
   Is there a simpler way to short circuit in those cases rather than trying the parse unnecessarily?\
   Maybe a `canDeserializeOptimally()` method which checks it upfront?



##########
processing/src/main/java/org/apache/druid/java/util/common/Intervals.java:
##########
@@ -53,6 +57,46 @@ public static Interval of(String format, Object... formatArgs)
     return of(StringUtils.format(format, formatArgs));
   }
 
+  /**
+   * @return Null if cannot parse with optimized strategy, else return the Interval.

Review Comment:
   ```suggestion
      * @return null if cannot parse with optimized strategy, else return the Interval.
   ```



##########
processing/src/main/java/org/apache/druid/java/util/common/Intervals.java:
##########
@@ -53,6 +57,46 @@ public static Interval of(String format, Object... formatArgs)
     return of(StringUtils.format(format, formatArgs));
   }
 
+  /**
+   * @return Null if cannot parse with optimized strategy, else return the Interval.
+   */
+  @Nullable
+  private static Interval tryOptimizedIntervalDeserialization(final String intervalText)
+  {
+    final int slashIndex = intervalText.indexOf('/');
+    if (slashIndex <= 0 || slashIndex >= intervalText.length() - 1) {
+      return null;
+    }
+    try {
+      final String startStr = intervalText.substring(0, slashIndex);
+      final long startMillis = FAST_ISO_UTC_FORMATTER.parseMillis(startStr);
+
+      final String endStr = intervalText.substring(slashIndex + 1);
+      final long endMillis = FAST_ISO_UTC_FORMATTER.parseMillis(endStr);
+      return Intervals.utc(startMillis, endMillis);
+    }
+    catch (IllegalArgumentException e) {
+      return null;
+    }
+  }
+
+  /**
+   * Parses a Joda-Time {@link Interval} from its string representation.
+   *
+   * Tries a fast path for ISO-8601 UTC, slash-delimited intervals (e.g.
+   * "2022-09-16T00:00:00.000Z/2022-09-17T00:00:00.000Z"); otherwise falls back to the
+   * general interval parser. The returned interval uses ISO chronology in UTC.
+   */
+  public static Interval deserialize(String string)

Review Comment:
   ```suggestion
     public static Interval fromString(String string)
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


Re: [PR] Optimize interval deserialization (druid)

Posted by "FrankChen021 (via GitHub)" <gi...@apache.org>.
FrankChen021 commented on PR #18477:
URL: https://github.com/apache/druid/pull/18477#issuecomment-3254294682

   ```
   // Zulu without millis
   "2022-01-01T00:00:00Z/2022-01-02T00:00:00Z",
   // Date-only
   "2022-01-01/2022-01-02",
   // start/period
   "2022-01-01T12:00:00Z/PT6H",
   // period/end
   "P2DT3H4M5S/2022-01-01T00:00:00Z"
   ```
   
   do we have such interval string for a segment ? I think considering all cases should be too complicated.
   the problem here is mainly related to the deserialization of interval of a segment because of extra large number of segment during loading.
   
   My opinion is that to create a new class named as `SegmentInterval` which inherits from current `Interval` class and then create a dedicated deserialization for this one to handle the segment interval format.
   
   This would simplify the change.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org