You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by GitBox <gi...@apache.org> on 2020/07/22 05:45:10 UTC

[GitHub] [druid] FrankChen021 opened a new pull request #10203: Support unit on byte-related properties

FrankChen021 opened a new pull request #10203:
URL: https://github.com/apache/druid/pull/10203


   ### Description
   
   This PR solves #9998 
   
   In Druid, there're lots of byte-related properties. In the world of big data, the value of these properties are usually very large.
   
   For example, 
   
   ````
   druid.server.maxSize=300000000000 
   ````
   
   At first glance of this configuration, it's hard to tell how many bytes `druid.server.maxSize` hold. User have to count the number digit by digit to know it's 300_000_000_000.  Changing its value is also error-prone.
   
   So this PR adds support of unit suffix to the value of these properties, like how we specify the value of `Xms` or `Xmx` properties of JVM,  to make configurations more convenient and intuitive. 
   
   With this PR, we could set `druid.server.maxSize` as below:
   
   ````
   druid.server.maxSize=300m
   # or
   druid.server.maxSize=300g
   # or
   druid.server.maxSize=3t
   ````
   
   Because K(or M/G/T) is ambiguous in computer, whether it's 1000 or 1024 is context related. This PR defines the supported units and their bases as follow
   
   | Unit | Description | Base |
   |---|---|---|
   | K | Kilo Decimal Byte | 1000 |
   | M | Mega Decimal Byte | 1000_000 |
   | G | Giga Decimal Byte | 1000_000_000 |
   | T | Tera Decimal Byte | 1000_000_000_1000 |
   | P | Peta Decimal Byte | 1000_000_000_1000_1000 |
   | KiB | Kilo Binary Byte | 1024 |
   | MiB  | Mega Binary Byte | 1024 * 1024 |
   | GiB | Giga Binary Byte | 1024 * 1024 * 1024 |
   | TiB  | Tera Binary Byte | 1024 * 1024 * 1024 * 1024 |
   | PiB  | Peta Binary Byte | 1024 * 1024 * 1024 * 1024 |
   
   So, 
   
   ````
   druid.server.maxSize=300m # 300_000_000 bytes
   ````
   ````
   druid.server.maxSize=300MiB # 300*1024*1024 bytes
   ````
   
   Of course, the old ways are still supported
   ````
   druid.server.maxSize=300000000 # 300M bytes
   ````
   
   ### Implementation
   
   This PR
   1.  adds a `Bytes` class to parse the new format of value and turns it into long-format value. With this class, it's easy to turn number formatted property into this new format
   2. adds a `BytesRange` annotation to constraint the range of a `Bytes` property
   3. changes 8 properties to support this feature
   
       - druid.cache.sizeInBytes
       - druid.indexer.runner.maxZnodeBytes
       - druid.processing.buffer.sizeBytes
       - druid.server.maxSize
       - druid.server.http.maxScatterGatherBytes
       - druid.broker.http.maxQueuedBytes
       - druid.processing.buffer.sizeBytes
       - maxSize within druid.segmentCache.locations
   
   4. changes the default value of properties above in example configuration files to use the simplified format
       
   
   <hr>
   
   This PR has:
   - [X] been self-reviewed.
   - [X] added documentation for new or modified features or behaviors.
   - [X] added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
   - [X] added unit tests or modified existing tests to cover new code paths, ensuring the threshold for [code coverage](https://github.com/apache/druid/blob/master/dev/code-review/code-coverage.md) is met.
   - [X] been tested in a test Druid cluster.
   
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] jihoonson commented on a change in pull request #10203: Support unit on byte-related properties

Posted by GitBox <gi...@apache.org>.
jihoonson commented on a change in pull request #10203:
URL: https://github.com/apache/druid/pull/10203#discussion_r462599164



##########
File path: core/src/main/java/org/apache/druid/java/util/common/HumanReadableBytes.java
##########
@@ -0,0 +1,210 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.java.util.common;
+
+import com.fasterxml.jackson.databind.annotation.JsonSerialize;
+
+import java.io.Serializable;
+
+@JsonSerialize(using = HumanReadableBytesSerializer.class)

Review comment:
       Ah I see. It's used in another Jackson magic. Thanks.

##########
File path: docs/configuration/human-readable-byte.md
##########
@@ -33,21 +33,22 @@ These properties can be configured through 2 ways:
 
 Given the cache size is 3G, there's a configuration as below
 
-````
-druid.cache.sizeInBytes=3000000000 #3 * 1000_000_000
-````
+```properties
+# 3G bytes = 3_000_000_000 bytes
+druid.cache.sizeInBytes=3000000000 
+```
 
 
 ## A number with a unit suffix
 
-Sometimes value in bytes could be very large in Druid, it's counterintuitive to set value of these properties as above.
+Sometimes value in byte could be very large in Druid, it's counterintuitive to set value of these properties as above.

Review comment:
       I think `in bytes` is correct.

##########
File path: docs/configuration/human-readable-byte.md
##########
@@ -33,21 +33,22 @@ These properties can be configured through 2 ways:
 
 Given the cache size is 3G, there's a configuration as below
 
-````
-druid.cache.sizeInBytes=3000000000 #3 * 1000_000_000
-````
+```properties
+# 3G bytes = 3_000_000_000 bytes
+druid.cache.sizeInBytes=3000000000 
+```
 
 
 ## A number with a unit suffix
 
-Sometimes value in bytes could be very large in Druid, it's counterintuitive to set value of these properties as above.
+Sometimes value in byte could be very large in Druid, it's counterintuitive to set value of these properties as above.
 Here comes another way, a number with a unit suffix.
 
-Given the total size of disks is 1T, the configuration can be
+Given the size of disk is 1T, the configuration can be

Review comment:
       Maybe `Given a disk of 1T` is better.

##########
File path: docs/configuration/human-readable-byte.md
##########
@@ -0,0 +1,88 @@
+---
+id: human-readable-byte
+title: "Human-readable Byte Configuration Reference"
+---
+
+<!--
+  ~ Licensed to the Apache Software Foundation (ASF) under one
+  ~ or more contributor license agreements.  See the NOTICE file
+  ~ distributed with this work for additional information
+  ~ regarding copyright ownership.  The ASF licenses this file
+  ~ to you under the Apache License, Version 2.0 (the
+  ~ "License"); you may not use this file except in compliance
+  ~ with the License.  You may obtain a copy of the License at
+  ~
+  ~   http://www.apache.org/licenses/LICENSE-2.0
+  ~
+  ~ Unless required by applicable law or agreed to in writing,
+  ~ software distributed under the License is distributed on an
+  ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+  ~ KIND, either express or implied.  See the License for the
+  ~ specific language governing permissions and limitations
+  ~ under the License.
+  -->
+
+
+This page documents all configuration properties related to bytes.
+
+These properties can be configured through 2 ways: 
+1. a simple number in byte

Review comment:
       `in bytes` here too.

##########
File path: docs/configuration/human-readable-byte.md
##########
@@ -0,0 +1,88 @@
+---
+id: human-readable-byte
+title: "Human-readable Byte Configuration Reference"
+---
+
+<!--
+  ~ Licensed to the Apache Software Foundation (ASF) under one
+  ~ or more contributor license agreements.  See the NOTICE file
+  ~ distributed with this work for additional information
+  ~ regarding copyright ownership.  The ASF licenses this file
+  ~ to you under the Apache License, Version 2.0 (the
+  ~ "License"); you may not use this file except in compliance
+  ~ with the License.  You may obtain a copy of the License at
+  ~
+  ~   http://www.apache.org/licenses/LICENSE-2.0
+  ~
+  ~ Unless required by applicable law or agreed to in writing,
+  ~ software distributed under the License is distributed on an
+  ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+  ~ KIND, either express or implied.  See the License for the
+  ~ specific language governing permissions and limitations
+  ~ under the License.
+  -->
+
+
+This page documents all configuration properties related to bytes.
+
+These properties can be configured through 2 ways: 
+1. a simple number in byte
+2. a number with a unit suffix
+
+## A number in byte

Review comment:
       Same here too.

##########
File path: docs/configuration/human-readable-byte.md
##########
@@ -0,0 +1,88 @@
+---
+id: human-readable-byte
+title: "Human-readable Byte Configuration Reference"
+---
+
+<!--
+  ~ Licensed to the Apache Software Foundation (ASF) under one
+  ~ or more contributor license agreements.  See the NOTICE file
+  ~ distributed with this work for additional information
+  ~ regarding copyright ownership.  The ASF licenses this file
+  ~ to you under the Apache License, Version 2.0 (the
+  ~ "License"); you may not use this file except in compliance
+  ~ with the License.  You may obtain a copy of the License at
+  ~
+  ~   http://www.apache.org/licenses/LICENSE-2.0
+  ~
+  ~ Unless required by applicable law or agreed to in writing,
+  ~ software distributed under the License is distributed on an
+  ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+  ~ KIND, either express or implied.  See the License for the
+  ~ specific language governing permissions and limitations
+  ~ under the License.
+  -->
+
+
+This page documents all configuration properties related to bytes.
+
+These properties can be configured through 2 ways: 
+1. a simple number in byte
+2. a number with a unit suffix
+
+## A number in byte
+
+Given the cache size is 3G, there's a configuration as below
+
+```properties
+# 3G bytes = 3_000_000_000 bytes
+druid.cache.sizeInBytes=3000000000 
+```
+
+
+## A number with a unit suffix
+
+Sometimes value in byte could be very large in Druid, it's counterintuitive to set value of these properties as above.
+Here comes another way, a number with a unit suffix.
+
+Given the size of disk is 1T, the configuration can be
+
+```properties
+druid.segmentCache.locations=[{"path":"/segment-cache","maxSize":"1t"}]
+```
+
+Note: in above example, both `1t` and `1T` are acceptable since it's case insensitive.

Review comment:
       I think it should be `case-insensitive` since it's a single word.

##########
File path: docs/configuration/human-readable-byte.md
##########
@@ -0,0 +1,88 @@
+---
+id: human-readable-byte
+title: "Human-readable Byte Configuration Reference"
+---
+
+<!--
+  ~ Licensed to the Apache Software Foundation (ASF) under one
+  ~ or more contributor license agreements.  See the NOTICE file
+  ~ distributed with this work for additional information
+  ~ regarding copyright ownership.  The ASF licenses this file
+  ~ to you under the Apache License, Version 2.0 (the
+  ~ "License"); you may not use this file except in compliance
+  ~ with the License.  You may obtain a copy of the License at
+  ~
+  ~   http://www.apache.org/licenses/LICENSE-2.0
+  ~
+  ~ Unless required by applicable law or agreed to in writing,
+  ~ software distributed under the License is distributed on an
+  ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+  ~ KIND, either express or implied.  See the License for the
+  ~ specific language governing permissions and limitations
+  ~ under the License.
+  -->
+
+
+This page documents all configuration properties related to bytes.
+
+These properties can be configured through 2 ways: 
+1. a simple number in byte
+2. a number with a unit suffix
+
+## A number in byte
+
+Given the cache size is 3G, there's a configuration as below
+
+```properties
+# 3G bytes = 3_000_000_000 bytes
+druid.cache.sizeInBytes=3000000000 
+```
+
+
+## A number with a unit suffix
+
+Sometimes value in byte could be very large in Druid, it's counterintuitive to set value of these properties as above.
+Here comes another way, a number with a unit suffix.

Review comment:
       Hmm.. I'm not sure whether or not `counterintuitive` is being used correctly here. AFAIK, it means something which doesn't seems true intuitively but is actually true. Maybe it would be more clear by rephrasing this paragraph to `When you have to put a large number for some configuration, it is easy to make a mistake such as extra or missing 0s. Druid supports a better way, a number with a unit suffix.`




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] lgtm-com[bot] commented on pull request #10203: Support unit on byte-related properties

Posted by GitBox <gi...@apache.org>.
lgtm-com[bot] commented on pull request #10203:
URL: https://github.com/apache/druid/pull/10203#issuecomment-664189732


   This pull request **introduces 1 alert** when merging 023b908d76192c6387dae54a6aeb900a2e0ea893 into 79dffefbf8d51e3f46e235ec6741a810adcbd8f2 - [view on LGTM.com](https://lgtm.com/projects/g/apache/druid/rev/pr-b74468afc2794cbfdc975827712e1bddd3ba37f2)
   
   **new alerts:**
   
   * 1 for Missing format argument


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] FrankChen021 commented on a change in pull request #10203: Support unit on byte-related properties

Posted by GitBox <gi...@apache.org>.
FrankChen021 commented on a change in pull request #10203:
URL: https://github.com/apache/druid/pull/10203#discussion_r462216106



##########
File path: docs/configuration/human-readable-byte.md
##########
@@ -0,0 +1,83 @@
+---
+id: human-readable-byte
+title: "Human-readable Byte Configuration Reference"
+---
+
+<!--
+  ~ Licensed to the Apache Software Foundation (ASF) under one
+  ~ or more contributor license agreements.  See the NOTICE file
+  ~ distributed with this work for additional information
+  ~ regarding copyright ownership.  The ASF licenses this file
+  ~ to you under the Apache License, Version 2.0 (the
+  ~ "License"); you may not use this file except in compliance
+  ~ with the License.  You may obtain a copy of the License at
+  ~
+  ~   http://www.apache.org/licenses/LICENSE-2.0
+  ~
+  ~ Unless required by applicable law or agreed to in writing,
+  ~ software distributed under the License is distributed on an
+  ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+  ~ KIND, either express or implied.  See the License for the
+  ~ specific language governing permissions and limitations
+  ~ under the License.
+  -->
+
+
+This page documents all configuration properties related to bytes.
+
+These properties can be configured through 2 ways: 
+1. a simple number in byte
+2. a number with a unit suffix
+
+## A number in byte
+
+Given the cache size is 3G, there's a configuration as below
+
+````
+druid.cache.sizeInBytes=3000000000 #3 * 1000_000_000
+````
+
+
+## A number with a unit suffix
+
+Sometimes value in bytes could be very large in Druid, it's counterintuitive to set value of these properties as above.
+Here comes another way, a number with a unit suffix.
+
+Given the total size of disks is 1T, the configuration can be
+
+````
+druid.segmentCache.locations=[{"path":"/segment-cache","maxSize":"1g"}]
+````
+
+### Supported Units
+In the world of computer, a unit like `K` is ambiguous. It means 1000 or 1024 in different contexts, for more information please see [Here](https://en.wikipedia.org/wiki/Binary_prefix).
+
+To make it clear, the base of units are defined as below
+
+| Unit | Description | Base |
+|---|---|---|
+| K | Kilo Decimal Byte | 1000 |

Review comment:
       Yes, no one wants to read doc until he encounters problems. So example configuration files have been updated to demonstrate the usage of both decimal byte and binary byte, I hope them help. 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] FrankChen021 commented on pull request #10203: Support unit on byte-related properties

Posted by GitBox <gi...@apache.org>.
FrankChen021 commented on pull request #10203:
URL: https://github.com/apache/druid/pull/10203#issuecomment-663361858


   Hi @jihoonson , Thanks for your comments. I'll check and fix them on Monday.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] AKarbas commented on pull request #10203: Support unit on byte-related properties

Posted by GitBox <gi...@apache.org>.
AKarbas commented on pull request #10203:
URL: https://github.com/apache/druid/pull/10203#issuecomment-771080634


   Great job. Just one thing that IMO is missing from the docs: Based on https://github.com/apache/druid/blob/c346ce64b1497f8bbd547bee16fa98cc0d3c74f2/core/src/main/java/org/apache/druid/java/util/common/HumanReadableBytes.java#L200
   
   Only integers are valid before the unit. E.g., `1.5GiB` is invalid.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] FrankChen021 commented on a change in pull request #10203: Support unit on byte-related properties

Posted by GitBox <gi...@apache.org>.
FrankChen021 commented on a change in pull request #10203:
URL: https://github.com/apache/druid/pull/10203#discussion_r460709226



##########
File path: processing/src/main/java/org/apache/druid/query/DruidProcessingConfig.java
##########
@@ -40,9 +41,14 @@
   private AtomicReference<Integer> computedBufferSizeBytes = new AtomicReference<>();
 
   @Config({"druid.computation.buffer.size", "${base_path}.buffer.sizeBytes"})
+  public String bufferSizeConfigured()
+  {
+    return "-1";
+  }
+
   public int intermediateComputeSizeBytesConfigured()
   {
-    return DEFAULT_PROCESSING_BUFFER_SIZE_BYTES;
+    return (int) Bytes.parse(bufferSizeConfigured(), DEFAULT_PROCESSING_BUFFER_SIZE_BYTES);

Review comment:
       Yes, it should return an instance of `Bytes`. I didn't understand how `@Config` works at first, so `String` was used.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] FrankChen021 commented on a change in pull request #10203: Support unit on byte-related properties

Posted by GitBox <gi...@apache.org>.
FrankChen021 commented on a change in pull request #10203:
URL: https://github.com/apache/druid/pull/10203#discussion_r460711990



##########
File path: core/src/main/java/org/apache/druid/java/util/common/Bytes.java
##########
@@ -0,0 +1,198 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.java.util.common;
+
+import com.fasterxml.jackson.databind.annotation.JsonSerialize;
+
+import java.io.Serializable;
+import java.util.Locale;
+
+@JsonSerialize(using = BytesSerializer.class)
+public class Bytes implements Serializable
+{
+  public static final Bytes ZERO = new Bytes(0L);
+
+  private long value;
+
+  public Bytes(String value)
+  {
+    this.value = Bytes.parse(value);
+  }
+
+  public Bytes(long value)
+  {
+    this.value = value;
+  }
+
+  public long getValue()
+  {
+    return value;
+  }
+
+  @Override
+  public boolean equals(Object thatObj)
+  {
+    if (thatObj == null) {
+      return false;
+    }
+    if (thatObj instanceof Bytes) {
+      return value == ((Bytes) thatObj).value;
+    } else {
+      return false;
+    }
+  }
+
+  @Override
+  public int hashCode()
+  {
+    return Long.hashCode(value);
+  }
+
+  @Override
+  public String toString()
+  {
+    return String.valueOf(value);
+  }
+
+  public static Bytes valueOf(int bytes)
+  {
+    return new Bytes(bytes);
+  }
+
+  public static Bytes valueOf(long bytes)
+  {
+    return new Bytes(bytes);
+  }
+
+  public static long parse(String number)
+  {
+    if (number == null) {
+      throw new IAE("number is null");
+    }
+
+    number = number.trim().toLowerCase(Locale.getDefault());
+    int len = number.length();
+    if (len == 0) {
+      throw new IAE("number is empty");
+    }
+
+    return parseInner(number);
+  }
+
+  /**
+   * parse the case-insensitive string number, which is either:
+   * <p>
+   * a number string
+   * <p>
+   * or
+   * <p>
+   * a number string with a suffix which indicates the unit the of number
+   * the unit must be one of following
+   * k - kilobyte = 1000
+   * m - megabyte = 1,000,000
+   * g - gigabyte = 1,000,000,000
+   * t - terabyte = 1,000,000,000,000
+   * p - petabyte = 1,000,000,000,000,000
+   * KiB - kilo binary byte = 1024
+   * MiB - mega binary byte = 1024*1204
+   * GiB - giga binary byte = 1024*1024*1024
+   * TiB - tera binary byte = 1024*1024*1024*1024
+   * PiB - peta binary byte = 1024*1024*1024*1024*1024
+   * <p>
+   *
+   * @param nullValue to be returned when given number is null or empty
+   * @return nullValue if input is null or empty
+   * value of number
+   * @throws IAE if the input is invalid
+   */
+  public static long parse(String number, long nullValue)
+  {
+    if (number == null) {
+      return nullValue;
+    }
+
+    number = number.trim().toLowerCase(Locale.getDefault());
+    if (number.length() == 0) {
+      return nullValue;
+    }
+    return parseInner(number);
+  }
+
+  private static long parseInner(String number)
+  {
+    int index = number.length() - 1;
+    boolean isBinary = false;
+    char unit = number.charAt(index--);
+    if (unit == 'b') {
+      if (index < 2) {
+        throw new IAE("invalid format of number[%s]", number);
+      }
+      if (number.charAt(index--) != 'i') {
+        throw new IAE("invalid format of number[%s]", number);
+      }
+
+      unit = number.charAt(index--);
+      isBinary = true;
+    }
+    long base = 1;
+    switch (unit) {
+      case 'k':
+        base = isBinary ? 1024 : 1_000;
+        break;
+
+      case 'm':
+        base = isBinary ? 1024 * 1024 : 1_000_000;
+        break;
+
+      case 'g':
+        base = isBinary ? 1024 * 1024 * 1024 : 1_000_000_000;
+        break;
+
+      case 't':
+        base = isBinary ? 1024 * 1024 * 1024 * 1024L : 1_000_000_000_000L;
+        break;
+
+      case 'p':
+        base = isBinary ? 1024L * 1024 * 1024 * 1024 * 1024 : 1_000_000_000_000_000L;
+        break;
+
+      default:
+        if (!Character.isDigit(unit)) {
+          throw new IAE("invalid character in number[%s]", number);
+        }
+        break;
+    }
+
+    try {
+      if (base > 1 && index >= 0) {
+        long value = Long.parseLong(number.substring(0, index + 1)) * base;
+        if (value < base) {

Review comment:
       it's tricky here, because when base equals to 1, `index`, `substring` should not be called. I've made some changes to eliminate the confusing code.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] FrankChen021 commented on a change in pull request #10203: Support unit on byte-related properties

Posted by GitBox <gi...@apache.org>.
FrankChen021 commented on a change in pull request #10203:
URL: https://github.com/apache/druid/pull/10203#discussion_r460707644



##########
File path: core/src/main/java/org/apache/druid/java/util/common/Bytes.java
##########
@@ -0,0 +1,198 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.java.util.common;
+
+import com.fasterxml.jackson.databind.annotation.JsonSerialize;
+
+import java.io.Serializable;
+import java.util.Locale;
+
+@JsonSerialize(using = BytesSerializer.class)
+public class Bytes implements Serializable

Review comment:
       I didn't come up with a nice name at first. `Bytes` is really wide used and has no specific meaning. Your suggestion is great.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] asdf2014 merged pull request #10203: Support unit on byte-related properties

Posted by GitBox <gi...@apache.org>.
asdf2014 merged pull request #10203:
URL: https://github.com/apache/druid/pull/10203


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] jon-wei commented on a change in pull request #10203: Support unit on byte-related properties

Posted by GitBox <gi...@apache.org>.
jon-wei commented on a change in pull request #10203:
URL: https://github.com/apache/druid/pull/10203#discussion_r462507356



##########
File path: core/src/main/java/org/apache/druid/java/util/common/HumanReadableBytes.java
##########
@@ -0,0 +1,210 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.java.util.common;
+
+import com.fasterxml.jackson.databind.annotation.JsonSerialize;
+
+import java.io.Serializable;
+
+@JsonSerialize(using = HumanReadableBytesSerializer.class)

Review comment:
       It's deserialized from configs in JettyServerModule:
   
   `JsonConfigProvider.bind(binder, "druid.server.http", ServerConfig.class);`




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] jihoonson commented on a change in pull request #10203: Support unit on byte-related properties

Posted by GitBox <gi...@apache.org>.
jihoonson commented on a change in pull request #10203:
URL: https://github.com/apache/druid/pull/10203#discussion_r461406956



##########
File path: core/src/main/java/org/apache/druid/java/util/common/HumanReadableBytes.java
##########
@@ -128,71 +135,76 @@ public static long parse(String number, long nullValue)
       return nullValue;
     }
 
-    number = number.trim().toLowerCase(Locale.getDefault());
+    number = number.trim();
     if (number.length() == 0) {
       return nullValue;
     }
     return parseInner(number);
   }
 
-  private static long parseInner(String number)
+  private static long parseInner(String rawNumber)
   {
-    int index = number.length() - 1;
-    boolean isBinary = false;
-    char unit = number.charAt(index--);
+    String number = StringUtils.toLowerCase(rawNumber);
+    if (number.charAt(0) == '-') {
+      throw new IAE("Invalid format of number: %s. Negative value is not allowed.", rawNumber);
+    }
+
+    int lastDigitIndex = number.length() - 1;
+    boolean isBinaryByte = false;
+    char unit = number.charAt(lastDigitIndex--);
     if (unit == 'b') {
-      if (index < 2) {
-        throw new IAE("invalid format of number[%s]", number);
+      //unit ends with 'b' must be format of KiB/MiB/GiB/TiB/PiB, so at least 3 extra characters are required
+      if (lastDigitIndex < 2) {
+        throw new IAE("Invalid format of number: %s", rawNumber);
       }
-      if (number.charAt(index--) != 'i') {
-        throw new IAE("invalid format of number[%s]", number);
+      if (number.charAt(lastDigitIndex--) != 'i') {
+        throw new IAE("Invalid format of number: %s", rawNumber);
       }
 
-      unit = number.charAt(index--);
-      isBinary = true;
+      unit = number.charAt(lastDigitIndex--);
+      isBinaryByte = true;
     }
+
     long base = 1;
     switch (unit) {
       case 'k':
-        base = isBinary ? 1024 : 1_000;
+        base = isBinaryByte ? 1024 : 1_000;
         break;
 
       case 'm':
-        base = isBinary ? 1024 * 1024 : 1_000_000;
+        base = isBinaryByte ? 1024 * 1024 : 1_000_000;
         break;
 
       case 'g':
-        base = isBinary ? 1024 * 1024 * 1024 : 1_000_000_000;
+        base = isBinaryByte ? 1024 * 1024 * 1024 : 1_000_000_000;
         break;
 
       case 't':
-        base = isBinary ? 1024 * 1024 * 1024 * 1024L : 1_000_000_000_000L;
+        base = isBinaryByte ? 1024 * 1024 * 1024 * 1024L : 1_000_000_000_000L;
         break;
 
       case 'p':
-        base = isBinary ? 1024L * 1024 * 1024 * 1024 * 1024 : 1_000_000_000_000_000L;
+        base = isBinaryByte ? 1024L * 1024 * 1024 * 1024 * 1024 : 1_000_000_000_000_000L;
         break;
 
       default:
+        lastDigitIndex++;
         if (!Character.isDigit(unit)) {
-          throw new IAE("invalid character in number[%s]", number);
+          throw new IAE("Invalid format of number: %s", rawNumber);
         }
         break;
     }
 
     try {
-      if (base > 1 && index >= 0) {
-        long value = Long.parseLong(number.substring(0, index + 1)) * base;
-        if (value < base) {
-          throw new IAE("number [%s] overflow", number);
-        }
-        return value;
-      } else {
-        return Long.parseLong(number);
+      long value = Long.parseLong(number.substring(0, lastDigitIndex + 1)) * base;
+      if (base > 1 && value < base) {
+        //for base == 1, overflow has been checked in parseLong
+        throw new IAE("Number overflow: %s", rawNumber);
       }
+      return value;
     }
     catch (NumberFormatException e) {
-      throw new IAE("invalid format of number[%s]", number);
+      throw new IAE("Invalid format of number: %s", rawNumber);

Review comment:
       It would be nice to log that the number might be out of range of `long`.

##########
File path: docs/configuration/human-readable-byte.md
##########
@@ -0,0 +1,83 @@
+---
+id: human-readable-byte
+title: "Human-readable Byte Configuration Reference"
+---
+
+<!--
+  ~ Licensed to the Apache Software Foundation (ASF) under one
+  ~ or more contributor license agreements.  See the NOTICE file
+  ~ distributed with this work for additional information
+  ~ regarding copyright ownership.  The ASF licenses this file
+  ~ to you under the Apache License, Version 2.0 (the
+  ~ "License"); you may not use this file except in compliance
+  ~ with the License.  You may obtain a copy of the License at
+  ~
+  ~   http://www.apache.org/licenses/LICENSE-2.0
+  ~
+  ~ Unless required by applicable law or agreed to in writing,
+  ~ software distributed under the License is distributed on an
+  ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+  ~ KIND, either express or implied.  See the License for the
+  ~ specific language governing permissions and limitations
+  ~ under the License.
+  -->
+
+
+This page documents all configuration properties related to bytes.
+
+These properties can be configured through 2 ways: 
+1. a simple number in byte
+2. a number with a unit suffix
+
+## A number in byte
+
+Given the cache size is 3G, there's a configuration as below
+
+````
+druid.cache.sizeInBytes=3000000000 #3 * 1000_000_000
+````
+
+
+## A number with a unit suffix
+
+Sometimes value in bytes could be very large in Druid, it's counterintuitive to set value of these properties as above.
+Here comes another way, a number with a unit suffix.
+
+Given the total size of disks is 1T, the configuration can be
+
+````
+druid.segmentCache.locations=[{"path":"/segment-cache","maxSize":"1g"}]

Review comment:
       There is a mismatch in this example and the above statement. It should be `1t`.

##########
File path: core/src/main/java/org/apache/druid/java/util/common/HumanReadableBytes.java
##########
@@ -0,0 +1,210 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.java.util.common;
+
+import com.fasterxml.jackson.databind.annotation.JsonSerialize;
+
+import java.io.Serializable;
+
+@JsonSerialize(using = HumanReadableBytesSerializer.class)
+public class HumanReadableBytes implements Serializable

Review comment:
       I missed this before. Does `Serializable` do anything here? It doesn't seem so.

##########
File path: core/src/main/java/org/apache/druid/java/util/common/HumanReadableBytes.java
##########
@@ -0,0 +1,210 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.java.util.common;
+
+import com.fasterxml.jackson.databind.annotation.JsonSerialize;
+
+import java.io.Serializable;
+
+@JsonSerialize(using = HumanReadableBytesSerializer.class)

Review comment:
       The custom serializer doesn't seem necessary. AFAIT, this class never gets serialized. In deserializing, Jackson will magically finds the proper constructor depending on the parameter type.

##########
File path: docs/configuration/human-readable-byte.md
##########
@@ -0,0 +1,83 @@
+---
+id: human-readable-byte
+title: "Human-readable Byte Configuration Reference"
+---
+
+<!--
+  ~ Licensed to the Apache Software Foundation (ASF) under one
+  ~ or more contributor license agreements.  See the NOTICE file
+  ~ distributed with this work for additional information
+  ~ regarding copyright ownership.  The ASF licenses this file
+  ~ to you under the Apache License, Version 2.0 (the
+  ~ "License"); you may not use this file except in compliance
+  ~ with the License.  You may obtain a copy of the License at
+  ~
+  ~   http://www.apache.org/licenses/LICENSE-2.0
+  ~
+  ~ Unless required by applicable law or agreed to in writing,
+  ~ software distributed under the License is distributed on an
+  ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+  ~ KIND, either express or implied.  See the License for the
+  ~ specific language governing permissions and limitations
+  ~ under the License.
+  -->
+
+
+This page documents all configuration properties related to bytes.
+
+These properties can be configured through 2 ways: 
+1. a simple number in byte
+2. a number with a unit suffix
+
+## A number in byte
+
+Given the cache size is 3G, there's a configuration as below
+
+````
+druid.cache.sizeInBytes=3000000000 #3 * 1000_000_000
+````
+
+
+## A number with a unit suffix
+
+Sometimes value in bytes could be very large in Druid, it's counterintuitive to set value of these properties as above.
+Here comes another way, a number with a unit suffix.
+
+Given the total size of disks is 1T, the configuration can be
+
+````
+druid.segmentCache.locations=[{"path":"/segment-cache","maxSize":"1g"}]
+````
+
+### Supported Units
+In the world of computer, a unit like `K` is ambiguous. It means 1000 or 1024 in different contexts, for more information please see [Here](https://en.wikipedia.org/wiki/Binary_prefix).
+
+To make it clear, the base of units are defined as below
+
+| Unit | Description | Base |
+|---|---|---|
+| K | Kilo Decimal Byte | 1000 |

Review comment:
       nit: this doesn't have to be done in this PR, but I think it would be nice to allow `KB`, `MB`, etc as aliases for decimal units.

##########
File path: core/src/main/java/org/apache/druid/java/util/common/HumanReadableBytes.java
##########
@@ -0,0 +1,210 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.java.util.common;
+
+import com.fasterxml.jackson.databind.annotation.JsonSerialize;
+
+import java.io.Serializable;
+
+@JsonSerialize(using = HumanReadableBytesSerializer.class)
+public class HumanReadableBytes implements Serializable
+{
+  public static final HumanReadableBytes ZERO = new HumanReadableBytes(0L);
+
+  private long bytes;

Review comment:
       Can be final.

##########
File path: core/src/main/java/org/apache/druid/java/util/common/HumanReadableBytes.java
##########
@@ -128,71 +135,76 @@ public static long parse(String number, long nullValue)
       return nullValue;
     }
 
-    number = number.trim().toLowerCase(Locale.getDefault());
+    number = number.trim();
     if (number.length() == 0) {
       return nullValue;
     }
     return parseInner(number);
   }
 
-  private static long parseInner(String number)
+  private static long parseInner(String rawNumber)
   {
-    int index = number.length() - 1;
-    boolean isBinary = false;
-    char unit = number.charAt(index--);
+    String number = StringUtils.toLowerCase(rawNumber);
+    if (number.charAt(0) == '-') {
+      throw new IAE("Invalid format of number: %s. Negative value is not allowed.", rawNumber);
+    }
+
+    int lastDigitIndex = number.length() - 1;
+    boolean isBinaryByte = false;
+    char unit = number.charAt(lastDigitIndex--);
     if (unit == 'b') {
-      if (index < 2) {
-        throw new IAE("invalid format of number[%s]", number);
+      //unit ends with 'b' must be format of KiB/MiB/GiB/TiB/PiB, so at least 3 extra characters are required
+      if (lastDigitIndex < 2) {
+        throw new IAE("Invalid format of number: %s", rawNumber);
       }
-      if (number.charAt(index--) != 'i') {
-        throw new IAE("invalid format of number[%s]", number);
+      if (number.charAt(lastDigitIndex--) != 'i') {
+        throw new IAE("Invalid format of number: %s", rawNumber);
       }
 
-      unit = number.charAt(index--);
-      isBinary = true;
+      unit = number.charAt(lastDigitIndex--);
+      isBinaryByte = true;
     }
+
     long base = 1;
     switch (unit) {
       case 'k':
-        base = isBinary ? 1024 : 1_000;
+        base = isBinaryByte ? 1024 : 1_000;
         break;
 
       case 'm':
-        base = isBinary ? 1024 * 1024 : 1_000_000;
+        base = isBinaryByte ? 1024 * 1024 : 1_000_000;
         break;
 
       case 'g':
-        base = isBinary ? 1024 * 1024 * 1024 : 1_000_000_000;
+        base = isBinaryByte ? 1024 * 1024 * 1024 : 1_000_000_000;
         break;
 
       case 't':
-        base = isBinary ? 1024 * 1024 * 1024 * 1024L : 1_000_000_000_000L;
+        base = isBinaryByte ? 1024 * 1024 * 1024 * 1024L : 1_000_000_000_000L;
         break;
 
       case 'p':
-        base = isBinary ? 1024L * 1024 * 1024 * 1024 * 1024 : 1_000_000_000_000_000L;
+        base = isBinaryByte ? 1024L * 1024 * 1024 * 1024 * 1024 : 1_000_000_000_000_000L;
         break;
 
       default:
+        lastDigitIndex++;
         if (!Character.isDigit(unit)) {
-          throw new IAE("invalid character in number[%s]", number);
+          throw new IAE("Invalid format of number: %s", rawNumber);
         }
         break;
     }
 
     try {
-      if (base > 1 && index >= 0) {
-        long value = Long.parseLong(number.substring(0, index + 1)) * base;
-        if (value < base) {
-          throw new IAE("number [%s] overflow", number);
-        }
-        return value;
-      } else {
-        return Long.parseLong(number);
+      long value = Long.parseLong(number.substring(0, lastDigitIndex + 1)) * base;
+      if (base > 1 && value < base) {
+        //for base == 1, overflow has been checked in parseLong
+        throw new IAE("Number overflow: %s", rawNumber);
       }
+      return value;
     }
     catch (NumberFormatException e) {
-      throw new IAE("invalid format of number[%s]", number);
+      throw new IAE("Invalid format of number: %s", rawNumber);

Review comment:
       I meant the error message should be understandable for Druid operators. The current message is confusing since it doesn't mention another possibility for out of range. Perhaps it's better to be `Invalid number format or out of range of long: %s`.

##########
File path: docs/configuration/human-readable-byte.md
##########
@@ -0,0 +1,83 @@
+---
+id: human-readable-byte
+title: "Human-readable Byte Configuration Reference"
+---
+
+<!--
+  ~ Licensed to the Apache Software Foundation (ASF) under one
+  ~ or more contributor license agreements.  See the NOTICE file
+  ~ distributed with this work for additional information
+  ~ regarding copyright ownership.  The ASF licenses this file
+  ~ to you under the Apache License, Version 2.0 (the
+  ~ "License"); you may not use this file except in compliance
+  ~ with the License.  You may obtain a copy of the License at
+  ~
+  ~   http://www.apache.org/licenses/LICENSE-2.0
+  ~
+  ~ Unless required by applicable law or agreed to in writing,
+  ~ software distributed under the License is distributed on an
+  ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+  ~ KIND, either express or implied.  See the License for the
+  ~ specific language governing permissions and limitations
+  ~ under the License.
+  -->
+
+
+This page documents all configuration properties related to bytes.
+
+These properties can be configured through 2 ways: 
+1. a simple number in byte
+2. a number with a unit suffix
+
+## A number in byte
+
+Given the cache size is 3G, there's a configuration as below
+
+````
+druid.cache.sizeInBytes=3000000000 #3 * 1000_000_000
+````
+
+
+## A number with a unit suffix
+
+Sometimes value in bytes could be very large in Druid, it's counterintuitive to set value of these properties as above.
+Here comes another way, a number with a unit suffix.
+
+Given the total size of disks is 1T, the configuration can be
+
+````
+druid.segmentCache.locations=[{"path":"/segment-cache","maxSize":"1g"}]
+````
+
+### Supported Units
+In the world of computer, a unit like `K` is ambiguous. It means 1000 or 1024 in different contexts, for more information please see [Here](https://en.wikipedia.org/wiki/Binary_prefix).
+
+To make it clear, the base of units are defined as below
+
+| Unit | Description | Base |
+|---|---|---|
+| K | Kilo Decimal Byte | 1000 |

Review comment:
       That's true, but I'm not sure how much it matters since `KB` and `KiB` are approximately same. Also, I bet there will be more people who try setting some value before reading the doc. It would be nice if it just works whatever they put.

##########
File path: core/src/main/java/org/apache/druid/java/util/common/HumanReadableBytes.java
##########
@@ -0,0 +1,210 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.java.util.common;
+
+import com.fasterxml.jackson.databind.annotation.JsonSerialize;
+
+import java.io.Serializable;
+
+@JsonSerialize(using = HumanReadableBytesSerializer.class)

Review comment:
       Hmm, I'm not sure why we have `ServerConfigTest.testSerde()` even though `ServerConfig` doesn't seem to be serialized/deserialized. @jon-wei do you have any idea?

##########
File path: core/src/main/java/org/apache/druid/java/util/common/HumanReadableBytes.java
##########
@@ -128,71 +135,76 @@ public static long parse(String number, long nullValue)
       return nullValue;
     }
 
-    number = number.trim().toLowerCase(Locale.getDefault());
+    number = number.trim();
     if (number.length() == 0) {
       return nullValue;
     }
     return parseInner(number);
   }
 
-  private static long parseInner(String number)
+  private static long parseInner(String rawNumber)
   {
-    int index = number.length() - 1;
-    boolean isBinary = false;
-    char unit = number.charAt(index--);
+    String number = StringUtils.toLowerCase(rawNumber);
+    if (number.charAt(0) == '-') {
+      throw new IAE("Invalid format of number: %s. Negative value is not allowed.", rawNumber);
+    }
+
+    int lastDigitIndex = number.length() - 1;
+    boolean isBinaryByte = false;
+    char unit = number.charAt(lastDigitIndex--);
     if (unit == 'b') {
-      if (index < 2) {
-        throw new IAE("invalid format of number[%s]", number);
+      //unit ends with 'b' must be format of KiB/MiB/GiB/TiB/PiB, so at least 3 extra characters are required
+      if (lastDigitIndex < 2) {
+        throw new IAE("Invalid format of number: %s", rawNumber);
       }
-      if (number.charAt(index--) != 'i') {
-        throw new IAE("invalid format of number[%s]", number);
+      if (number.charAt(lastDigitIndex--) != 'i') {
+        throw new IAE("Invalid format of number: %s", rawNumber);
       }
 
-      unit = number.charAt(index--);
-      isBinary = true;
+      unit = number.charAt(lastDigitIndex--);
+      isBinaryByte = true;
     }
+
     long base = 1;
     switch (unit) {
       case 'k':
-        base = isBinary ? 1024 : 1_000;
+        base = isBinaryByte ? 1024 : 1_000;
         break;
 
       case 'm':
-        base = isBinary ? 1024 * 1024 : 1_000_000;
+        base = isBinaryByte ? 1024 * 1024 : 1_000_000;
         break;
 
       case 'g':
-        base = isBinary ? 1024 * 1024 * 1024 : 1_000_000_000;
+        base = isBinaryByte ? 1024 * 1024 * 1024 : 1_000_000_000;
         break;
 
       case 't':
-        base = isBinary ? 1024 * 1024 * 1024 * 1024L : 1_000_000_000_000L;
+        base = isBinaryByte ? 1024 * 1024 * 1024 * 1024L : 1_000_000_000_000L;
         break;
 
       case 'p':
-        base = isBinary ? 1024L * 1024 * 1024 * 1024 * 1024 : 1_000_000_000_000_000L;
+        base = isBinaryByte ? 1024L * 1024 * 1024 * 1024 * 1024 : 1_000_000_000_000_000L;
         break;
 
       default:
+        lastDigitIndex++;
         if (!Character.isDigit(unit)) {
-          throw new IAE("invalid character in number[%s]", number);
+          throw new IAE("Invalid format of number: %s", rawNumber);
         }
         break;
     }
 
     try {
-      if (base > 1 && index >= 0) {
-        long value = Long.parseLong(number.substring(0, index + 1)) * base;
-        if (value < base) {
-          throw new IAE("number [%s] overflow", number);
-        }
-        return value;
-      } else {
-        return Long.parseLong(number);
+      long value = Long.parseLong(number.substring(0, lastDigitIndex + 1)) * base;
+      if (base > 1 && value < base) {
+        //for base == 1, overflow has been checked in parseLong
+        throw new IAE("Number overflow: %s", rawNumber);
       }
+      return value;
     }
     catch (NumberFormatException e) {
-      throw new IAE("invalid format of number[%s]", number);
+      throw new IAE("Invalid format of number: %s", rawNumber);

Review comment:
       I meant the error message should be correct so that Druid operators can fix the issue. The current message can be misleading since it doesn't mention another possibility for out of range. Perhaps it's better to be `Invalid number format or out of range of long: %s`.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] FrankChen021 commented on a change in pull request #10203: Support unit on byte-related properties

Posted by GitBox <gi...@apache.org>.
FrankChen021 commented on a change in pull request #10203:
URL: https://github.com/apache/druid/pull/10203#discussion_r462010802



##########
File path: core/src/main/java/org/apache/druid/java/util/common/HumanReadableBytes.java
##########
@@ -128,71 +135,76 @@ public static long parse(String number, long nullValue)
       return nullValue;
     }
 
-    number = number.trim().toLowerCase(Locale.getDefault());
+    number = number.trim();
     if (number.length() == 0) {
       return nullValue;
     }
     return parseInner(number);
   }
 
-  private static long parseInner(String number)
+  private static long parseInner(String rawNumber)
   {
-    int index = number.length() - 1;
-    boolean isBinary = false;
-    char unit = number.charAt(index--);
+    String number = StringUtils.toLowerCase(rawNumber);
+    if (number.charAt(0) == '-') {
+      throw new IAE("Invalid format of number: %s. Negative value is not allowed.", rawNumber);
+    }
+
+    int lastDigitIndex = number.length() - 1;
+    boolean isBinaryByte = false;
+    char unit = number.charAt(lastDigitIndex--);
     if (unit == 'b') {
-      if (index < 2) {
-        throw new IAE("invalid format of number[%s]", number);
+      //unit ends with 'b' must be format of KiB/MiB/GiB/TiB/PiB, so at least 3 extra characters are required
+      if (lastDigitIndex < 2) {
+        throw new IAE("Invalid format of number: %s", rawNumber);
       }
-      if (number.charAt(index--) != 'i') {
-        throw new IAE("invalid format of number[%s]", number);
+      if (number.charAt(lastDigitIndex--) != 'i') {
+        throw new IAE("Invalid format of number: %s", rawNumber);
       }
 
-      unit = number.charAt(index--);
-      isBinary = true;
+      unit = number.charAt(lastDigitIndex--);
+      isBinaryByte = true;
     }
+
     long base = 1;
     switch (unit) {
       case 'k':
-        base = isBinary ? 1024 : 1_000;
+        base = isBinaryByte ? 1024 : 1_000;
         break;
 
       case 'm':
-        base = isBinary ? 1024 * 1024 : 1_000_000;
+        base = isBinaryByte ? 1024 * 1024 : 1_000_000;
         break;
 
       case 'g':
-        base = isBinary ? 1024 * 1024 * 1024 : 1_000_000_000;
+        base = isBinaryByte ? 1024 * 1024 * 1024 : 1_000_000_000;
         break;
 
       case 't':
-        base = isBinary ? 1024 * 1024 * 1024 * 1024L : 1_000_000_000_000L;
+        base = isBinaryByte ? 1024 * 1024 * 1024 * 1024L : 1_000_000_000_000L;
         break;
 
       case 'p':
-        base = isBinary ? 1024L * 1024 * 1024 * 1024 * 1024 : 1_000_000_000_000_000L;
+        base = isBinaryByte ? 1024L * 1024 * 1024 * 1024 * 1024 : 1_000_000_000_000_000L;
         break;
 
       default:
+        lastDigitIndex++;
         if (!Character.isDigit(unit)) {
-          throw new IAE("invalid character in number[%s]", number);
+          throw new IAE("Invalid format of number: %s", rawNumber);
         }
         break;
     }
 
     try {
-      if (base > 1 && index >= 0) {
-        long value = Long.parseLong(number.substring(0, index + 1)) * base;
-        if (value < base) {
-          throw new IAE("number [%s] overflow", number);
-        }
-        return value;
-      } else {
-        return Long.parseLong(number);
+      long value = Long.parseLong(number.substring(0, lastDigitIndex + 1)) * base;
+      if (base > 1 && value < base) {
+        //for base == 1, overflow has been checked in parseLong
+        throw new IAE("Number overflow: %s", rawNumber);
       }
+      return value;
     }
     catch (NumberFormatException e) {
-      throw new IAE("invalid format of number[%s]", number);
+      throw new IAE("Invalid format of number: %s", rawNumber);

Review comment:
       NumberFormatException throws when 1) there're invalid characters ; 2) number overflows. It's unknown for us which case it is when it's thrown. 

##########
File path: core/src/main/java/org/apache/druid/java/util/common/HumanReadableBytes.java
##########
@@ -0,0 +1,210 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.java.util.common;
+
+import com.fasterxml.jackson.databind.annotation.JsonSerialize;
+
+import java.io.Serializable;
+
+@JsonSerialize(using = HumanReadableBytesSerializer.class)

Review comment:
       `HumanReadableBytes` is referenced by other classes such as `ServerConfig`. The test cases of the latter ones test serialization and deserialization. So this serializer annotation currently works for test cases only.

##########
File path: docs/configuration/human-readable-byte.md
##########
@@ -0,0 +1,83 @@
+---
+id: human-readable-byte
+title: "Human-readable Byte Configuration Reference"
+---
+
+<!--
+  ~ Licensed to the Apache Software Foundation (ASF) under one
+  ~ or more contributor license agreements.  See the NOTICE file
+  ~ distributed with this work for additional information
+  ~ regarding copyright ownership.  The ASF licenses this file
+  ~ to you under the Apache License, Version 2.0 (the
+  ~ "License"); you may not use this file except in compliance
+  ~ with the License.  You may obtain a copy of the License at
+  ~
+  ~   http://www.apache.org/licenses/LICENSE-2.0
+  ~
+  ~ Unless required by applicable law or agreed to in writing,
+  ~ software distributed under the License is distributed on an
+  ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+  ~ KIND, either express or implied.  See the License for the
+  ~ specific language governing permissions and limitations
+  ~ under the License.
+  -->
+
+
+This page documents all configuration properties related to bytes.
+
+These properties can be configured through 2 ways: 
+1. a simple number in byte
+2. a number with a unit suffix
+
+## A number in byte
+
+Given the cache size is 3G, there's a configuration as below
+
+````
+druid.cache.sizeInBytes=3000000000 #3 * 1000_000_000
+````
+
+
+## A number with a unit suffix
+
+Sometimes value in bytes could be very large in Druid, it's counterintuitive to set value of these properties as above.
+Here comes another way, a number with a unit suffix.
+
+Given the total size of disks is 1T, the configuration can be
+
+````
+druid.segmentCache.locations=[{"path":"/segment-cache","maxSize":"1g"}]
+````
+
+### Supported Units
+In the world of computer, a unit like `K` is ambiguous. It means 1000 or 1024 in different contexts, for more information please see [Here](https://en.wikipedia.org/wiki/Binary_prefix).
+
+To make it clear, the base of units are defined as below
+
+| Unit | Description | Base |
+|---|---|---|
+| K | Kilo Decimal Byte | 1000 |

Review comment:
       Formally, it should be `KB` or `MB`.  K/M/G... are commonly accepted abbreviations. My opinion is we can choose one of `KB` ,`K`, but not both of them. To support both `K` and `KB` might introduce complexity and ambiguity




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] jihoonson commented on a change in pull request #10203: Support unit on byte-related properties

Posted by GitBox <gi...@apache.org>.
jihoonson commented on a change in pull request #10203:
URL: https://github.com/apache/druid/pull/10203#discussion_r459852843



##########
File path: core/src/main/java/org/apache/druid/java/util/common/Bytes.java
##########
@@ -0,0 +1,198 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.java.util.common;
+
+import com.fasterxml.jackson.databind.annotation.JsonSerialize;
+
+import java.io.Serializable;
+import java.util.Locale;
+
+@JsonSerialize(using = BytesSerializer.class)
+public class Bytes implements Serializable
+{
+  public static final Bytes ZERO = new Bytes(0L);
+
+  private long value;
+
+  public Bytes(String value)
+  {
+    this.value = Bytes.parse(value);
+  }
+
+  public Bytes(long value)
+  {
+    this.value = value;
+  }
+
+  public long getValue()
+  {
+    return value;
+  }
+
+  @Override
+  public boolean equals(Object thatObj)
+  {
+    if (thatObj == null) {
+      return false;
+    }
+    if (thatObj instanceof Bytes) {
+      return value == ((Bytes) thatObj).value;
+    } else {
+      return false;
+    }
+  }
+
+  @Override
+  public int hashCode()
+  {
+    return Long.hashCode(value);
+  }
+
+  @Override
+  public String toString()
+  {
+    return String.valueOf(value);
+  }
+
+  public static Bytes valueOf(int bytes)
+  {
+    return new Bytes(bytes);
+  }
+
+  public static Bytes valueOf(long bytes)
+  {
+    return new Bytes(bytes);
+  }
+
+  public static long parse(String number)
+  {
+    if (number == null) {
+      throw new IAE("number is null");
+    }
+
+    number = number.trim().toLowerCase(Locale.getDefault());

Review comment:
       Please use `StringUtils.toLowerCase()` instead.

##########
File path: processing/src/main/java/org/apache/druid/query/DruidProcessingConfig.java
##########
@@ -40,9 +41,14 @@
   private AtomicReference<Integer> computedBufferSizeBytes = new AtomicReference<>();
 
   @Config({"druid.computation.buffer.size", "${base_path}.buffer.sizeBytes"})
+  public String bufferSizeConfigured()
+  {
+    return "-1";
+  }
+
   public int intermediateComputeSizeBytesConfigured()
   {
-    return DEFAULT_PROCESSING_BUFFER_SIZE_BYTES;
+    return (int) Bytes.parse(bufferSizeConfigured(), DEFAULT_PROCESSING_BUFFER_SIZE_BYTES);

Review comment:
       Hmm, this method can be called several times during processing a query. Why not returning `Bytes` in `bufferSizeConfigured()` instead of `String`?

##########
File path: core/src/main/java/org/apache/druid/java/util/common/Bytes.java
##########
@@ -0,0 +1,198 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.java.util.common;
+
+import com.fasterxml.jackson.databind.annotation.JsonSerialize;
+
+import java.io.Serializable;
+import java.util.Locale;
+
+@JsonSerialize(using = BytesSerializer.class)
+public class Bytes implements Serializable

Review comment:
       nit: `Bytes` seems too broad. For example, `Bytes.getValue()` doesn't seem intuitive enough to me what it would return. Maybe `HumanReadableBytes` could be a more readable name.

##########
File path: core/src/main/java/org/apache/druid/java/util/common/Bytes.java
##########
@@ -0,0 +1,198 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.java.util.common;
+
+import com.fasterxml.jackson.databind.annotation.JsonSerialize;
+
+import java.io.Serializable;
+import java.util.Locale;
+
+@JsonSerialize(using = BytesSerializer.class)
+public class Bytes implements Serializable
+{
+  public static final Bytes ZERO = new Bytes(0L);
+
+  private long value;
+
+  public Bytes(String value)
+  {
+    this.value = Bytes.parse(value);
+  }
+
+  public Bytes(long value)
+  {
+    this.value = value;
+  }
+
+  public long getValue()
+  {
+    return value;
+  }
+
+  @Override
+  public boolean equals(Object thatObj)
+  {
+    if (thatObj == null) {
+      return false;
+    }
+    if (thatObj instanceof Bytes) {
+      return value == ((Bytes) thatObj).value;
+    } else {
+      return false;
+    }
+  }
+
+  @Override
+  public int hashCode()
+  {
+    return Long.hashCode(value);
+  }
+
+  @Override
+  public String toString()
+  {
+    return String.valueOf(value);
+  }
+
+  public static Bytes valueOf(int bytes)
+  {
+    return new Bytes(bytes);
+  }
+
+  public static Bytes valueOf(long bytes)
+  {
+    return new Bytes(bytes);
+  }
+
+  public static long parse(String number)
+  {
+    if (number == null) {
+      throw new IAE("number is null");
+    }
+
+    number = number.trim().toLowerCase(Locale.getDefault());
+    int len = number.length();
+    if (len == 0) {
+      throw new IAE("number is empty");
+    }
+
+    return parseInner(number);
+  }
+
+  /**
+   * parse the case-insensitive string number, which is either:
+   * <p>
+   * a number string
+   * <p>
+   * or
+   * <p>
+   * a number string with a suffix which indicates the unit the of number
+   * the unit must be one of following
+   * k - kilobyte = 1000
+   * m - megabyte = 1,000,000
+   * g - gigabyte = 1,000,000,000
+   * t - terabyte = 1,000,000,000,000
+   * p - petabyte = 1,000,000,000,000,000
+   * KiB - kilo binary byte = 1024
+   * MiB - mega binary byte = 1024*1204
+   * GiB - giga binary byte = 1024*1024*1024
+   * TiB - tera binary byte = 1024*1024*1024*1024
+   * PiB - peta binary byte = 1024*1024*1024*1024*1024
+   * <p>
+   *
+   * @param nullValue to be returned when given number is null or empty
+   * @return nullValue if input is null or empty
+   * value of number
+   * @throws IAE if the input is invalid
+   */
+  public static long parse(String number, long nullValue)
+  {
+    if (number == null) {
+      return nullValue;
+    }
+
+    number = number.trim().toLowerCase(Locale.getDefault());
+    if (number.length() == 0) {
+      return nullValue;
+    }
+    return parseInner(number);
+  }
+
+  private static long parseInner(String number)
+  {
+    int index = number.length() - 1;
+    boolean isBinary = false;
+    char unit = number.charAt(index--);
+    if (unit == 'b') {
+      if (index < 2) {
+        throw new IAE("invalid format of number[%s]", number);
+      }
+      if (number.charAt(index--) != 'i') {
+        throw new IAE("invalid format of number[%s]", number);
+      }
+
+      unit = number.charAt(index--);
+      isBinary = true;
+    }
+    long base = 1;
+    switch (unit) {
+      case 'k':
+        base = isBinary ? 1024 : 1_000;
+        break;
+
+      case 'm':
+        base = isBinary ? 1024 * 1024 : 1_000_000;
+        break;
+
+      case 'g':
+        base = isBinary ? 1024 * 1024 * 1024 : 1_000_000_000;
+        break;
+
+      case 't':
+        base = isBinary ? 1024 * 1024 * 1024 * 1024L : 1_000_000_000_000L;
+        break;
+
+      case 'p':
+        base = isBinary ? 1024L * 1024 * 1024 * 1024 * 1024 : 1_000_000_000_000_000L;
+        break;
+
+      default:
+        if (!Character.isDigit(unit)) {
+          throw new IAE("invalid character in number[%s]", number);
+        }
+        break;
+    }
+
+    try {
+      if (base > 1 && index >= 0) {
+        long value = Long.parseLong(number.substring(0, index + 1)) * base;
+        if (value < base) {
+          throw new IAE("number [%s] overflow", number);
+        }
+        return value;
+      } else {
+        return Long.parseLong(number);

Review comment:
       Why not checking overflow here too?

##########
File path: core/src/main/java/org/apache/druid/java/util/common/Bytes.java
##########
@@ -0,0 +1,198 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.java.util.common;
+
+import com.fasterxml.jackson.databind.annotation.JsonSerialize;
+
+import java.io.Serializable;
+import java.util.Locale;
+
+@JsonSerialize(using = BytesSerializer.class)
+public class Bytes implements Serializable
+{
+  public static final Bytes ZERO = new Bytes(0L);
+
+  private long value;
+
+  public Bytes(String value)
+  {
+    this.value = Bytes.parse(value);
+  }
+
+  public Bytes(long value)
+  {
+    this.value = value;
+  }
+
+  public long getValue()
+  {
+    return value;
+  }
+
+  @Override
+  public boolean equals(Object thatObj)
+  {
+    if (thatObj == null) {
+      return false;
+    }
+    if (thatObj instanceof Bytes) {
+      return value == ((Bytes) thatObj).value;
+    } else {
+      return false;
+    }
+  }
+
+  @Override
+  public int hashCode()
+  {
+    return Long.hashCode(value);
+  }
+
+  @Override
+  public String toString()
+  {
+    return String.valueOf(value);
+  }
+
+  public static Bytes valueOf(int bytes)
+  {
+    return new Bytes(bytes);
+  }
+
+  public static Bytes valueOf(long bytes)
+  {
+    return new Bytes(bytes);
+  }
+
+  public static long parse(String number)
+  {
+    if (number == null) {
+      throw new IAE("number is null");
+    }
+
+    number = number.trim().toLowerCase(Locale.getDefault());
+    int len = number.length();
+    if (len == 0) {
+      throw new IAE("number is empty");
+    }
+
+    return parseInner(number);
+  }
+
+  /**
+   * parse the case-insensitive string number, which is either:
+   * <p>
+   * a number string
+   * <p>
+   * or
+   * <p>
+   * a number string with a suffix which indicates the unit the of number
+   * the unit must be one of following
+   * k - kilobyte = 1000
+   * m - megabyte = 1,000,000
+   * g - gigabyte = 1,000,000,000
+   * t - terabyte = 1,000,000,000,000
+   * p - petabyte = 1,000,000,000,000,000
+   * KiB - kilo binary byte = 1024
+   * MiB - mega binary byte = 1024*1204
+   * GiB - giga binary byte = 1024*1024*1024
+   * TiB - tera binary byte = 1024*1024*1024*1024
+   * PiB - peta binary byte = 1024*1024*1024*1024*1024
+   * <p>
+   *
+   * @param nullValue to be returned when given number is null or empty
+   * @return nullValue if input is null or empty
+   * value of number
+   * @throws IAE if the input is invalid
+   */
+  public static long parse(String number, long nullValue)
+  {
+    if (number == null) {
+      return nullValue;
+    }
+
+    number = number.trim().toLowerCase(Locale.getDefault());
+    if (number.length() == 0) {
+      return nullValue;
+    }
+    return parseInner(number);
+  }
+
+  private static long parseInner(String number)
+  {
+    int index = number.length() - 1;
+    boolean isBinary = false;
+    char unit = number.charAt(index--);
+    if (unit == 'b') {
+      if (index < 2) {
+        throw new IAE("invalid format of number[%s]", number);
+      }
+      if (number.charAt(index--) != 'i') {
+        throw new IAE("invalid format of number[%s]", number);
+      }
+
+      unit = number.charAt(index--);
+      isBinary = true;
+    }
+    long base = 1;
+    switch (unit) {
+      case 'k':
+        base = isBinary ? 1024 : 1_000;
+        break;
+
+      case 'm':
+        base = isBinary ? 1024 * 1024 : 1_000_000;
+        break;
+
+      case 'g':
+        base = isBinary ? 1024 * 1024 * 1024 : 1_000_000_000;
+        break;
+
+      case 't':
+        base = isBinary ? 1024 * 1024 * 1024 * 1024L : 1_000_000_000_000L;
+        break;
+
+      case 'p':
+        base = isBinary ? 1024L * 1024 * 1024 * 1024 * 1024 : 1_000_000_000_000_000L;
+        break;
+
+      default:
+        if (!Character.isDigit(unit)) {
+          throw new IAE("invalid character in number[%s]", number);
+        }
+        break;
+    }
+
+    try {
+      if (base > 1 && index >= 0) {
+        long value = Long.parseLong(number.substring(0, index + 1)) * base;
+        if (value < base) {

Review comment:
       Out of curiosity, why does this compare against `base` instead of 0?

##########
File path: processing/src/main/java/org/apache/druid/query/DruidProcessingConfig.java
##########
@@ -40,9 +41,14 @@
   private AtomicReference<Integer> computedBufferSizeBytes = new AtomicReference<>();
 
   @Config({"druid.computation.buffer.size", "${base_path}.buffer.sizeBytes"})
+  public String bufferSizeConfigured()
+  {
+    return "-1";
+  }
+
   public int intermediateComputeSizeBytesConfigured()
   {
-    return DEFAULT_PROCESSING_BUFFER_SIZE_BYTES;
+    return (int) Bytes.parse(bufferSizeConfigured(), DEFAULT_PROCESSING_BUFFER_SIZE_BYTES);

Review comment:
       I guess we can call `Bytes.parse()` without default value since `bufferSizeConfigured()` has default?

##########
File path: processing/src/main/java/org/apache/druid/query/DruidProcessingConfig.java
##########
@@ -40,9 +41,14 @@
   private AtomicReference<Integer> computedBufferSizeBytes = new AtomicReference<>();
 
   @Config({"druid.computation.buffer.size", "${base_path}.buffer.sizeBytes"})
+  public String bufferSizeConfigured()
+  {
+    return "-1";

Review comment:
       Please make it as a static variable.

##########
File path: core/src/main/java/org/apache/druid/java/util/common/BytesRange.java
##########
@@ -0,0 +1,79 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.java.util.common;
+
+import javax.validation.Constraint;
+import javax.validation.ConstraintValidator;
+import javax.validation.ConstraintValidatorContext;
+import javax.validation.Payload;
+import java.lang.annotation.Documented;
+import java.lang.annotation.ElementType;
+import java.lang.annotation.Retention;
+import java.lang.annotation.RetentionPolicy;
+import java.lang.annotation.Target;
+
+@Target({
+    ElementType.FIELD,
+    ElementType.PARAMETER
+})
+@Retention(RetentionPolicy.RUNTIME)
+@Documented
+@Constraint(validatedBy = BytesRange.DateRangeValidator.class)
+public @interface BytesRange

Review comment:
       Please add simple javadoc about what this annotation is and how it is used.

##########
File path: core/src/main/java/org/apache/druid/java/util/common/Bytes.java
##########
@@ -0,0 +1,198 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.java.util.common;
+
+import com.fasterxml.jackson.databind.annotation.JsonSerialize;
+
+import java.io.Serializable;
+import java.util.Locale;
+
+@JsonSerialize(using = BytesSerializer.class)
+public class Bytes implements Serializable
+{
+  public static final Bytes ZERO = new Bytes(0L);
+
+  private long value;
+
+  public Bytes(String value)
+  {
+    this.value = Bytes.parse(value);
+  }
+
+  public Bytes(long value)
+  {
+    this.value = value;
+  }
+
+  public long getValue()
+  {
+    return value;
+  }
+
+  @Override
+  public boolean equals(Object thatObj)
+  {
+    if (thatObj == null) {
+      return false;
+    }
+    if (thatObj instanceof Bytes) {
+      return value == ((Bytes) thatObj).value;
+    } else {
+      return false;
+    }
+  }
+
+  @Override
+  public int hashCode()
+  {
+    return Long.hashCode(value);
+  }
+
+  @Override
+  public String toString()
+  {
+    return String.valueOf(value);
+  }
+
+  public static Bytes valueOf(int bytes)
+  {
+    return new Bytes(bytes);
+  }
+
+  public static Bytes valueOf(long bytes)
+  {
+    return new Bytes(bytes);
+  }
+
+  public static long parse(String number)
+  {
+    if (number == null) {
+      throw new IAE("number is null");
+    }
+
+    number = number.trim().toLowerCase(Locale.getDefault());
+    int len = number.length();
+    if (len == 0) {
+      throw new IAE("number is empty");
+    }
+
+    return parseInner(number);
+  }
+
+  /**
+   * parse the case-insensitive string number, which is either:
+   * <p>
+   * a number string
+   * <p>
+   * or
+   * <p>
+   * a number string with a suffix which indicates the unit the of number
+   * the unit must be one of following
+   * k - kilobyte = 1000
+   * m - megabyte = 1,000,000
+   * g - gigabyte = 1,000,000,000
+   * t - terabyte = 1,000,000,000,000
+   * p - petabyte = 1,000,000,000,000,000
+   * KiB - kilo binary byte = 1024
+   * MiB - mega binary byte = 1024*1204
+   * GiB - giga binary byte = 1024*1024*1024
+   * TiB - tera binary byte = 1024*1024*1024*1024
+   * PiB - peta binary byte = 1024*1024*1024*1024*1024
+   * <p>
+   *
+   * @param nullValue to be returned when given number is null or empty
+   * @return nullValue if input is null or empty
+   * value of number
+   * @throws IAE if the input is invalid
+   */
+  public static long parse(String number, long nullValue)
+  {
+    if (number == null) {
+      return nullValue;
+    }
+
+    number = number.trim().toLowerCase(Locale.getDefault());

Review comment:
       Please use `StringUtils.toLowerCase()` instead.

##########
File path: indexing-service/src/main/java/org/apache/druid/indexing/overlord/config/RemoteTaskRunnerConfig.java
##########
@@ -81,15 +83,14 @@ public Period getTaskCleanupTimeout()
 
   public int getMaxZnodeBytes()
   {
-    return maxZnodeBytes;
+    return (int) maxZnodeBytes.getValue();

Review comment:
       How about adding `asInt()` which has a check for integer overflow?

##########
File path: core/src/test/java/org/apache/druid/java/util/common/BytesTest.java
##########
@@ -0,0 +1,300 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.java.util.common;
+
+import com.fasterxml.jackson.core.JsonProcessingException;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.junit.Assert;
+import org.junit.Test;
+
+import javax.validation.ConstraintViolation;
+import javax.validation.Validation;
+import javax.validation.Validator;
+import javax.validation.groups.Default;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Set;
+
+public class BytesTest
+{
+  @Test
+  public void testNumberString()
+  {
+    Assert.assertEquals(1, Bytes.parse("1", 0L));
+    Assert.assertEquals(10000000, Bytes.parse("10000000", 0));
+  }
+
+  @Test
+  public void testWithWhiteSpace()
+  {
+    Assert.assertEquals(12345, Bytes.parse(" 12345 ", 0));
+    Assert.assertEquals(12345, Bytes.parse("\t12345\t", 0));
+  }
+
+  @Test
+  public void testK()
+  {
+    Assert.assertEquals(1000, Bytes.parse("1k", 0));
+    Assert.assertEquals(1000, Bytes.parse("1K", 0));
+  }
+
+  @Test
+  public void testM()
+  {
+    Assert.assertEquals(1000_000, Bytes.parse("1m", 0));
+    Assert.assertEquals(1000_000, Bytes.parse("1M", 0));
+  }
+
+  @Test
+  public void testG()
+  {
+    Assert.assertEquals(1000_000_000, Bytes.parse("1g", 0));
+    Assert.assertEquals(1000_000_000, Bytes.parse("1G", 0));
+  }
+
+  @Test
+  public void testT()
+  {
+    Assert.assertEquals(1000_000_000_000L, Bytes.parse("1t", 0));
+    Assert.assertEquals(1000_000_000_000L, Bytes.parse("1T", 0));
+  }
+
+  @Test
+  public void testKiB()
+  {
+    Assert.assertEquals(1024, Bytes.parse("1kib", 0));
+    Assert.assertEquals(9 * 1024, Bytes.parse("9KiB", 0));
+    Assert.assertEquals(9 * 1024, Bytes.parse("9Kib", 0));
+  }
+
+  @Test
+  public void testMb()
+  {
+    Assert.assertEquals(1024 * 1024, Bytes.parse("1mib", 0));
+    Assert.assertEquals(9 * 1024 * 1024, Bytes.parse("9MiB", 0));
+    Assert.assertEquals(9 * 1024 * 1024, Bytes.parse("9Mib", 0));
+  }
+
+  @Test
+  public void testGiB()
+  {
+    Assert.assertEquals(1024 * 1024 * 1024, Bytes.parse("1gib", 0));
+    Assert.assertEquals(1024 * 1024 * 1024, Bytes.parse("1GiB", 0));
+    Assert.assertEquals(9L * 1024 * 1024 * 1024, Bytes.parse("9Gib", 0));
+  }
+
+  @Test
+  public void testTiB()
+  {
+    Assert.assertEquals(1024 * 1024 * 1024 * 1024L, Bytes.parse("1tib", 0));
+    Assert.assertEquals(9L * 1024 * 1024 * 1024 * 1024, Bytes.parse("9TiB", 0));
+    Assert.assertEquals(9L * 1024 * 1024 * 1024 * 1024, Bytes.parse("9Tib", 0));
+  }
+
+  @Test
+  public void testPiB()
+  {
+    Assert.assertEquals(1024L * 1024 * 1024 * 1024 * 1024, Bytes.parse("1pib", 0));
+    Assert.assertEquals(9L * 1024 * 1024 * 1024 * 1024 * 1024, Bytes.parse("9PiB", 0));
+    Assert.assertEquals(9L * 1024 * 1024 * 1024 * 1024 * 1024, Bytes.parse("9Pib", 0));
+  }
+
+  @Test
+  public void testDefault()
+  {
+    Assert.assertEquals(-123, Bytes.parse(" ", -123));
+    Assert.assertEquals(-456, Bytes.parse(null, -456));
+    Assert.assertEquals(-789, Bytes.parse("\t", -789));
+  }
+
+  @Test
+  public void testException()
+  {
+    try {
+      Bytes.parse("b", 0);
+      Assert.assertFalse("IAE should be thrown", true);

Review comment:
       nit: `ExpectedException` is preferred since you can verify the error message as well as the exception type. This is useful since even when the same type of exception is thrown, exceptions from different area of code can have different messages.

##########
File path: core/src/test/java/org/apache/druid/java/util/common/BytesTest.java
##########
@@ -0,0 +1,300 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.java.util.common;
+
+import com.fasterxml.jackson.core.JsonProcessingException;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import org.junit.Assert;
+import org.junit.Test;
+
+import javax.validation.ConstraintViolation;
+import javax.validation.Validation;
+import javax.validation.Validator;
+import javax.validation.groups.Default;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Set;
+
+public class BytesTest
+{
+  @Test
+  public void testNumberString()
+  {
+    Assert.assertEquals(1, Bytes.parse("1", 0L));
+    Assert.assertEquals(10000000, Bytes.parse("10000000", 0));
+  }
+
+  @Test
+  public void testWithWhiteSpace()
+  {
+    Assert.assertEquals(12345, Bytes.parse(" 12345 ", 0));
+    Assert.assertEquals(12345, Bytes.parse("\t12345\t", 0));
+  }
+
+  @Test
+  public void testK()
+  {
+    Assert.assertEquals(1000, Bytes.parse("1k", 0));
+    Assert.assertEquals(1000, Bytes.parse("1K", 0));
+  }
+
+  @Test
+  public void testM()
+  {
+    Assert.assertEquals(1000_000, Bytes.parse("1m", 0));
+    Assert.assertEquals(1000_000, Bytes.parse("1M", 0));
+  }
+
+  @Test
+  public void testG()
+  {
+    Assert.assertEquals(1000_000_000, Bytes.parse("1g", 0));
+    Assert.assertEquals(1000_000_000, Bytes.parse("1G", 0));
+  }
+
+  @Test
+  public void testT()
+  {
+    Assert.assertEquals(1000_000_000_000L, Bytes.parse("1t", 0));
+    Assert.assertEquals(1000_000_000_000L, Bytes.parse("1T", 0));
+  }
+
+  @Test
+  public void testKiB()
+  {
+    Assert.assertEquals(1024, Bytes.parse("1kib", 0));
+    Assert.assertEquals(9 * 1024, Bytes.parse("9KiB", 0));
+    Assert.assertEquals(9 * 1024, Bytes.parse("9Kib", 0));
+  }
+
+  @Test
+  public void testMb()
+  {
+    Assert.assertEquals(1024 * 1024, Bytes.parse("1mib", 0));
+    Assert.assertEquals(9 * 1024 * 1024, Bytes.parse("9MiB", 0));
+    Assert.assertEquals(9 * 1024 * 1024, Bytes.parse("9Mib", 0));
+  }
+
+  @Test
+  public void testGiB()
+  {
+    Assert.assertEquals(1024 * 1024 * 1024, Bytes.parse("1gib", 0));
+    Assert.assertEquals(1024 * 1024 * 1024, Bytes.parse("1GiB", 0));
+    Assert.assertEquals(9L * 1024 * 1024 * 1024, Bytes.parse("9Gib", 0));
+  }
+
+  @Test
+  public void testTiB()
+  {
+    Assert.assertEquals(1024 * 1024 * 1024 * 1024L, Bytes.parse("1tib", 0));
+    Assert.assertEquals(9L * 1024 * 1024 * 1024 * 1024, Bytes.parse("9TiB", 0));
+    Assert.assertEquals(9L * 1024 * 1024 * 1024 * 1024, Bytes.parse("9Tib", 0));
+  }
+
+  @Test
+  public void testPiB()
+  {
+    Assert.assertEquals(1024L * 1024 * 1024 * 1024 * 1024, Bytes.parse("1pib", 0));
+    Assert.assertEquals(9L * 1024 * 1024 * 1024 * 1024 * 1024, Bytes.parse("9PiB", 0));
+    Assert.assertEquals(9L * 1024 * 1024 * 1024 * 1024 * 1024, Bytes.parse("9Pib", 0));
+  }
+
+  @Test
+  public void testDefault()
+  {
+    Assert.assertEquals(-123, Bytes.parse(" ", -123));
+    Assert.assertEquals(-456, Bytes.parse(null, -456));
+    Assert.assertEquals(-789, Bytes.parse("\t", -789));
+  }
+
+  @Test
+  public void testException()
+  {
+    try {
+      Bytes.parse("b", 0);
+      Assert.assertFalse("IAE should be thrown", true);
+    }
+    catch (IAE e) {
+      Assert.assertTrue(true);
+    }
+
+    try {
+      Bytes.parse("1 b", 0);
+      Assert.assertFalse("IAE should be thrown", true);
+    }
+    catch (IAE e) {
+      Assert.assertTrue(true);
+    }
+
+    try {
+      Bytes.parse("tb", 0);
+      Assert.assertFalse("IAE should be thrown", true);
+    }
+    catch (IAE e) {
+      Assert.assertTrue(true);
+    }
+
+    try {
+      Bytes.parse("1 mb", 0);
+      Assert.assertFalse("IAE should be thrown", true);
+    }
+    catch (IAE e) {
+      Assert.assertTrue(true);
+    }
+
+    try {
+      Bytes.parse("gb", 0);
+      Assert.assertFalse("IAE should be thrown", true);
+    }
+    catch (IAE e) {
+      Assert.assertTrue(true);
+    }
+
+    try {
+      Bytes.parse("tb", 0);
+      Assert.assertFalse("IAE should be thrown", true);
+    }
+    catch (IAE e) {
+      Assert.assertTrue(true);
+    }
+  }
+
+  @Test
+  public void testOverflow()
+  {
+    try {
+      Bytes.parse("123456789123456789123", 0);
+      Assert.assertFalse("IAE should be thrown", true);
+    }
+    catch (IAE e) {
+    }
+
+    try {
+      String max = (Long.MAX_VALUE / 1000 + 1) + "k";
+      Bytes.parse(max, 0);
+      Assert.assertFalse("IAE should be thrown", true);
+    }
+    catch (IAE e) {
+    }
+
+    try {
+      String max = (Long.MAX_VALUE / 1000_000 + 1) + "m";
+      Bytes.parse(max, 0);
+      Assert.assertFalse("IAE should be thrown", true);
+    }
+    catch (IAE e) {
+    }
+
+    try {
+      String max = (Long.MAX_VALUE / 1000_000_000L + 1) + "g";
+      Bytes.parse(max, 0);
+      Assert.assertFalse("IAE should be thrown", true);
+    }
+    catch (IAE e) {
+    }
+
+    try {
+      String max = (Long.MAX_VALUE / 1000_000_000_000L + 1) + "t";
+      Bytes.parse(max, 0);
+      Assert.assertFalse("IAE should be thrown", true);
+    }
+    catch (IAE e) {
+    }
+
+    try {
+      String max = (Long.MAX_VALUE / 1024 + 1) + "kb";
+      Bytes.parse(max, 0);
+      Assert.assertFalse("IAE should be thrown", true);
+    }
+    catch (IAE e) {
+    }
+
+    try {
+      String max = (Long.MAX_VALUE / (1024 * 1024) + 1) + "mb";
+      Bytes.parse(max, 0);
+      Assert.assertFalse("IAE should be thrown", true);
+    }
+    catch (IAE e) {
+    }
+
+    try {
+      String max = (Long.MAX_VALUE / (1024L * 1024 * 1024) + 1) + "gb";
+      Bytes.parse(max, 0);
+      Assert.assertFalse("IAE should be thrown", true);
+    }
+    catch (IAE e) {
+    }
+
+    try {
+      String max = (Long.MAX_VALUE / (1024L * 1024 * 1024 * 1024) + 1) + "tb";
+      Bytes.parse(max, 0);
+      Assert.assertFalse("IAE should be thrown", true);
+    }
+    catch (IAE e) {
+    }
+  }
+
+  @Test
+  public void testJSON() throws JsonProcessingException
+  {
+    ObjectMapper mapper = new ObjectMapper();
+    Bytes bytes = new Bytes("5m");
+    String serialized = mapper.writeValueAsString(bytes);
+    Bytes deserialized = mapper.readValue(serialized, Bytes.class);
+    Assert.assertEquals(bytes, deserialized);
+  }
+
+  public static class BytesRangeTest

Review comment:
       We usually name a test class to have the `Test` suffix if it has unit tests. How about `TestBytesRange`?

##########
File path: core/src/main/java/org/apache/druid/java/util/common/Bytes.java
##########
@@ -0,0 +1,198 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.java.util.common;
+
+import com.fasterxml.jackson.databind.annotation.JsonSerialize;
+
+import java.io.Serializable;
+import java.util.Locale;
+
+@JsonSerialize(using = BytesSerializer.class)
+public class Bytes implements Serializable
+{
+  public static final Bytes ZERO = new Bytes(0L);
+
+  private long value;
+
+  public Bytes(String value)
+  {
+    this.value = Bytes.parse(value);
+  }
+
+  public Bytes(long value)
+  {
+    this.value = value;
+  }
+
+  public long getValue()
+  {
+    return value;
+  }
+
+  @Override
+  public boolean equals(Object thatObj)
+  {
+    if (thatObj == null) {
+      return false;
+    }
+    if (thatObj instanceof Bytes) {
+      return value == ((Bytes) thatObj).value;
+    } else {
+      return false;
+    }
+  }
+
+  @Override
+  public int hashCode()
+  {
+    return Long.hashCode(value);
+  }
+
+  @Override
+  public String toString()
+  {
+    return String.valueOf(value);
+  }
+
+  public static Bytes valueOf(int bytes)
+  {
+    return new Bytes(bytes);
+  }
+
+  public static Bytes valueOf(long bytes)
+  {
+    return new Bytes(bytes);
+  }
+
+  public static long parse(String number)
+  {
+    if (number == null) {
+      throw new IAE("number is null");

Review comment:
       All error messages should start with a capital letter. Please fix other places too.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] FrankChen021 commented on pull request #10203: Support unit on byte-related properties

Posted by GitBox <gi...@apache.org>.
FrankChen021 commented on pull request #10203:
URL: https://github.com/apache/druid/pull/10203#issuecomment-666903817


   @jihoonson @asdf2014 Thank you for your detailed review! 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] FrankChen021 commented on a change in pull request #10203: Support unit on byte-related properties

Posted by GitBox <gi...@apache.org>.
FrankChen021 commented on a change in pull request #10203:
URL: https://github.com/apache/druid/pull/10203#discussion_r460713315



##########
File path: indexing-service/src/main/java/org/apache/druid/indexing/overlord/config/RemoteTaskRunnerConfig.java
##########
@@ -81,15 +83,14 @@ public Period getTaskCleanupTimeout()
 
   public int getMaxZnodeBytes()
   {
-    return maxZnodeBytes;
+    return (int) maxZnodeBytes.getValue();

Review comment:
       A new `getBytesInInt` is provided to check the potential overflow.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] asdf2014 commented on a change in pull request #10203: Support unit on byte-related properties

Posted by GitBox <gi...@apache.org>.
asdf2014 commented on a change in pull request #10203:
URL: https://github.com/apache/druid/pull/10203#discussion_r462263540



##########
File path: core/src/main/java/org/apache/druid/java/util/common/HumanReadableBytes.java
##########
@@ -0,0 +1,208 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.java.util.common;
+
+import com.fasterxml.jackson.databind.annotation.JsonSerialize;
+
+@JsonSerialize(using = HumanReadableBytesSerializer.class)
+public class HumanReadableBytes
+{
+  public static final HumanReadableBytes ZERO = new HumanReadableBytes(0L);
+
+  private final long bytes;
+
+  public HumanReadableBytes(String bytes)
+  {
+    this.bytes = HumanReadableBytes.parse(bytes);
+  }
+
+  public HumanReadableBytes(long bytes)
+  {
+    this.bytes = bytes;
+  }
+
+  public long getBytes()
+  {
+    return bytes;
+  }
+
+  public int getBytesInInt()
+  {
+    if (bytes > Integer.MAX_VALUE) {
+      throw new ISE("Number overflow");
+    }
+
+    return (int) bytes;
+  }
+
+  @Override
+  public boolean equals(Object thatObj)
+  {
+    if (thatObj == null) {
+      return false;
+    }
+    if (thatObj instanceof HumanReadableBytes) {
+      return bytes == ((HumanReadableBytes) thatObj).bytes;
+    } else {
+      return false;
+    }
+  }
+
+  @Override
+  public int hashCode()
+  {
+    return Long.hashCode(bytes);
+  }
+
+  @Override
+  public String toString()
+  {
+    return String.valueOf(bytes);
+  }
+
+  public static HumanReadableBytes valueOf(int bytes)
+  {
+    return new HumanReadableBytes(bytes);
+  }
+
+  public static HumanReadableBytes valueOf(long bytes)
+  {
+    return new HumanReadableBytes(bytes);
+  }
+
+  public static long parse(String number)
+  {
+    if (number == null) {
+      throw new IAE("Invalid format of number: number is null");
+    }
+
+    number = number.trim();
+    if (number.length() == 0) {
+      throw new IAE("Invalid format of number: Number is empty");

Review comment:
       The error message is best to be consistent with the previous one. In addition, perhaps it is more appropriate to use `blank` instead of `empty` here.

##########
File path: docs/configuration/human-readable-byte.md
##########
@@ -0,0 +1,85 @@
+---
+id: human-readable-byte
+title: "Human-readable Byte Configuration Reference"
+---
+
+<!--
+  ~ Licensed to the Apache Software Foundation (ASF) under one
+  ~ or more contributor license agreements.  See the NOTICE file
+  ~ distributed with this work for additional information
+  ~ regarding copyright ownership.  The ASF licenses this file
+  ~ to you under the Apache License, Version 2.0 (the
+  ~ "License"); you may not use this file except in compliance
+  ~ with the License.  You may obtain a copy of the License at
+  ~
+  ~   http://www.apache.org/licenses/LICENSE-2.0
+  ~
+  ~ Unless required by applicable law or agreed to in writing,
+  ~ software distributed under the License is distributed on an
+  ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+  ~ KIND, either express or implied.  See the License for the
+  ~ specific language governing permissions and limitations
+  ~ under the License.
+  -->
+
+
+This page documents all configuration properties related to bytes.
+
+These properties can be configured through 2 ways: 
+1. a simple number in byte
+2. a number with a unit suffix
+
+## A number in byte
+
+Given the cache size is 3G, there's a configuration as below
+
+````

Review comment:
       There should not be four ` symbols but three.

##########
File path: docs/configuration/human-readable-byte.md
##########
@@ -0,0 +1,85 @@
+---
+id: human-readable-byte
+title: "Human-readable Byte Configuration Reference"
+---
+
+<!--
+  ~ Licensed to the Apache Software Foundation (ASF) under one
+  ~ or more contributor license agreements.  See the NOTICE file
+  ~ distributed with this work for additional information
+  ~ regarding copyright ownership.  The ASF licenses this file
+  ~ to you under the Apache License, Version 2.0 (the
+  ~ "License"); you may not use this file except in compliance
+  ~ with the License.  You may obtain a copy of the License at
+  ~
+  ~   http://www.apache.org/licenses/LICENSE-2.0
+  ~
+  ~ Unless required by applicable law or agreed to in writing,
+  ~ software distributed under the License is distributed on an
+  ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+  ~ KIND, either express or implied.  See the License for the
+  ~ specific language governing permissions and limitations
+  ~ under the License.
+  -->
+
+
+This page documents all configuration properties related to bytes.
+
+These properties can be configured through 2 ways: 
+1. a simple number in byte
+2. a number with a unit suffix
+
+## A number in byte
+
+Given the cache size is 3G, there's a configuration as below
+
+````
+druid.cache.sizeInBytes=3000000000 #3 * 1000_000_000
+````
+
+
+## A number with a unit suffix
+
+Sometimes value in bytes could be very large in Druid, it's counterintuitive to set value of these properties as above.
+Here comes another way, a number with a unit suffix.
+
+Given the total size of disks is 1T, the configuration can be
+
+````

Review comment:
       The type of code block should be specified as `properties`. The others should be the same.

##########
File path: website/.spelling
##########
@@ -1771,3 +1774,17 @@ CVE-2019-17571
 CVE-2019-12399
 CVE-2018-17196
 bin.tar.gz
+ - ../docs/configuration/human-readable-byte.md
+3G
+1000_000
+1000_000_000
+1000_000_000_000
+1000_000_000_000_000
+Giga
+Tera
+Peta
+KiB
+MiB
+GiB
+TiB
+PiB

Review comment:
       Please keep this blank line.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] jihoonson commented on a change in pull request #10203: Support unit on byte-related properties

Posted by GitBox <gi...@apache.org>.
jihoonson commented on a change in pull request #10203:
URL: https://github.com/apache/druid/pull/10203#discussion_r462032214



##########
File path: docs/configuration/human-readable-byte.md
##########
@@ -0,0 +1,83 @@
+---
+id: human-readable-byte
+title: "Human-readable Byte Configuration Reference"
+---
+
+<!--
+  ~ Licensed to the Apache Software Foundation (ASF) under one
+  ~ or more contributor license agreements.  See the NOTICE file
+  ~ distributed with this work for additional information
+  ~ regarding copyright ownership.  The ASF licenses this file
+  ~ to you under the Apache License, Version 2.0 (the
+  ~ "License"); you may not use this file except in compliance
+  ~ with the License.  You may obtain a copy of the License at
+  ~
+  ~   http://www.apache.org/licenses/LICENSE-2.0
+  ~
+  ~ Unless required by applicable law or agreed to in writing,
+  ~ software distributed under the License is distributed on an
+  ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+  ~ KIND, either express or implied.  See the License for the
+  ~ specific language governing permissions and limitations
+  ~ under the License.
+  -->
+
+
+This page documents all configuration properties related to bytes.
+
+These properties can be configured through 2 ways: 
+1. a simple number in byte
+2. a number with a unit suffix
+
+## A number in byte
+
+Given the cache size is 3G, there's a configuration as below
+
+````
+druid.cache.sizeInBytes=3000000000 #3 * 1000_000_000
+````
+
+
+## A number with a unit suffix
+
+Sometimes value in bytes could be very large in Druid, it's counterintuitive to set value of these properties as above.
+Here comes another way, a number with a unit suffix.
+
+Given the total size of disks is 1T, the configuration can be
+
+````
+druid.segmentCache.locations=[{"path":"/segment-cache","maxSize":"1g"}]
+````
+
+### Supported Units
+In the world of computer, a unit like `K` is ambiguous. It means 1000 or 1024 in different contexts, for more information please see [Here](https://en.wikipedia.org/wiki/Binary_prefix).
+
+To make it clear, the base of units are defined as below
+
+| Unit | Description | Base |
+|---|---|---|
+| K | Kilo Decimal Byte | 1000 |

Review comment:
       That's true, but I'm not sure how much it matters since `KB` and `KiB` are approximately same and so there would be not much difference no matter what K means. Also, I bet there will be more people who try setting some value before reading the doc. It would be nice if it just works whatever they put.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] FrankChen021 commented on a change in pull request #10203: Support unit on byte-related properties

Posted by GitBox <gi...@apache.org>.
FrankChen021 commented on a change in pull request #10203:
URL: https://github.com/apache/druid/pull/10203#discussion_r460712932



##########
File path: core/src/main/java/org/apache/druid/java/util/common/Bytes.java
##########
@@ -0,0 +1,198 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.java.util.common;
+
+import com.fasterxml.jackson.databind.annotation.JsonSerialize;
+
+import java.io.Serializable;
+import java.util.Locale;
+
+@JsonSerialize(using = BytesSerializer.class)
+public class Bytes implements Serializable
+{
+  public static final Bytes ZERO = new Bytes(0L);
+
+  private long value;
+
+  public Bytes(String value)
+  {
+    this.value = Bytes.parse(value);
+  }
+
+  public Bytes(long value)
+  {
+    this.value = value;
+  }
+
+  public long getValue()
+  {
+    return value;
+  }
+
+  @Override
+  public boolean equals(Object thatObj)
+  {
+    if (thatObj == null) {
+      return false;
+    }
+    if (thatObj instanceof Bytes) {
+      return value == ((Bytes) thatObj).value;
+    } else {
+      return false;
+    }
+  }
+
+  @Override
+  public int hashCode()
+  {
+    return Long.hashCode(value);
+  }
+
+  @Override
+  public String toString()
+  {
+    return String.valueOf(value);
+  }
+
+  public static Bytes valueOf(int bytes)
+  {
+    return new Bytes(bytes);
+  }
+
+  public static Bytes valueOf(long bytes)
+  {
+    return new Bytes(bytes);
+  }
+
+  public static long parse(String number)
+  {
+    if (number == null) {
+      throw new IAE("number is null");
+    }
+
+    number = number.trim().toLowerCase(Locale.getDefault());
+    int len = number.length();
+    if (len == 0) {
+      throw new IAE("number is empty");
+    }
+
+    return parseInner(number);
+  }
+
+  /**
+   * parse the case-insensitive string number, which is either:
+   * <p>
+   * a number string
+   * <p>
+   * or
+   * <p>
+   * a number string with a suffix which indicates the unit the of number
+   * the unit must be one of following
+   * k - kilobyte = 1000
+   * m - megabyte = 1,000,000
+   * g - gigabyte = 1,000,000,000
+   * t - terabyte = 1,000,000,000,000
+   * p - petabyte = 1,000,000,000,000,000
+   * KiB - kilo binary byte = 1024
+   * MiB - mega binary byte = 1024*1204
+   * GiB - giga binary byte = 1024*1024*1024
+   * TiB - tera binary byte = 1024*1024*1024*1024
+   * PiB - peta binary byte = 1024*1024*1024*1024*1024
+   * <p>
+   *
+   * @param nullValue to be returned when given number is null or empty
+   * @return nullValue if input is null or empty
+   * value of number
+   * @throws IAE if the input is invalid
+   */
+  public static long parse(String number, long nullValue)
+  {
+    if (number == null) {
+      return nullValue;
+    }
+
+    number = number.trim().toLowerCase(Locale.getDefault());
+    if (number.length() == 0) {
+      return nullValue;
+    }
+    return parseInner(number);
+  }
+
+  private static long parseInner(String number)
+  {
+    int index = number.length() - 1;
+    boolean isBinary = false;
+    char unit = number.charAt(index--);
+    if (unit == 'b') {
+      if (index < 2) {
+        throw new IAE("invalid format of number[%s]", number);
+      }
+      if (number.charAt(index--) != 'i') {
+        throw new IAE("invalid format of number[%s]", number);
+      }
+
+      unit = number.charAt(index--);
+      isBinary = true;
+    }
+    long base = 1;
+    switch (unit) {
+      case 'k':
+        base = isBinary ? 1024 : 1_000;
+        break;
+
+      case 'm':
+        base = isBinary ? 1024 * 1024 : 1_000_000;
+        break;
+
+      case 'g':
+        base = isBinary ? 1024 * 1024 * 1024 : 1_000_000_000;
+        break;
+
+      case 't':
+        base = isBinary ? 1024 * 1024 * 1024 * 1024L : 1_000_000_000_000L;
+        break;
+
+      case 'p':
+        base = isBinary ? 1024L * 1024 * 1024 * 1024 * 1024 : 1_000_000_000_000_000L;
+        break;
+
+      default:
+        if (!Character.isDigit(unit)) {
+          throw new IAE("invalid character in number[%s]", number);
+        }
+        break;
+    }
+
+    try {
+      if (base > 1 && index >= 0) {
+        long value = Long.parseLong(number.substring(0, index + 1)) * base;
+        if (value < base) {
+          throw new IAE("number [%s] overflow", number);
+        }
+        return value;
+      } else {
+        return Long.parseLong(number);

Review comment:
       Overflow has been checked inside `parseLong` . If overflowed, `NumberFormatException` will be thrown. I've add a comment in the new code.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] jihoonson commented on a change in pull request #10203: Support unit on byte-related properties

Posted by GitBox <gi...@apache.org>.
jihoonson commented on a change in pull request #10203:
URL: https://github.com/apache/druid/pull/10203#discussion_r459856645



##########
File path: indexing-service/src/main/java/org/apache/druid/indexing/overlord/config/RemoteTaskRunnerConfig.java
##########
@@ -81,15 +83,14 @@ public Period getTaskCleanupTimeout()
 
   public int getMaxZnodeBytes()
   {
-    return maxZnodeBytes;
+    return (int) maxZnodeBytes.getValue();

Review comment:
       This wouldn't cause integer overflow because the max is limited as `Integer.MAX_VALUE`, but this seems fragile and easy to make a mistake. How about adding `asInt()` which has a check for integer overflow?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org