You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by GitBox <gi...@apache.org> on 2020/12/26 17:24:33 UTC
[GitHub] [incubator-pinot] yupeng9 opened a new pull request #6387: Allow multiple H3 index resolutions
yupeng9 opened a new pull request #6387:
URL: https://github.com/apache/incubator-pinot/pull/6387
Encode the index resolutions (which can be multiple) in two bytes, and save them in the index header. This allows multiple H3 index resolution creation in the index.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] yupeng9 commented on a change in pull request #6387: Allow multiple H3 index resolutions
Posted by GitBox <gi...@apache.org>.
yupeng9 commented on a change in pull request #6387:
URL: https://github.com/apache/incubator-pinot/pull/6387#discussion_r549013820
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/segment/creator/impl/geospatial/H3IndexCreator.java
##########
@@ -172,6 +175,7 @@ public void seal()
//write header file
headerStream.writeInt(VERSION);
headerStream.writeInt(writer.getNumUniqueIds());
+ headerStream.writeShort(_resolution.size());
Review comment:
Good catch
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] kishoreg commented on a change in pull request #6387: Allow multiple H3 index resolutions
Posted by GitBox <gi...@apache.org>.
kishoreg commented on a change in pull request #6387:
URL: https://github.com/apache/incubator-pinot/pull/6387#discussion_r549013323
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/segment/creator/impl/geospatial/H3IndexResolution.java
##########
@@ -0,0 +1,53 @@
+package org.apache.pinot.core.segment.creator.impl.geospatial;
+
+import java.util.ArrayList;
+import java.util.List;
+
+
+/**
+ * Stores the resolutions for an index. There are in total of H3 resolutions https://h3geo.org/#/documentation/core-library/resolution-table
+ * To efficiently serialize the resolutions, we use two bytes for encoding th enabled resolutions. The resolution level
+ * maps to the corresponding bit.
+ */
+public class H3IndexResolution {
+ private short _resolutions;
Review comment:
use INT
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/segment/creator/impl/geospatial/H3IndexCreator.java
##########
@@ -172,6 +175,7 @@ public void seal()
//write header file
headerStream.writeInt(VERSION);
headerStream.writeInt(writer.getNumUniqueIds());
+ headerStream.writeShort(_resolution.size());
Review comment:
why size, we should write the serialized INT right?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] kishoreg commented on a change in pull request #6387: Allow multiple H3 index resolutions
Posted by GitBox <gi...@apache.org>.
kishoreg commented on a change in pull request #6387:
URL: https://github.com/apache/incubator-pinot/pull/6387#discussion_r549013279
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/segment/creator/impl/geospatial/H3IndexResolution.java
##########
@@ -0,0 +1,53 @@
+package org.apache.pinot.core.segment.creator.impl.geospatial;
+
+import java.util.ArrayList;
+import java.util.List;
+
+
+/**
+ * Stores the resolutions for an index. There are in total of H3 resolutions https://h3geo.org/#/documentation/core-library/resolution-table
+ * To efficiently serialize the resolutions, we use two bytes for encoding th enabled resolutions. The resolution level
+ * maps to the corresponding bit.
+ */
+public class H3IndexResolution {
+ private short _resolutions;
+
+ public H3IndexResolution(List<Integer> resolutions) {
+ for (int resolution : resolutions) {
+ _resolutions |= 1 << resolution;
+ }
+ }
+
+ /**
+ * Creates the resolutions with the serialized short value
+ * @param resolutions
+ */
+ public H3IndexResolution(short resolutions) {
+ _resolutions = resolutions;
+ }
+
+ /**
+ * @return the encoding of the resolutions into a short value (two bytes)
+ */
+ public short serialize() {
Review comment:
make it int
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] kishoreg merged pull request #6387: Allow multiple H3 index resolutions
Posted by GitBox <gi...@apache.org>.
kishoreg merged pull request #6387:
URL: https://github.com/apache/incubator-pinot/pull/6387
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] yupeng9 commented on a change in pull request #6387: Allow multiple H3 index resolutions
Posted by GitBox <gi...@apache.org>.
yupeng9 commented on a change in pull request #6387:
URL: https://github.com/apache/incubator-pinot/pull/6387#discussion_r549013755
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/segment/creator/impl/geospatial/H3IndexResolution.java
##########
@@ -0,0 +1,53 @@
+package org.apache.pinot.core.segment.creator.impl.geospatial;
+
+import java.util.ArrayList;
+import java.util.List;
+
+
+/**
+ * Stores the resolutions for an index. There are in total of H3 resolutions https://h3geo.org/#/documentation/core-library/resolution-table
+ * To efficiently serialize the resolutions, we use two bytes for encoding th enabled resolutions. The resolution level
+ * maps to the corresponding bit.
+ */
+public class H3IndexResolution {
+ private short _resolutions;
Review comment:
I feel short is easier since all resolutions can be represented in two bytes, so I don't have to reason about the first leftmost two bytes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org