You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by GitBox <gi...@apache.org> on 2021/01/03 00:48:29 UTC
[GitHub] [incubator-pinot] yupeng9 opened a new pull request #6400: Add real-time H3 index reader
yupeng9 opened a new pull request #6400:
URL: https://github.com/apache/incubator-pinot/pull/6400
Add the H3 index support to real-time segments.
Updated the RealTimeQuickStart example to utilize the H3 index. For example
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] kishoreg commented on a change in pull request #6400: Add real-time H3 index reader
Posted by GitBox <gi...@apache.org>.
kishoreg commented on a change in pull request #6400:
URL: https://github.com/apache/incubator-pinot/pull/6400#discussion_r550945424
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/realtime/impl/geospatial/RealtimeH3IndexReader.java
##########
@@ -0,0 +1,63 @@
+package org.apache.pinot.core.realtime.impl.geospatial;
+
+import com.uber.h3core.H3Core;
+import java.io.IOException;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import org.apache.pinot.core.realtime.impl.ThreadSafeMutableRoaringBitmap;
+import org.apache.pinot.core.segment.creator.GeoSpatialIndexCreator;
+import org.apache.pinot.core.segment.creator.impl.geospatial.H3IndexResolution;
+import org.apache.pinot.core.segment.index.readers.H3IndexReader;
+import org.roaringbitmap.buffer.ImmutableRoaringBitmap;
+import org.roaringbitmap.buffer.MutableRoaringBitmap;
+
+
+/**
+ * A H3 index reader for the real-time H3 index values on the fly.
+ * <p>This class is thread-safe for single writer multiple readers.
+ */
+public class RealtimeH3IndexReader implements GeoSpatialIndexCreator, H3IndexReader {
+ private final H3Core _h3Core;
+ private final Map<Long, ThreadSafeMutableRoaringBitmap> _h3IndexMap = new ConcurrentHashMap<>();
+ private final H3IndexResolution _resolution;
+ private int _lowestResolution;
+
+ public RealtimeH3IndexReader(List<Integer> resolutions)
Review comment:
why multiple constructors? lets stick with H3IndexConfig?
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/realtime/impl/geospatial/RealtimeH3IndexReader.java
##########
@@ -0,0 +1,63 @@
+package org.apache.pinot.core.realtime.impl.geospatial;
+
+import com.uber.h3core.H3Core;
+import java.io.IOException;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import org.apache.pinot.core.realtime.impl.ThreadSafeMutableRoaringBitmap;
+import org.apache.pinot.core.segment.creator.GeoSpatialIndexCreator;
+import org.apache.pinot.core.segment.creator.impl.geospatial.H3IndexResolution;
+import org.apache.pinot.core.segment.index.readers.H3IndexReader;
+import org.roaringbitmap.buffer.ImmutableRoaringBitmap;
+import org.roaringbitmap.buffer.MutableRoaringBitmap;
+
+
+/**
+ * A H3 index reader for the real-time H3 index values on the fly.
+ * <p>This class is thread-safe for single writer multiple readers.
+ */
+public class RealtimeH3IndexReader implements GeoSpatialIndexCreator, H3IndexReader {
+ private final H3Core _h3Core;
+ private final Map<Long, ThreadSafeMutableRoaringBitmap> _h3IndexMap = new ConcurrentHashMap<>();
Review comment:
may be start with something bigger depending on the resolution to avoid resizes
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/segment/index/column/ColumnIndexContainer.java
##########
@@ -51,7 +51,7 @@
/**
* Returns the H3 index for the column, or {@code null} if it does not exist.
*/
- H3IndexReader getH3Index();
+ ImmutableH3IndexReader getH3Index();
Review comment:
why is this change needed?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] kishoreg merged pull request #6400: Add real-time H3 index reader
Posted by GitBox <gi...@apache.org>.
kishoreg merged pull request #6400:
URL: https://github.com/apache/incubator-pinot/pull/6400
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] yupeng9 commented on a change in pull request #6400: Add real-time H3 index reader
Posted by GitBox <gi...@apache.org>.
yupeng9 commented on a change in pull request #6400:
URL: https://github.com/apache/incubator-pinot/pull/6400#discussion_r550962098
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/realtime/impl/geospatial/RealtimeH3IndexReader.java
##########
@@ -0,0 +1,63 @@
+package org.apache.pinot.core.realtime.impl.geospatial;
+
+import com.uber.h3core.H3Core;
+import java.io.IOException;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import org.apache.pinot.core.realtime.impl.ThreadSafeMutableRoaringBitmap;
+import org.apache.pinot.core.segment.creator.GeoSpatialIndexCreator;
+import org.apache.pinot.core.segment.creator.impl.geospatial.H3IndexResolution;
+import org.apache.pinot.core.segment.index.readers.H3IndexReader;
+import org.roaringbitmap.buffer.ImmutableRoaringBitmap;
+import org.roaringbitmap.buffer.MutableRoaringBitmap;
+
+
+/**
+ * A H3 index reader for the real-time H3 index values on the fly.
+ * <p>This class is thread-safe for single writer multiple readers.
+ */
+public class RealtimeH3IndexReader implements GeoSpatialIndexCreator, H3IndexReader {
+ private final H3Core _h3Core;
+ private final Map<Long, ThreadSafeMutableRoaringBitmap> _h3IndexMap = new ConcurrentHashMap<>();
Review comment:
there is a tradeoff of memory usage?
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/realtime/impl/geospatial/RealtimeH3IndexReader.java
##########
@@ -0,0 +1,63 @@
+package org.apache.pinot.core.realtime.impl.geospatial;
+
+import com.uber.h3core.H3Core;
+import java.io.IOException;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import org.apache.pinot.core.realtime.impl.ThreadSafeMutableRoaringBitmap;
+import org.apache.pinot.core.segment.creator.GeoSpatialIndexCreator;
+import org.apache.pinot.core.segment.creator.impl.geospatial.H3IndexResolution;
+import org.apache.pinot.core.segment.index.readers.H3IndexReader;
+import org.roaringbitmap.buffer.ImmutableRoaringBitmap;
+import org.roaringbitmap.buffer.MutableRoaringBitmap;
+
+
+/**
+ * A H3 index reader for the real-time H3 index values on the fly.
+ * <p>This class is thread-safe for single writer multiple readers.
+ */
+public class RealtimeH3IndexReader implements GeoSpatialIndexCreator, H3IndexReader {
+ private final H3Core _h3Core;
+ private final Map<Long, ThreadSafeMutableRoaringBitmap> _h3IndexMap = new ConcurrentHashMap<>();
+ private final H3IndexResolution _resolution;
+ private int _lowestResolution;
+
+ public RealtimeH3IndexReader(List<Integer> resolutions)
Review comment:
done
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/segment/index/column/ColumnIndexContainer.java
##########
@@ -51,7 +51,7 @@
/**
* Returns the H3 index for the column, or {@code null} if it does not exist.
*/
- H3IndexReader getH3Index();
+ ImmutableH3IndexReader getH3Index();
Review comment:
H3IndexReader is changed to interface
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] yupeng9 commented on pull request #6400: Add real-time H3 index reader
Posted by GitBox <gi...@apache.org>.
yupeng9 commented on pull request #6400:
URL: https://github.com/apache/incubator-pinot/pull/6400#issuecomment-753649503
@kishoreg Updated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] kishoreg commented on pull request #6400: Add real-time H3 index reader
Posted by GitBox <gi...@apache.org>.
kishoreg commented on pull request #6400:
URL: https://github.com/apache/incubator-pinot/pull/6400#issuecomment-753576708
@yupeng9 missing header
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org