You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by GitBox <gi...@apache.org> on 2022/06/14 04:48:35 UTC

[GitHub] [druid] imply-cheddar opened a new pull request, #12646: Poison stupid pool

imply-cheddar opened a new pull request, #12646:
URL: https://github.com/apache/druid/pull/12646

   There are various resource leaks from test setup as well as some
   corners in query processing.  We poison the StupidPool to start failing
   tests when the leaks come and fix any issues uncovered from that so
   that we can start from a clean baseline.
   
   Unfortunately, because of how poisoning works,
   we can only fail future checkouts from the same pool,
   which means that there is a natural race between a
   leak happening -> GC occurs -> leak detected -> pool poisoned.
   
   This race means that, depending on interleaving of tests,
   if the very last time that an object is checked out
   from the pool leaks, then it won't get caught.
   At some point in the future, something will catch it,
    however and from that point on it will be deterministic.
   
   
   This PR has:
   - [x] been self-reviewed.
   - [x] added unit tests or modified existing tests to cover new code paths, ensuring the threshold for [code coverage](https://github.com/apache/druid/blob/master/dev/code-review/code-coverage.md) is met.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] imply-cheddar commented on a diff in pull request #12646: Poison stupid pool

Posted by GitBox <gi...@apache.org>.
imply-cheddar commented on code in PR #12646:
URL: https://github.com/apache/druid/pull/12646#discussion_r899696209


##########
processing/src/main/java/org/apache/druid/segment/QueryableIndexCursorSequenceBuilder.java:
##########
@@ -50,51 +56,66 @@
 
 import javax.annotation.Nullable;
 import java.io.IOException;
-import java.util.HashMap;
-import java.util.Map;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
 
 public class QueryableIndexCursorSequenceBuilder
 {
   private final QueryableIndex index;
   private final Interval interval;
   private final VirtualColumns virtualColumns;
-  @Nullable
-  private final ImmutableBitmap filterBitmap;
+  private final Filter filter;
+  private final QueryMetrics<? extends Query> metrics;
   private final long minDataTimestamp;
   private final long maxDataTimestamp;
   private final boolean descending;
-  @Nullable
-  private final Filter postFilter;
-  @Nullable
-  private final ColumnSelectorColumnIndexSelector bitmapIndexSelector;
 
   public QueryableIndexCursorSequenceBuilder(
       QueryableIndex index,
       Interval interval,
       VirtualColumns virtualColumns,
-      @Nullable ImmutableBitmap filterBitmap,
+      @Nullable Filter filter,
+      @Nullable QueryMetrics<? extends Query> metrics,
       long minDataTimestamp,
       long maxDataTimestamp,
-      boolean descending,
-      @Nullable Filter postFilter,
-      @Nullable ColumnSelectorColumnIndexSelector bitmapIndexSelector
+      boolean descending
   )
   {
     this.index = index;
     this.interval = interval;
     this.virtualColumns = virtualColumns;
-    this.filterBitmap = filterBitmap;
+    this.filter = filter;
+    this.metrics = metrics;
     this.minDataTimestamp = minDataTimestamp;
     this.maxDataTimestamp = maxDataTimestamp;
     this.descending = descending;
-    this.postFilter = postFilter;
-    this.bitmapIndexSelector = bitmapIndexSelector;
   }
 
   public Sequence<Cursor> build(final Granularity gran)
   {
+    final Closer closer = Closer.create();
+
+    // Column caches shared amongst all cursors in this sequence.
+    final ColumnCache columnCache = new ColumnCache(index, closer);
+
     final Offset baseOffset;
 
+    final ColumnSelectorColumnIndexSelector bitmapIndexSelector = new ColumnSelectorColumnIndexSelector(
+        index.getBitmapFactoryForDimensions(),
+        virtualColumns,
+        columnCache

Review Comment:
   Hrm, actually, no, if I wrap the column objects here, then that will break the contract around the class of object being returned from the `getColumn()`.  We'd need to introduce an `as()` style method to be able to work around that, which is, again, expanding the scope of this PR a bit too much



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] gianm commented on a diff in pull request #12646: Poison stupid pool

Posted by GitBox <gi...@apache.org>.
gianm commented on code in PR #12646:
URL: https://github.com/apache/druid/pull/12646#discussion_r912234997


##########
processing/src/main/java/org/apache/druid/segment/DeprecatedQueryableIndexColumnSelector.java:
##########
@@ -0,0 +1,66 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.segment;
+
+import org.apache.druid.segment.column.ColumnHolder;
+
+import javax.annotation.Nullable;
+import java.util.List;
+
+/**
+ * It likely looks weird that we are creating a new instance of ColumnSelector here that begins its life deprecated
+ * and only delegates methods to the Queryable Index. This is done intentionally so that the QueryableIndex doesn't
+ * accidentally get used as a ColumnSelector.
+ *
+ * The lifecycle of the QueryableIndex is over the lifetime of the segment on a specific process, while
+ * the ColumnSelector's lifecycle is for a given query.  When we don't use the same ColumnSelector for an
+ * entire query, we defeat caching and use a lot more resources than necessary for queries.
+ *
+ * Places that use this class are intentionally circumventing column caching and column lifecycle management,
+ * ostensibly because those code locations know that they are only looking at metadata.  If a code path uses this
+ * and actually accesses a column instead of just looking at metadata, it will leak any resources that said column
+ * requires.
+ *
+ * The ColumnCache is the preferred implementation of a ColumnSelector, it takes a Closer and that closer can be used
+ * to ensure that resources are cleaned up.
+ */
+@Deprecated
+public class DeprecatedQueryableIndexColumnSelector implements ColumnSelector

Review Comment:
   Sounds reasonable.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] imply-cheddar commented on a diff in pull request #12646: Poison stupid pool

Posted by GitBox <gi...@apache.org>.
imply-cheddar commented on code in PR #12646:
URL: https://github.com/apache/druid/pull/12646#discussion_r899698434


##########
processing/src/main/java/org/apache/druid/segment/QueryableIndex.java:
##########
@@ -35,7 +38,7 @@
  * @see QueryableIndexStorageAdapter for query path adapter
  * @see QueryableIndexIndexableAdapter for indexing path adapter
  */
-public interface QueryableIndex extends ColumnSelector, Closeable
+public interface QueryableIndex extends Closeable

Review Comment:
   It was not considered.  It might work and might not.  I don't remember places that were creating a cache just to throw it away, IIRC, they were all created in places that could actually have resources that need cleanup, so I think it's fine as is.  If not, I'd put that cleanup in scope for another PR.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] gianm merged pull request #12646: Poison stupid pool

Posted by GitBox <gi...@apache.org>.
gianm merged PR #12646:
URL: https://github.com/apache/druid/pull/12646


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] clintropolis commented on a diff in pull request #12646: Poison stupid pool

Posted by GitBox <gi...@apache.org>.
clintropolis commented on code in PR #12646:
URL: https://github.com/apache/druid/pull/12646#discussion_r896379882


##########
processing/src/main/java/org/apache/druid/segment/QueryableIndex.java:
##########
@@ -35,7 +38,7 @@
  * @see QueryableIndexStorageAdapter for query path adapter
  * @see QueryableIndexIndexableAdapter for indexing path adapter
  */
-public interface QueryableIndex extends ColumnSelector, Closeable
+public interface QueryableIndex extends Closeable

Review Comment:
   since this still implements `getColumnCapabilities` should it still extend `ColumnInspector`? it might save some of the churn of making a column cache for things that only needed to get capabilities (like expressions trying to compute their output type). Otoh it doesn't really seem necessary either, mostly just wondering if it was considered.



##########
processing/src/main/java/org/apache/druid/query/timeseries/TimeseriesQueryEngine.java:
##########
@@ -161,7 +161,7 @@ private Sequence<Result<TimeseriesResultValue>> processVectorized(
       );
 
       if (granularizer == null) {
-        return Sequences.empty();
+        return Sequences.withBaggage(Sequences.empty(), closer);

Review Comment:
   nice catch :+1:



##########
core/pom.xml:
##########
@@ -430,12 +430,6 @@
         <configuration>
           <!-- use normal classpath instead of manifest jar for JvmUtilsTest.testSystemClassPath -->
           <useManifestOnlyJar>false</useManifestOnlyJar>
-          <argLine>
-            @{jacocoArgLine}
-            ${jdk.surefire.argLine}
-            -Djava.library.path=${project.build.directory}/hyperic-sigar-${sigar.base.version}/sigar-bin/lib/

Review Comment:
   i guess this line didn't matter? (i don't see it in parent pom)



##########
processing/src/main/java/org/apache/druid/segment/ColumnCache.java:
##########
@@ -0,0 +1,119 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.segment;
+
+import org.apache.druid.java.util.common.io.Closer;
+import org.apache.druid.segment.column.BaseColumn;
+import org.apache.druid.segment.column.ColumnCapabilities;
+import org.apache.druid.segment.column.ColumnHolder;
+import org.apache.druid.segment.column.ColumnIndexSupplier;
+import org.apache.druid.segment.selector.settable.SettableColumnValueSelector;
+
+import javax.annotation.Nullable;
+import java.util.HashMap;
+import java.util.List;
+
+public class ColumnCache implements ColumnSelector
+{
+  private final HashMap<String, ColumnHolder> holderCache;
+  private final QueryableIndex index;
+  private final Closer closer;
+
+  public ColumnCache(QueryableIndex index, Closer closer)
+  {
+    this.index = index;
+    this.closer = closer;
+
+    this.holderCache = new HashMap<>();
+  }
+
+
+  @Override
+  public List<String> getColumnNames()
+  {
+    return index.getColumnNames();
+  }
+
+  @Nullable
+  @Override
+  public ColumnHolder getColumnHolder(String columnName)
+  {
+    return holderCache.computeIfAbsent(columnName, dimension -> {
+      // Here we do a funny little dance to memoize the BaseColumn and register it with the closer.
+      // It would probably be cleaner if the ColumnHolder itself was `Closeable` and did its own memoization,
+      // but that change is much wider and runs the risk of even more things that need to close the thing
+      // not actually closing it.  So, maybe this is a hack, maybe it's a wise decision, who knows, but at
+      // least for now, we grab the holder, grab the column, register the column with the closer and then return
+      // a new holder that always returns the same reference for the column.
+
+      final ColumnHolder holder = index.getColumnHolder(columnName);
+      if (holder == null) {
+        return null;
+      }
+
+      return new ColumnHolder()
+      {
+        @Nullable
+        private BaseColumn theColumn = null;
+
+        @Override
+        public ColumnCapabilities getCapabilities()
+        {
+          return holder.getCapabilities();
+        }
+
+        @Override
+        public int getLength()
+        {
+          return holder.getLength();
+        }
+
+        @Override
+        public BaseColumn getColumn()
+        {
+          if (theColumn == null) {
+            theColumn = closer.register(holder.getColumn());
+          }
+          return theColumn;
+        }
+
+        @Nullable
+        @Override
+        public ColumnIndexSupplier getIndexSupplier()
+        {
+          return holder.getIndexSupplier();

Review Comment:
   Afaik they currently don't have anything to close, but it makes sense to me to make them closable eventually so that they could if necessary. It could also possibly be part of the `BaseColumn`, though that might limit or require a different mechanism if we ever add indexes that are for multiple columns, and also requires consideration for how virtual columns would close their indexes since they don't currently close stuff, so maybe is better to just make the index supplier closable. I'm not sure it needs to be done in this PR though.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] imply-cheddar commented on a diff in pull request #12646: Poison stupid pool

Posted by GitBox <gi...@apache.org>.
imply-cheddar commented on code in PR #12646:
URL: https://github.com/apache/druid/pull/12646#discussion_r899685702


##########
pom.xml:
##########
@@ -1500,8 +1500,9 @@
                         <!-- set default options -->
                         <argLine>
                             @{jacocoArgLine}
+                            ${jdk.surefire.argLine}
                             -Xmx1500m
-                            -XX:MaxDirectMemorySize=512m
+                            -XX:MaxDirectMemorySize=2500m

Review Comment:
   Because one of the configs that I removed had it set to that value and, well, this setting is meaningless anyway (it has absolutely no functional impact on the process, all it does it set a threshold for when exceptions get thrown)



##########
pom.xml:
##########
@@ -1500,8 +1500,9 @@
                         <!-- set default options -->
                         <argLine>
                             @{jacocoArgLine}
+                            ${jdk.surefire.argLine}
                             -Xmx1500m
-                            -XX:MaxDirectMemorySize=512m
+                            -XX:MaxDirectMemorySize=2500m

Review Comment:
   Because one of the configs that I removed on a child pom had it set to that value and, well, this setting is meaningless anyway (it has absolutely no functional impact on the process, all it does it set a threshold for when exceptions get thrown)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] clintropolis commented on a diff in pull request #12646: Poison stupid pool

Posted by GitBox <gi...@apache.org>.
clintropolis commented on code in PR #12646:
URL: https://github.com/apache/druid/pull/12646#discussion_r910763365


##########
processing/src/main/java/org/apache/druid/segment/QueryableIndexCursorSequenceBuilder.java:
##########
@@ -50,51 +56,66 @@
 
 import javax.annotation.Nullable;
 import java.io.IOException;
-import java.util.HashMap;
-import java.util.Map;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
 
 public class QueryableIndexCursorSequenceBuilder
 {
   private final QueryableIndex index;
   private final Interval interval;
   private final VirtualColumns virtualColumns;
-  @Nullable
-  private final ImmutableBitmap filterBitmap;
+  private final Filter filter;
+  private final QueryMetrics<? extends Query> metrics;
   private final long minDataTimestamp;
   private final long maxDataTimestamp;
   private final boolean descending;
-  @Nullable
-  private final Filter postFilter;
-  @Nullable
-  private final ColumnSelectorColumnIndexSelector bitmapIndexSelector;
 
   public QueryableIndexCursorSequenceBuilder(
       QueryableIndex index,
       Interval interval,
       VirtualColumns virtualColumns,
-      @Nullable ImmutableBitmap filterBitmap,
+      @Nullable Filter filter,
+      @Nullable QueryMetrics<? extends Query> metrics,
       long minDataTimestamp,
       long maxDataTimestamp,
-      boolean descending,
-      @Nullable Filter postFilter,
-      @Nullable ColumnSelectorColumnIndexSelector bitmapIndexSelector
+      boolean descending
   )
   {
     this.index = index;
     this.interval = interval;
     this.virtualColumns = virtualColumns;
-    this.filterBitmap = filterBitmap;
+    this.filter = filter;
+    this.metrics = metrics;
     this.minDataTimestamp = minDataTimestamp;
     this.maxDataTimestamp = maxDataTimestamp;
     this.descending = descending;
-    this.postFilter = postFilter;
-    this.bitmapIndexSelector = bitmapIndexSelector;
   }
 
   public Sequence<Cursor> build(final Granularity gran)
   {
+    final Closer closer = Closer.create();
+
+    // Column caches shared amongst all cursors in this sequence.
+    final ColumnCache columnCache = new ColumnCache(index, closer);
+
     final Offset baseOffset;
 
+    final ColumnSelectorColumnIndexSelector bitmapIndexSelector = new ColumnSelectorColumnIndexSelector(
+        index.getBitmapFactoryForDimensions(),
+        virtualColumns,
+        columnCache

Review Comment:
   nothing to change now, but thinking out loud i wish there was a better way to get the number of rows than checking the time column, its not very accessible and seems a bit overkill for something so useful



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] gianm commented on a diff in pull request #12646: Poison stupid pool

Posted by GitBox <gi...@apache.org>.
gianm commented on code in PR #12646:
URL: https://github.com/apache/druid/pull/12646#discussion_r912234822


##########
processing/src/main/java/org/apache/druid/segment/QueryableIndexCursorSequenceBuilder.java:
##########
@@ -50,51 +56,66 @@
 
 import javax.annotation.Nullable;
 import java.io.IOException;
-import java.util.HashMap;
-import java.util.Map;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
 
 public class QueryableIndexCursorSequenceBuilder
 {
   private final QueryableIndex index;
   private final Interval interval;
   private final VirtualColumns virtualColumns;
-  @Nullable
-  private final ImmutableBitmap filterBitmap;
+  private final Filter filter;
+  private final QueryMetrics<? extends Query> metrics;
   private final long minDataTimestamp;
   private final long maxDataTimestamp;
   private final boolean descending;
-  @Nullable
-  private final Filter postFilter;
-  @Nullable
-  private final ColumnSelectorColumnIndexSelector bitmapIndexSelector;
 
   public QueryableIndexCursorSequenceBuilder(
       QueryableIndex index,
       Interval interval,
       VirtualColumns virtualColumns,
-      @Nullable ImmutableBitmap filterBitmap,
+      @Nullable Filter filter,
+      @Nullable QueryMetrics<? extends Query> metrics,
       long minDataTimestamp,
       long maxDataTimestamp,
-      boolean descending,
-      @Nullable Filter postFilter,
-      @Nullable ColumnSelectorColumnIndexSelector bitmapIndexSelector
+      boolean descending
   )
   {
     this.index = index;
     this.interval = interval;
     this.virtualColumns = virtualColumns;
-    this.filterBitmap = filterBitmap;
+    this.filter = filter;
+    this.metrics = metrics;
     this.minDataTimestamp = minDataTimestamp;
     this.maxDataTimestamp = maxDataTimestamp;
     this.descending = descending;
-    this.postFilter = postFilter;
-    this.bitmapIndexSelector = bitmapIndexSelector;
   }
 
   public Sequence<Cursor> build(final Granularity gran)
   {
+    final Closer closer = Closer.create();
+
+    // Column caches shared amongst all cursors in this sequence.
+    final ColumnCache columnCache = new ColumnCache(index, closer);
+
     final Offset baseOffset;
 
+    final ColumnSelectorColumnIndexSelector bitmapIndexSelector = new ColumnSelectorColumnIndexSelector(
+        index.getBitmapFactoryForDimensions(),
+        virtualColumns,
+        columnCache

Review Comment:
   Based on the conversation here I agree it's OK to leave it as-is, since the `__time` column is currently resilient to double-close, and the changes to avoid it add more scope than we want to bite off right now.
   
   Can you add a comment about this on `ColumnSelectorColumnIndexSelector.getNumRows`, @imply-cheddar, so whoever looks at this code in the future understands what's happening? Something that mentions that the column is being closed here, which will lead to a double-close if it comes from a ColumnCache, but that's fine because the `__time` column is expected to be able to handle a double-close.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] gianm commented on pull request #12646: Poison stupid pool

Posted by GitBox <gi...@apache.org>.
gianm commented on PR #12646:
URL: https://github.com/apache/druid/pull/12646#issuecomment-1173176835

   Nevermind, I committed this patch and did another one adding the comment as a follow-up: https://github.com/apache/druid/pull/12735


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] imply-cheddar commented on a diff in pull request #12646: Poison stupid pool

Posted by GitBox <gi...@apache.org>.
imply-cheddar commented on code in PR #12646:
URL: https://github.com/apache/druid/pull/12646#discussion_r899709313


##########
processing/src/main/java/org/apache/druid/query/timeseries/TimeseriesQueryEngine.java:
##########
@@ -161,7 +161,7 @@ private Sequence<Result<TimeseriesResultValue>> processVectorized(
       );
 
       if (granularizer == null) {
-        return Sequences.empty();
+        return Sequences.withBaggage(Sequences.empty(), closer);

Review Comment:
   🤷  close with the Sequence is how all of the other code works, so keeping it consistent.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] gianm commented on a diff in pull request #12646: Poison stupid pool

Posted by GitBox <gi...@apache.org>.
gianm commented on code in PR #12646:
URL: https://github.com/apache/druid/pull/12646#discussion_r897253582


##########
processing/src/main/java/org/apache/druid/segment/DeprecatedQueryableIndexColumnSelector.java:
##########
@@ -0,0 +1,66 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.segment;
+
+import org.apache.druid.segment.column.ColumnHolder;
+
+import javax.annotation.Nullable;
+import java.util.List;
+
+/**
+ * It likely looks weird that we are creating a new instance of ColumnSelector here that begins its life deprecated
+ * and only delegates methods to the Queryable Index. This is done intentionally so that the QueryableIndex doesn't
+ * accidentally get used as a ColumnSelector.
+ *
+ * The lifecycle of the QueryableIndex is over the lifetime of the segment on a specific process, while
+ * the ColumnSelector's lifecycle is for a given query.  When we don't use the same ColumnSelector for an
+ * entire query, we defeat caching and use a lot more resources than necessary for queries.
+ *
+ * Places that use this class are intentionally circumventing column caching and column lifecycle management,
+ * ostensibly because those code locations know that they are only looking at metadata.  If a code path uses this
+ * and actually accesses a column instead of just looking at metadata, it will leak any resources that said column
+ * requires.
+ *
+ * The ColumnCache is the preferred implementation of a ColumnSelector, it takes a Closer and that closer can be used
+ * to ensure that resources are cleaned up.
+ */
+@Deprecated
+public class DeprecatedQueryableIndexColumnSelector implements ColumnSelector

Review Comment:
   I'm wondering we can get away without having this class, especially after fixing the double-close issue with ColumnSelectorColumnIndexSelector. All but one of the usages of DeprecatedQueryableIndexColumnSelector are as an input to ColumnSelectorColumnIndexSelector. Once that's adjusted, it may be that we don't actually need this adapter anymore.
   
   The other usage is in a call to `Filter.supportsSelectivityEstimation` in AutoStrategy, where it would be fine (and preferable?) to pass in a ColumnCache instead of a DeprecatedQueryableIndexColumnSelector.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] imply-cheddar commented on a diff in pull request #12646: Poison stupid pool

Posted by GitBox <gi...@apache.org>.
imply-cheddar commented on code in PR #12646:
URL: https://github.com/apache/druid/pull/12646#discussion_r899697562


##########
processing/src/main/java/org/apache/druid/segment/DeprecatedQueryableIndexColumnSelector.java:
##########
@@ -0,0 +1,66 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.segment;
+
+import org.apache.druid.segment.column.ColumnHolder;
+
+import javax.annotation.Nullable;
+import java.util.List;
+
+/**
+ * It likely looks weird that we are creating a new instance of ColumnSelector here that begins its life deprecated
+ * and only delegates methods to the Queryable Index. This is done intentionally so that the QueryableIndex doesn't
+ * accidentally get used as a ColumnSelector.
+ *
+ * The lifecycle of the QueryableIndex is over the lifetime of the segment on a specific process, while
+ * the ColumnSelector's lifecycle is for a given query.  When we don't use the same ColumnSelector for an
+ * entire query, we defeat caching and use a lot more resources than necessary for queries.
+ *
+ * Places that use this class are intentionally circumventing column caching and column lifecycle management,
+ * ostensibly because those code locations know that they are only looking at metadata.  If a code path uses this
+ * and actually accesses a column instead of just looking at metadata, it will leak any resources that said column
+ * requires.
+ *
+ * The ColumnCache is the preferred implementation of a ColumnSelector, it takes a Closer and that closer can be used
+ * to ensure that resources are cleaned up.
+ */
+@Deprecated
+public class DeprecatedQueryableIndexColumnSelector implements ColumnSelector

Review Comment:
   The `AutoStrategy` one could definitely be adjusted, the other places are all significantly larger changes than the words used to describe them.  So, while I agree that I wish this didn't exist, the goal of the PR is to get the pool poisoned so that we are sure that future leaks get caught, I don't think we should block that goal on perfection here.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] imply-cheddar commented on a diff in pull request #12646: Poison stupid pool

Posted by GitBox <gi...@apache.org>.
imply-cheddar commented on code in PR #12646:
URL: https://github.com/apache/druid/pull/12646#discussion_r899687246


##########
processing/src/main/java/org/apache/druid/segment/ColumnSelector.java:
##########
@@ -29,6 +29,10 @@
  */
 public interface ColumnSelector extends ColumnInspector
 {
+  /**
+   * This method is apparently no longer used anymore, so deprecating it.
+   */
+  @Deprecated

Review Comment:
   Was just being conservative.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] imply-cheddar commented on a diff in pull request #12646: Poison stupid pool

Posted by GitBox <gi...@apache.org>.
imply-cheddar commented on code in PR #12646:
URL: https://github.com/apache/druid/pull/12646#discussion_r899694756


##########
processing/src/main/java/org/apache/druid/segment/QueryableIndexCursorSequenceBuilder.java:
##########
@@ -50,51 +56,66 @@
 
 import javax.annotation.Nullable;
 import java.io.IOException;
-import java.util.HashMap;
-import java.util.Map;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
 
 public class QueryableIndexCursorSequenceBuilder
 {
   private final QueryableIndex index;
   private final Interval interval;
   private final VirtualColumns virtualColumns;
-  @Nullable
-  private final ImmutableBitmap filterBitmap;
+  private final Filter filter;
+  private final QueryMetrics<? extends Query> metrics;
   private final long minDataTimestamp;
   private final long maxDataTimestamp;
   private final boolean descending;
-  @Nullable
-  private final Filter postFilter;
-  @Nullable
-  private final ColumnSelectorColumnIndexSelector bitmapIndexSelector;
 
   public QueryableIndexCursorSequenceBuilder(
       QueryableIndex index,
       Interval interval,
       VirtualColumns virtualColumns,
-      @Nullable ImmutableBitmap filterBitmap,
+      @Nullable Filter filter,
+      @Nullable QueryMetrics<? extends Query> metrics,
       long minDataTimestamp,
       long maxDataTimestamp,
-      boolean descending,
-      @Nullable Filter postFilter,
-      @Nullable ColumnSelectorColumnIndexSelector bitmapIndexSelector
+      boolean descending
   )
   {
     this.index = index;
     this.interval = interval;
     this.virtualColumns = virtualColumns;
-    this.filterBitmap = filterBitmap;
+    this.filter = filter;
+    this.metrics = metrics;
     this.minDataTimestamp = minDataTimestamp;
     this.maxDataTimestamp = maxDataTimestamp;
     this.descending = descending;
-    this.postFilter = postFilter;
-    this.bitmapIndexSelector = bitmapIndexSelector;
   }
 
   public Sequence<Cursor> build(final Granularity gran)
   {
+    final Closer closer = Closer.create();
+
+    // Column caches shared amongst all cursors in this sequence.
+    final ColumnCache columnCache = new ColumnCache(index, closer);
+
     final Offset baseOffset;
 
+    final ColumnSelectorColumnIndexSelector bitmapIndexSelector = new ColumnSelectorColumnIndexSelector(
+        index.getBitmapFactoryForDimensions(),
+        virtualColumns,
+        columnCache

Review Comment:
   (2) is not really an option at this point in time.  The whole reason that the column cache is being passed in is because there are a plethora of legitimate bugs where we are not closing columns that need to be closed and passing the columnCache is the way to fix that.  Of course, we could go and change all of the different places where that is broken, but that is a lot more intrusive of a change.
   
   (1) is possible, though, the real fundamental issue is that the `Closer` pattern got added to the `Closeables` that are in the interfaces in Druid.  The addition of the `Closer` meant that we started losing track of where objects are actually being closed as sometimes they are magically closed elsewhere and sometimes they aren't.  We likely need to just have a single pattern (either, it's always via `Closeable`s or it's always via a `Closer`), but the goal of this PR is to get us to where we proactively catch and fix leaks by failing tests, so getting us to a place of consistency in how we deal with closing objects is a bit too big of a thing.
   
   That said, I think I can make it so that the cached Column objects from the columnCache ignore their `close()` method so that the only way to close the underlying object is to do it from the `Closer` which avoids the issue for now.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] imply-cheddar commented on a diff in pull request #12646: Poison stupid pool

Posted by GitBox <gi...@apache.org>.
imply-cheddar commented on code in PR #12646:
URL: https://github.com/apache/druid/pull/12646#discussion_r899687023


##########
processing/src/main/java/org/apache/druid/segment/ColumnCache.java:
##########
@@ -0,0 +1,119 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.segment;
+
+import org.apache.druid.java.util.common.io.Closer;
+import org.apache.druid.segment.column.BaseColumn;
+import org.apache.druid.segment.column.ColumnCapabilities;
+import org.apache.druid.segment.column.ColumnHolder;
+import org.apache.druid.segment.column.ColumnIndexSupplier;
+import org.apache.druid.segment.selector.settable.SettableColumnValueSelector;
+
+import javax.annotation.Nullable;
+import java.util.HashMap;
+import java.util.List;
+
+public class ColumnCache implements ColumnSelector
+{
+  private final HashMap<String, ColumnHolder> holderCache;
+  private final QueryableIndex index;
+  private final Closer closer;
+
+  public ColumnCache(QueryableIndex index, Closer closer)
+  {
+    this.index = index;
+    this.closer = closer;
+
+    this.holderCache = new HashMap<>();
+  }
+
+
+  @Override
+  public List<String> getColumnNames()
+  {
+    return index.getColumnNames();
+  }
+
+  @Nullable
+  @Override
+  public ColumnHolder getColumnHolder(String columnName)
+  {
+    return holderCache.computeIfAbsent(columnName, dimension -> {
+      // Here we do a funny little dance to memoize the BaseColumn and register it with the closer.
+      // It would probably be cleaner if the ColumnHolder itself was `Closeable` and did its own memoization,
+      // but that change is much wider and runs the risk of even more things that need to close the thing
+      // not actually closing it.  So, maybe this is a hack, maybe it's a wise decision, who knows, but at
+      // least for now, we grab the holder, grab the column, register the column with the closer and then return
+      // a new holder that always returns the same reference for the column.
+
+      final ColumnHolder holder = index.getColumnHolder(columnName);
+      if (holder == null) {
+        return null;
+      }
+
+      return new ColumnHolder()
+      {
+        @Nullable
+        private BaseColumn theColumn = null;
+
+        @Override
+        public ColumnCapabilities getCapabilities()
+        {
+          return holder.getCapabilities();
+        }
+
+        @Override
+        public int getLength()
+        {
+          return holder.getLength();
+        }
+
+        @Override
+        public BaseColumn getColumn()
+        {
+          if (theColumn == null) {
+            theColumn = closer.register(holder.getColumn());
+          }
+          return theColumn;
+        }
+
+        @Nullable
+        @Override
+        public ColumnIndexSupplier getIndexSupplier()
+        {
+          return holder.getIndexSupplier();

Review Comment:
   Yeah, this is a gap that we uncovered in doing this.  `getIndexSupplier()` likely should've been put on `BaseColumn` instead.
   
   Indexes on multiple columns won't come from this interface anyway, `ColumnHolder` won't carry indexes on multiple columns.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] imply-cheddar commented on a diff in pull request #12646: Poison stupid pool

Posted by GitBox <gi...@apache.org>.
imply-cheddar commented on code in PR #12646:
URL: https://github.com/apache/druid/pull/12646#discussion_r899699490


##########
core/pom.xml:
##########
@@ -430,12 +430,6 @@
         <configuration>
           <!-- use normal classpath instead of manifest jar for JvmUtilsTest.testSystemClassPath -->
           <useManifestOnlyJar>false</useManifestOnlyJar>
-          <argLine>
-            @{jacocoArgLine}
-            ${jdk.surefire.argLine}
-            -Djava.library.path=${project.build.directory}/hyperic-sigar-${sigar.base.version}/sigar-bin/lib/

Review Comment:
   It's not failing the tests, so 🤷 
   
   A magical line that must exist yet doesn't fail tests when it doesn't exist might as well not exist...



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] gianm commented on a diff in pull request #12646: Poison stupid pool

Posted by GitBox <gi...@apache.org>.
gianm commented on code in PR #12646:
URL: https://github.com/apache/druid/pull/12646#discussion_r897204563


##########
.idea/misc.xml:
##########
@@ -84,7 +87,7 @@
     <resource url="http://maven.apache.org/ASSEMBLY/2.0.0" location="$PROJECT_DIR$/.idea/xml-schemas/assembly-2.0.0.xsd" />
     <resource url="http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd" location="$PROJECT_DIR$/.idea/xml-schemas/svg11.dtd" />
   </component>
-  <component name="ProjectRootManager" version="2" languageLevel="JDK_1_8" default="false" project-jdk-name="1.8" project-jdk-type="JavaSDK">
+  <component name="ProjectRootManager" version="2" languageLevel="JDK_1_8" project-jdk-name="corretto-15" project-jdk-type="JavaSDK">

Review Comment:
   This change doesn't look intentional. When I check out the branch, it blows up my IDE because I don't have Corretto installed.



##########
core/src/main/java/org/apache/druid/collections/StupidPool.java:
##########
@@ -39,6 +39,18 @@
 public class StupidPool<T> implements NonBlockingPool<T>
 {
   private static final Logger log = new Logger(StupidPool.class);
+  private static final AtomicBoolean POISONED = new AtomicBoolean(false);

Review Comment:
   Could use some javadocs here explaining how poisoning works.



##########
processing/src/main/java/org/apache/druid/segment/QueryableIndexStorageAdapter.java:
##########
@@ -227,21 +227,15 @@ public VectorCursor makeVectorCursor(
     if (actualInterval == null) {
       return null;
     }
-
-    final ColumnSelectorColumnIndexSelector bitmapIndexSelector = makeBitmapIndexSelector(virtualColumns);
-
-    final FilterAnalysis filterAnalysis = analyzeFilter(filter, bitmapIndexSelector, queryMetrics);

Review Comment:
   I think this means the `analyzeFilter` method can be deleted from this file.



##########
processing/src/main/java/org/apache/druid/segment/ColumnSelector.java:
##########
@@ -29,6 +29,10 @@
  */
 public interface ColumnSelector extends ColumnInspector
 {
+  /**
+   * This method is apparently no longer used anymore, so deprecating it.
+   */
+  @Deprecated

Review Comment:
   This isn't a `@PublicApi` so why not delete it?



##########
processing/src/main/java/org/apache/druid/segment/QueryableIndexCursorSequenceBuilder.java:
##########
@@ -50,51 +56,66 @@
 
 import javax.annotation.Nullable;
 import java.io.IOException;
-import java.util.HashMap;
-import java.util.Map;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
 
 public class QueryableIndexCursorSequenceBuilder
 {
   private final QueryableIndex index;
   private final Interval interval;
   private final VirtualColumns virtualColumns;
-  @Nullable
-  private final ImmutableBitmap filterBitmap;
+  private final Filter filter;
+  private final QueryMetrics<? extends Query> metrics;
   private final long minDataTimestamp;
   private final long maxDataTimestamp;
   private final boolean descending;
-  @Nullable
-  private final Filter postFilter;
-  @Nullable
-  private final ColumnSelectorColumnIndexSelector bitmapIndexSelector;
 
   public QueryableIndexCursorSequenceBuilder(
       QueryableIndex index,
       Interval interval,
       VirtualColumns virtualColumns,
-      @Nullable ImmutableBitmap filterBitmap,
+      @Nullable Filter filter,
+      @Nullable QueryMetrics<? extends Query> metrics,
       long minDataTimestamp,
       long maxDataTimestamp,
-      boolean descending,
-      @Nullable Filter postFilter,
-      @Nullable ColumnSelectorColumnIndexSelector bitmapIndexSelector
+      boolean descending
   )
   {
     this.index = index;
     this.interval = interval;
     this.virtualColumns = virtualColumns;
-    this.filterBitmap = filterBitmap;
+    this.filter = filter;
+    this.metrics = metrics;
     this.minDataTimestamp = minDataTimestamp;
     this.maxDataTimestamp = maxDataTimestamp;
     this.descending = descending;
-    this.postFilter = postFilter;
-    this.bitmapIndexSelector = bitmapIndexSelector;
   }
 
   public Sequence<Cursor> build(final Granularity gran)
   {
+    final Closer closer = Closer.create();
+
+    // Column caches shared amongst all cursors in this sequence.
+    final ColumnCache columnCache = new ColumnCache(index, closer);
+
     final Offset baseOffset;
 
+    final ColumnSelectorColumnIndexSelector bitmapIndexSelector = new ColumnSelectorColumnIndexSelector(
+        index.getBitmapFactoryForDimensions(),
+        virtualColumns,
+        columnCache

Review Comment:
   There's a double-close that can happen as a result of the caching here: `ColumnSelectorColumnIndexSelector.getNumRows` gets the `__time` column and then immediately closes it. So calls to `bitmapIndexSelector.getNumRows()` subsequent to the first one will get an already-closed column. `not` filters call it once, so it'll happen on queries that have multiple `not` filters. And maybe other cases too?
   
   Seems like there's two ways we can fix it:
   
   1. Modify ColumnSelectorColumnIndexSelector to not close the time column, and instead assume it will get closed some other way. Modify places that construct ColumnSelectorColumnIndexSelector to ensure that this happens.
   
   2. Modify places that construct ColumnSelectorColumnIndexSelector to *not* pass in a ColumnCache, but to instead pass in something that doesn't cache columns. Keep the closing behavior of ColumnSelectorColumnIndexSelector the way it is.



##########
processing/src/main/java/org/apache/druid/segment/DeprecatedQueryableIndexColumnSelector.java:
##########
@@ -0,0 +1,66 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.segment;
+
+import org.apache.druid.segment.column.ColumnHolder;
+
+import javax.annotation.Nullable;
+import java.util.List;
+
+/**
+ * It likely looks weird that we are creating a new instance of ColumnSelector here that begins its life deprecated
+ * and only delegates methods to the Queryable Index. This is done intentionally so that the QueryableIndex doesn't
+ * accidentally get used as a ColumnSelector.
+ *
+ * The lifecycle of the QueryableIndex is over the lifetime of the segment on a specific process, while
+ * the ColumnSelector's lifecycle is for a given query.  When we don't use the same ColumnSelector for an
+ * entire query, we defeat caching and use a lot more resources than necessary for queries.
+ *
+ * Places that use this class are intentionally circumventing column caching and column lifecycle management,
+ * ostensibly because those code locations know that they are only looking at metadata.  If a code path uses this
+ * and actually accesses a column instead of just looking at metadata, it will leak any resources that said column
+ * requires.
+ *
+ * The ColumnCache is the preferred implementation of a ColumnSelector, it takes a Closer and that closer can be used
+ * to ensure that resources are cleaned up.
+ */
+@Deprecated
+public class DeprecatedQueryableIndexColumnSelector implements ColumnSelector

Review Comment:
   I'm wondering we can get away without having this class, especially after fixing the double-free issue with ColumnSelectorColumnIndexSelector. All but one of the usages of DeprecatedQueryableIndexColumnSelector are as an input to ColumnSelectorColumnIndexSelector. Once that's adjusted, it may be that we don't actually need this adapter anymore.
   
   The other usage is in a call to `Filter.supportsSelectivityEstimation` in AutoStrategy, where it would be fine (and preferable?) to pass in a ColumnCache instead of a DeprecatedQueryableIndexColumnSelector.



##########
pom.xml:
##########
@@ -1500,8 +1500,9 @@
                         <!-- set default options -->
                         <argLine>
                             @{jacocoArgLine}
+                            ${jdk.surefire.argLine}
                             -Xmx1500m
-                            -XX:MaxDirectMemorySize=512m
+                            -XX:MaxDirectMemorySize=2500m

Review Comment:
   Why?



##########
processing/src/main/java/org/apache/druid/query/timeseries/TimeseriesQueryEngine.java:
##########
@@ -161,7 +161,7 @@ private Sequence<Result<TimeseriesResultValue>> processVectorized(
       );
 
       if (granularizer == null) {
-        return Sequences.empty();
+        return Sequences.withBaggage(Sequences.empty(), closer);

Review Comment:
   Might as well close the closer immediately and then return `Sequences.empty()`? I guess either way is fine.



##########
processing/src/main/java/org/apache/druid/segment/DeprecatedQueryableIndexColumnSelector.java:
##########
@@ -0,0 +1,66 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.segment;
+
+import org.apache.druid.segment.column.ColumnHolder;
+
+import javax.annotation.Nullable;
+import java.util.List;
+
+/**
+ * It likely looks weird that we are creating a new instance of ColumnSelector here that begins its life deprecated
+ * and only delegates methods to the Queryable Index. This is done intentionally so that the QueryableIndex doesn't
+ * accidentally get used as a ColumnSelector.
+ *
+ * The lifecycle of the QueryableIndex is over the lifetime of the segment on a specific process, while
+ * the ColumnSelector's lifecycle is for a given query.  When we don't use the same ColumnSelector for an
+ * entire query, we defeat caching and use a lot more resources than necessary for queries.
+ *
+ * Places that use this class are intentionally circumventing column caching and column lifecycle management,
+ * ostensibly because those code locations know that they are only looking at metadata.  If a code path uses this

Review Comment:
   It won't necessarily leak, will it? Callers can close the BaseColumn they get from the ColumnHolder. For example, `ColumnSelectorColumnIndexSelector.getNumRows` does this, which is called by AutoStrategy in a way that seems OK.



##########
processing/src/main/java/org/apache/druid/segment/ColumnCache.java:
##########
@@ -0,0 +1,119 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.segment;
+
+import org.apache.druid.java.util.common.io.Closer;
+import org.apache.druid.segment.column.BaseColumn;
+import org.apache.druid.segment.column.ColumnCapabilities;
+import org.apache.druid.segment.column.ColumnHolder;
+import org.apache.druid.segment.column.ColumnIndexSupplier;
+import org.apache.druid.segment.selector.settable.SettableColumnValueSelector;
+
+import javax.annotation.Nullable;
+import java.util.HashMap;
+import java.util.List;
+
+public class ColumnCache implements ColumnSelector
+{
+  private final HashMap<String, ColumnHolder> holderCache;
+  private final QueryableIndex index;
+  private final Closer closer;
+
+  public ColumnCache(QueryableIndex index, Closer closer)
+  {
+    this.index = index;
+    this.closer = closer;
+
+    this.holderCache = new HashMap<>();
+  }
+
+
+  @Override
+  public List<String> getColumnNames()
+  {
+    return index.getColumnNames();
+  }
+
+  @Nullable
+  @Override
+  public ColumnHolder getColumnHolder(String columnName)
+  {
+    return holderCache.computeIfAbsent(columnName, dimension -> {
+      // Here we do a funny little dance to memoize the BaseColumn and register it with the closer.
+      // It would probably be cleaner if the ColumnHolder itself was `Closeable` and did its own memoization,
+      // but that change is much wider and runs the risk of even more things that need to close the thing
+      // not actually closing it.  So, maybe this is a hack, maybe it's a wise decision, who knows, but at
+      // least for now, we grab the holder, grab the column, register the column with the closer and then return
+      // a new holder that always returns the same reference for the column.
+
+      final ColumnHolder holder = index.getColumnHolder(columnName);
+      if (holder == null) {
+        return null;
+      }
+
+      return new ColumnHolder()
+      {
+        @Nullable
+        private BaseColumn theColumn = null;
+
+        @Override
+        public ColumnCapabilities getCapabilities()
+        {
+          return holder.getCapabilities();
+        }
+
+        @Override
+        public int getLength()
+        {
+          return holder.getLength();
+        }
+
+        @Override
+        public BaseColumn getColumn()
+        {
+          if (theColumn == null) {
+            theColumn = closer.register(holder.getColumn());
+          }
+          return theColumn;
+        }
+
+        @Nullable
+        @Override
+        public ColumnIndexSupplier getIndexSupplier()
+        {
+          return holder.getIndexSupplier();

Review Comment:
   BaseColumns need to be closed but ColumnIndexSuppliers don't?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] imply-cheddar commented on a diff in pull request #12646: Poison stupid pool

Posted by GitBox <gi...@apache.org>.
imply-cheddar commented on code in PR #12646:
URL: https://github.com/apache/druid/pull/12646#discussion_r899685702


##########
pom.xml:
##########
@@ -1500,8 +1500,9 @@
                         <!-- set default options -->
                         <argLine>
                             @{jacocoArgLine}
+                            ${jdk.surefire.argLine}
                             -Xmx1500m
-                            -XX:MaxDirectMemorySize=512m
+                            -XX:MaxDirectMemorySize=2500m

Review Comment:
   Because the config from the processing pom had it set to that value and, well, this setting is meaningless anyway (it has absolutely no functional impact on the process, all it does it set a threshold for when exceptions get thrown) so setting it on all of the test runs doesn't change anything.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] imply-cheddar commented on a diff in pull request #12646: Poison stupid pool

Posted by GitBox <gi...@apache.org>.
imply-cheddar commented on code in PR #12646:
URL: https://github.com/apache/druid/pull/12646#discussion_r899692983


##########
processing/src/main/java/org/apache/druid/segment/DeprecatedQueryableIndexColumnSelector.java:
##########
@@ -0,0 +1,66 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.druid.segment;
+
+import org.apache.druid.segment.column.ColumnHolder;
+
+import javax.annotation.Nullable;
+import java.util.List;
+
+/**
+ * It likely looks weird that we are creating a new instance of ColumnSelector here that begins its life deprecated
+ * and only delegates methods to the Queryable Index. This is done intentionally so that the QueryableIndex doesn't
+ * accidentally get used as a ColumnSelector.
+ *
+ * The lifecycle of the QueryableIndex is over the lifetime of the segment on a specific process, while
+ * the ColumnSelector's lifecycle is for a given query.  When we don't use the same ColumnSelector for an
+ * entire query, we defeat caching and use a lot more resources than necessary for queries.
+ *
+ * Places that use this class are intentionally circumventing column caching and column lifecycle management,
+ * ostensibly because those code locations know that they are only looking at metadata.  If a code path uses this

Review Comment:
   Yes, that's true.  The comments are hyperbolic in that way, but in general, I wish this class didn't exist and any source of using this class should likely be adjusted, so claiming that they are all bad, even if it's not for a technically correct reason, doesn't seem to horrible.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] imply-cheddar commented on a diff in pull request #12646: Poison stupid pool

Posted by GitBox <gi...@apache.org>.
imply-cheddar commented on code in PR #12646:
URL: https://github.com/apache/druid/pull/12646#discussion_r899696671


##########
processing/src/main/java/org/apache/druid/segment/QueryableIndexCursorSequenceBuilder.java:
##########
@@ -50,51 +56,66 @@
 
 import javax.annotation.Nullable;
 import java.io.IOException;
-import java.util.HashMap;
-import java.util.Map;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.List;
 
 public class QueryableIndexCursorSequenceBuilder
 {
   private final QueryableIndex index;
   private final Interval interval;
   private final VirtualColumns virtualColumns;
-  @Nullable
-  private final ImmutableBitmap filterBitmap;
+  private final Filter filter;
+  private final QueryMetrics<? extends Query> metrics;
   private final long minDataTimestamp;
   private final long maxDataTimestamp;
   private final boolean descending;
-  @Nullable
-  private final Filter postFilter;
-  @Nullable
-  private final ColumnSelectorColumnIndexSelector bitmapIndexSelector;
 
   public QueryableIndexCursorSequenceBuilder(
       QueryableIndex index,
       Interval interval,
       VirtualColumns virtualColumns,
-      @Nullable ImmutableBitmap filterBitmap,
+      @Nullable Filter filter,
+      @Nullable QueryMetrics<? extends Query> metrics,
       long minDataTimestamp,
       long maxDataTimestamp,
-      boolean descending,
-      @Nullable Filter postFilter,
-      @Nullable ColumnSelectorColumnIndexSelector bitmapIndexSelector
+      boolean descending
   )
   {
     this.index = index;
     this.interval = interval;
     this.virtualColumns = virtualColumns;
-    this.filterBitmap = filterBitmap;
+    this.filter = filter;
+    this.metrics = metrics;
     this.minDataTimestamp = minDataTimestamp;
     this.maxDataTimestamp = maxDataTimestamp;
     this.descending = descending;
-    this.postFilter = postFilter;
-    this.bitmapIndexSelector = bitmapIndexSelector;
   }
 
   public Sequence<Cursor> build(final Granularity gran)
   {
+    final Closer closer = Closer.create();
+
+    // Column caches shared amongst all cursors in this sequence.
+    final ColumnCache columnCache = new ColumnCache(index, closer);
+
     final Offset baseOffset;
 
+    final ColumnSelectorColumnIndexSelector bitmapIndexSelector = new ColumnSelectorColumnIndexSelector(
+        index.getBitmapFactoryForDimensions(),
+        virtualColumns,
+        columnCache

Review Comment:
   For now, the `__time` column tends to be resilient to double close, so maybe it's okay.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org