You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by GitBox <gi...@apache.org> on 2021/04/16 19:02:59 UTC

[GitHub] [druid] gianm opened a new pull request #11124: Remove OffheapIncrementalIndex and clarify aggregator thread-safety needs.

gianm opened a new pull request #11124:
URL: https://github.com/apache/druid/pull/11124


   This patch does the following:
   
   - Removes OffheapIncrementalIndex.
   - Clarifies that Aggregators are required to be thread safe.
   - Clarifies that BufferAggregators and VectorAggregators are not
     required to be thread safe.
   - Removes thread safety code from some DataSketches aggregators that
     had it. (Not all of them did, and that's OK, because it wasn't necessary
     anyway.)
   - Makes enabling "useOffheap" with groupBy v1 an error.
   
   Rationale for removing the offheap incremental index:
   
   - It is only used in one rare scenario: groupBy v1 (which is non-default)
     in "useOffheap" mode (also non-default). So you have to go pretty deep
     into the wilderness to get this code to activate in production. It is
     never used during ingestion.
   - Its existence complicates developer efforts to reason about how
     aggregators get used, because the way it uses buffer aggregators is so
     different from how every other query engine uses them.
   - It doesn't have meaningful testing.
   
   By the way, I do believe that the given way the offheap incremental index
   works, it actually didn't require buffer aggregators to be thread-safe.
   It synchronizes on "aggregate" and doesn't call "get" until it has
   stopped calling "aggregate". Nevertheless, this is a bother to think about,
   and for the above reasons I think it makes sense to remove the code anyway.
   
   <!-- Thanks for trying to help us make Apache Druid be the best it can be! Please fill out as much of the following information as is possible (where relevant, and remove it when irrelevant) to help make the intention and scope of this PR clear in order to ease review. -->
   
   <!-- Please read the doc for contribution (https://github.com/apache/druid/blob/master/CONTRIBUTING.md) before making this PR. Also, once you open a PR, please _avoid using force pushes and rebasing_ since these make it difficult for reviewers to see what you've changed in response to their reviews. See [the 'If your pull request shows conflicts with master' section](https://github.com/apache/druid/blob/master/CONTRIBUTING.md#if-your-pull-request-shows-conflicts-with-master) for more details. -->
   
   Fixes #XXXX.
   
   <!-- Replace XXXX with the id of the issue fixed in this PR. Remove this section if there is no corresponding issue. Don't reference the issue in the title of this pull-request. -->
   
   <!-- If you are a committer, follow the PR action item checklist for committers:
   https://github.com/apache/druid/blob/master/dev/committer-instructions.md#pr-and-issue-action-item-checklist-for-committers. -->
   
   ### Description
   
   <!-- Describe the goal of this PR, what problem are you fixing. If there is a corresponding issue (referenced above), it's not necessary to repeat the description here, however, you may choose to keep one summary sentence. -->
   
   <!-- Describe your patch: what did you change in code? How did you fix the problem? -->
   
   <!-- If there are several relatively logically separate changes in this PR, create a mini-section for each of them. For example: -->
   
   #### Fixed the bug ...
   #### Renamed the class ...
   #### Added a forbidden-apis entry ...
   
   <!--
   In each section, please describe design decisions made, including:
    - Choice of algorithms
    - Behavioral aspects. What configuration values are acceptable? How are corner cases and error conditions handled, such as when there are insufficient resources?
    - Class organization and design (how the logic is split between classes, inheritance, composition, design patterns)
    - Method organization and design (how the logic is split between methods, parameters and return types)
    - Naming (class, method, API, configuration, HTTP endpoint, names of emitted metrics)
   -->
   
   
   <!-- It's good to describe an alternative design (or mention an alternative name) for every design (or naming) decision point and compare the alternatives with the designs that you've implemented (or the names you've chosen) to highlight the advantages of the chosen designs and names. -->
   
   <!-- If there was a discussion of the design of the feature implemented in this PR elsewhere (e. g. a "Proposal" issue, any other issue, or a thread in the development mailing list), link to that discussion from this PR description and explain what have changed in your final design compared to your original proposal or the consensus version in the end of the discussion. If something hasn't changed since the original discussion, you can omit a detailed discussion of those aspects of the design here, perhaps apart from brief mentioning for the sake of readability of this PR description. -->
   
   <!-- Some of the aspects mentioned above may be omitted for simple and small changes. -->
   
   <hr>
   
   ##### Key changed/added classes in this PR
    * `MyFoo`
    * `OurBar`
    * `TheirBaz`
   
   <hr>
   
   <!-- Check the items by putting "x" in the brackets for the done things. Not all of these items apply to every PR. Remove the items which are not done or not relevant to the PR. None of the items from the checklist below are strictly necessary, but it would be very helpful if you at least self-review the PR. -->
   
   This PR has:
   - [ ] been self-reviewed.
      - [ ] using the [concurrency checklist](https://github.com/apache/druid/blob/master/dev/code-review/concurrency.md) (Remove this item if the PR doesn't have any relation to concurrency.)
   - [ ] added documentation for new or modified features or behaviors.
   - [ ] added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
   - [ ] added or updated version, license, or notice information in [licenses.yaml](https://github.com/apache/druid/blob/master/dev/license.md)
   - [ ] added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
   - [ ] added unit tests or modified existing tests to cover new code paths, ensuring the threshold for [code coverage](https://github.com/apache/druid/blob/master/dev/code-review/code-coverage.md) is met.
   - [ ] added integration tests.
   - [ ] been tested in a test Druid cluster.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] gianm commented on pull request #11124: Remove OffheapIncrementalIndex and clarify aggregator thread-safety needs.

Posted by GitBox <gi...@apache.org>.
gianm commented on pull request #11124:
URL: https://github.com/apache/druid/pull/11124#issuecomment-824403679


   I'm ok either way @liran-funaro. It depends on which patch is merged first, I think.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] gianm merged pull request #11124: Remove OffheapIncrementalIndex and clarify aggregator thread-safety needs.

Posted by GitBox <gi...@apache.org>.
gianm merged pull request #11124:
URL: https://github.com/apache/druid/pull/11124


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] liran-funaro commented on pull request #11124: Remove OffheapIncrementalIndex and clarify aggregator thread-safety needs.

Posted by GitBox <gi...@apache.org>.
liran-funaro commented on pull request #11124:
URL: https://github.com/apache/druid/pull/11124#issuecomment-951610536


   Sure. Go ahead.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] gianm commented on pull request #11124: Remove OffheapIncrementalIndex and clarify aggregator thread-safety needs.

Posted by GitBox <gi...@apache.org>.
gianm commented on pull request #11124:
URL: https://github.com/apache/druid/pull/11124#issuecomment-821737321


   Wow, there was a lot of thread to pull on behind the intellij "unused item" inspections. The primitive get methods of BufferAggregator are no longer needed; nor is the BufferPool passed to the groupBy v1 merger. I removed them in the latest commit. I added an "unused" suppression to the stuff in IncrementalIndex that is unused now, but that #10001 may need.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] clintropolis commented on pull request #11124: Remove OffheapIncrementalIndex and clarify aggregator thread-safety needs.

Posted by GitBox <gi...@apache.org>.
clintropolis commented on pull request #11124:
URL: https://github.com/apache/druid/pull/11124#issuecomment-821720100


   overall +1 on removing this. This PR is failing some intellij inspections, but we might want to suppress them to accommodate #10001, which does appear to use some of these methods.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] liran-funaro commented on a change in pull request #11124: Remove OffheapIncrementalIndex and clarify aggregator thread-safety needs.

Posted by GitBox <gi...@apache.org>.
liran-funaro commented on a change in pull request #11124:
URL: https://github.com/apache/druid/pull/11124#discussion_r620163635



##########
File path: processing/src/main/java/org/apache/druid/query/aggregation/BufferAggregator.java
##########
@@ -93,64 +96,6 @@
   @Nullable
   Object get(ByteBuffer buf, int position);
 
-  /**
-   * Returns the float representation of the given aggregate byte array
-   *
-   * Converts the given byte buffer representation into the intermediate aggregate value.
-   *
-   * <b>Implementations must not change the position, limit or mark of the given buffer</b>
-   *
-   * Implementations are only required to support this method if they are aggregations which
-   * have an {@link AggregatorFactory#getType()} ()} of {@link org.apache.druid.segment.column.ValueType#FLOAT}.
-   * If unimplemented, throwing an {@link UnsupportedOperationException} is common and recommended.
-   *
-   * @param buf byte buffer storing the byte array representation of the aggregate
-   * @param position offset within the byte buffer at which the aggregate value is stored
-   * @return the float representation of the aggregate
-   */
-  float getFloat(ByteBuffer buf, int position);
-
-  /**
-   * Returns the long representation of the given aggregate byte array
-   *
-   * Converts the given byte buffer representation into the intermediate aggregate value.
-   *
-   * <b>Implementations must not change the position, limit or mark of the given buffer</b>
-   *
-   * Implementations are only required to support this method if they are aggregations which
-   * have an {@link AggregatorFactory#getType()} of  of {@link org.apache.druid.segment.column.ValueType#LONG}.
-   * If unimplemented, throwing an {@link UnsupportedOperationException} is common and recommended.
-   *
-   * @param buf byte buffer storing the byte array representation of the aggregate
-   * @param position offset within the byte buffer at which the aggregate value is stored
-   * @return the long representation of the aggregate
-   */
-  long getLong(ByteBuffer buf, int position);
-
-  /**
-   * Returns the double representation of the given aggregate byte array
-   *
-   * Converts the given byte buffer representation into the intermediate aggregate value.
-   *
-   * <b>Implementations must not change the position, limit or mark of the given buffer</b>
-   *
-   * Implementations are only required to support this method if they are aggregations which
-   * have an {@link AggregatorFactory#getType()} of  of {@link org.apache.druid.segment.column.ValueType#DOUBLE}.
-   * If unimplemented, throwing an {@link UnsupportedOperationException} is common and recommended.
-   *
-   * The default implementation casts {@link BufferAggregator#getFloat(ByteBuffer, int)} to double.
-   * This default method is added to enable smooth backward compatibility, please re-implement it if your aggregators
-   * work with numeric double columns.
-   *
-   * @param buf byte buffer storing the byte array representation of the aggregate
-   * @param position offset within the byte buffer at which the aggregate value is stored
-   * @return the double representation of the aggregate
-   */
-  default double getDouble(ByteBuffer buf, int position)
-  {
-    return (double) getFloat(buf, position);
-  }
-

Review comment:
       It is important for #10001 ( `OakIncrementalIndex` ) that `BufferAggregator` will have the same API as `Aggregator`.
   So please keep these methods. Also please keep `isNull()`.
   




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] gianm commented on pull request #11124: Remove OffheapIncrementalIndex and clarify aggregator thread-safety needs.

Posted by GitBox <gi...@apache.org>.
gianm commented on pull request #11124:
URL: https://github.com/apache/druid/pull/11124#issuecomment-821729611


   > I took a look at #10001, and I don't see usage of `getAggVal` there, so I'll convert it to a private method and remove the unused params. It does use `getAggs` though, so I'll suppress the warning on that one.
   
   Oh, nevermind, it's used by the parent class. But the OakIncrementalIndex doesn't support that method anyway: https://github.com/apache/druid/pull/10001/files#diff-8a952057b2e59239cbd4cbff14c3202ea7d8cd85b4bb4c174d9375b00a0e7b40R244-R250


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] liran-funaro commented on pull request #11124: Remove OffheapIncrementalIndex and clarify aggregator thread-safety needs.

Posted by GitBox <gi...@apache.org>.
liran-funaro commented on pull request #11124:
URL: https://github.com/apache/druid/pull/11124#issuecomment-823177125


   > I added an "unused" suppression to the stuff in IncrementalIndex that is unused now, but that #10001 may need.
   
   You can avoid using "unused" annotation by allowing `AggregatorType[] aggs` to be defined as `protected`.
   This might be a more straightforward approach.
   
   I can rebase #10001 after this PR will be merged.
   Or alternatively, if you prefer to merge #10001 before this PR, I can modify `aggs` to be protected in #10001. Then you won't have to deal with this at all in this PR.
   
   Please let me know what you think.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] jihoonson commented on pull request #11124: Remove OffheapIncrementalIndex and clarify aggregator thread-safety needs.

Posted by GitBox <gi...@apache.org>.
jihoonson commented on pull request #11124:
URL: https://github.com/apache/druid/pull/11124#issuecomment-951417403


   @liran-funaro thank you for taking a look at this PR and sorry for taking a long time to review your PRs. I quickly reviewed https://github.com/apache/druid/pull/11160 and left one question on it. Since this PR now has enough approvals, would you mind if we merge this PR before #11160 gets merged?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] gianm commented on pull request #11124: Remove OffheapIncrementalIndex and clarify aggregator thread-safety needs.

Posted by GitBox <gi...@apache.org>.
gianm commented on pull request #11124:
URL: https://github.com/apache/druid/pull/11124#issuecomment-949951003


   Per https://github.com/apache/druid/pull/11124#discussion_r620163635 I reverted the BufferAggregator and IncrementalIndex API changes mentioned in https://github.com/apache/druid/pull/11124#issuecomment-821737321.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] gianm commented on pull request #11124: Remove OffheapIncrementalIndex and clarify aggregator thread-safety needs.

Posted by GitBox <gi...@apache.org>.
gianm commented on pull request #11124:
URL: https://github.com/apache/druid/pull/11124#issuecomment-952034097


   Thanks!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] gianm commented on pull request #11124: Remove OffheapIncrementalIndex and clarify aggregator thread-safety needs.

Posted by GitBox <gi...@apache.org>.
gianm commented on pull request #11124:
URL: https://github.com/apache/druid/pull/11124#issuecomment-821728575


   > overall +1 on removing this. This PR is failing some intellij inspections, but we might want to suppress them to accommodate #10001, which does appear to use some of these methods.
   
   I took a look at #10001, and I don't see usage of `getAggVal` there, so I'll convert it to a private method and remove the unused params. It does use `getAggs` though, so I'll suppress the warning on that one.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org