You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by GitBox <gi...@apache.org> on 2022/02/02 01:05:31 UTC
[GitHub] [druid] AlexanderSaydakov opened a new pull request #12224: latest datasketches-java-3.1.0
AlexanderSaydakov opened a new pull request #12224:
URL: https://github.com/apache/druid/pull/12224
These changes are to use the latest datasketches-java-3.1.0 and also to restore support for quantile and HLL4 sketches to be able to grow larger than a given buffer in a buffer aggregator and move to heap in rare cases. This was discussed in #11544.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] AlexanderSaydakov commented on a change in pull request #12224: latest datasketches-java-3.1.0
Posted by GitBox <gi...@apache.org>.
AlexanderSaydakov commented on a change in pull request #12224:
URL: https://github.com/apache/druid/pull/12224#discussion_r817264530
##########
File path: extensions-core/datasketches/src/main/java/org/apache/druid/query/aggregation/datasketches/hll/HllSketchBuildBufferAggregatorHelper.java
##########
@@ -123,7 +126,8 @@ public int getLgK()
private WritableMemory getMemory(final ByteBuffer buf)
{
- return memCache.computeIfAbsent(buf, b -> WritableMemory.writableWrap(b, ByteOrder.LITTLE_ENDIAN));
+ return memCache.computeIfAbsent(buf,
+ b -> WritableMemory.writableWrap(b, ByteOrder.LITTLE_ENDIAN, MEM_REQ_SERVER));
Review comment:
This is to restore support for quantile and HLL4 sketches to be able to grow larger than a given buffer in a buffer aggregator and move to heap in rare cases. This used to be a default before. The problem with this support arose because datasketches-memory was substantially redesigned some time ago, and the default behavior was removed. At that time we should have changed this Druid extension accordingly, but we did not. And because this support is needed in rare circumstances, the problem was not noticed for quite a while.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] AlexanderSaydakov commented on a change in pull request #12224: latest datasketches-java-3.1.0
Posted by GitBox <gi...@apache.org>.
AlexanderSaydakov commented on a change in pull request #12224:
URL: https://github.com/apache/druid/pull/12224#discussion_r817269308
##########
File path: extensions-core/datasketches/src/main/java/org/apache/druid/query/aggregation/datasketches/hll/HllSketchBuildBufferAggregatorHelper.java
##########
@@ -123,7 +126,8 @@ public int getLgK()
private WritableMemory getMemory(final ByteBuffer buf)
{
- return memCache.computeIfAbsent(buf, b -> WritableMemory.writableWrap(b, ByteOrder.LITTLE_ENDIAN));
+ return memCache.computeIfAbsent(buf,
+ b -> WritableMemory.writableWrap(b, ByteOrder.LITTLE_ENDIAN, MEM_REQ_SERVER));
Review comment:
Unit tests were added to make sure this support works. If you look at the changes, you can see that I changed a few tests from asserting failure to asserting success. For instance testFailureWhenMaxStreamLengthHit to testSuccessWhenMaxStreamLengthHit in DoublesSketchAggregatorTest.java
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] xvrl merged pull request #12224: latest datasketches-java-3.1.0
Posted by GitBox <gi...@apache.org>.
xvrl merged pull request #12224:
URL: https://github.com/apache/druid/pull/12224
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] xvrl commented on a change in pull request #12224: latest datasketches-java-3.1.0
Posted by GitBox <gi...@apache.org>.
xvrl commented on a change in pull request #12224:
URL: https://github.com/apache/druid/pull/12224#discussion_r816966121
##########
File path: extensions-core/datasketches/src/main/java/org/apache/druid/query/aggregation/datasketches/hll/HllSketchBuildBufferAggregatorHelper.java
##########
@@ -123,7 +126,8 @@ public int getLgK()
private WritableMemory getMemory(final ByteBuffer buf)
{
- return memCache.computeIfAbsent(buf, b -> WritableMemory.writableWrap(b, ByteOrder.LITTLE_ENDIAN));
+ return memCache.computeIfAbsent(buf,
+ b -> WritableMemory.writableWrap(b, ByteOrder.LITTLE_ENDIAN, MEM_REQ_SERVER));
Review comment:
memory server doesn't seem to be a required argument, do you mind explaining why we need to pass it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] xvrl commented on a change in pull request #12224: latest datasketches-java-3.1.0
Posted by GitBox <gi...@apache.org>.
xvrl commented on a change in pull request #12224:
URL: https://github.com/apache/druid/pull/12224#discussion_r817269674
##########
File path: extensions-core/datasketches/src/main/java/org/apache/druid/query/aggregation/datasketches/hll/HllSketchBuildBufferAggregatorHelper.java
##########
@@ -123,7 +126,8 @@ public int getLgK()
private WritableMemory getMemory(final ByteBuffer buf)
{
- return memCache.computeIfAbsent(buf, b -> WritableMemory.writableWrap(b, ByteOrder.LITTLE_ENDIAN));
+ return memCache.computeIfAbsent(buf,
+ b -> WritableMemory.writableWrap(b, ByteOrder.LITTLE_ENDIAN, MEM_REQ_SERVER));
Review comment:
thanks for the explanation!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org