You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by "gianm (via GitHub)" <gi...@apache.org> on 2023/03/07 02:41:13 UTC
[GitHub] [druid] gianm opened a new pull request, #13889: Improve memory efficiency of WrappedRoaringBitmap.
gianm opened a new pull request, #13889:
URL: https://github.com/apache/druid/pull/13889
Two changes:
1) Use an int[] for sizes 4 or below.
2) Remove the boolean compressRunOnSerialization. Doesn't save much
space, but it does save a little, and it isn't adding a ton of value
to have it be configurable. It was originally configurable in case
anything broke when enabling it, but it's been a while and nothing
has broken.
Helps reduce memory usage when building dictionaries.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] gianm commented on a diff in pull request #13889: Improve memory efficiency of WrappedRoaringBitmap.
Posted by "gianm (via GitHub)" <gi...@apache.org>.
gianm commented on code in PR #13889:
URL: https://github.com/apache/druid/pull/13889#discussion_r1127312423
##########
processing/src/main/java/org/apache/druid/segment/nested/GlobalDictionaryEncodedFieldColumnWriter.java:
##########
@@ -224,6 +224,7 @@ public void writeTo(int finalRowCount, FileSmoosher smoosher) throws IOException
bitmapIndexWriter.write(
indexSpec.getBitmapSerdeFactory().getBitmapFactory().makeImmutableBitmap(bitmap)
);
+ bitmap.clear(); // Reclaim memory
Review Comment:
Works for me, I changed it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] clintropolis commented on a diff in pull request #13889: Improve memory efficiency of WrappedRoaringBitmap.
Posted by "clintropolis (via GitHub)" <gi...@apache.org>.
clintropolis commented on code in PR #13889:
URL: https://github.com/apache/druid/pull/13889#discussion_r1127288591
##########
processing/src/main/java/org/apache/druid/segment/nested/GlobalDictionaryEncodedFieldColumnWriter.java:
##########
@@ -224,6 +224,7 @@ public void writeTo(int finalRowCount, FileSmoosher smoosher) throws IOException
bitmapIndexWriter.write(
indexSpec.getBitmapSerdeFactory().getBitmapFactory().makeImmutableBitmap(bitmap)
);
+ bitmap.clear(); // Reclaim memory
Review Comment:
i suppose alternatively we could change to classic for loop and just set bitmaps[i] to be null to let the gc reclaim it all?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] clintropolis commented on a diff in pull request #13889: Improve memory efficiency of WrappedRoaringBitmap.
Posted by "clintropolis (via GitHub)" <gi...@apache.org>.
clintropolis commented on code in PR #13889:
URL: https://github.com/apache/druid/pull/13889#discussion_r1131794073
##########
website/.spelling:
##########
@@ -1099,7 +1099,7 @@ KinesisSupervisorIOConfig
KinesisSupervisorTuningConfig
Resharding
resharding
-LZ4LZFuncompressedLZ4LZ4LZFuncompressednoneLZ4autolongsautolongslongstypeconcisetyperoaringcompressRunOnSerializationtruetypestreamendpointreplicastaskCounttaskCount
+LZ4LZFuncompressedLZ4LZ4LZFuncompressednoneLZ4autolongsautolongslongstypeconcisetyperoaringtypestreamendpointreplicastaskCounttaskCount
Review Comment:
hm, what is this i wonder? i know not new in this PR, but wondering did this file get messed up or something along the way? there are a few other lines like this in this section
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] gianm merged pull request #13889: Improve memory efficiency of WrappedRoaringBitmap.
Posted by "gianm (via GitHub)" <gi...@apache.org>.
gianm merged PR #13889:
URL: https://github.com/apache/druid/pull/13889
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] clintropolis commented on a diff in pull request #13889: Improve memory efficiency of WrappedRoaringBitmap.
Posted by "clintropolis (via GitHub)" <gi...@apache.org>.
clintropolis commented on code in PR #13889:
URL: https://github.com/apache/druid/pull/13889#discussion_r1131834632
##########
website/.spelling:
##########
@@ -1099,7 +1099,7 @@ KinesisSupervisorIOConfig
KinesisSupervisorTuningConfig
Resharding
resharding
-LZ4LZFuncompressedLZ4LZ4LZFuncompressednoneLZ4autolongsautolongslongstypeconcisetyperoaringcompressRunOnSerializationtruetypestreamendpointreplicastaskCounttaskCount
+LZ4LZFuncompressedLZ4LZ4LZFuncompressednoneLZ4autolongsautolongslongstypeconcisetyperoaringtypestreamendpointreplicastaskCounttaskCount
Review Comment:
it has apparently always been like that https://github.com/apache/druid/pull/8548/files#diff-f0ad954629586689c64c31c7467c9ca21e5832971048aaa83f3b00532b7d05aaR643 :shrug:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org