You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "junegunn (via GitHub)" <gi...@apache.org> on 2025/01/31 05:32:50 UTC
[PR] HBASE-29107 shell: Improve 'count' performance [hbase]
junegunn opened a new pull request, #6656:
URL: https://github.com/apache/hbase/pull/6656
(no comment)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-29107 shell: Improve 'count' performance [hbase]
Posted by "NihalJain (via GitHub)" <gi...@apache.org>.
NihalJain commented on code in PR #6656:
URL: https://github.com/apache/hbase/pull/6656#discussion_r1942288483
##########
hbase-shell/src/main/ruby/hbase/table.rb:
##########
@@ -321,24 +321,21 @@ def _append_internal(row, column, value, args = {})
def _count_internal(interval = 1000, scan = nil, cacheBlocks=false)
raise(ArgumentError, 'Scan argument should be org.apache.hadoop.hbase.client.Scan') \
unless scan.nil? || scan.is_a?(org.apache.hadoop.hbase.client.Scan)
- # We can safely set scanner caching with the first key only filter
- if scan.nil?
- scan = org.apache.hadoop.hbase.client.Scan.new
- scan.setCacheBlocks(cacheBlocks)
- scan.setCaching(10)
- scan.setFilter(org.apache.hadoop.hbase.filter.FirstKeyOnlyFilter.new)
- else
- scan.setCacheBlocks(cacheBlocks)
- filter = scan.getFilter
- firstKeyOnlyFilter = org.apache.hadoop.hbase.filter.FirstKeyOnlyFilter.new
- if filter.nil?
- scan.setFilter(firstKeyOnlyFilter)
- else
- firstKeyOnlyFilter.setReversed(filter.isReversed)
- scan.setFilter(org.apache.hadoop.hbase.filter.FilterList.new(filter, firstKeyOnlyFilter))
- end
+ scan ||= org.apache.hadoop.hbase.client.Scan.new
+ scan.setCacheBlocks(cacheBlocks)
+
+ # Optimize counting by using FirstKeyOnlyFilter and KeyOnlyFilter
+ filters = [
+ org.apache.hadoop.hbase.filter.FirstKeyOnlyFilter.new,
+ org.apache.hadoop.hbase.filter.KeyOnlyFilter.new
Review Comment:
Interesting! May be we would want to re-visit RowCounter mapreduce job (with another jira of course) post this change.
Ref: https://github.com/apache/hbase/blob/master/hbase-mapreduce/src/main/java/org/apache/hadoop/hbase/mapreduce/RowCounter.java#L318
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-29107 shell: Improve 'count' performance [hbase]
Posted by "junegunn (via GitHub)" <gi...@apache.org>.
junegunn commented on code in PR #6656:
URL: https://github.com/apache/hbase/pull/6656#discussion_r1942357008
##########
hbase-shell/src/main/ruby/hbase/table.rb:
##########
@@ -321,24 +321,21 @@ def _append_internal(row, column, value, args = {})
def _count_internal(interval = 1000, scan = nil, cacheBlocks=false)
raise(ArgumentError, 'Scan argument should be org.apache.hadoop.hbase.client.Scan') \
unless scan.nil? || scan.is_a?(org.apache.hadoop.hbase.client.Scan)
- # We can safely set scanner caching with the first key only filter
- if scan.nil?
- scan = org.apache.hadoop.hbase.client.Scan.new
- scan.setCacheBlocks(cacheBlocks)
- scan.setCaching(10)
- scan.setFilter(org.apache.hadoop.hbase.filter.FirstKeyOnlyFilter.new)
- else
- scan.setCacheBlocks(cacheBlocks)
- filter = scan.getFilter
- firstKeyOnlyFilter = org.apache.hadoop.hbase.filter.FirstKeyOnlyFilter.new
- if filter.nil?
- scan.setFilter(firstKeyOnlyFilter)
- else
- firstKeyOnlyFilter.setReversed(filter.isReversed)
- scan.setFilter(org.apache.hadoop.hbase.filter.FilterList.new(filter, firstKeyOnlyFilter))
- end
+ scan ||= org.apache.hadoop.hbase.client.Scan.new
+ scan.setCacheBlocks(cacheBlocks)
+
+ # Optimize counting by using FirstKeyOnlyFilter and KeyOnlyFilter
+ filters = [
+ org.apache.hadoop.hbase.filter.FirstKeyOnlyFilter.new,
+ org.apache.hadoop.hbase.filter.KeyOnlyFilter.new
Review Comment:
@NihalJain Good point! It's a low-hanging fruit. I'll go ahead and open an issue for it.
https://issues.apache.org/jira/browse/HBASE-29112
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-29107 shell: Improve 'count' performance [hbase]
Posted by "junegunn (via GitHub)" <gi...@apache.org>.
junegunn commented on code in PR #6656:
URL: https://github.com/apache/hbase/pull/6656#discussion_r1941539696
##########
hbase-shell/src/main/ruby/hbase/table.rb:
##########
@@ -321,24 +321,21 @@ def _append_internal(row, column, value, args = {})
def _count_internal(interval = 1000, scan = nil, cacheBlocks=false)
raise(ArgumentError, 'Scan argument should be org.apache.hadoop.hbase.client.Scan') \
unless scan.nil? || scan.is_a?(org.apache.hadoop.hbase.client.Scan)
- # We can safely set scanner caching with the first key only filter
- if scan.nil?
- scan = org.apache.hadoop.hbase.client.Scan.new
- scan.setCacheBlocks(cacheBlocks)
- scan.setCaching(10)
- scan.setFilter(org.apache.hadoop.hbase.filter.FirstKeyOnlyFilter.new)
- else
- scan.setCacheBlocks(cacheBlocks)
- filter = scan.getFilter
- firstKeyOnlyFilter = org.apache.hadoop.hbase.filter.FirstKeyOnlyFilter.new
- if filter.nil?
- scan.setFilter(firstKeyOnlyFilter)
- else
- firstKeyOnlyFilter.setReversed(filter.isReversed)
- scan.setFilter(org.apache.hadoop.hbase.filter.FilterList.new(filter, firstKeyOnlyFilter))
- end
+ scan ||= org.apache.hadoop.hbase.client.Scan.new
+ scan.setCacheBlocks(cacheBlocks)
+
+ # Optimize counting by using FirstKeyOnlyFilter and KeyOnlyFilter
+ filters = [
+ org.apache.hadoop.hbase.filter.FirstKeyOnlyFilter.new,
+ org.apache.hadoop.hbase.filter.KeyOnlyFilter.new
Review Comment:
True, FirstCellOnlyFilter should be a more fitting name, but the ship has sailed long ago.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-29107 shell: Improve 'count' performance [hbase]
Posted by "virajjasani (via GitHub)" <gi...@apache.org>.
virajjasani commented on code in PR #6656:
URL: https://github.com/apache/hbase/pull/6656#discussion_r1942260500
##########
hbase-shell/src/main/ruby/hbase/table.rb:
##########
@@ -321,24 +321,21 @@ def _append_internal(row, column, value, args = {})
def _count_internal(interval = 1000, scan = nil, cacheBlocks=false)
raise(ArgumentError, 'Scan argument should be org.apache.hadoop.hbase.client.Scan') \
unless scan.nil? || scan.is_a?(org.apache.hadoop.hbase.client.Scan)
- # We can safely set scanner caching with the first key only filter
- if scan.nil?
- scan = org.apache.hadoop.hbase.client.Scan.new
- scan.setCacheBlocks(cacheBlocks)
- scan.setCaching(10)
- scan.setFilter(org.apache.hadoop.hbase.filter.FirstKeyOnlyFilter.new)
- else
- scan.setCacheBlocks(cacheBlocks)
- filter = scan.getFilter
- firstKeyOnlyFilter = org.apache.hadoop.hbase.filter.FirstKeyOnlyFilter.new
- if filter.nil?
- scan.setFilter(firstKeyOnlyFilter)
- else
- firstKeyOnlyFilter.setReversed(filter.isReversed)
- scan.setFilter(org.apache.hadoop.hbase.filter.FilterList.new(filter, firstKeyOnlyFilter))
- end
+ scan ||= org.apache.hadoop.hbase.client.Scan.new
+ scan.setCacheBlocks(cacheBlocks)
+
+ # Optimize counting by using FirstKeyOnlyFilter and KeyOnlyFilter
+ filters = [
+ org.apache.hadoop.hbase.filter.FirstKeyOnlyFilter.new,
+ org.apache.hadoop.hbase.filter.KeyOnlyFilter.new
Review Comment:
If KeyOnlyFilter alone works, many applications can take advantage of it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-29107 shell: Improve 'count' performance [hbase]
Posted by "Apache9 (via GitHub)" <gi...@apache.org>.
Apache9 commented on code in PR #6656:
URL: https://github.com/apache/hbase/pull/6656#discussion_r1941293004
##########
hbase-shell/src/main/ruby/hbase/table.rb:
##########
@@ -321,24 +321,21 @@ def _append_internal(row, column, value, args = {})
def _count_internal(interval = 1000, scan = nil, cacheBlocks=false)
raise(ArgumentError, 'Scan argument should be org.apache.hadoop.hbase.client.Scan') \
unless scan.nil? || scan.is_a?(org.apache.hadoop.hbase.client.Scan)
- # We can safely set scanner caching with the first key only filter
- if scan.nil?
- scan = org.apache.hadoop.hbase.client.Scan.new
- scan.setCacheBlocks(cacheBlocks)
- scan.setCaching(10)
- scan.setFilter(org.apache.hadoop.hbase.filter.FirstKeyOnlyFilter.new)
- else
- scan.setCacheBlocks(cacheBlocks)
- filter = scan.getFilter
- firstKeyOnlyFilter = org.apache.hadoop.hbase.filter.FirstKeyOnlyFilter.new
- if filter.nil?
- scan.setFilter(firstKeyOnlyFilter)
- else
- firstKeyOnlyFilter.setReversed(filter.isReversed)
- scan.setFilter(org.apache.hadoop.hbase.filter.FilterList.new(filter, firstKeyOnlyFilter))
- end
+ scan ||= org.apache.hadoop.hbase.client.Scan.new
+ scan.setCacheBlocks(cacheBlocks)
+
+ # Optimize counting by using FirstKeyOnlyFilter and KeyOnlyFilter
+ filters = [
+ org.apache.hadoop.hbase.filter.FirstKeyOnlyFilter.new,
+ org.apache.hadoop.hbase.filter.KeyOnlyFilter.new
Review Comment:
The FirstKeyOnlyFilter means returning the first key and value, not only the key, the name is a bit misleading...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-29107 shell: Improve 'count' performance [hbase]
Posted by "junegunn (via GitHub)" <gi...@apache.org>.
junegunn commented on code in PR #6656:
URL: https://github.com/apache/hbase/pull/6656#discussion_r1942268264
##########
hbase-shell/src/main/ruby/hbase/table.rb:
##########
@@ -321,24 +321,21 @@ def _append_internal(row, column, value, args = {})
def _count_internal(interval = 1000, scan = nil, cacheBlocks=false)
raise(ArgumentError, 'Scan argument should be org.apache.hadoop.hbase.client.Scan') \
unless scan.nil? || scan.is_a?(org.apache.hadoop.hbase.client.Scan)
- # We can safely set scanner caching with the first key only filter
- if scan.nil?
- scan = org.apache.hadoop.hbase.client.Scan.new
- scan.setCacheBlocks(cacheBlocks)
- scan.setCaching(10)
- scan.setFilter(org.apache.hadoop.hbase.filter.FirstKeyOnlyFilter.new)
- else
- scan.setCacheBlocks(cacheBlocks)
- filter = scan.getFilter
- firstKeyOnlyFilter = org.apache.hadoop.hbase.filter.FirstKeyOnlyFilter.new
- if filter.nil?
- scan.setFilter(firstKeyOnlyFilter)
- else
- firstKeyOnlyFilter.setReversed(filter.isReversed)
- scan.setFilter(org.apache.hadoop.hbase.filter.FilterList.new(filter, firstKeyOnlyFilter))
- end
+ scan ||= org.apache.hadoop.hbase.client.Scan.new
+ scan.setCacheBlocks(cacheBlocks)
+
+ # Optimize counting by using FirstKeyOnlyFilter and KeyOnlyFilter
+ filters = [
+ org.apache.hadoop.hbase.filter.FirstKeyOnlyFilter.new,
+ org.apache.hadoop.hbase.filter.KeyOnlyFilter.new
Review Comment:
> Why not only use KeyOnlyFilter?
A record can have many number of columns. If we only use KeyOnlyFilter, we'll fetch that many cells. Their values will be empty, but they're still unnecessary for counting, we just need one cell.
```sh
# No filter
R1: CF:CQ1="foo" CF:CQ2="bar" CF:CQ3="baz"
# KeyOnlyFilter
R1: CF:CQ1="" CF:CQ2="" CF:CQ3=""
# FirstKeyOnlyFilter
R1: CF:CQ1="foo"
# FirstKeyOnlyFilter + KeyOnlyFilter
R1: CF:CQ1=""
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-29107 shell: Improve 'count' performance [hbase]
Posted by "Apache-HBase (via GitHub)" <gi...@apache.org>.
Apache-HBase commented on PR #6656:
URL: https://github.com/apache/hbase/pull/6656#issuecomment-2626367465
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 27s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
|||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 3m 13s | | master passed |
| +1 :green_heart: | spotless | 0m 45s | | branch has no errors when running spotless:check. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 2m 57s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks issues. |
| -0 :warning: | rubocop | 0m 16s | [/results-rubocop.txt](https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6656/1/artifact/yetus-general-check/output/results-rubocop.txt) | The patch generated 4 new + 152 unchanged - 10 fixed = 156 total (was 162) |
| +1 :green_heart: | spotless | 0m 43s | | patch has no errors when running spotless:check. |
|||| _ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 9s | | The patch does not generate ASF License warnings. |
| | | 9m 52s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6656/1/artifact/yetus-general-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/6656 |
| Optional Tests | dupname asflicense javac codespell detsecrets rubocop spotless |
| uname | Linux e6b4dc24f192 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 915d721da16fd0f08ed8727e91b0f20f17923985 |
| Default Java | Eclipse Adoptium-17.0.11+9 |
| Max. process+thread count | 83 (vs. ulimit of 30000) |
| modules | C: hbase-shell U: hbase-shell |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6656/1/console |
| versions | git=2.34.1 maven=3.9.8 rubocop=1.37.1 |
| Powered by | Apache Yetus 0.15.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-29107 shell: Improve 'count' performance [hbase]
Posted by "virajjasani (via GitHub)" <gi...@apache.org>.
virajjasani commented on code in PR #6656:
URL: https://github.com/apache/hbase/pull/6656#discussion_r1938212555
##########
hbase-shell/src/main/ruby/hbase/table.rb:
##########
@@ -321,24 +321,21 @@ def _append_internal(row, column, value, args = {})
def _count_internal(interval = 1000, scan = nil, cacheBlocks=false)
raise(ArgumentError, 'Scan argument should be org.apache.hadoop.hbase.client.Scan') \
unless scan.nil? || scan.is_a?(org.apache.hadoop.hbase.client.Scan)
- # We can safely set scanner caching with the first key only filter
- if scan.nil?
- scan = org.apache.hadoop.hbase.client.Scan.new
- scan.setCacheBlocks(cacheBlocks)
- scan.setCaching(10)
- scan.setFilter(org.apache.hadoop.hbase.filter.FirstKeyOnlyFilter.new)
- else
- scan.setCacheBlocks(cacheBlocks)
- filter = scan.getFilter
- firstKeyOnlyFilter = org.apache.hadoop.hbase.filter.FirstKeyOnlyFilter.new
- if filter.nil?
- scan.setFilter(firstKeyOnlyFilter)
- else
- firstKeyOnlyFilter.setReversed(filter.isReversed)
- scan.setFilter(org.apache.hadoop.hbase.filter.FilterList.new(filter, firstKeyOnlyFilter))
- end
+ scan ||= org.apache.hadoop.hbase.client.Scan.new
+ scan.setCacheBlocks(cacheBlocks)
+
+ # Optimize counting by using FirstKeyOnlyFilter and KeyOnlyFilter
+ filters = [
+ org.apache.hadoop.hbase.filter.FirstKeyOnlyFilter.new,
+ org.apache.hadoop.hbase.filter.KeyOnlyFilter.new
Review Comment:
Is FirstKeyOnlyFilter not sufficient?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-29107 shell: Improve 'count' performance [hbase]
Posted by "junegunn (via GitHub)" <gi...@apache.org>.
junegunn commented on code in PR #6656:
URL: https://github.com/apache/hbase/pull/6656#discussion_r1942357008
##########
hbase-shell/src/main/ruby/hbase/table.rb:
##########
@@ -321,24 +321,21 @@ def _append_internal(row, column, value, args = {})
def _count_internal(interval = 1000, scan = nil, cacheBlocks=false)
raise(ArgumentError, 'Scan argument should be org.apache.hadoop.hbase.client.Scan') \
unless scan.nil? || scan.is_a?(org.apache.hadoop.hbase.client.Scan)
- # We can safely set scanner caching with the first key only filter
- if scan.nil?
- scan = org.apache.hadoop.hbase.client.Scan.new
- scan.setCacheBlocks(cacheBlocks)
- scan.setCaching(10)
- scan.setFilter(org.apache.hadoop.hbase.filter.FirstKeyOnlyFilter.new)
- else
- scan.setCacheBlocks(cacheBlocks)
- filter = scan.getFilter
- firstKeyOnlyFilter = org.apache.hadoop.hbase.filter.FirstKeyOnlyFilter.new
- if filter.nil?
- scan.setFilter(firstKeyOnlyFilter)
- else
- firstKeyOnlyFilter.setReversed(filter.isReversed)
- scan.setFilter(org.apache.hadoop.hbase.filter.FilterList.new(filter, firstKeyOnlyFilter))
- end
+ scan ||= org.apache.hadoop.hbase.client.Scan.new
+ scan.setCacheBlocks(cacheBlocks)
+
+ # Optimize counting by using FirstKeyOnlyFilter and KeyOnlyFilter
+ filters = [
+ org.apache.hadoop.hbase.filter.FirstKeyOnlyFilter.new,
+ org.apache.hadoop.hbase.filter.KeyOnlyFilter.new
Review Comment:
@NihalJain Good point! It's a low-hanging fruit. I'll go ahead and open an issue for it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-29107 shell: Improve 'count' performance [hbase]
Posted by "junegunn (via GitHub)" <gi...@apache.org>.
junegunn commented on code in PR #6656:
URL: https://github.com/apache/hbase/pull/6656#discussion_r1938222667
##########
hbase-shell/src/main/ruby/hbase/table.rb:
##########
@@ -321,24 +321,21 @@ def _append_internal(row, column, value, args = {})
def _count_internal(interval = 1000, scan = nil, cacheBlocks=false)
raise(ArgumentError, 'Scan argument should be org.apache.hadoop.hbase.client.Scan') \
unless scan.nil? || scan.is_a?(org.apache.hadoop.hbase.client.Scan)
- # We can safely set scanner caching with the first key only filter
- if scan.nil?
- scan = org.apache.hadoop.hbase.client.Scan.new
- scan.setCacheBlocks(cacheBlocks)
- scan.setCaching(10)
- scan.setFilter(org.apache.hadoop.hbase.filter.FirstKeyOnlyFilter.new)
- else
- scan.setCacheBlocks(cacheBlocks)
- filter = scan.getFilter
- firstKeyOnlyFilter = org.apache.hadoop.hbase.filter.FirstKeyOnlyFilter.new
- if filter.nil?
- scan.setFilter(firstKeyOnlyFilter)
- else
- firstKeyOnlyFilter.setReversed(filter.isReversed)
- scan.setFilter(org.apache.hadoop.hbase.filter.FilterList.new(filter, firstKeyOnlyFilter))
- end
+ scan ||= org.apache.hadoop.hbase.client.Scan.new
+ scan.setCacheBlocks(cacheBlocks)
+
+ # Optimize counting by using FirstKeyOnlyFilter and KeyOnlyFilter
+ filters = [
+ org.apache.hadoop.hbase.filter.FirstKeyOnlyFilter.new,
+ org.apache.hadoop.hbase.filter.KeyOnlyFilter.new
Review Comment:
If the first cell is large. You can find the test result in the issue description.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-29107 shell: Improve 'count' performance [hbase]
Posted by "virajjasani (via GitHub)" <gi...@apache.org>.
virajjasani commented on code in PR #6656:
URL: https://github.com/apache/hbase/pull/6656#discussion_r1942259652
##########
hbase-shell/src/main/ruby/hbase/table.rb:
##########
@@ -321,24 +321,21 @@ def _append_internal(row, column, value, args = {})
def _count_internal(interval = 1000, scan = nil, cacheBlocks=false)
raise(ArgumentError, 'Scan argument should be org.apache.hadoop.hbase.client.Scan') \
unless scan.nil? || scan.is_a?(org.apache.hadoop.hbase.client.Scan)
- # We can safely set scanner caching with the first key only filter
- if scan.nil?
- scan = org.apache.hadoop.hbase.client.Scan.new
- scan.setCacheBlocks(cacheBlocks)
- scan.setCaching(10)
- scan.setFilter(org.apache.hadoop.hbase.filter.FirstKeyOnlyFilter.new)
- else
- scan.setCacheBlocks(cacheBlocks)
- filter = scan.getFilter
- firstKeyOnlyFilter = org.apache.hadoop.hbase.filter.FirstKeyOnlyFilter.new
- if filter.nil?
- scan.setFilter(firstKeyOnlyFilter)
- else
- firstKeyOnlyFilter.setReversed(filter.isReversed)
- scan.setFilter(org.apache.hadoop.hbase.filter.FilterList.new(filter, firstKeyOnlyFilter))
- end
+ scan ||= org.apache.hadoop.hbase.client.Scan.new
+ scan.setCacheBlocks(cacheBlocks)
+
+ # Optimize counting by using FirstKeyOnlyFilter and KeyOnlyFilter
+ filters = [
+ org.apache.hadoop.hbase.filter.FirstKeyOnlyFilter.new,
+ org.apache.hadoop.hbase.filter.KeyOnlyFilter.new
Review Comment:
Why not only use KeyOnlyFilter? Do we need FirstKeyOnlyFilter too if we already get perf improvement for large cells with just KeyOnlyFilter?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-29107 shell: Improve 'count' performance [hbase]
Posted by "junegunn (via GitHub)" <gi...@apache.org>.
junegunn commented on code in PR #6656:
URL: https://github.com/apache/hbase/pull/6656#discussion_r1942268264
##########
hbase-shell/src/main/ruby/hbase/table.rb:
##########
@@ -321,24 +321,21 @@ def _append_internal(row, column, value, args = {})
def _count_internal(interval = 1000, scan = nil, cacheBlocks=false)
raise(ArgumentError, 'Scan argument should be org.apache.hadoop.hbase.client.Scan') \
unless scan.nil? || scan.is_a?(org.apache.hadoop.hbase.client.Scan)
- # We can safely set scanner caching with the first key only filter
- if scan.nil?
- scan = org.apache.hadoop.hbase.client.Scan.new
- scan.setCacheBlocks(cacheBlocks)
- scan.setCaching(10)
- scan.setFilter(org.apache.hadoop.hbase.filter.FirstKeyOnlyFilter.new)
- else
- scan.setCacheBlocks(cacheBlocks)
- filter = scan.getFilter
- firstKeyOnlyFilter = org.apache.hadoop.hbase.filter.FirstKeyOnlyFilter.new
- if filter.nil?
- scan.setFilter(firstKeyOnlyFilter)
- else
- firstKeyOnlyFilter.setReversed(filter.isReversed)
- scan.setFilter(org.apache.hadoop.hbase.filter.FilterList.new(filter, firstKeyOnlyFilter))
- end
+ scan ||= org.apache.hadoop.hbase.client.Scan.new
+ scan.setCacheBlocks(cacheBlocks)
+
+ # Optimize counting by using FirstKeyOnlyFilter and KeyOnlyFilter
+ filters = [
+ org.apache.hadoop.hbase.filter.FirstKeyOnlyFilter.new,
+ org.apache.hadoop.hbase.filter.KeyOnlyFilter.new
Review Comment:
> Why not only use KeyOnlyFilter?
A record can have many number of columns. If we only use KeyOnlyFilter, we'll fetch that many cells. Their values will be empty, but they're still unnecessary for counting, we only need one cell per row.
```sh
# No filter
R1: CF:CQ1="foo" CF:CQ2="bar" CF:CQ3="baz"
# KeyOnlyFilter
R1: CF:CQ1="" CF:CQ2="" CF:CQ3=""
# FirstKeyOnlyFilter
R1: CF:CQ1="foo"
# FirstKeyOnlyFilter + KeyOnlyFilter
R1: CF:CQ1=""
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-29107 shell: Improve 'count' performance [hbase]
Posted by "Apache9 (via GitHub)" <gi...@apache.org>.
Apache9 commented on code in PR #6656:
URL: https://github.com/apache/hbase/pull/6656#discussion_r1942310771
##########
hbase-shell/src/main/ruby/hbase/table.rb:
##########
@@ -321,24 +321,21 @@ def _append_internal(row, column, value, args = {})
def _count_internal(interval = 1000, scan = nil, cacheBlocks=false)
raise(ArgumentError, 'Scan argument should be org.apache.hadoop.hbase.client.Scan') \
unless scan.nil? || scan.is_a?(org.apache.hadoop.hbase.client.Scan)
- # We can safely set scanner caching with the first key only filter
- if scan.nil?
- scan = org.apache.hadoop.hbase.client.Scan.new
- scan.setCacheBlocks(cacheBlocks)
- scan.setCaching(10)
- scan.setFilter(org.apache.hadoop.hbase.filter.FirstKeyOnlyFilter.new)
- else
- scan.setCacheBlocks(cacheBlocks)
- filter = scan.getFilter
- firstKeyOnlyFilter = org.apache.hadoop.hbase.filter.FirstKeyOnlyFilter.new
- if filter.nil?
- scan.setFilter(firstKeyOnlyFilter)
- else
- firstKeyOnlyFilter.setReversed(filter.isReversed)
- scan.setFilter(org.apache.hadoop.hbase.filter.FilterList.new(filter, firstKeyOnlyFilter))
- end
+ scan ||= org.apache.hadoop.hbase.client.Scan.new
+ scan.setCacheBlocks(cacheBlocks)
+
+ # Optimize counting by using FirstKeyOnlyFilter and KeyOnlyFilter
+ filters = [
+ org.apache.hadoop.hbase.filter.FirstKeyOnlyFilter.new,
+ org.apache.hadoop.hbase.filter.KeyOnlyFilter.new
Review Comment:
> Why not only use KeyOnlyFilter? Do we need FirstKeyOnlyFilter too if we already get perf improvement for large cells with just KeyOnlyFilter?
The 'key' in KeyOnlyFilter means the key of a Cell, not row key...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-29107 shell: Improve 'count' performance [hbase]
Posted by "Apache-HBase (via GitHub)" <gi...@apache.org>.
Apache-HBase commented on PR #6656:
URL: https://github.com/apache/hbase/pull/6656#issuecomment-2626373877
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 27s | | Docker mode activated. |
| -0 :warning: | yetus | 0m 2s | | Unprocessed flag(s): --brief-report-file --spotbugs-strict-precheck --author-ignore-list --blanks-eol-ignore-file --blanks-tabs-ignore-file --quick-hadoopcheck |
|||| _ Prechecks _ |
|||| _ master Compile Tests _ |
| +1 :green_heart: | mvninstall | 3m 16s | | master passed |
| +1 :green_heart: | javadoc | 0m 11s | | master passed |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 3m 4s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 9s | | the patch passed |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 7m 36s | | hbase-shell in the patch passed. |
| | | 15m 45s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6656/1/artifact/yetus-jdk17-hadoop3-check/output/Dockerfile |
| GITHUB PR | https://github.com/apache/hbase/pull/6656 |
| Optional Tests | javac javadoc unit |
| uname | Linux 68e743d08fe0 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/hbase-personality.sh |
| git revision | master / 915d721da16fd0f08ed8727e91b0f20f17923985 |
| Default Java | Eclipse Adoptium-17.0.11+9 |
| Test Results | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6656/1/testReport/ |
| Max. process+thread count | 2031 (vs. ulimit of 30000) |
| modules | C: hbase-shell U: hbase-shell |
| Console output | https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6656/1/console |
| versions | git=2.34.1 maven=3.9.8 |
| Powered by | Apache Yetus 0.15.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-29107 shell: Improve 'count' performance [hbase]
Posted by "virajjasani (via GitHub)" <gi...@apache.org>.
virajjasani commented on code in PR #6656:
URL: https://github.com/apache/hbase/pull/6656#discussion_r1938212555
##########
hbase-shell/src/main/ruby/hbase/table.rb:
##########
@@ -321,24 +321,21 @@ def _append_internal(row, column, value, args = {})
def _count_internal(interval = 1000, scan = nil, cacheBlocks=false)
raise(ArgumentError, 'Scan argument should be org.apache.hadoop.hbase.client.Scan') \
unless scan.nil? || scan.is_a?(org.apache.hadoop.hbase.client.Scan)
- # We can safely set scanner caching with the first key only filter
- if scan.nil?
- scan = org.apache.hadoop.hbase.client.Scan.new
- scan.setCacheBlocks(cacheBlocks)
- scan.setCaching(10)
- scan.setFilter(org.apache.hadoop.hbase.filter.FirstKeyOnlyFilter.new)
- else
- scan.setCacheBlocks(cacheBlocks)
- filter = scan.getFilter
- firstKeyOnlyFilter = org.apache.hadoop.hbase.filter.FirstKeyOnlyFilter.new
- if filter.nil?
- scan.setFilter(firstKeyOnlyFilter)
- else
- firstKeyOnlyFilter.setReversed(filter.isReversed)
- scan.setFilter(org.apache.hadoop.hbase.filter.FilterList.new(filter, firstKeyOnlyFilter))
- end
+ scan ||= org.apache.hadoop.hbase.client.Scan.new
+ scan.setCacheBlocks(cacheBlocks)
+
+ # Optimize counting by using FirstKeyOnlyFilter and KeyOnlyFilter
+ filters = [
+ org.apache.hadoop.hbase.filter.FirstKeyOnlyFilter.new,
+ org.apache.hadoop.hbase.filter.KeyOnlyFilter.new
Review Comment:
Is FirstKeyOnlyFilter alone not sufficient?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-29107 shell: Improve 'count' performance [hbase]
Posted by "junegunn (via GitHub)" <gi...@apache.org>.
junegunn commented on PR #6656:
URL: https://github.com/apache/hbase/pull/6656#issuecomment-2634517343
> So here we still support specifying CACHE, and we just remove it from the example?
Yes, the option probably shouldn't be used in most cases, so we don't show it in the examples, but we don't need to break backward compatibility.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] HBASE-29107 shell: Improve 'count' performance [hbase]
Posted by "Apache9 (via GitHub)" <gi...@apache.org>.
Apache9 merged PR #6656:
URL: https://github.com/apache/hbase/pull/6656
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@hbase.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org