You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@accumulo.apache.org by z11373 <z1...@outlook.com> on 2016/01/20 17:22:26 UTC
question on data block cache
Hi,
I see the data cache hit rate percentage is in 80s.
My client app sends a lot of small scan requests, and looking at per tablet
details, I see the same number of tablets (where the data being queried
from). We set the data cache size to 16GB, so I'd think it should have
plenty of room. Is there a way for me to see if the percentage doesn't get
higher (would expect in 90s if not 100%) because of 16GB not big enough, or
it's just there are many data/tablets to load from?
Thanks,
Z
--
View this message in context: http://apache-accumulo.1065345.n5.nabble.com/question-on-data-block-cache-tp15906.html
Sent from the Developers mailing list archive at Nabble.com.
Re: question on data block cache
Posted by Eric Newton <er...@gmail.com>.
There are some debug-level cache statistics printed to the tserver logs
periodically. It will take some digging around to figure out what they
mean. They might help.
Re: question on data block cache
Posted by Keith Turner <ke...@deenlo.com>.
On Fri, Jan 22, 2016 at 2:22 PM, z11373 <z1...@outlook.com> wrote:
> Another observation which I really want to know why.
>
> I see 1 tablet server consistently shows high number of waiting scans, i.e.
> '16 (435)' or '14 (476)' on Running Scans column, while the other 19 tablet
> servers all showing '0 (0)' or sometimes showing single digit number. The
> thing I don't understand is what I've seen from the Monitor UI for a table
> at one snapshot for example, and this is more or less consistent from time
> to time (when the test clients are still running).
>
> Server | Query | Running Scans
> ====================
> TServer1 | 24 | 0 (0)
> TServer2 | 37 | 0 (0)
> TServer3 | 40 | 0 (0)
> TServer4 | 1 | 0 (0)
> TServer5 | 0 | 0 (0)
> TServer6 | 57 | 0 (0)
> TServer7 | 5 | 0 (0)
> TServer8 | 4 | 0 (0)
> TServer9 | 7 | 0 (0)
> TServer10 | 0 | 0 (0)
> TServer11 | 56 | 0 (0)
> TServer12 | 124 | 0 (0)
> TServer13 | 25 | 0 (0)
> TServer14 | 121 | 16 (435)
> TServer15 | 36 | 0 (0)
> TServer16 | 21 | 0 (0)
> TServer17 | 3 | 0 (0)
> TServer18 | 107 | 0 (0)
> TServer19 | 21 | 0 (0)
> TServer20 | 92 | 0 (0)
>
> I'd really appreciate if someone could explain what may have happened, or
> can point me to the right direction for starting the investigation. For the
> record, there are 50 test clients, each may create multiple scanners to
> query different data from that table.
>
One starting point is to see what tablets are on that tablet server. One
way to do this a grep like the following in the accumulo shell (note this
command is from memory so it may have mistakes)
grep -t accumulo.metadata -c loc <TServer14 addr as it would appear in
metadata table>
In the output of this, rows will contain <table id>:<tablet end row>.
Maybe the tablet end rows can give you a clue.
You can narrow the grep by restricting the row range using the table id.
grep -t accumulo.metadata -b <table id> -e <table id>~ -c loc
<TServer14 addr as it would appear in metadata table>
There is a command in the shell to get table ids.
> Thanks,
> Z
>
>
>
> --
> View this message in context:
> http://apache-accumulo.1065345.n5.nabble.com/question-on-data-block-cache-tp15906p15934.html
> Sent from the Developers mailing list archive at Nabble.com.
>
Re: question on data block cache
Posted by Josh Elser <jo...@gmail.com>.
z11373 wrote:
> Thanks Josh!
> Ok, here I add column Hosted Tablets and Entries to the table below for
> additional information.
> As we can see the tablets are distributed evenly to all tablet servers, and
> the one with highest load has the highest number of entries (> 1B), there
> are few tablet servers have> 700M entries, which are not really far away.
> I'd admit the data distribution likely not great, because URL is used as row
> id value (so many of them share same prefix), and it's almost impossible to
> set the presplit points, unless we know what the data value would be.
> Instead of specifying split point strings, I wish Accumulo has feature to
> allow us to specify x number of tablets, and it will automatically split y
> entries across those x tablets :-)
Remember that tables are not static entities. As you add more data, you
want them to keep splitting, that's why splitting is intended to run off
of size of tablets. The manual split generation by row is meant to help
you "seed" a table.
> Follow up questions:
> 1. The test queries are generated randomly, so theoretically I'd say the
> likelihood most requests coming to 1 tablet server should be slim, but with
> the fact of URL is used as row id value, then that may be possible. What
> does the number in Query column indicate? Is that the number of entries
> returned, or number of reads?
You should know what your distribution of data looks like and what you
queries look like. It sounds like you created a big hotspot in your
table. The Query column is number of entries per second, IIRC. There's a
collapsed legend on the data shown on the monitor which you can check.
> 2. Looking at sample table below, is there a way to find out the ranges of
> all tablets hosted on TServer14? I am thinking to write a small program to
> scan all row ids from that tablet server, and find the values which would
> become the split points, which then I can add the splits to the table, and
> re-run my tests to see if it resolves the issue.
This information can be extracted from the accumulo.metadata table, but
I'm not sure if we have a utility for it I can directly refer you to.
But yes, find the hotspot of data on that server, and add splits.
You can also try to use the listscans shell command to see the open
scans going to particular tservers.
> Regarding your other question, yes, I saw a few occasion when refreshing the
> page, which it shows number of active scans was not 16, and yet there were
> waiting scans, so it's not like 1-2 times.
>
> Server | Hosted Tablets | Entries | Query | Running Scans
> ====================================
> TServer1 | 47 | 548.43M | 24 | 0 (0)
> TServer2 | 47 | 708.70M | 37 | 0 (0)
> TServer3 | 47 | 597.88M | 40 | 0 (0)
> TServer4 | 47 | 382.72M | 1 | 0 (0)
> TServer5 | 47 | 756.77M | 0 | 0 (0)
> TServer6 | 47 | 654.38M | 57 | 0 (0)
> TServer7 | 47 | 695.09M | 5 | 0 (0)
> TServer8 | 47 | 637.94M | 4 | 0 (0)
> TServer9 | 47 | 541.74M | 7 | 0 (0)
> TServer10 | 46 | 625.12M | 0 | 0 (0)
> TServer11 | 46 | 248.75M | 56 | 0 (0)
> TServer12 | 46 | 368.87M | 124 | 0 (0)
> TServer13 | 46 | 292.73M | 25 | 0 (0)
> TServer14 | 46 | 1.05B | 121 | 16 (435)
> TServer15 | 46 | 442.23M | 36 | 0 (0)
> TServer16 | 46 | 800.67M | 21 | 0 (0)
> TServer17 | 46 | 689.81M | 3 | 0 (0)
> TServer18 | 46 | 351.86M | 107 | 0 (0)
> TServer19 | 47 | 941.17M | 21 | 0 (0)
> TServer20 | 47 | 257.99M | 92 | 0 (0)
>
>
> Thanks,
> Z
>
>
>
>
> --
> View this message in context: http://apache-accumulo.1065345.n5.nabble.com/question-on-data-block-cache-tp15906p15937.html
> Sent from the Developers mailing list archive at Nabble.com.
Re: question on data block cache
Posted by z11373 <z1...@outlook.com>.
Thanks Josh!
Ok, here I add column Hosted Tablets and Entries to the table below for
additional information.
As we can see the tablets are distributed evenly to all tablet servers, and
the one with highest load has the highest number of entries (> 1B), there
are few tablet servers have > 700M entries, which are not really far away.
I'd admit the data distribution likely not great, because URL is used as row
id value (so many of them share same prefix), and it's almost impossible to
set the presplit points, unless we know what the data value would be.
Instead of specifying split point strings, I wish Accumulo has feature to
allow us to specify x number of tablets, and it will automatically split y
entries across those x tablets :-)
Follow up questions:
1. The test queries are generated randomly, so theoretically I'd say the
likelihood most requests coming to 1 tablet server should be slim, but with
the fact of URL is used as row id value, then that may be possible. What
does the number in Query column indicate? Is that the number of entries
returned, or number of reads?
2. Looking at sample table below, is there a way to find out the ranges of
all tablets hosted on TServer14? I am thinking to write a small program to
scan all row ids from that tablet server, and find the values which would
become the split points, which then I can add the splits to the table, and
re-run my tests to see if it resolves the issue.
Regarding your other question, yes, I saw a few occasion when refreshing the
page, which it shows number of active scans was not 16, and yet there were
waiting scans, so it's not like 1-2 times.
Server | Hosted Tablets | Entries | Query | Running Scans
====================================
TServer1 | 47 | 548.43M | 24 | 0 (0)
TServer2 | 47 | 708.70M | 37 | 0 (0)
TServer3 | 47 | 597.88M | 40 | 0 (0)
TServer4 | 47 | 382.72M | 1 | 0 (0)
TServer5 | 47 | 756.77M | 0 | 0 (0)
TServer6 | 47 | 654.38M | 57 | 0 (0)
TServer7 | 47 | 695.09M | 5 | 0 (0)
TServer8 | 47 | 637.94M | 4 | 0 (0)
TServer9 | 47 | 541.74M | 7 | 0 (0)
TServer10 | 46 | 625.12M | 0 | 0 (0)
TServer11 | 46 | 248.75M | 56 | 0 (0)
TServer12 | 46 | 368.87M | 124 | 0 (0)
TServer13 | 46 | 292.73M | 25 | 0 (0)
TServer14 | 46 | 1.05B | 121 | 16 (435)
TServer15 | 46 | 442.23M | 36 | 0 (0)
TServer16 | 46 | 800.67M | 21 | 0 (0)
TServer17 | 46 | 689.81M | 3 | 0 (0)
TServer18 | 46 | 351.86M | 107 | 0 (0)
TServer19 | 47 | 941.17M | 21 | 0 (0)
TServer20 | 47 | 257.99M | 92 | 0 (0)
Thanks,
Z
--
View this message in context: http://apache-accumulo.1065345.n5.nabble.com/question-on-data-block-cache-tp15906p15937.html
Sent from the Developers mailing list archive at Nabble.com.
Re: question on data block cache
Posted by Josh Elser <jo...@gmail.com>.
A Tablet is hosted by one and only one TabletServer at any time. The
only data you are querying is in Tablet(s) hosted by that one TabletServer.
Add more splits to the table which you are heavily querying to spread
the load across many TabletServers. Load is parallelized in Accumulo by
creating multiple Tablets (via splits) which can be spread across
multiple servers.
z11373 wrote:
> Another observation which I really want to know why.
>
> I see 1 tablet server consistently shows high number of waiting scans, i.e.
> '16 (435)' or '14 (476)' on Running Scans column, while the other 19 tablet
> servers all showing '0 (0)' or sometimes showing single digit number. The
> thing I don't understand is what I've seen from the Monitor UI for a table
> at one snapshot for example, and this is more or less consistent from time
> to time (when the test clients are still running).
>
> Server | Query | Running Scans
> ====================
> TServer1 | 24 | 0 (0)
> TServer2 | 37 | 0 (0)
> TServer3 | 40 | 0 (0)
> TServer4 | 1 | 0 (0)
> TServer5 | 0 | 0 (0)
> TServer6 | 57 | 0 (0)
> TServer7 | 5 | 0 (0)
> TServer8 | 4 | 0 (0)
> TServer9 | 7 | 0 (0)
> TServer10 | 0 | 0 (0)
> TServer11 | 56 | 0 (0)
> TServer12 | 124 | 0 (0)
> TServer13 | 25 | 0 (0)
> TServer14 | 121 | 16 (435)
> TServer15 | 36 | 0 (0)
> TServer16 | 21 | 0 (0)
> TServer17 | 3 | 0 (0)
> TServer18 | 107 | 0 (0)
> TServer19 | 21 | 0 (0)
> TServer20 | 92 | 0 (0)
>
> I'd really appreciate if someone could explain what may have happened, or
> can point me to the right direction for starting the investigation. For the
> record, there are 50 test clients, each may create multiple scanners to
> query different data from that table.
>
> Thanks,
> Z
>
>
>
> --
> View this message in context: http://apache-accumulo.1065345.n5.nabble.com/question-on-data-block-cache-tp15906p15934.html
> Sent from the Developers mailing list archive at Nabble.com.
Re: question on data block cache
Posted by z11373 <z1...@outlook.com>.
Another observation which I really want to know why.
I see 1 tablet server consistently shows high number of waiting scans, i.e.
'16 (435)' or '14 (476)' on Running Scans column, while the other 19 tablet
servers all showing '0 (0)' or sometimes showing single digit number. The
thing I don't understand is what I've seen from the Monitor UI for a table
at one snapshot for example, and this is more or less consistent from time
to time (when the test clients are still running).
Server | Query | Running Scans
====================
TServer1 | 24 | 0 (0)
TServer2 | 37 | 0 (0)
TServer3 | 40 | 0 (0)
TServer4 | 1 | 0 (0)
TServer5 | 0 | 0 (0)
TServer6 | 57 | 0 (0)
TServer7 | 5 | 0 (0)
TServer8 | 4 | 0 (0)
TServer9 | 7 | 0 (0)
TServer10 | 0 | 0 (0)
TServer11 | 56 | 0 (0)
TServer12 | 124 | 0 (0)
TServer13 | 25 | 0 (0)
TServer14 | 121 | 16 (435)
TServer15 | 36 | 0 (0)
TServer16 | 21 | 0 (0)
TServer17 | 3 | 0 (0)
TServer18 | 107 | 0 (0)
TServer19 | 21 | 0 (0)
TServer20 | 92 | 0 (0)
I'd really appreciate if someone could explain what may have happened, or
can point me to the right direction for starting the investigation. For the
record, there are 50 test clients, each may create multiple scanners to
query different data from that table.
Thanks,
Z
--
View this message in context: http://apache-accumulo.1065345.n5.nabble.com/question-on-data-block-cache-tp15906p15934.html
Sent from the Developers mailing list archive at Nabble.com.
Re: question on data block cache
Posted by Josh Elser <jo...@gmail.com>.
A max of 16 but only 8 running is peculiar. It could be a scheduling
thing, but I'm not entirely sure. Do you see this case regularly? (not
the one you replied with later and I responded about already)
z11373 wrote:
> Thanks Josh!
> Another question came to my mind while observing the Monitor UI :-)
>
> I noticed on the Running Scans column, it occasionally showing '16(5)',
> which I interpret it as there are 21 scans, 16 are running and 5 are
> waiting. What I don't understand is I also saw something like '8(3)', which
> seems weird to me, because default max is 16, so how come there's only 8
> running and 3 have to wait, why not '11(0)'?
>
> Btw, that number represents each Scanner object created by the client,
> right?
>
> Thanks,
> Z
>
>
>
> --
> View this message in context: http://apache-accumulo.1065345.n5.nabble.com/question-on-data-block-cache-tp15906p15932.html
> Sent from the Developers mailing list archive at Nabble.com.
Re: question on data block cache
Posted by z11373 <z1...@outlook.com>.
Thanks Josh!
Another question came to my mind while observing the Monitor UI :-)
I noticed on the Running Scans column, it occasionally showing '16(5)',
which I interpret it as there are 21 scans, 16 are running and 5 are
waiting. What I don't understand is I also saw something like '8(3)', which
seems weird to me, because default max is 16, so how come there's only 8
running and 3 have to wait, why not '11(0)'?
Btw, that number represents each Scanner object created by the client,
right?
Thanks,
Z
--
View this message in context: http://apache-accumulo.1065345.n5.nabble.com/question-on-data-block-cache-tp15906p15932.html
Sent from the Developers mailing list archive at Nabble.com.
Re: question on data block cache
Posted by Josh Elser <jo...@gmail.com>.
Patches welcome for trace-level block cache logging improvements :)
We don't have a lot of granular insight into the block cache. This would
be nice.
z11373 wrote:
> Hi Eric,
> I actually care less about that, at least for now.
> Is there something simpler, like Accumulo telling the LRU data in the cache
> being pushed out?
> I don't need the exact detail, just want to know that indeed happened (and
> perhaps pretty frequent, hence won't hit 99% data cache hit ratio)
>
> Thanks,
> Z
>
>
>
> --
> View this message in context: http://apache-accumulo.1065345.n5.nabble.com/question-on-data-block-cache-tp15906p15929.html
> Sent from the Developers mailing list archive at Nabble.com.
Re: question on data block cache
Posted by z11373 <z1...@outlook.com>.
Hi Eric,
I actually care less about that, at least for now.
Is there something simpler, like Accumulo telling the LRU data in the cache
being pushed out?
I don't need the exact detail, just want to know that indeed happened (and
perhaps pretty frequent, hence won't hit 99% data cache hit ratio)
Thanks,
Z
--
View this message in context: http://apache-accumulo.1065345.n5.nabble.com/question-on-data-block-cache-tp15906p15929.html
Sent from the Developers mailing list archive at Nabble.com.
Re: question on data block cache
Posted by Eric Newton <er...@gmail.com>.
There's an easy way, but it's not exactly fast.
Change the compression type on the table and compact it. Then the numbers
provided by the "df" in the shell will be very reliable for the
uncompressed size of the data.
-Eric
On Fri, Jan 22, 2016 at 10:55 AM, z11373 <z1...@outlook.com> wrote:
> Is there a quick and sure way to find this out?
> I just want to confirm if that is the case, can't really tell from the data
> shown on Monitor UI.
>
> Thanks,
> Z
>
>
>
> --
> View this message in context:
> http://apache-accumulo.1065345.n5.nabble.com/question-on-data-block-cache-tp15906p15927.html
> Sent from the Developers mailing list archive at Nabble.com.
>
Re: question on data block cache
Posted by z11373 <z1...@outlook.com>.
Is there a quick and sure way to find this out?
I just want to confirm if that is the case, can't really tell from the data
shown on Monitor UI.
Thanks,
Z
--
View this message in context: http://apache-accumulo.1065345.n5.nabble.com/question-on-data-block-cache-tp15906p15927.html
Sent from the Developers mailing list archive at Nabble.com.
Re: question on data block cache
Posted by Eric Newton <er...@gmail.com>.
By default, the files used to store the data are compressed. The block
cache stores the data in uncompressed blocks. So, even if your stored data
is <16G/tserver, the amount it might take in memory could be much larger.
There might be bugs in computing this number, too. :-)
However, I see lots of systems running with a 100% data block cache hit
rate.
-Eric
On Fri, Jan 22, 2016 at 9:49 AM, z11373 <z1...@outlook.com> wrote:
> Thanks Eric! Yes, the data block cache is already enabled. From the table
> config, I see:
>
> default | table.cache.block.enable .................. | false
> site | @override .............................. | true
>
> Actually, there is no ingestion task running at that time when I ran the
> tests (which only do read).
> What is other possibility? Looking at the total tablets and total entries
> being queried from the Monitor UI, they are not like big number, so I have
> some doubt that 16GB data cache is not enough space, but I could be wrong.
>
> Thanks,
> Z
>
>
>
> --
> View this message in context:
> http://apache-accumulo.1065345.n5.nabble.com/question-on-data-block-cache-tp15906p15925.html
> Sent from the Developers mailing list archive at Nabble.com.
>
Re: question on data block cache
Posted by z11373 <z1...@outlook.com>.
Thanks Eric! Yes, the data block cache is already enabled. From the table
config, I see:
default | table.cache.block.enable .................. | false
site | @override .............................. | true
Actually, there is no ingestion task running at that time when I ran the
tests (which only do read).
What is other possibility? Looking at the total tablets and total entries
being queried from the Monitor UI, they are not like big number, so I have
some doubt that 16GB data cache is not enough space, but I could be wrong.
Thanks,
Z
--
View this message in context: http://apache-accumulo.1065345.n5.nabble.com/question-on-data-block-cache-tp15906p15925.html
Sent from the Developers mailing list archive at Nabble.com.
Re: question on data block cache
Posted by Eric Newton <er...@gmail.com>.
Assuming you have on-going ingest, accumulo is going to re-sort/organize
your data. The new files that result will cause cache hit failures.
Make sure you enable data caching for your table since it is not turned on
for user tables by default.
-Eric
On Wed, Jan 20, 2016 at 11:22 AM, z11373 <z1...@outlook.com> wrote:
> Hi,
> I see the data cache hit rate percentage is in 80s.
> My client app sends a lot of small scan requests, and looking at per tablet
> details, I see the same number of tablets (where the data being queried
> from). We set the data cache size to 16GB, so I'd think it should have
> plenty of room. Is there a way for me to see if the percentage doesn't get
> higher (would expect in 90s if not 100%) because of 16GB not big enough, or
> it's just there are many data/tablets to load from?
>
> Thanks,
> Z
>
>
>
> --
> View this message in context:
> http://apache-accumulo.1065345.n5.nabble.com/question-on-data-block-cache-tp15906.html
> Sent from the Developers mailing list archive at Nabble.com.
>