You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@phoenix.apache.org by "Riesland, Zack" <Za...@sensus.com> on 2015/06/26 18:37:32 UTC
How to count table rows from Java?
I wrote a Java program that runs nightly and collects metrics about our hive tables.
I would like to include HBase tables in this as well.
Since select count(*) is slow and not recommended on Phoenix, what are my alternatives from Java?
Is there a way to call org.apache.hadoop.hbase.mapreduce.RowCounter from java and get results in some kind of result set?
Thanks for any info!
RE: How to count table rows from Java?
Posted by "Riesland, Zack" <Za...@sensus.com>.
Perfect.
Thank you so much!
-----Original Message-----
From: James Taylor [mailto:jamestaylor@apache.org]
Sent: Monday, June 29, 2015 11:51 AM
To: user
Subject: Re: How to count table rows from Java?
Server side HBase properties such as hbase.rpc.timeout need to be set in your hbase-sites.xml. There's no API to set these.
The phoenix.query.timeoutMs can be set using
http://docs.oracle.com/javase/7/docs/api/java/sql/Statement.html#setQueryTimeout(int)
prior to executing the query. The units in this case is seconds, not milliseconds.
The phoenix.query.timeoutMs can also be set at connection-time in the properties like this:
Properties props = new Properties();
props.setProperty("phoenix.query.timeoutMs", Long.toString("600000"));
Connection conn = DriverManager.getConnection(url, props);
This would set the timeout of a query for any statement executing through the connection to 600000 milliseconds (10mins).
You can also set the phoenix.query.timeoutMs in your client-side hbase-sites.xml and it'll be used for the query timeout for all connections.
Thanks,
James
On Mon, Jun 29, 2015 at 2:44 AM, Riesland, Zack <Za...@sensus.com> wrote:
> Thanks, James!
>
>
>
> Can you point me to some instructions or some syntax for setting those
> timeout values in Java code?
>
>
>
> I’ve seen lots of information about setting them on the classpath when
> launching different query clients. But how do I set a high timeout so
> that I can do a large query via Java/JDBC?
>
>
>
> Thanks!
>
>
>
> From: James Taylor [mailto:jamestaylor@apache.org]
> Sent: Friday, June 26, 2015 2:14 PM
> To: user@phoenix.apache.org
> Subject: Re: How to count table rows from Java?
>
>
>
> Zach,
>
> I wouldn't at all say that doing a count(*) is not recommended. It's
> important to know that 1) this requires a full table scan and 2) this
> is done by Phoenix asynchronously. You'll need to set the timeouts
> high enough for this to complete. Phoenix will be much faster than
> running a MR job, but the MR job runs asynchronously.
>
>
>
> To ensure your stats are up to date, run a major compaction on your
> table as that updates the stats. Also, if you have wide rows, consider
> using multiple column families so that the count(*) doesn't have to
> traverse all of your data.
>
>
>
> Thanks,
>
> James
>
>
> On Friday, June 26, 2015, Nick Dimiduk <nd...@gmail.com> wrote:
>
> RowCounter is s mapreduce program. After the program completes
> execution of the job, it returns information about that job, including job counters.
> RowCounter includes its counts in the job counters, so they're easily
> accessed programmatically from the returned object. It's not a
> ResultSet, but it should work none the less.
>
> On Friday, June 26, 2015, Riesland, Zack <Za...@sensus.com> wrote:
>
> I wrote a Java program that runs nightly and collects metrics about
> our hive tables.
>
>
>
> I would like to include HBase tables in this as well.
>
>
>
> Since select count(*) is slow and not recommended on Phoenix, what are
> my alternatives from Java?
>
>
>
> Is there a way to call org.apache.hadoop.hbase.mapreduce.RowCounter
> from java and get results in some kind of result set?
>
>
>
> Thanks for any info!
>
>
>
>
Re: How to count table rows from Java?
Posted by James Taylor <ja...@apache.org>.
Support for setQueryTimeout was added in our 4.3 release. Support for
setting the connection property has been around for a while, though. Best
bet is to add it to your hbase-sites.xml on the client side and make sure
it's on the class path (search our mailing list, as this has been discussed
many times).
On Mon, Jun 29, 2015 at 11:32 AM, Riesland, Zack <Za...@sensus.com>
wrote:
> I REALLY hate to be a bother, but I just can't seem to get this timeout
> stuff to work for me.
>
> It looks like setQueryTimeout() is not supported (I get a
> java.sql.SQLFeatureNotSupportedException). I'm guessing maybe we need a
> newer version of Phoenix ?
>
> And I also tried the property approach:
> props.setProperty("phoenix.query.timeoutMs", "900000");
>
> But my query (a simple select count(*) ) still fails after a few minutes
> with the errors below.
>
> Anybody know what's going on here?
>
>
> org.apache.phoenix.exception.PhoenixIOException:
> org.apache.phoenix.exception.PhoenixIOException: Failed after attempts=36,
> exceptions:
> Mon Jun 29 14:08:49 EDT 2015, null, java.net.SocketTimeoutException:
> callTimeout=60000, callDuration=63169: row 'ABRDN_B7' on table
> 'FMA.MY_TABLE' at
> region=FMA.MY_TABLE,ABRDN_B7\x00\x00\x00\x00\x00,1435243688700.e4a6e5a437db9de9826a6d9d466a6a66.,
> hostname=MY_HOST,60020,1435572929067, seqNum=23
>
> at
> org.apache.phoenix.util.ServerUtil.parseServerException(ServerUtil.java:107)
> at
> org.apache.phoenix.iterate.ParallelIterators.getIterators(ParallelIterators.java:527)
> at
> org.apache.phoenix.iterate.ConcatResultIterator.getIterators(ConcatResultIterator.java:44)
> at
> org.apache.phoenix.iterate.ConcatResultIterator.currentIterator(ConcatResultIterator.java:66)
> at
> org.apache.phoenix.iterate.ConcatResultIterator.next(ConcatResultIterator.java:86)
> at
> org.apache.phoenix.iterate.GroupedAggregatingResultIterator.next(GroupedAggregatingResultIterator.java:68)
> at
> org.apache.phoenix.iterate.UngroupedAggregatingResultIterator.next(UngroupedAggregatingResultIterator.java:39)
> at
> org.apache.phoenix.jdbc.PhoenixResultSet.next(PhoenixResultSet.java:739)
> at
> com.sensus.fma.IngestProgressMonitor.collectStats(IngestProgressMonitor.java:475)
> at
> com.sensus.fma.IngestProgressMonitor.main(IngestProgressMonitor.java:21)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.eclipse.jdt.internal.jarinjarloader.JarRsrcLoader.main(JarRsrcLoader.java:58)
> Caused by: java.util.concurrent.ExecutionException:
> org.apache.phoenix.exception.PhoenixIOException: Failed after attempts=36,
> exceptions:
> Mon Jun 29 14:08:49 EDT 2015, null, java.net.SocketTimeoutException:
> callTimeout=60000, callDuration=63169: row 'ABRDN_B7' on table'MY_TABLE' at
> region=MYTABLE,ABRDN_B7\x00\x00\x00\x00\x00,1435243688700.e4a6e5a437db9de9826a6d9d466a6a66.,
> hostname=MY_HOST,60020,1435572929067, seqNum=23
>
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> at java.util.concurrent.FutureTask.get(FutureTask.java:202)
> at
> org.apache.phoenix.iterate.ParallelIterators.getIterators(ParallelIterators.java:523)
> ... 13 more
> Caused by: org.apache.phoenix.exception.PhoenixIOException: Failed after
> attempts=36, exceptions:
> Mon Jun 29 14:08:49 EDT 2015, null, java.net.SocketTimeoutException:
> callTimeout=60000, callDuration=63169: row 'ABRDN_B7' on table 'MY_TABLE'
> at
> region=MY_TABLE,ABRDN_B7\x00\x00\x00\x00\x00,1435243688700.e4a6e5a437db9de9826a6d9d466a6a66.,
> hostname=DA-CLS-02-002.davis.sensus.lab,60020,1435572929067, seqNum=23
>
> at
> org.apache.phoenix.util.ServerUtil.parseServerException(ServerUtil.java:107)
> at
> org.apache.phoenix.iterate.TableResultIterator.<init>(TableResultIterator.java:57)
> at
> org.apache.phoenix.iterate.ParallelIterators$2.call(ParallelIterators.java:627)
> at
> org.apache.phoenix.iterate.ParallelIterators$2.call(ParallelIterators.java:622)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> -----Original Message-----
> From: James Taylor [mailto:jamestaylor@apache.org]
> Sent: Monday, June 29, 2015 11:51 AM
> To: user
> Subject: Re: How to count table rows from Java?
>
> Server side HBase properties such as hbase.rpc.timeout need to be set in
> your hbase-sites.xml. There's no API to set these.
>
> The phoenix.query.timeoutMs can be set using
>
> http://docs.oracle.com/javase/7/docs/api/java/sql/Statement.html#setQueryTimeout(int)
> prior to executing the query. The units in this case is seconds, not
> milliseconds.
>
> The phoenix.query.timeoutMs can also be set at connection-time in the
> properties like this:
>
> Properties props = new Properties();
> props.setProperty("phoenix.query.timeoutMs",
> Long.toString("600000"));
> Connection conn = DriverManager.getConnection(url, props);
>
> This would set the timeout of a query for any statement executing through
> the connection to 600000 milliseconds (10mins).
>
> You can also set the phoenix.query.timeoutMs in your client-side
> hbase-sites.xml and it'll be used for the query timeout for all connections.
>
> Thanks,
> James
>
>
> On Mon, Jun 29, 2015 at 2:44 AM, Riesland, Zack <Za...@sensus.com>
> wrote:
> > Thanks, James!
> >
> >
> >
> > Can you point me to some instructions or some syntax for setting those
> > timeout values in Java code?
> >
> >
> >
> > I’ve seen lots of information about setting them on the classpath when
> > launching different query clients. But how do I set a high timeout so
> > that I can do a large query via Java/JDBC?
> >
> >
> >
> > Thanks!
> >
> >
> >
> > From: James Taylor [mailto:jamestaylor@apache.org]
> > Sent: Friday, June 26, 2015 2:14 PM
> > To: user@phoenix.apache.org
> > Subject: Re: How to count table rows from Java?
> >
> >
> >
> > Zach,
> >
> > I wouldn't at all say that doing a count(*) is not recommended. It's
> > important to know that 1) this requires a full table scan and 2) this
> > is done by Phoenix asynchronously. You'll need to set the timeouts
> > high enough for this to complete. Phoenix will be much faster than
> > running a MR job, but the MR job runs asynchronously.
> >
> >
> >
> > To ensure your stats are up to date, run a major compaction on your
> > table as that updates the stats. Also, if you have wide rows, consider
> > using multiple column families so that the count(*) doesn't have to
> > traverse all of your data.
> >
> >
> >
> > Thanks,
> >
> > James
> >
> >
> > On Friday, June 26, 2015, Nick Dimiduk <nd...@gmail.com> wrote:
> >
> > RowCounter is s mapreduce program. After the program completes
> > execution of the job, it returns information about that job, including
> job counters.
> > RowCounter includes its counts in the job counters, so they're easily
> > accessed programmatically from the returned object. It's not a
> > ResultSet, but it should work none the less.
> >
> > On Friday, June 26, 2015, Riesland, Zack <Za...@sensus.com>
> wrote:
> >
> > I wrote a Java program that runs nightly and collects metrics about
> > our hive tables.
> >
> >
> >
> > I would like to include HBase tables in this as well.
> >
> >
> >
> > Since select count(*) is slow and not recommended on Phoenix, what are
> > my alternatives from Java?
> >
> >
> >
> > Is there a way to call org.apache.hadoop.hbase.mapreduce.RowCounter
> > from java and get results in some kind of result set?
> >
> >
> >
> > Thanks for any info!
> >
> >
> >
> >
>
RE: How to count table rows from Java?
Posted by "Riesland, Zack" <Za...@sensus.com>.
I REALLY hate to be a bother, but I just can't seem to get this timeout stuff to work for me.
It looks like setQueryTimeout() is not supported (I get a java.sql.SQLFeatureNotSupportedException). I'm guessing maybe we need a newer version of Phoenix ?
And I also tried the property approach: props.setProperty("phoenix.query.timeoutMs", "900000");
But my query (a simple select count(*) ) still fails after a few minutes with the errors below.
Anybody know what's going on here?
org.apache.phoenix.exception.PhoenixIOException: org.apache.phoenix.exception.PhoenixIOException: Failed after attempts=36, exceptions:
Mon Jun 29 14:08:49 EDT 2015, null, java.net.SocketTimeoutException: callTimeout=60000, callDuration=63169: row 'ABRDN_B7' on table 'FMA.MY_TABLE' at region=FMA.MY_TABLE,ABRDN_B7\x00\x00\x00\x00\x00,1435243688700.e4a6e5a437db9de9826a6d9d466a6a66., hostname=MY_HOST,60020,1435572929067, seqNum=23
at org.apache.phoenix.util.ServerUtil.parseServerException(ServerUtil.java:107)
at org.apache.phoenix.iterate.ParallelIterators.getIterators(ParallelIterators.java:527)
at org.apache.phoenix.iterate.ConcatResultIterator.getIterators(ConcatResultIterator.java:44)
at org.apache.phoenix.iterate.ConcatResultIterator.currentIterator(ConcatResultIterator.java:66)
at org.apache.phoenix.iterate.ConcatResultIterator.next(ConcatResultIterator.java:86)
at org.apache.phoenix.iterate.GroupedAggregatingResultIterator.next(GroupedAggregatingResultIterator.java:68)
at org.apache.phoenix.iterate.UngroupedAggregatingResultIterator.next(UngroupedAggregatingResultIterator.java:39)
at org.apache.phoenix.jdbc.PhoenixResultSet.next(PhoenixResultSet.java:739)
at com.sensus.fma.IngestProgressMonitor.collectStats(IngestProgressMonitor.java:475)
at com.sensus.fma.IngestProgressMonitor.main(IngestProgressMonitor.java:21)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.eclipse.jdt.internal.jarinjarloader.JarRsrcLoader.main(JarRsrcLoader.java:58)
Caused by: java.util.concurrent.ExecutionException: org.apache.phoenix.exception.PhoenixIOException: Failed after attempts=36, exceptions:
Mon Jun 29 14:08:49 EDT 2015, null, java.net.SocketTimeoutException: callTimeout=60000, callDuration=63169: row 'ABRDN_B7' on table'MY_TABLE' at region=MYTABLE,ABRDN_B7\x00\x00\x00\x00\x00,1435243688700.e4a6e5a437db9de9826a6d9d466a6a66., hostname=MY_HOST,60020,1435572929067, seqNum=23
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:202)
at org.apache.phoenix.iterate.ParallelIterators.getIterators(ParallelIterators.java:523)
... 13 more
Caused by: org.apache.phoenix.exception.PhoenixIOException: Failed after attempts=36, exceptions:
Mon Jun 29 14:08:49 EDT 2015, null, java.net.SocketTimeoutException: callTimeout=60000, callDuration=63169: row 'ABRDN_B7' on table 'MY_TABLE' at region=MY_TABLE,ABRDN_B7\x00\x00\x00\x00\x00,1435243688700.e4a6e5a437db9de9826a6d9d466a6a66., hostname=DA-CLS-02-002.davis.sensus.lab,60020,1435572929067, seqNum=23
at org.apache.phoenix.util.ServerUtil.parseServerException(ServerUtil.java:107)
at org.apache.phoenix.iterate.TableResultIterator.<init>(TableResultIterator.java:57)
at org.apache.phoenix.iterate.ParallelIterators$2.call(ParallelIterators.java:627)
at org.apache.phoenix.iterate.ParallelIterators$2.call(ParallelIterators.java:622)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
-----Original Message-----
From: James Taylor [mailto:jamestaylor@apache.org]
Sent: Monday, June 29, 2015 11:51 AM
To: user
Subject: Re: How to count table rows from Java?
Server side HBase properties such as hbase.rpc.timeout need to be set in your hbase-sites.xml. There's no API to set these.
The phoenix.query.timeoutMs can be set using
http://docs.oracle.com/javase/7/docs/api/java/sql/Statement.html#setQueryTimeout(int)
prior to executing the query. The units in this case is seconds, not milliseconds.
The phoenix.query.timeoutMs can also be set at connection-time in the properties like this:
Properties props = new Properties();
props.setProperty("phoenix.query.timeoutMs", Long.toString("600000"));
Connection conn = DriverManager.getConnection(url, props);
This would set the timeout of a query for any statement executing through the connection to 600000 milliseconds (10mins).
You can also set the phoenix.query.timeoutMs in your client-side hbase-sites.xml and it'll be used for the query timeout for all connections.
Thanks,
James
On Mon, Jun 29, 2015 at 2:44 AM, Riesland, Zack <Za...@sensus.com> wrote:
> Thanks, James!
>
>
>
> Can you point me to some instructions or some syntax for setting those
> timeout values in Java code?
>
>
>
> I’ve seen lots of information about setting them on the classpath when
> launching different query clients. But how do I set a high timeout so
> that I can do a large query via Java/JDBC?
>
>
>
> Thanks!
>
>
>
> From: James Taylor [mailto:jamestaylor@apache.org]
> Sent: Friday, June 26, 2015 2:14 PM
> To: user@phoenix.apache.org
> Subject: Re: How to count table rows from Java?
>
>
>
> Zach,
>
> I wouldn't at all say that doing a count(*) is not recommended. It's
> important to know that 1) this requires a full table scan and 2) this
> is done by Phoenix asynchronously. You'll need to set the timeouts
> high enough for this to complete. Phoenix will be much faster than
> running a MR job, but the MR job runs asynchronously.
>
>
>
> To ensure your stats are up to date, run a major compaction on your
> table as that updates the stats. Also, if you have wide rows, consider
> using multiple column families so that the count(*) doesn't have to
> traverse all of your data.
>
>
>
> Thanks,
>
> James
>
>
> On Friday, June 26, 2015, Nick Dimiduk <nd...@gmail.com> wrote:
>
> RowCounter is s mapreduce program. After the program completes
> execution of the job, it returns information about that job, including job counters.
> RowCounter includes its counts in the job counters, so they're easily
> accessed programmatically from the returned object. It's not a
> ResultSet, but it should work none the less.
>
> On Friday, June 26, 2015, Riesland, Zack <Za...@sensus.com> wrote:
>
> I wrote a Java program that runs nightly and collects metrics about
> our hive tables.
>
>
>
> I would like to include HBase tables in this as well.
>
>
>
> Since select count(*) is slow and not recommended on Phoenix, what are
> my alternatives from Java?
>
>
>
> Is there a way to call org.apache.hadoop.hbase.mapreduce.RowCounter
> from java and get results in some kind of result set?
>
>
>
> Thanks for any info!
>
>
>
>
Re: How to count table rows from Java?
Posted by James Taylor <ja...@apache.org>.
Server side HBase properties such as hbase.rpc.timeout need to be set
in your hbase-sites.xml. There's no API to set these.
The phoenix.query.timeoutMs can be set using
http://docs.oracle.com/javase/7/docs/api/java/sql/Statement.html#setQueryTimeout(int)
prior to executing the query. The units in this case is seconds, not
milliseconds.
The phoenix.query.timeoutMs can also be set at connection-time in the
properties like this:
Properties props = new Properties();
props.setProperty("phoenix.query.timeoutMs", Long.toString("600000"));
Connection conn = DriverManager.getConnection(url, props);
This would set the timeout of a query for any statement executing
through the connection to 600000 milliseconds (10mins).
You can also set the phoenix.query.timeoutMs in your client-side
hbase-sites.xml and it'll be used for the query timeout for all
connections.
Thanks,
James
On Mon, Jun 29, 2015 at 2:44 AM, Riesland, Zack
<Za...@sensus.com> wrote:
> Thanks, James!
>
>
>
> Can you point me to some instructions or some syntax for setting those
> timeout values in Java code?
>
>
>
> I’ve seen lots of information about setting them on the classpath when
> launching different query clients. But how do I set a high timeout so that I
> can do a large query via Java/JDBC?
>
>
>
> Thanks!
>
>
>
> From: James Taylor [mailto:jamestaylor@apache.org]
> Sent: Friday, June 26, 2015 2:14 PM
> To: user@phoenix.apache.org
> Subject: Re: How to count table rows from Java?
>
>
>
> Zach,
>
> I wouldn't at all say that doing a count(*) is not recommended. It's
> important to know that 1) this requires a full table scan and 2) this is
> done by Phoenix asynchronously. You'll need to set the timeouts high enough
> for this to complete. Phoenix will be much faster than running a MR job, but
> the MR job runs asynchronously.
>
>
>
> To ensure your stats are up to date, run a major compaction on your table as
> that updates the stats. Also, if you have wide rows, consider using multiple
> column families so that the count(*) doesn't have to traverse all of your
> data.
>
>
>
> Thanks,
>
> James
>
>
> On Friday, June 26, 2015, Nick Dimiduk <nd...@gmail.com> wrote:
>
> RowCounter is s mapreduce program. After the program completes execution of
> the job, it returns information about that job, including job counters.
> RowCounter includes its counts in the job counters, so they're easily
> accessed programmatically from the returned object. It's not a ResultSet,
> but it should work none the less.
>
> On Friday, June 26, 2015, Riesland, Zack <Za...@sensus.com> wrote:
>
> I wrote a Java program that runs nightly and collects metrics about our hive
> tables.
>
>
>
> I would like to include HBase tables in this as well.
>
>
>
> Since select count(*) is slow and not recommended on Phoenix, what are my
> alternatives from Java?
>
>
>
> Is there a way to call org.apache.hadoop.hbase.mapreduce.RowCounter from
> java and get results in some kind of result set?
>
>
>
> Thanks for any info!
>
>
>
>
RE: How to count table rows from Java?
Posted by "Riesland, Zack" <Za...@sensus.com>.
Thanks, James!
Can you point me to some instructions or some syntax for setting those timeout values in Java code?
I’ve seen lots of information about setting them on the classpath when launching different query clients. But how do I set a high timeout so that I can do a large query via Java/JDBC?
Thanks!
From: James Taylor [mailto:jamestaylor@apache.org]
Sent: Friday, June 26, 2015 2:14 PM
To: user@phoenix.apache.org
Subject: Re: How to count table rows from Java?
Zach,
I wouldn't at all say that doing a count(*) is not recommended. It's important to know that 1) this requires a full table scan and 2) this is done by Phoenix asynchronously. You'll need to set the timeouts high enough for this to complete. Phoenix will be much faster than running a MR job, but the MR job runs asynchronously.
To ensure your stats are up to date, run a major compaction on your table as that updates the stats. Also, if you have wide rows, consider using multiple column families so that the count(*) doesn't have to traverse all of your data.
Thanks,
James
On Friday, June 26, 2015, Nick Dimiduk <nd...@gmail.com>> wrote:
RowCounter is s mapreduce program. After the program completes execution of the job, it returns information about that job, including job counters. RowCounter includes its counts in the job counters, so they're easily accessed programmatically from the returned object. It's not a ResultSet, but it should work none the less.
On Friday, June 26, 2015, Riesland, Zack <Zack.Riesland@sensus.com<javascript:_e(%7B%7D,'cvml','Zack.Riesland@sensus.com');>> wrote:
I wrote a Java program that runs nightly and collects metrics about our hive tables.
I would like to include HBase tables in this as well.
Since select count(*) is slow and not recommended on Phoenix, what are my alternatives from Java?
Is there a way to call org.apache.hadoop.hbase.mapreduce.RowCounter from java and get results in some kind of result set?
Thanks for any info!
Re: How to count table rows from Java?
Posted by James Taylor <ja...@apache.org>.
Zach,
I wouldn't at all say that doing a count(*) is not recommended. It's
important to know that 1) this requires a full table scan and 2) this is
done by Phoenix asynchronously. You'll need to set the timeouts high enough
for this to complete. Phoenix will be much faster than running a MR job,
but the MR job runs asynchronously.
To ensure your stats are up to date, run a major compaction on your table
as that updates the stats. Also, if you have wide rows, consider using
multiple column families so that the count(*) doesn't have to traverse all
of your data.
Thanks,
James
On Friday, June 26, 2015, Nick Dimiduk <nd...@gmail.com> wrote:
> RowCounter is s mapreduce program. After the program completes execution
> of the job, it returns information about that job, including job counters.
> RowCounter includes its counts in the job counters, so they're easily
> accessed programmatically from the returned object. It's not a ResultSet,
> but it should work none the less.
>
> On Friday, June 26, 2015, Riesland, Zack <Zack.Riesland@sensus.com
> <javascript:_e(%7B%7D,'cvml','Zack.Riesland@sensus.com');>> wrote:
>
>> I wrote a Java program that runs nightly and collects metrics about our
>> hive tables.
>>
>>
>>
>> I would like to include HBase tables in this as well.
>>
>>
>>
>> Since select count(*) is slow and not recommended on Phoenix, what are my
>> alternatives from Java?
>>
>>
>>
>> Is there a way to call org.apache.hadoop.hbase.mapreduce.RowCounter from
>> java and get results in some kind of result set?
>>
>>
>>
>> Thanks for any info!
>>
>>
>>
>>
>>
>
Re: How to count table rows from Java?
Posted by Nick Dimiduk <nd...@gmail.com>.
RowCounter is s mapreduce program. After the program completes execution of
the job, it returns information about that job, including job counters.
RowCounter includes its counts in the job counters, so they're easily
accessed programmatically from the returned object. It's not a ResultSet,
but it should work none the less.
On Friday, June 26, 2015, Riesland, Zack <Za...@sensus.com> wrote:
> I wrote a Java program that runs nightly and collects metrics about our
> hive tables.
>
>
>
> I would like to include HBase tables in this as well.
>
>
>
> Since select count(*) is slow and not recommended on Phoenix, what are my
> alternatives from Java?
>
>
>
> Is there a way to call org.apache.hadoop.hbase.mapreduce.RowCounter from
> java and get results in some kind of result set?
>
>
>
> Thanks for any info!
>
>
>
>
>