Phoenix query tracing issue

[Stepan Migunov <St...@firstlinesoftware.com>]

We were trying to enable Phoenix query tracing and ran into problem. Traces
are not being written to SYSTEM.TRACING_STATS table.



We're using HBase 1.3 and Phoenix 4.13.1.

We were trying to do this according to this official manual:

https://phoenix.apache.org/tracing.html

.

We've set up hadoop-metrics2-phoenix.properties and
hadoop-metrics2-hbase.properties as is from the manual. These files are
located at $PHOENIX_HOME/bin/config/hadoop-metrics2-phoenix.properties and
$HBASE_HOME/conf/hadoop-metrics2-hbase.properties . hbase-site.xml at
$PHOENIX_HOME/bin contains phoenix.trace.frequency property which is set to
"always". Despite that when we perform some queries from sqlline.py, traces
are not being written to SYSTEM.TRACING_STATS. This table is not even
created.



During investigation of this problem we dived into Phoenix sources. What
confuses us is that we can't find
org.apache.phoenix.trace.PhoenixTableMetricsWriter class in sources of
Phoenix 4.13.1. Is that correct name of metrics writer class? And what
confuses us even more is that turned out that we're specifying unexistent
class to HBase, but despite that HBase Master starts and there are no
errors in logs. I would expect NoClassDefFoundError or
ClassNotFoundException or something like that. Looks like HBase is not even
trying to load or instantiate this class.



In 4.13.1 org.apache.phoenix.trace.TableWriter class looks like metrics
writer class, but after specifying it in phoenix.sink.tracing.writer-class
and hbase.sink.tracing.writer-class properties in
hadoop-metrics2-phoenix.properties and hadoop-metrics2-hbase.properties
there is still no luck: SYSTEM.TRACING_STATS table is not created after we
perform some queries in sqlline.py .



We've also tried to trace queries performed by our Spark application that
uses spark phoenix plugin, but ran into same problem: SYSTEM.TRACING_STATS
table is not even initialized. phoenix.trace.frequency = always is
specified in HBase config object in our Spark application.



Also we've tried to trace queries performed by IntelliJ IDEA via JDBC
connection, phoenix.trace.frequency property was set as custom JDBC
connection property.



Can you please advice what we're doing wrong?



Thanks,

Stepan.