You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Stephen Boesch <ja...@gmail.com> on 2014/11/14 01:57:51 UTC
Logging for HBase tests
How can logging be enabled/viewed when launching the hbase tests via
command line maven?
Given the following mvn command, I am able to set breakpoints within an IDE
(intellij):
mvn -Dmaven.surefire.debug="-Xdebug
-Xrunjdwp:transport=dt_socket,server=y,suspend=y,address=5005 -Xnoagent
-Djava.compiler=NONE" test
-Dtest=TestTleSnapshotInputFormat#testInitTableSnapshotMapperJobConfig
However, there is no stdout/stderr or log4j output anywhere.
The only output is the following (Note there an exception deep inside a
create table test but the details are not displayed!)
Listening for transport dt_socket at address: 5005
Running org.apache.hadoop.hbase.mapred.TestTableSnapshotInputFormat
2014-11-13 16:40:27.560 java[47177:2496198] Unable to load realm info from
SCDynamicStore
Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 178.141 sec
<<< FAILURE!
Note: the hbase-server/src/test/resources/log4j.properties has settings
that are friendly to printing plenty of logging messages. But why are the
messages not generated?
# Define some default values that can be overridden by system properties
hbase.root.logger=INFO,console
hbase.log.dir=.
hbase.log.file=hbase.log
..
# Logging Threshold
log4j.threshhold=ALL
..
log4j.logger.org.apache.hadoop.hbase=DEBUG
BTW I also have a related post also on StackOverflow - this one focuses
more on starting/debugging a test within the Intellij IDE:
http://stackoverflow.com/questions/26920330/logging-and-debugging-using-maven-within-intellij
Re: Logging for HBase tests
Posted by Nick Dimiduk <nd...@gmail.com>.
This stuff lands by default under
<module>/target/surefire-reports/testName.xml. The content is only there
when a test fails. I don't know how to make surefire always preserve the
run logs, so when I need to check them I just add an Assert.fail() to the
end of the test. This trick of overriding the logger looks handy for
running a single test, but won't all logs from all tests run land in the
same file, without indication as to the test origin?
On Saturday, November 15, 2014, Stephen Boesch <ja...@gmail.com> wrote:
> Thanks Sean the last option -Dhbase.root.logger=DEBUG,DRFA does the trick.
>
> 2014-11-14 14:58 GMT-08:00 Sean Busbey <busbey@cloudera.com <javascript:;>
> >:
>
> > You needn't specify the log4j configuration file, the one in
> > hbase-server/src/test/resources will get put in the test classpath for
> you.
> >
> > The default appender in that file is on stderr, which surefire will
> swallow
> > somewhere in the temp directory it creates for running things.
> >
> > Usually when I need log output from tests, I activate the daily rolling
> > file appender from the command line. ex:
> >
> > mvn
> > -Dtest=TestTableSnapshotInputFormat#testInitTableSnapshotMapperJobConfig
> > -Dhbase.root.logger=DEBUG,DRFA package
> >
> > this will place the log output in hbase-server/hbase.log
> >
> >
> >
> > On Fri, Nov 14, 2014 at 3:15 PM, Stephen Boesch <javadba@gmail.com
> <javascript:;>> wrote:
> >
> > > I still have not been able to locate the testing logs output. There
> are
> > no
> > > new directories/files created under the hbase distro directory at the
> > time
> > > the test is run, or under hbase-server. The hbase book does not
> mention
> > > this either. http://hbase.apache.org/0.94/book/hbase.tests.html
> > >
> > > The command line is the same as mentioned earlier:
> > >
> > > mvn -Dmaven.surefire.debug="-Xdebug
> > > -Xrunjdwp:transport=dt_socket,server=y,suspend=y,address=5005 -Xnoagent
> > > -Djava.compiler=NONE" test
> > >
> -Dtest=TestTableSnapshotInputFormat#testInitTableSnapshotMapperJobConfig
> > >
> > >
> >
> -Dlog4j.configuration=file:$HBASE_HOME/hbase-server/src/test/resources/log4j.properties
> > >
> > > 2014-11-13 17:18 GMT-08:00 Stephen Boesch <javadba@gmail.com
> <javascript:;>>:
> > >
> > > > Hi Ted,
> > > > as mentioned in that SOF post (a) breakpoints are not respected
> when
> > > > launched inside IJ (but they ARE respected when launching mvn command
> > > line)
> > > > and (b) logging is not working in both IJ and command line
> > > >
> > > > 2014-11-13 17:13 GMT-08:00 Ted Yu <yuzhihong@gmail.com
> <javascript:;>>:
> > > >
> > > > TestTableSnapshotInputFormat is a unit test which you can run from
> your
> > > IDE
> > > >> directly.
> > > >>
> > > >> BTW there was a typo in the command line below w.r.t. the test name.
> > > >>
> > > >> Cheers
> > > >>
> > > >> On Thu, Nov 13, 2014 at 4:57 PM, Stephen Boesch <javadba@gmail.com
> <javascript:;>>
> > > >> wrote:
> > > >>
> > > >> > How can logging be enabled/viewed when launching the hbase tests
> via
> > > >> > command line maven?
> > > >> >
> > > >> > Given the following mvn command, I am able to set breakpoints
> within
> > > an
> > > >> IDE
> > > >> > (intellij):
> > > >> >
> > > >> > mvn -Dmaven.surefire.debug="-Xdebug
> > > >> > -Xrunjdwp:transport=dt_socket,server=y,suspend=y,address=5005
> > > -Xnoagent
> > > >> > -Djava.compiler=NONE" test
> > > >> >
> > -Dtest=TestTleSnapshotInputFormat#testInitTableSnapshotMapperJobConfig
> > > >> >
> > > >> >
> > > >> > However, there is no stdout/stderr or log4j output anywhere.
> > > >> >
> > > >> > The only output is the following (Note there an exception deep
> > inside
> > > a
> > > >> > create table test but the details are not displayed!)
> > > >> >
> > > >> > Listening for transport dt_socket at address: 5005
> > > >> > Running
> org.apache.hadoop.hbase.mapred.TestTableSnapshotInputFormat
> > > >> > 2014-11-13 16:40:27.560 java[47177:2496198] Unable to load realm
> > info
> > > >> from
> > > >> > SCDynamicStore
> > > >> > Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed:
> > > 178.141
> > > >> sec
> > > >> > <<< FAILURE!
> > > >> >
> > > >> > Note: the hbase-server/src/test/resources/log4j.properties has
> > > settings
> > > >> > that are friendly to printing plenty of logging messages. But why
> > are
> > > >> the
> > > >> > messages not generated?
> > > >> >
> > > >> > # Define some default values that can be overridden by system
> > > properties
> > > >> > hbase.root.logger=INFO,console
> > > >> > hbase.log.dir=.
> > > >> > hbase.log.file=hbase.log
> > > >> > ..
> > > >> > # Logging Threshold
> > > >> > log4j.threshhold=ALL
> > > >> > ..
> > > >> > log4j.logger.org.apache.hadoop.hbase=DEBUG
> > > >> >
> > > >> >
> > > >> > BTW I also have a related post also on StackOverflow - this one
> > > focuses
> > > >> > more on starting/debugging a test within the Intellij IDE:
> > > >> >
> > > >> >
> > > >> >
> > > >>
> > >
> >
> http://stackoverflow.com/questions/26920330/logging-and-debugging-using-maven-within-intellij
> > > >> >
> > > >>
> > > >
> > > >
> > >
> >
> >
> >
> > --
> > Sean
> >
>
Re: Logging for HBase tests
Posted by Stephen Boesch <ja...@gmail.com>.
Thanks Sean the last option -Dhbase.root.logger=DEBUG,DRFA does the trick.
2014-11-14 14:58 GMT-08:00 Sean Busbey <bu...@cloudera.com>:
> You needn't specify the log4j configuration file, the one in
> hbase-server/src/test/resources will get put in the test classpath for you.
>
> The default appender in that file is on stderr, which surefire will swallow
> somewhere in the temp directory it creates for running things.
>
> Usually when I need log output from tests, I activate the daily rolling
> file appender from the command line. ex:
>
> mvn
> -Dtest=TestTableSnapshotInputFormat#testInitTableSnapshotMapperJobConfig
> -Dhbase.root.logger=DEBUG,DRFA package
>
> this will place the log output in hbase-server/hbase.log
>
>
>
> On Fri, Nov 14, 2014 at 3:15 PM, Stephen Boesch <ja...@gmail.com> wrote:
>
> > I still have not been able to locate the testing logs output. There are
> no
> > new directories/files created under the hbase distro directory at the
> time
> > the test is run, or under hbase-server. The hbase book does not mention
> > this either. http://hbase.apache.org/0.94/book/hbase.tests.html
> >
> > The command line is the same as mentioned earlier:
> >
> > mvn -Dmaven.surefire.debug="-Xdebug
> > -Xrunjdwp:transport=dt_socket,server=y,suspend=y,address=5005 -Xnoagent
> > -Djava.compiler=NONE" test
> > -Dtest=TestTableSnapshotInputFormat#testInitTableSnapshotMapperJobConfig
> >
> >
> -Dlog4j.configuration=file:$HBASE_HOME/hbase-server/src/test/resources/log4j.properties
> >
> > 2014-11-13 17:18 GMT-08:00 Stephen Boesch <ja...@gmail.com>:
> >
> > > Hi Ted,
> > > as mentioned in that SOF post (a) breakpoints are not respected when
> > > launched inside IJ (but they ARE respected when launching mvn command
> > line)
> > > and (b) logging is not working in both IJ and command line
> > >
> > > 2014-11-13 17:13 GMT-08:00 Ted Yu <yu...@gmail.com>:
> > >
> > > TestTableSnapshotInputFormat is a unit test which you can run from your
> > IDE
> > >> directly.
> > >>
> > >> BTW there was a typo in the command line below w.r.t. the test name.
> > >>
> > >> Cheers
> > >>
> > >> On Thu, Nov 13, 2014 at 4:57 PM, Stephen Boesch <ja...@gmail.com>
> > >> wrote:
> > >>
> > >> > How can logging be enabled/viewed when launching the hbase tests via
> > >> > command line maven?
> > >> >
> > >> > Given the following mvn command, I am able to set breakpoints within
> > an
> > >> IDE
> > >> > (intellij):
> > >> >
> > >> > mvn -Dmaven.surefire.debug="-Xdebug
> > >> > -Xrunjdwp:transport=dt_socket,server=y,suspend=y,address=5005
> > -Xnoagent
> > >> > -Djava.compiler=NONE" test
> > >> >
> -Dtest=TestTleSnapshotInputFormat#testInitTableSnapshotMapperJobConfig
> > >> >
> > >> >
> > >> > However, there is no stdout/stderr or log4j output anywhere.
> > >> >
> > >> > The only output is the following (Note there an exception deep
> inside
> > a
> > >> > create table test but the details are not displayed!)
> > >> >
> > >> > Listening for transport dt_socket at address: 5005
> > >> > Running org.apache.hadoop.hbase.mapred.TestTableSnapshotInputFormat
> > >> > 2014-11-13 16:40:27.560 java[47177:2496198] Unable to load realm
> info
> > >> from
> > >> > SCDynamicStore
> > >> > Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed:
> > 178.141
> > >> sec
> > >> > <<< FAILURE!
> > >> >
> > >> > Note: the hbase-server/src/test/resources/log4j.properties has
> > settings
> > >> > that are friendly to printing plenty of logging messages. But why
> are
> > >> the
> > >> > messages not generated?
> > >> >
> > >> > # Define some default values that can be overridden by system
> > properties
> > >> > hbase.root.logger=INFO,console
> > >> > hbase.log.dir=.
> > >> > hbase.log.file=hbase.log
> > >> > ..
> > >> > # Logging Threshold
> > >> > log4j.threshhold=ALL
> > >> > ..
> > >> > log4j.logger.org.apache.hadoop.hbase=DEBUG
> > >> >
> > >> >
> > >> > BTW I also have a related post also on StackOverflow - this one
> > focuses
> > >> > more on starting/debugging a test within the Intellij IDE:
> > >> >
> > >> >
> > >> >
> > >>
> >
> http://stackoverflow.com/questions/26920330/logging-and-debugging-using-maven-within-intellij
> > >> >
> > >>
> > >
> > >
> >
>
>
>
> --
> Sean
>
Re: Logging for HBase tests
Posted by Sean Busbey <bu...@cloudera.com>.
You needn't specify the log4j configuration file, the one in
hbase-server/src/test/resources will get put in the test classpath for you.
The default appender in that file is on stderr, which surefire will swallow
somewhere in the temp directory it creates for running things.
Usually when I need log output from tests, I activate the daily rolling
file appender from the command line. ex:
mvn -Dtest=TestTableSnapshotInputFormat#testInitTableSnapshotMapperJobConfig
-Dhbase.root.logger=DEBUG,DRFA package
this will place the log output in hbase-server/hbase.log
On Fri, Nov 14, 2014 at 3:15 PM, Stephen Boesch <ja...@gmail.com> wrote:
> I still have not been able to locate the testing logs output. There are no
> new directories/files created under the hbase distro directory at the time
> the test is run, or under hbase-server. The hbase book does not mention
> this either. http://hbase.apache.org/0.94/book/hbase.tests.html
>
> The command line is the same as mentioned earlier:
>
> mvn -Dmaven.surefire.debug="-Xdebug
> -Xrunjdwp:transport=dt_socket,server=y,suspend=y,address=5005 -Xnoagent
> -Djava.compiler=NONE" test
> -Dtest=TestTableSnapshotInputFormat#testInitTableSnapshotMapperJobConfig
>
> -Dlog4j.configuration=file:$HBASE_HOME/hbase-server/src/test/resources/log4j.properties
>
> 2014-11-13 17:18 GMT-08:00 Stephen Boesch <ja...@gmail.com>:
>
> > Hi Ted,
> > as mentioned in that SOF post (a) breakpoints are not respected when
> > launched inside IJ (but they ARE respected when launching mvn command
> line)
> > and (b) logging is not working in both IJ and command line
> >
> > 2014-11-13 17:13 GMT-08:00 Ted Yu <yu...@gmail.com>:
> >
> > TestTableSnapshotInputFormat is a unit test which you can run from your
> IDE
> >> directly.
> >>
> >> BTW there was a typo in the command line below w.r.t. the test name.
> >>
> >> Cheers
> >>
> >> On Thu, Nov 13, 2014 at 4:57 PM, Stephen Boesch <ja...@gmail.com>
> >> wrote:
> >>
> >> > How can logging be enabled/viewed when launching the hbase tests via
> >> > command line maven?
> >> >
> >> > Given the following mvn command, I am able to set breakpoints within
> an
> >> IDE
> >> > (intellij):
> >> >
> >> > mvn -Dmaven.surefire.debug="-Xdebug
> >> > -Xrunjdwp:transport=dt_socket,server=y,suspend=y,address=5005
> -Xnoagent
> >> > -Djava.compiler=NONE" test
> >> > -Dtest=TestTleSnapshotInputFormat#testInitTableSnapshotMapperJobConfig
> >> >
> >> >
> >> > However, there is no stdout/stderr or log4j output anywhere.
> >> >
> >> > The only output is the following (Note there an exception deep inside
> a
> >> > create table test but the details are not displayed!)
> >> >
> >> > Listening for transport dt_socket at address: 5005
> >> > Running org.apache.hadoop.hbase.mapred.TestTableSnapshotInputFormat
> >> > 2014-11-13 16:40:27.560 java[47177:2496198] Unable to load realm info
> >> from
> >> > SCDynamicStore
> >> > Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed:
> 178.141
> >> sec
> >> > <<< FAILURE!
> >> >
> >> > Note: the hbase-server/src/test/resources/log4j.properties has
> settings
> >> > that are friendly to printing plenty of logging messages. But why are
> >> the
> >> > messages not generated?
> >> >
> >> > # Define some default values that can be overridden by system
> properties
> >> > hbase.root.logger=INFO,console
> >> > hbase.log.dir=.
> >> > hbase.log.file=hbase.log
> >> > ..
> >> > # Logging Threshold
> >> > log4j.threshhold=ALL
> >> > ..
> >> > log4j.logger.org.apache.hadoop.hbase=DEBUG
> >> >
> >> >
> >> > BTW I also have a related post also on StackOverflow - this one
> focuses
> >> > more on starting/debugging a test within the Intellij IDE:
> >> >
> >> >
> >> >
> >>
> http://stackoverflow.com/questions/26920330/logging-and-debugging-using-maven-within-intellij
> >> >
> >>
> >
> >
>
--
Sean
Re: Logging for HBase tests
Posted by Stephen Boesch <ja...@gmail.com>.
I still have not been able to locate the testing logs output. There are no
new directories/files created under the hbase distro directory at the time
the test is run, or under hbase-server. The hbase book does not mention
this either. http://hbase.apache.org/0.94/book/hbase.tests.html
The command line is the same as mentioned earlier:
mvn -Dmaven.surefire.debug="-Xdebug
-Xrunjdwp:transport=dt_socket,server=y,suspend=y,address=5005 -Xnoagent
-Djava.compiler=NONE" test
-Dtest=TestTableSnapshotInputFormat#testInitTableSnapshotMapperJobConfig
-Dlog4j.configuration=file:$HBASE_HOME/hbase-server/src/test/resources/log4j.properties
2014-11-13 17:18 GMT-08:00 Stephen Boesch <ja...@gmail.com>:
> Hi Ted,
> as mentioned in that SOF post (a) breakpoints are not respected when
> launched inside IJ (but they ARE respected when launching mvn command line)
> and (b) logging is not working in both IJ and command line
>
> 2014-11-13 17:13 GMT-08:00 Ted Yu <yu...@gmail.com>:
>
> TestTableSnapshotInputFormat is a unit test which you can run from your IDE
>> directly.
>>
>> BTW there was a typo in the command line below w.r.t. the test name.
>>
>> Cheers
>>
>> On Thu, Nov 13, 2014 at 4:57 PM, Stephen Boesch <ja...@gmail.com>
>> wrote:
>>
>> > How can logging be enabled/viewed when launching the hbase tests via
>> > command line maven?
>> >
>> > Given the following mvn command, I am able to set breakpoints within an
>> IDE
>> > (intellij):
>> >
>> > mvn -Dmaven.surefire.debug="-Xdebug
>> > -Xrunjdwp:transport=dt_socket,server=y,suspend=y,address=5005 -Xnoagent
>> > -Djava.compiler=NONE" test
>> > -Dtest=TestTleSnapshotInputFormat#testInitTableSnapshotMapperJobConfig
>> >
>> >
>> > However, there is no stdout/stderr or log4j output anywhere.
>> >
>> > The only output is the following (Note there an exception deep inside a
>> > create table test but the details are not displayed!)
>> >
>> > Listening for transport dt_socket at address: 5005
>> > Running org.apache.hadoop.hbase.mapred.TestTableSnapshotInputFormat
>> > 2014-11-13 16:40:27.560 java[47177:2496198] Unable to load realm info
>> from
>> > SCDynamicStore
>> > Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 178.141
>> sec
>> > <<< FAILURE!
>> >
>> > Note: the hbase-server/src/test/resources/log4j.properties has settings
>> > that are friendly to printing plenty of logging messages. But why are
>> the
>> > messages not generated?
>> >
>> > # Define some default values that can be overridden by system properties
>> > hbase.root.logger=INFO,console
>> > hbase.log.dir=.
>> > hbase.log.file=hbase.log
>> > ..
>> > # Logging Threshold
>> > log4j.threshhold=ALL
>> > ..
>> > log4j.logger.org.apache.hadoop.hbase=DEBUG
>> >
>> >
>> > BTW I also have a related post also on StackOverflow - this one focuses
>> > more on starting/debugging a test within the Intellij IDE:
>> >
>> >
>> >
>> http://stackoverflow.com/questions/26920330/logging-and-debugging-using-maven-within-intellij
>> >
>>
>
>
Re: Logging for HBase tests
Posted by Stephen Boesch <ja...@gmail.com>.
Hi Ted,
as mentioned in that SOF post (a) breakpoints are not respected when
launched inside IJ (but they ARE respected when launching mvn command line)
and (b) logging is not working in both IJ and command line
2014-11-13 17:13 GMT-08:00 Ted Yu <yu...@gmail.com>:
> TestTableSnapshotInputFormat is a unit test which you can run from your IDE
> directly.
>
> BTW there was a typo in the command line below w.r.t. the test name.
>
> Cheers
>
> On Thu, Nov 13, 2014 at 4:57 PM, Stephen Boesch <ja...@gmail.com> wrote:
>
> > How can logging be enabled/viewed when launching the hbase tests via
> > command line maven?
> >
> > Given the following mvn command, I am able to set breakpoints within an
> IDE
> > (intellij):
> >
> > mvn -Dmaven.surefire.debug="-Xdebug
> > -Xrunjdwp:transport=dt_socket,server=y,suspend=y,address=5005 -Xnoagent
> > -Djava.compiler=NONE" test
> > -Dtest=TestTleSnapshotInputFormat#testInitTableSnapshotMapperJobConfig
> >
> >
> > However, there is no stdout/stderr or log4j output anywhere.
> >
> > The only output is the following (Note there an exception deep inside a
> > create table test but the details are not displayed!)
> >
> > Listening for transport dt_socket at address: 5005
> > Running org.apache.hadoop.hbase.mapred.TestTableSnapshotInputFormat
> > 2014-11-13 16:40:27.560 java[47177:2496198] Unable to load realm info
> from
> > SCDynamicStore
> > Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 178.141
> sec
> > <<< FAILURE!
> >
> > Note: the hbase-server/src/test/resources/log4j.properties has settings
> > that are friendly to printing plenty of logging messages. But why are the
> > messages not generated?
> >
> > # Define some default values that can be overridden by system properties
> > hbase.root.logger=INFO,console
> > hbase.log.dir=.
> > hbase.log.file=hbase.log
> > ..
> > # Logging Threshold
> > log4j.threshhold=ALL
> > ..
> > log4j.logger.org.apache.hadoop.hbase=DEBUG
> >
> >
> > BTW I also have a related post also on StackOverflow - this one focuses
> > more on starting/debugging a test within the Intellij IDE:
> >
> >
> >
> http://stackoverflow.com/questions/26920330/logging-and-debugging-using-maven-within-intellij
> >
>
Re: Logging for HBase tests
Posted by Ted Yu <yu...@gmail.com>.
TestTableSnapshotInputFormat is a unit test which you can run from your IDE
directly.
BTW there was a typo in the command line below w.r.t. the test name.
Cheers
On Thu, Nov 13, 2014 at 4:57 PM, Stephen Boesch <ja...@gmail.com> wrote:
> How can logging be enabled/viewed when launching the hbase tests via
> command line maven?
>
> Given the following mvn command, I am able to set breakpoints within an IDE
> (intellij):
>
> mvn -Dmaven.surefire.debug="-Xdebug
> -Xrunjdwp:transport=dt_socket,server=y,suspend=y,address=5005 -Xnoagent
> -Djava.compiler=NONE" test
> -Dtest=TestTleSnapshotInputFormat#testInitTableSnapshotMapperJobConfig
>
>
> However, there is no stdout/stderr or log4j output anywhere.
>
> The only output is the following (Note there an exception deep inside a
> create table test but the details are not displayed!)
>
> Listening for transport dt_socket at address: 5005
> Running org.apache.hadoop.hbase.mapred.TestTableSnapshotInputFormat
> 2014-11-13 16:40:27.560 java[47177:2496198] Unable to load realm info from
> SCDynamicStore
> Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 178.141 sec
> <<< FAILURE!
>
> Note: the hbase-server/src/test/resources/log4j.properties has settings
> that are friendly to printing plenty of logging messages. But why are the
> messages not generated?
>
> # Define some default values that can be overridden by system properties
> hbase.root.logger=INFO,console
> hbase.log.dir=.
> hbase.log.file=hbase.log
> ..
> # Logging Threshold
> log4j.threshhold=ALL
> ..
> log4j.logger.org.apache.hadoop.hbase=DEBUG
>
>
> BTW I also have a related post also on StackOverflow - this one focuses
> more on starting/debugging a test within the Intellij IDE:
>
>
> http://stackoverflow.com/questions/26920330/logging-and-debugging-using-maven-within-intellij
>