You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hive.apache.org by Pradeep Kamath <pr...@yahoo-inc.com> on 2010/06/10 00:09:37 UTC
Logs when running metastore
Hi,
Are there any logs created when running the metastore using "bin/hive
-service metastore" ? From what I can tell this command generates output
on the screen and runs in the foreground. Would the right way to run
this as a server be "bin/hive -service metstatore > metastore.log 2>&1
&" ? Is there any other logfile which is written to, while the metastore
server is running?
Thanks,
Pradeep
RE: Logs when running metastore
Posted by Paul Yang <py...@facebook.com>.
When calling "bin/hive -service metastore", all the error messages are printed to stdout/err. The command you proposed is a workable approach (but maybe replace > with >> so that it appends, instead of overwriting). Ideally, you would use log4j to configure logging to a file e.g. with an argument like "-Dlog4j.configuration=configuration_file" when starting the server. But the way that the startup scripts work now, it looks like you might need to make modifications to bin/ext/metastore.sh to make that happen. The advantage to using log4j is that it can be configured to do daily roll-over logs, so they never get too big.
There's not really a good way to identify the metastore process once it's started - the command you ran is similar to what I use too. One way to identify the process might be to record to a file the PID of the metastore when you start it up -
nohup bin/hive -service metstatore > metastore.log 2>&1 &
echo $! > metastore_pid
From: Pradeep Kamath [mailto:pradeepk@yahoo-inc.com]
Sent: Wednesday, June 09, 2010 4:11 PM
To: hive-user@hadoop.apache.org
Subject: RE: Logs when running metastore
Also a related question is how do I stop the server if have started the server with "nohup bin/hive -service metstatore > metastore.log 2>&1 &" command ? When I run "ps -aef |grep hive", I get :
pradeepk 5463 14236 0 16:00 pts/6 00:00:00 grep hive
pradeepk 27803 14226 0 13:47 pts/5 00:00:04 /usr/java/jdk1.6.0_01/bin/java -Xmx256m -Dhadoop.log.dir=/homes/pradeepk/hadoopcluster/hadoop/bin/../logs -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/homes/pradeepk/hadoopcluster/hadoop/bin/.. -Dhadoop.id.str= -Dhadoop.root.logger=INFO,console -Djava.library.path=/homes/pradeepk/hadoopcluster/hadoop/bin/../lib/native/Linux-i386-32 -Dhadoop.policy.file=hadoop-policy.xml -classpath /homes/pradeepk/hadoopcluster/hadoop/conf:/usr/java/jdk1.6.0_01/lib/tools.jar:/homes/pradeepk/hadoopcluster/hadoop/bin/..:/homes/pradeepk/hadoopcluster/hadoop/bin/../hadoop-0.20.1-core.jar:/homes/pradeepk/hadoopcluster/hadoop/bin/../lib/commons-cli-1.2.jar:/homes/pradeepk/hadoopcluster/hadoop/bin/../lib/commons-codec-1.3.jar:/homes/pradeepk/hadoopcluster/hadoop/bin/../lib/commons-el-1.0.jar:/homes/pradeepk/hadoopcluster/hadoop/bin/../lib/commons-httpclient-3.0.1.jar:/homes/pradeepk/hadoopcluster/hadoop/bin/../lib/commons-logging-1.0.4.jar:/homes/pradeepk/hadoopcluster/hadoop/bin/../lib/commons-logging-api-1.0.4.jar:/homes/pradeepk/hadoopcluster/hadoop/bin/../lib/commons-net-1.4.1.jar:/homes/pradeepk/hadoopcluster/hadoop/bin/../lib/core-3.1.1.jar:/homes/pradeepk/hadoopcluster/hadoop/bin/../lib/hsqldb-1.8.0.10.jar:/homes/pradeepk/hadoopcluster/hadoop/bin/../lib/jasper-compiler-5.5.12.jar:/homes/pradeepk/hadoopcluster/hadoop/bin/../lib/jasper-runtime-5.5.12.jar:/homes/pradeepk/hadoopcluster/hadoop/bin/../lib/jets3t-0.6.1.jar:/homes/pradeepk/hadoopcluster/hadoop/bin/../lib/jetty-6.1.14.jar:/homes/pradeepk/hadoopcluster/hadoop/bin/../lib/jetty-util-6.1.14.jar:/homes/pradeepk/hadoopcluster/hadoop/bin/../lib/junit-3.8.1.jar:/homes/pradeepk/hadoopcluster/hadoop/bin/../lib/kfs-0.2.2.jar:/homes/pradeepk/hadoopcluster/hadoop/bin/../lib/log4j-1.2.15.jar:/homes/pradeepk/hadoopcluster/hadoop/bin/../lib/oro-2.0.8.jar:/homes/pradeepk/hadoopcluster/hadoop/bin/../lib/servlet-api-2.5-6.1.14.jar:/homes/pradeepk/hadoopcluster/hadoop/bin/../lib/slf4j-api-1.4.3.jar:/homes/pradeepk/hadoopcluster/hadoop/bin/../lib/slf4j-log4j12-1.4.3.jar:/homes/pradeepk/hadoopcluster/hadoop/bin/../lib/xmlenc-0.52.jar:/homes/pradeepk/hadoopcluster/hadoop/bin/../lib/jsp-2.1/jsp-2.1.jar:/homes/pradeepk/hadoopcluster/hadoop/bin/../lib/jsp-2.1/jsp-api-2.1.jar:/homes/pradeepk/dev/hive/build/dist/conf:/homes/pradeepk/dev/hive/build/dist/lib/antlr-runtime-3.0.1.jar:/homes/pradeepk/dev/hive/build/dist/lib/asm-3.1.jar:/homes/pradeepk/dev/hive/build/dist/lib/commons-cli-2.0-SNAPSHOT.jar:/homes/pradeepk/dev/hive/build/dist/lib/commons-codec-1.3.jar:/homes/pradeepk/dev/hive/build/dist/lib/commons-collections-3.2.1.jar:/homes/pradeepk/dev/hive/build/dist/lib/commons-dbcp-1.2.2.jar:/homes/pradeepk/dev/hive/build/dist/lib/commons-lang-2.4.jar:/homes/pradeepk/dev/hive/build/dist/lib/commons-logging-1.0.4.jar:/homes/pradeepk/dev/hive/build/dist/lib/commons-logging-api-1.0.4.jar:/homes/pradeepk/dev/hive/build/dist/lib/commons-pool-1.2.jar:/homes/pradeepk/dev/hive/build/dist/lib/datanucleus-connectionpool-1.0.2.jar:/homes/pradeepk/dev/hive/build/dist/lib/datanucleus-core-1.1.2.jar:/homes/pradeepk/dev/hive/build/dist/lib/datanucleus-enhancer-1.1.2.jar:/homes/pradeepk/dev/hive/build/dist/lib/datanucleus-rdbms-1.1.2.jar:/homes/pradeepk/dev/hive/build/dist/lib/derby.jar:/homes/pradeepk/dev/hive/build/dist/lib/hbase-0.20.3.jar:/homes/pradeepk/dev/hive/build/dist/lib/hbase-0.20.3-test.jar:/homes/pradeepk/dev/hive/build/dist/lib/hive-anttasks-0.6.0.jar:/homes/pradeepk/dev/hive/build/dist/lib/hive-cli-0.6.0.jar:/homes/pradeepk/dev/hive/build/dist/lib/hive-common-0.6.0.jar:/homes/pradeepk/dev/hive/build/dist/lib/hive_contrib.jar:/homes/pradeepk/dev/hive/build/dist/lib/hive-exec-0.6.0.jar:/homes/pradeepk/dev/hive/build/dist/lib/hive_hbase-handler.jar:/homes/pradeepk/dev/hive/build/dist/lib/hive-hwi-0.6.0.jar:/homes/pradeepk/dev/hive/build/dist/lib/hive-jdbc-0.6.0.jar:/homes/pradeepk/dev/hive/build/dist/lib/hive-metastore-0.6.0.jar:/homes/pradeepk/dev/hive/build/dist/lib/hive-serde-0.6.0.jar:/homes/pradeepk/dev/hive/build/dist/lib/hive-service-0.6.0.jar:/homes/pradeepk/dev/hive/build/dist/lib/hive-shims-0.6.0.jar
Is there a recommended way to identify the metastore server process?
Thanks,
Pradeep
________________________________
From: Pradeep Kamath [mailto:pradeepk@yahoo-inc.com]
Sent: Wednesday, June 09, 2010 3:10 PM
To: hive-user@hadoop.apache.org
Subject: Logs when running metastore
Hi,
Are there any logs created when running the metastore using "bin/hive -service metastore" ? From what I can tell this command generates output on the screen and runs in the foreground. Would the right way to run this as a server be "bin/hive -service metstatore > metastore.log 2>&1 &" ? Is there any other logfile which is written to, while the metastore server is running?
Thanks,
Pradeep
Re: Logs when running metastore
Posted by Edward Capriolo <ed...@gmail.com>.
On Wed, Jun 9, 2010 at 7:10 PM, Pradeep Kamath <pr...@yahoo-inc.com>wrote:
> Also a related question is how do I stop the server if have started the
> server with “nohup bin/hive –service metstatore > metastore.log 2>&1 &”
> command ? When I run “ps -aef |grep hive”, I get :
>
>
>
> pradeepk 5463 14236 0 16:00 pts/6 00:00:00 grep hive
>
> pradeepk 27803 14226 0 13:47 pts/5 00:00:04
> /usr/java/jdk1.6.0_01/bin/java -Xmx256m
> -Dhadoop.log.dir=/homes/pradeepk/hadoopcluster/hadoop/bin/../logs
> -Dhadoop.log.file=hadoop.log
> -Dhadoop.home.dir=/homes/pradeepk/hadoopcluster/hadoop/bin/..
> -Dhadoop.id.str= -Dhadoop.root.logger=INFO,console
> -Djava.library.path=/homes/pradeepk/hadoopcluster/hadoop/bin/../lib/native/Linux-i386-32
> -Dhadoop.policy.file=hadoop-policy.xml -classpath
> /homes/pradeepk/hadoopcluster/hadoop/conf:/usr/java/jdk1.6.0_01/lib/tools.jar:/homes/pradeepk/hadoopcluster/hadoop/bin/..:/homes/pradeepk/hadoopcluster/hadoop/bin/../hadoop-0.20.1-core.jar:/homes/pradeepk/hadoopcluster/hadoop/bin/../lib/commons-cli-1.2.jar:/homes/pradeepk/hadoopcluster/hadoop/bin/../lib/commons-codec-1.3.jar:/homes/pradeepk/hadoopcluster/hadoop/bin/../lib/commons-el-1.0.jar:/homes/pradeepk/hadoopcluster/hadoop/bin/../lib/commons-httpclient-3.0.1.jar:/homes/pradeepk/hadoopcluster/hadoop/bin/../lib/commons-logging-1.0.4.jar:/homes/pradeepk/hadoopcluster/hadoop/bin/../lib/commons-logging-api-1.0.4.jar:/homes/pradeepk/hadoopcluster/hadoop/bin/../lib/commons-net-1.4.1.jar:/homes/pradeepk/hadoopcluster/hadoop/bin/../lib/core-3.1.1.jar:/homes/pradeepk/hadoopcluster/hadoop/bin/../lib/hsqldb-1.8.0.10.jar:/homes/pradeepk/hadoopcluster/hadoop/bin/../lib/jasper-compiler-5.5.12.jar:/homes/pradeepk/hadoopcluster/hadoop/bin/../lib/jasper-runtime-5.5.12.jar:/homes/pradeepk/hadoopcluster/hadoop/bin/../lib/jets3t-0.6.1.jar:/homes/pradeepk/hadoopcluster/hadoop/bin/../lib/jetty-6.1.14.jar:/homes/pradeepk/hadoopcluster/hadoop/bin/../lib/jetty-util-6.1.14.jar:/homes/pradeepk/hadoopcluster/hadoop/bin/../lib/junit-3.8.1.jar:/homes/pradeepk/hadoopcluster/hadoop/bin/../lib/kfs-0.2.2.jar:/homes/pradeepk/hadoopcluster/hadoop/bin/../lib/log4j-1.2.15.jar:/homes/pradeepk/hadoopcluster/hadoop/bin/../lib/oro-2.0.8.jar:/homes/pradeepk/hadoopcluster/hadoop/bin/../lib/servlet-api-2.5-6.1.14.jar:/homes/pradeepk/hadoopcluster/hadoop/bin/../lib/slf4j-api-1.4.3.jar:/homes/pradeepk/hadoopcluster/hadoop/bin/../lib/slf4j-log4j12-1.4.3.jar:/homes/pradeepk/hadoopcluster/hadoop/bin/../lib/xmlenc-0.52.jar:/homes/pradeepk/hadoopcluster/hadoop/bin/../lib/jsp-2.1/jsp-2.1.jar:/homes/pradeepk/hadoopcluster/hadoop/bin/../lib/jsp-2.1/jsp-api-2.1.jar:/homes/pradeepk/dev/hive/build/dist/conf:/homes/pradeepk/dev/hive/build/dist/lib/antlr-runtime-3.0.1.jar:/homes/pradeepk/dev/hive/build/dist/lib/asm-3.1.jar:/homes/pradeepk/dev/hive/build/dist/lib/commons-cli-2.0-SNAPSHOT.jar:/homes/pradeepk/dev/hive/build/dist/lib/commons-codec-1.3.jar:/homes/pradeepk/dev/hive/build/dist/lib/commons-collections-3.2.1.jar:/homes/pradeepk/dev/hive/build/dist/lib/commons-dbcp-1.2.2.jar:/homes/pradeepk/dev/hive/build/dist/lib/commons-lang-2.4.jar:/homes/pradeepk/dev/hive/build/dist/lib/commons-logging-1.0.4.jar:/homes/pradeepk/dev/hive/build/dist/lib/commons-logging-api-1.0.4.jar:/homes/pradeepk/dev/hive/build/dist/lib/commons-pool-1.2.jar:/homes/pradeepk/dev/hive/build/dist/lib/datanucleus-connectionpool-1.0.2.jar:/homes/pradeepk/dev/hive/build/dist/lib/datanucleus-core-1.1.2.jar:/homes/pradeepk/dev/hive/build/dist/lib/datanucleus-enhancer-1.1.2.jar:/homes/pradeepk/dev/hive/build/dist/lib/datanucleus-rdbms-1.1.2.jar:/homes/pradeepk/dev/hive/build/dist/lib/derby.jar:/homes/pradeepk/dev/hive/build/dist/lib/hbase-0.20.3.jar:/homes/pradeepk/dev/hive/build/dist/lib/hbase-0.20.3-test.jar:/homes/pradeepk/dev/hive/build/dist/lib/hive-anttasks-0.6.0.jar:/homes/pradeepk/dev/hive/build/dist/lib/hive-cli-0.6.0.jar:/homes/pradeepk/dev/hive/build/dist/lib/hive-common-0.6.0.jar:/homes/pradeepk/dev/hive/build/dist/lib/hive_contrib.jar:/homes/pradeepk/dev/hive/build/dist/lib/hive-exec-0.6.0.jar:/homes/pradeepk/dev/hive/build/dist/lib/hive_hbase-handler.jar:/homes/pradeepk/dev/hive/build/dist/lib/hive-hwi-0.6.0.jar:/homes/pradeepk/dev/hive/build/dist/lib/hive-jdbc-0.6.0.jar:/homes/pradeepk/dev/hive/build/dist/lib/hive-metastore-0.6.0.jar:/homes/pradeepk/dev/hive/build/dist/lib/hive-serde-0.6.0.jar:/homes/pradeepk/dev/hive/build/dist/lib/hive-service-0.6.0.jar:/homes/pradeepk/dev/hive/build/dist/lib/hive-shims-0.6.0.jar
>
>
>
> Is there a recommended way to identify the metastore server process?
>
>
>
> Thanks,
>
> Pradeep
> ------------------------------
>
> *From:* Pradeep Kamath [mailto:pradeepk@yahoo-inc.com]
> *Sent:* Wednesday, June 09, 2010 3:10 PM
> *To:* hive-user@hadoop.apache.org
> *Subject:* Logs when running metastore
>
>
>
> Hi,
>
> Are there any logs created when running the metastore using “bin/hive
> –service metastore” ? From what I can tell this command generates output on
> the screen and runs in the foreground. Would the right way to run this as a
> server be “bin/hive –service metstatore > metastore.log 2>&1 &” ? Is there
> any other logfile which is written to, while the metastore server is
> running?
>
>
>
> Thanks,
>
> Pradeep
>
>
>
>
>
haha, There is no official way. A few suggestions I have seen
1) do not use nohup use runit svc (daemon tools)
2) use screen
3) start it as its own user (easer to identify)
4) throw some bogus option in the startup like -Dthisishiveserver=true
:)
RE: Logs when running metastore
Posted by Pradeep Kamath <pr...@yahoo-inc.com>.
Also a related question is how do I stop the server if have started the
server with "nohup bin/hive -service metstatore > metastore.log 2>&1 &"
command ? When I run "ps -aef |grep hive", I get :
pradeepk 5463 14236 0 16:00 pts/6 00:00:00 grep hive
pradeepk 27803 14226 0 13:47 pts/5 00:00:04
/usr/java/jdk1.6.0_01/bin/java -Xmx256m
-Dhadoop.log.dir=/homes/pradeepk/hadoopcluster/hadoop/bin/../logs
-Dhadoop.log.file=hadoop.log
-Dhadoop.home.dir=/homes/pradeepk/hadoopcluster/hadoop/bin/..
-Dhadoop.id.str= -Dhadoop.root.logger=INFO,console
-Djava.library.path=/homes/pradeepk/hadoopcluster/hadoop/bin/../lib/nati
ve/Linux-i386-32 -Dhadoop.policy.file=hadoop-policy.xml -classpath
/homes/pradeepk/hadoopcluster/hadoop/conf:/usr/java/jdk1.6.0_01/lib/tool
s.jar:/homes/pradeepk/hadoopcluster/hadoop/bin/..:/homes/pradeepk/hadoop
cluster/hadoop/bin/../hadoop-0.20.1-core.jar:/homes/pradeepk/hadoopclust
er/hadoop/bin/../lib/commons-cli-1.2.jar:/homes/pradeepk/hadoopcluster/h
adoop/bin/../lib/commons-codec-1.3.jar:/homes/pradeepk/hadoopcluster/had
oop/bin/../lib/commons-el-1.0.jar:/homes/pradeepk/hadoopcluster/hadoop/b
in/../lib/commons-httpclient-3.0.1.jar:/homes/pradeepk/hadoopcluster/had
oop/bin/../lib/commons-logging-1.0.4.jar:/homes/pradeepk/hadoopcluster/h
adoop/bin/../lib/commons-logging-api-1.0.4.jar:/homes/pradeepk/hadoopclu
ster/hadoop/bin/../lib/commons-net-1.4.1.jar:/homes/pradeepk/hadoopclust
er/hadoop/bin/../lib/core-3.1.1.jar:/homes/pradeepk/hadoopcluster/hadoop
/bin/../lib/hsqldb-1.8.0.10.jar:/homes/pradeepk/hadoopcluster/hadoop/bin
/../lib/jasper-compiler-5.5.12.jar:/homes/pradeepk/hadoopcluster/hadoop/
bin/../lib/jasper-runtime-5.5.12.jar:/homes/pradeepk/hadoopcluster/hadoo
p/bin/../lib/jets3t-0.6.1.jar:/homes/pradeepk/hadoopcluster/hadoop/bin/.
./lib/jetty-6.1.14.jar:/homes/pradeepk/hadoopcluster/hadoop/bin/../lib/j
etty-util-6.1.14.jar:/homes/pradeepk/hadoopcluster/hadoop/bin/../lib/jun
it-3.8.1.jar:/homes/pradeepk/hadoopcluster/hadoop/bin/../lib/kfs-0.2.2.j
ar:/homes/pradeepk/hadoopcluster/hadoop/bin/../lib/log4j-1.2.15.jar:/hom
es/pradeepk/hadoopcluster/hadoop/bin/../lib/oro-2.0.8.jar:/homes/pradeep
k/hadoopcluster/hadoop/bin/../lib/servlet-api-2.5-6.1.14.jar:/homes/prad
eepk/hadoopcluster/hadoop/bin/../lib/slf4j-api-1.4.3.jar:/homes/pradeepk
/hadoopcluster/hadoop/bin/../lib/slf4j-log4j12-1.4.3.jar:/homes/pradeepk
/hadoopcluster/hadoop/bin/../lib/xmlenc-0.52.jar:/homes/pradeepk/hadoopc
luster/hadoop/bin/../lib/jsp-2.1/jsp-2.1.jar:/homes/pradeepk/hadoopclust
er/hadoop/bin/../lib/jsp-2.1/jsp-api-2.1.jar:/homes/pradeepk/dev/hive/bu
ild/dist/conf:/homes/pradeepk/dev/hive/build/dist/lib/antlr-runtime-3.0.
1.jar:/homes/pradeepk/dev/hive/build/dist/lib/asm-3.1.jar:/homes/pradeep
k/dev/hive/build/dist/lib/commons-cli-2.0-SNAPSHOT.jar:/homes/pradeepk/d
ev/hive/build/dist/lib/commons-codec-1.3.jar:/homes/pradeepk/dev/hive/bu
ild/dist/lib/commons-collections-3.2.1.jar:/homes/pradeepk/dev/hive/buil
d/dist/lib/commons-dbcp-1.2.2.jar:/homes/pradeepk/dev/hive/build/dist/li
b/commons-lang-2.4.jar:/homes/pradeepk/dev/hive/build/dist/lib/commons-l
ogging-1.0.4.jar:/homes/pradeepk/dev/hive/build/dist/lib/commons-logging
-api-1.0.4.jar:/homes/pradeepk/dev/hive/build/dist/lib/commons-pool-1.2.
jar:/homes/pradeepk/dev/hive/build/dist/lib/datanucleus-connectionpool-1
.0.2.jar:/homes/pradeepk/dev/hive/build/dist/lib/datanucleus-core-1.1.2.
jar:/homes/pradeepk/dev/hive/build/dist/lib/datanucleus-enhancer-1.1.2.j
ar:/homes/pradeepk/dev/hive/build/dist/lib/datanucleus-rdbms-1.1.2.jar:/
homes/pradeepk/dev/hive/build/dist/lib/derby.jar:/homes/pradeepk/dev/hiv
e/build/dist/lib/hbase-0.20.3.jar:/homes/pradeepk/dev/hive/build/dist/li
b/hbase-0.20.3-test.jar:/homes/pradeepk/dev/hive/build/dist/lib/hive-ant
tasks-0.6.0.jar:/homes/pradeepk/dev/hive/build/dist/lib/hive-cli-0.6.0.j
ar:/homes/pradeepk/dev/hive/build/dist/lib/hive-common-0.6.0.jar:/homes/
pradeepk/dev/hive/build/dist/lib/hive_contrib.jar:/homes/pradeepk/dev/hi
ve/build/dist/lib/hive-exec-0.6.0.jar:/homes/pradeepk/dev/hive/build/dis
t/lib/hive_hbase-handler.jar:/homes/pradeepk/dev/hive/build/dist/lib/hiv
e-hwi-0.6.0.jar:/homes/pradeepk/dev/hive/build/dist/lib/hive-jdbc-0.6.0.
jar:/homes/pradeepk/dev/hive/build/dist/lib/hive-metastore-0.6.0.jar:/ho
mes/pradeepk/dev/hive/build/dist/lib/hive-serde-0.6.0.jar:/homes/pradeep
k/dev/hive/build/dist/lib/hive-service-0.6.0.jar:/homes/pradeepk/dev/hiv
e/build/dist/lib/hive-shims-0.6.0.jar
Is there a recommended way to identify the metastore server process?
Thanks,
Pradeep
________________________________
From: Pradeep Kamath [mailto:pradeepk@yahoo-inc.com]
Sent: Wednesday, June 09, 2010 3:10 PM
To: hive-user@hadoop.apache.org
Subject: Logs when running metastore
Hi,
Are there any logs created when running the metastore using "bin/hive
-service metastore" ? From what I can tell this command generates output
on the screen and runs in the foreground. Would the right way to run
this as a server be "bin/hive -service metstatore > metastore.log 2>&1
&" ? Is there any other logfile which is written to, while the metastore
server is running?
Thanks,
Pradeep