You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-user@hadoop.apache.org by "Jie Zhou (周杰)" <ji...@allyes.com> on 2013/05/20 23:23:10 UTC
答复: Unable to stop Thrift Server
The direct solution is that first find the pid of hive service and then just kill it.
netstat -anp |grep 10000
kill -9 pid
________________________________
发件人: Raj Hadoop [hadoopraj@yahoo.com]
发送时间: 2013年5月21日 5:17
收件人: Sanjay Subramanian; user@hive.apache.org; User
主题: Re: Unable to stop Thrift Server
Hi Sanjay,
I am using 0.9 version.
I do not have a sudo access. is there any other command to stop the service.
thanks,
raj
________________________________
From: Sanjay Subramanian <Sa...@wizecommerce.com>
To: "user@hive.apache.org" <us...@hive.apache.org>; Raj Hadoop <ha...@yahoo.com>; User <us...@hadoop.apache.org>
Sent: Monday, May 20, 2013 5:11 PM
Subject: Re: Unable to stop Thrift Server
Raj
Which version r u using ?
I think from 0.9+ onwards its best to use service to stop and start and NOT hive
sudo service hive-metastore stop
sudo service hive-server stop
sudo service hive-metastore start
sudo service hive-server start
Couple of general things that might help
1. Use linux screens : then u can start many screen sessions and u don't have to give the synch mode "&" of execution
Its very easy to manage several screen sessions and they keep running till your server restarts….and generally u can ssh to some jumhost and create your screen sessions there
2. Run the following
pstree -pulac | less
U can possible search for hive or your username or root which was used to start the service…and kill the process
sanjay
From: Raj Hadoop <ha...@yahoo.com>>
Reply-To: "user@hive.apache.org<ma...@hive.apache.org>" <us...@hive.apache.org>>, Raj Hadoop <ha...@yahoo.com>>
Date: Monday, May 20, 2013 2:03 PM
To: Hive <us...@hive.apache.org>>, User <us...@hadoop.apache.org>>
Subject: Unable to stop Thrift Server
Hi,
I was not able to stopThrift Server after performing the following steps.
$ bin/hive --service hiveserver &
Starting Hive Thrift Server
$ netstat -nl | grep 10000
tcp 0 0 :::10000 :::* LISTEN
I gave the following to stop. but not working.
hive --service hiveserver --action stop 10000
How can I stop this service?
Thanks,
Raj
CONFIDENTIALITY NOTICE
======================
This email message and any attachments are for the exclusive use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message along with any attachments, from your computer system. If you are the intended recipient, please be advised that the content of this message is subject to access, review and disclosure by the sender's Email System Administrator.
Request perm to edit wiki
Posted by Sanjay Subramanian <Sa...@wizecommerce.com>.
Hi
I'd like to add documentation on how to use LZO compression in Hive
May I get edit permissions?
Thanks.
Sanjay
Confluence username: sanjaysubramanian@yahoo.com<ma...@yahoo.com>
Apache JIRA username - sanjaysubramanian
From: Lefty Leverenz <le...@hortonworks.com>>
Reply-To: "user@hive.apache.org<ma...@hive.apache.org>" <us...@hive.apache.org>>
Date: Tuesday, June 18, 2013 2:12 AM
To: "user@hive.apache.org<ma...@hive.apache.org>" <us...@hive.apache.org>>
Subject: Re: LZO compression implementation in Hive
You can email hive-dev@hadoop.apache.org<ma...@hadoop.apache.org> asking for edit privileges on the Hive wiki.
Here's an example from the archives<http://mail-archives.apache.org/mod_mbox/hive-dev/201304.mbox/%3cCAKHgJS42yQqj54RC8YvzGXbUbiEs7mpV9fDofq2Z3Onfo67zwg@mail.gmail.com%3e>.
Once you have permission, you can use the Edit and Add symbols in the upper right corner of each wiki page.
For page location, I'd have to think it over and check the existing docs. Initial thoughts:
* The Language Manual<https://cwiki.apache.org/confluence/display/Hive/LanguageManual> has a placeholder for file formats which only contains ORC files now.
* Perhaps the DDL section is more appropriate. If so, add a subbullet on the Language Manual page.
* If you don't want it in the Language Manual, just put it on the home page<https://cwiki.apache.org/confluence/display/Hive/Home> and add a link under User Documentation.
* As long as it's somewhere in the Hive wiki tree, the apparent organization can be changed by adding/removing links.
* Many people don't pay attention to the page structure, they just google the topic they're looking for.
– Lefty
On Tue, Jun 18, 2013 at 2:56 AM, Sanjay Subramanian <Sa...@wizecommerce.com>> wrote:
:-) Not sure how to add a page…may be the Admin needs to grant me permission
From: Sanjay Subramanian <sa...@wizecommerce.com>>
Reply-To: "user@hive.apache.org<ma...@hive.apache.org>" <us...@hive.apache.org>>
Date: Monday, June 17, 2013 11:50 PM
To: "user@hive.apache.org<ma...@hive.apache.org>" <us...@hive.apache.org>>
Subject: Re: LZO compression implementation in Hive
Sure…would love to add the LZO compression in Hive
Is there a specific page structure u want me to add to in confluence?
https://cwiki.apache.org/confluence<https://cwiki.apache.org/confluence/dashboard.action>
thanks
sanjay
From: Lefty Leverenz <le...@hortonworks.com>>
Reply-To: "user@hive.apache.org<ma...@hive.apache.org>" <us...@hive.apache.org>>
Date: Monday, June 17, 2013 7:08 PM
To: "user@hive.apache.org<ma...@hive.apache.org>" <us...@hive.apache.org>>
Subject: Re: LZO compression implementation in Hive
Perhaps you'd like to write up your insights in the Hive wiki, and others could add their insights. Then the information would be available to all, immediately.
– Lefty
On Mon, Jun 17, 2013 at 4:39 PM, Ramki Palle <ra...@gmail.com>> wrote:
Hi Sanjay,
Can you quickly give your insights on thip topic, if possible?
Regards,
Ramki.
On Mon, May 20, 2013 at 2:51 PM, Sanjay Subramanian <Sa...@wizecommerce.com>> wrote:
Hi Programming Hive Book authors
Maybe a lot of u have already successfully implemented this but only these last two weeks , we implemented our aggregations using LZO compression in Hive - MR jobs creating LZO files as Input for Hive ---> Therafter Hive aggregations creating more LZO files as output.
As usual nothing was straight forward :-) Also the other challenge was to neatly tie all into actions in Oozie workflows….but after being underwater for weeks I think I am able to rise above water and breathe !
In the next version of the book , If u guys r planning to add detailed sections on using lzo compression in Hive , let me know…my experiences might be useful :-)
Thanks
sanjay
CONFIDENTIALITY NOTICE
======================
This email message and any attachments are for the exclusive use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message along with any attachments, from your computer system. If you are the intended recipient, please be advised that the content of this message is subject to access, review and disclosure by the sender's Email System Administrator.
CONFIDENTIALITY NOTICE
======================
This email message and any attachments are for the exclusive use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message along with any attachments, from your computer system. If you are the intended recipient, please be advised that the content of this message is subject to access, review and disclosure by the sender's Email System Administrator.
CONFIDENTIALITY NOTICE
======================
This email message and any attachments are for the exclusive use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message along with any attachments, from your computer system. If you are the intended recipient, please be advised that the content of this message is subject to access, review and disclosure by the sender's Email System Administrator.
CONFIDENTIALITY NOTICE
======================
This email message and any attachments are for the exclusive use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message along with any attachments, from your computer system. If you are the intended recipient, please be advised that the content of this message is subject to access, review and disclosure by the sender's Email System Administrator.
Request perm to edit wiki
Posted by Sanjay Subramanian <Sa...@wizecommerce.com>.
Hi
I'd like to add documentation on how to use LZO compression in Hive
May I get edit permissions?
Thanks.
Sanjay
Confluence username: sanjaysubramanian@yahoo.com<ma...@yahoo.com>
Apache JIRA username - sanjaysubramanian
From: Lefty Leverenz <le...@hortonworks.com>>
Reply-To: "user@hive.apache.org<ma...@hive.apache.org>" <us...@hive.apache.org>>
Date: Tuesday, June 18, 2013 2:12 AM
To: "user@hive.apache.org<ma...@hive.apache.org>" <us...@hive.apache.org>>
Subject: Re: LZO compression implementation in Hive
You can email hive-dev@hadoop.apache.org<ma...@hadoop.apache.org> asking for edit privileges on the Hive wiki.
Here's an example from the archives<http://mail-archives.apache.org/mod_mbox/hive-dev/201304.mbox/%3cCAKHgJS42yQqj54RC8YvzGXbUbiEs7mpV9fDofq2Z3Onfo67zwg@mail.gmail.com%3e>.
Once you have permission, you can use the Edit and Add symbols in the upper right corner of each wiki page.
For page location, I'd have to think it over and check the existing docs. Initial thoughts:
* The Language Manual<https://cwiki.apache.org/confluence/display/Hive/LanguageManual> has a placeholder for file formats which only contains ORC files now.
* Perhaps the DDL section is more appropriate. If so, add a subbullet on the Language Manual page.
* If you don't want it in the Language Manual, just put it on the home page<https://cwiki.apache.org/confluence/display/Hive/Home> and add a link under User Documentation.
* As long as it's somewhere in the Hive wiki tree, the apparent organization can be changed by adding/removing links.
* Many people don't pay attention to the page structure, they just google the topic they're looking for.
– Lefty
On Tue, Jun 18, 2013 at 2:56 AM, Sanjay Subramanian <Sa...@wizecommerce.com>> wrote:
:-) Not sure how to add a page…may be the Admin needs to grant me permission
From: Sanjay Subramanian <sa...@wizecommerce.com>>
Reply-To: "user@hive.apache.org<ma...@hive.apache.org>" <us...@hive.apache.org>>
Date: Monday, June 17, 2013 11:50 PM
To: "user@hive.apache.org<ma...@hive.apache.org>" <us...@hive.apache.org>>
Subject: Re: LZO compression implementation in Hive
Sure…would love to add the LZO compression in Hive
Is there a specific page structure u want me to add to in confluence?
https://cwiki.apache.org/confluence<https://cwiki.apache.org/confluence/dashboard.action>
thanks
sanjay
From: Lefty Leverenz <le...@hortonworks.com>>
Reply-To: "user@hive.apache.org<ma...@hive.apache.org>" <us...@hive.apache.org>>
Date: Monday, June 17, 2013 7:08 PM
To: "user@hive.apache.org<ma...@hive.apache.org>" <us...@hive.apache.org>>
Subject: Re: LZO compression implementation in Hive
Perhaps you'd like to write up your insights in the Hive wiki, and others could add their insights. Then the information would be available to all, immediately.
– Lefty
On Mon, Jun 17, 2013 at 4:39 PM, Ramki Palle <ra...@gmail.com>> wrote:
Hi Sanjay,
Can you quickly give your insights on thip topic, if possible?
Regards,
Ramki.
On Mon, May 20, 2013 at 2:51 PM, Sanjay Subramanian <Sa...@wizecommerce.com>> wrote:
Hi Programming Hive Book authors
Maybe a lot of u have already successfully implemented this but only these last two weeks , we implemented our aggregations using LZO compression in Hive - MR jobs creating LZO files as Input for Hive ---> Therafter Hive aggregations creating more LZO files as output.
As usual nothing was straight forward :-) Also the other challenge was to neatly tie all into actions in Oozie workflows….but after being underwater for weeks I think I am able to rise above water and breathe !
In the next version of the book , If u guys r planning to add detailed sections on using lzo compression in Hive , let me know…my experiences might be useful :-)
Thanks
sanjay
CONFIDENTIALITY NOTICE
======================
This email message and any attachments are for the exclusive use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message along with any attachments, from your computer system. If you are the intended recipient, please be advised that the content of this message is subject to access, review and disclosure by the sender's Email System Administrator.
CONFIDENTIALITY NOTICE
======================
This email message and any attachments are for the exclusive use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message along with any attachments, from your computer system. If you are the intended recipient, please be advised that the content of this message is subject to access, review and disclosure by the sender's Email System Administrator.
CONFIDENTIALITY NOTICE
======================
This email message and any attachments are for the exclusive use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message along with any attachments, from your computer system. If you are the intended recipient, please be advised that the content of this message is subject to access, review and disclosure by the sender's Email System Administrator.
CONFIDENTIALITY NOTICE
======================
This email message and any attachments are for the exclusive use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message along with any attachments, from your computer system. If you are the intended recipient, please be advised that the content of this message is subject to access, review and disclosure by the sender's Email System Administrator.
Request perm to edit wiki
Posted by Sanjay Subramanian <Sa...@wizecommerce.com>.
I'd like to add documentation on how to use LZO compression in Hive
Can I get edit permissions?
Thanks.
Sanjay
username: sanjaysubramanian@yahoo.com
From: Lefty Leverenz <le...@hortonworks.com>>
Reply-To: "user@hive.apache.org<ma...@hive.apache.org>" <us...@hive.apache.org>>
Date: Tuesday, June 18, 2013 2:12 AM
To: "user@hive.apache.org<ma...@hive.apache.org>" <us...@hive.apache.org>>
Subject: Re: LZO compression implementation in Hive
You can email hive-dev@hadoop.apache.org<ma...@hadoop.apache.org> asking for edit privileges on the Hive wiki.
Here's an example from the archives<http://mail-archives.apache.org/mod_mbox/hive-dev/201304.mbox/%3cCAKHgJS42yQqj54RC8YvzGXbUbiEs7mpV9fDofq2Z3Onfo67zwg@mail.gmail.com%3e>.
Once you have permission, you can use the Edit and Add symbols in the upper right corner of each wiki page.
For page location, I'd have to think it over and check the existing docs. Initial thoughts:
* The Language Manual<https://cwiki.apache.org/confluence/display/Hive/LanguageManual> has a placeholder for file formats which only contains ORC files now.
* Perhaps the DDL section is more appropriate. If so, add a subbullet on the Language Manual page.
* If you don't want it in the Language Manual, just put it on the home page<https://cwiki.apache.org/confluence/display/Hive/Home> and add a link under User Documentation.
* As long as it's somewhere in the Hive wiki tree, the apparent organization can be changed by adding/removing links.
* Many people don't pay attention to the page structure, they just google the topic they're looking for.
– Lefty
On Tue, Jun 18, 2013 at 2:56 AM, Sanjay Subramanian <Sa...@wizecommerce.com>> wrote:
:-) Not sure how to add a page…may be the Admin needs to grant me permission
From: Sanjay Subramanian <sa...@wizecommerce.com>>
Reply-To: "user@hive.apache.org<ma...@hive.apache.org>" <us...@hive.apache.org>>
Date: Monday, June 17, 2013 11:50 PM
To: "user@hive.apache.org<ma...@hive.apache.org>" <us...@hive.apache.org>>
Subject: Re: LZO compression implementation in Hive
Sure…would love to add the LZO compression in Hive
Is there a specific page structure u want me to add to in confluence?
https://cwiki.apache.org/confluence<https://cwiki.apache.org/confluence/dashboard.action>
thanks
sanjay
From: Lefty Leverenz <le...@hortonworks.com>>
Reply-To: "user@hive.apache.org<ma...@hive.apache.org>" <us...@hive.apache.org>>
Date: Monday, June 17, 2013 7:08 PM
To: "user@hive.apache.org<ma...@hive.apache.org>" <us...@hive.apache.org>>
Subject: Re: LZO compression implementation in Hive
Perhaps you'd like to write up your insights in the Hive wiki, and others could add their insights. Then the information would be available to all, immediately.
– Lefty
On Mon, Jun 17, 2013 at 4:39 PM, Ramki Palle <ra...@gmail.com>> wrote:
Hi Sanjay,
Can you quickly give your insights on thip topic, if possible?
Regards,
Ramki.
On Mon, May 20, 2013 at 2:51 PM, Sanjay Subramanian <Sa...@wizecommerce.com>> wrote:
Hi Programming Hive Book authors
Maybe a lot of u have already successfully implemented this but only these last two weeks , we implemented our aggregations using LZO compression in Hive - MR jobs creating LZO files as Input for Hive ---> Therafter Hive aggregations creating more LZO files as output.
As usual nothing was straight forward :-) Also the other challenge was to neatly tie all into actions in Oozie workflows….but after being underwater for weeks I think I am able to rise above water and breathe !
In the next version of the book , If u guys r planning to add detailed sections on using lzo compression in Hive , let me know…my experiences might be useful :-)
Thanks
sanjay
CONFIDENTIALITY NOTICE
======================
This email message and any attachments are for the exclusive use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message along with any attachments, from your computer system. If you are the intended recipient, please be advised that the content of this message is subject to access, review and disclosure by the sender's Email System Administrator.
CONFIDENTIALITY NOTICE
======================
This email message and any attachments are for the exclusive use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message along with any attachments, from your computer system. If you are the intended recipient, please be advised that the content of this message is subject to access, review and disclosure by the sender's Email System Administrator.
CONFIDENTIALITY NOTICE
======================
This email message and any attachments are for the exclusive use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message along with any attachments, from your computer system. If you are the intended recipient, please be advised that the content of this message is subject to access, review and disclosure by the sender's Email System Administrator.
CONFIDENTIALITY NOTICE
======================
This email message and any attachments are for the exclusive use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message along with any attachments, from your computer system. If you are the intended recipient, please be advised that the content of this message is subject to access, review and disclosure by the sender's Email System Administrator.
Re: LZO compression implementation in Hive
Posted by Sanjay Subramanian <Sa...@wizecommerce.com>.
Thanks I sent it
From: Lefty Leverenz <le...@hortonworks.com>>
Reply-To: "user@hive.apache.org<ma...@hive.apache.org>" <us...@hive.apache.org>>
Date: Tuesday, June 18, 2013 2:12 AM
To: "user@hive.apache.org<ma...@hive.apache.org>" <us...@hive.apache.org>>
Subject: Re: LZO compression implementation in Hive
You can email hive-dev@hadoop.apache.org<ma...@hadoop.apache.org> asking for edit privileges on the Hive wiki.
Here's an example from the archives<http://mail-archives.apache.org/mod_mbox/hive-dev/201304.mbox/%3cCAKHgJS42yQqj54RC8YvzGXbUbiEs7mpV9fDofq2Z3Onfo67zwg@mail.gmail.com%3e>.
Once you have permission, you can use the Edit and Add symbols in the upper right corner of each wiki page.
For page location, I'd have to think it over and check the existing docs. Initial thoughts:
* The Language Manual<https://cwiki.apache.org/confluence/display/Hive/LanguageManual> has a placeholder for file formats which only contains ORC files now.
* Perhaps the DDL section is more appropriate. If so, add a subbullet on the Language Manual page.
* If you don't want it in the Language Manual, just put it on the home page<https://cwiki.apache.org/confluence/display/Hive/Home> and add a link under User Documentation.
* As long as it's somewhere in the Hive wiki tree, the apparent organization can be changed by adding/removing links.
* Many people don't pay attention to the page structure, they just google the topic they're looking for.
– Lefty
On Tue, Jun 18, 2013 at 2:56 AM, Sanjay Subramanian <Sa...@wizecommerce.com>> wrote:
:-) Not sure how to add a page…may be the Admin needs to grant me permission
From: Sanjay Subramanian <sa...@wizecommerce.com>>
Reply-To: "user@hive.apache.org<ma...@hive.apache.org>" <us...@hive.apache.org>>
Date: Monday, June 17, 2013 11:50 PM
To: "user@hive.apache.org<ma...@hive.apache.org>" <us...@hive.apache.org>>
Subject: Re: LZO compression implementation in Hive
Sure…would love to add the LZO compression in Hive
Is there a specific page structure u want me to add to in confluence?
https://cwiki.apache.org/confluence<https://cwiki.apache.org/confluence/dashboard.action>
thanks
sanjay
From: Lefty Leverenz <le...@hortonworks.com>>
Reply-To: "user@hive.apache.org<ma...@hive.apache.org>" <us...@hive.apache.org>>
Date: Monday, June 17, 2013 7:08 PM
To: "user@hive.apache.org<ma...@hive.apache.org>" <us...@hive.apache.org>>
Subject: Re: LZO compression implementation in Hive
Perhaps you'd like to write up your insights in the Hive wiki, and others could add their insights. Then the information would be available to all, immediately.
– Lefty
On Mon, Jun 17, 2013 at 4:39 PM, Ramki Palle <ra...@gmail.com>> wrote:
Hi Sanjay,
Can you quickly give your insights on thip topic, if possible?
Regards,
Ramki.
On Mon, May 20, 2013 at 2:51 PM, Sanjay Subramanian <Sa...@wizecommerce.com>> wrote:
Hi Programming Hive Book authors
Maybe a lot of u have already successfully implemented this but only these last two weeks , we implemented our aggregations using LZO compression in Hive - MR jobs creating LZO files as Input for Hive ---> Therafter Hive aggregations creating more LZO files as output.
As usual nothing was straight forward :-) Also the other challenge was to neatly tie all into actions in Oozie workflows….but after being underwater for weeks I think I am able to rise above water and breathe !
In the next version of the book , If u guys r planning to add detailed sections on using lzo compression in Hive , let me know…my experiences might be useful :-)
Thanks
sanjay
CONFIDENTIALITY NOTICE
======================
This email message and any attachments are for the exclusive use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message along with any attachments, from your computer system. If you are the intended recipient, please be advised that the content of this message is subject to access, review and disclosure by the sender's Email System Administrator.
CONFIDENTIALITY NOTICE
======================
This email message and any attachments are for the exclusive use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message along with any attachments, from your computer system. If you are the intended recipient, please be advised that the content of this message is subject to access, review and disclosure by the sender's Email System Administrator.
CONFIDENTIALITY NOTICE
======================
This email message and any attachments are for the exclusive use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message along with any attachments, from your computer system. If you are the intended recipient, please be advised that the content of this message is subject to access, review and disclosure by the sender's Email System Administrator.
CONFIDENTIALITY NOTICE
======================
This email message and any attachments are for the exclusive use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message along with any attachments, from your computer system. If you are the intended recipient, please be advised that the content of this message is subject to access, review and disclosure by the sender's Email System Administrator.
Re: LZO compression implementation in Hive
Posted by Lefty Leverenz <le...@hortonworks.com>.
You can email hive-dev@hadoop.apache.org asking for edit privileges on the
Hive wiki.
Here's an example from the
archives<http://mail-archives.apache.org/mod_mbox/hive-dev/201304.mbox/%3cCAKHgJS42yQqj54RC8YvzGXbUbiEs7mpV9fDofq2Z3Onfo67zwg@mail.gmail.com%3e>
.
Once you have permission, you can use the Edit and Add symbols in the upper
right corner of each wiki page.
For page location, I'd have to think it over and check the existing docs.
Initial thoughts:
- The Language
Manual<https://cwiki.apache.org/confluence/display/Hive/LanguageManual>has
a placeholder for file formats which only contains ORC files now.
- Perhaps the DDL section is more appropriate. If so, add a subbullet
on the Language Manual page.
- If you don't want it in the Language Manual, just put it on the home
page <https://cwiki.apache.org/confluence/display/Hive/Home> and add a
link under User Documentation.
- As long as it's somewhere in the Hive wiki tree, the apparent
organization can be changed by adding/removing links.
- Many people don't pay attention to the page structure, they just
google the topic they're looking for.
– Lefty
On Tue, Jun 18, 2013 at 2:56 AM, Sanjay Subramanian <
Sanjay.Subramanian@wizecommerce.com> wrote:
> :-) Not sure how to add a page…may be the Admin needs to grant me
> permission
>
> From: Sanjay Subramanian <sa...@wizecommerce.com>
> Reply-To: "user@hive.apache.org" <us...@hive.apache.org>
> Date: Monday, June 17, 2013 11:50 PM
>
> To: "user@hive.apache.org" <us...@hive.apache.org>
> Subject: Re: LZO compression implementation in Hive
>
> Sure…would love to add the LZO compression in Hive
> Is there a specific page structure u want me to add to in confluence?
>
> https://cwiki.apache.org/confluence<https://cwiki.apache.org/confluence/dashboard.action>
>
> thanks
> sanjay
>
>
> From: Lefty Leverenz <le...@hortonworks.com>
> Reply-To: "user@hive.apache.org" <us...@hive.apache.org>
> Date: Monday, June 17, 2013 7:08 PM
> To: "user@hive.apache.org" <us...@hive.apache.org>
> Subject: Re: LZO compression implementation in Hive
>
> Perhaps you'd like to write up your insights in the Hive wiki, and
> others could add their insights. Then the information would be available
> to all, immediately.
>
> – Lefty
>
>
>
> On Mon, Jun 17, 2013 at 4:39 PM, Ramki Palle <ra...@gmail.com>wrote:
>
>> Hi Sanjay,
>>
>> Can you quickly give your insights on thip topic, if possible?
>>
>> Regards,
>> Ramki.
>>
>>
>> On Mon, May 20, 2013 at 2:51 PM, Sanjay Subramanian <
>> Sanjay.Subramanian@wizecommerce.com> wrote:
>>
>>> Hi Programming Hive Book authors
>>>
>>> Maybe a lot of u have already successfully implemented this but only
>>> these last two weeks , we implemented our aggregations using LZO
>>> compression in Hive - MR jobs creating LZO files as Input for Hive --->
>>> Therafter Hive aggregations creating more LZO files as output.
>>> As usual nothing was straight forward :-) Also the other challenge was
>>> to neatly tie all into actions in Oozie workflows….but after being
>>> underwater for weeks I think I am able to rise above water and breathe !
>>>
>>> In the next version of the book , If u guys r planning to add detailed
>>> sections on using lzo compression in Hive , let me know…my experiences
>>> might be useful :-)
>>>
>>> Thanks
>>>
>>> sanjay
>>>
>>>
>>>
>>> CONFIDENTIALITY NOTICE
>>> ======================
>>> This email message and any attachments are for the exclusive use of the
>>> intended recipient(s) and may contain confidential and privileged
>>> information. Any unauthorized review, use, disclosure or distribution is
>>> prohibited. If you are not the intended recipient, please contact the
>>> sender by reply email and destroy all copies of the original message along
>>> with any attachments, from your computer system. If you are the intended
>>> recipient, please be advised that the content of this message is subject to
>>> access, review and disclosure by the sender's Email System Administrator.
>>>
>>
>>
>
> CONFIDENTIALITY NOTICE
> ======================
> This email message and any attachments are for the exclusive use of the
> intended recipient(s) and may contain confidential and privileged
> information. Any unauthorized review, use, disclosure or distribution is
> prohibited. If you are not the intended recipient, please contact the
> sender by reply email and destroy all copies of the original message along
> with any attachments, from your computer system. If you are the intended
> recipient, please be advised that the content of this message is subject to
> access, review and disclosure by the sender's Email System Administrator.
>
> CONFIDENTIALITY NOTICE
> ======================
> This email message and any attachments are for the exclusive use of the
> intended recipient(s) and may contain confidential and privileged
> information. Any unauthorized review, use, disclosure or distribution is
> prohibited. If you are not the intended recipient, please contact the
> sender by reply email and destroy all copies of the original message along
> with any attachments, from your computer system. If you are the intended
> recipient, please be advised that the content of this message is subject to
> access, review and disclosure by the sender's Email System Administrator.
>
Re: Errors in one Hive script using LZO compression
Posted by Sanjay Subramanian <Sa...@wizecommerce.com>.
Yes I am going to start debugging from the inner query working my way outwards….starting tomorrow AM… :-)
From: Sanjay Subramanian <sa...@wizecommerce.com>>
Date: Monday, June 17, 2013 11:59 PM
To: "user@hive.apache.org<ma...@hive.apache.org>" <us...@hive.apache.org>>
Subject: Errors in one Hive script using LZO compression
Hi
I am using LZO compression in our scripts but one script is still creating errors
Diagnostic Messages for this Task:
Error: java.io.IOException: java.io.EOFException: Premature EOF from inputStream
at org.apache.hadoop.hive.io.HiveIOExceptionHandlerChain.handleRecordReaderCreationException(HiveIOExceptionHandlerChain.java:97)
at org.apache.hadoop.hive.io.HiveIOExceptionHandlerUtil.handleRecordReaderCreationException(HiveIOExceptionHandlerUtil.java:57)
at org.apache.hadoop.hive.ql.io.HiveInputFormat.getRecordReader(HiveInputFormat.java:243)
at org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getRecordReader(CombineHiveInputFormat.java:522)
at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.<init>(MapTask.java:160)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:381)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:334)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:152)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:147)
Caused by: java.io.EOFException: Premature EOF from inputStream
at com.hadoop.compression.lzo.LzopInputStream.readFully(LzopInputStream.java:75)
at com.hadoop.compression.lzo.LzopInputStream.readHeader(LzopInputStream.java:114)
at com.hadoop.compression.lzo.LzopInputStream.<init>(LzopInputStream.java:54)
at com.hadoop.compression.lzo.LzopCodec.createInputStream(LzopCodec.java:83)
at org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1871)
at org.apache.hadoop.io.SequenceFile$Reader.initialize(SequenceFile.java:1765)
at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1714)
at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1728)
at org.apache.hadoop.mapred.SequenceFileRecordReader.<init>(SequenceFileRecordReader.java:49)
at org.apache.hadoop.mapred.SequenceFileInputFormat.getRecordReader(SequenceFileInputFormat.java:64)
at org.apache.hadoop.hive.ql.io.HiveInputFormat.getRecordReader(HiveInputFormat.java:240)
... 9 more
SCRIPT
=======
set hiveconf mapred.output.compression.type=BLOCK;
set mapred.map.output.compression.codec=org.apache.hadoop.io.compress.SnappyCodec;
set mapreduce.map.output.compress=true;
set hive.exec.compress.output=true;
set mapreduce.output.fileoutputformat.compress.codec=com.hadoop.compression.lzo.LzopCodec;
set mapreduce.output.fileoutputformat.compress=true;
set hive.exec.compress.intermediate=true;
set mapreduce.job.maps=500;
set mapreduce.job.reduces=8;
set mapreduce.tasktracker.map.tasks.maximum=12;
set mapreduce.tasktracker.reduce.tasks.maximum=8;
add jar /home/nextag/sasubramanian/mycode/impressions/jar/impressions-hiveudfs-1.0-20130615-155038.jar;
create temporary function collect as 'com.wizecommerce.utils.hive.udf.GenericUDAFCollect';
create temporary function isnextagip as 'com.wizecommerce.utils.hive.udf.IsNextagIP';
create temporary function isfrombot as 'com.wizecommerce.utils.hive.udf.IsFromBot';
create temporary function processblankkeyword as 'com.wizecommerce.utils.hive.udf.ProcessBlankKeyword';
create temporary function getValidHiddenSellers as 'com.wizecommerce.utils.hive.udf.GetValidHiddenSellers';
INSERT OVERWRITE DIRECTORY '/user/beeswax/warehouse/keyword_impressions_ptitles_log/2013-03-19'
SELECT
hp.header_date,
hp.impression_id,
hp.header_searchsessionid,
hp.cached_visit_id,
split(hp.header_servername,'[\.]')[0],
hp.cached_ip,
hp.header_adnode,
IF (concat_ws(',' , collect_set(concat_ws('|', cast(hp.seller_id as STRING), cast(IF(hp.seller_pricetier IS NULL, -1L, hp.seller_pricetier) as STRING), cast(hp.seller_price as STRING), cast(IF(hp.ptitle_rank IS NULL, -1L, hp.ptitle_rank) as STRING)))) = '-1|-1',NULL,concat_ws(',' , collect_set(concat_ws('|', cast(hp.seller_id as STRING), cast(IF(hp.seller_pricetier IS NULL, -1L, hp.seller_pricetier) as STRING), cast(hp.seller_price as STRING), cast(IF(hp.ptitle_rank IS NULL, -1L, hp.ptitle_rank) as STRING))))),
IF(concat_ws(',' , getValidHiddenSellers(collect_set(concat_ws('|', cast(sh.seller_id as STRING), cast(sh.ptitle_id as STRING), cast(sh.tag_id as STRING), cast(IF(sh.price_tier IS NULL, -1L, sh.price_tier) as STRING))))) = '',NULL, concat_ws(',' , getValidHiddenSellers(collect_set(concat_ws('|', cast(sh.seller_id as STRING), cast(sh.ptitle_id as STRING), cast(sh.tag_id as STRING), cast(IF(sh.price_tier IS NULL, -1L, sh.price_tier) as STRING))))))
FROM
(SELECT
h.header_date,
h.header_servername,
h.impression_id,
h.header_searchsessionid,
h.cached_visit_id,
h.cached_ip,
h.header_adnode,
p.ptitle_ptitleid,
p.seller_id,
p.seller_pricetier,
p.seller_price,
p.ptitle_rank
FROM
(SELECT
header_date,
header_servername,
impression_id,
header_searchsessionid,
cached_ip,
header_adnode,
cached_recordid,
cached_visit_id
FROM
outpdir_impressions_header
WHERE
header_date_partition='2013-03-19'
AND
header_rbabsentsellers = 1L
AND
cached_recordid IS NOT NULL
AND
isnextagip(cached_ip) = FALSE
AND
isfrombot(cached_visit_id) = FALSE
) h
LEFT OUTER JOIN
(SELECT
po.impression_id,
po.ptitle_ptitleid,
po.header_date,
po.seller_id,
po.seller_pricetier,
po.seller_price,
po.ptitle_rank
FROM
(SELECT
impression_id,
ptitle_ptitleid,
header_date,
seller_id,
seller_pricetier,
seller_price,
ptitle_rank
FROM
outpdir_impressions_ptitle
WHERE
header_date_partition = '2013-03-19'
AND
seller_id IS NOT NULL
) po
JOIN
(SELECT
impression_id,
ptitle_ptitleid,
ptitle_rank,
COUNT(DISTINCT seller_id, seller_pricetier, seller_price, ptitle_rank)
FROM
outpdir_impressions_ptitle pi
WHERE
header_date_partition = '2013-03-19'
AND
seller_id IS NOT NULL
GROUP BY
impression_id,
ptitle_ptitleid,
ptitle_rank
HAVING
COUNT(DISTINCT seller_id, seller_pricetier, seller_price, ptitle_rank) = 1
) pi
ON
po.impression_id = pi.impression_id
AND
po.ptitle_ptitleid = pi.ptitle_ptitleid
AND
po.ptitle_rank = pi.ptitle_rank
) p
ON
h.impression_id = p.impression_id
AND
h.header_date=p.header_date
) hp
LEFT OUTER JOIN
(SELECT
*
FROM
outpdir_seller_hidden
WHERE
header_date_partition='2013-03-19'
) sh
ON
hp.impression_id = sh.impression_id
AND
hp.header_date = sh.header_date
GROUP BY
hp.header_date,
hp.impression_id,
hp.header_searchsessionid,
hp.cached_visit_id,
hp.header_servername,
hp.cached_ip,
hp.header_adnode
CONFIDENTIALITY NOTICE
======================
This email message and any attachments are for the exclusive use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message along with any attachments, from your computer system. If you are the intended recipient, please be advised that the content of this message is subject to access, review and disclosure by the sender's Email System Administrator.
Re: Errors in one Hive script using LZO compression
Posted by Sanjay Subramanian <Sa...@wizecommerce.com>.
Ok guys I solved it in not so elegant way but I need to go forward in production and deploy this because of time constraints :-)
I divided the scripts into two stages
Stage 1 : The hive script creates TXT files and writes to HDFS
Stage 2 : I wrote a Lzo file creator and indexer that will convert the TXT files on HDFS to .lzo and .lzo.index
I still don't know what makes this specific hive script throw this error….but I got to keep going ahead….
Perhaps if anyone can shed more light on this error in the future , I will STILL be interested in knowing the root cause
Thanks
sanjay
From: Sanjay Subramanian <sa...@wizecommerce.com>>
Reply-To: "user@hive.apache.org<ma...@hive.apache.org>" <us...@hive.apache.org>>
Date: Monday, June 17, 2013 11:59 PM
To: "user@hive.apache.org<ma...@hive.apache.org>" <us...@hive.apache.org>>
Subject: Errors in one Hive script using LZO compression
Hi
I am using LZO compression in our scripts but one script is still creating errors
Diagnostic Messages for this Task:
Error: java.io.IOException: java.io.EOFException: Premature EOF from inputStream
at org.apache.hadoop.hive.io.HiveIOExceptionHandlerChain.handleRecordReaderCreationException(HiveIOExceptionHandlerChain.java:97)
at org.apache.hadoop.hive.io.HiveIOExceptionHandlerUtil.handleRecordReaderCreationException(HiveIOExceptionHandlerUtil.java:57)
at org.apache.hadoop.hive.ql.io.HiveInputFormat.getRecordReader(HiveInputFormat.java:243)
at org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getRecordReader(CombineHiveInputFormat.java:522)
at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.<init>(MapTask.java:160)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:381)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:334)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:152)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:147)
Caused by: java.io.EOFException: Premature EOF from inputStream
at com.hadoop.compression.lzo.LzopInputStream.readFully(LzopInputStream.java:75)
at com.hadoop.compression.lzo.LzopInputStream.readHeader(LzopInputStream.java:114)
at com.hadoop.compression.lzo.LzopInputStream.<init>(LzopInputStream.java:54)
at com.hadoop.compression.lzo.LzopCodec.createInputStream(LzopCodec.java:83)
at org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1871)
at org.apache.hadoop.io.SequenceFile$Reader.initialize(SequenceFile.java:1765)
at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1714)
at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1728)
at org.apache.hadoop.mapred.SequenceFileRecordReader.<init>(SequenceFileRecordReader.java:49)
at org.apache.hadoop.mapred.SequenceFileInputFormat.getRecordReader(SequenceFileInputFormat.java:64)
at org.apache.hadoop.hive.ql.io.HiveInputFormat.getRecordReader(HiveInputFormat.java:240)
... 9 more
SCRIPT
=======
set hiveconf mapred.output.compression.type=BLOCK;
set mapred.map.output.compression.codec=org.apache.hadoop.io.compress.SnappyCodec;
set mapreduce.map.output.compress=true;
set hive.exec.compress.output=true;
set mapreduce.output.fileoutputformat.compress.codec=com.hadoop.compression.lzo.LzopCodec;
set mapreduce.output.fileoutputformat.compress=true;
set hive.exec.compress.intermediate=true;
set mapreduce.job.maps=500;
set mapreduce.job.reduces=8;
set mapreduce.tasktracker.map.tasks.maximum=12;
set mapreduce.tasktracker.reduce.tasks.maximum=8;
add jar /home/nextag/sasubramanian/mycode/impressions/jar/impressions-hiveudfs-1.0-20130615-155038.jar;
create temporary function collect as 'com.wizecommerce.utils.hive.udf.GenericUDAFCollect';
create temporary function isnextagip as 'com.wizecommerce.utils.hive.udf.IsNextagIP';
create temporary function isfrombot as 'com.wizecommerce.utils.hive.udf.IsFromBot';
create temporary function processblankkeyword as 'com.wizecommerce.utils.hive.udf.ProcessBlankKeyword';
create temporary function getValidHiddenSellers as 'com.wizecommerce.utils.hive.udf.GetValidHiddenSellers';
INSERT OVERWRITE DIRECTORY '/user/beeswax/warehouse/keyword_impressions_ptitles_log/2013-03-19'
SELECT
hp.header_date,
hp.impression_id,
hp.header_searchsessionid,
hp.cached_visit_id,
split(hp.header_servername,'[\.]')[0],
hp.cached_ip,
hp.header_adnode,
IF (concat_ws(',' , collect_set(concat_ws('|', cast(hp.seller_id as STRING), cast(IF(hp.seller_pricetier IS NULL, -1L, hp.seller_pricetier) as STRING), cast(hp.seller_price as STRING), cast(IF(hp.ptitle_rank IS NULL, -1L, hp.ptitle_rank) as STRING)))) = '-1|-1',NULL,concat_ws(',' , collect_set(concat_ws('|', cast(hp.seller_id as STRING), cast(IF(hp.seller_pricetier IS NULL, -1L, hp.seller_pricetier) as STRING), cast(hp.seller_price as STRING), cast(IF(hp.ptitle_rank IS NULL, -1L, hp.ptitle_rank) as STRING))))),
IF(concat_ws(',' , getValidHiddenSellers(collect_set(concat_ws('|', cast(sh.seller_id as STRING), cast(sh.ptitle_id as STRING), cast(sh.tag_id as STRING), cast(IF(sh.price_tier IS NULL, -1L, sh.price_tier) as STRING))))) = '',NULL, concat_ws(',' , getValidHiddenSellers(collect_set(concat_ws('|', cast(sh.seller_id as STRING), cast(sh.ptitle_id as STRING), cast(sh.tag_id as STRING), cast(IF(sh.price_tier IS NULL, -1L, sh.price_tier) as STRING))))))
FROM
(SELECT
h.header_date,
h.header_servername,
h.impression_id,
h.header_searchsessionid,
h.cached_visit_id,
h.cached_ip,
h.header_adnode,
p.ptitle_ptitleid,
p.seller_id,
p.seller_pricetier,
p.seller_price,
p.ptitle_rank
FROM
(SELECT
header_date,
header_servername,
impression_id,
header_searchsessionid,
cached_ip,
header_adnode,
cached_recordid,
cached_visit_id
FROM
outpdir_impressions_header
WHERE
header_date_partition='2013-03-19'
AND
header_rbabsentsellers = 1L
AND
cached_recordid IS NOT NULL
AND
isnextagip(cached_ip) = FALSE
AND
isfrombot(cached_visit_id) = FALSE
) h
LEFT OUTER JOIN
(SELECT
po.impression_id,
po.ptitle_ptitleid,
po.header_date,
po.seller_id,
po.seller_pricetier,
po.seller_price,
po.ptitle_rank
FROM
(SELECT
impression_id,
ptitle_ptitleid,
header_date,
seller_id,
seller_pricetier,
seller_price,
ptitle_rank
FROM
outpdir_impressions_ptitle
WHERE
header_date_partition = '2013-03-19'
AND
seller_id IS NOT NULL
) po
JOIN
(SELECT
impression_id,
ptitle_ptitleid,
ptitle_rank,
COUNT(DISTINCT seller_id, seller_pricetier, seller_price, ptitle_rank)
FROM
outpdir_impressions_ptitle pi
WHERE
header_date_partition = '2013-03-19'
AND
seller_id IS NOT NULL
GROUP BY
impression_id,
ptitle_ptitleid,
ptitle_rank
HAVING
COUNT(DISTINCT seller_id, seller_pricetier, seller_price, ptitle_rank) = 1
) pi
ON
po.impression_id = pi.impression_id
AND
po.ptitle_ptitleid = pi.ptitle_ptitleid
AND
po.ptitle_rank = pi.ptitle_rank
) p
ON
h.impression_id = p.impression_id
AND
h.header_date=p.header_date
) hp
LEFT OUTER JOIN
(SELECT
*
FROM
outpdir_seller_hidden
WHERE
header_date_partition='2013-03-19'
) sh
ON
hp.impression_id = sh.impression_id
AND
hp.header_date = sh.header_date
GROUP BY
hp.header_date,
hp.impression_id,
hp.header_searchsessionid,
hp.cached_visit_id,
hp.header_servername,
hp.cached_ip,
hp.header_adnode
CONFIDENTIALITY NOTICE
======================
This email message and any attachments are for the exclusive use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message along with any attachments, from your computer system. If you are the intended recipient, please be advised that the content of this message is subject to access, review and disclosure by the sender's Email System Administrator.
CONFIDENTIALITY NOTICE
======================
This email message and any attachments are for the exclusive use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message along with any attachments, from your computer system. If you are the intended recipient, please be advised that the content of this message is subject to access, review and disclosure by the sender's Email System Administrator.
Errors in one Hive script using LZO compression
Posted by Sanjay Subramanian <Sa...@wizecommerce.com>.
Hi
I am using LZO compression in our scripts but one script is still creating errors
Diagnostic Messages for this Task:
Error: java.io.IOException: java.io.EOFException: Premature EOF from inputStream
at org.apache.hadoop.hive.io.HiveIOExceptionHandlerChain.handleRecordReaderCreationException(HiveIOExceptionHandlerChain.java:97)
at org.apache.hadoop.hive.io.HiveIOExceptionHandlerUtil.handleRecordReaderCreationException(HiveIOExceptionHandlerUtil.java:57)
at org.apache.hadoop.hive.ql.io.HiveInputFormat.getRecordReader(HiveInputFormat.java:243)
at org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getRecordReader(CombineHiveInputFormat.java:522)
at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.<init>(MapTask.java:160)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:381)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:334)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:152)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:147)
Caused by: java.io.EOFException: Premature EOF from inputStream
at com.hadoop.compression.lzo.LzopInputStream.readFully(LzopInputStream.java:75)
at com.hadoop.compression.lzo.LzopInputStream.readHeader(LzopInputStream.java:114)
at com.hadoop.compression.lzo.LzopInputStream.<init>(LzopInputStream.java:54)
at com.hadoop.compression.lzo.LzopCodec.createInputStream(LzopCodec.java:83)
at org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1871)
at org.apache.hadoop.io.SequenceFile$Reader.initialize(SequenceFile.java:1765)
at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1714)
at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1728)
at org.apache.hadoop.mapred.SequenceFileRecordReader.<init>(SequenceFileRecordReader.java:49)
at org.apache.hadoop.mapred.SequenceFileInputFormat.getRecordReader(SequenceFileInputFormat.java:64)
at org.apache.hadoop.hive.ql.io.HiveInputFormat.getRecordReader(HiveInputFormat.java:240)
... 9 more
SCRIPT
=======
set hiveconf mapred.output.compression.type=BLOCK;
set mapred.map.output.compression.codec=org.apache.hadoop.io.compress.SnappyCodec;
set mapreduce.map.output.compress=true;
set hive.exec.compress.output=true;
set mapreduce.output.fileoutputformat.compress.codec=com.hadoop.compression.lzo.LzopCodec;
set mapreduce.output.fileoutputformat.compress=true;
set hive.exec.compress.intermediate=true;
set mapreduce.job.maps=500;
set mapreduce.job.reduces=8;
set mapreduce.tasktracker.map.tasks.maximum=12;
set mapreduce.tasktracker.reduce.tasks.maximum=8;
add jar /home/nextag/sasubramanian/mycode/impressions/jar/impressions-hiveudfs-1.0-20130615-155038.jar;
create temporary function collect as 'com.wizecommerce.utils.hive.udf.GenericUDAFCollect';
create temporary function isnextagip as 'com.wizecommerce.utils.hive.udf.IsNextagIP';
create temporary function isfrombot as 'com.wizecommerce.utils.hive.udf.IsFromBot';
create temporary function processblankkeyword as 'com.wizecommerce.utils.hive.udf.ProcessBlankKeyword';
create temporary function getValidHiddenSellers as 'com.wizecommerce.utils.hive.udf.GetValidHiddenSellers';
INSERT OVERWRITE DIRECTORY '/user/beeswax/warehouse/keyword_impressions_ptitles_log/2013-03-19'
SELECT
hp.header_date,
hp.impression_id,
hp.header_searchsessionid,
hp.cached_visit_id,
split(hp.header_servername,'[\.]')[0],
hp.cached_ip,
hp.header_adnode,
IF (concat_ws(',' , collect_set(concat_ws('|', cast(hp.seller_id as STRING), cast(IF(hp.seller_pricetier IS NULL, -1L, hp.seller_pricetier) as STRING), cast(hp.seller_price as STRING), cast(IF(hp.ptitle_rank IS NULL, -1L, hp.ptitle_rank) as STRING)))) = '-1|-1',NULL,concat_ws(',' , collect_set(concat_ws('|', cast(hp.seller_id as STRING), cast(IF(hp.seller_pricetier IS NULL, -1L, hp.seller_pricetier) as STRING), cast(hp.seller_price as STRING), cast(IF(hp.ptitle_rank IS NULL, -1L, hp.ptitle_rank) as STRING))))),
IF(concat_ws(',' , getValidHiddenSellers(collect_set(concat_ws('|', cast(sh.seller_id as STRING), cast(sh.ptitle_id as STRING), cast(sh.tag_id as STRING), cast(IF(sh.price_tier IS NULL, -1L, sh.price_tier) as STRING))))) = '',NULL, concat_ws(',' , getValidHiddenSellers(collect_set(concat_ws('|', cast(sh.seller_id as STRING), cast(sh.ptitle_id as STRING), cast(sh.tag_id as STRING), cast(IF(sh.price_tier IS NULL, -1L, sh.price_tier) as STRING))))))
FROM
(SELECT
h.header_date,
h.header_servername,
h.impression_id,
h.header_searchsessionid,
h.cached_visit_id,
h.cached_ip,
h.header_adnode,
p.ptitle_ptitleid,
p.seller_id,
p.seller_pricetier,
p.seller_price,
p.ptitle_rank
FROM
(SELECT
header_date,
header_servername,
impression_id,
header_searchsessionid,
cached_ip,
header_adnode,
cached_recordid,
cached_visit_id
FROM
outpdir_impressions_header
WHERE
header_date_partition='2013-03-19'
AND
header_rbabsentsellers = 1L
AND
cached_recordid IS NOT NULL
AND
isnextagip(cached_ip) = FALSE
AND
isfrombot(cached_visit_id) = FALSE
) h
LEFT OUTER JOIN
(SELECT
po.impression_id,
po.ptitle_ptitleid,
po.header_date,
po.seller_id,
po.seller_pricetier,
po.seller_price,
po.ptitle_rank
FROM
(SELECT
impression_id,
ptitle_ptitleid,
header_date,
seller_id,
seller_pricetier,
seller_price,
ptitle_rank
FROM
outpdir_impressions_ptitle
WHERE
header_date_partition = '2013-03-19'
AND
seller_id IS NOT NULL
) po
JOIN
(SELECT
impression_id,
ptitle_ptitleid,
ptitle_rank,
COUNT(DISTINCT seller_id, seller_pricetier, seller_price, ptitle_rank)
FROM
outpdir_impressions_ptitle pi
WHERE
header_date_partition = '2013-03-19'
AND
seller_id IS NOT NULL
GROUP BY
impression_id,
ptitle_ptitleid,
ptitle_rank
HAVING
COUNT(DISTINCT seller_id, seller_pricetier, seller_price, ptitle_rank) = 1
) pi
ON
po.impression_id = pi.impression_id
AND
po.ptitle_ptitleid = pi.ptitle_ptitleid
AND
po.ptitle_rank = pi.ptitle_rank
) p
ON
h.impression_id = p.impression_id
AND
h.header_date=p.header_date
) hp
LEFT OUTER JOIN
(SELECT
*
FROM
outpdir_seller_hidden
WHERE
header_date_partition='2013-03-19'
) sh
ON
hp.impression_id = sh.impression_id
AND
hp.header_date = sh.header_date
GROUP BY
hp.header_date,
hp.impression_id,
hp.header_searchsessionid,
hp.cached_visit_id,
hp.header_servername,
hp.cached_ip,
hp.header_adnode
CONFIDENTIALITY NOTICE
======================
This email message and any attachments are for the exclusive use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message along with any attachments, from your computer system. If you are the intended recipient, please be advised that the content of this message is subject to access, review and disclosure by the sender's Email System Administrator.
Re: LZO compression implementation in Hive
Posted by Sanjay Subramanian <Sa...@wizecommerce.com>.
:-) Not sure how to add a page…may be the Admin needs to grant me permission
From: Sanjay Subramanian <sa...@wizecommerce.com>>
Reply-To: "user@hive.apache.org<ma...@hive.apache.org>" <us...@hive.apache.org>>
Date: Monday, June 17, 2013 11:50 PM
To: "user@hive.apache.org<ma...@hive.apache.org>" <us...@hive.apache.org>>
Subject: Re: LZO compression implementation in Hive
Sure…would love to add the LZO compression in Hive
Is there a specific page structure u want me to add to in confluence?
https://cwiki.apache.org/confluence<https://cwiki.apache.org/confluence/dashboard.action>
thanks
sanjay
From: Lefty Leverenz <le...@hortonworks.com>>
Reply-To: "user@hive.apache.org<ma...@hive.apache.org>" <us...@hive.apache.org>>
Date: Monday, June 17, 2013 7:08 PM
To: "user@hive.apache.org<ma...@hive.apache.org>" <us...@hive.apache.org>>
Subject: Re: LZO compression implementation in Hive
Perhaps you'd like to write up your insights in the Hive wiki, and others could add their insights. Then the information would be available to all, immediately.
– Lefty
On Mon, Jun 17, 2013 at 4:39 PM, Ramki Palle <ra...@gmail.com>> wrote:
Hi Sanjay,
Can you quickly give your insights on thip topic, if possible?
Regards,
Ramki.
On Mon, May 20, 2013 at 2:51 PM, Sanjay Subramanian <Sa...@wizecommerce.com>> wrote:
Hi Programming Hive Book authors
Maybe a lot of u have already successfully implemented this but only these last two weeks , we implemented our aggregations using LZO compression in Hive - MR jobs creating LZO files as Input for Hive ---> Therafter Hive aggregations creating more LZO files as output.
As usual nothing was straight forward :-) Also the other challenge was to neatly tie all into actions in Oozie workflows….but after being underwater for weeks I think I am able to rise above water and breathe !
In the next version of the book , If u guys r planning to add detailed sections on using lzo compression in Hive , let me know…my experiences might be useful :-)
Thanks
sanjay
CONFIDENTIALITY NOTICE
======================
This email message and any attachments are for the exclusive use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message along with any attachments, from your computer system. If you are the intended recipient, please be advised that the content of this message is subject to access, review and disclosure by the sender's Email System Administrator.
CONFIDENTIALITY NOTICE
======================
This email message and any attachments are for the exclusive use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message along with any attachments, from your computer system. If you are the intended recipient, please be advised that the content of this message is subject to access, review and disclosure by the sender's Email System Administrator.
CONFIDENTIALITY NOTICE
======================
This email message and any attachments are for the exclusive use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message along with any attachments, from your computer system. If you are the intended recipient, please be advised that the content of this message is subject to access, review and disclosure by the sender's Email System Administrator.
Re: LZO compression implementation in Hive
Posted by Sanjay Subramanian <Sa...@wizecommerce.com>.
Sure…would love to add the LZO compression in Hive
Is there a specific page structure u want me to add to in confluence?
https://cwiki.apache.org/confluence<https://cwiki.apache.org/confluence/dashboard.action>
thanks
sanjay
From: Lefty Leverenz <le...@hortonworks.com>>
Reply-To: "user@hive.apache.org<ma...@hive.apache.org>" <us...@hive.apache.org>>
Date: Monday, June 17, 2013 7:08 PM
To: "user@hive.apache.org<ma...@hive.apache.org>" <us...@hive.apache.org>>
Subject: Re: LZO compression implementation in Hive
Perhaps you'd like to write up your insights in the Hive wiki, and others could add their insights. Then the information would be available to all, immediately.
– Lefty
On Mon, Jun 17, 2013 at 4:39 PM, Ramki Palle <ra...@gmail.com>> wrote:
Hi Sanjay,
Can you quickly give your insights on thip topic, if possible?
Regards,
Ramki.
On Mon, May 20, 2013 at 2:51 PM, Sanjay Subramanian <Sa...@wizecommerce.com>> wrote:
Hi Programming Hive Book authors
Maybe a lot of u have already successfully implemented this but only these last two weeks , we implemented our aggregations using LZO compression in Hive - MR jobs creating LZO files as Input for Hive ---> Therafter Hive aggregations creating more LZO files as output.
As usual nothing was straight forward :-) Also the other challenge was to neatly tie all into actions in Oozie workflows….but after being underwater for weeks I think I am able to rise above water and breathe !
In the next version of the book , If u guys r planning to add detailed sections on using lzo compression in Hive , let me know…my experiences might be useful :-)
Thanks
sanjay
CONFIDENTIALITY NOTICE
======================
This email message and any attachments are for the exclusive use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message along with any attachments, from your computer system. If you are the intended recipient, please be advised that the content of this message is subject to access, review and disclosure by the sender's Email System Administrator.
CONFIDENTIALITY NOTICE
======================
This email message and any attachments are for the exclusive use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message along with any attachments, from your computer system. If you are the intended recipient, please be advised that the content of this message is subject to access, review and disclosure by the sender's Email System Administrator.
Re: LZO compression implementation in Hive
Posted by Lefty Leverenz <le...@hortonworks.com>.
Perhaps you'd like to write up your insights in the Hive wiki, and others
could add their insights. Then the information would be available to all,
immediately.
– Lefty
On Mon, Jun 17, 2013 at 4:39 PM, Ramki Palle <ra...@gmail.com> wrote:
> Hi Sanjay,
>
> Can you quickly give your insights on thip topic, if possible?
>
> Regards,
> Ramki.
>
>
> On Mon, May 20, 2013 at 2:51 PM, Sanjay Subramanian <
> Sanjay.Subramanian@wizecommerce.com> wrote:
>
>> Hi Programming Hive Book authors
>>
>> Maybe a lot of u have already successfully implemented this but only
>> these last two weeks , we implemented our aggregations using LZO
>> compression in Hive - MR jobs creating LZO files as Input for Hive --->
>> Therafter Hive aggregations creating more LZO files as output.
>> As usual nothing was straight forward :-) Also the other challenge was
>> to neatly tie all into actions in Oozie workflows….but after being
>> underwater for weeks I think I am able to rise above water and breathe !
>>
>> In the next version of the book , If u guys r planning to add detailed
>> sections on using lzo compression in Hive , let me know…my experiences
>> might be useful :-)
>>
>> Thanks
>>
>> sanjay
>>
>>
>>
>> CONFIDENTIALITY NOTICE
>> ======================
>> This email message and any attachments are for the exclusive use of the
>> intended recipient(s) and may contain confidential and privileged
>> information. Any unauthorized review, use, disclosure or distribution is
>> prohibited. If you are not the intended recipient, please contact the
>> sender by reply email and destroy all copies of the original message along
>> with any attachments, from your computer system. If you are the intended
>> recipient, please be advised that the content of this message is subject to
>> access, review and disclosure by the sender's Email System Administrator.
>>
>
>
Re: LZO compression implementation in Hive
Posted by Ramki Palle <ra...@gmail.com>.
Hi Sanjay,
Can you quickly give your insights on thip topic, if possible?
Regards,
Ramki.
On Mon, May 20, 2013 at 2:51 PM, Sanjay Subramanian <
Sanjay.Subramanian@wizecommerce.com> wrote:
> Hi Programming Hive Book authors
>
> Maybe a lot of u have already successfully implemented this but only
> these last two weeks , we implemented our aggregations using LZO
> compression in Hive - MR jobs creating LZO files as Input for Hive --->
> Therafter Hive aggregations creating more LZO files as output.
> As usual nothing was straight forward :-) Also the other challenge was to
> neatly tie all into actions in Oozie workflows….but after being underwater
> for weeks I think I am able to rise above water and breathe !
>
> In the next version of the book , If u guys r planning to add detailed
> sections on using lzo compression in Hive , let me know…my experiences
> might be useful :-)
>
> Thanks
>
> sanjay
>
>
>
> CONFIDENTIALITY NOTICE
> ======================
> This email message and any attachments are for the exclusive use of the
> intended recipient(s) and may contain confidential and privileged
> information. Any unauthorized review, use, disclosure or distribution is
> prohibited. If you are not the intended recipient, please contact the
> sender by reply email and destroy all copies of the original message along
> with any attachments, from your computer system. If you are the intended
> recipient, please be advised that the content of this message is subject to
> access, review and disclosure by the sender's Email System Administrator.
>
LZO compression implementation in Hive
Posted by Sanjay Subramanian <Sa...@wizecommerce.com>.
Hi Programming Hive Book authors
Maybe a lot of u have already successfully implemented this but only these last two weeks , we implemented our aggregations using LZO compression in Hive - MR jobs creating LZO files as Input for Hive ---> Therafter Hive aggregations creating more LZO files as output.
As usual nothing was straight forward :-) Also the other challenge was to neatly tie all into actions in Oozie workflows….but after being underwater for weeks I think I am able to rise above water and breathe !
In the next version of the book , If u guys r planning to add detailed sections on using lzo compression in Hive , let me know…my experiences might be useful :-)
Thanks
sanjay
CONFIDENTIALITY NOTICE
======================
This email message and any attachments are for the exclusive use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message along with any attachments, from your computer system. If you are the intended recipient, please be advised that the content of this message is subject to access, review and disclosure by the sender's Email System Administrator.
LZO compression implementation in Hive
Posted by Sanjay Subramanian <Sa...@wizecommerce.com>.
Hi Programming Hive Book authors
Maybe a lot of u have already successfully implemented this but only these last two weeks , we implemented our aggregations using LZO compression in Hive - MR jobs creating LZO files as Input for Hive ---> Therafter Hive aggregations creating more LZO files as output.
As usual nothing was straight forward :-) Also the other challenge was to neatly tie all into actions in Oozie workflows….but after being underwater for weeks I think I am able to rise above water and breathe !
In the next version of the book , If u guys r planning to add detailed sections on using lzo compression in Hive , let me know…my experiences might be useful :-)
Thanks
sanjay
CONFIDENTIALITY NOTICE
======================
This email message and any attachments are for the exclusive use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message along with any attachments, from your computer system. If you are the intended recipient, please be advised that the content of this message is subject to access, review and disclosure by the sender's Email System Administrator.
LZO compression implementation in Hive
Posted by Sanjay Subramanian <Sa...@wizecommerce.com>.
Hi Programming Hive Book authors
Maybe a lot of u have already successfully implemented this but only these last two weeks , we implemented our aggregations using LZO compression in Hive - MR jobs creating LZO files as Input for Hive ---> Therafter Hive aggregations creating more LZO files as output.
As usual nothing was straight forward :-) Also the other challenge was to neatly tie all into actions in Oozie workflows….but after being underwater for weeks I think I am able to rise above water and breathe !
In the next version of the book , If u guys r planning to add detailed sections on using lzo compression in Hive , let me know…my experiences might be useful :-)
Thanks
sanjay
CONFIDENTIALITY NOTICE
======================
This email message and any attachments are for the exclusive use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message along with any attachments, from your computer system. If you are the intended recipient, please be advised that the content of this message is subject to access, review and disclosure by the sender's Email System Administrator.
LZO compression implementation in Hive
Posted by Sanjay Subramanian <Sa...@wizecommerce.com>.
Hi Programming Hive Book authors
Maybe a lot of u have already successfully implemented this but only these last two weeks , we implemented our aggregations using LZO compression in Hive - MR jobs creating LZO files as Input for Hive ---> Therafter Hive aggregations creating more LZO files as output.
As usual nothing was straight forward :-) Also the other challenge was to neatly tie all into actions in Oozie workflows….but after being underwater for weeks I think I am able to rise above water and breathe !
In the next version of the book , If u guys r planning to add detailed sections on using lzo compression in Hive , let me know…my experiences might be useful :-)
Thanks
sanjay
CONFIDENTIALITY NOTICE
======================
This email message and any attachments are for the exclusive use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message along with any attachments, from your computer system. If you are the intended recipient, please be advised that the content of this message is subject to access, review and disclosure by the sender's Email System Administrator.
LZO compression implementation in Hive
Posted by Sanjay Subramanian <Sa...@wizecommerce.com>.
Hi Programming Hive Book authors
Maybe a lot of u have already successfully implemented this but only these last two weeks , we implemented our aggregations using LZO compression in Hive - MR jobs creating LZO files as Input for Hive ---> Therafter Hive aggregations creating more LZO files as output.
As usual nothing was straight forward :-) Also the other challenge was to neatly tie all into actions in Oozie workflows….but after being underwater for weeks I think I am able to rise above water and breathe !
In the next version of the book , If u guys r planning to add detailed sections on using lzo compression in Hive , let me know…my experiences might be useful :-)
Thanks
sanjay
CONFIDENTIALITY NOTICE
======================
This email message and any attachments are for the exclusive use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message along with any attachments, from your computer system. If you are the intended recipient, please be advised that the content of this message is subject to access, review and disclosure by the sender's Email System Administrator.
Re: 答复: Unable to stop Thrift Server
Posted by Raj Hadoop <ha...@yahoo.com>.
Hi,
I had to do a kill -9. I am very surprised even 'Programming Hive' book has not given details on how to stop the thrift service. It just metioned on how to start.
Thanks,
Raj
________________________________
From: Jie Zhou (周杰) <ji...@allyes.com>
To: "user@hadoop.apache.org" <us...@hadoop.apache.org>; Raj Hadoop <ha...@yahoo.com>; Sanjay Subramanian <Sa...@wizecommerce.com>; "user@hive.apache.org" <us...@hive.apache.org>
Sent: Monday, May 20, 2013 5:23 PM
Subject: 答复: Unable to stop Thrift Server
The direct solution is that first find the pid of hive service and then just kill it.
netstat -anp |grep 10000
kill -9 pid
________________________________
发件人: Raj Hadoop [hadoopraj@yahoo.com]
发送时间: 2013年5月21日 5:17
收件人: Sanjay Subramanian; user@hive.apache.org; User
主题: Re: Unable to stop Thrift Server
Hi Sanjay,
I am using 0.9 version.
I do not have a sudo access. is there any other command to stop the service.
thanks,
raj
________________________________
From: Sanjay Subramanian <Sa...@wizecommerce.com>
To: "user@hive.apache.org" <us...@hive.apache.org>; Raj Hadoop <ha...@yahoo.com>; User <us...@hadoop.apache.org>
Sent: Monday, May 20, 2013 5:11 PM
Subject: Re: Unable to stop Thrift Server
Raj
Which version r u using ?
I think from 0.9+ onwards its best to use service to stop and start and NOT hive
sudo service hive-metastore stop
sudo service hive-server stop
sudo service hive-metastore start
sudo service hive-server start
Couple of general things that might help
1. Use linux screens : then u can start many screen sessions and u don't have to give the synch mode "&" of execution
Its very easy to manage several screen sessions and they keep running till your server restarts….and generally u can ssh to some jumhost and create your screen sessions there
2. Run the following
pstree -pulac | less
U can possible search for hive or your username or root which was used to start the service…and kill the process
sanjay
From: Raj Hadoop <ha...@yahoo.com>
Reply-To: "user@hive.apache.org" <us...@hive.apache.org>, Raj Hadoop <ha...@yahoo.com>
Date: Monday, May 20, 2013 2:03 PM
To: Hive <us...@hive.apache.org>, User <us...@hadoop.apache.org>
Subject: Unable to stop Thrift Server
Hi,
I was not able to stopThrift Server after performing the following steps.
$ bin/hive --service hiveserver &
Starting Hive Thrift Server
$ netstat -nl | grep 10000
tcp 0 0 :::10000 :::* LISTEN
I gave the following to stop. but not working.
hive --service hiveserver --action stop 10000
How can I stop this service?
Thanks,
Raj
CONFIDENTIALITY NOTICE
======================
This email message and any attachments are for the exclusive use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient,
please contact the sender by reply email and destroy all copies of the original message along with any attachments, from your computer system. If you are the intended recipient, please be advised that the content of this message is subject to access, review
and disclosure by the sender's Email System Administrator.
Re: 答复: Unable to stop Thrift Server
Posted by Raj Hadoop <ha...@yahoo.com>.
Hi,
I had to do a kill -9. I am very surprised even 'Programming Hive' book has not given details on how to stop the thrift service. It just metioned on how to start.
Thanks,
Raj
________________________________
From: Jie Zhou (周杰) <ji...@allyes.com>
To: "user@hadoop.apache.org" <us...@hadoop.apache.org>; Raj Hadoop <ha...@yahoo.com>; Sanjay Subramanian <Sa...@wizecommerce.com>; "user@hive.apache.org" <us...@hive.apache.org>
Sent: Monday, May 20, 2013 5:23 PM
Subject: 答复: Unable to stop Thrift Server
The direct solution is that first find the pid of hive service and then just kill it.
netstat -anp |grep 10000
kill -9 pid
________________________________
发件人: Raj Hadoop [hadoopraj@yahoo.com]
发送时间: 2013年5月21日 5:17
收件人: Sanjay Subramanian; user@hive.apache.org; User
主题: Re: Unable to stop Thrift Server
Hi Sanjay,
I am using 0.9 version.
I do not have a sudo access. is there any other command to stop the service.
thanks,
raj
________________________________
From: Sanjay Subramanian <Sa...@wizecommerce.com>
To: "user@hive.apache.org" <us...@hive.apache.org>; Raj Hadoop <ha...@yahoo.com>; User <us...@hadoop.apache.org>
Sent: Monday, May 20, 2013 5:11 PM
Subject: Re: Unable to stop Thrift Server
Raj
Which version r u using ?
I think from 0.9+ onwards its best to use service to stop and start and NOT hive
sudo service hive-metastore stop
sudo service hive-server stop
sudo service hive-metastore start
sudo service hive-server start
Couple of general things that might help
1. Use linux screens : then u can start many screen sessions and u don't have to give the synch mode "&" of execution
Its very easy to manage several screen sessions and they keep running till your server restarts….and generally u can ssh to some jumhost and create your screen sessions there
2. Run the following
pstree -pulac | less
U can possible search for hive or your username or root which was used to start the service…and kill the process
sanjay
From: Raj Hadoop <ha...@yahoo.com>
Reply-To: "user@hive.apache.org" <us...@hive.apache.org>, Raj Hadoop <ha...@yahoo.com>
Date: Monday, May 20, 2013 2:03 PM
To: Hive <us...@hive.apache.org>, User <us...@hadoop.apache.org>
Subject: Unable to stop Thrift Server
Hi,
I was not able to stopThrift Server after performing the following steps.
$ bin/hive --service hiveserver &
Starting Hive Thrift Server
$ netstat -nl | grep 10000
tcp 0 0 :::10000 :::* LISTEN
I gave the following to stop. but not working.
hive --service hiveserver --action stop 10000
How can I stop this service?
Thanks,
Raj
CONFIDENTIALITY NOTICE
======================
This email message and any attachments are for the exclusive use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient,
please contact the sender by reply email and destroy all copies of the original message along with any attachments, from your computer system. If you are the intended recipient, please be advised that the content of this message is subject to access, review
and disclosure by the sender's Email System Administrator.
Re: 答复: Unable to stop Thrift Server
Posted by Raj Hadoop <ha...@yahoo.com>.
Hi,
I had to do a kill -9. I am very surprised even 'Programming Hive' book has not given details on how to stop the thrift service. It just metioned on how to start.
Thanks,
Raj
________________________________
From: Jie Zhou (周杰) <ji...@allyes.com>
To: "user@hadoop.apache.org" <us...@hadoop.apache.org>; Raj Hadoop <ha...@yahoo.com>; Sanjay Subramanian <Sa...@wizecommerce.com>; "user@hive.apache.org" <us...@hive.apache.org>
Sent: Monday, May 20, 2013 5:23 PM
Subject: 答复: Unable to stop Thrift Server
The direct solution is that first find the pid of hive service and then just kill it.
netstat -anp |grep 10000
kill -9 pid
________________________________
发件人: Raj Hadoop [hadoopraj@yahoo.com]
发送时间: 2013年5月21日 5:17
收件人: Sanjay Subramanian; user@hive.apache.org; User
主题: Re: Unable to stop Thrift Server
Hi Sanjay,
I am using 0.9 version.
I do not have a sudo access. is there any other command to stop the service.
thanks,
raj
________________________________
From: Sanjay Subramanian <Sa...@wizecommerce.com>
To: "user@hive.apache.org" <us...@hive.apache.org>; Raj Hadoop <ha...@yahoo.com>; User <us...@hadoop.apache.org>
Sent: Monday, May 20, 2013 5:11 PM
Subject: Re: Unable to stop Thrift Server
Raj
Which version r u using ?
I think from 0.9+ onwards its best to use service to stop and start and NOT hive
sudo service hive-metastore stop
sudo service hive-server stop
sudo service hive-metastore start
sudo service hive-server start
Couple of general things that might help
1. Use linux screens : then u can start many screen sessions and u don't have to give the synch mode "&" of execution
Its very easy to manage several screen sessions and they keep running till your server restarts….and generally u can ssh to some jumhost and create your screen sessions there
2. Run the following
pstree -pulac | less
U can possible search for hive or your username or root which was used to start the service…and kill the process
sanjay
From: Raj Hadoop <ha...@yahoo.com>
Reply-To: "user@hive.apache.org" <us...@hive.apache.org>, Raj Hadoop <ha...@yahoo.com>
Date: Monday, May 20, 2013 2:03 PM
To: Hive <us...@hive.apache.org>, User <us...@hadoop.apache.org>
Subject: Unable to stop Thrift Server
Hi,
I was not able to stopThrift Server after performing the following steps.
$ bin/hive --service hiveserver &
Starting Hive Thrift Server
$ netstat -nl | grep 10000
tcp 0 0 :::10000 :::* LISTEN
I gave the following to stop. but not working.
hive --service hiveserver --action stop 10000
How can I stop this service?
Thanks,
Raj
CONFIDENTIALITY NOTICE
======================
This email message and any attachments are for the exclusive use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient,
please contact the sender by reply email and destroy all copies of the original message along with any attachments, from your computer system. If you are the intended recipient, please be advised that the content of this message is subject to access, review
and disclosure by the sender's Email System Administrator.
Re: 答复: Unable to stop Thrift Server
Posted by Raj Hadoop <ha...@yahoo.com>.
Hi,
I had to do a kill -9. I am very surprised even 'Programming Hive' book has not given details on how to stop the thrift service. It just metioned on how to start.
Thanks,
Raj
________________________________
From: Jie Zhou (周杰) <ji...@allyes.com>
To: "user@hadoop.apache.org" <us...@hadoop.apache.org>; Raj Hadoop <ha...@yahoo.com>; Sanjay Subramanian <Sa...@wizecommerce.com>; "user@hive.apache.org" <us...@hive.apache.org>
Sent: Monday, May 20, 2013 5:23 PM
Subject: 答复: Unable to stop Thrift Server
The direct solution is that first find the pid of hive service and then just kill it.
netstat -anp |grep 10000
kill -9 pid
________________________________
发件人: Raj Hadoop [hadoopraj@yahoo.com]
发送时间: 2013年5月21日 5:17
收件人: Sanjay Subramanian; user@hive.apache.org; User
主题: Re: Unable to stop Thrift Server
Hi Sanjay,
I am using 0.9 version.
I do not have a sudo access. is there any other command to stop the service.
thanks,
raj
________________________________
From: Sanjay Subramanian <Sa...@wizecommerce.com>
To: "user@hive.apache.org" <us...@hive.apache.org>; Raj Hadoop <ha...@yahoo.com>; User <us...@hadoop.apache.org>
Sent: Monday, May 20, 2013 5:11 PM
Subject: Re: Unable to stop Thrift Server
Raj
Which version r u using ?
I think from 0.9+ onwards its best to use service to stop and start and NOT hive
sudo service hive-metastore stop
sudo service hive-server stop
sudo service hive-metastore start
sudo service hive-server start
Couple of general things that might help
1. Use linux screens : then u can start many screen sessions and u don't have to give the synch mode "&" of execution
Its very easy to manage several screen sessions and they keep running till your server restarts….and generally u can ssh to some jumhost and create your screen sessions there
2. Run the following
pstree -pulac | less
U can possible search for hive or your username or root which was used to start the service…and kill the process
sanjay
From: Raj Hadoop <ha...@yahoo.com>
Reply-To: "user@hive.apache.org" <us...@hive.apache.org>, Raj Hadoop <ha...@yahoo.com>
Date: Monday, May 20, 2013 2:03 PM
To: Hive <us...@hive.apache.org>, User <us...@hadoop.apache.org>
Subject: Unable to stop Thrift Server
Hi,
I was not able to stopThrift Server after performing the following steps.
$ bin/hive --service hiveserver &
Starting Hive Thrift Server
$ netstat -nl | grep 10000
tcp 0 0 :::10000 :::* LISTEN
I gave the following to stop. but not working.
hive --service hiveserver --action stop 10000
How can I stop this service?
Thanks,
Raj
CONFIDENTIALITY NOTICE
======================
This email message and any attachments are for the exclusive use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient,
please contact the sender by reply email and destroy all copies of the original message along with any attachments, from your computer system. If you are the intended recipient, please be advised that the content of this message is subject to access, review
and disclosure by the sender's Email System Administrator.
Re: 答复: Unable to stop Thrift Server
Posted by Raj Hadoop <ha...@yahoo.com>.
Hi,
I had to do a kill -9. I am very surprised even 'Programming Hive' book has not given details on how to stop the thrift service. It just metioned on how to start.
Thanks,
Raj
________________________________
From: Jie Zhou (周杰) <ji...@allyes.com>
To: "user@hadoop.apache.org" <us...@hadoop.apache.org>; Raj Hadoop <ha...@yahoo.com>; Sanjay Subramanian <Sa...@wizecommerce.com>; "user@hive.apache.org" <us...@hive.apache.org>
Sent: Monday, May 20, 2013 5:23 PM
Subject: 答复: Unable to stop Thrift Server
The direct solution is that first find the pid of hive service and then just kill it.
netstat -anp |grep 10000
kill -9 pid
________________________________
发件人: Raj Hadoop [hadoopraj@yahoo.com]
发送时间: 2013年5月21日 5:17
收件人: Sanjay Subramanian; user@hive.apache.org; User
主题: Re: Unable to stop Thrift Server
Hi Sanjay,
I am using 0.9 version.
I do not have a sudo access. is there any other command to stop the service.
thanks,
raj
________________________________
From: Sanjay Subramanian <Sa...@wizecommerce.com>
To: "user@hive.apache.org" <us...@hive.apache.org>; Raj Hadoop <ha...@yahoo.com>; User <us...@hadoop.apache.org>
Sent: Monday, May 20, 2013 5:11 PM
Subject: Re: Unable to stop Thrift Server
Raj
Which version r u using ?
I think from 0.9+ onwards its best to use service to stop and start and NOT hive
sudo service hive-metastore stop
sudo service hive-server stop
sudo service hive-metastore start
sudo service hive-server start
Couple of general things that might help
1. Use linux screens : then u can start many screen sessions and u don't have to give the synch mode "&" of execution
Its very easy to manage several screen sessions and they keep running till your server restarts….and generally u can ssh to some jumhost and create your screen sessions there
2. Run the following
pstree -pulac | less
U can possible search for hive or your username or root which was used to start the service…and kill the process
sanjay
From: Raj Hadoop <ha...@yahoo.com>
Reply-To: "user@hive.apache.org" <us...@hive.apache.org>, Raj Hadoop <ha...@yahoo.com>
Date: Monday, May 20, 2013 2:03 PM
To: Hive <us...@hive.apache.org>, User <us...@hadoop.apache.org>
Subject: Unable to stop Thrift Server
Hi,
I was not able to stopThrift Server after performing the following steps.
$ bin/hive --service hiveserver &
Starting Hive Thrift Server
$ netstat -nl | grep 10000
tcp 0 0 :::10000 :::* LISTEN
I gave the following to stop. but not working.
hive --service hiveserver --action stop 10000
How can I stop this service?
Thanks,
Raj
CONFIDENTIALITY NOTICE
======================
This email message and any attachments are for the exclusive use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient,
please contact the sender by reply email and destroy all copies of the original message along with any attachments, from your computer system. If you are the intended recipient, please be advised that the content of this message is subject to access, review
and disclosure by the sender's Email System Administrator.