You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hive.apache.org by Suhail Doshi <su...@mixpanel.com> on 2009/03/15 21:22:47 UTC
Java/Python examples
I was browsing around the Hive wiki and couldn't manage to find any examples
on how people might be connecting to hive and executing queries through
java/python. Anyone have anything available? (Python preferable)
--
http://mixpanel.com
Blog: http://blog.mixpanel.com
Re: Java/Python examples
Posted by Edward Capriolo <ed...@gmail.com>.
Also have you looked at. Running the meta store as a stand alone service?
http://wiki.apache.org/hadoop/HiveDerbyServerMode?
Re: Java/Python examples
Posted by Suhail Doshi <di...@gmail.com>.
Nevermind, seems to be working now by placing it in the same dir, I guess my
query was just bad.
Suhail
On Mon, Mar 23, 2009 at 12:08 PM, Suhail Doshi <di...@gmail.com>wrote:
> I understand why the metastore_db directory is generated everywhere you
> execute ./hive but why is that useful? It starts generate tons of
> metastore_db's if you put hive in your path. It just seems really messy.
>
> The steps I take are:
> start-all.sh (to start hadoop)
> then I navigate to the build/dist/bin directory and do: nohup ./hive
> --service hiveserver &
> Then I navigate to my test client in ~/test/hive_test.py and run it and it
> generates an exception
>
> It seems a bit frustrating to have to be in the exact place where the
> metastore_db is generated in order to interact with your table...
>
> Either way, how do I get my python client to use the metastore_db in its
> own directory so it's not using some other one.
>
> Edward, I'll take a look at setting up derby later if this all fails.
>
>
> On Mon, Mar 23, 2009 at 11:13 AM, Raghu Murthy <ra...@facebook.com> wrote:
>
>> Looks like you are trying to run the server and tests simultaneously as
>> per
>> the error:
>> > Caused by: ERROR XSDB6: Another instance of Derby may have already
>> booted the
>> > database /home/hadoop/hive/build/dist/conf/metastore_db.
>>
>> Can you make sure nothing other than the server is running? Or, point the
>> server to a different metastore directory by copying over the dist
>> directory
>> to a different directory (after a fresh 'ant clean package') and starting
>> the server from there.
>>
>>
>>
>>
>> On 3/23/09 11:04 AM, "Suhail Doshi" <di...@gmail.com> wrote:
>>
>> > Has anyone had problems when you start the hiveserver you simply can't
>> execute
>> > queries because of the following error:
>> > Caused by: ERROR XSDB6: Another instance of Derby may have already
>> booted the
>> > database /home/hadoop/hive/build/dist/conf/metastore_db.
>> >
>> > li57-125 ~/hive/build/dist/conf/metastore_db: hive
>> > Hive history
>> file=/tmp/hadoop/hive_job_log_hadoop_200903231803_720530131.txt
>> > hive> show tables;
>> > FAILED: Error in metadata: javax.jdo.JDOFatalDataStoreException: Failed
>> to
>> > start database '/home/hadoop/hive/build/dist/conf/metastore_db', see the
>> next
>> > exception for details.
>> > NestedThrowables:
>> > java.sql.SQLException: Failed to start database
>> > '/home/hadoop/hive/build/dist/conf/metastore_db', see the next exception
>> for
>> > details.
>> > FAILED: Execution Error, return code 1 from
>> > org.apache.hadoop.hive.ql.exec.DDLTask
>> > Time taken: 1.018 seconds
>> >
>> > Not sure what to do, is my metastore_db corrupted? Anyway to start from
>> > scratch or a fix?
>> >
>> > Suhail
>> >
>> > On Sun, Mar 22, 2009 at 4:38 PM, Suhail Doshi <digitalwarfare@gmail.com
>> >
>> > wrote:
>> >> Hi Ragu,
>> >>
>> >> Thanks for the help thus far, seem to have one last problem:
>> >>
>> >>
>> >> from thrift import Thrift
>> >> from thrift.transport import TSocket
>> >> from thrift.transport import TTransport
>> >> from thrift.protocol import TBinaryProtocol
>> >>
>> >> from hive_service import ThriftHive
>> >>
>> >>
>> >> transport = TSocket.TSocket('localhost', 9090)
>> >> transport = TTransport.TBufferedTransport(transport)
>> >> protocol = TBinaryProtocol.TBinaryProtocol(transport)
>> >>
>> >> client = ThriftHive.Client(protocol)
>> >>
>> >> Do you know the correct way to execute a query? I've tried:
>> >>
>> >>>>>>>> client.execute("select * from page_views")
>> >>
>> >> However received the following error:
>> >>
>> >>
>> >> Traceback (most recent call last):
>> >> File "<stdin>", line 1, in <module>
>> >> File
>> "/home/hadoop/hive/build/dist/lib/py/hive_service/ThriftHive.py", line
>> >> 41, in execute
>> >> self.send_execute(query)
>> >> File
>> "/home/hadoop/hive/build/dist/lib/py/hive_service/ThriftHive.py", line
>> >> 50, in send_execute
>> >> self._oprot.trans.flush()
>> >> File
>> "/usr/lib/python2.5/site-packages/thrift/transport/TTransport.py",
>> >> line
>> >> 152, in flush
>> >> self.__trans.write(out)
>> >> File "/usr/lib/python2.5/site-packages/thrift/transport/TSocket.py",
>> line
>> >> 88, in write
>> >> plus = self.handle.send(buff)
>> >> AttributeError: 'NoneType' object has no attribute 'send'
>> >>
>> >>
>> >>
>> >> On Sat, Mar 21, 2009 at 6:38 PM, Raghu Murthy <ra...@facebook.com>
>> wrote:
>> >>> The hive module is in hive_service.
>> >>>
>> >>> Try from hive_service import ThriftHive
>> >>>
>> >>>
>> >>> On 3/21/09 5:20 PM, "Suhail Doshi" <di...@gmail.com> wrote:
>> >>>
>> >>>>> Actually the hive generate py files for thrift don't appear in that
>> >>>>> dir...should I just make a symlink?
>> >>>>>
>> >>>>> On Sat, Mar 21, 2009 at 5:04 PM, Raghu Murthy <ra...@facebook.com>
>> wrote:
>> >>>>>>> Use build/dist/lib/py instead of gen-py. All modules are copied to
>> that
>> >>>>>>> directory once you run ant package.
>> >>>>>>>
>> >>>>>>>
>> >>>>>>> On 3/21/09 2:17 PM, "Suhail Doshi" <di...@gmail.com>
>> wrote:
>> >>>>>>>
>> >>>>>>>>>>> I seem to be getting the following error:
>> >>>>>>>>>>>
>> >>>>>>>>>>> Traceback (most recent call last):
>> >>>>>>>>>>> File "hive_test.py", line 13, in <module>
>> >>>>>>>>>>> from hive import ThriftHive
>> >>>>>>>>>>> File "ThriftHive.py", line 8, in <module>
>> >>>>>>>>>>> ImportError: No module named
>> hive_metastore.ThriftHiveMetastore
>> >>>>>>>>>>>
>> >>>>>>>>>>> There's a symlink gen-py ->
>> /home/hadoop/hive/service/src/gen-py in
>> >>>>>>>>>>> the
>> >>>>>>>>> folder
>> >>>>>>>>>>> with the py script.
>> >>>>>>>>>>>
>> >>>>>>>>>>> With the following code:
>> >>>>>>>>>>>
>> >>>>>>>>>>> #!/usr/bin/evn python
>> >>>>>>>>>>> import sys
>> >>>>>>>>>>> import os
>> >>>>>>>>>>>
>> >>>>>>>>>>> dir = os.path.dirname(os.path.realpath(sys.argv[0]))
>> >>>>>>>>>>> sys.path.insert(0, dir + '/gen-py')
>> >>>>>>>>>>>
>> >>>>>>>>>>> from thrift import Thrift
>> >>>>>>>>>>> from thrift.transport import TSocket
>> >>>>>>>>>>> from thrift.transport import TTransport
>> >>>>>>>>>>> from thrift.protocol import TBinaryProtocol
>> >>>>>>>>>>>
>> >>>>>>>>>>> from hive import ThriftHive
>> >>>>>>>>>>>
>> >>>>>>>>>>> transport = TSocket.TSocket('localhost', 9090)
>> >>>>>>>>>>> transport = TTransport.TBufferedTransport(transport)
>> >>>>>>>>>>> protocol = TBinaryProtocol.TBinaryProtocol(transport)
>> >>>>>>>>>>>
>> >>>>>>>>>>> client = ThriftHive.Client(protocol)
>> >>>>>>>>>>>
>> >>>>>>>>>>> print client
>> >>>>>>>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>> On Fri, Mar 20, 2009 at 4:06 PM, Suhail Doshi
>> >>>>>>> <di...@gmail.com>
>> >>>>>>>>>>> wrote:
>> >>>>>>>>>>>> Whoops, silly mistake, nevermind =)
>> >>>>>>>>>>>>
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> On Fri, Mar 20, 2009 at 3:55 PM, Suhail Doshi
>> >>>>>>>>> <di...@gmail.com>
>> >>>>>>>>>>>> wrote:
>> >>>>>>>>>>>> Namit,
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> For some reason when I apply:
>> >>>>>>>>>>>> li57-125 ~/hive: patch -p0 hive-py.patch
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> It seems just hang for a while.
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> Suhail
>> >>>>>>>>>>>>
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> On Fri, Mar 20, 2009 at 12:58 PM, Namit Jain
>> >>>>>>>>>>> <nj...@facebook.com>
>> >>>>>>>>>>>> wrote:
>> >>>>>>>>>>>> Download the patch file <patch_file>
>> >>>>>>>>>>>>
>> >>>>>>>>>>>>
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> Go to the truck directory, and perform
>> >>>>>>>>>>>>
>> >>>>>>>>>>>>
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> patch -p0 < <patch_file>
>> >>>>>>>>>>>>
>> >>>>>>>>>>>>
>> >>>>>>>>>>>>
>> >>>>>>>>>>>>
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> Then compile and run tests.
>> >>>>>>>>>>>>
>> >>>>>>>>>>>>
>> >>>>>>>>>>>>
>> >>>>>>>>>>>>
>> >>>>>>>>>>>>
>> >>>>>>>>>>>>
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> From: Suhail Doshi [mailto:digitalwarfare@gmail.com]
>> >>>>>>>>>>>> Sent: Friday, March 20, 2009 12:52 PM
>> >>>>>>>>>>>> To: hive-user@hadoop.apache.org
>> >>>>>>>>>>>> Subject: Re: Java/Python examples
>> >>>>>>>>>>>>
>> >>>>>>>>>>>>
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> Ragu,
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> How is exactly do you apply the patches?
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> Suhail
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> On Sun, Mar 15, 2009 at 3:57 PM, Raghu Murthy
>> >>>>>>>>>>>> <ra...@facebook.com>
>> >>>>>>>>>>>> wrote:
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> You can call Hive in Java either directly or in
>> >>>>>>>>>>>> client-server mode
>> >>>>>>>>>>>> where
>> >>>>>>>>>>>> you
>> >>>>>>>>>>>> would have to run a thrift server.
>> >>>>>>>>>>>> Python and php clients are currently only available in
>> patches at
>> >>>>>>>>>>>> https://issues.apache.org/jira/browse/HIVE-343 and
>> >>>>>>>>>>>> https://issues.apache.org/jira/browse/HIVE-299
>> >>>>>>>>>>>> respectively.
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> If you apply the patches and then run ant package, the
>> >>>>>>>>>>>> documentation
>> >>>>>>>>>>>> at
>> >>>>>>>>>>>> http://wiki.apache.org/hadoop/Hive/HiveServer should be
>> >>>>>>>>>>>> >>>>>>>>>>>> useful. Hope
>> >>>>>>>>>>>> this
>> >>>>>>>>>>>> helps.
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> raghu
>> >>>>>>>>>>>>
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> On 3/15/09 12:22 PM, "Suhail Doshi"
>> >>>>>>>>>>>> <su...@mixpanel.com> wrote:
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> I was browsing around the Hive wiki and couldn't
>> >>>>>>>>>>>> manage to find any
>> >>>>>>>>>>>> examples
>> >>>>>>>>>>>> on how people might be connecting to hive and
>> >>>>>>>>>>>> executing queries
>> >>>>>>>>>>>> through
>> >>>>>>>>>>>> java/python. Anyone have anything available? (Python
>> preferable)
>> >>>>>>>>>>>>
>> >>>>>>>>>>>>
>> >>>>>>>>>>>>
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> --
>> >>>>>>>>>>>> http://mixpanel.com
>> >>>>>>>>>>>> Blog: http://blog.mixpanel.com
>> >>>>>>>>>>>>
>> >>>>>>>>>>>>
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> --
>> >>>>>>>>>>>> http://mixpanel.com
>> >>>>>>>>>>>> Blog: http://blog.mixpanel.com
>> >>>>>>>>>>>>
>> >>>>>>>>>>>>
>> >>>>>>>>>>>>
>> >>>>>>>>>>>> --
>> >>>>>>>>>>>> http://mixpanel.com
>> >>>>>>>>>>>> Blog: http://blog.mixpanel.com
>> >>>>>>>>>>>
>> >>>>>>>>>>>
>> >>>>>>>
>> >>>>>
>> >>>>>
>> >>>
>> >>
>> >>
>> >>
>> >> --
>> >> http://mixpanel.com
>> >> Blog: http://blog.mixpanel.com
>> >
>> >
>>
>>
>
>
> --
> http://mixpanel.com
> Blog: http://blog.mixpanel.com
>
--
http://mixpanel.com
Blog: http://blog.mixpanel.com
Re: Java/Python examples
Posted by Suhail Doshi <di...@gmail.com>.
I understand why the metastore_db directory is generated everywhere you
execute ./hive but why is that useful? It starts generate tons of
metastore_db's if you put hive in your path. It just seems really messy.
The steps I take are:
start-all.sh (to start hadoop)
then I navigate to the build/dist/bin directory and do: nohup ./hive
--service hiveserver &
Then I navigate to my test client in ~/test/hive_test.py and run it and it
generates an exception
It seems a bit frustrating to have to be in the exact place where the
metastore_db is generated in order to interact with your table...
Either way, how do I get my python client to use the metastore_db in its own
directory so it's not using some other one.
Edward, I'll take a look at setting up derby later if this all fails.
On Mon, Mar 23, 2009 at 11:13 AM, Raghu Murthy <ra...@facebook.com> wrote:
> Looks like you are trying to run the server and tests simultaneously as per
> the error:
> > Caused by: ERROR XSDB6: Another instance of Derby may have already booted
> the
> > database /home/hadoop/hive/build/dist/conf/metastore_db.
>
> Can you make sure nothing other than the server is running? Or, point the
> server to a different metastore directory by copying over the dist
> directory
> to a different directory (after a fresh 'ant clean package') and starting
> the server from there.
>
>
>
>
> On 3/23/09 11:04 AM, "Suhail Doshi" <di...@gmail.com> wrote:
>
> > Has anyone had problems when you start the hiveserver you simply can't
> execute
> > queries because of the following error:
> > Caused by: ERROR XSDB6: Another instance of Derby may have already booted
> the
> > database /home/hadoop/hive/build/dist/conf/metastore_db.
> >
> > li57-125 ~/hive/build/dist/conf/metastore_db: hive
> > Hive history
> file=/tmp/hadoop/hive_job_log_hadoop_200903231803_720530131.txt
> > hive> show tables;
> > FAILED: Error in metadata: javax.jdo.JDOFatalDataStoreException: Failed
> to
> > start database '/home/hadoop/hive/build/dist/conf/metastore_db', see the
> next
> > exception for details.
> > NestedThrowables:
> > java.sql.SQLException: Failed to start database
> > '/home/hadoop/hive/build/dist/conf/metastore_db', see the next exception
> for
> > details.
> > FAILED: Execution Error, return code 1 from
> > org.apache.hadoop.hive.ql.exec.DDLTask
> > Time taken: 1.018 seconds
> >
> > Not sure what to do, is my metastore_db corrupted? Anyway to start from
> > scratch or a fix?
> >
> > Suhail
> >
> > On Sun, Mar 22, 2009 at 4:38 PM, Suhail Doshi <di...@gmail.com>
> > wrote:
> >> Hi Ragu,
> >>
> >> Thanks for the help thus far, seem to have one last problem:
> >>
> >>
> >> from thrift import Thrift
> >> from thrift.transport import TSocket
> >> from thrift.transport import TTransport
> >> from thrift.protocol import TBinaryProtocol
> >>
> >> from hive_service import ThriftHive
> >>
> >>
> >> transport = TSocket.TSocket('localhost', 9090)
> >> transport = TTransport.TBufferedTransport(transport)
> >> protocol = TBinaryProtocol.TBinaryProtocol(transport)
> >>
> >> client = ThriftHive.Client(protocol)
> >>
> >> Do you know the correct way to execute a query? I've tried:
> >>
> >>>>>>>> client.execute("select * from page_views")
> >>
> >> However received the following error:
> >>
> >>
> >> Traceback (most recent call last):
> >> File "<stdin>", line 1, in <module>
> >> File "/home/hadoop/hive/build/dist/lib/py/hive_service/ThriftHive.py",
> line
> >> 41, in execute
> >> self.send_execute(query)
> >> File "/home/hadoop/hive/build/dist/lib/py/hive_service/ThriftHive.py",
> line
> >> 50, in send_execute
> >> self._oprot.trans.flush()
> >> File
> "/usr/lib/python2.5/site-packages/thrift/transport/TTransport.py",
> >> line
> >> 152, in flush
> >> self.__trans.write(out)
> >> File "/usr/lib/python2.5/site-packages/thrift/transport/TSocket.py",
> line
> >> 88, in write
> >> plus = self.handle.send(buff)
> >> AttributeError: 'NoneType' object has no attribute 'send'
> >>
> >>
> >>
> >> On Sat, Mar 21, 2009 at 6:38 PM, Raghu Murthy <ra...@facebook.com>
> wrote:
> >>> The hive module is in hive_service.
> >>>
> >>> Try from hive_service import ThriftHive
> >>>
> >>>
> >>> On 3/21/09 5:20 PM, "Suhail Doshi" <di...@gmail.com> wrote:
> >>>
> >>>>> Actually the hive generate py files for thrift don't appear in that
> >>>>> dir...should I just make a symlink?
> >>>>>
> >>>>> On Sat, Mar 21, 2009 at 5:04 PM, Raghu Murthy <ra...@facebook.com>
> wrote:
> >>>>>>> Use build/dist/lib/py instead of gen-py. All modules are copied to
> that
> >>>>>>> directory once you run ant package.
> >>>>>>>
> >>>>>>>
> >>>>>>> On 3/21/09 2:17 PM, "Suhail Doshi" <di...@gmail.com>
> wrote:
> >>>>>>>
> >>>>>>>>>>> I seem to be getting the following error:
> >>>>>>>>>>>
> >>>>>>>>>>> Traceback (most recent call last):
> >>>>>>>>>>> File "hive_test.py", line 13, in <module>
> >>>>>>>>>>> from hive import ThriftHive
> >>>>>>>>>>> File "ThriftHive.py", line 8, in <module>
> >>>>>>>>>>> ImportError: No module named hive_metastore.ThriftHiveMetastore
> >>>>>>>>>>>
> >>>>>>>>>>> There's a symlink gen-py ->
> /home/hadoop/hive/service/src/gen-py in
> >>>>>>>>>>> the
> >>>>>>>>> folder
> >>>>>>>>>>> with the py script.
> >>>>>>>>>>>
> >>>>>>>>>>> With the following code:
> >>>>>>>>>>>
> >>>>>>>>>>> #!/usr/bin/evn python
> >>>>>>>>>>> import sys
> >>>>>>>>>>> import os
> >>>>>>>>>>>
> >>>>>>>>>>> dir = os.path.dirname(os.path.realpath(sys.argv[0]))
> >>>>>>>>>>> sys.path.insert(0, dir + '/gen-py')
> >>>>>>>>>>>
> >>>>>>>>>>> from thrift import Thrift
> >>>>>>>>>>> from thrift.transport import TSocket
> >>>>>>>>>>> from thrift.transport import TTransport
> >>>>>>>>>>> from thrift.protocol import TBinaryProtocol
> >>>>>>>>>>>
> >>>>>>>>>>> from hive import ThriftHive
> >>>>>>>>>>>
> >>>>>>>>>>> transport = TSocket.TSocket('localhost', 9090)
> >>>>>>>>>>> transport = TTransport.TBufferedTransport(transport)
> >>>>>>>>>>> protocol = TBinaryProtocol.TBinaryProtocol(transport)
> >>>>>>>>>>>
> >>>>>>>>>>> client = ThriftHive.Client(protocol)
> >>>>>>>>>>>
> >>>>>>>>>>> print client
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> On Fri, Mar 20, 2009 at 4:06 PM, Suhail Doshi
> >>>>>>> <di...@gmail.com>
> >>>>>>>>>>> wrote:
> >>>>>>>>>>>> Whoops, silly mistake, nevermind =)
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> On Fri, Mar 20, 2009 at 3:55 PM, Suhail Doshi
> >>>>>>>>> <di...@gmail.com>
> >>>>>>>>>>>> wrote:
> >>>>>>>>>>>> Namit,
> >>>>>>>>>>>>
> >>>>>>>>>>>> For some reason when I apply:
> >>>>>>>>>>>> li57-125 ~/hive: patch -p0 hive-py.patch
> >>>>>>>>>>>>
> >>>>>>>>>>>> It seems just hang for a while.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Suhail
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> On Fri, Mar 20, 2009 at 12:58 PM, Namit Jain
> >>>>>>>>>>> <nj...@facebook.com>
> >>>>>>>>>>>> wrote:
> >>>>>>>>>>>> Download the patch file <patch_file>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> Go to the truck directory, and perform
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> patch -p0 < <patch_file>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> Then compile and run tests.
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> From: Suhail Doshi [mailto:digitalwarfare@gmail.com]
> >>>>>>>>>>>> Sent: Friday, March 20, 2009 12:52 PM
> >>>>>>>>>>>> To: hive-user@hadoop.apache.org
> >>>>>>>>>>>> Subject: Re: Java/Python examples
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> Ragu,
> >>>>>>>>>>>>
> >>>>>>>>>>>> How is exactly do you apply the patches?
> >>>>>>>>>>>>
> >>>>>>>>>>>> Suhail
> >>>>>>>>>>>>
> >>>>>>>>>>>> On Sun, Mar 15, 2009 at 3:57 PM, Raghu Murthy
> >>>>>>>>>>>> <ra...@facebook.com>
> >>>>>>>>>>>> wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>> You can call Hive in Java either directly or in
> >>>>>>>>>>>> client-server mode
> >>>>>>>>>>>> where
> >>>>>>>>>>>> you
> >>>>>>>>>>>> would have to run a thrift server.
> >>>>>>>>>>>> Python and php clients are currently only available in patches
> at
> >>>>>>>>>>>> https://issues.apache.org/jira/browse/HIVE-343 and
> >>>>>>>>>>>> https://issues.apache.org/jira/browse/HIVE-299
> >>>>>>>>>>>> respectively.
> >>>>>>>>>>>>
> >>>>>>>>>>>> If you apply the patches and then run ant package, the
> >>>>>>>>>>>> documentation
> >>>>>>>>>>>> at
> >>>>>>>>>>>> http://wiki.apache.org/hadoop/Hive/HiveServer should be
> >>>>>>>>>>>> >>>>>>>>>>>> useful. Hope
> >>>>>>>>>>>> this
> >>>>>>>>>>>> helps.
> >>>>>>>>>>>>
> >>>>>>>>>>>> raghu
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> On 3/15/09 12:22 PM, "Suhail Doshi"
> >>>>>>>>>>>> <su...@mixpanel.com> wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>> I was browsing around the Hive wiki and couldn't
> >>>>>>>>>>>> manage to find any
> >>>>>>>>>>>> examples
> >>>>>>>>>>>> on how people might be connecting to hive and
> >>>>>>>>>>>> executing queries
> >>>>>>>>>>>> through
> >>>>>>>>>>>> java/python. Anyone have anything available? (Python
> preferable)
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> --
> >>>>>>>>>>>> http://mixpanel.com
> >>>>>>>>>>>> Blog: http://blog.mixpanel.com
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> --
> >>>>>>>>>>>> http://mixpanel.com
> >>>>>>>>>>>> Blog: http://blog.mixpanel.com
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> --
> >>>>>>>>>>>> http://mixpanel.com
> >>>>>>>>>>>> Blog: http://blog.mixpanel.com
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>
> >>>>>
> >>>>>
> >>>
> >>
> >>
> >>
> >> --
> >> http://mixpanel.com
> >> Blog: http://blog.mixpanel.com
> >
> >
>
>
--
http://mixpanel.com
Blog: http://blog.mixpanel.com
Re: Java/Python examples
Posted by Raghu Murthy <ra...@facebook.com>.
Looks like you are trying to run the server and tests simultaneously as per
the error:
> Caused by: ERROR XSDB6: Another instance of Derby may have already booted the
> database /home/hadoop/hive/build/dist/conf/metastore_db.
Can you make sure nothing other than the server is running? Or, point the
server to a different metastore directory by copying over the dist directory
to a different directory (after a fresh 'ant clean package') and starting
the server from there.
On 3/23/09 11:04 AM, "Suhail Doshi" <di...@gmail.com> wrote:
> Has anyone had problems when you start the hiveserver you simply can't execute
> queries because of the following error:
> Caused by: ERROR XSDB6: Another instance of Derby may have already booted the
> database /home/hadoop/hive/build/dist/conf/metastore_db.
>
> li57-125 ~/hive/build/dist/conf/metastore_db: hive
> Hive history file=/tmp/hadoop/hive_job_log_hadoop_200903231803_720530131.txt
> hive> show tables;
> FAILED: Error in metadata: javax.jdo.JDOFatalDataStoreException: Failed to
> start database '/home/hadoop/hive/build/dist/conf/metastore_db', see the next
> exception for details.
> NestedThrowables:
> java.sql.SQLException: Failed to start database
> '/home/hadoop/hive/build/dist/conf/metastore_db', see the next exception for
> details.
> FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.DDLTask
> Time taken: 1.018 seconds
>
> Not sure what to do, is my metastore_db corrupted? Anyway to start from
> scratch or a fix?
>
> Suhail
>
> On Sun, Mar 22, 2009 at 4:38 PM, Suhail Doshi <di...@gmail.com>
> wrote:
>> Hi Ragu,
>>
>> Thanks for the help thus far, seem to have one last problem:
>>
>>
>> from thrift import Thrift
>> from thrift.transport import TSocket
>> from thrift.transport import TTransport
>> from thrift.protocol import TBinaryProtocol
>>
>> from hive_service import ThriftHive
>>
>>
>> transport = TSocket.TSocket('localhost', 9090)
>> transport = TTransport.TBufferedTransport(transport)
>> protocol = TBinaryProtocol.TBinaryProtocol(transport)
>>
>> client = ThriftHive.Client(protocol)
>>
>> Do you know the correct way to execute a query? I've tried:
>>
>>>>>>>> client.execute("select * from page_views")
>>
>> However received the following error:
>>
>>
>> Traceback (most recent call last):
>> File "<stdin>", line 1, in <module>
>> File "/home/hadoop/hive/build/dist/lib/py/hive_service/ThriftHive.py", line
>> 41, in execute
>> self.send_execute(query)
>> File "/home/hadoop/hive/build/dist/lib/py/hive_service/ThriftHive.py", line
>> 50, in send_execute
>> self._oprot.trans.flush()
>> File "/usr/lib/python2.5/site-packages/thrift/transport/TTransport.py",
>> line
>> 152, in flush
>> self.__trans.write(out)
>> File "/usr/lib/python2.5/site-packages/thrift/transport/TSocket.py", line
>> 88, in write
>> plus = self.handle.send(buff)
>> AttributeError: 'NoneType' object has no attribute 'send'
>>
>>
>>
>> On Sat, Mar 21, 2009 at 6:38 PM, Raghu Murthy <ra...@facebook.com> wrote:
>>> The hive module is in hive_service.
>>>
>>> Try from hive_service import ThriftHive
>>>
>>>
>>> On 3/21/09 5:20 PM, "Suhail Doshi" <di...@gmail.com> wrote:
>>>
>>>>> Actually the hive generate py files for thrift don't appear in that
>>>>> dir...should I just make a symlink?
>>>>>
>>>>> On Sat, Mar 21, 2009 at 5:04 PM, Raghu Murthy <ra...@facebook.com> wrote:
>>>>>>> Use build/dist/lib/py instead of gen-py. All modules are copied to that
>>>>>>> directory once you run ant package.
>>>>>>>
>>>>>>>
>>>>>>> On 3/21/09 2:17 PM, "Suhail Doshi" <di...@gmail.com> wrote:
>>>>>>>
>>>>>>>>>>> I seem to be getting the following error:
>>>>>>>>>>>
>>>>>>>>>>> Traceback (most recent call last):
>>>>>>>>>>> File "hive_test.py", line 13, in <module>
>>>>>>>>>>> from hive import ThriftHive
>>>>>>>>>>> File "ThriftHive.py", line 8, in <module>
>>>>>>>>>>> ImportError: No module named hive_metastore.ThriftHiveMetastore
>>>>>>>>>>>
>>>>>>>>>>> There's a symlink gen-py -> /home/hadoop/hive/service/src/gen-py in
>>>>>>>>>>> the
>>>>>>>>> folder
>>>>>>>>>>> with the py script.
>>>>>>>>>>>
>>>>>>>>>>> With the following code:
>>>>>>>>>>>
>>>>>>>>>>> #!/usr/bin/evn python
>>>>>>>>>>> import sys
>>>>>>>>>>> import os
>>>>>>>>>>>
>>>>>>>>>>> dir = os.path.dirname(os.path.realpath(sys.argv[0]))
>>>>>>>>>>> sys.path.insert(0, dir + '/gen-py')
>>>>>>>>>>>
>>>>>>>>>>> from thrift import Thrift
>>>>>>>>>>> from thrift.transport import TSocket
>>>>>>>>>>> from thrift.transport import TTransport
>>>>>>>>>>> from thrift.protocol import TBinaryProtocol
>>>>>>>>>>>
>>>>>>>>>>> from hive import ThriftHive
>>>>>>>>>>>
>>>>>>>>>>> transport = TSocket.TSocket('localhost', 9090)
>>>>>>>>>>> transport = TTransport.TBufferedTransport(transport)
>>>>>>>>>>> protocol = TBinaryProtocol.TBinaryProtocol(transport)
>>>>>>>>>>>
>>>>>>>>>>> client = ThriftHive.Client(protocol)
>>>>>>>>>>>
>>>>>>>>>>> print client
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Fri, Mar 20, 2009 at 4:06 PM, Suhail Doshi
>>>>>>> <di...@gmail.com>
>>>>>>>>>>> wrote:
>>>>>>>>>>>> Whoops, silly mistake, nevermind =)
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Fri, Mar 20, 2009 at 3:55 PM, Suhail Doshi
>>>>>>>>> <di...@gmail.com>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>> Namit,
>>>>>>>>>>>>
>>>>>>>>>>>> For some reason when I apply:
>>>>>>>>>>>> li57-125 ~/hive: patch -p0 hive-py.patch
>>>>>>>>>>>>
>>>>>>>>>>>> It seems just hang for a while.
>>>>>>>>>>>>
>>>>>>>>>>>> Suhail
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Fri, Mar 20, 2009 at 12:58 PM, Namit Jain
>>>>>>>>>>> <nj...@facebook.com>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>> Download the patch file <patch_file>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Go to the truck directory, and perform
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> patch -p0 < <patch_file>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Then compile and run tests.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> From: Suhail Doshi [mailto:digitalwarfare@gmail.com]
>>>>>>>>>>>> Sent: Friday, March 20, 2009 12:52 PM
>>>>>>>>>>>> To: hive-user@hadoop.apache.org
>>>>>>>>>>>> Subject: Re: Java/Python examples
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Ragu,
>>>>>>>>>>>>
>>>>>>>>>>>> How is exactly do you apply the patches?
>>>>>>>>>>>>
>>>>>>>>>>>> Suhail
>>>>>>>>>>>>
>>>>>>>>>>>> On Sun, Mar 15, 2009 at 3:57 PM, Raghu Murthy
>>>>>>>>>>>> <ra...@facebook.com>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> You can call Hive in Java either directly or in
>>>>>>>>>>>> client-server mode
>>>>>>>>>>>> where
>>>>>>>>>>>> you
>>>>>>>>>>>> would have to run a thrift server.
>>>>>>>>>>>> Python and php clients are currently only available in patches at
>>>>>>>>>>>> https://issues.apache.org/jira/browse/HIVE-343 and
>>>>>>>>>>>> https://issues.apache.org/jira/browse/HIVE-299
>>>>>>>>>>>> respectively.
>>>>>>>>>>>>
>>>>>>>>>>>> If you apply the patches and then run ant package, the
>>>>>>>>>>>> documentation
>>>>>>>>>>>> at
>>>>>>>>>>>> http://wiki.apache.org/hadoop/Hive/HiveServer should be
>>>>>>>>>>>> >>>>>>>>>>>> useful. Hope
>>>>>>>>>>>> this
>>>>>>>>>>>> helps.
>>>>>>>>>>>>
>>>>>>>>>>>> raghu
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On 3/15/09 12:22 PM, "Suhail Doshi"
>>>>>>>>>>>> <su...@mixpanel.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> I was browsing around the Hive wiki and couldn't
>>>>>>>>>>>> manage to find any
>>>>>>>>>>>> examples
>>>>>>>>>>>> on how people might be connecting to hive and
>>>>>>>>>>>> executing queries
>>>>>>>>>>>> through
>>>>>>>>>>>> java/python. Anyone have anything available? (Python preferable)
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> http://mixpanel.com
>>>>>>>>>>>> Blog: http://blog.mixpanel.com
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> http://mixpanel.com
>>>>>>>>>>>> Blog: http://blog.mixpanel.com
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> http://mixpanel.com
>>>>>>>>>>>> Blog: http://blog.mixpanel.com
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>
>>>>>
>>>>>
>>>
>>
>>
>>
>> --
>> http://mixpanel.com
>> Blog: http://blog.mixpanel.com
>
>
Re: Java/Python examples
Posted by Suhail Doshi <di...@gmail.com>.
Has anyone had problems when you start the hiveserver you simply can't
execute queries because of the following error:
Caused by: ERROR XSDB6: Another instance of Derby may have already booted
the database /home/hadoop/hive/build/dist/conf/metastore_db.
li57-125 ~/hive/build/dist/conf/metastore_db: hive
Hive history file=/tmp/hadoop/hive_job_log_hadoop_200903231803_720530131.txt
hive> show tables;
FAILED: Error in metadata: javax.jdo.JDOFatalDataStoreException: Failed to
start database '/home/hadoop/hive/build/dist/conf/metastore_db', see the
next exception for details.
NestedThrowables:
java.sql.SQLException: Failed to start database
'/home/hadoop/hive/build/dist/conf/metastore_db', see the next exception for
details.
FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.DDLTask
Time taken: 1.018 seconds
Not sure what to do, is my metastore_db corrupted? Anyway to start from
scratch or a fix?
Suhail
On Sun, Mar 22, 2009 at 4:38 PM, Suhail Doshi <di...@gmail.com>wrote:
> Hi Ragu,
>
> Thanks for the help thus far, seem to have one last problem:
>
> from thrift import Thrift
> from thrift.transport import TSocket
> from thrift.transport import TTransport
> from thrift.protocol import TBinaryProtocol
>
> from hive_service import ThriftHive
>
> transport = TSocket.TSocket('localhost', 9090)
> transport = TTransport.TBufferedTransport(transport)
> protocol = TBinaryProtocol.TBinaryProtocol(transport)
>
> client = ThriftHive.Client(protocol)
>
> Do you know the correct way to execute a query? I've tried:
>
> >>> client.execute("select * from page_views")
>
> However received the following error:
>
> Traceback (most recent call last):
> File "<stdin>", line 1, in <module>
> File "/home/hadoop/hive/build/dist/lib/py/hive_service/ThriftHive.py",
> line 41, in execute
> self.send_execute(query)
> File "/home/hadoop/hive/build/dist/lib/py/hive_service/ThriftHive.py",
> line 50, in send_execute
> self._oprot.trans.flush()
> File "/usr/lib/python2.5/site-packages/thrift/transport/TTransport.py",
> line 152, in flush
> self.__trans.write(out)
> File "/usr/lib/python2.5/site-packages/thrift/transport/TSocket.py", line
> 88, in write
> plus = self.handle.send(buff)
> AttributeError: 'NoneType' object has no attribute 'send'
>
>
>
> On Sat, Mar 21, 2009 at 6:38 PM, Raghu Murthy <ra...@facebook.com> wrote:
>
>> The hive module is in hive_service.
>>
>> Try from hive_service import ThriftHive
>>
>>
>> On 3/21/09 5:20 PM, "Suhail Doshi" <di...@gmail.com> wrote:
>>
>> > Actually the hive generate py files for thrift don't appear in that
>> > dir...should I just make a symlink?
>> >
>> > On Sat, Mar 21, 2009 at 5:04 PM, Raghu Murthy <ra...@facebook.com>
>> wrote:
>> >> Use build/dist/lib/py instead of gen-py. All modules are copied to that
>> >> directory once you run ant package.
>> >>
>> >>
>> >> On 3/21/09 2:17 PM, "Suhail Doshi" <di...@gmail.com> wrote:
>> >>
>> >>>> I seem to be getting the following error:
>> >>>>
>> >>>> Traceback (most recent call last):
>> >>>> File "hive_test.py", line 13, in <module>
>> >>>> from hive import ThriftHive
>> >>>> File "ThriftHive.py", line 8, in <module>
>> >>>> ImportError: No module named hive_metastore.ThriftHiveMetastore
>> >>>>
>> >>>> There's a symlink gen-py -> /home/hadoop/hive/service/src/gen-py in
>> the
>> >>> folder
>> >>>> with the py script.
>> >>>>
>> >>>> With the following code:
>> >>>>
>> >>>> #!/usr/bin/evn python
>> >>>> import sys
>> >>>> import os
>> >>>>
>> >>>> dir = os.path.dirname(os.path.realpath(sys.argv[0]))
>> >>>> sys.path.insert(0, dir + '/gen-py')
>> >>>>
>> >>>> from thrift import Thrift
>> >>>> from thrift.transport import TSocket
>> >>>> from thrift.transport import TTransport
>> >>>> from thrift.protocol import TBinaryProtocol
>> >>>>
>> >>>> from hive import ThriftHive
>> >>>>
>> >>>> transport = TSocket.TSocket('localhost', 9090)
>> >>>> transport = TTransport.TBufferedTransport(transport)
>> >>>> protocol = TBinaryProtocol.TBinaryProtocol(transport)
>> >>>>
>> >>>> client = ThriftHive.Client(protocol)
>> >>>>
>> >>>> print client
>> >>>>
>> >>>>
>> >>>>
>> >>>> On Fri, Mar 20, 2009 at 4:06 PM, Suhail Doshi <
>> digitalwarfare@gmail.com>
>> >>>> wrote:
>> >>>>>> Whoops, silly mistake, nevermind =)
>> >>>>>>
>> >>>>>>
>> >>>>>> On Fri, Mar 20, 2009 at 3:55 PM, Suhail Doshi <
>> digitalwarfare@gmail.com>
>> >>>>>> wrote:
>> >>>>>>>> Namit,
>> >>>>>>>>
>> >>>>>>>> For some reason when I apply:
>> >>>>>>>> li57-125 ~/hive: patch -p0 hive-py.patch
>> >>>>>>>>
>> >>>>>>>> It seems just hang for a while.
>> >>>>>>>>
>> >>>>>>>> Suhail
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>> On Fri, Mar 20, 2009 at 12:58 PM, Namit Jain <njain@facebook.com
>> >
>> >>>>> wrote:
>> >>>>>>>>>> Download the patch file <patch_file>
>> >>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>> Go to the truck directory, and perform
>> >>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>> patch -p0 < <patch_file>
>> >>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>> Then compile and run tests.
>> >>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>> From: Suhail Doshi [mailto:digitalwarfare@gmail.com]
>> >>>>>>>>>> Sent: Friday, March 20, 2009 12:52 PM
>> >>>>>>>>>> To: hive-user@hadoop.apache.org
>> >>>>>>>>>> Subject: Re: Java/Python examples
>> >>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>> Ragu,
>> >>>>>>>>>>
>> >>>>>>>>>> How is exactly do you apply the patches?
>> >>>>>>>>>>
>> >>>>>>>>>> Suhail
>> >>>>>>>>>>
>> >>>>>>>>>> On Sun, Mar 15, 2009 at 3:57 PM, Raghu Murthy <
>> raghu@facebook.com>
>> >>>>>>>>>> >>>>> wrote:
>> >>>>>>>>>>
>> >>>>>>>>>> You can call Hive in Java either directly or in client-server
>> mode
>> >>>>>>>>>> >>>>> where
>> >>>>>>>>>> you
>> >>>>>>>>>> would have to run a thrift server.
>> >>>>>>>>>> Python and php clients are currently only available in patches
>> at
>> >>>>>>>>>> https://issues.apache.org/jira/browse/HIVE-343 and
>> >>>>>>>>>> https://issues.apache.org/jira/browse/HIVE-299 respectively.
>> >>>>>>>>>>
>> >>>>>>>>>> If you apply the patches and then run ant package, the
>> documentation
>> >>>>>>>>>> at
>> >>>>>>>>>> http://wiki.apache.org/hadoop/Hive/HiveServer should be
>> useful. Hope
>> >>>>>>>>>> this
>> >>>>>>>>>> helps.
>> >>>>>>>>>>
>> >>>>>>>>>> raghu
>> >>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>> On 3/15/09 12:22 PM, "Suhail Doshi" <su...@mixpanel.com>
>> wrote:
>> >>>>>>>>>>
>> >>>>>>>>>>>> I was browsing around the Hive wiki and couldn't manage to
>> find any
>> >>>>>>>>>>>> examples
>> >>>>>>>>>>>> on how people might be connecting to hive and executing
>> queries
>> >>>>>>>>>>>> >>>>>>> through
>> >>>>>>>>>>>> java/python. Anyone have anything available? (Python
>> preferable)
>> >>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>> --
>> >>>>>>>>>> http://mixpanel.com
>> >>>>>>>>>> Blog: http://blog.mixpanel.com
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>> --
>> >>>>>>>> http://mixpanel.com
>> >>>>>>>> Blog: http://blog.mixpanel.com
>> >>>>>>
>> >>>>>>
>> >>>>>>
>> >>>>>> --
>> >>>>>> http://mixpanel.com
>> >>>>>> Blog: http://blog.mixpanel.com
>> >>>>
>> >>>>
>> >>
>> >
>> >
>>
>>
>
>
> --
> http://mixpanel.com
> Blog: http://blog.mixpanel.com
>
--
http://mixpanel.com
Blog: http://blog.mixpanel.com
Re: Java/Python examples
Posted by Suhail Doshi <di...@gmail.com>.
Hi Ragu,
Thanks for the help thus far, seem to have one last problem:
from thrift import Thrift
from thrift.transport import TSocket
from thrift.transport import TTransport
from thrift.protocol import TBinaryProtocol
from hive_service import ThriftHive
transport = TSocket.TSocket('localhost', 9090)
transport = TTransport.TBufferedTransport(transport)
protocol = TBinaryProtocol.TBinaryProtocol(transport)
client = ThriftHive.Client(protocol)
Do you know the correct way to execute a query? I've tried:
>>> client.execute("select * from page_views")
However received the following error:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/hadoop/hive/build/dist/lib/py/hive_service/ThriftHive.py",
line 41, in execute
self.send_execute(query)
File "/home/hadoop/hive/build/dist/lib/py/hive_service/ThriftHive.py",
line 50, in send_execute
self._oprot.trans.flush()
File "/usr/lib/python2.5/site-packages/thrift/transport/TTransport.py",
line 152, in flush
self.__trans.write(out)
File "/usr/lib/python2.5/site-packages/thrift/transport/TSocket.py", line
88, in write
plus = self.handle.send(buff)
AttributeError: 'NoneType' object has no attribute 'send'
On Sat, Mar 21, 2009 at 6:38 PM, Raghu Murthy <ra...@facebook.com> wrote:
> The hive module is in hive_service.
>
> Try from hive_service import ThriftHive
>
>
> On 3/21/09 5:20 PM, "Suhail Doshi" <di...@gmail.com> wrote:
>
> > Actually the hive generate py files for thrift don't appear in that
> > dir...should I just make a symlink?
> >
> > On Sat, Mar 21, 2009 at 5:04 PM, Raghu Murthy <ra...@facebook.com>
> wrote:
> >> Use build/dist/lib/py instead of gen-py. All modules are copied to that
> >> directory once you run ant package.
> >>
> >>
> >> On 3/21/09 2:17 PM, "Suhail Doshi" <di...@gmail.com> wrote:
> >>
> >>>> I seem to be getting the following error:
> >>>>
> >>>> Traceback (most recent call last):
> >>>> File "hive_test.py", line 13, in <module>
> >>>> from hive import ThriftHive
> >>>> File "ThriftHive.py", line 8, in <module>
> >>>> ImportError: No module named hive_metastore.ThriftHiveMetastore
> >>>>
> >>>> There's a symlink gen-py -> /home/hadoop/hive/service/src/gen-py in
> the
> >>> folder
> >>>> with the py script.
> >>>>
> >>>> With the following code:
> >>>>
> >>>> #!/usr/bin/evn python
> >>>> import sys
> >>>> import os
> >>>>
> >>>> dir = os.path.dirname(os.path.realpath(sys.argv[0]))
> >>>> sys.path.insert(0, dir + '/gen-py')
> >>>>
> >>>> from thrift import Thrift
> >>>> from thrift.transport import TSocket
> >>>> from thrift.transport import TTransport
> >>>> from thrift.protocol import TBinaryProtocol
> >>>>
> >>>> from hive import ThriftHive
> >>>>
> >>>> transport = TSocket.TSocket('localhost', 9090)
> >>>> transport = TTransport.TBufferedTransport(transport)
> >>>> protocol = TBinaryProtocol.TBinaryProtocol(transport)
> >>>>
> >>>> client = ThriftHive.Client(protocol)
> >>>>
> >>>> print client
> >>>>
> >>>>
> >>>>
> >>>> On Fri, Mar 20, 2009 at 4:06 PM, Suhail Doshi <
> digitalwarfare@gmail.com>
> >>>> wrote:
> >>>>>> Whoops, silly mistake, nevermind =)
> >>>>>>
> >>>>>>
> >>>>>> On Fri, Mar 20, 2009 at 3:55 PM, Suhail Doshi <
> digitalwarfare@gmail.com>
> >>>>>> wrote:
> >>>>>>>> Namit,
> >>>>>>>>
> >>>>>>>> For some reason when I apply:
> >>>>>>>> li57-125 ~/hive: patch -p0 hive-py.patch
> >>>>>>>>
> >>>>>>>> It seems just hang for a while.
> >>>>>>>>
> >>>>>>>> Suhail
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> On Fri, Mar 20, 2009 at 12:58 PM, Namit Jain <nj...@facebook.com>
> >>>>> wrote:
> >>>>>>>>>> Download the patch file <patch_file>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> Go to the truck directory, and perform
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> patch -p0 < <patch_file>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> Then compile and run tests.
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> From: Suhail Doshi [mailto:digitalwarfare@gmail.com]
> >>>>>>>>>> Sent: Friday, March 20, 2009 12:52 PM
> >>>>>>>>>> To: hive-user@hadoop.apache.org
> >>>>>>>>>> Subject: Re: Java/Python examples
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> Ragu,
> >>>>>>>>>>
> >>>>>>>>>> How is exactly do you apply the patches?
> >>>>>>>>>>
> >>>>>>>>>> Suhail
> >>>>>>>>>>
> >>>>>>>>>> On Sun, Mar 15, 2009 at 3:57 PM, Raghu Murthy <
> raghu@facebook.com>
> >>>>>>>>>> >>>>> wrote:
> >>>>>>>>>>
> >>>>>>>>>> You can call Hive in Java either directly or in client-server
> mode
> >>>>>>>>>> >>>>> where
> >>>>>>>>>> you
> >>>>>>>>>> would have to run a thrift server.
> >>>>>>>>>> Python and php clients are currently only available in patches
> at
> >>>>>>>>>> https://issues.apache.org/jira/browse/HIVE-343 and
> >>>>>>>>>> https://issues.apache.org/jira/browse/HIVE-299 respectively.
> >>>>>>>>>>
> >>>>>>>>>> If you apply the patches and then run ant package, the
> documentation
> >>>>>>>>>> at
> >>>>>>>>>> http://wiki.apache.org/hadoop/Hive/HiveServer should be useful.
> Hope
> >>>>>>>>>> this
> >>>>>>>>>> helps.
> >>>>>>>>>>
> >>>>>>>>>> raghu
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> On 3/15/09 12:22 PM, "Suhail Doshi" <su...@mixpanel.com>
> wrote:
> >>>>>>>>>>
> >>>>>>>>>>>> I was browsing around the Hive wiki and couldn't manage to
> find any
> >>>>>>>>>>>> examples
> >>>>>>>>>>>> on how people might be connecting to hive and executing
> queries
> >>>>>>>>>>>> >>>>>>> through
> >>>>>>>>>>>> java/python. Anyone have anything available? (Python
> preferable)
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> --
> >>>>>>>>>> http://mixpanel.com
> >>>>>>>>>> Blog: http://blog.mixpanel.com
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> --
> >>>>>>>> http://mixpanel.com
> >>>>>>>> Blog: http://blog.mixpanel.com
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> --
> >>>>>> http://mixpanel.com
> >>>>>> Blog: http://blog.mixpanel.com
> >>>>
> >>>>
> >>
> >
> >
>
>
--
http://mixpanel.com
Blog: http://blog.mixpanel.com
Re: Java/Python examples
Posted by Raghu Murthy <ra...@facebook.com>.
The hive module is in hive_service.
Try from hive_service import ThriftHive
On 3/21/09 5:20 PM, "Suhail Doshi" <di...@gmail.com> wrote:
> Actually the hive generate py files for thrift don't appear in that
> dir...should I just make a symlink?
>
> On Sat, Mar 21, 2009 at 5:04 PM, Raghu Murthy <ra...@facebook.com> wrote:
>> Use build/dist/lib/py instead of gen-py. All modules are copied to that
>> directory once you run ant package.
>>
>>
>> On 3/21/09 2:17 PM, "Suhail Doshi" <di...@gmail.com> wrote:
>>
>>>> I seem to be getting the following error:
>>>>
>>>> Traceback (most recent call last):
>>>> File "hive_test.py", line 13, in <module>
>>>> from hive import ThriftHive
>>>> File "ThriftHive.py", line 8, in <module>
>>>> ImportError: No module named hive_metastore.ThriftHiveMetastore
>>>>
>>>> There's a symlink gen-py -> /home/hadoop/hive/service/src/gen-py in the
>>> folder
>>>> with the py script.
>>>>
>>>> With the following code:
>>>>
>>>> #!/usr/bin/evn python
>>>> import sys
>>>> import os
>>>>
>>>> dir = os.path.dirname(os.path.realpath(sys.argv[0]))
>>>> sys.path.insert(0, dir + '/gen-py')
>>>>
>>>> from thrift import Thrift
>>>> from thrift.transport import TSocket
>>>> from thrift.transport import TTransport
>>>> from thrift.protocol import TBinaryProtocol
>>>>
>>>> from hive import ThriftHive
>>>>
>>>> transport = TSocket.TSocket('localhost', 9090)
>>>> transport = TTransport.TBufferedTransport(transport)
>>>> protocol = TBinaryProtocol.TBinaryProtocol(transport)
>>>>
>>>> client = ThriftHive.Client(protocol)
>>>>
>>>> print client
>>>>
>>>>
>>>>
>>>> On Fri, Mar 20, 2009 at 4:06 PM, Suhail Doshi <di...@gmail.com>
>>>> wrote:
>>>>>> Whoops, silly mistake, nevermind =)
>>>>>>
>>>>>>
>>>>>> On Fri, Mar 20, 2009 at 3:55 PM, Suhail Doshi <di...@gmail.com>
>>>>>> wrote:
>>>>>>>> Namit,
>>>>>>>>
>>>>>>>> For some reason when I apply:
>>>>>>>> li57-125 ~/hive: patch -p0 hive-py.patch
>>>>>>>>
>>>>>>>> It seems just hang for a while.
>>>>>>>>
>>>>>>>> Suhail
>>>>>>>>
>>>>>>>>
>>>>>>>> On Fri, Mar 20, 2009 at 12:58 PM, Namit Jain <nj...@facebook.com>
>>>>> wrote:
>>>>>>>>>> Download the patch file <patch_file>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Go to the truck directory, and perform
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> patch -p0 < <patch_file>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Then compile and run tests.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> From: Suhail Doshi [mailto:digitalwarfare@gmail.com]
>>>>>>>>>> Sent: Friday, March 20, 2009 12:52 PM
>>>>>>>>>> To: hive-user@hadoop.apache.org
>>>>>>>>>> Subject: Re: Java/Python examples
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Ragu,
>>>>>>>>>>
>>>>>>>>>> How is exactly do you apply the patches?
>>>>>>>>>>
>>>>>>>>>> Suhail
>>>>>>>>>>
>>>>>>>>>> On Sun, Mar 15, 2009 at 3:57 PM, Raghu Murthy <ra...@facebook.com>
>>>>>>>>>> >>>>> wrote:
>>>>>>>>>>
>>>>>>>>>> You can call Hive in Java either directly or in client-server mode
>>>>>>>>>> >>>>> where
>>>>>>>>>> you
>>>>>>>>>> would have to run a thrift server.
>>>>>>>>>> Python and php clients are currently only available in patches at
>>>>>>>>>> https://issues.apache.org/jira/browse/HIVE-343 and
>>>>>>>>>> https://issues.apache.org/jira/browse/HIVE-299 respectively.
>>>>>>>>>>
>>>>>>>>>> If you apply the patches and then run ant package, the documentation
>>>>>>>>>> at
>>>>>>>>>> http://wiki.apache.org/hadoop/Hive/HiveServer should be useful. Hope
>>>>>>>>>> this
>>>>>>>>>> helps.
>>>>>>>>>>
>>>>>>>>>> raghu
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On 3/15/09 12:22 PM, "Suhail Doshi" <su...@mixpanel.com> wrote:
>>>>>>>>>>
>>>>>>>>>>>> I was browsing around the Hive wiki and couldn't manage to find any
>>>>>>>>>>>> examples
>>>>>>>>>>>> on how people might be connecting to hive and executing queries
>>>>>>>>>>>> >>>>>>> through
>>>>>>>>>>>> java/python. Anyone have anything available? (Python preferable)
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> http://mixpanel.com
>>>>>>>>>> Blog: http://blog.mixpanel.com
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> http://mixpanel.com
>>>>>>>> Blog: http://blog.mixpanel.com
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> http://mixpanel.com
>>>>>> Blog: http://blog.mixpanel.com
>>>>
>>>>
>>
>
>
Re: Java/Python examples
Posted by Suhail Doshi <di...@gmail.com>.
Actually the hive generate py files for thrift don't appear in that
dir...should I just make a symlink?
On Sat, Mar 21, 2009 at 5:04 PM, Raghu Murthy <ra...@facebook.com> wrote:
> Use build/dist/lib/py instead of gen-py. All modules are copied to that
> directory once you run ant package.
>
>
> On 3/21/09 2:17 PM, "Suhail Doshi" <di...@gmail.com> wrote:
>
> > I seem to be getting the following error:
> >
> > Traceback (most recent call last):
> > File "hive_test.py", line 13, in <module>
> > from hive import ThriftHive
> > File "ThriftHive.py", line 8, in <module>
> > ImportError: No module named hive_metastore.ThriftHiveMetastore
> >
> > There's a symlink gen-py -> /home/hadoop/hive/service/src/gen-py in the
> folder
> > with the py script.
> >
> > With the following code:
> >
> > #!/usr/bin/evn python
> > import sys
> > import os
> >
> > dir = os.path.dirname(os.path.realpath(sys.argv[0]))
> > sys.path.insert(0, dir + '/gen-py')
> >
> > from thrift import Thrift
> > from thrift.transport import TSocket
> > from thrift.transport import TTransport
> > from thrift.protocol import TBinaryProtocol
> >
> > from hive import ThriftHive
> >
> > transport = TSocket.TSocket('localhost', 9090)
> > transport = TTransport.TBufferedTransport(transport)
> > protocol = TBinaryProtocol.TBinaryProtocol(transport)
> >
> > client = ThriftHive.Client(protocol)
> >
> > print client
> >
> >
> >
> > On Fri, Mar 20, 2009 at 4:06 PM, Suhail Doshi <di...@gmail.com>
> > wrote:
> >> Whoops, silly mistake, nevermind =)
> >>
> >>
> >> On Fri, Mar 20, 2009 at 3:55 PM, Suhail Doshi <digitalwarfare@gmail.com
> >
> >> wrote:
> >>> Namit,
> >>>
> >>> For some reason when I apply:
> >>> li57-125 ~/hive: patch -p0 hive-py.patch
> >>>
> >>> It seems just hang for a while.
> >>>
> >>> Suhail
> >>>
> >>>
> >>> On Fri, Mar 20, 2009 at 12:58 PM, Namit Jain <nj...@facebook.com>
> wrote:
> >>>> Download the patch file <patch_file>
> >>>>
> >>>>
> >>>>
> >>>> Go to the truck directory, and perform
> >>>>
> >>>>
> >>>>
> >>>> patch -p0 < <patch_file>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> Then compile and run tests.
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> From: Suhail Doshi [mailto:digitalwarfare@gmail.com]
> >>>> Sent: Friday, March 20, 2009 12:52 PM
> >>>> To: hive-user@hadoop.apache.org
> >>>> Subject: Re: Java/Python examples
> >>>>
> >>>>
> >>>>
> >>>> Ragu,
> >>>>
> >>>> How is exactly do you apply the patches?
> >>>>
> >>>> Suhail
> >>>>
> >>>> On Sun, Mar 15, 2009 at 3:57 PM, Raghu Murthy <ra...@facebook.com>
> wrote:
> >>>>
> >>>> You can call Hive in Java either directly or in client-server mode
> where
> >>>> you
> >>>> would have to run a thrift server.
> >>>> Python and php clients are currently only available in patches at
> >>>> https://issues.apache.org/jira/browse/HIVE-343 and
> >>>> https://issues.apache.org/jira/browse/HIVE-299 respectively.
> >>>>
> >>>> If you apply the patches and then run ant package, the documentation
> at
> >>>> http://wiki.apache.org/hadoop/Hive/HiveServer should be useful. Hope
> this
> >>>> helps.
> >>>>
> >>>> raghu
> >>>>
> >>>>
> >>>> On 3/15/09 12:22 PM, "Suhail Doshi" <su...@mixpanel.com> wrote:
> >>>>
> >>>>>> I was browsing around the Hive wiki and couldn't manage to find any
> >>>>> examples
> >>>>>> on how people might be connecting to hive and executing queries
> through
> >>>>>> java/python. Anyone have anything available? (Python preferable)
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> --
> >>>> http://mixpanel.com
> >>>> Blog: http://blog.mixpanel.com
> >>>
> >>>
> >>>
> >>> --
> >>> http://mixpanel.com
> >>> Blog: http://blog.mixpanel.com
> >>
> >>
> >>
> >> --
> >> http://mixpanel.com
> >> Blog: http://blog.mixpanel.com
> >
> >
>
>
--
http://mixpanel.com
Blog: http://blog.mixpanel.com
Re: Java/Python examples
Posted by Suhail Doshi <di...@gmail.com>.
Perhaps I am incorrectly setting/using my pythonpath:
PYTHONPATH=/home/hadoop/hive/build/dist/lib/py/
li57-125 ~/hive: python hive_test.py
Traceback (most recent call last):
File "hive_test.py", line 13, in <module>
from hive import ThriftHive
ImportError: No module named hive
li57-125 ~/hive/build/dist/lib/py: ls -l
total 32
-rw-r--r-- 1 hadoop hadoop 0 Mar 15 01:35 __init__.py
drwxr-xr-x 2 hadoop hadoop 4096 Mar 22 00:10 fb303
drwxr-xr-x 2 hadoop hadoop 4096 Mar 20 23:06 fb303_scripts
drwxr-xr-x 2 hadoop hadoop 4096 Mar 22 00:10 hive_metastore
drwxr-xr-x 2 hadoop hadoop 4096 Mar 20 23:06 hive_serde
drwxr-xr-x 2 hadoop hadoop 4096 Mar 20 23:06 hive_service
drwxr-xr-x 2 hadoop hadoop 4096 Mar 15 01:35 org_apache_hadoop_hive_serde
drwxr-xr-x 2 hadoop hadoop 4096 Mar 15 01:35 serde
drwxr-xr-x 6 hadoop hadoop 4096 Mar 22 00:10 thrift
On Sat, Mar 21, 2009 at 5:04 PM, Raghu Murthy <ra...@facebook.com> wrote:
> Use build/dist/lib/py instead of gen-py. All modules are copied to that
> directory once you run ant package.
>
>
> On 3/21/09 2:17 PM, "Suhail Doshi" <di...@gmail.com> wrote:
>
> > I seem to be getting the following error:
> >
> > Traceback (most recent call last):
> > File "hive_test.py", line 13, in <module>
> > from hive import ThriftHive
> > File "ThriftHive.py", line 8, in <module>
> > ImportError: No module named hive_metastore.ThriftHiveMetastore
> >
> > There's a symlink gen-py -> /home/hadoop/hive/service/src/gen-py in the
> folder
> > with the py script.
> >
> > With the following code:
> >
> > #!/usr/bin/evn python
> > import sys
> > import os
> >
> > dir = os.path.dirname(os.path.realpath(sys.argv[0]))
> > sys.path.insert(0, dir + '/gen-py')
> >
> > from thrift import Thrift
> > from thrift.transport import TSocket
> > from thrift.transport import TTransport
> > from thrift.protocol import TBinaryProtocol
> >
> > from hive import ThriftHive
> >
> > transport = TSocket.TSocket('localhost', 9090)
> > transport = TTransport.TBufferedTransport(transport)
> > protocol = TBinaryProtocol.TBinaryProtocol(transport)
> >
> > client = ThriftHive.Client(protocol)
> >
> > print client
> >
> >
> >
> > On Fri, Mar 20, 2009 at 4:06 PM, Suhail Doshi <di...@gmail.com>
> > wrote:
> >> Whoops, silly mistake, nevermind =)
> >>
> >>
> >> On Fri, Mar 20, 2009 at 3:55 PM, Suhail Doshi <digitalwarfare@gmail.com
> >
> >> wrote:
> >>> Namit,
> >>>
> >>> For some reason when I apply:
> >>> li57-125 ~/hive: patch -p0 hive-py.patch
> >>>
> >>> It seems just hang for a while.
> >>>
> >>> Suhail
> >>>
> >>>
> >>> On Fri, Mar 20, 2009 at 12:58 PM, Namit Jain <nj...@facebook.com>
> wrote:
> >>>> Download the patch file <patch_file>
> >>>>
> >>>>
> >>>>
> >>>> Go to the truck directory, and perform
> >>>>
> >>>>
> >>>>
> >>>> patch -p0 < <patch_file>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> Then compile and run tests.
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> From: Suhail Doshi [mailto:digitalwarfare@gmail.com]
> >>>> Sent: Friday, March 20, 2009 12:52 PM
> >>>> To: hive-user@hadoop.apache.org
> >>>> Subject: Re: Java/Python examples
> >>>>
> >>>>
> >>>>
> >>>> Ragu,
> >>>>
> >>>> How is exactly do you apply the patches?
> >>>>
> >>>> Suhail
> >>>>
> >>>> On Sun, Mar 15, 2009 at 3:57 PM, Raghu Murthy <ra...@facebook.com>
> wrote:
> >>>>
> >>>> You can call Hive in Java either directly or in client-server mode
> where
> >>>> you
> >>>> would have to run a thrift server.
> >>>> Python and php clients are currently only available in patches at
> >>>> https://issues.apache.org/jira/browse/HIVE-343 and
> >>>> https://issues.apache.org/jira/browse/HIVE-299 respectively.
> >>>>
> >>>> If you apply the patches and then run ant package, the documentation
> at
> >>>> http://wiki.apache.org/hadoop/Hive/HiveServer should be useful. Hope
> this
> >>>> helps.
> >>>>
> >>>> raghu
> >>>>
> >>>>
> >>>> On 3/15/09 12:22 PM, "Suhail Doshi" <su...@mixpanel.com> wrote:
> >>>>
> >>>>>> I was browsing around the Hive wiki and couldn't manage to find any
> >>>>> examples
> >>>>>> on how people might be connecting to hive and executing queries
> through
> >>>>>> java/python. Anyone have anything available? (Python preferable)
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> --
> >>>> http://mixpanel.com
> >>>> Blog: http://blog.mixpanel.com
> >>>
> >>>
> >>>
> >>> --
> >>> http://mixpanel.com
> >>> Blog: http://blog.mixpanel.com
> >>
> >>
> >>
> >> --
> >> http://mixpanel.com
> >> Blog: http://blog.mixpanel.com
> >
> >
>
>
--
http://mixpanel.com
Blog: http://blog.mixpanel.com
Re: Java/Python examples
Posted by Raghu Murthy <ra...@facebook.com>.
Use build/dist/lib/py instead of gen-py. All modules are copied to that
directory once you run ant package.
On 3/21/09 2:17 PM, "Suhail Doshi" <di...@gmail.com> wrote:
> I seem to be getting the following error:
>
> Traceback (most recent call last):
> File "hive_test.py", line 13, in <module>
> from hive import ThriftHive
> File "ThriftHive.py", line 8, in <module>
> ImportError: No module named hive_metastore.ThriftHiveMetastore
>
> There's a symlink gen-py -> /home/hadoop/hive/service/src/gen-py in the folder
> with the py script.
>
> With the following code:
>
> #!/usr/bin/evn python
> import sys
> import os
>
> dir = os.path.dirname(os.path.realpath(sys.argv[0]))
> sys.path.insert(0, dir + '/gen-py')
>
> from thrift import Thrift
> from thrift.transport import TSocket
> from thrift.transport import TTransport
> from thrift.protocol import TBinaryProtocol
>
> from hive import ThriftHive
>
> transport = TSocket.TSocket('localhost', 9090)
> transport = TTransport.TBufferedTransport(transport)
> protocol = TBinaryProtocol.TBinaryProtocol(transport)
>
> client = ThriftHive.Client(protocol)
>
> print client
>
>
>
> On Fri, Mar 20, 2009 at 4:06 PM, Suhail Doshi <di...@gmail.com>
> wrote:
>> Whoops, silly mistake, nevermind =)
>>
>>
>> On Fri, Mar 20, 2009 at 3:55 PM, Suhail Doshi <di...@gmail.com>
>> wrote:
>>> Namit,
>>>
>>> For some reason when I apply:
>>> li57-125 ~/hive: patch -p0 hive-py.patch
>>>
>>> It seems just hang for a while.
>>>
>>> Suhail
>>>
>>>
>>> On Fri, Mar 20, 2009 at 12:58 PM, Namit Jain <nj...@facebook.com> wrote:
>>>> Download the patch file <patch_file>
>>>>
>>>>
>>>>
>>>> Go to the truck directory, and perform
>>>>
>>>>
>>>>
>>>> patch -p0 < <patch_file>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> Then compile and run tests.
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> From: Suhail Doshi [mailto:digitalwarfare@gmail.com]
>>>> Sent: Friday, March 20, 2009 12:52 PM
>>>> To: hive-user@hadoop.apache.org
>>>> Subject: Re: Java/Python examples
>>>>
>>>>
>>>>
>>>> Ragu,
>>>>
>>>> How is exactly do you apply the patches?
>>>>
>>>> Suhail
>>>>
>>>> On Sun, Mar 15, 2009 at 3:57 PM, Raghu Murthy <ra...@facebook.com> wrote:
>>>>
>>>> You can call Hive in Java either directly or in client-server mode where
>>>> you
>>>> would have to run a thrift server.
>>>> Python and php clients are currently only available in patches at
>>>> https://issues.apache.org/jira/browse/HIVE-343 and
>>>> https://issues.apache.org/jira/browse/HIVE-299 respectively.
>>>>
>>>> If you apply the patches and then run ant package, the documentation at
>>>> http://wiki.apache.org/hadoop/Hive/HiveServer should be useful. Hope this
>>>> helps.
>>>>
>>>> raghu
>>>>
>>>>
>>>> On 3/15/09 12:22 PM, "Suhail Doshi" <su...@mixpanel.com> wrote:
>>>>
>>>>>> I was browsing around the Hive wiki and couldn't manage to find any
>>>>> examples
>>>>>> on how people might be connecting to hive and executing queries through
>>>>>> java/python. Anyone have anything available? (Python preferable)
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> http://mixpanel.com
>>>> Blog: http://blog.mixpanel.com
>>>
>>>
>>>
>>> --
>>> http://mixpanel.com
>>> Blog: http://blog.mixpanel.com
>>
>>
>>
>> --
>> http://mixpanel.com
>> Blog: http://blog.mixpanel.com
>
>
Re: Java/Python examples
Posted by Suhail Doshi <di...@gmail.com>.
I seem to be getting the following error:
Traceback (most recent call last):
File "hive_test.py", line 13, in <module>
from hive import ThriftHive
File "ThriftHive.py", line 8, in <module>
ImportError: No module named hive_metastore.ThriftHiveMetastore
There's a symlink gen-py -> /home/hadoop/hive/service/src/gen-py in the
folder with the py script.
With the following code:
#!/usr/bin/evn python
import sys
import os
dir = os.path.dirname(os.path.realpath(sys.argv[0]))
sys.path.insert(0, dir + '/gen-py')
from thrift import Thrift
from thrift.transport import TSocket
from thrift.transport import TTransport
from thrift.protocol import TBinaryProtocol
from hive import ThriftHive
transport = TSocket.TSocket('localhost', 9090)
transport = TTransport.TBufferedTransport(transport)
protocol = TBinaryProtocol.TBinaryProtocol(transport)
client = ThriftHive.Client(protocol)
print client
On Fri, Mar 20, 2009 at 4:06 PM, Suhail Doshi <di...@gmail.com>wrote:
> Whoops, silly mistake, nevermind =)
>
>
> On Fri, Mar 20, 2009 at 3:55 PM, Suhail Doshi <di...@gmail.com>wrote:
>
>> Namit,
>>
>> For some reason when I apply:
>> li57-125 ~/hive: patch -p0 hive-py.patch
>>
>> It seems just hang for a while.
>>
>> Suhail
>>
>>
>> On Fri, Mar 20, 2009 at 12:58 PM, Namit Jain <nj...@facebook.com> wrote:
>>
>>> Download the patch file <patch_file>
>>>
>>>
>>>
>>> Go to the truck directory, and perform
>>>
>>>
>>>
>>> patch -p0 < <patch_file>
>>>
>>>
>>>
>>>
>>>
>>> Then compile and run tests.
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> *From:* Suhail Doshi [mailto:digitalwarfare@gmail.com]
>>> *Sent:* Friday, March 20, 2009 12:52 PM
>>> *To:* hive-user@hadoop.apache.org
>>> *Subject:* Re: Java/Python examples
>>>
>>>
>>>
>>> Ragu,
>>>
>>> How is exactly do you apply the patches?
>>>
>>> Suhail
>>>
>>> On Sun, Mar 15, 2009 at 3:57 PM, Raghu Murthy <ra...@facebook.com>
>>> wrote:
>>>
>>> You can call Hive in Java either directly or in client-server mode where
>>> you
>>> would have to run a thrift server.
>>> Python and php clients are currently only available in patches at
>>> https://issues.apache.org/jira/browse/HIVE-343 and
>>> https://issues.apache.org/jira/browse/HIVE-299 respectively.
>>>
>>> If you apply the patches and then run ant package, the documentation at
>>> http://wiki.apache.org/hadoop/Hive/HiveServer should be useful. Hope
>>> this
>>> helps.
>>>
>>> raghu
>>>
>>>
>>> On 3/15/09 12:22 PM, "Suhail Doshi" <su...@mixpanel.com> wrote:
>>>
>>> > I was browsing around the Hive wiki and couldn't manage to find any
>>> examples
>>> > on how people might be connecting to hive and executing queries through
>>> > java/python. Anyone have anything available? (Python preferable)
>>>
>>>
>>>
>>>
>>> --
>>> http://mixpanel.com
>>> Blog: http://blog.mixpanel.com
>>>
>>
>>
>>
>> --
>> http://mixpanel.com
>> Blog: http://blog.mixpanel.com
>>
>
>
>
> --
> http://mixpanel.com
> Blog: http://blog.mixpanel.com
>
--
http://mixpanel.com
Blog: http://blog.mixpanel.com
Re: Java/Python examples
Posted by Suhail Doshi <di...@gmail.com>.
Whoops, silly mistake, nevermind =)
On Fri, Mar 20, 2009 at 3:55 PM, Suhail Doshi <di...@gmail.com>wrote:
> Namit,
>
> For some reason when I apply:
> li57-125 ~/hive: patch -p0 hive-py.patch
>
> It seems just hang for a while.
>
> Suhail
>
>
> On Fri, Mar 20, 2009 at 12:58 PM, Namit Jain <nj...@facebook.com> wrote:
>
>> Download the patch file <patch_file>
>>
>>
>>
>> Go to the truck directory, and perform
>>
>>
>>
>> patch -p0 < <patch_file>
>>
>>
>>
>>
>>
>> Then compile and run tests.
>>
>>
>>
>>
>>
>>
>>
>> *From:* Suhail Doshi [mailto:digitalwarfare@gmail.com]
>> *Sent:* Friday, March 20, 2009 12:52 PM
>> *To:* hive-user@hadoop.apache.org
>> *Subject:* Re: Java/Python examples
>>
>>
>>
>> Ragu,
>>
>> How is exactly do you apply the patches?
>>
>> Suhail
>>
>> On Sun, Mar 15, 2009 at 3:57 PM, Raghu Murthy <ra...@facebook.com> wrote:
>>
>> You can call Hive in Java either directly or in client-server mode where
>> you
>> would have to run a thrift server.
>> Python and php clients are currently only available in patches at
>> https://issues.apache.org/jira/browse/HIVE-343 and
>> https://issues.apache.org/jira/browse/HIVE-299 respectively.
>>
>> If you apply the patches and then run ant package, the documentation at
>> http://wiki.apache.org/hadoop/Hive/HiveServer should be useful. Hope this
>> helps.
>>
>> raghu
>>
>>
>> On 3/15/09 12:22 PM, "Suhail Doshi" <su...@mixpanel.com> wrote:
>>
>> > I was browsing around the Hive wiki and couldn't manage to find any
>> examples
>> > on how people might be connecting to hive and executing queries through
>> > java/python. Anyone have anything available? (Python preferable)
>>
>>
>>
>>
>> --
>> http://mixpanel.com
>> Blog: http://blog.mixpanel.com
>>
>
>
>
> --
> http://mixpanel.com
> Blog: http://blog.mixpanel.com
>
--
http://mixpanel.com
Blog: http://blog.mixpanel.com
Re: Java/Python examples
Posted by Suhail Doshi <di...@gmail.com>.
Namit,
For some reason when I apply:
li57-125 ~/hive: patch -p0 hive-py.patch
It seems just hang for a while.
Suhail
On Fri, Mar 20, 2009 at 12:58 PM, Namit Jain <nj...@facebook.com> wrote:
> Download the patch file <patch_file>
>
>
>
> Go to the truck directory, and perform
>
>
>
> patch -p0 < <patch_file>
>
>
>
>
>
> Then compile and run tests.
>
>
>
>
>
>
>
> *From:* Suhail Doshi [mailto:digitalwarfare@gmail.com]
> *Sent:* Friday, March 20, 2009 12:52 PM
> *To:* hive-user@hadoop.apache.org
> *Subject:* Re: Java/Python examples
>
>
>
> Ragu,
>
> How is exactly do you apply the patches?
>
> Suhail
>
> On Sun, Mar 15, 2009 at 3:57 PM, Raghu Murthy <ra...@facebook.com> wrote:
>
> You can call Hive in Java either directly or in client-server mode where
> you
> would have to run a thrift server.
> Python and php clients are currently only available in patches at
> https://issues.apache.org/jira/browse/HIVE-343 and
> https://issues.apache.org/jira/browse/HIVE-299 respectively.
>
> If you apply the patches and then run ant package, the documentation at
> http://wiki.apache.org/hadoop/Hive/HiveServer should be useful. Hope this
> helps.
>
> raghu
>
>
> On 3/15/09 12:22 PM, "Suhail Doshi" <su...@mixpanel.com> wrote:
>
> > I was browsing around the Hive wiki and couldn't manage to find any
> examples
> > on how people might be connecting to hive and executing queries through
> > java/python. Anyone have anything available? (Python preferable)
>
>
>
>
> --
> http://mixpanel.com
> Blog: http://blog.mixpanel.com
>
--
http://mixpanel.com
Blog: http://blog.mixpanel.com
RE: Java/Python examples
Posted by Namit Jain <nj...@facebook.com>.
Download the patch file <patch_file>
Go to the truck directory, and perform
patch -p0 < <patch_file>
Then compile and run tests.
From: Suhail Doshi [mailto:digitalwarfare@gmail.com]
Sent: Friday, March 20, 2009 12:52 PM
To: hive-user@hadoop.apache.org
Subject: Re: Java/Python examples
Ragu,
How is exactly do you apply the patches?
Suhail
On Sun, Mar 15, 2009 at 3:57 PM, Raghu Murthy <ra...@facebook.com>> wrote:
You can call Hive in Java either directly or in client-server mode where you
would have to run a thrift server.
Python and php clients are currently only available in patches at
https://issues.apache.org/jira/browse/HIVE-343 and
https://issues.apache.org/jira/browse/HIVE-299 respectively.
If you apply the patches and then run ant package, the documentation at
http://wiki.apache.org/hadoop/Hive/HiveServer should be useful. Hope this
helps.
raghu
On 3/15/09 12:22 PM, "Suhail Doshi" <su...@mixpanel.com>> wrote:
> I was browsing around the Hive wiki and couldn't manage to find any examples
> on how people might be connecting to hive and executing queries through
> java/python. Anyone have anything available? (Python preferable)
--
http://mixpanel.com
Blog: http://blog.mixpanel.com
Re: Java/Python examples
Posted by Suhail Doshi <di...@gmail.com>.
Ragu,
How is exactly do you apply the patches?
Suhail
On Sun, Mar 15, 2009 at 3:57 PM, Raghu Murthy <ra...@facebook.com> wrote:
> You can call Hive in Java either directly or in client-server mode where
> you
> would have to run a thrift server.
> Python and php clients are currently only available in patches at
> https://issues.apache.org/jira/browse/HIVE-343 and
> https://issues.apache.org/jira/browse/HIVE-299 respectively.
>
> If you apply the patches and then run ant package, the documentation at
> http://wiki.apache.org/hadoop/Hive/HiveServer should be useful. Hope this
> helps.
>
> raghu
>
> On 3/15/09 12:22 PM, "Suhail Doshi" <su...@mixpanel.com> wrote:
>
> > I was browsing around the Hive wiki and couldn't manage to find any
> examples
> > on how people might be connecting to hive and executing queries through
> > java/python. Anyone have anything available? (Python preferable)
>
>
--
http://mixpanel.com
Blog: http://blog.mixpanel.com
Re: Java/Python examples
Posted by Raghu Murthy <ra...@facebook.com>.
You can call Hive in Java either directly or in client-server mode where you
would have to run a thrift server.
Python and php clients are currently only available in patches at
https://issues.apache.org/jira/browse/HIVE-343 and
https://issues.apache.org/jira/browse/HIVE-299 respectively.
If you apply the patches and then run ant package, the documentation at
http://wiki.apache.org/hadoop/Hive/HiveServer should be useful. Hope this
helps.
raghu
On 3/15/09 12:22 PM, "Suhail Doshi" <su...@mixpanel.com> wrote:
> I was browsing around the Hive wiki and couldn't manage to find any examples
> on how people might be connecting to hive and executing queries through
> java/python. Anyone have anything available? (Python preferable)