You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hive.apache.org by Jay Vyas <ja...@gmail.com> on 2013/12/28 04:27:45 UTC
Hive, datanucleus, jdbc, localmode.
Hi Hive:
I'm attempting to create a robust eclipse based dev environment for testing my hive jobs in localmode however I run into classnotfound errors depending on which maven dependencies I use. Also, it seems when I change these dependencies from hive 0.12 to hive 0.11, I get other errors related to hive trying to launch jobs via calling /usr/bin/hadoop.
This I am stuck: I can't run hive 12 in local java mode because of subtle datanucleus class and API inconsistencies which are tough to resolve, and when going to hive 11, it seems local mode is not natively detected via the jdbc URL...
So I have 2 questions:
0) how does hive 12 versus 11 implement local mode differently ?
And
1) What is the right way to in hive in pure java/ local environments?
The hive book suggests modifying configuration properties, for local mode..
but I also have found that in hive 0.12 , using the jdbc://hive connection URL automagically launches jobs in local mode..
However in 0.11 , I see calls to /usr/bin/hadoop when running java classes in local eclipse environment.
Thanks!
FYI to see an example of my pom.xml, you can checkout the github://jayunit100/bigpetstore pom.xml file.
Re: Hive, datanucleus, jdbc, localmode.
Posted by Jay Vyas <ja...@gmail.com>.
Yes this blog + hive_test should be merged into a jira and then officially integrated into hive in some way I think.
> On Dec 29, 2013, at 8:44 AM, Edward Capriolo <ed...@gmail.com> wrote:
>
> This article describes exactly what hive test does.
>
> :)
>
>
>> On Sat, Dec 28, 2013 at 9:18 PM, Jay Vyas <ja...@gmail.com> wrote:
>> Anyone try this yet :
>>
>> http://hadoop-pig-hive-thejas.blogspot.com/2013/04/running-hive-in-local-mode.html
>>
>> ?
>>
>>
>>> On Sat, Dec 28, 2013 at 8:09 PM, Jay Vyas <ja...@gmail.com> wrote:
>>> -Local mode should have clear instructions on how to run fully local hive jobs, with no hadoop installation.
>>>
>>> -I like the hive_test repo but I'm not yet sure hive_test is 100% up to date with the simplest strategy for testing hive workflows on the JVM.
>>>
>>>
>>>> On Dec 28, 2013, at 4:19 PM, Lefty Leverenz <le...@gmail.com> wrote:
>>>>
>>>> This sounds like something the documentation should cover.
>>>> What information should be added to the Local Mode section?
>>>> Should the wiki have a link to hive_test (for example, in Hive Developer FAQ)?
>>>>
>>>> -- Lefty
>>>>
>>>>
>>>>> On Sat, Dec 28, 2013 at 8:02 AM, Edward Capriolo <ed...@gmail.com> wrote:
>>>>> I do not think so. Local mode, just implies the job tracker is local (and some of the temp storage directories) it does not imply hive will use hadoop without forking.
>>>>>
>>>>>
>>>>>> On Sat, Dec 28, 2013 at 10:43 AM, Jay Vyas <ja...@gmail.com> wrote:
>>>>>> Thanks... But are you sure this is the only way? Or is there some magic way to run hive in local mode that we both are missing out on ?:)...
>>>>>>
>>>>>> - isn't hive in local mode supposed to be run simply via the jdbc://hive URL which runs local mode... Or maybe by the fork config parameter?
>>>>>>
>>>>>> - For example see the parameters in this file:
>>>>>>
>>>>>> https://github.com/riptano/brisk/blob/master/resources/hive/conf/hive-site.xml
>>>>>>
>>>>>>
>>>>>>> On Dec 28, 2013, at 10:22 AM, Edward Capriolo <ed...@gmail.com> wrote:
>>>>>>>
>>>>>>> You can follow along to what I do here.
>>>>>>>
>>>>>>> https://github.com/edwardcapriolo/hive_test
>>>>>>>
>>>>>>> Essentially hive requires a HADOOP_HOME because it always wants to fork a bin/hadoop process. Hive-test helps you unpack hadoop inside target and change your hadoop_home to some other directory.
>>>>>>>
>>>>>>> It would be nice if there was some other way to do this.
>>>>>>>
>>>>>>>
>>>>>>>> On Fri, Dec 27, 2013 at 10:27 PM, Jay Vyas <ja...@gmail.com> wrote:
>>>>>>>> Hi Hive:
>>>>>>>>
>>>>>>>> I'm attempting to create a robust eclipse based dev environment for testing my hive jobs in localmode however I run into classnotfound errors depending on which maven dependencies I use. Also, it seems when I change these dependencies from hive 0.12 to hive 0.11, I get other errors related to hive trying to launch jobs via calling /usr/bin/hadoop.
>>>>>>>>
>>>>>>>> This I am stuck: I can't run hive 12 in local java mode because of subtle datanucleus class and API inconsistencies which are tough to resolve, and when going to hive 11, it seems local mode is not natively detected via the jdbc URL...
>>>>>>>>
>>>>>>>> So I have 2 questions:
>>>>>>>>
>>>>>>>> 0) how does hive 12 versus 11 implement local mode differently ?
>>>>>>>>
>>>>>>>> And
>>>>>>>>
>>>>>>>> 1) What is the right way to in hive in pure java/ local environments?
>>>>>>>>
>>>>>>>> The hive book suggests modifying configuration properties, for local mode..
>>>>>>>>
>>>>>>>> but I also have found that in hive 0.12 , using the jdbc://hive connection URL automagically launches jobs in local mode..
>>>>>>>>
>>>>>>>> However in 0.11 , I see calls to /usr/bin/hadoop when running java classes in local eclipse environment.
>>>>>>>>
>>>>>>>> Thanks!
>>>>>>>>
>>>>>>>> FYI to see an example of my pom.xml, you can checkout the github://jayunit100/bigpetstore pom.xml file.
>>
>>
>>
>> --
>> Jay Vyas
>> http://jayunit100.blogspot.com
>
Re: Hive, datanucleus, jdbc, localmode.
Posted by Edward Capriolo <ed...@gmail.com>.
This article describes exactly what hive test does.
:)
On Sat, Dec 28, 2013 at 9:18 PM, Jay Vyas <ja...@gmail.com> wrote:
> Anyone try this yet :
>
>
> http://hadoop-pig-hive-thejas.blogspot.com/2013/04/running-hive-in-local-mode.html
>
> ?
>
>
> On Sat, Dec 28, 2013 at 8:09 PM, Jay Vyas <ja...@gmail.com> wrote:
>
>> -Local mode should have clear instructions on how to run fully local hive
>> jobs, with no hadoop installation.
>>
>> -I like the hive_test repo but I'm not yet sure hive_test is 100% up to
>> date with the simplest strategy for testing hive workflows on the JVM.
>>
>>
>> On Dec 28, 2013, at 4:19 PM, Lefty Leverenz <le...@gmail.com>
>> wrote:
>>
>> This sounds like something the documentation should cover.
>>
>> - What information should be added to the Local Mode<https://cwiki.apache.org/confluence/display/Hive/AdminManual+MetastoreAdmin#AdminManualMetastoreAdmin-LocalMetastore>section?
>> - Should the wiki have a link to hive_test<https://github.com/edwardcapriolo/hive_test> (for
>> example, in Hive Developer FAQ<https://cwiki.apache.org/confluence/display/Hive/HiveDeveloperFAQ>
>> )?
>>
>>
>> -- Lefty
>>
>>
>> On Sat, Dec 28, 2013 at 8:02 AM, Edward Capriolo <ed...@gmail.com>wrote:
>>
>>> I do not think so. Local mode, just implies the job tracker is local
>>> (and some of the temp storage directories) it does not imply hive will use
>>> hadoop without forking.
>>>
>>>
>>> On Sat, Dec 28, 2013 at 10:43 AM, Jay Vyas <ja...@gmail.com> wrote:
>>>
>>>> Thanks... But are you sure this is the only way? Or is there some magic
>>>> way to run hive in local mode that we both are missing out on ?:)...
>>>>
>>>> - isn't hive in local mode supposed to be run simply via the
>>>> jdbc://hive URL which runs local mode... Or maybe by the fork config
>>>> parameter?
>>>>
>>>> - For example see the parameters in this file:
>>>>
>>>>
>>>> https://github.com/riptano/brisk/blob/master/resources/hive/conf/hive-site.xml
>>>>
>>>>
>>>> On Dec 28, 2013, at 10:22 AM, Edward Capriolo <ed...@gmail.com>
>>>> wrote:
>>>>
>>>> You can follow along to what I do here.
>>>>
>>>> https://github.com/edwardcapriolo/hive_test
>>>>
>>>> Essentially hive requires a HADOOP_HOME because it always wants to fork
>>>> a bin/hadoop process. Hive-test helps you unpack hadoop inside target and
>>>> change your hadoop_home to some other directory.
>>>>
>>>> It would be nice if there was some other way to do this.
>>>>
>>>>
>>>> On Fri, Dec 27, 2013 at 10:27 PM, Jay Vyas <ja...@gmail.com>wrote:
>>>>
>>>>> Hi Hive:
>>>>>
>>>>> I'm attempting to create a robust eclipse based dev environment for
>>>>> testing my hive jobs in localmode however I run into classnotfound errors
>>>>> depending on which maven dependencies I use. Also, it seems when I change
>>>>> these dependencies from hive 0.12 to hive 0.11, I get other errors related
>>>>> to hive trying to launch jobs via calling /usr/bin/hadoop.
>>>>>
>>>>> This I am stuck: I can't run hive 12 in local java mode because of
>>>>> subtle datanucleus class and API inconsistencies which are tough to
>>>>> resolve, and when going to hive 11, it seems local mode is not natively
>>>>> detected via the jdbc URL...
>>>>>
>>>>> So I have 2 questions:
>>>>>
>>>>> 0) how does hive 12 versus 11 implement local mode differently ?
>>>>>
>>>>> And
>>>>>
>>>>> 1) What is the right way to in hive in pure java/ local environments?
>>>>>
>>>>> The hive book suggests modifying configuration properties, for local
>>>>> mode..
>>>>>
>>>>> but I also have found that in hive 0.12 , using the jdbc://hive
>>>>> connection URL automagically launches jobs in local mode..
>>>>>
>>>>> However in 0.11 , I see calls to /usr/bin/hadoop when running java
>>>>> classes in local eclipse environment.
>>>>>
>>>>> Thanks!
>>>>>
>>>>> FYI to see an example of my pom.xml, you can checkout the
>>>>> github://jayunit100/bigpetstore pom.xml file.
>>>>>
>>>>
>>>>
>>>
>>
>
>
> --
> Jay Vyas
> http://jayunit100.blogspot.com
>
Re: Hive, datanucleus, jdbc, localmode.
Posted by Jay Vyas <ja...@gmail.com>.
Anyone try this yet :
http://hadoop-pig-hive-thejas.blogspot.com/2013/04/running-hive-in-local-mode.html
?
On Sat, Dec 28, 2013 at 8:09 PM, Jay Vyas <ja...@gmail.com> wrote:
> -Local mode should have clear instructions on how to run fully local hive
> jobs, with no hadoop installation.
>
> -I like the hive_test repo but I'm not yet sure hive_test is 100% up to
> date with the simplest strategy for testing hive workflows on the JVM.
>
>
> On Dec 28, 2013, at 4:19 PM, Lefty Leverenz <le...@gmail.com>
> wrote:
>
> This sounds like something the documentation should cover.
>
> - What information should be added to the Local Mode<https://cwiki.apache.org/confluence/display/Hive/AdminManual+MetastoreAdmin#AdminManualMetastoreAdmin-LocalMetastore>section?
> - Should the wiki have a link to hive_test<https://github.com/edwardcapriolo/hive_test> (for
> example, in Hive Developer FAQ<https://cwiki.apache.org/confluence/display/Hive/HiveDeveloperFAQ>
> )?
>
>
> -- Lefty
>
>
> On Sat, Dec 28, 2013 at 8:02 AM, Edward Capriolo <ed...@gmail.com>wrote:
>
>> I do not think so. Local mode, just implies the job tracker is local (and
>> some of the temp storage directories) it does not imply hive will use
>> hadoop without forking.
>>
>>
>> On Sat, Dec 28, 2013 at 10:43 AM, Jay Vyas <ja...@gmail.com> wrote:
>>
>>> Thanks... But are you sure this is the only way? Or is there some magic
>>> way to run hive in local mode that we both are missing out on ?:)...
>>>
>>> - isn't hive in local mode supposed to be run simply via the jdbc://hive
>>> URL which runs local mode... Or maybe by the fork config parameter?
>>>
>>> - For example see the parameters in this file:
>>>
>>>
>>> https://github.com/riptano/brisk/blob/master/resources/hive/conf/hive-site.xml
>>>
>>>
>>> On Dec 28, 2013, at 10:22 AM, Edward Capriolo <ed...@gmail.com>
>>> wrote:
>>>
>>> You can follow along to what I do here.
>>>
>>> https://github.com/edwardcapriolo/hive_test
>>>
>>> Essentially hive requires a HADOOP_HOME because it always wants to fork
>>> a bin/hadoop process. Hive-test helps you unpack hadoop inside target and
>>> change your hadoop_home to some other directory.
>>>
>>> It would be nice if there was some other way to do this.
>>>
>>>
>>> On Fri, Dec 27, 2013 at 10:27 PM, Jay Vyas <ja...@gmail.com> wrote:
>>>
>>>> Hi Hive:
>>>>
>>>> I'm attempting to create a robust eclipse based dev environment for
>>>> testing my hive jobs in localmode however I run into classnotfound errors
>>>> depending on which maven dependencies I use. Also, it seems when I change
>>>> these dependencies from hive 0.12 to hive 0.11, I get other errors related
>>>> to hive trying to launch jobs via calling /usr/bin/hadoop.
>>>>
>>>> This I am stuck: I can't run hive 12 in local java mode because of
>>>> subtle datanucleus class and API inconsistencies which are tough to
>>>> resolve, and when going to hive 11, it seems local mode is not natively
>>>> detected via the jdbc URL...
>>>>
>>>> So I have 2 questions:
>>>>
>>>> 0) how does hive 12 versus 11 implement local mode differently ?
>>>>
>>>> And
>>>>
>>>> 1) What is the right way to in hive in pure java/ local environments?
>>>>
>>>> The hive book suggests modifying configuration properties, for local
>>>> mode..
>>>>
>>>> but I also have found that in hive 0.12 , using the jdbc://hive
>>>> connection URL automagically launches jobs in local mode..
>>>>
>>>> However in 0.11 , I see calls to /usr/bin/hadoop when running java
>>>> classes in local eclipse environment.
>>>>
>>>> Thanks!
>>>>
>>>> FYI to see an example of my pom.xml, you can checkout the
>>>> github://jayunit100/bigpetstore pom.xml file.
>>>>
>>>
>>>
>>
>
--
Jay Vyas
http://jayunit100.blogspot.com
Re: Hive, datanucleus, jdbc, localmode.
Posted by Jay Vyas <ja...@gmail.com>.
-Local mode should have clear instructions on how to run fully local hive jobs, with no hadoop installation.
-I like the hive_test repo but I'm not yet sure hive_test is 100% up to date with the simplest strategy for testing hive workflows on the JVM.
> On Dec 28, 2013, at 4:19 PM, Lefty Leverenz <le...@gmail.com> wrote:
>
> This sounds like something the documentation should cover.
> What information should be added to the Local Mode section?
> Should the wiki have a link to hive_test (for example, in Hive Developer FAQ)?
>
> -- Lefty
>
>
>> On Sat, Dec 28, 2013 at 8:02 AM, Edward Capriolo <ed...@gmail.com> wrote:
>> I do not think so. Local mode, just implies the job tracker is local (and some of the temp storage directories) it does not imply hive will use hadoop without forking.
>>
>>
>>> On Sat, Dec 28, 2013 at 10:43 AM, Jay Vyas <ja...@gmail.com> wrote:
>>> Thanks... But are you sure this is the only way? Or is there some magic way to run hive in local mode that we both are missing out on ?:)...
>>>
>>> - isn't hive in local mode supposed to be run simply via the jdbc://hive URL which runs local mode... Or maybe by the fork config parameter?
>>>
>>> - For example see the parameters in this file:
>>>
>>> https://github.com/riptano/brisk/blob/master/resources/hive/conf/hive-site.xml
>>>
>>>
>>>> On Dec 28, 2013, at 10:22 AM, Edward Capriolo <ed...@gmail.com> wrote:
>>>>
>>>> You can follow along to what I do here.
>>>>
>>>> https://github.com/edwardcapriolo/hive_test
>>>>
>>>> Essentially hive requires a HADOOP_HOME because it always wants to fork a bin/hadoop process. Hive-test helps you unpack hadoop inside target and change your hadoop_home to some other directory.
>>>>
>>>> It would be nice if there was some other way to do this.
>>>>
>>>>
>>>>> On Fri, Dec 27, 2013 at 10:27 PM, Jay Vyas <ja...@gmail.com> wrote:
>>>>> Hi Hive:
>>>>>
>>>>> I'm attempting to create a robust eclipse based dev environment for testing my hive jobs in localmode however I run into classnotfound errors depending on which maven dependencies I use. Also, it seems when I change these dependencies from hive 0.12 to hive 0.11, I get other errors related to hive trying to launch jobs via calling /usr/bin/hadoop.
>>>>>
>>>>> This I am stuck: I can't run hive 12 in local java mode because of subtle datanucleus class and API inconsistencies which are tough to resolve, and when going to hive 11, it seems local mode is not natively detected via the jdbc URL...
>>>>>
>>>>> So I have 2 questions:
>>>>>
>>>>> 0) how does hive 12 versus 11 implement local mode differently ?
>>>>>
>>>>> And
>>>>>
>>>>> 1) What is the right way to in hive in pure java/ local environments?
>>>>>
>>>>> The hive book suggests modifying configuration properties, for local mode..
>>>>>
>>>>> but I also have found that in hive 0.12 , using the jdbc://hive connection URL automagically launches jobs in local mode..
>>>>>
>>>>> However in 0.11 , I see calls to /usr/bin/hadoop when running java classes in local eclipse environment.
>>>>>
>>>>> Thanks!
>>>>>
>>>>> FYI to see an example of my pom.xml, you can checkout the github://jayunit100/bigpetstore pom.xml file.
>
Re: Hive, datanucleus, jdbc, localmode.
Posted by Lefty Leverenz <le...@gmail.com>.
This sounds like something the documentation should cover.
- What information should be added to the Local
Mode<https://cwiki.apache.org/confluence/display/Hive/AdminManual+MetastoreAdmin#AdminManualMetastoreAdmin-LocalMetastore>section?
- Should the wiki have a link to
hive_test<https://github.com/edwardcapriolo/hive_test> (for
example, in Hive Developer
FAQ<https://cwiki.apache.org/confluence/display/Hive/HiveDeveloperFAQ>
)?
-- Lefty
On Sat, Dec 28, 2013 at 8:02 AM, Edward Capriolo <ed...@gmail.com>wrote:
> I do not think so. Local mode, just implies the job tracker is local (and
> some of the temp storage directories) it does not imply hive will use
> hadoop without forking.
>
>
> On Sat, Dec 28, 2013 at 10:43 AM, Jay Vyas <ja...@gmail.com> wrote:
>
>> Thanks... But are you sure this is the only way? Or is there some magic
>> way to run hive in local mode that we both are missing out on ?:)...
>>
>> - isn't hive in local mode supposed to be run simply via the jdbc://hive
>> URL which runs local mode... Or maybe by the fork config parameter?
>>
>> - For example see the parameters in this file:
>>
>>
>> https://github.com/riptano/brisk/blob/master/resources/hive/conf/hive-site.xml
>>
>>
>> On Dec 28, 2013, at 10:22 AM, Edward Capriolo <ed...@gmail.com>
>> wrote:
>>
>> You can follow along to what I do here.
>>
>> https://github.com/edwardcapriolo/hive_test
>>
>> Essentially hive requires a HADOOP_HOME because it always wants to fork a
>> bin/hadoop process. Hive-test helps you unpack hadoop inside target and
>> change your hadoop_home to some other directory.
>>
>> It would be nice if there was some other way to do this.
>>
>>
>> On Fri, Dec 27, 2013 at 10:27 PM, Jay Vyas <ja...@gmail.com> wrote:
>>
>>> Hi Hive:
>>>
>>> I'm attempting to create a robust eclipse based dev environment for
>>> testing my hive jobs in localmode however I run into classnotfound errors
>>> depending on which maven dependencies I use. Also, it seems when I change
>>> these dependencies from hive 0.12 to hive 0.11, I get other errors related
>>> to hive trying to launch jobs via calling /usr/bin/hadoop.
>>>
>>> This I am stuck: I can't run hive 12 in local java mode because of
>>> subtle datanucleus class and API inconsistencies which are tough to
>>> resolve, and when going to hive 11, it seems local mode is not natively
>>> detected via the jdbc URL...
>>>
>>> So I have 2 questions:
>>>
>>> 0) how does hive 12 versus 11 implement local mode differently ?
>>>
>>> And
>>>
>>> 1) What is the right way to in hive in pure java/ local environments?
>>>
>>> The hive book suggests modifying configuration properties, for local
>>> mode..
>>>
>>> but I also have found that in hive 0.12 , using the jdbc://hive
>>> connection URL automagically launches jobs in local mode..
>>>
>>> However in 0.11 , I see calls to /usr/bin/hadoop when running java
>>> classes in local eclipse environment.
>>>
>>> Thanks!
>>>
>>> FYI to see an example of my pom.xml, you can checkout the
>>> github://jayunit100/bigpetstore pom.xml file.
>>>
>>
>>
>
Re: Hive, datanucleus, jdbc, localmode.
Posted by Edward Capriolo <ed...@gmail.com>.
I do not think so. Local mode, just implies the job tracker is local (and
some of the temp storage directories) it does not imply hive will use
hadoop without forking.
On Sat, Dec 28, 2013 at 10:43 AM, Jay Vyas <ja...@gmail.com> wrote:
> Thanks... But are you sure this is the only way? Or is there some magic
> way to run hive in local mode that we both are missing out on ?:)...
>
> - isn't hive in local mode supposed to be run simply via the jdbc://hive
> URL which runs local mode... Or maybe by the fork config parameter?
>
> - For example see the parameters in this file:
>
>
> https://github.com/riptano/brisk/blob/master/resources/hive/conf/hive-site.xml
>
>
> On Dec 28, 2013, at 10:22 AM, Edward Capriolo <ed...@gmail.com>
> wrote:
>
> You can follow along to what I do here.
>
> https://github.com/edwardcapriolo/hive_test
>
> Essentially hive requires a HADOOP_HOME because it always wants to fork a
> bin/hadoop process. Hive-test helps you unpack hadoop inside target and
> change your hadoop_home to some other directory.
>
> It would be nice if there was some other way to do this.
>
>
> On Fri, Dec 27, 2013 at 10:27 PM, Jay Vyas <ja...@gmail.com> wrote:
>
>> Hi Hive:
>>
>> I'm attempting to create a robust eclipse based dev environment for
>> testing my hive jobs in localmode however I run into classnotfound errors
>> depending on which maven dependencies I use. Also, it seems when I change
>> these dependencies from hive 0.12 to hive 0.11, I get other errors related
>> to hive trying to launch jobs via calling /usr/bin/hadoop.
>>
>> This I am stuck: I can't run hive 12 in local java mode because of subtle
>> datanucleus class and API inconsistencies which are tough to resolve, and
>> when going to hive 11, it seems local mode is not natively detected via the
>> jdbc URL...
>>
>> So I have 2 questions:
>>
>> 0) how does hive 12 versus 11 implement local mode differently ?
>>
>> And
>>
>> 1) What is the right way to in hive in pure java/ local environments?
>>
>> The hive book suggests modifying configuration properties, for local
>> mode..
>>
>> but I also have found that in hive 0.12 , using the jdbc://hive
>> connection URL automagically launches jobs in local mode..
>>
>> However in 0.11 , I see calls to /usr/bin/hadoop when running java
>> classes in local eclipse environment.
>>
>> Thanks!
>>
>> FYI to see an example of my pom.xml, you can checkout the
>> github://jayunit100/bigpetstore pom.xml file.
>>
>
>
Re: Hive, datanucleus, jdbc, localmode.
Posted by Jay Vyas <ja...@gmail.com>.
Thanks... But are you sure this is the only way? Or is there some magic way to run hive in local mode that we both are missing out on ?:)...
- isn't hive in local mode supposed to be run simply via the jdbc://hive URL which runs local mode... Or maybe by the fork config parameter?
- For example see the parameters in this file:
https://github.com/riptano/brisk/blob/master/resources/hive/conf/hive-site.xml
> On Dec 28, 2013, at 10:22 AM, Edward Capriolo <ed...@gmail.com> wrote:
>
> You can follow along to what I do here.
>
> https://github.com/edwardcapriolo/hive_test
>
> Essentially hive requires a HADOOP_HOME because it always wants to fork a bin/hadoop process. Hive-test helps you unpack hadoop inside target and change your hadoop_home to some other directory.
>
> It would be nice if there was some other way to do this.
>
>
>> On Fri, Dec 27, 2013 at 10:27 PM, Jay Vyas <ja...@gmail.com> wrote:
>> Hi Hive:
>>
>> I'm attempting to create a robust eclipse based dev environment for testing my hive jobs in localmode however I run into classnotfound errors depending on which maven dependencies I use. Also, it seems when I change these dependencies from hive 0.12 to hive 0.11, I get other errors related to hive trying to launch jobs via calling /usr/bin/hadoop.
>>
>> This I am stuck: I can't run hive 12 in local java mode because of subtle datanucleus class and API inconsistencies which are tough to resolve, and when going to hive 11, it seems local mode is not natively detected via the jdbc URL...
>>
>> So I have 2 questions:
>>
>> 0) how does hive 12 versus 11 implement local mode differently ?
>>
>> And
>>
>> 1) What is the right way to in hive in pure java/ local environments?
>>
>> The hive book suggests modifying configuration properties, for local mode..
>>
>> but I also have found that in hive 0.12 , using the jdbc://hive connection URL automagically launches jobs in local mode..
>>
>> However in 0.11 , I see calls to /usr/bin/hadoop when running java classes in local eclipse environment.
>>
>> Thanks!
>>
>> FYI to see an example of my pom.xml, you can checkout the github://jayunit100/bigpetstore pom.xml file.
>
Re: Hive, datanucleus, jdbc, localmode.
Posted by Edward Capriolo <ed...@gmail.com>.
You can follow along to what I do here.
https://github.com/edwardcapriolo/hive_test
Essentially hive requires a HADOOP_HOME because it always wants to fork a
bin/hadoop process. Hive-test helps you unpack hadoop inside target and
change your hadoop_home to some other directory.
It would be nice if there was some other way to do this.
On Fri, Dec 27, 2013 at 10:27 PM, Jay Vyas <ja...@gmail.com> wrote:
> Hi Hive:
>
> I'm attempting to create a robust eclipse based dev environment for
> testing my hive jobs in localmode however I run into classnotfound errors
> depending on which maven dependencies I use. Also, it seems when I change
> these dependencies from hive 0.12 to hive 0.11, I get other errors related
> to hive trying to launch jobs via calling /usr/bin/hadoop.
>
> This I am stuck: I can't run hive 12 in local java mode because of subtle
> datanucleus class and API inconsistencies which are tough to resolve, and
> when going to hive 11, it seems local mode is not natively detected via the
> jdbc URL...
>
> So I have 2 questions:
>
> 0) how does hive 12 versus 11 implement local mode differently ?
>
> And
>
> 1) What is the right way to in hive in pure java/ local environments?
>
> The hive book suggests modifying configuration properties, for local mode..
>
> but I also have found that in hive 0.12 , using the jdbc://hive
> connection URL automagically launches jobs in local mode..
>
> However in 0.11 , I see calls to /usr/bin/hadoop when running java classes
> in local eclipse environment.
>
> Thanks!
>
> FYI to see an example of my pom.xml, you can checkout the
> github://jayunit100/bigpetstore pom.xml file.
>