You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hive.apache.org by Sai Sai <sa...@yahoo.in> on 2013/03/04 11:24:40 UTC
Re: hive commands from a file
Just wondering if it is possible to run a bunch of hive commands from a file rather than one a time.
For ex:
1. Create external...
2. Load ...
3. Select * from ...
4....
Thanks
Sai
Re: hive commands from a file
Posted by Krishna Rao <kr...@gmail.com>.
Hi Sai,
just use the "-f" arg together with the file name. For details see:
https://cwiki.apache.org/Hive/languagemanual-cli.html
Krishna
On 4 March 2013 10:24, Sai Sai <sa...@yahoo.in> wrote:
> Just wondering if it is possible to run a bunch of hive commands from a
> file rather than one a time.
> For ex:
> 1. Create external...
> 2. Load ...
> 3. Select * from ...
> 4....
>
> Thanks
> Sai
>
Re: show tables in bin does not display the tables
Posted by Mark Grover <gr...@gmail.com>.
Sai,
This is because you are using the default embedded derby database as
metastore. When using the embedded derby metastore, the metadata is
stored in a relative location.
See the value of javax.jdo.option.ConnectionURL. By default, its value
is jdbc:derby:;databaseName=metastore_db;create=true
metastore_db is the directory that gets created to store the metadata.
If you put an absolute path there instead, e.g.
jdbc:derby:;databaseName=/a/path/that/exists/metastore_db;create=true
that would ensure that the same metadata is shared.
I, however, would recommend moving away from embedded derby metastore
and use MySQL or PostgreSQL for metastore instead. Googling should
give you some nice articles on how to do that.
Mark
On Tue, Mar 5, 2013 at 3:48 AM, Sai Sai <sa...@yahoo.in> wrote:
> Hello
>
> I have noticed when i execute the following command from hive shell in diff
> folders it behaves in diff ways and was wondering if this is right:
>
> show tables;
>
> from the bin folder under my hive install folder it just shows tab_name:
> ****************************
> myUser@ubuntu:~/work/hive-0.10.0-bin/bin$ ./hive
>
> hive> show tables;
>
> OK
> tab_name
> Time taken: 5.268 seconds
> ****************************
>
> But when i excecute the same command from my install folder:
>
> ****************************
> myUser@ubuntu:~/work/hive-0.10.0-bin/bin$ cd ..
>
> hive> show tables;
>
> OK
> tab_name
> employees
> sample_pages
> Time taken: 13.547 seconds
> ****************************
>
> Please let me know.
> Thanks
> Sai
Re: Done SemanticException Line 1:17 issue
Posted by Sai Sai <sa...@yahoo.in>.
Thanks for your help Nitin.
I have restarted my VM and tried again and it appears to work.
Thanks again.
Sai
________________________________
From: Sai Sai <sa...@yahoo.in>
To: "user@hive.apache.org" <us...@hive.apache.org>
Sent: Tuesday, 5 March 2013 4:42 AM
Subject: Re: SemanticException Line 1:17 issue
Thanks for your help Nitin, here is what it displays:
satish@ubuntu:~/work/hadoop-1.0.4/bin$ $HADOOP_HOME/bin/hadoop dfs -ls /tmp/
Warning: $HADOOP_HOME is deprecated.
Found 3 items
drwxr-xr-x - satish supergroup 0 2013-03-05 04:12 /tmp/hive-satish
-rw-r--r-- 1 satish supergroup 654 2013-03-04 02:41 /tmp/states.txt
drwxr-xr-x - satish supergroup 0 2013-02-16 00:46 /tmp/temp-1850940621
**************
I have done a search for the file states.txt and it refers to 3 places 2 of em refer to
proc/2693/cwd
but none of them refer to tmp folder.
Please let me know if you have any other suggestions.
In the meantime i will try with the [LOCAL] file and let you know.
Thanks
Sai
________________________________
From: Nitin Pawar <ni...@gmail.com>
To: user@hive.apache.org; Sai Sai <sa...@yahoo.in>
Sent: Tuesday, 5 March 2013 4:24 AM
Subject: Re: SemanticException Line 1:17 issue
it exists but where? on your hdfs or local linux filesystem ? so if you are checking the file with ls -l /tmp/ then add word local
ls can you provide output of $HADOOP_HOME/bin/hadoop dfs -ls /tmp/
LOAD DATA [LOCAL] INPATH 'filepath' [OVERWRITE] INTO TABLE tablename
If the keyword LOCAL is specified, then:
* the load command will look for filepath in the local file system. If a relative path is specified - it will be interpreted relative to the current directory of the user
On Tue, Mar 5, 2013 at 5:48 PM, Sai Sai <sa...@yahoo.in> wrote:
Yes Nitin it exists... but still getting the same issue.
>
>
>
>
>________________________________
> From: Nitin Pawar <ni...@gmail.com>
>To: user@hive.apache.org; Sai Sai <sa...@yahoo.in>
>Sent: Tuesday, 5 March 2013 4:14 AM
>Subject: Re: SemanticException Line 1:17 issue
>
>
>
>this file /tmp/o_small.tsv is on your local filesystem or hdfs?
>
>
>
>On Tue, Mar 5, 2013 at 5:39 PM, Sai Sai <sa...@yahoo.in> wrote:
>
>Hello
>>
>>
>>I have been stuck on this issue for quite some time and was wondering if anyone sees any problem with this that i am not seeing:
>>
>>
>>I have verified the file exists here and have also manually pasted the file into the tmp folder but still running into the same issue.
>>
>>
>>I am also wondering which folder this maps to in my local drive:
>>hdfs://ubuntu:9000/
>>
>>
>>***********************************
>>
>>
>>hive> LOAD DATA INPATH '/tmp/o_small.tsv' OVERWRITE INTO TABLE odata ;
>>FAILED: SemanticException Line 1:17 Invalid path ''/tmp/o_small.tsv'': No files matching path hdfs://ubuntu:9000/tmp/o_small.tsv
>>
>>
>>***********************************
>>I have verified the file exists here and have also manually pasted the file here but still running into the same issue.
>>Please let me know if u have any suggestions will be really appreciated.
>>ThanksSai
>>
>
>
>
>--
>Nitin Pawar
>
>
>
--
Nitin Pawar
Re: SemanticException Line 1:17 issue
Posted by Nitin Pawar <ni...@gmail.com>.
this file /tmp/o_small.tsv looks like is existing on your local filesystem
try load data local inpath
it should work
On Tue, Mar 5, 2013 at 6:12 PM, Sai Sai <sa...@yahoo.in> wrote:
> Thanks for your help Nitin, here is what it displays:
>
> satish@ubuntu:~/work/hadoop-1.0.4/bin$ $HADOOP_HOME/bin/hadoop dfs -ls
> /tmp/
>
> Warning: $HADOOP_HOME is deprecated.
> Found 3 items
>
> drwxr-xr-x - satish supergroup 0 2013-03-05 04:12
> /tmp/hive-satish
> -rw-r--r-- 1 satish supergroup 654 2013-03-04 02:41
> /tmp/states.txt
> drwxr-xr-x - satish supergroup 0 2013-02-16 00:46
> /tmp/temp-1850940621
>
> **************
> I have done a search for the file states.txt and it refers to 3 places 2
> of em refer to
> proc/2693/cwd
>
> but none of them refer to tmp folder.
>
> Please let me know if you have any other suggestions.
> In the meantime i will try with the [LOCAL] file and let you know.
> Thanks
> Sai
>
> ------------------------------
> *From:* Nitin Pawar <ni...@gmail.com>
> *To:* user@hive.apache.org; Sai Sai <sa...@yahoo.in>
> *Sent:* Tuesday, 5 March 2013 4:24 AM
>
> *Subject:* Re: SemanticException Line 1:17 issue
>
> it exists but where? on your hdfs or local linux filesystem ? so if you
> are checking the file with ls -l /tmp/ then add word local
>
> ls can you provide output of $HADOOP_HOME/bin/hadoop dfs -ls /tmp/
>
> LOAD DATA [LOCAL] INPATH 'filepath' [OVERWRITE] INTO TABLE tablename
>
> If the keyword LOCAL is specified, then:
>
> - the load command will look for *filepath* in the local file system. If a relative path is specified - it will be interpreted relative to the current directory of the user
>
>
>
> On Tue, Mar 5, 2013 at 5:48 PM, Sai Sai <sa...@yahoo.in> wrote:
>
> Yes Nitin it exists... but still getting the same issue.
>
> ------------------------------
> *From:* Nitin Pawar <ni...@gmail.com>
> *To:* user@hive.apache.org; Sai Sai <sa...@yahoo.in>
> *Sent:* Tuesday, 5 March 2013 4:14 AM
> *Subject:* Re: SemanticException Line 1:17 issue
>
> this file /tmp/o_small.tsv is on your local filesystem or hdfs?
>
>
> On Tue, Mar 5, 2013 at 5:39 PM, Sai Sai <sa...@yahoo.in> wrote:
>
> Hello
>
> I have been stuck on this issue for quite some time and was wondering if
> anyone sees any problem with this that i am not seeing:
>
> I have verified the file exists here and have also manually pasted the
> file into the tmp folder but still running into the same issue.
>
> I am also wondering which folder this maps to in my local drive:
> hdfs://ubuntu:9000/
>
> ***********************************
>
> hive> LOAD DATA INPATH '/tmp/o_small.tsv' OVERWRITE INTO TABLE odata ;
> FAILED: SemanticException Line 1:17 Invalid path ''/tmp/o_small.tsv'': No
> files matching path hdfs://ubuntu:9000/tmp/o_small.tsv
>
> ***********************************
> I have verified the file exists here and have also manually pasted the
> file here but still running into the same issue.
> Please let me know if u have any suggestions will be really appreciated.
> Thanks
> Sai
>
>
>
>
> --
> Nitin Pawar
>
>
>
>
>
> --
> Nitin Pawar
>
>
>
--
Nitin Pawar
Re: SemanticException Line 1:17 issue
Posted by Sai Sai <sa...@yahoo.in>.
Thanks for your help Nitin, here is what it displays:
satish@ubuntu:~/work/hadoop-1.0.4/bin$ $HADOOP_HOME/bin/hadoop dfs -ls /tmp/
Warning: $HADOOP_HOME is deprecated.
Found 3 items
drwxr-xr-x - satish supergroup 0 2013-03-05 04:12 /tmp/hive-satish
-rw-r--r-- 1 satish supergroup 654 2013-03-04 02:41 /tmp/states.txt
drwxr-xr-x - satish supergroup 0 2013-02-16 00:46 /tmp/temp-1850940621
**************
I have done a search for the file states.txt and it refers to 3 places 2 of em refer to
proc/2693/cwd
but none of them refer to tmp folder.
Please let me know if you have any other suggestions.
In the meantime i will try with the [LOCAL] file and let you know.
Thanks
Sai
________________________________
From: Nitin Pawar <ni...@gmail.com>
To: user@hive.apache.org; Sai Sai <sa...@yahoo.in>
Sent: Tuesday, 5 March 2013 4:24 AM
Subject: Re: SemanticException Line 1:17 issue
it exists but where? on your hdfs or local linux filesystem ? so if you are checking the file with ls -l /tmp/ then add word local
ls can you provide output of $HADOOP_HOME/bin/hadoop dfs -ls /tmp/
LOAD DATA [LOCAL] INPATH 'filepath' [OVERWRITE] INTO TABLE tablename
If the keyword LOCAL is specified, then:
* the load command will look for filepath in the local file system. If a relative path is specified - it will be interpreted relative to the current directory of the user
On Tue, Mar 5, 2013 at 5:48 PM, Sai Sai <sa...@yahoo.in> wrote:
Yes Nitin it exists... but still getting the same issue.
>
>
>
>
>________________________________
> From: Nitin Pawar <ni...@gmail.com>
>To: user@hive.apache.org; Sai Sai <sa...@yahoo.in>
>Sent: Tuesday, 5 March 2013 4:14 AM
>Subject: Re: SemanticException Line 1:17 issue
>
>
>
>this file /tmp/o_small.tsv is on your local filesystem or hdfs?
>
>
>
>On Tue, Mar 5, 2013 at 5:39 PM, Sai Sai <sa...@yahoo.in> wrote:
>
>Hello
>>
>>
>>I have been stuck on this issue for quite some time and was wondering if anyone sees any problem with this that i am not seeing:
>>
>>
>>I have verified the file exists here and have also manually pasted the file into the tmp folder but still running into the same issue.
>>
>>
>>I am also wondering which folder this maps to in my local drive:
>>hdfs://ubuntu:9000/
>>
>>
>>***********************************
>>
>>
>>hive> LOAD DATA INPATH '/tmp/o_small.tsv' OVERWRITE INTO TABLE odata ;
>>FAILED: SemanticException Line 1:17 Invalid path ''/tmp/o_small.tsv'': No files matching path hdfs://ubuntu:9000/tmp/o_small.tsv
>>
>>
>>***********************************
>>I have verified the file exists here and have also manually pasted the file here but still running into the same issue.
>>Please let me know if u have any suggestions will be really appreciated.
>>ThanksSai
>>
>
>
>
>--
>Nitin Pawar
>
>
>
--
Nitin Pawar
Re: Location of external table in hdfs
Posted by Dean Wampler <de...@thinkbiganalytics.com>.
/tmp/states in HDFS.
On Tue, Mar 5, 2013 at 10:56 AM, Sai Sai <sa...@yahoo.in> wrote:
> I have created an external table like below and wondering where (folder)
> in hdfs i can find this:
>
> CREATE EXTERNAL TABLE states(abbreviation string, full_name string) ROW
> FORMAT DELIMITED FIELDS TERMINATED BY '\t' LOCATION '/tmp/states' ;
>
> Any help is really appreciated.
> Thanks
> Sai
>
--
*Dean Wampler, Ph.D.*
thinkbiganalytics.com
+1-312-339-1330
Re: Where is the location of hive queries
Posted by Dean Wampler <de...@thinkbiganalytics.com>.
Or use a variant of the INSERT statement to write to a directory or a table.
On Wed, Mar 6, 2013 at 10:05 AM, Nitin Pawar <ni...@gmail.com>wrote:
> the results are not stored to any file .. they are available on console
> only
>
> if you want to save to the results then write execute your query like hive
> -e "query" > file
>
>
> On Wed, Mar 6, 2013 at 9:32 PM, Sai Sai <sa...@yahoo.in> wrote:
>
>> After we run a query in hive shell as:
>> Select * from myTable;
>>
>> Are these results getting saved to any file apart from the
>> console/terminal display.
>> If so where is the location of the results.
>> Thanks
>> Sai
>>
>
>
>
> --
> Nitin Pawar
>
--
*Dean Wampler, Ph.D.*
thinkbiganalytics.com
+1-312-339-1330
Re: Where is the location of hive queries
Posted by Nitin Pawar <ni...@gmail.com>.
the results are not stored to any file .. they are available on console only
if you want to save to the results then write execute your query like hive
-e "query" > file
On Wed, Mar 6, 2013 at 9:32 PM, Sai Sai <sa...@yahoo.in> wrote:
> After we run a query in hive shell as:
> Select * from myTable;
>
> Are these results getting saved to any file apart from the
> console/terminal display.
> If so where is the location of the results.
> Thanks
> Sai
>
--
Nitin Pawar
Re: Accessing sub column in hive
Posted by Dean Wampler <de...@thinkbiganalytics.com>.
I recognize this example ;)
You reference struct elements with the dot notation, as Bejoy said, map
elements with what you tried, deductions['Federal taxes'], and arrays by
index, starting from zero, subordinates[0].
On Fri, Mar 8, 2013 at 6:35 AM, <be...@yahoo.com> wrote:
> Hi Sai
>
>
> You can do it as
> Select address.country from employees;
>
> Regards
> Bejoy KS
>
> Sent from remote device, Please excuse typos
> ------------------------------
> *From: * Bennie Schut <bs...@ebuddy.com>
> *Date: *Fri, 8 Mar 2013 09:09:49 +0100
> *To: *user@hive.apache.org<us...@hive.apache.org>; 'Sai Sai'<
> saigraph@yahoo.in>
> *ReplyTo: * user@hive.apache.org
> *Subject: *RE: Accessing sub column in hive
>
> Perhaps worth posting the error. Some might know what the error means.****
>
> ** **
>
> Also a bit unrelated to hive but please do yourself a favor and don’t use
> float to store monetary values like salary. You will get rounding issues at
> some point in time when you do arithmetic on them. Considering you are
> using hadoop you probably have a lot of data so adding it all up will get
> you there really really fast.
> http://stackoverflow.com/questions/3730019/why-not-use-double-or-float-to-represent-currency
> ****
>
> ** **
>
> ** **
>
> *From:* Sai Sai [mailto:saigraph@yahoo.in]
> *Sent:* Thursday, March 07, 2013 12:54 PM
> *To:* user@hive.apache.org
> *Subject:* Re: Accessing sub column in hive****
>
> ** **
>
> I have a table created like this successfully:****
>
> ** **
>
> CREATE TABLE IF NOT EXISTS employees (name STRING,salary
> FLOAT,subordinates ARRAY<STRING>,deductions MAP<STRING,FLOAT>,address
> STRUCT<street:STRING, city:STRING, state:STRING, zip:INT, country:STRING>)
> ****
>
> ** **
>
> I would like to access/display country column from my address struct.****
>
> I have tried this:****
>
> ** **
>
> select address["country"] from employees;****
>
> ** **
>
> I get an error.****
>
> ** **
>
> Please help.****
>
> ** **
>
> Thanks****
>
> Sai****
>
--
*Dean Wampler, Ph.D.*
thinkbiganalytics.com
+1-312-339-1330
Re: Accessing sub column in hive
Posted by be...@yahoo.com.
Hi Sai
You can do it as
Select address.country from employees;
Regards
Bejoy KS
Sent from remote device, Please excuse typos
-----Original Message-----
From: Bennie Schut <bs...@ebuddy.com>
Date: Fri, 8 Mar 2013 09:09:49
To: user@hive.apache.org<us...@hive.apache.org>; 'Sai Sai'<sa...@yahoo.in>
Reply-To: user@hive.apache.org
Subject: RE: Accessing sub column in hive
Perhaps worth posting the error. Some might know what the error means.
Also a bit unrelated to hive but please do yourself a favor and don't use float to store monetary values like salary. You will get rounding issues at some point in time when you do arithmetic on them. Considering you are using hadoop you probably have a lot of data so adding it all up will get you there really really fast. http://stackoverflow.com/questions/3730019/why-not-use-double-or-float-to-represent-currency
From: Sai Sai [mailto:saigraph@yahoo.in]
Sent: Thursday, March 07, 2013 12:54 PM
To: user@hive.apache.org
Subject: Re: Accessing sub column in hive
I have a table created like this successfully:
CREATE TABLE IF NOT EXISTS employees (name STRING,salary FLOAT,subordinates ARRAY<STRING>,deductions MAP<STRING,FLOAT>,address STRUCT<street:STRING, city:STRING, state:STRING, zip:INT, country:STRING>)
I would like to access/display country column from my address struct.
I have tried this:
select address["country"] from employees;
I get an error.
Please help.
Thanks
Sai
RE: Accessing sub column in hive
Posted by Bennie Schut <bs...@ebuddy.com>.
Perhaps worth posting the error. Some might know what the error means.
Also a bit unrelated to hive but please do yourself a favor and don't use float to store monetary values like salary. You will get rounding issues at some point in time when you do arithmetic on them. Considering you are using hadoop you probably have a lot of data so adding it all up will get you there really really fast. http://stackoverflow.com/questions/3730019/why-not-use-double-or-float-to-represent-currency
From: Sai Sai [mailto:saigraph@yahoo.in]
Sent: Thursday, March 07, 2013 12:54 PM
To: user@hive.apache.org
Subject: Re: Accessing sub column in hive
I have a table created like this successfully:
CREATE TABLE IF NOT EXISTS employees (name STRING,salary FLOAT,subordinates ARRAY<STRING>,deductions MAP<STRING,FLOAT>,address STRUCT<street:STRING, city:STRING, state:STRING, zip:INT, country:STRING>)
I would like to access/display country column from my address struct.
I have tried this:
select address["country"] from employees;
I get an error.
Please help.
Thanks
Sai
Re: java.lang.NoClassDefFoundError: com/jayway/jsonpath/PathUtil
Posted by john smith <js...@gmail.com>.
Hi,
It clearly is a classpath issue! When you do a select * from tab , it
works because HIve just fetches the data from HDFS using a FetchTask and
doesn't start any MR job (It probably uses json jar in your local hive lib
directory to deserialize and limit the rows to 5 and hence not raising any
error).
However the error you are facing in the 2nd query involving an MR job is
because mapper (on some remote machine which doesn't have json jar in its
class path) is unable to locate your json jar ! Did u try doing what Dean
suggested?
Thanks
On Sun, Mar 10, 2013 at 1:49 PM, Sai Sai <sa...@yahoo.in> wrote:
> Just wondering if anyone has any suggestions:
>
> This executes successfully:
>
> hive> select * from twitter limit 5;
>
> This does not work:
>
> hive> select tweet_id from twitter limit 5; // I have given the exception
> info below:
>
> Here is the output of this:
>
> hive> select * from twitter limit 5;
> OK
>
> tweet_id created_at text user_id user_screen_name user_lang
> 122106088022745088 Fri Oct 07 00:28:54 +0000 2011 wkwkw -_- ayo saja
> mba RT @yullyunet: Sepupuuu, kita lanjalan yok.. Kita karokoe-an.. Ajak mas
> galih jg kalo dia mau.. "@Dindnf: doremifas 124735434 Dindnf en
> 122106088018558976 Fri Oct 07 00:28:54 +0000 2011 @egg486 특별히
> 준비했습니다! 252828803 CocaCola_Korea ko
> 122106088026939392 Fri Oct 07 00:28:54 +0000 2011 My offer of free
> gobbies for all if @amityaffliction play Blair snitch project still
> stands. 168590073 SarahYoungBlood en
> 122106088035328001 Fri Oct 07 00:28:54 +0000 2011 the girl nxt to me
> in the lib got her headphones in dancing and singing loud af like she the
> only one here haha 267296295 MONEYyDREAMS_ en
> 122106088005971968 Fri Oct 07 00:28:54 +0000 2011 @KUnYoong_B2UTY
> Bị lsao đấy 269182160 b2st_b2utyhp en
> Time taken: 0.154 seconds
>
> This does not work:
>
> hive> select tweet_id from twitter limit 5;
>
>
> Total MapReduce jobs = 1
> Launching Job 1 out of 1
> Number of reduce tasks is set to 0 since there's no reduce operator
> Starting Job = job_201303050432_0094, Tracking URL =
> http://ubuntu:50030/jobdetails.jsp?jobid=job_201303050432_0094
> Kill Command = /home/satish/work/hadoop-1.0.4/libexec/../bin/hadoop job
> -kill job_201303050432_0094
> Hadoop job information for Stage-1: number of mappers: 1; number of
> reducers: 0
> 2013-03-10 00:14:44,509 Stage-1 map = 0%, reduce = 0%
> 2013-03-10 00:15:14,613 Stage-1 map = 100%, reduce = 100%
> Ended Job = job_201303050432_0094 with errors
> Error during job, obtaining debugging information...
> Job Tracking URL:
> http://ubuntu:50030/jobdetails.jsp?jobid=job_201303050432_0094
> Examining task ID: task_201303050432_0094_m_000002 (and more) from job
> job_201303050432_0094
>
> Task with the most failures(4):
> -----
> Task ID:
> task_201303050432_0094_m_000000
>
> URL:
>
> http://ubuntu:50030/taskdetails.jsp?jobid=job_201303050432_0094&tipid=task_201303050432_0094_m_000000
> -----
> Diagnostic Messages for this Task:
> java.lang.RuntimeException: Error in configuring object
> at
> org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:93)
> at
> org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64)
> at
> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:432)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372)
> at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:416)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
> at org.apache.hadoop.mapred.Child.main(Child.java:249)
> Caused by: java.lang.reflect.InvocationTargetException
>
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at
> org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88)
> ... 9 more
> Caused by: java.lang.RuntimeException: Error in configuring object
> at
> org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:93)
> at
> org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64)
> at
> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
> at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:34)
> ... 14 more
> Caused by: java.lang.reflect.InvocationTargetException
>
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at
> org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88)
> ... 17 more
> Caused by: java.lang.RuntimeException: Map operator initialization failed
> at
> org.apache.hadoop.hive.ql.exec.ExecMapper.configure(ExecMapper.java:121)
> ... 22 more
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
> java.lang.ClassNotFoundException:
> org.apache.hadoop.hive.contrib.serde2.JsonSerde
> at
> org.apache.hadoop.hive.ql.exec.MapOperator.setChildren(MapOperator.java:420)
> at
> org.apache.hadoop.hive.ql.exec.ExecMapper.configure(ExecMapper.java:90)
> ... 22 more
> Caused by: java.lang.ClassNotFoundException:
> org.apache.hadoop.hive.contrib.serde2.JsonSerde
>
> at java.net.URLClassLoader$1.run(URLClassLoader.java:217)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:205)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:321)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:294)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:266)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:264)
> at
> org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:820)
> at
> org.apache.hadoop.hive.ql.exec.MapOperator.initObjectInspector(MapOperator.java:243)
> at
> org.apache.hadoop.hive.ql.exec.MapOperator.setChildren(MapOperator.java:380)
> ... 23 more
>
>
> FAILED: Execution Error, return code 2 from
> org.apache.hadoop.hive.ql.exec.MapRedTask
> MapReduce Jobs Launched:
> Job 0: Map: 1 HDFS Read: 0 HDFS Write: 0 FAIL
> Total MapReduce CPU Time Spent: 0 msec
>
> Thanks
> Sai
> ------------------------------
> *From:* Dean Wampler <de...@thinkbiganalytics.com>
> *To:* user@hive.apache.org; Sai Sai <sa...@yahoo.in>
> *Sent:* Friday, 8 March 2013 5:22 AM
> *Subject:* Re: java.lang.NoClassDefFoundError:
> com/jayway/jsonpath/PathUtil
>
> Unfortunately, you have to also add the json jars to Hive's class path
> before it starts, e.g.,
>
> env HADOOP_CLASSPATH=/path/to/lib/*.jar hive
>
> Use the appropriate path to your lib directory.
>
> On Fri, Mar 8, 2013 at 4:53 AM, Sai Sai <sa...@yahoo.in> wrote:
>
> I have added the jar files successfully like this:
>
>
> hive (testdb)> ADD JAR lib/hive-json-serde-0.3.jar;
> Added lib/hive-json-serde-0.3.jar to class path
> Added resource: lib/hive-json-serde-0.3.jar
>
>
> hive (testdb)> ADD JAR lib/json-path-0.5.4.jar;
> Added lib/json-path-0.5.4.jar to class path
> Added resource: lib/json-path-0.5.4.jar
>
>
> hive (testdb)> ADD JAR lib/json-smart-1.0.6.3.jar;
> Added lib/json-smart-1.0.6.3.jar to class path
> Added resource: lib/json-smart-1.0.6.3.jar
>
>
> After this i am getting this error:
>
>
> CREATE EXTERNAL TABLE IF NOT EXISTS twitter (tweet_id BIGINT,created_at
> STRING,text STRING,user_id BIGINT, user_screen_name STRING,user_lang
> STRING) ROW FORMAT SERDE "org.apache.hadoop.hive.contrib.serde2.JsonSerde"
> WITH SERDEPROPERTIES (
> "tweet_id"="$.id","created_at"="$.created_at","text"="$.text","user_id"="$.
> user.id","user_screen_name"="$.user.screen_name",
> "user_lang"="$.user.lang") LOCATION '/home/satish/data/twitter/input';
> java.lang.NoClassDefFoundError: com/jayway/jsonpath/PathUtil
> at org.apache.hadoop.hive.contrib.serde2.JsonSerde.initialize(Unknown
> Source)
> at
> org.apache.hadoop.hive.metastore.MetaStoreUtils.getDeserializer(MetaStoreUtils.java:207)
> at
> org.apache.hadoop.hive.ql.metadata.Table.getDeserializerFromMetaStore(Table.java:266)
> at
> org.apache.hadoop.hive.ql.metadata.Table.getDeserializer(Table.java:259)
> at org.apache.hadoop.hive.ql.metadata.Table.getCols(Table.java:585)
> at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:550)
> at
> org.apache.hadoop.hive.ql.exec.DDLTask.createTable(DDLTask.java:3698)
> at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:253)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:138)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:57)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1336)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1122)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:935)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:259)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:216)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:412)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:755)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:613)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
> Caused by: java.lang.ClassNotFoundException: com.jayway.jsonpath.PathUtil
> at java.net.URLClassLoader$1.run(URLClassLoader.java:217)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:205)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:321)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:266)
> ... 23 more
> FAILED: Execution Error, return code -101 from
> org.apache.hadoop.hive.ql.exec.DDLTask
>
>
> Any help would be really appreciated.
> Thanks
> Sai
>
>
>
>
> --
> *Dean Wampler, Ph.D.*
> thinkbiganalytics.com
> +1-312-339-1330
>
>
>
>
Re: Where to find the external table file in HDFS
Posted by Sai Sai <sa...@yahoo.in>.
I have created an external table states and loaded it from a file under /tmp/states.txt
Then in the url:
http://localhost.localdomain:50070/dfshealth.jsp
I have looked to see if this file states table exists and do not see it.
Just wondering if it is saved in hdfs or not.
How many days will the files exist under /tmp folder.
Thanks
Sai
Re: Where can we see the results of Select * from states
Posted by Jov <am...@amutu.com>.
you can write data into filesystem from query using "INSERT OVERWRITE
[LOCAL] DIRECTORY directory1 SELECT ... FROM ..."
more detail:
https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DML#LanguageManualDML-Writingdataintofilesystemfromqueries
2013/5/24 Sai Sai <sa...@yahoo.in>
> I have created an external table called states under a database called
> test,
> Then loaded the table successfully;
> The i have tried:
>
> Select * from states;
>
> It successfully executes MR and displays the results in the console but
> wondering where to look in hdfs to see these results.
>
> I have looked under all the dirs in filesystem for the below url but
> cannot see the results part file.
>
> http://localhost.localdomain:50070/dfshealth.jsp
>
> Also if i would like the results to save to a specific file from a query
> how to do it?
>
> For Ex:
> Select * from states > myStates.txt ;
> Is there something like this.
> Thanks
> Sai
>
>
>
--
Jov
blog: http:amutu.com/blog <http://amutu.com/blog>
Re: Where can we see the results of Select * from states
Posted by Sai Sai <sa...@yahoo.in>.
I have created an external table called states under a database called test,
Then loaded the table successfully;
The i have tried:
Select * from states;
It successfully executes MR and displays the results in the console but wondering where to look in hdfs to see these results.
I have looked under all the dirs in filesystem for the below url but cannot see the results part file.
http://localhost.localdomain:50070/dfshealth.jsp
Also if i would like the results to save to a specific file from a query how to do it?
For Ex:
Select * from states > myStates.txt ;
Is there something like this.
Thanks
Sai
Re: Difference between like %A% and %a%
Posted by Anthony Urso <an...@cs.ucla.edu>.
Postgres/Vertica and their ilk have ILIKE which is a case-insensitive
version of LIKE, in addition to the case-sensitive LIKE. Works well having
both.
Cheers,
Anthony
On Fri, May 24, 2013 at 8:58 AM, Edward Capriolo <ed...@gmail.com>wrote:
> It is not as simple of a problem as you think. Mysql has the same problem
> just most everyone uses a default charset and comparator.
>
> http://www.bluebox.net/about/blog/2009/07/mysql_encoding/
>
> You do you account for foreign characters like the a~ etc. is that > then
> A and less then <
>
>
> On Fri, May 24, 2013 at 11:41 AM, Dean Wampler <de...@gmail.com>wrote:
>
>> If backwards compatibility wasn't an issue, the hive code that implements
>> LIKE could be changed to convert the fields and LIKE strings to lower case
>> before comparing ;) Of course, there is overhead doing that.
>>
>> On Fri, May 24, 2013 at 9:50 AM, Edward Capriolo <ed...@gmail.com>wrote:
>>
>>> Also I am thinking that the rlike is based on regex and can be told to
>>> do case insensitive matching.
>>>
>>>
>>> On Fri, May 24, 2013 at 9:16 AM, Dean Wampler <de...@gmail.com>wrote:
>>>
>>>> Hortonworks has announced plans to make Hive more SQL compliant. I
>>>> suspect bugs like this will be addressed sooner or later. It will be
>>>> necessary to handle backwards compatibility, but that could be handled with
>>>> a hive property that enables one or the other behaviors.
>>>>
>>>> On Fri, May 24, 2013 at 8:07 AM, John Omernik <jo...@omernik.com> wrote:
>>>>
>>>>> I have mentioned this before, and I think this a big miss by the Hive
>>>>> team. Like, by default in many SQL RDBMS (like MSSQL or MYSQL) is not
>>>>> case sensitive. Thus when you have new users moving over to Hive, if they
>>>>> see a command like "like" they will assume similarity (like many other SQL
>>>>> like qualities) and thus false negatives may ensue. Even though it's
>>>>> different by default (I am ok with this ... I guess, my personal preference
>>>>> is that it matches the defaults on other systems, and outside of that
>>>>> (which I am, in in the end fine with, just grumbly :) ) give us the ability
>>>>> to set that behavior in the hive-site.xml. That way when an org realizes
>>>>> that it is different, and their users are all getting false negatives, they
>>>>> can just update the hive-site and fix the problem rather than have to
>>>>> include it in training that may or may not work. I've added this comment
>>>>> to https://issues.apache.org/jira/browse/HIVE-4070#comment-13666278 for fun. :)
>>>>>
>>>>> Please? :)
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Fri, May 24, 2013 at 7:53 AM, Dean Wampler <de...@gmail.com>wrote:
>>>>>
>>>>>> Your where clause looks at the abbreviation, requiring 'A', not the
>>>>>> state name. You got the correct answer.
>>>>>>
>>>>>>
>>>>>> On Fri, May 24, 2013 at 6:21 AM, Sai Sai <sa...@yahoo.in> wrote:
>>>>>>
>>>>>>> But it should get more results for this:
>>>>>>>
>>>>>>> %a%
>>>>>>>
>>>>>>> than for
>>>>>>>
>>>>>>> %A%
>>>>>>>
>>>>>>> Please let me know if i am missing something.
>>>>>>> Thanks
>>>>>>> Sai
>>>>>>>
>>>>>>>
>>>>>>> ------------------------------
>>>>>>> *From:* Jov <am...@amutu.com>
>>>>>>> *To:* user@hive.apache.org; Sai Sai <sa...@yahoo.in>
>>>>>>> *Sent:* Friday, 24 May 2013 4:39 PM
>>>>>>> *Subject:* Re: Difference between like %A% and %a%
>>>>>>>
>>>>>>>
>>>>>>> 2013/5/24 Sai Sai <sa...@yahoo.in>
>>>>>>>
>>>>>>> abbreviation l
>>>>>>>
>>>>>>>
>>>>>>> unlike MySQL, string in Hive is case sensitive，so '%A%' is not equal
>>>>>>> with '%a%'.
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Jov
>>>>>>> blog: http:amutu.com/blog <http://amutu.com/blog>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Dean Wampler, Ph.D.
>>>>>> @deanwampler
>>>>>> http://polyglotprogramming.com
>>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Dean Wampler, Ph.D.
>>>> @deanwampler
>>>> http://polyglotprogramming.com
>>>
>>>
>>>
>>
>>
>> --
>> Dean Wampler, Ph.D.
>> @deanwampler
>> http://polyglotprogramming.com
>
>
>
Re: Difference between like %A% and %a%
Posted by Edward Capriolo <ed...@gmail.com>.
It is not as simple of a problem as you think. Mysql has the same problem
just most everyone uses a default charset and comparator.
http://www.bluebox.net/about/blog/2009/07/mysql_encoding/
You do you account for foreign characters like the a~ etc. is that > then A
and less then <
On Fri, May 24, 2013 at 11:41 AM, Dean Wampler <de...@gmail.com>wrote:
> If backwards compatibility wasn't an issue, the hive code that implements
> LIKE could be changed to convert the fields and LIKE strings to lower case
> before comparing ;) Of course, there is overhead doing that.
>
> On Fri, May 24, 2013 at 9:50 AM, Edward Capriolo <ed...@gmail.com>wrote:
>
>> Also I am thinking that the rlike is based on regex and can be told to do
>> case insensitive matching.
>>
>>
>> On Fri, May 24, 2013 at 9:16 AM, Dean Wampler <de...@gmail.com>wrote:
>>
>>> Hortonworks has announced plans to make Hive more SQL compliant. I
>>> suspect bugs like this will be addressed sooner or later. It will be
>>> necessary to handle backwards compatibility, but that could be handled with
>>> a hive property that enables one or the other behaviors.
>>>
>>> On Fri, May 24, 2013 at 8:07 AM, John Omernik <jo...@omernik.com> wrote:
>>>
>>>> I have mentioned this before, and I think this a big miss by the Hive
>>>> team. Like, by default in many SQL RDBMS (like MSSQL or MYSQL) is not
>>>> case sensitive. Thus when you have new users moving over to Hive, if they
>>>> see a command like "like" they will assume similarity (like many other SQL
>>>> like qualities) and thus false negatives may ensue. Even though it's
>>>> different by default (I am ok with this ... I guess, my personal preference
>>>> is that it matches the defaults on other systems, and outside of that
>>>> (which I am, in in the end fine with, just grumbly :) ) give us the ability
>>>> to set that behavior in the hive-site.xml. That way when an org realizes
>>>> that it is different, and their users are all getting false negatives, they
>>>> can just update the hive-site and fix the problem rather than have to
>>>> include it in training that may or may not work. I've added this comment
>>>> to https://issues.apache.org/jira/browse/HIVE-4070#comment-13666278 for fun. :)
>>>>
>>>> Please? :)
>>>>
>>>>
>>>>
>>>>
>>>> On Fri, May 24, 2013 at 7:53 AM, Dean Wampler <de...@gmail.com>wrote:
>>>>
>>>>> Your where clause looks at the abbreviation, requiring 'A', not the
>>>>> state name. You got the correct answer.
>>>>>
>>>>>
>>>>> On Fri, May 24, 2013 at 6:21 AM, Sai Sai <sa...@yahoo.in> wrote:
>>>>>
>>>>>> But it should get more results for this:
>>>>>>
>>>>>> %a%
>>>>>>
>>>>>> than for
>>>>>>
>>>>>> %A%
>>>>>>
>>>>>> Please let me know if i am missing something.
>>>>>> Thanks
>>>>>> Sai
>>>>>>
>>>>>>
>>>>>> ------------------------------
>>>>>> *From:* Jov <am...@amutu.com>
>>>>>> *To:* user@hive.apache.org; Sai Sai <sa...@yahoo.in>
>>>>>> *Sent:* Friday, 24 May 2013 4:39 PM
>>>>>> *Subject:* Re: Difference between like %A% and %a%
>>>>>>
>>>>>>
>>>>>> 2013/5/24 Sai Sai <sa...@yahoo.in>
>>>>>>
>>>>>> abbreviation l
>>>>>>
>>>>>>
>>>>>> unlike MySQL, string in Hive is case sensitive，so '%A%' is not equal
>>>>>> with '%a%'.
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Jov
>>>>>> blog: http:amutu.com/blog <http://amutu.com/blog>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Dean Wampler, Ph.D.
>>>>> @deanwampler
>>>>> http://polyglotprogramming.com
>>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> Dean Wampler, Ph.D.
>>> @deanwampler
>>> http://polyglotprogramming.com
>>
>>
>>
>
>
> --
> Dean Wampler, Ph.D.
> @deanwampler
> http://polyglotprogramming.com
Re: Difference between like %A% and %a%
Posted by Dean Wampler <de...@gmail.com>.
If backwards compatibility wasn't an issue, the hive code that implements
LIKE could be changed to convert the fields and LIKE strings to lower case
before comparing ;) Of course, there is overhead doing that.
On Fri, May 24, 2013 at 9:50 AM, Edward Capriolo <ed...@gmail.com>wrote:
> Also I am thinking that the rlike is based on regex and can be told to do
> case insensitive matching.
>
>
> On Fri, May 24, 2013 at 9:16 AM, Dean Wampler <de...@gmail.com>wrote:
>
>> Hortonworks has announced plans to make Hive more SQL compliant. I
>> suspect bugs like this will be addressed sooner or later. It will be
>> necessary to handle backwards compatibility, but that could be handled with
>> a hive property that enables one or the other behaviors.
>>
>> On Fri, May 24, 2013 at 8:07 AM, John Omernik <jo...@omernik.com> wrote:
>>
>>> I have mentioned this before, and I think this a big miss by the Hive
>>> team. Like, by default in many SQL RDBMS (like MSSQL or MYSQL) is not
>>> case sensitive. Thus when you have new users moving over to Hive, if they
>>> see a command like "like" they will assume similarity (like many other SQL
>>> like qualities) and thus false negatives may ensue. Even though it's
>>> different by default (I am ok with this ... I guess, my personal preference
>>> is that it matches the defaults on other systems, and outside of that
>>> (which I am, in in the end fine with, just grumbly :) ) give us the ability
>>> to set that behavior in the hive-site.xml. That way when an org realizes
>>> that it is different, and their users are all getting false negatives, they
>>> can just update the hive-site and fix the problem rather than have to
>>> include it in training that may or may not work. I've added this comment
>>> to https://issues.apache.org/jira/browse/HIVE-4070#comment-13666278 for fun. :)
>>>
>>> Please? :)
>>>
>>>
>>>
>>>
>>> On Fri, May 24, 2013 at 7:53 AM, Dean Wampler <de...@gmail.com>wrote:
>>>
>>>> Your where clause looks at the abbreviation, requiring 'A', not the
>>>> state name. You got the correct answer.
>>>>
>>>>
>>>> On Fri, May 24, 2013 at 6:21 AM, Sai Sai <sa...@yahoo.in> wrote:
>>>>
>>>>> But it should get more results for this:
>>>>>
>>>>> %a%
>>>>>
>>>>> than for
>>>>>
>>>>> %A%
>>>>>
>>>>> Please let me know if i am missing something.
>>>>> Thanks
>>>>> Sai
>>>>>
>>>>>
>>>>> ------------------------------
>>>>> *From:* Jov <am...@amutu.com>
>>>>> *To:* user@hive.apache.org; Sai Sai <sa...@yahoo.in>
>>>>> *Sent:* Friday, 24 May 2013 4:39 PM
>>>>> *Subject:* Re: Difference between like %A% and %a%
>>>>>
>>>>>
>>>>> 2013/5/24 Sai Sai <sa...@yahoo.in>
>>>>>
>>>>> abbreviation l
>>>>>
>>>>>
>>>>> unlike MySQL, string in Hive is case sensitive，so '%A%' is not equal
>>>>> with '%a%'.
>>>>>
>>>>>
>>>>> --
>>>>> Jov
>>>>> blog: http:amutu.com/blog <http://amutu.com/blog>
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Dean Wampler, Ph.D.
>>>> @deanwampler
>>>> http://polyglotprogramming.com
>>>>
>>>
>>>
>>
>>
>> --
>> Dean Wampler, Ph.D.
>> @deanwampler
>> http://polyglotprogramming.com
>
>
>
--
Dean Wampler, Ph.D.
@deanwampler
http://polyglotprogramming.com
Re: Difference between like %A% and %a%
Posted by Edward Capriolo <ed...@gmail.com>.
Also I am thinking that the rlike is based on regex and can be told to do
case insensitive matching.
On Fri, May 24, 2013 at 9:16 AM, Dean Wampler <de...@gmail.com> wrote:
> Hortonworks has announced plans to make Hive more SQL compliant. I suspect
> bugs like this will be addressed sooner or later. It will be necessary to
> handle backwards compatibility, but that could be handled with a hive
> property that enables one or the other behaviors.
>
> On Fri, May 24, 2013 at 8:07 AM, John Omernik <jo...@omernik.com> wrote:
>
>> I have mentioned this before, and I think this a big miss by the Hive
>> team. Like, by default in many SQL RDBMS (like MSSQL or MYSQL) is not
>> case sensitive. Thus when you have new users moving over to Hive, if they
>> see a command like "like" they will assume similarity (like many other SQL
>> like qualities) and thus false negatives may ensue. Even though it's
>> different by default (I am ok with this ... I guess, my personal preference
>> is that it matches the defaults on other systems, and outside of that
>> (which I am, in in the end fine with, just grumbly :) ) give us the ability
>> to set that behavior in the hive-site.xml. That way when an org realizes
>> that it is different, and their users are all getting false negatives, they
>> can just update the hive-site and fix the problem rather than have to
>> include it in training that may or may not work. I've added this comment
>> to https://issues.apache.org/jira/browse/HIVE-4070#comment-13666278 for
>> fun. :)
>>
>> Please? :)
>>
>>
>>
>>
>> On Fri, May 24, 2013 at 7:53 AM, Dean Wampler <de...@gmail.com>wrote:
>>
>>> Your where clause looks at the abbreviation, requiring 'A', not the
>>> state name. You got the correct answer.
>>>
>>>
>>> On Fri, May 24, 2013 at 6:21 AM, Sai Sai <sa...@yahoo.in> wrote:
>>>
>>>> But it should get more results for this:
>>>>
>>>> %a%
>>>>
>>>> than for
>>>>
>>>> %A%
>>>>
>>>> Please let me know if i am missing something.
>>>> Thanks
>>>> Sai
>>>>
>>>>
>>>> ------------------------------
>>>> *From:* Jov <am...@amutu.com>
>>>> *To:* user@hive.apache.org; Sai Sai <sa...@yahoo.in>
>>>> *Sent:* Friday, 24 May 2013 4:39 PM
>>>> *Subject:* Re: Difference between like %A% and %a%
>>>>
>>>>
>>>> 2013/5/24 Sai Sai <sa...@yahoo.in>
>>>>
>>>> abbreviation l
>>>>
>>>>
>>>> unlike MySQL, string in Hive is case sensitive，so '%A%' is not equal
>>>> with '%a%'.
>>>>
>>>>
>>>> --
>>>> Jov
>>>> blog: http:amutu.com/blog <http://amutu.com/blog>
>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> Dean Wampler, Ph.D.
>>> @deanwampler
>>> http://polyglotprogramming.com
>>>
>>
>>
>
>
> --
> Dean Wampler, Ph.D.
> @deanwampler
> http://polyglotprogramming.com
Re: Difference between like %A% and %a%
Posted by Edward Capriolo <ed...@gmail.com>.
It is not really a bug, as must as it is the way hive is designed.
https://issues.apache.org/jira/browse/HIVE-4070#comment-13666362
So there already is a 'like' and an 'rlike', mlike is a good idea. It seems
like an easier UDF (low hanging fruit) type issue anyone could tackle.
On Fri, May 24, 2013 at 9:16 AM, Dean Wampler <de...@gmail.com> wrote:
> Hortonworks has announced plans to make Hive more SQL compliant. I suspect
> bugs like this will be addressed sooner or later. It will be necessary to
> handle backwards compatibility, but that could be handled with a hive
> property that enables one or the other behaviors.
>
> On Fri, May 24, 2013 at 8:07 AM, John Omernik <jo...@omernik.com> wrote:
>
>> I have mentioned this before, and I think this a big miss by the Hive
>> team. Like, by default in many SQL RDBMS (like MSSQL or MYSQL) is not
>> case sensitive. Thus when you have new users moving over to Hive, if they
>> see a command like "like" they will assume similarity (like many other SQL
>> like qualities) and thus false negatives may ensue. Even though it's
>> different by default (I am ok with this ... I guess, my personal preference
>> is that it matches the defaults on other systems, and outside of that
>> (which I am, in in the end fine with, just grumbly :) ) give us the ability
>> to set that behavior in the hive-site.xml. That way when an org realizes
>> that it is different, and their users are all getting false negatives, they
>> can just update the hive-site and fix the problem rather than have to
>> include it in training that may or may not work. I've added this comment
>> to https://issues.apache.org/jira/browse/HIVE-4070#comment-13666278 for
>> fun. :)
>>
>> Please? :)
>>
>>
>>
>>
>> On Fri, May 24, 2013 at 7:53 AM, Dean Wampler <de...@gmail.com>wrote:
>>
>>> Your where clause looks at the abbreviation, requiring 'A', not the
>>> state name. You got the correct answer.
>>>
>>>
>>> On Fri, May 24, 2013 at 6:21 AM, Sai Sai <sa...@yahoo.in> wrote:
>>>
>>>> But it should get more results for this:
>>>>
>>>> %a%
>>>>
>>>> than for
>>>>
>>>> %A%
>>>>
>>>> Please let me know if i am missing something.
>>>> Thanks
>>>> Sai
>>>>
>>>>
>>>> ------------------------------
>>>> *From:* Jov <am...@amutu.com>
>>>> *To:* user@hive.apache.org; Sai Sai <sa...@yahoo.in>
>>>> *Sent:* Friday, 24 May 2013 4:39 PM
>>>> *Subject:* Re: Difference between like %A% and %a%
>>>>
>>>>
>>>> 2013/5/24 Sai Sai <sa...@yahoo.in>
>>>>
>>>> abbreviation l
>>>>
>>>>
>>>> unlike MySQL, string in Hive is case sensitive，so '%A%' is not equal
>>>> with '%a%'.
>>>>
>>>>
>>>> --
>>>> Jov
>>>> blog: http:amutu.com/blog <http://amutu.com/blog>
>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> Dean Wampler, Ph.D.
>>> @deanwampler
>>> http://polyglotprogramming.com
>>>
>>
>>
>
>
> --
> Dean Wampler, Ph.D.
> @deanwampler
> http://polyglotprogramming.com
Re: Difference between like %A% and %a%
Posted by Dean Wampler <de...@gmail.com>.
Hortonworks has announced plans to make Hive more SQL compliant. I suspect
bugs like this will be addressed sooner or later. It will be necessary to
handle backwards compatibility, but that could be handled with a hive
property that enables one or the other behaviors.
On Fri, May 24, 2013 at 8:07 AM, John Omernik <jo...@omernik.com> wrote:
> I have mentioned this before, and I think this a big miss by the Hive
> team. Like, by default in many SQL RDBMS (like MSSQL or MYSQL) is not
> case sensitive. Thus when you have new users moving over to Hive, if they
> see a command like "like" they will assume similarity (like many other SQL
> like qualities) and thus false negatives may ensue. Even though it's
> different by default (I am ok with this ... I guess, my personal preference
> is that it matches the defaults on other systems, and outside of that
> (which I am, in in the end fine with, just grumbly :) ) give us the ability
> to set that behavior in the hive-site.xml. That way when an org realizes
> that it is different, and their users are all getting false negatives, they
> can just update the hive-site and fix the problem rather than have to
> include it in training that may or may not work. I've added this comment
> to https://issues.apache.org/jira/browse/HIVE-4070#comment-13666278 for
> fun. :)
>
> Please? :)
>
>
>
>
> On Fri, May 24, 2013 at 7:53 AM, Dean Wampler <de...@gmail.com>wrote:
>
>> Your where clause looks at the abbreviation, requiring 'A', not the state
>> name. You got the correct answer.
>>
>>
>> On Fri, May 24, 2013 at 6:21 AM, Sai Sai <sa...@yahoo.in> wrote:
>>
>>> But it should get more results for this:
>>>
>>> %a%
>>>
>>> than for
>>>
>>> %A%
>>>
>>> Please let me know if i am missing something.
>>> Thanks
>>> Sai
>>>
>>>
>>> ------------------------------
>>> *From:* Jov <am...@amutu.com>
>>> *To:* user@hive.apache.org; Sai Sai <sa...@yahoo.in>
>>> *Sent:* Friday, 24 May 2013 4:39 PM
>>> *Subject:* Re: Difference between like %A% and %a%
>>>
>>>
>>> 2013/5/24 Sai Sai <sa...@yahoo.in>
>>>
>>> abbreviation l
>>>
>>>
>>> unlike MySQL, string in Hive is case sensitive，so '%A%' is not equal
>>> with '%a%'.
>>>
>>>
>>> --
>>> Jov
>>> blog: http:amutu.com/blog <http://amutu.com/blog>
>>>
>>>
>>>
>>
>>
>> --
>> Dean Wampler, Ph.D.
>> @deanwampler
>> http://polyglotprogramming.com
>>
>
>
--
Dean Wampler, Ph.D.
@deanwampler
http://polyglotprogramming.com
Re: Difference between like %A% and %a%
Posted by John Omernik <jo...@omernik.com>.
I have mentioned this before, and I think this a big miss by the Hive team.
Like, by default in many SQL RDBMS (like MSSQL or MYSQL) is not case
sensitive. Thus when you have new users moving over to Hive, if they see a
command like "like" they will assume similarity (like many other SQL like
qualities) and thus false negatives may ensue. Even though it's different
by default (I am ok with this ... I guess, my personal preference is that
it matches the defaults on other systems, and outside of that (which I am,
in in the end fine with, just grumbly :) ) give us the ability to set
that behavior in the hive-site.xml. That way when an org realizes that it
is different, and their users are all getting false negatives, they can
just update the hive-site and fix the problem rather than have to include
it in training that may or may not work. I've added this comment to
https://issues.apache.org/jira/browse/HIVE-4070#comment-13666278 for fun.
:)
Please? :)
On Fri, May 24, 2013 at 7:53 AM, Dean Wampler <de...@gmail.com> wrote:
> Your where clause looks at the abbreviation, requiring 'A', not the state
> name. You got the correct answer.
>
>
> On Fri, May 24, 2013 at 6:21 AM, Sai Sai <sa...@yahoo.in> wrote:
>
>> But it should get more results for this:
>>
>> %a%
>>
>> than for
>>
>> %A%
>>
>> Please let me know if i am missing something.
>> Thanks
>> Sai
>>
>>
>> ------------------------------
>> *From:* Jov <am...@amutu.com>
>> *To:* user@hive.apache.org; Sai Sai <sa...@yahoo.in>
>> *Sent:* Friday, 24 May 2013 4:39 PM
>> *Subject:* Re: Difference between like %A% and %a%
>>
>>
>> 2013/5/24 Sai Sai <sa...@yahoo.in>
>>
>> abbreviation l
>>
>>
>> unlike MySQL, string in Hive is case sensitive，so '%A%' is not equal with
>> '%a%'.
>>
>>
>> --
>> Jov
>> blog: http:amutu.com/blog <http://amutu.com/blog>
>>
>>
>>
>
>
> --
> Dean Wampler, Ph.D.
> @deanwampler
> http://polyglotprogramming.com
>
Re: Difference between like %A% and %a%
Posted by Dean Wampler <de...@gmail.com>.
Your where clause looks at the abbreviation, requiring 'A', not the state
name. You got the correct answer.
On Fri, May 24, 2013 at 6:21 AM, Sai Sai <sa...@yahoo.in> wrote:
> But it should get more results for this:
>
> %a%
>
> than for
>
> %A%
>
> Please let me know if i am missing something.
> Thanks
> Sai
>
>
> ------------------------------
> *From:* Jov <am...@amutu.com>
> *To:* user@hive.apache.org; Sai Sai <sa...@yahoo.in>
> *Sent:* Friday, 24 May 2013 4:39 PM
> *Subject:* Re: Difference between like %A% and %a%
>
>
> 2013/5/24 Sai Sai <sa...@yahoo.in>
>
> abbreviation l
>
>
> unlike MySQL, string in Hive is case sensitive，so '%A%' is not equal with
> '%a%'.
>
>
> --
> Jov
> blog: http:amutu.com/blog <http://amutu.com/blog>
>
>
>
--
Dean Wampler, Ph.D.
@deanwampler
http://polyglotprogramming.com
Re: How to look at the metadata of the tables we have created.
Posted by Sanjay Subramanian <Sa...@wizecommerce.com>.
I have an earlier version ER diagram of Hive….There are not too many tables and u can navigate to them and poke around easily
For example I created a hive locations alias for myself (and I think this is already implemented in 10.0) where u can get the HDFS location for each partition
select t.TBL_NAME, p.PART_NAME, s.LOCATION from PARTITIONS p, SDS s, TBLS t where t.TBL_ID=p.TBL_ID and p.SD_ID=s.SD_ID
From: Stephen Sprague <sp...@gmail.com>>
Reply-To: "user@hive.apache.org<ma...@hive.apache.org>" <us...@hive.apache.org>>
Date: Friday, May 24, 2013 3:19 PM
To: "user@hive.apache.org<ma...@hive.apache.org>" <us...@hive.apache.org>>, Sai Sai <sa...@yahoo.in>>
Subject: Re: How to look at the metadata of the tables we have created.
yes. there's this high faluntin' concept called the 'metastore'. should you google it in conjunction with 'hive' the answer shall be before you.
On Fri, May 24, 2013 at 4:30 AM, Sai Sai <sa...@yahoo.in>> wrote:
Is it possible to look at the metadata of the databases/tables/views we have created in hive.
Is there some thing like sysobjects in hive.
Thanks
Sai
CONFIDENTIALITY NOTICE
======================
This email message and any attachments are for the exclusive use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message along with any attachments, from your computer system. If you are the intended recipient, please be advised that the content of this message is subject to access, review and disclosure by the sender's Email System Administrator.
Re: How to look at the metadata of the tables we have created.
Posted by Stephen Sprague <sp...@gmail.com>.
yes. there's this high faluntin' concept called the 'metastore'. should
you google it in conjunction with 'hive' the answer shall be before you.
On Fri, May 24, 2013 at 4:30 AM, Sai Sai <sa...@yahoo.in> wrote:
> Is it possible to look at the metadata of the databases/tables/views we
> have created in hive.
> Is there some thing like sysobjects in hive.
> Thanks
> Sai
>
Re: Partitioning confusion
Posted by Nitin Pawar <ni...@gmail.com>.
if you have a 250GB file, then how did it become 2.5TB ?
if you can not write a mapreduce job to process to write your data into
specific partitions, then the other way around would be load the entire
data into a temporary table and then load data into partitioned tabled and
then drop the temporary table.
approach would be
1) Create a temporary table
CREATE TABLE temp_employees (name STRING, salary FLOAT, subordinates
ARRAY<STRING>, deductions MAP<STRING, FLOAT>, address STRUCT<street:STRING,
city:STRING, state:STRING, zip:INT, country:STRING> );
2) Load data into this table
LOAD DATA LOCAL INPATH
'/home/satish/data/employees/input/employees-country.txt'
INTO TABLE temp_employees;
3) Create partitoned table
CREATE TABLE employees (name STRING, salary FLOAT, subordinates
ARRAY<STRING>, deductions MAP<STRING, FLOAT>, address STRUCT<street:STRING,
city:STRING, state:STRING, zip:INT, country:STRING> ) PARTITIONED BY
(country STRING, state STRING);
2) Load data from temporary data into partitioned tables
2a) if there are already partitions existing then be careful giving a
blanket query or it will overwrite data into exisiting partitions
insert overwrite employees partition(country='USA',state='IL') select *
from temp_employees where country="USA" and state ="IL";
this you will need to do for all of them
2b) if your partitioned table is empty currently then you can just
enable dynamic partitioning and hive will take care of adding correct data
into individual partitions
set hive.exec.dynamic.partition.mode=nonstrict;
set hive.exec.dynamic.partition=true;
insert overwrite table employees select * from employees;
3) drop the temporary table
by doing above approach you will have your data being replicated 2x times
of replication factor for sometime but then once you drop temporary table
it will be normal HDFS replication factor size data.
Its been sometime I have worked on hive so others may have better
approaches as well, so wait for someone to correct me before going further
:)
On Mon, May 27, 2013 at 2:13 PM, Sai Sai <sa...@yahoo.in> wrote:
> Nitin
> I am still confused, from the below data that i have given should the
> file which sits in the folder Country=USA and state=IL have only the rows
> where Country=USA and state=IL or will it have rows of other countries also.
> The reason i ask is because if we have a 250GB file and would like to
> create 10 partitions that would end up in 2.5 TB * 3 = 7.5TB. Is this
> expected.
> Thanks
> S
>
> ------------------------------
> *From:* Nitin Pawar <ni...@gmail.com>
> *To:* user@hive.apache.org; Sai Sai <sa...@yahoo.in>
> *Sent:* Monday, 27 May 2013 2:08 PM
> *Subject:* Re: Partitioning confusion
>
> when you specify the load data query with specific partition, it will put
> the entire data into that partition.
>
>
>
> On Mon, May 27, 2013 at 1:08 PM, Sai Sai <sa...@yahoo.in> wrote:
>
>
> After creating a partition for a country (USA) and state (IL) and when we
> go to the the hdfs site to look at the partition in the browser we r seeing
> all the records for all the countries and states rather than just for the
> partition created for US and IL given below, is this correct behavior:
> ********************
> Here is my commands:
> ********************
>
> CREATE TABLE employees (name STRING, salary FLOAT, subordinates
> ARRAY<STRING>, deductions MAP<STRING, FLOAT>, address STRUCT<street:STRING,
> city:STRING, state:STRING, zip:INT, country:STRING> ) PARTITIONED BY
> (country STRING, state STRING);
>
> LOAD DATA LOCAL INPATH
> '/home/satish/data/employees/input/employees-country.txt' INTO TABLE
> employees PARTITION (country='USA',state='IL');
>
> ********************
> Here is my original data file, where i have a few countries data such as
> USA, INDIA, UK, AUS:
> ********************
>
> John Doe100000.0Mary SmithTodd JonesFederal Taxes.2State
> Taxes.05Insurance.11 Michigan Ave.ChicagoIL60600USA
> Mary Smith80000.0Bill KingFederal Taxes.2State Taxes.05Insurance.1100
> Ontario St.ChicagoIL60601USA
> Todd Jones70000.0Federal Taxes.15State Taxes.03Insurance.1200 Chicago
> Ave.Oak ParkIL60700USA
> Bill King60000.0Federal Taxes.15State Taxes.03Insurance.1300 Obscure
> Dr.ObscuriaIL60100USA
> Boss Man200000.0John DoeFred FinanceFederal Taxes.3State
> Taxes.07Insurance.051 Pretentious Drive.ChicagoIL60500USA
> Fred Finance150000.0Stacy AccountantFederal Taxes.3State
> Taxes.07Insurance.052 Pretentious Drive.ChicagoIL60500USA
> Stacy Accountant60000.0Federal Taxes.15State Taxes.03Insurance.1300 Main
> St.NapervilleIL60563USA
> John Doe 2100000.0Mary SmithTodd JonesFederal Taxes.2State
> Taxes.05Insurance.11 Michigan Ave.ChicagoIL60600INDIA
> Mary Smith 280000.0Bill KingFederal Taxes.2State Taxes.05Insurance.1100
> Ontario St.ChicagoIL60601INDIA
> Todd Jones 270000.0Federal Taxes.15State Taxes.03Insurance.1200 Chicago
> Ave.Oak ParkIL60700AUSTRALIA
> Bill King 260000.0Federal Taxes.15State Taxes.03Insurance.1300 Obscure
> Dr.ObscuriaIL60100AUSTRALIA
> Boss Man2 200000.0John DoeFred FinanceFederal Taxes.3State
> Taxes.07Insurance.051 Pretentious Drive.ChicagoIL60500UK
> Fred Finance 2150000.0Stacy AccountantFederal Taxes.3State
> Taxes.07Insurance.052 Pretentious Drive.ChicagoIL60500UK
> Stacy Accountant 260000.0Federal Taxes.15State Taxes.03Insurance.1300 Main
> St.NapervilleIL60563UK
> ********************
> Now when i navigate to:
> Contents of directory
> /user/hive/warehouse/db1.db/employees/country=USA/state=IL
> ********************
> I see all the records and was wondering if it should have only USA & IL
> records.
> Please help.
>
>
>
>
> --
> Nitin Pawar
>
>
>
--
Nitin Pawar
Re: Partitioning confusion
Posted by Sai Sai <sa...@yahoo.in>.
Nitin
I am still confused, from the below data that i have given should the file which sits in the folder Country=USA and state=IL have only the rows where Country=USA and state=IL or will it have rows of other countries also.
The reason i ask is because if we have a 250GB file and would like to create 10 partitions that would end up in 2.5 TB * 3 = 7.5TB. Is this expected.
Thanks
S
________________________________
From: Nitin Pawar <ni...@gmail.com>
To: user@hive.apache.org; Sai Sai <sa...@yahoo.in>
Sent: Monday, 27 May 2013 2:08 PM
Subject: Re: Partitioning confusion
when you specify the load data query with specific partition, it will put the entire data into that partition.
On Mon, May 27, 2013 at 1:08 PM, Sai Sai <sa...@yahoo.in> wrote:
>
>After creating a partition for a country (USA) and state (IL) and when we go to the the hdfs site to look at the partition in the browser we r seeing all the records for all the countries and states rather than just for the partition created for US and IL given below, is this correct behavior:
>********************
>Here is my commands:
>********************
>
>
>
>CREATE TABLE employees (name STRING, salary FLOAT, subordinates ARRAY<STRING>, deductions MAP<STRING, FLOAT>, address STRUCT<street:STRING, city:STRING, state:STRING, zip:INT, country:STRING> ) PARTITIONED BY (country STRING, state STRING);
>
>
>LOAD DATA LOCAL INPATH '/home/satish/data/employees/input/employees-country.txt' INTO TABLE employees PARTITION (country='USA',state='IL');
>
>
>********************
>
>Here is my original data file, where i have a few countries data such as USA, INDIA, UK, AUS:
>********************
>
>
>
>John Doe100000.0Mary SmithTodd JonesFederal Taxes.2State Taxes.05Insurance.11 Michigan Ave.ChicagoIL60600USA
>Mary Smith80000.0Bill KingFederal Taxes.2State Taxes.05Insurance.1100 Ontario St.ChicagoIL60601USA
>Todd Jones70000.0Federal Taxes.15State Taxes.03Insurance.1200 Chicago Ave.Oak ParkIL60700USA
>Bill King60000.0Federal Taxes.15State Taxes.03Insurance.1300 Obscure Dr.ObscuriaIL60100USA
>Boss Man200000.0John DoeFred FinanceFederal Taxes.3State Taxes.07Insurance.051 Pretentious Drive.ChicagoIL60500USA
>Fred Finance150000.0Stacy AccountantFederal Taxes.3State Taxes.07Insurance.052 Pretentious Drive.ChicagoIL60500USA
>Stacy Accountant60000.0Federal Taxes.15State Taxes.03Insurance.1300 Main St.NapervilleIL60563USA
>John Doe 2100000.0Mary SmithTodd JonesFederal Taxes.2State Taxes.05Insurance.11 Michigan Ave.ChicagoIL60600INDIA
>Mary Smith 280000.0Bill KingFederal Taxes.2State Taxes.05Insurance.1100 Ontario St.ChicagoIL60601INDIA
>Todd Jones 270000.0Federal Taxes.15State Taxes.03Insurance.1200 Chicago Ave.Oak ParkIL60700AUSTRALIA
>Bill King 260000.0Federal Taxes.15State Taxes.03Insurance.1300 Obscure Dr.ObscuriaIL60100AUSTRALIA
>Boss Man2 200000.0John DoeFred FinanceFederal Taxes.3State Taxes.07Insurance.051 Pretentious Drive.ChicagoIL60500UK
>Fred Finance 2150000.0Stacy AccountantFederal Taxes.3State Taxes.07Insurance.052 Pretentious Drive.ChicagoIL60500UK
>Stacy Accountant 260000.0Federal Taxes.15State Taxes.03Insurance.1300 Main St.NapervilleIL60563UK
>********************
>
>Now when i navigate to:
>Contents of directory /user/hive/warehouse/db1.db/employees/country=USA/state=IL
>
>********************
>
>I see all the records and was wondering if it should have only USA & IL records.
>Please help.
--
Nitin Pawar
Re: Partitioning confusion
Posted by Nitin Pawar <ni...@gmail.com>.
when you specify the load data query with specific partition, it will put
the entire data into that partition.
On Mon, May 27, 2013 at 1:08 PM, Sai Sai <sa...@yahoo.in> wrote:
>
> After creating a partition for a country (USA) and state (IL) and when we
> go to the the hdfs site to look at the partition in the browser we r seeing
> all the records for all the countries and states rather than just for the
> partition created for US and IL given below, is this correct behavior:
> ********************
> Here is my commands:
> ********************
>
> CREATE TABLE employees (name STRING, salary FLOAT, subordinates
> ARRAY<STRING>, deductions MAP<STRING, FLOAT>, address STRUCT<street:STRING,
> city:STRING, state:STRING, zip:INT, country:STRING> ) PARTITIONED BY
> (country STRING, state STRING);
>
> LOAD DATA LOCAL INPATH
> '/home/satish/data/employees/input/employees-country.txt' INTO TABLE
> employees PARTITION (country='USA',state='IL');
>
> ********************
> Here is my original data file, where i have a few countries data such as
> USA, INDIA, UK, AUS:
> ********************
>
> John Doe100000.0Mary SmithTodd JonesFederal Taxes.2State
> Taxes.05Insurance.11 Michigan Ave.ChicagoIL60600USA
> Mary Smith80000.0Bill KingFederal Taxes.2State Taxes.05Insurance.1100
> Ontario St.ChicagoIL60601USA
> Todd Jones70000.0Federal Taxes.15State Taxes.03Insurance.1200 Chicago
> Ave.Oak ParkIL60700USA
> Bill King60000.0Federal Taxes.15State Taxes.03Insurance.1300 Obscure
> Dr.ObscuriaIL60100USA
> Boss Man200000.0John DoeFred FinanceFederal Taxes.3State
> Taxes.07Insurance.051 Pretentious Drive.ChicagoIL60500USA
> Fred Finance150000.0Stacy AccountantFederal Taxes.3State
> Taxes.07Insurance.052 Pretentious Drive.ChicagoIL60500USA
> Stacy Accountant60000.0Federal Taxes.15State Taxes.03Insurance.1300 Main
> St.NapervilleIL60563USA
> John Doe 2100000.0Mary SmithTodd JonesFederal Taxes.2State
> Taxes.05Insurance.11 Michigan Ave.ChicagoIL60600INDIA
> Mary Smith 280000.0Bill KingFederal Taxes.2State Taxes.05Insurance.1100
> Ontario St.ChicagoIL60601INDIA
> Todd Jones 270000.0Federal Taxes.15State Taxes.03Insurance.1200 Chicago
> Ave.Oak ParkIL60700AUSTRALIA
> Bill King 260000.0Federal Taxes.15State Taxes.03Insurance.1300 Obscure
> Dr.ObscuriaIL60100AUSTRALIA
> Boss Man2 200000.0John DoeFred FinanceFederal Taxes.3State
> Taxes.07Insurance.051 Pretentious Drive.ChicagoIL60500UK
> Fred Finance 2150000.0Stacy AccountantFederal Taxes.3State
> Taxes.07Insurance.052 Pretentious Drive.ChicagoIL60500UK
> Stacy Accountant 260000.0Federal Taxes.15State Taxes.03Insurance.1300 Main
> St.NapervilleIL60563UK
> ********************
> Now when i navigate to:
> Contents of directory
> /user/hive/warehouse/db1.db/employees/country=USA/state=IL
> ********************
> I see all the records and was wondering if it should have only USA & IL
> records.
> Please help.
>
--
Nitin Pawar
Re:Partitioning confusion
Posted by Sai Sai <sa...@yahoo.in>.
After creating a partition for a country (USA) and state (IL) and when we go to the the hdfs site to look at the partition in the browser we r seeing all the records for all the countries and states rather than just for the partition created for US and IL given below, is this correct behavior:
********************
Here is my commands:
********************
CREATE TABLE employees (name STRING, salary FLOAT, subordinates ARRAY<STRING>, deductions MAP<STRING, FLOAT>, address STRUCT<street:STRING, city:STRING, state:STRING, zip:INT, country:STRING> ) PARTITIONED BY (country STRING, state STRING);
LOAD DATA LOCAL INPATH '/home/satish/data/employees/input/employees-country.txt' INTO TABLE employees PARTITION (country='USA',state='IL');
********************
Here is my original data file, where i have a few countries data such as USA, INDIA, UK, AUS:
********************
John Doe100000.0Mary SmithTodd JonesFederal Taxes.2State Taxes.05Insurance.11 Michigan Ave.ChicagoIL60600USA
Mary Smith80000.0Bill KingFederal Taxes.2State Taxes.05Insurance.1100 Ontario St.ChicagoIL60601USA
Todd Jones70000.0Federal Taxes.15State Taxes.03Insurance.1200 Chicago Ave.Oak ParkIL60700USA
Bill King60000.0Federal Taxes.15State Taxes.03Insurance.1300 Obscure Dr.ObscuriaIL60100USA
Boss Man200000.0John DoeFred FinanceFederal Taxes.3State Taxes.07Insurance.051 Pretentious Drive.ChicagoIL60500USA
Fred Finance150000.0Stacy AccountantFederal Taxes.3State Taxes.07Insurance.052 Pretentious Drive.ChicagoIL60500USA
Stacy Accountant60000.0Federal Taxes.15State Taxes.03Insurance.1300 Main St.NapervilleIL60563USA
John Doe 2100000.0Mary SmithTodd JonesFederal Taxes.2State Taxes.05Insurance.11 Michigan Ave.ChicagoIL60600INDIA
Mary Smith 280000.0Bill KingFederal Taxes.2State Taxes.05Insurance.1100 Ontario St.ChicagoIL60601INDIA
Todd Jones 270000.0Federal Taxes.15State Taxes.03Insurance.1200 Chicago Ave.Oak ParkIL60700AUSTRALIA
Bill King 260000.0Federal Taxes.15State Taxes.03Insurance.1300 Obscure Dr.ObscuriaIL60100AUSTRALIA
Boss Man2 200000.0John DoeFred FinanceFederal Taxes.3State Taxes.07Insurance.051 Pretentious Drive.ChicagoIL60500UK
Fred Finance 2150000.0Stacy AccountantFederal Taxes.3State Taxes.07Insurance.052 Pretentious Drive.ChicagoIL60500UK
Stacy Accountant 260000.0Federal Taxes.15State Taxes.03Insurance.1300 Main St.NapervilleIL60563UK
********************
Now when i navigate to:
Contents of directory /user/hive/warehouse/db1.db/employees/country=USA/state=IL
********************
I see all the records and was wondering if it should have only USA & IL records.
Please help.
Re: Issue with Json tuple lateral view
Posted by Sai Sai <sa...@yahoo.in>.
Thanks Navis
________________________________
From: Navis류승우 <na...@nexr.com>
To: user@hive.apache.org; Sai Sai <sa...@yahoo.in>
Sent: Monday, 27 May 2013 12:15 PM
Subject: Re: Issue with Json tuple lateral view
Removing last ',' in second row would make result as you expected.
I can't tell it's bug or not.
2013/5/27 Sai Sai <sa...@yahoo.in>:
>
> *************************
> Here is the json-data that i load:
> *************************
>
> { "blogID" : "FJY26J1333", "date" : "2012-04-01", "name" : "vpxnksu",
> "comment" : "good stuff", "contact" : { "email" : "vpxnksu@gmail.com",
> "website" : "vpxnksu.wordpress.com" } }
> { "blogID" : "VSAUMDFGSD", "date" : "2012-04-01", "name" : "yhftrcx",
> "comment" : "another comment",}
>
> *************************
> Here is the hive commands :
> *************************
>
> CREATE EXTERNAL TABLE json_comments(value STRING) LOCATION
> '/user/json-comments';
>
> LOAD DATA LOCAL INPATH '/home/satish/data/inputSai/json-comments.txt'
> OVERWRITE INTO TABLE json_comments;
>
> SELECT b.blogID, c.email FROM json_comments a LATERAL VIEW
> json_tuple(a.value, 'blogID', 'contact') b AS blogID, contact LATERAL VIEW
> json_tuple(b.contact, 'email', 'website') c AS email, website;
>
> *************************
> Here r the results of map reduce:
> *************************
>
> blogid email
> FJY26J1333 vpxnksu@gmail.com
> NULL NULL
>
> *************************
> My question is why the 2nd row is coming up as Null values, i was expecting
> the results to be like this:
> *************************
>
> blogid email
> FJY26J1333 vpxnksu@gmail.com
> VSAUMDFGSD NULL
>
> Any input is appreciated in understanding this.
> Thanks
> S
Re: Issue with Json tuple lateral view
Posted by Navis류승우 <na...@nexr.com>.
Removing last ',' in second row would make result as you expected.
I can't tell it's bug or not.
2013/5/27 Sai Sai <sa...@yahoo.in>:
>
> *************************
> Here is the json-data that i load:
> *************************
>
> { "blogID" : "FJY26J1333", "date" : "2012-04-01", "name" : "vpxnksu",
> "comment" : "good stuff", "contact" : { "email" : "vpxnksu@gmail.com",
> "website" : "vpxnksu.wordpress.com" } }
> { "blogID" : "VSAUMDFGSD", "date" : "2012-04-01", "name" : "yhftrcx",
> "comment" : "another comment",}
>
> *************************
> Here is the hive commands :
> *************************
>
> CREATE EXTERNAL TABLE json_comments(value STRING) LOCATION
> '/user/json-comments';
>
> LOAD DATA LOCAL INPATH '/home/satish/data/inputSai/json-comments.txt'
> OVERWRITE INTO TABLE json_comments;
>
> SELECT b.blogID, c.email FROM json_comments a LATERAL VIEW
> json_tuple(a.value, 'blogID', 'contact') b AS blogID, contact LATERAL VIEW
> json_tuple(b.contact, 'email', 'website') c AS email, website;
>
> *************************
> Here r the results of map reduce:
> *************************
>
> blogid email
> FJY26J1333 vpxnksu@gmail.com
> NULL NULL
>
> *************************
> My question is why the 2nd row is coming up as Null values, i was expecting
> the results to be like this:
> *************************
>
> blogid email
> FJY26J1333 vpxnksu@gmail.com
> VSAUMDFGSD NULL
>
> Any input is appreciated in understanding this.
> Thanks
> S
Re: Issue with Json tuple lateral view
Posted by Sai Sai <sa...@yahoo.in>.
*************************
Here is the json-data that i load:
*************************
{ "blogID" : "FJY26J1333", "date" : "2012-04-01", "name" : "vpxnksu", "comment" : "good stuff", "contact" : { "email" : "vpxnksu@gmail.com", "website" : "vpxnksu.wordpress.com" } }
{ "blogID" : "VSAUMDFGSD", "date" : "2012-04-01", "name" : "yhftrcx", "comment" : "another comment",}
*************************
Here is the hive commands :
*************************
CREATE EXTERNAL TABLE json_comments(value STRING) LOCATION '/user/json-comments';
LOAD DATA LOCAL INPATH '/home/satish/data/inputSai/json-comments.txt' OVERWRITE INTO TABLE json_comments;
SELECT b.blogID, c.email FROM json_comments a LATERAL VIEW json_tuple(a.value, 'blogID', 'contact') b AS blogID, contact LATERAL VIEW json_tuple(b.contact, 'email', 'website') c AS email, website;
*************************
Here r the results of map reduce:
*************************
blogidemail
FJY26J1333vpxnksu@gmail.com
NULLNULL
*************************
My question is why the 2nd row is coming up as Null values, i was expecting the results to be like this:
*************************
blogidemail
FJY26J1333vpxnksu@gmail.com
VSAUMDFGSDNULL
Any input is appreciated in understanding this.
Thanks
S
Re: How to look at the metadata of the tables we have created.
Posted by Sai Sai <sa...@yahoo.in>.
Is it possible to look at the metadata of the databases/tables/views we have created in hive.
Is there some thing like sysobjects in hive.
Thanks
Sai
Re: Difference between like %A% and %a%
Posted by Sai Sai <sa...@yahoo.in>.
But it should get more results for this:
%a%
than for
%A%
Please let me know if i am missing something.
Thanks
Sai
________________________________
From: Jov <am...@amutu.com>
To: user@hive.apache.org; Sai Sai <sa...@yahoo.in>
Sent: Friday, 24 May 2013 4:39 PM
Subject: Re: Difference between like %A% and %a%
2013/5/24 Sai Sai <sa...@yahoo.in>
abbreviation l
unlike MySQL, string in Hive is case sensitive，so '%A%' is not equal with '%a%'.
--
Jov
blog: http:amutu.com/blog
Re: Difference between like %A% and %a%
Posted by Jov <am...@amutu.com>.
2013/5/24 Sai Sai <sa...@yahoo.in>
> abbreviation l
unlike MySQL, string in Hive is case sensitive，so '%A%' is not equal with
'%a%'.
--
Jov
blog: http:amutu.com/blog <http://amutu.com/blog>
Re: Difference between like %A% and %a%
Posted by Sai Sai <sa...@yahoo.in>.
Just wondering about this, please let me know if you have any suggestions why we r getting these results:
This query does not return any data:
Query1:hive (test)> select full_name from states where abbreviation like '%a%';
But this query returns data successfully:
Query2:hive (test)> select full_name from states where abbreviation like '%A%';
Result of Query 1:
Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks is set to 0 since there's no reduce operator
Starting Job = job_201305240156_0012, Tracking URL = http://ubuntu:50030/jobdetails.jsp?jobid=job_201305240156_0012
Kill Command = /home/satish/work/hadoop-1.0.4/libexec/../bin/hadoop job -kill job_201305240156_0012
Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 0
2013-05-24 03:51:04,939 Stage-1 map = 0%, reduce = 0%
2013-05-24 03:51:10,970 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 0.46 sec
2013-05-24 03:51:11,983 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 0.46 sec
2013-05-24 03:51:12,988 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 0.46 sec
2013-05-24 03:51:13,995 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 0.46 sec
2013-05-24 03:51:15,004 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 0.46 sec
2013-05-24 03:51:16,013 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 0.46 sec
2013-05-24 03:51:17,020 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 0.46 sec
MapReduce Total cumulative CPU time: 460 msec
Ended Job = job_201305240156_0012
MapReduce Jobs Launched:
Job 0: Map: 1 Cumulative CPU: 0.46 sec HDFS Read: 848 HDFS Write: 0 SUCCESS
Total MapReduce CPU Time Spent: 460 msec
OK
full_name
Time taken: 19.558 seconds
But this query returns data successfully:
hive (test)> select full_name from states where abbreviation like '%A%';
Result of Query2:
Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks is set to 0 since there's no reduce operator
Starting Job = job_201305240156_0011, Tracking URL = http://ubuntu:50030/jobdetails.jsp?jobid=job_201305240156_0011
Kill Command = /home/satish/work/hadoop-1.0.4/libexec/../bin/hadoop job -kill job_201305240156_0011
Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 0
2013-05-24 03:50:32,163 Stage-1 map = 0%, reduce = 0%
2013-05-24 03:50:38,193 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 0.47 sec
2013-05-24 03:50:39,196 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 0.47 sec
2013-05-24 03:50:40,199 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 0.47 sec
2013-05-24 03:50:41,206 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 0.47 sec
2013-05-24 03:50:42,210 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 0.47 sec
2013-05-24 03:50:43,221 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 0.47 sec
2013-05-24 03:50:44,227 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 0.47 sec
MapReduce Total cumulative CPU time: 470 msec
Ended Job = job_201305240156_0011
MapReduce Jobs Launched:
Job 0: Map: 1 Cumulative CPU: 0.47 sec HDFS Read: 848 HDFS Write: 115 SUCCESS
Total MapReduce CPU Time Spent: 470 msec
OK
full_name
Alabama
Alaska
Arizona
Arkansas
California
Georgia
Iowa
Louisiana
Massachusetts
Pennsylvania
Virginia
Washington
Time taken: 20.551 seconds
Thanks
Sai
Re: java.lang.NoClassDefFoundError: com/jayway/jsonpath/PathUtil
Posted by Sai Sai <sa...@yahoo.in>.
Many Thanks Guys, you guys r really helpful. Really appreciate it.
Thanks
Sai
________________________________
From: "bejoy_ks@yahoo.com" <be...@yahoo.com>
To: user@hive.apache.org; Sai Sai <sa...@yahoo.in>
Sent: Sunday, 10 March 2013 12:06 PM
Subject: Re: java.lang.NoClassDefFoundError: com/jayway/jsonpath/PathUtil
Hi Sai
Local mode is just for trials, for any pre prod/production environment you need MR jobs.
Hive under the hood stores data in HDFS (mostly) and definitely we use hadoop/hive for larger data volumes. So MR should be in there to process them.
Regards
Bejoy KS
Sent from remote device, Please excuse typos
________________________________
From: Ramki Palle <ra...@gmail.com>
Date: Sun, 10 Mar 2013 06:58:57 -0700
To: <us...@hive.apache.org>; Sai Sai<sa...@yahoo.in>
ReplyTo: user@hive.apache.org
Subject: Re: java.lang.NoClassDefFoundError: com/jayway/jsonpath/PathUtil
Well, you get the results faster.
Please check this:
https://cwiki.apache.org/Hive/gettingstarted.html#GettingStarted-Runtimeconfiguration
Under section "Hive, Map-Reduce and Local-Mode", it says
This can be very useful to run queries over small data sets - in such
cases local mode execution is usually significantly faster than
submitting jobs to a large cluster.
-Ramki.
On Sun, Mar 10, 2013 at 5:26 AM, Sai Sai <sa...@yahoo.in> wrote:
Ramki/John
>Many Thanks, that really helped. I have run the add jars in the new session and it appears to be running. However i was wondering about by passing MR, why would we do it and what is the use of it. Will appreciate any input.
>Thanks
>Sai
>
>
>
>
>
>
>________________________________
> From: Ramki Palle <ra...@gmail.com>
>
>To: user@hive.apache.org; Sai Sai <sa...@yahoo.in>
>Sent: Sunday, 10 March 2013 4:22 AM
>Subject: Re: java.lang.NoClassDefFoundError: com/jayway/jsonpath/PathUtil
>
>
>
>When you execute the following query,
>
>hive> select * from twitter limit 5;
>
>Hive runs it in local mode and not use MapReduce.
>
>For the query,
>
>hive> select tweet_id from twitter limit 5;
>
>I think you need to add JSON jars to overcome this error. You might have added these in a previous session. If you want these jars available for all sessions, insert the add jar statements to your $HOME/.hiverc file.
>
>
>
>To bypass MapReduce
>
>set hive.exec.mode.local.auto = true;
>
>to suggest Hive to use local mode to execute the query. If it still uses MR, try
>
>set hive.fetch.task.conversion = more;.
>
>
>-Ramki.
>
>
>
>
>
>
>On Sun, Mar 10, 2013 at 12:19 AM, Sai Sai <sa...@yahoo.in> wrote:
>
>Just wondering if anyone has any suggestions:
>>
>>
>>This executes successfully:
>>
>>
>>hive> select * from twitter limit 5;
>>
>>
>>This does not work:
>>
>>
>>hive> select tweet_id from twitter limit 5; // I have given the exception info below:
>>
>>
>>
>>Here is the output of this:
>>
>>
>>hive> select * from twitter limit 5;
>>OK
>>
>>
>>
>>tweet_id created_at text user_id user_screen_name user_lang
>>122106088022745088 Fri Oct 07 00:28:54 +0000 2011 wkwkw -_- ayo saja mba RT @yullyunet: Sepupuuu, kita lanjalan yok.. Kita karokoe-an.. Ajak mas galih jg kalo dia mau.. "@Dindnf: doremifas 124735434 Dindnf en
>>122106088018558976 Fri Oct 07 00:28:54 +0000 2011 @egg486 특별히 준비했습니다! 252828803 CocaCola_Korea ko
>>122106088026939392 Fri Oct 07 00:28:54 +0000 2011 My offer of free gobbies for all if @amityaffliction play Blair snitch project still
stands. 168590073 SarahYoungBlood en
>>122106088035328001 Fri Oct 07 00:28:54 +0000 2011 the girl nxt to me in the lib got her headphones in dancing and singing loud af like she the only one here haha 267296295 MONEYyDREAMS_ en
>>122106088005971968 Fri Oct 07 00:28:54 +0000 2011 @KUnYoong_B2UTY Bị lsao đấy 269182160 b2st_b2utyhp en
>>Time taken: 0.154 seconds
>>
>>
>>
>>This does not work:
>>
>>
>>hive> select tweet_id from twitter limit 5;
>>
>>
>>
>>
>>
>>Total MapReduce jobs = 1
>>Launching Job 1 out of 1
>>Number of reduce tasks is set to 0 since there's no reduce operator
>>Starting Job = job_201303050432_0094, Tracking URL = http://ubuntu:50030/jobdetails.jsp?jobid=job_201303050432_0094
>>Kill Command = /home/satish/work/hadoop-1.0.4/libexec/../bin/hadoop job -kill job_201303050432_0094
>>Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 0
>>2013-03-10 00:14:44,509 Stage-1 map = 0%, reduce = 0%
>>2013-03-10 00:15:14,613 Stage-1 map = 100%, reduce = 100%
>>Ended Job = job_201303050432_0094 with errors
>>Error during job, obtaining debugging information...
>>Job Tracking URL: http://ubuntu:50030/jobdetails.jsp?jobid=job_201303050432_0094
>>Examining task ID: task_201303050432_0094_m_000002 (and more) from job job_201303050432_0094
>>
>>Task with the most failures(4):
>>-----
>>Task ID:
>> task_201303050432_0094_m_000000
>>
>>URL:
>> http://ubuntu:50030/taskdetails.jsp?jobid=job_201303050432_0094&tipid=task_201303050432_0094_m_000000
>>-----
>>Diagnostic Messages for this Task:
>>java.lang.RuntimeException: Error in configuring object
>> at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:93)
>> at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64)
>> at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
>> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:432)
>> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372)
>> at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at javax.security.auth.Subject.doAs(Subject.java:416)
>> at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
>> at org.apache.hadoop.mapred.Child.main(Child.java:249)
>>Caused by: java.lang.reflect.InvocationTargetException
>>
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:616)
>> at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88)
>> ... 9 more
>>Caused by: java.lang.RuntimeException: Error in configuring object
>> at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:93)
>> at
org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64)
>> at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
>> at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:34)
>> ... 14 more
>>Caused by: java.lang.reflect.InvocationTargetException
>>
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:616)
>> at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88)
>> ... 17 more
>>Caused by: java.lang.RuntimeException: Map operator initialization failed
>> at
org.apache.hadoop.hive.ql.exec.ExecMapper.configure(ExecMapper.java:121)
>> ... 22 more
>>Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.ClassNotFoundException: org.apache.hadoop.hive.contrib.serde2.JsonSerde
>> at org.apache.hadoop.hive.ql.exec.MapOperator.setChildren(MapOperator.java:420)
>> at org.apache.hadoop.hive.ql.exec.ExecMapper.configure(ExecMapper.java:90)
>> ... 22 more
>>Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hive.contrib.serde2.JsonSerde
>>
>> at java.net.URLClassLoader$1.run(URLClassLoader.java:217)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at java.net.URLClassLoader.findClass(URLClassLoader.java:205)
>> at java.lang.ClassLoader.loadClass(ClassLoader.java:321)
>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:294)
>> at java.lang.ClassLoader.loadClass(ClassLoader.java:266)
>> at java.lang.Class.forName0(Native Method)
>> at java.lang.Class.forName(Class.java:264)
>> at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:820)
>> at org.apache.hadoop.hive.ql.exec.MapOperator.initObjectInspector(MapOperator.java:243)
>> at org.apache.hadoop.hive.ql.exec.MapOperator.setChildren(MapOperator.java:380)
>> ... 23 more
>>
>>
>>FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.MapRedTask
>>MapReduce Jobs Launched:
>>Job 0: Map: 1 HDFS Read: 0 HDFS Write: 0 FAIL
>>Total MapReduce CPU Time Spent: 0 msec
>>
>>
>>Thanks
>>Sai
>>
>>
>>________________________________
>> From: Dean Wampler <de...@thinkbiganalytics.com>
>>To: user@hive.apache.org; Sai Sai <sa...@yahoo.in>
>>Sent: Friday, 8 March 2013 5:22 AM
>>Subject: Re: java.lang.NoClassDefFoundError: com/jayway/jsonpath/PathUtil
>>
>>
>>
>>Unfortunately, you have to also add the json jars to Hive's class path before it starts, e.g.,
>>
>>
>>env HADOOP_CLASSPATH=/path/to/lib/*.jar hive
>>
>>Use the appropriate path to your lib directory.
>>
>>
>>On Fri, Mar 8, 2013 at 4:53 AM, Sai Sai <sa...@yahoo.in> wrote:
>>
>>I have added the jar files successfully like this:
>>>
>>>
>>>
>>>
>>>hive (testdb)> ADD JAR lib/hive-json-serde-0.3.jar;
>>> Added lib/hive-json-serde-0.3.jar to class path
>>> Added resource: lib/hive-json-serde-0.3.jar
>>>
>>>
>>>
>>>
>>>
>>>hive (testdb)> ADD JAR lib/json-path-0.5.4.jar;
>>> Added lib/json-path-0.5.4.jar to class path
>>> Added resource: lib/json-path-0.5.4.jar
>>>
>>>
>>>
>>>
>>>
>>>hive (testdb)> ADD JAR lib/json-smart-1.0.6.3.jar;
>>> Added lib/json-smart-1.0.6.3.jar to class path
>>> Added resource: lib/json-smart-1.0.6.3.jar
>>>
>>>
>>>
>>>
>>>After this i am getting this error:
>>>
>>>
>>>
>>>
>>>CREATE EXTERNAL TABLE IF NOT EXISTS twitter (tweet_id BIGINT,created_at STRING,text STRING,user_id BIGINT, user_screen_name STRING,user_lang STRING) ROW FORMAT SERDE "org.apache.hadoop.hive.contrib.serde2.JsonSerde" WITH SERDEPROPERTIES ( "tweet_id"="$.id","created_at"="$.created_at","text"="$.text","user_id"="$.user.id","user_screen_name"="$.user.screen_name", "user_lang"="$.user.lang") LOCATION '/home/satish/data/twitter/input';
>>>java.lang.NoClassDefFoundError: com/jayway/jsonpath/PathUtil
>>> at org.apache.hadoop.hive.contrib.serde2.JsonSerde.initialize(Unknown Source)
>>> at org.apache.hadoop.hive.metastore.MetaStoreUtils.getDeserializer(MetaStoreUtils.java:207)
>>> at org.apache.hadoop.hive.ql.metadata.Table.getDeserializerFromMetaStore(Table.java:266)
>>> at org.apache.hadoop.hive.ql.metadata.Table.getDeserializer(Table.java:259)
>>> at org.apache.hadoop.hive.ql.metadata.Table.getCols(Table.java:585)
>>> at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:550)
>>> at org.apache.hadoop.hive.ql.exec.DDLTask.createTable(DDLTask.java:3698)
>>> at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:253)
>>> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:138)
>>> at
org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:57)
>>> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1336)
>>> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1122)
>>> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:935)
>>> at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:259)
>>> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:216)
>>> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:412)
>>> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:755)
>>> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:613)
>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>> at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>> at java.lang.reflect.Method.invoke(Method.java:616)
>>> at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
>>>Caused by: java.lang.ClassNotFoundException: com.jayway.jsonpath.PathUtil
>>> at java.net.URLClassLoader$1.run(URLClassLoader.java:217)
>>> at java.security.AccessController.doPrivileged(Native Method)
>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:205)
>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:321)
>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:266)
>>> ... 23 more
>>>FAILED: Execution Error, return code -101 from org.apache.hadoop.hive.ql.exec.DDLTask
>>>
>>>
>>>
>>>
>>>
>>>Any help would be really appreciated.
>>>ThanksSai
>>>
>>
>>
>>
>>--
>>Dean Wampler, Ph.D.
>>thinkbiganalytics.com
>>+1-312-339-1330
>>
>>
>>
>>
>
>
>
Re: java.lang.NoClassDefFoundError: com/jayway/jsonpath/PathUtil
Posted by be...@yahoo.com.
Hi Sai
Local mode is just for trials, for any pre prod/production environment you need MR jobs.
Hive under the hood stores data in HDFS (mostly) and definitely we use hadoop/hive for larger data volumes. So MR should be in there to process them.
Regards
Bejoy KS
Sent from remote device, Please excuse typos
-----Original Message-----
From: Ramki Palle <ra...@gmail.com>
Date: Sun, 10 Mar 2013 06:58:57
To: <us...@hive.apache.org>; Sai Sai<sa...@yahoo.in>
Reply-To: user@hive.apache.org
Subject: Re: java.lang.NoClassDefFoundError: com/jayway/jsonpath/PathUtil
Well, you get the results faster.
Please check this:
https://cwiki.apache.org/Hive/gettingstarted.html#GettingStarted-Runtimeconfiguration
Under section "Hive, Map-Reduce and Local-Mode", it says
This can be very useful to run queries over small data sets - in such cases
local mode execution is usually significantly faster than submitting jobs
to a large cluster.
-Ramki.
On Sun, Mar 10, 2013 at 5:26 AM, Sai Sai <sa...@yahoo.in> wrote:
> Ramki/John
> Many Thanks, that really helped. I have run the add jars in the new
> session and it appears to be running. However i was wondering about by
> passing MR, why would we do it and what is the use of it. Will appreciate
> any input.
> Thanks
> Sai
>
>
> ------------------------------
> *From:* Ramki Palle <ra...@gmail.com>
>
> *To:* user@hive.apache.org; Sai Sai <sa...@yahoo.in>
> *Sent:* Sunday, 10 March 2013 4:22 AM
> *Subject:* Re: java.lang.NoClassDefFoundError:
> com/jayway/jsonpath/PathUtil
>
> When you execute the following query,
>
> hive> select * from twitter limit 5;
>
> Hive runs it in local mode and not use MapReduce.
>
> For the query,
>
> hive> select tweet_id from twitter limit 5;
>
> I think you need to add JSON jars to overcome this error. You might have
> added these in a previous session. If you want these jars available for all
> sessions, insert the add jar statements to your $HOME/.hiverc file.
>
>
> To bypass MapReduce
>
> set hive.exec.mode.local.auto = true;
>
> to suggest Hive to use local mode to execute the query. If it still uses
> MR, try
>
> set hive.fetch.task.conversion = more;.
>
>
> -Ramki.
>
>
>
> On Sun, Mar 10, 2013 at 12:19 AM, Sai Sai <sa...@yahoo.in> wrote:
>
> Just wondering if anyone has any suggestions:
>
> This executes successfully:
>
> hive> select * from twitter limit 5;
>
> This does not work:
>
> hive> select tweet_id from twitter limit 5; // I have given the exception
> info below:
>
> Here is the output of this:
>
> hive> select * from twitter limit 5;
> OK
>
> tweet_id created_at text user_id user_screen_name user_lang
> 122106088022745088 Fri Oct 07 00:28:54 +0000 2011 wkwkw -_- ayo saja
> mba RT @yullyunet: Sepupuuu, kita lanjalan yok.. Kita karokoe-an.. Ajak mas
> galih jg kalo dia mau.. "@Dindnf: doremifas 124735434 Dindnf en
> 122106088018558976 Fri Oct 07 00:28:54 +0000 2011 @egg486 특별히
> 준비했습니다! 252828803 CocaCola_Korea ko
> 122106088026939392 Fri Oct 07 00:28:54 +0000 2011 My offer of free
> gobbies for all if @amityaffliction play Blair snitch project still
> stands. 168590073 SarahYoungBlood en
> 122106088035328001 Fri Oct 07 00:28:54 +0000 2011 the girl nxt to me
> in the lib got her headphones in dancing and singing loud af like she the
> only one here haha 267296295 MONEYyDREAMS_ en
> 122106088005971968 Fri Oct 07 00:28:54 +0000 2011 @KUnYoong_B2UTY
> Bị lsao đấy 269182160 b2st_b2utyhp en
> Time taken: 0.154 seconds
>
> This does not work:
>
> hive> select tweet_id from twitter limit 5;
>
>
> Total MapReduce jobs = 1
> Launching Job 1 out of 1
> Number of reduce tasks is set to 0 since there's no reduce operator
> Starting Job = job_201303050432_0094, Tracking URL =
> http://ubuntu:50030/jobdetails.jsp?jobid=job_201303050432_0094
> Kill Command = /home/satish/work/hadoop-1.0.4/libexec/../bin/hadoop job
> -kill job_201303050432_0094
> Hadoop job information for Stage-1: number of mappers: 1; number of
> reducers: 0
> 2013-03-10 00:14:44,509 Stage-1 map = 0%, reduce = 0%
> 2013-03-10 00:15:14,613 Stage-1 map = 100%, reduce = 100%
> Ended Job = job_201303050432_0094 with errors
> Error during job, obtaining debugging information...
> Job Tracking URL:
> http://ubuntu:50030/jobdetails.jsp?jobid=job_201303050432_0094
> Examining task ID: task_201303050432_0094_m_000002 (and more) from job
> job_201303050432_0094
>
> Task with the most failures(4):
> -----
> Task ID:
> task_201303050432_0094_m_000000
>
> URL:
>
> http://ubuntu:50030/taskdetails.jsp?jobid=job_201303050432_0094&tipid=task_201303050432_0094_m_000000
> -----
> Diagnostic Messages for this Task:
> java.lang.RuntimeException: Error in configuring object
> at
> org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:93)
> at
> org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64)
> at
> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:432)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372)
> at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:416)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
> at org.apache.hadoop.mapred.Child.main(Child.java:249)
> Caused by: java.lang.reflect.InvocationTargetException
>
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at
> org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88)
> ... 9 more
> Caused by: java.lang.RuntimeException: Error in configuring object
> at
> org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:93)
> at
> org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64)
> at
> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
> at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:34)
> ... 14 more
> Caused by: java.lang.reflect.InvocationTargetException
>
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at
> org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88)
> ... 17 more
> Caused by: java.lang.RuntimeException: Map operator initialization failed
> at
> org.apache.hadoop.hive.ql.exec.ExecMapper.configure(ExecMapper.java:121)
> ... 22 more
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
> java.lang.ClassNotFoundException:
> org.apache.hadoop.hive.contrib.serde2.JsonSerde
> at
> org.apache.hadoop.hive.ql.exec.MapOperator.setChildren(MapOperator.java:420)
> at
> org.apache.hadoop.hive.ql.exec.ExecMapper.configure(ExecMapper.java:90)
> ... 22 more
> Caused by: java.lang.ClassNotFoundException:
> org.apache.hadoop.hive.contrib.serde2.JsonSerde
>
> at java.net.URLClassLoader$1.run(URLClassLoader.java:217)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:205)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:321)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:294)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:266)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:264)
> at
> org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:820)
> at
> org.apache.hadoop.hive.ql.exec.MapOperator.initObjectInspector(MapOperator.java:243)
> at
> org.apache.hadoop.hive.ql.exec.MapOperator.setChildren(MapOperator.java:380)
> ... 23 more
>
>
> FAILED: Execution Error, return code 2 from
> org.apache.hadoop.hive.ql.exec.MapRedTask
> MapReduce Jobs Launched:
> Job 0: Map: 1 HDFS Read: 0 HDFS Write: 0 FAIL
> Total MapReduce CPU Time Spent: 0 msec
>
> Thanks
> Sai
> ------------------------------
> *From:* Dean Wampler <de...@thinkbiganalytics.com>
> *To:* user@hive.apache.org; Sai Sai <sa...@yahoo.in>
> *Sent:* Friday, 8 March 2013 5:22 AM
> *Subject:* Re: java.lang.NoClassDefFoundError:
> com/jayway/jsonpath/PathUtil
>
> Unfortunately, you have to also add the json jars to Hive's class path
> before it starts, e.g.,
>
> env HADOOP_CLASSPATH=/path/to/lib/*.jar hive
>
> Use the appropriate path to your lib directory.
>
> On Fri, Mar 8, 2013 at 4:53 AM, Sai Sai <sa...@yahoo.in> wrote:
>
> I have added the jar files successfully like this:
>
>
> hive (testdb)> ADD JAR lib/hive-json-serde-0.3.jar;
> Added lib/hive-json-serde-0.3.jar to class path
> Added resource: lib/hive-json-serde-0.3.jar
>
>
> hive (testdb)> ADD JAR lib/json-path-0.5.4.jar;
> Added lib/json-path-0.5.4.jar to class path
> Added resource: lib/json-path-0.5.4.jar
>
>
> hive (testdb)> ADD JAR lib/json-smart-1.0.6.3.jar;
> Added lib/json-smart-1.0.6.3.jar to class path
> Added resource: lib/json-smart-1.0.6.3.jar
>
>
> After this i am getting this error:
>
>
> CREATE EXTERNAL TABLE IF NOT EXISTS twitter (tweet_id BIGINT,created_at
> STRING,text STRING,user_id BIGINT, user_screen_name STRING,user_lang
> STRING) ROW FORMAT SERDE "org.apache.hadoop.hive.contrib.serde2.JsonSerde"
> WITH SERDEPROPERTIES (
> "tweet_id"="$.id","created_at"="$.created_at","text"="$.text","user_id"="$.
> user.id","user_screen_name"="$.user.screen_name",
> "user_lang"="$.user.lang") LOCATION '/home/satish/data/twitter/input';
> java.lang.NoClassDefFoundError: com/jayway/jsonpath/PathUtil
> at org.apache.hadoop.hive.contrib.serde2.JsonSerde.initialize(Unknown
> Source)
> at
> org.apache.hadoop.hive.metastore.MetaStoreUtils.getDeserializer(MetaStoreUtils.java:207)
> at
> org.apache.hadoop.hive.ql.metadata.Table.getDeserializerFromMetaStore(Table.java:266)
> at
> org.apache.hadoop.hive.ql.metadata.Table.getDeserializer(Table.java:259)
> at org.apache.hadoop.hive.ql.metadata.Table.getCols(Table.java:585)
> at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:550)
> at
> org.apache.hadoop.hive.ql.exec.DDLTask.createTable(DDLTask.java:3698)
> at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:253)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:138)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:57)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1336)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1122)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:935)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:259)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:216)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:412)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:755)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:613)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
> Caused by: java.lang.ClassNotFoundException: com.jayway.jsonpath.PathUtil
> at java.net.URLClassLoader$1.run(URLClassLoader.java:217)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:205)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:321)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:266)
> ... 23 more
> FAILED: Execution Error, return code -101 from
> org.apache.hadoop.hive.ql.exec.DDLTask
>
>
> Any help would be really appreciated.
> Thanks
> Sai
>
>
>
>
> --
> *Dean Wampler, Ph.D.*
> thinkbiganalytics.com
> +1-312-339-1330
>
>
>
>
>
>
>
Re: java.lang.NoClassDefFoundError: com/jayway/jsonpath/PathUtil
Posted by Ramki Palle <ra...@gmail.com>.
Well, you get the results faster.
Please check this:
https://cwiki.apache.org/Hive/gettingstarted.html#GettingStarted-Runtimeconfiguration
Under section "Hive, Map-Reduce and Local-Mode", it says
This can be very useful to run queries over small data sets - in such cases
local mode execution is usually significantly faster than submitting jobs
to a large cluster.
-Ramki.
On Sun, Mar 10, 2013 at 5:26 AM, Sai Sai <sa...@yahoo.in> wrote:
> Ramki/John
> Many Thanks, that really helped. I have run the add jars in the new
> session and it appears to be running. However i was wondering about by
> passing MR, why would we do it and what is the use of it. Will appreciate
> any input.
> Thanks
> Sai
>
>
> ------------------------------
> *From:* Ramki Palle <ra...@gmail.com>
>
> *To:* user@hive.apache.org; Sai Sai <sa...@yahoo.in>
> *Sent:* Sunday, 10 March 2013 4:22 AM
> *Subject:* Re: java.lang.NoClassDefFoundError:
> com/jayway/jsonpath/PathUtil
>
> When you execute the following query,
>
> hive> select * from twitter limit 5;
>
> Hive runs it in local mode and not use MapReduce.
>
> For the query,
>
> hive> select tweet_id from twitter limit 5;
>
> I think you need to add JSON jars to overcome this error. You might have
> added these in a previous session. If you want these jars available for all
> sessions, insert the add jar statements to your $HOME/.hiverc file.
>
>
> To bypass MapReduce
>
> set hive.exec.mode.local.auto = true;
>
> to suggest Hive to use local mode to execute the query. If it still uses
> MR, try
>
> set hive.fetch.task.conversion = more;.
>
>
> -Ramki.
>
>
>
> On Sun, Mar 10, 2013 at 12:19 AM, Sai Sai <sa...@yahoo.in> wrote:
>
> Just wondering if anyone has any suggestions:
>
> This executes successfully:
>
> hive> select * from twitter limit 5;
>
> This does not work:
>
> hive> select tweet_id from twitter limit 5; // I have given the exception
> info below:
>
> Here is the output of this:
>
> hive> select * from twitter limit 5;
> OK
>
> tweet_id created_at text user_id user_screen_name user_lang
> 122106088022745088 Fri Oct 07 00:28:54 +0000 2011 wkwkw -_- ayo saja
> mba RT @yullyunet: Sepupuuu, kita lanjalan yok.. Kita karokoe-an.. Ajak mas
> galih jg kalo dia mau.. "@Dindnf: doremifas 124735434 Dindnf en
> 122106088018558976 Fri Oct 07 00:28:54 +0000 2011 @egg486 특별히
> 준비했습니다! 252828803 CocaCola_Korea ko
> 122106088026939392 Fri Oct 07 00:28:54 +0000 2011 My offer of free
> gobbies for all if @amityaffliction play Blair snitch project still
> stands. 168590073 SarahYoungBlood en
> 122106088035328001 Fri Oct 07 00:28:54 +0000 2011 the girl nxt to me
> in the lib got her headphones in dancing and singing loud af like she the
> only one here haha 267296295 MONEYyDREAMS_ en
> 122106088005971968 Fri Oct 07 00:28:54 +0000 2011 @KUnYoong_B2UTY
> Bị lsao đấy 269182160 b2st_b2utyhp en
> Time taken: 0.154 seconds
>
> This does not work:
>
> hive> select tweet_id from twitter limit 5;
>
>
> Total MapReduce jobs = 1
> Launching Job 1 out of 1
> Number of reduce tasks is set to 0 since there's no reduce operator
> Starting Job = job_201303050432_0094, Tracking URL =
> http://ubuntu:50030/jobdetails.jsp?jobid=job_201303050432_0094
> Kill Command = /home/satish/work/hadoop-1.0.4/libexec/../bin/hadoop job
> -kill job_201303050432_0094
> Hadoop job information for Stage-1: number of mappers: 1; number of
> reducers: 0
> 2013-03-10 00:14:44,509 Stage-1 map = 0%, reduce = 0%
> 2013-03-10 00:15:14,613 Stage-1 map = 100%, reduce = 100%
> Ended Job = job_201303050432_0094 with errors
> Error during job, obtaining debugging information...
> Job Tracking URL:
> http://ubuntu:50030/jobdetails.jsp?jobid=job_201303050432_0094
> Examining task ID: task_201303050432_0094_m_000002 (and more) from job
> job_201303050432_0094
>
> Task with the most failures(4):
> -----
> Task ID:
> task_201303050432_0094_m_000000
>
> URL:
>
> http://ubuntu:50030/taskdetails.jsp?jobid=job_201303050432_0094&tipid=task_201303050432_0094_m_000000
> -----
> Diagnostic Messages for this Task:
> java.lang.RuntimeException: Error in configuring object
> at
> org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:93)
> at
> org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64)
> at
> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:432)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372)
> at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:416)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
> at org.apache.hadoop.mapred.Child.main(Child.java:249)
> Caused by: java.lang.reflect.InvocationTargetException
>
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at
> org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88)
> ... 9 more
> Caused by: java.lang.RuntimeException: Error in configuring object
> at
> org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:93)
> at
> org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64)
> at
> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
> at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:34)
> ... 14 more
> Caused by: java.lang.reflect.InvocationTargetException
>
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at
> org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88)
> ... 17 more
> Caused by: java.lang.RuntimeException: Map operator initialization failed
> at
> org.apache.hadoop.hive.ql.exec.ExecMapper.configure(ExecMapper.java:121)
> ... 22 more
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
> java.lang.ClassNotFoundException:
> org.apache.hadoop.hive.contrib.serde2.JsonSerde
> at
> org.apache.hadoop.hive.ql.exec.MapOperator.setChildren(MapOperator.java:420)
> at
> org.apache.hadoop.hive.ql.exec.ExecMapper.configure(ExecMapper.java:90)
> ... 22 more
> Caused by: java.lang.ClassNotFoundException:
> org.apache.hadoop.hive.contrib.serde2.JsonSerde
>
> at java.net.URLClassLoader$1.run(URLClassLoader.java:217)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:205)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:321)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:294)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:266)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:264)
> at
> org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:820)
> at
> org.apache.hadoop.hive.ql.exec.MapOperator.initObjectInspector(MapOperator.java:243)
> at
> org.apache.hadoop.hive.ql.exec.MapOperator.setChildren(MapOperator.java:380)
> ... 23 more
>
>
> FAILED: Execution Error, return code 2 from
> org.apache.hadoop.hive.ql.exec.MapRedTask
> MapReduce Jobs Launched:
> Job 0: Map: 1 HDFS Read: 0 HDFS Write: 0 FAIL
> Total MapReduce CPU Time Spent: 0 msec
>
> Thanks
> Sai
> ------------------------------
> *From:* Dean Wampler <de...@thinkbiganalytics.com>
> *To:* user@hive.apache.org; Sai Sai <sa...@yahoo.in>
> *Sent:* Friday, 8 March 2013 5:22 AM
> *Subject:* Re: java.lang.NoClassDefFoundError:
> com/jayway/jsonpath/PathUtil
>
> Unfortunately, you have to also add the json jars to Hive's class path
> before it starts, e.g.,
>
> env HADOOP_CLASSPATH=/path/to/lib/*.jar hive
>
> Use the appropriate path to your lib directory.
>
> On Fri, Mar 8, 2013 at 4:53 AM, Sai Sai <sa...@yahoo.in> wrote:
>
> I have added the jar files successfully like this:
>
>
> hive (testdb)> ADD JAR lib/hive-json-serde-0.3.jar;
> Added lib/hive-json-serde-0.3.jar to class path
> Added resource: lib/hive-json-serde-0.3.jar
>
>
> hive (testdb)> ADD JAR lib/json-path-0.5.4.jar;
> Added lib/json-path-0.5.4.jar to class path
> Added resource: lib/json-path-0.5.4.jar
>
>
> hive (testdb)> ADD JAR lib/json-smart-1.0.6.3.jar;
> Added lib/json-smart-1.0.6.3.jar to class path
> Added resource: lib/json-smart-1.0.6.3.jar
>
>
> After this i am getting this error:
>
>
> CREATE EXTERNAL TABLE IF NOT EXISTS twitter (tweet_id BIGINT,created_at
> STRING,text STRING,user_id BIGINT, user_screen_name STRING,user_lang
> STRING) ROW FORMAT SERDE "org.apache.hadoop.hive.contrib.serde2.JsonSerde"
> WITH SERDEPROPERTIES (
> "tweet_id"="$.id","created_at"="$.created_at","text"="$.text","user_id"="$.
> user.id","user_screen_name"="$.user.screen_name",
> "user_lang"="$.user.lang") LOCATION '/home/satish/data/twitter/input';
> java.lang.NoClassDefFoundError: com/jayway/jsonpath/PathUtil
> at org.apache.hadoop.hive.contrib.serde2.JsonSerde.initialize(Unknown
> Source)
> at
> org.apache.hadoop.hive.metastore.MetaStoreUtils.getDeserializer(MetaStoreUtils.java:207)
> at
> org.apache.hadoop.hive.ql.metadata.Table.getDeserializerFromMetaStore(Table.java:266)
> at
> org.apache.hadoop.hive.ql.metadata.Table.getDeserializer(Table.java:259)
> at org.apache.hadoop.hive.ql.metadata.Table.getCols(Table.java:585)
> at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:550)
> at
> org.apache.hadoop.hive.ql.exec.DDLTask.createTable(DDLTask.java:3698)
> at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:253)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:138)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:57)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1336)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1122)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:935)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:259)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:216)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:412)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:755)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:613)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
> Caused by: java.lang.ClassNotFoundException: com.jayway.jsonpath.PathUtil
> at java.net.URLClassLoader$1.run(URLClassLoader.java:217)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:205)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:321)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:266)
> ... 23 more
> FAILED: Execution Error, return code -101 from
> org.apache.hadoop.hive.ql.exec.DDLTask
>
>
> Any help would be really appreciated.
> Thanks
> Sai
>
>
>
>
> --
> *Dean Wampler, Ph.D.*
> thinkbiganalytics.com
> +1-312-339-1330
>
>
>
>
>
>
>
Re: java.lang.NoClassDefFoundError: com/jayway/jsonpath/PathUtil
Posted by Sai Sai <sa...@yahoo.in>.
Ramki/John
Many Thanks, that really helped. I have run the add jars in the new session and it appears to be running. However i was wondering about by passing MR, why would we do it and what is the use of it. Will appreciate any input.
Thanks
Sai
________________________________
From: Ramki Palle <ra...@gmail.com>
To: user@hive.apache.org; Sai Sai <sa...@yahoo.in>
Sent: Sunday, 10 March 2013 4:22 AM
Subject: Re: java.lang.NoClassDefFoundError: com/jayway/jsonpath/PathUtil
When you execute the following query,
hive> select * from twitter limit 5;
Hive runs it in local mode and not use MapReduce.
For the query,
hive> select tweet_id from twitter limit 5;
I think you need to add JSON jars to overcome this error. You might have added these in a previous session. If you want these jars available for all sessions, insert the add jar statements to your $HOME/.hiverc file.
To bypass MapReduce
set hive.exec.mode.local.auto = true;
to suggest Hive to use local mode to execute the query. If it still uses MR, try
set hive.fetch.task.conversion = more;.
-Ramki.
On Sun, Mar 10, 2013 at 12:19 AM, Sai Sai <sa...@yahoo.in> wrote:
Just wondering if anyone has any suggestions:
>
>
>This executes successfully:
>
>
>hive> select * from twitter limit 5;
>
>
>This does not work:
>
>
>hive> select tweet_id from twitter limit 5; // I have given the exception info below:
>
>
>
>Here is the output of this:
>
>
>hive> select * from twitter limit 5;
>OK
>
>
>
>tweet_id created_at text user_id user_screen_name user_lang
>122106088022745088 Fri Oct 07 00:28:54 +0000 2011 wkwkw -_- ayo saja mba RT @yullyunet: Sepupuuu, kita lanjalan yok.. Kita karokoe-an.. Ajak mas galih jg kalo dia mau.. "@Dindnf: doremifas 124735434 Dindnf en
>122106088018558976 Fri Oct 07 00:28:54 +0000 2011 @egg486 특별히 준비했습니다! 252828803 CocaCola_Korea ko
>122106088026939392 Fri Oct 07 00:28:54 +0000 2011 My offer of free gobbies for all if @amityaffliction play Blair snitch project still
stands. 168590073 SarahYoungBlood en
>122106088035328001 Fri Oct 07 00:28:54 +0000 2011 the girl nxt to me in the lib got her headphones in dancing and singing loud af like she the only one here haha 267296295 MONEYyDREAMS_ en
>122106088005971968 Fri Oct 07 00:28:54 +0000 2011 @KUnYoong_B2UTY Bị lsao đấy 269182160 b2st_b2utyhp en
>Time taken: 0.154 seconds
>
>
>
>This does not work:
>
>
>hive> select tweet_id from twitter limit 5;
>
>
>
>
>
>Total MapReduce jobs = 1
>Launching Job 1 out of 1
>Number of reduce tasks is set to 0 since there's no reduce operator
>Starting Job = job_201303050432_0094, Tracking URL = http://ubuntu:50030/jobdetails.jsp?jobid=job_201303050432_0094
>Kill Command = /home/satish/work/hadoop-1.0.4/libexec/../bin/hadoop job -kill job_201303050432_0094
>Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 0
>2013-03-10 00:14:44,509 Stage-1 map = 0%, reduce = 0%
>2013-03-10 00:15:14,613 Stage-1 map = 100%, reduce = 100%
>Ended Job = job_201303050432_0094 with errors
>Error during job, obtaining debugging information...
>Job Tracking URL: http://ubuntu:50030/jobdetails.jsp?jobid=job_201303050432_0094
>Examining task ID: task_201303050432_0094_m_000002 (and more) from job job_201303050432_0094
>
>Task with the most failures(4):
>-----
>Task ID:
> task_201303050432_0094_m_000000
>
>URL:
> http://ubuntu:50030/taskdetails.jsp?jobid=job_201303050432_0094&tipid=task_201303050432_0094_m_000000
>-----
>Diagnostic Messages for this Task:
>java.lang.RuntimeException: Error in configuring object
> at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:93)
> at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64)
> at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:432)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372)
> at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:416)
> at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
> at org.apache.hadoop.mapred.Child.main(Child.java:249)
>Caused by: java.lang.reflect.InvocationTargetException
>
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88)
> ... 9 more
>Caused by: java.lang.RuntimeException: Error in configuring object
> at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:93)
> at
org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64)
> at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
> at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:34)
> ... 14 more
>Caused by: java.lang.reflect.InvocationTargetException
>
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88)
> ... 17 more
>Caused by: java.lang.RuntimeException: Map operator initialization failed
> at
org.apache.hadoop.hive.ql.exec.ExecMapper.configure(ExecMapper.java:121)
> ... 22 more
>Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.ClassNotFoundException: org.apache.hadoop.hive.contrib.serde2.JsonSerde
> at org.apache.hadoop.hive.ql.exec.MapOperator.setChildren(MapOperator.java:420)
> at org.apache.hadoop.hive.ql.exec.ExecMapper.configure(ExecMapper.java:90)
> ... 22 more
>Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hive.contrib.serde2.JsonSerde
>
> at java.net.URLClassLoader$1.run(URLClassLoader.java:217)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:205)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:321)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:294)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:266)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:264)
> at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:820)
> at org.apache.hadoop.hive.ql.exec.MapOperator.initObjectInspector(MapOperator.java:243)
> at org.apache.hadoop.hive.ql.exec.MapOperator.setChildren(MapOperator.java:380)
> ... 23 more
>
>
>FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.MapRedTask
>MapReduce Jobs Launched:
>Job 0: Map: 1 HDFS Read: 0 HDFS Write: 0 FAIL
>Total MapReduce CPU Time Spent: 0 msec
>
>
>Thanks
>Sai
>
>
>________________________________
> From: Dean Wampler <de...@thinkbiganalytics.com>
>To: user@hive.apache.org; Sai Sai <sa...@yahoo.in>
>Sent: Friday, 8 March 2013 5:22 AM
>Subject: Re: java.lang.NoClassDefFoundError: com/jayway/jsonpath/PathUtil
>
>
>
>Unfortunately, you have to also add the json jars to Hive's class path before it starts, e.g.,
>
>
>env HADOOP_CLASSPATH=/path/to/lib/*.jar hive
>
>Use the appropriate path to your lib directory.
>
>
>On Fri, Mar 8, 2013 at 4:53 AM, Sai Sai <sa...@yahoo.in> wrote:
>
>I have added the jar files successfully like this:
>>
>>
>>
>>
>>hive (testdb)> ADD JAR lib/hive-json-serde-0.3.jar;
>> Added lib/hive-json-serde-0.3.jar to class path
>> Added resource: lib/hive-json-serde-0.3.jar
>>
>>
>>
>>
>>
>>hive (testdb)> ADD JAR lib/json-path-0.5.4.jar;
>> Added lib/json-path-0.5.4.jar to class path
>> Added resource: lib/json-path-0.5.4.jar
>>
>>
>>
>>
>>
>>hive (testdb)> ADD JAR lib/json-smart-1.0.6.3.jar;
>> Added lib/json-smart-1.0.6.3.jar to class path
>> Added resource: lib/json-smart-1.0.6.3.jar
>>
>>
>>
>>
>>After this i am getting this error:
>>
>>
>>
>>
>>CREATE EXTERNAL TABLE IF NOT EXISTS twitter (tweet_id BIGINT,created_at STRING,text STRING,user_id BIGINT, user_screen_name STRING,user_lang STRING) ROW FORMAT SERDE "org.apache.hadoop.hive.contrib.serde2.JsonSerde" WITH SERDEPROPERTIES ( "tweet_id"="$.id","created_at"="$.created_at","text"="$.text","user_id"="$.user.id","user_screen_name"="$.user.screen_name", "user_lang"="$.user.lang") LOCATION '/home/satish/data/twitter/input';
>>java.lang.NoClassDefFoundError: com/jayway/jsonpath/PathUtil
>> at org.apache.hadoop.hive.contrib.serde2.JsonSerde.initialize(Unknown Source)
>> at org.apache.hadoop.hive.metastore.MetaStoreUtils.getDeserializer(MetaStoreUtils.java:207)
>> at org.apache.hadoop.hive.ql.metadata.Table.getDeserializerFromMetaStore(Table.java:266)
>> at org.apache.hadoop.hive.ql.metadata.Table.getDeserializer(Table.java:259)
>> at org.apache.hadoop.hive.ql.metadata.Table.getCols(Table.java:585)
>> at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:550)
>> at org.apache.hadoop.hive.ql.exec.DDLTask.createTable(DDLTask.java:3698)
>> at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:253)
>> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:138)
>> at
org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:57)
>> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1336)
>> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1122)
>> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:935)
>> at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:259)
>> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:216)
>> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:412)
>> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:755)
>> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:613)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:616)
>> at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
>>Caused by: java.lang.ClassNotFoundException: com.jayway.jsonpath.PathUtil
>> at java.net.URLClassLoader$1.run(URLClassLoader.java:217)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at java.net.URLClassLoader.findClass(URLClassLoader.java:205)
>> at java.lang.ClassLoader.loadClass(ClassLoader.java:321)
>> at java.lang.ClassLoader.loadClass(ClassLoader.java:266)
>> ... 23 more
>>FAILED: Execution Error, return code -101 from org.apache.hadoop.hive.ql.exec.DDLTask
>>
>>
>>
>>
>>
>>Any help would be really appreciated.
>>ThanksSai
>>
>
>
>
>--
>Dean Wampler, Ph.D.
>thinkbiganalytics.com
>+1-312-339-1330
>
>
>
>
Re: java.lang.NoClassDefFoundError: com/jayway/jsonpath/PathUtil
Posted by Ramki Palle <ra...@gmail.com>.
When you execute the following query,
hive> select * from twitter limit 5;
Hive runs it in local mode and not use MapReduce.
For the query,
hive> select tweet_id from twitter limit 5;
I think you need to add JSON jars to overcome this error. You might have
added these in a previous session. If you want these jars available for all
sessions, insert the add jar statements to your $HOME/.hiverc file.
To bypass MapReduce
set hive.exec.mode.local.auto = true;
to suggest Hive to use local mode to execute the query. If it still uses
MR, try
set hive.fetch.task.conversion = more;.
-Ramki.
On Sun, Mar 10, 2013 at 12:19 AM, Sai Sai <sa...@yahoo.in> wrote:
> Just wondering if anyone has any suggestions:
>
> This executes successfully:
>
> hive> select * from twitter limit 5;
>
> This does not work:
>
> hive> select tweet_id from twitter limit 5; // I have given the exception
> info below:
>
> Here is the output of this:
>
> hive> select * from twitter limit 5;
> OK
>
> tweet_id created_at text user_id user_screen_name user_lang
> 122106088022745088 Fri Oct 07 00:28:54 +0000 2011 wkwkw -_- ayo saja
> mba RT @yullyunet: Sepupuuu, kita lanjalan yok.. Kita karokoe-an.. Ajak mas
> galih jg kalo dia mau.. "@Dindnf: doremifas 124735434 Dindnf en
> 122106088018558976 Fri Oct 07 00:28:54 +0000 2011 @egg486 특별히
> 준비했습니다! 252828803 CocaCola_Korea ko
> 122106088026939392 Fri Oct 07 00:28:54 +0000 2011 My offer of free
> gobbies for all if @amityaffliction play Blair snitch project still
> stands. 168590073 SarahYoungBlood en
> 122106088035328001 Fri Oct 07 00:28:54 +0000 2011 the girl nxt to me
> in the lib got her headphones in dancing and singing loud af like she the
> only one here haha 267296295 MONEYyDREAMS_ en
> 122106088005971968 Fri Oct 07 00:28:54 +0000 2011 @KUnYoong_B2UTY
> Bị lsao đấy 269182160 b2st_b2utyhp en
> Time taken: 0.154 seconds
>
> This does not work:
>
> hive> select tweet_id from twitter limit 5;
>
>
> Total MapReduce jobs = 1
> Launching Job 1 out of 1
> Number of reduce tasks is set to 0 since there's no reduce operator
> Starting Job = job_201303050432_0094, Tracking URL =
> http://ubuntu:50030/jobdetails.jsp?jobid=job_201303050432_0094
> Kill Command = /home/satish/work/hadoop-1.0.4/libexec/../bin/hadoop job
> -kill job_201303050432_0094
> Hadoop job information for Stage-1: number of mappers: 1; number of
> reducers: 0
> 2013-03-10 00:14:44,509 Stage-1 map = 0%, reduce = 0%
> 2013-03-10 00:15:14,613 Stage-1 map = 100%, reduce = 100%
> Ended Job = job_201303050432_0094 with errors
> Error during job, obtaining debugging information...
> Job Tracking URL:
> http://ubuntu:50030/jobdetails.jsp?jobid=job_201303050432_0094
> Examining task ID: task_201303050432_0094_m_000002 (and more) from job
> job_201303050432_0094
>
> Task with the most failures(4):
> -----
> Task ID:
> task_201303050432_0094_m_000000
>
> URL:
>
> http://ubuntu:50030/taskdetails.jsp?jobid=job_201303050432_0094&tipid=task_201303050432_0094_m_000000
> -----
> Diagnostic Messages for this Task:
> java.lang.RuntimeException: Error in configuring object
> at
> org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:93)
> at
> org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64)
> at
> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:432)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372)
> at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:416)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
> at org.apache.hadoop.mapred.Child.main(Child.java:249)
> Caused by: java.lang.reflect.InvocationTargetException
>
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at
> org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88)
> ... 9 more
> Caused by: java.lang.RuntimeException: Error in configuring object
> at
> org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:93)
> at
> org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64)
> at
> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
> at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:34)
> ... 14 more
> Caused by: java.lang.reflect.InvocationTargetException
>
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at
> org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88)
> ... 17 more
> Caused by: java.lang.RuntimeException: Map operator initialization failed
> at
> org.apache.hadoop.hive.ql.exec.ExecMapper.configure(ExecMapper.java:121)
> ... 22 more
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
> java.lang.ClassNotFoundException:
> org.apache.hadoop.hive.contrib.serde2.JsonSerde
> at
> org.apache.hadoop.hive.ql.exec.MapOperator.setChildren(MapOperator.java:420)
> at
> org.apache.hadoop.hive.ql.exec.ExecMapper.configure(ExecMapper.java:90)
> ... 22 more
> Caused by: java.lang.ClassNotFoundException:
> org.apache.hadoop.hive.contrib.serde2.JsonSerde
>
> at java.net.URLClassLoader$1.run(URLClassLoader.java:217)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:205)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:321)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:294)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:266)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:264)
> at
> org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:820)
> at
> org.apache.hadoop.hive.ql.exec.MapOperator.initObjectInspector(MapOperator.java:243)
> at
> org.apache.hadoop.hive.ql.exec.MapOperator.setChildren(MapOperator.java:380)
> ... 23 more
>
>
> FAILED: Execution Error, return code 2 from
> org.apache.hadoop.hive.ql.exec.MapRedTask
> MapReduce Jobs Launched:
> Job 0: Map: 1 HDFS Read: 0 HDFS Write: 0 FAIL
> Total MapReduce CPU Time Spent: 0 msec
>
> Thanks
> Sai
> ------------------------------
> *From:* Dean Wampler <de...@thinkbiganalytics.com>
> *To:* user@hive.apache.org; Sai Sai <sa...@yahoo.in>
> *Sent:* Friday, 8 March 2013 5:22 AM
> *Subject:* Re: java.lang.NoClassDefFoundError:
> com/jayway/jsonpath/PathUtil
>
> Unfortunately, you have to also add the json jars to Hive's class path
> before it starts, e.g.,
>
> env HADOOP_CLASSPATH=/path/to/lib/*.jar hive
>
> Use the appropriate path to your lib directory.
>
> On Fri, Mar 8, 2013 at 4:53 AM, Sai Sai <sa...@yahoo.in> wrote:
>
> I have added the jar files successfully like this:
>
>
> hive (testdb)> ADD JAR lib/hive-json-serde-0.3.jar;
> Added lib/hive-json-serde-0.3.jar to class path
> Added resource: lib/hive-json-serde-0.3.jar
>
>
> hive (testdb)> ADD JAR lib/json-path-0.5.4.jar;
> Added lib/json-path-0.5.4.jar to class path
> Added resource: lib/json-path-0.5.4.jar
>
>
> hive (testdb)> ADD JAR lib/json-smart-1.0.6.3.jar;
> Added lib/json-smart-1.0.6.3.jar to class path
> Added resource: lib/json-smart-1.0.6.3.jar
>
>
> After this i am getting this error:
>
>
> CREATE EXTERNAL TABLE IF NOT EXISTS twitter (tweet_id BIGINT,created_at
> STRING,text STRING,user_id BIGINT, user_screen_name STRING,user_lang
> STRING) ROW FORMAT SERDE "org.apache.hadoop.hive.contrib.serde2.JsonSerde"
> WITH SERDEPROPERTIES (
> "tweet_id"="$.id","created_at"="$.created_at","text"="$.text","user_id"="$.
> user.id","user_screen_name"="$.user.screen_name",
> "user_lang"="$.user.lang") LOCATION '/home/satish/data/twitter/input';
> java.lang.NoClassDefFoundError: com/jayway/jsonpath/PathUtil
> at org.apache.hadoop.hive.contrib.serde2.JsonSerde.initialize(Unknown
> Source)
> at
> org.apache.hadoop.hive.metastore.MetaStoreUtils.getDeserializer(MetaStoreUtils.java:207)
> at
> org.apache.hadoop.hive.ql.metadata.Table.getDeserializerFromMetaStore(Table.java:266)
> at
> org.apache.hadoop.hive.ql.metadata.Table.getDeserializer(Table.java:259)
> at org.apache.hadoop.hive.ql.metadata.Table.getCols(Table.java:585)
> at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:550)
> at
> org.apache.hadoop.hive.ql.exec.DDLTask.createTable(DDLTask.java:3698)
> at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:253)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:138)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:57)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1336)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1122)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:935)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:259)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:216)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:412)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:755)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:613)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
> Caused by: java.lang.ClassNotFoundException: com.jayway.jsonpath.PathUtil
> at java.net.URLClassLoader$1.run(URLClassLoader.java:217)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:205)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:321)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:266)
> ... 23 more
> FAILED: Execution Error, return code -101 from
> org.apache.hadoop.hive.ql.exec.DDLTask
>
>
> Any help would be really appreciated.
> Thanks
> Sai
>
>
>
>
> --
> *Dean Wampler, Ph.D.*
> thinkbiganalytics.com
> +1-312-339-1330
>
>
>
>
Re: java.lang.NoClassDefFoundError: com/jayway/jsonpath/PathUtil
Posted by Sai Sai <sa...@yahoo.in>.
Just wondering if anyone has any suggestions:
This executes successfully:
hive> select * from twitter limit 5;
This does not work:
hive> select tweet_id from twitter limit 5; // I have given the exception info below:
Here is the output of this:
hive> select * from twitter limit 5;
OK
tweet_id created_at text user_id user_screen_name user_lang
122106088022745088 Fri Oct 07 00:28:54 +0000 2011 wkwkw -_- ayo saja mba RT @yullyunet: Sepupuuu, kita lanjalan yok.. Kita karokoe-an.. Ajak mas galih jg kalo dia mau.. "@Dindnf: doremifas 124735434 Dindnf en
122106088018558976 Fri Oct 07 00:28:54 +0000 2011 @egg486 특별히 준비했습니다! 252828803 CocaCola_Korea ko
122106088026939392 Fri Oct 07 00:28:54 +0000 2011 My offer of free gobbies for all if @amityaffliction play Blair snitch project still stands. 168590073 SarahYoungBlood en
122106088035328001 Fri Oct 07 00:28:54 +0000 2011 the girl nxt to me in the lib got her headphones in dancing and singing loud af like she the only one here haha 267296295 MONEYyDREAMS_ en
122106088005971968 Fri Oct 07 00:28:54 +0000 2011 @KUnYoong_B2UTY Bị lsao đấy 269182160 b2st_b2utyhp en
Time taken: 0.154 seconds
This does not work:
hive> select tweet_id from twitter limit 5;
Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks is set to 0 since there's no reduce operator
Starting Job = job_201303050432_0094, Tracking URL = http://ubuntu:50030/jobdetails.jsp?jobid=job_201303050432_0094
Kill Command = /home/satish/work/hadoop-1.0.4/libexec/../bin/hadoop job -kill job_201303050432_0094
Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 0
2013-03-10 00:14:44,509 Stage-1 map = 0%, reduce = 0%
2013-03-10 00:15:14,613 Stage-1 map = 100%, reduce = 100%
Ended Job = job_201303050432_0094 with errors
Error during job, obtaining debugging information...
Job Tracking URL: http://ubuntu:50030/jobdetails.jsp?jobid=job_201303050432_0094
Examining task ID: task_201303050432_0094_m_000002 (and more) from job job_201303050432_0094
Task with the most failures(4):
-----
Task ID:
task_201303050432_0094_m_000000
URL:
http://ubuntu:50030/taskdetails.jsp?jobid=job_201303050432_0094&tipid=task_201303050432_0094_m_000000
-----
Diagnostic Messages for this Task:
java.lang.RuntimeException: Error in configuring object
at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:93)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:432)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:416)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121)
at org.apache.hadoop.mapred.Child.main(Child.java:249)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88)
... 9 more
Caused by: java.lang.RuntimeException: Error in configuring object
at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:93)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:34)
... 14 more
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88)
... 17 more
Caused by: java.lang.RuntimeException: Map operator initialization failed
at org.apache.hadoop.hive.ql.exec.ExecMapper.configure(ExecMapper.java:121)
... 22 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.ClassNotFoundException: org.apache.hadoop.hive.contrib.serde2.JsonSerde
at org.apache.hadoop.hive.ql.exec.MapOperator.setChildren(MapOperator.java:420)
at org.apache.hadoop.hive.ql.exec.ExecMapper.configure(ExecMapper.java:90)
... 22 more
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hive.contrib.serde2.JsonSerde
at java.net.URLClassLoader$1.run(URLClassLoader.java:217)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:205)
at java.lang.ClassLoader.loadClass(ClassLoader.java:321)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:294)
at java.lang.ClassLoader.loadClass(ClassLoader.java:266)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:264)
at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:820)
at org.apache.hadoop.hive.ql.exec.MapOperator.initObjectInspector(MapOperator.java:243)
at org.apache.hadoop.hive.ql.exec.MapOperator.setChildren(MapOperator.java:380)
... 23 more
FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.MapRedTask
MapReduce Jobs Launched:
Job 0: Map: 1 HDFS Read: 0 HDFS Write: 0 FAIL
Total MapReduce CPU Time Spent: 0 msec
Thanks
Sai
________________________________
From: Dean Wampler <de...@thinkbiganalytics.com>
To: user@hive.apache.org; Sai Sai <sa...@yahoo.in>
Sent: Friday, 8 March 2013 5:22 AM
Subject: Re: java.lang.NoClassDefFoundError: com/jayway/jsonpath/PathUtil
Unfortunately, you have to also add the json jars to Hive's class path before it starts, e.g.,
env HADOOP_CLASSPATH=/path/to/lib/*.jar hive
Use the appropriate path to your lib directory.
On Fri, Mar 8, 2013 at 4:53 AM, Sai Sai <sa...@yahoo.in> wrote:
I have added the jar files successfully like this:
>
>
>
>
>hive (testdb)> ADD JAR lib/hive-json-serde-0.3.jar;
> Added lib/hive-json-serde-0.3.jar to class path
> Added resource: lib/hive-json-serde-0.3.jar
>
>
>
>
>
>hive (testdb)> ADD JAR lib/json-path-0.5.4.jar;
> Added lib/json-path-0.5.4.jar to class path
> Added resource: lib/json-path-0.5.4.jar
>
>
>
>
>
>hive (testdb)> ADD JAR lib/json-smart-1.0.6.3.jar;
> Added lib/json-smart-1.0.6.3.jar to class path
> Added resource: lib/json-smart-1.0.6.3.jar
>
>
>
>
>After this i am getting this error:
>
>
>
>
>CREATE EXTERNAL TABLE IF NOT EXISTS twitter (tweet_id BIGINT,created_at STRING,text STRING,user_id BIGINT, user_screen_name STRING,user_lang STRING) ROW FORMAT SERDE "org.apache.hadoop.hive.contrib.serde2.JsonSerde" WITH SERDEPROPERTIES ( "tweet_id"="$.id","created_at"="$.created_at","text"="$.text","user_id"="$.user.id","user_screen_name"="$.user.screen_name", "user_lang"="$.user.lang") LOCATION '/home/satish/data/twitter/input';
>java.lang.NoClassDefFoundError: com/jayway/jsonpath/PathUtil
> at org.apache.hadoop.hive.contrib.serde2.JsonSerde.initialize(Unknown Source)
> at org.apache.hadoop.hive.metastore.MetaStoreUtils.getDeserializer(MetaStoreUtils.java:207)
> at org.apache.hadoop.hive.ql.metadata.Table.getDeserializerFromMetaStore(Table.java:266)
> at org.apache.hadoop.hive.ql.metadata.Table.getDeserializer(Table.java:259)
> at org.apache.hadoop.hive.ql.metadata.Table.getCols(Table.java:585)
> at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:550)
> at org.apache.hadoop.hive.ql.exec.DDLTask.createTable(DDLTask.java:3698)
> at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:253)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:138)
> at
org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:57)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1336)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1122)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:935)
> at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:259)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:216)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:412)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:755)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:613)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
>Caused by: java.lang.ClassNotFoundException: com.jayway.jsonpath.PathUtil
> at java.net.URLClassLoader$1.run(URLClassLoader.java:217)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:205)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:321)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:266)
> ... 23 more
>FAILED: Execution Error, return code -101 from org.apache.hadoop.hive.ql.exec.DDLTask
>
>
>
>
>
>Any help would be really appreciated.
>ThanksSai
>
--
Dean Wampler, Ph.D.
thinkbiganalytics.com
+1-312-339-1330
Re: java.lang.NoClassDefFoundError: com/jayway/jsonpath/PathUtil
Posted by Dean Wampler <de...@thinkbiganalytics.com>.
Unfortunately, you have to also add the json jars to Hive's class path
before it starts, e.g.,
env HADOOP_CLASSPATH=/path/to/lib/*.jar hive
Use the appropriate path to your lib directory.
On Fri, Mar 8, 2013 at 4:53 AM, Sai Sai <sa...@yahoo.in> wrote:
> I have added the jar files successfully like this:
>
>
> hive (testdb)> ADD JAR lib/hive-json-serde-0.3.jar;
> Added lib/hive-json-serde-0.3.jar to class path
> Added resource: lib/hive-json-serde-0.3.jar
>
>
> hive (testdb)> ADD JAR lib/json-path-0.5.4.jar;
> Added lib/json-path-0.5.4.jar to class path
> Added resource: lib/json-path-0.5.4.jar
>
>
> hive (testdb)> ADD JAR lib/json-smart-1.0.6.3.jar;
> Added lib/json-smart-1.0.6.3.jar to class path
> Added resource: lib/json-smart-1.0.6.3.jar
>
>
> After this i am getting this error:
>
>
> CREATE EXTERNAL TABLE IF NOT EXISTS twitter (tweet_id BIGINT,created_at
> STRING,text STRING,user_id BIGINT, user_screen_name STRING,user_lang
> STRING) ROW FORMAT SERDE "org.apache.hadoop.hive.contrib.serde2.JsonSerde"
> WITH SERDEPROPERTIES (
> "tweet_id"="$.id","created_at"="$.created_at","text"="$.text","user_id"="$.
> user.id","user_screen_name"="$.user.screen_name",
> "user_lang"="$.user.lang") LOCATION '/home/satish/data/twitter/input';
> java.lang.NoClassDefFoundError: com/jayway/jsonpath/PathUtil
> at org.apache.hadoop.hive.contrib.serde2.JsonSerde.initialize(Unknown
> Source)
> at
> org.apache.hadoop.hive.metastore.MetaStoreUtils.getDeserializer(MetaStoreUtils.java:207)
> at
> org.apache.hadoop.hive.ql.metadata.Table.getDeserializerFromMetaStore(Table.java:266)
> at
> org.apache.hadoop.hive.ql.metadata.Table.getDeserializer(Table.java:259)
> at org.apache.hadoop.hive.ql.metadata.Table.getCols(Table.java:585)
> at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:550)
> at
> org.apache.hadoop.hive.ql.exec.DDLTask.createTable(DDLTask.java:3698)
> at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:253)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:138)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:57)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1336)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1122)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:935)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:259)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:216)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:412)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:755)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:613)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:616)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
> Caused by: java.lang.ClassNotFoundException: com.jayway.jsonpath.PathUtil
> at java.net.URLClassLoader$1.run(URLClassLoader.java:217)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:205)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:321)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:266)
> ... 23 more
> FAILED: Execution Error, return code -101 from
> org.apache.hadoop.hive.ql.exec.DDLTask
>
>
> Any help would be really appreciated.
> Thanks
> Sai
>
--
*Dean Wampler, Ph.D.*
thinkbiganalytics.com
+1-312-339-1330
Re: java.lang.NoClassDefFoundError: com/jayway/jsonpath/PathUtil
Posted by Sai Sai <sa...@yahoo.in>.
I have added the jar files successfully like this:
hive (testdb)> ADD JAR lib/hive-json-serde-0.3.jar;
Added lib/hive-json-serde-0.3.jar to class path
Added resource: lib/hive-json-serde-0.3.jar
hive (testdb)> ADD JAR lib/json-path-0.5.4.jar;
Added lib/json-path-0.5.4.jar to class path
Added resource: lib/json-path-0.5.4.jar
hive (testdb)> ADD JAR lib/json-smart-1.0.6.3.jar;
Added lib/json-smart-1.0.6.3.jar to class path
Added resource: lib/json-smart-1.0.6.3.jar
After this i am getting this error:
CREATE EXTERNAL TABLE IF NOT EXISTS twitter (tweet_id BIGINT,created_at STRING,text STRING,user_id BIGINT, user_screen_name STRING,user_lang STRING) ROW FORMAT SERDE "org.apache.hadoop.hive.contrib.serde2.JsonSerde" WITH SERDEPROPERTIES ( "tweet_id"="$.id","created_at"="$.created_at","text"="$.text","user_id"="$.user.id","user_screen_name"="$.user.screen_name", "user_lang"="$.user.lang") LOCATION '/home/satish/data/twitter/input';
java.lang.NoClassDefFoundError: com/jayway/jsonpath/PathUtil
at org.apache.hadoop.hive.contrib.serde2.JsonSerde.initialize(Unknown Source)
at org.apache.hadoop.hive.metastore.MetaStoreUtils.getDeserializer(MetaStoreUtils.java:207)
at org.apache.hadoop.hive.ql.metadata.Table.getDeserializerFromMetaStore(Table.java:266)
at org.apache.hadoop.hive.ql.metadata.Table.getDeserializer(Table.java:259)
at org.apache.hadoop.hive.ql.metadata.Table.getCols(Table.java:585)
at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:550)
at org.apache.hadoop.hive.ql.exec.DDLTask.createTable(DDLTask.java:3698)
at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:253)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:138)
at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:57)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1336)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1122)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:935)
at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:259)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:216)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:412)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:755)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:613)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
Caused by: java.lang.ClassNotFoundException: com.jayway.jsonpath.PathUtil
at java.net.URLClassLoader$1.run(URLClassLoader.java:217)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:205)
at java.lang.ClassLoader.loadClass(ClassLoader.java:321)
at java.lang.ClassLoader.loadClass(ClassLoader.java:266)
... 23 more
FAILED: Execution Error, return code -101 from org.apache.hadoop.hive.ql.exec.DDLTask
Any help would be really appreciated.
Thanks
Sai
Re: Find current db we r using in Hive
Posted by Nagarjuna Kanamarlapudi <na...@gmail.com>.
Another alternative way, just say
Desc extended <table_name>
There we can see the db name of the table
—
Sent from iPhone
On Sun, Mar 10, 2013 at 1:02 AM, Mark Grover <gr...@gmail.com>
wrote:
> Created https://issues.apache.org/jira/browse/HIVE-4144
> On Fri, Mar 8, 2013 at 5:25 AM, Dean Wampler
> <de...@thinkbiganalytics.com> wrote:
>> It's odd that there is no such command. The trick Ramki mentioned is the
>> only one I know of. Two points about it, though:
>>
>> 1. It only works on Hive v0.8+.
>> 2. I've seen a few cases where the prompt did NOT change when first used,
>> but starting working a little later! I have no idea why and of course, it
>> happened while teaching a class where I'm supposed to be the expert ;)
>>
>> dean
>>
>>
>> On Fri, Mar 8, 2013 at 12:36 AM, Ramki Palle <ra...@gmail.com> wrote:
>>>
>>> Sai,
>>>
>>> I do not think there is any command to show the current db in Hive. One
>>> alternative for you is to set a property so that the current database is
>>> shown as part of the prompt:
>>>
>>> set hive.cli.print.current.db=true;
>>>
>>> This one shows your current db as part of your hive prompt.
>>>
>>> Regards,
>>> Ramki.
>>>
>>>
>>> On Fri, Mar 8, 2013 at 11:13 AM, Sai Sai <sa...@yahoo.in> wrote:
>>>>
>>>> Just wondering if there is any command in Hive which will show us the
>>>> current db we r using similar to pwd in Unix.
>>>> Thanks
>>>> Sai
>>>>
>>>
>>
>>
>>
>> --
>> Dean Wampler, Ph.D.
>> thinkbiganalytics.com
>> +1-312-339-1330
>>
Re: Find current db we r using in Hive
Posted by Mark Grover <gr...@gmail.com>.
Created https://issues.apache.org/jira/browse/HIVE-4144
On Fri, Mar 8, 2013 at 5:25 AM, Dean Wampler
<de...@thinkbiganalytics.com> wrote:
> It's odd that there is no such command. The trick Ramki mentioned is the
> only one I know of. Two points about it, though:
>
> 1. It only works on Hive v0.8+.
> 2. I've seen a few cases where the prompt did NOT change when first used,
> but starting working a little later! I have no idea why and of course, it
> happened while teaching a class where I'm supposed to be the expert ;)
>
> dean
>
>
> On Fri, Mar 8, 2013 at 12:36 AM, Ramki Palle <ra...@gmail.com> wrote:
>>
>> Sai,
>>
>> I do not think there is any command to show the current db in Hive. One
>> alternative for you is to set a property so that the current database is
>> shown as part of the prompt:
>>
>> set hive.cli.print.current.db=true;
>>
>> This one shows your current db as part of your hive prompt.
>>
>> Regards,
>> Ramki.
>>
>>
>> On Fri, Mar 8, 2013 at 11:13 AM, Sai Sai <sa...@yahoo.in> wrote:
>>>
>>> Just wondering if there is any command in Hive which will show us the
>>> current db we r using similar to pwd in Unix.
>>> Thanks
>>> Sai
>>>
>>
>
>
>
> --
> Dean Wampler, Ph.D.
> thinkbiganalytics.com
> +1-312-339-1330
>
Re: Find current db we r using in Hive
Posted by Dean Wampler <de...@thinkbiganalytics.com>.
It's odd that there is no such command. The trick Ramki mentioned is the
only one I know of. Two points about it, though:
1. It only works on Hive v0.8+.
2. I've seen a few cases where the prompt did NOT change when first used,
but starting working a little later! I have no idea why and of course, it
happened while teaching a class where I'm supposed to be the expert ;)
dean
On Fri, Mar 8, 2013 at 12:36 AM, Ramki Palle <ra...@gmail.com> wrote:
> Sai,
>
> I do not think there is any command to show the current db in Hive. One
> alternative for you is to set a property so that the current database is
> shown as part of the prompt:
>
> set hive.cli.print.current.db=true;
>
> This one shows your current db as part of your hive prompt.
>
> Regards,
> Ramki.
>
>
> On Fri, Mar 8, 2013 at 11:13 AM, Sai Sai <sa...@yahoo.in> wrote:
>
>> Just wondering if there is any command in Hive which will show us the
>> current db we r using similar to pwd in Unix.
>> Thanks
>> Sai
>>
>>
>
--
*Dean Wampler, Ph.D.*
thinkbiganalytics.com
+1-312-339-1330
Re: Find current db we r using in Hive
Posted by Ramki Palle <ra...@gmail.com>.
Sai,
I do not think there is any command to show the current db in Hive. One
alternative for you is to set a property so that the current database is
shown as part of the prompt:
set hive.cli.print.current.db=true;
This one shows your current db as part of your hive prompt.
Regards,
Ramki.
On Fri, Mar 8, 2013 at 11:13 AM, Sai Sai <sa...@yahoo.in> wrote:
> Just wondering if there is any command in Hive which will show us the
> current db we r using similar to pwd in Unix.
> Thanks
> Sai
>
>
Re: Find current db we r using in Hive
Posted by Sai Sai <sa...@yahoo.in>.
Just wondering if there is any command in Hive which will show us the current db we r using similar to pwd in Unix.
Thanks
Sai
Re: Partition not displaying in the browser
Posted by Sai Sai <sa...@yahoo.in>.
Many Thanks for your help Venkatesh.
I have verified the partition exists and also the data displays successfully in the partition when i execute the select in the console.
But it does not appear in the web browser.
I have verified multiple time the path i have given and is given below:
Here is the first partition i have created and viewing successfully in both console & web browser:
ALTER TABLE dividends ADD PARTITION(exchange = 'NASDAQ', symbol = 'AAPL') LOCATION '/home/satish/data/dividends/input/plain-text/NASDAQ/AAPL';
LOAD DATA LOCAL INPATH '/home/satish/data/dividends/input/plain-text/NASDAQ/AAPL/dividends.csv' INTO TABLE dividends Partition(exchange='NASDAQ',symbol='AAPL');
Here is the one i can view only in the console but not in the browser:
ALTER TABLE dividends ADD PARTITION(exchange = 'NASDAQ', symbol = 'INTC') LOCATION '/home/satish/data/dividends/input/plain-text/NASDAQ/INTC';
LOAD DATA LOCAL INPATH '/home/satish/data/dividends/input/plain-text/NASDAQ/INTC/dividends.csv' INTO TABLE dividends Partition(exchange='NASDAQ',symbol='INTC');
When i run the command:
select * from dividends where exchange='NASDAQ' and symbol='INTC';
I succesfully see the data.
I am wondering if it is possible to bounce/restart the server with any command or
is it possible to look into the hive metadata directly using a command.
Any help is appreciated.
Thanks
Sai
________________________________
From: Venkatesh Kavuluri <vk...@outlook.com>
To: "user@hive.apache.org" <us...@hive.apache.org>
Sent: Thursday, 7 March 2013 1:44 PM
Subject: RE: Partition not displaying in the browser
The partitions info you see on 'show partitions' is fetched from Hive metadata tables. The reason you are not seeing the path you are expecting might be either
1) the path got deleted after the data load (do a simple select and verify you see some data) or
2) you have loaded the data from some other path to this partition
-
Venkatesh
________________________________
Date: Fri, 8 Mar 2013 00:54:17 +0800
From: saigraph@yahoo.in
Subject: Re: Partition not displaying in the browser
To: user@hive.apache.org
I get this ouput for:
hive> show partitions dividends;
OK
partition
exchange=NASDAQ/symbol=AAPL
exchange=NASDAQ/symbol=INTC
Time taken: 0.133 seconds
But when i navigate to my browser folder the partition INTC is not displayed even after refreshing it a bunch of times, any suggestions will be appreciated:
*************************
Contents of directory /home/satish/data/dividends/input/plain-text/NASDAQ
________________________________
Goto :
________________________________
Go to parent directory
Name
Type
Size
Replication
Block Size
Modification Time
Permission
Owner
Group
AAPL
dir
2013-03-07 08:46
rwxr-xr-x
satish
supergroup
Go back to DFS home
________________________________
Local logs
Any suggestions will be appreciated.
Thanks
Sai
RE: Partition not displaying in the browser
Posted by Venkatesh Kavuluri <vk...@outlook.com>.
The partitions info you see on 'show partitions' is fetched from Hive metadata tables. The reason you are not seeing the path you are expecting might be either 1) the path got deleted after the data load (do a simple select and verify you see some data) or2) you have loaded the data from some other path to this partition
-Venkatesh
Date: Fri, 8 Mar 2013 00:54:17 +0800
From: saigraph@yahoo.in
Subject: Re: Partition not displaying in the browser
To: user@hive.apache.org
I get this ouput for:
hive> show partitions dividends;
OK
partition
exchange=NASDAQ/symbol=AAPL
exchange=NASDAQ/symbol=INTC
Time taken: 0.133 seconds
But when i navigate to my browser folder the partition INTC is not displayed even after refreshing it a bunch of times, any suggestions will be appreciated:
*************************Contents of directory /home/satish/data/dividends/input/plain-text/NASDAQGoto : Go to parent directory
Name
Type
Size
Replication
Block Size
Modification Time
Permission
Owner
Group
AAPL
dir
2013-03-07 08:46
rwxr-xr-x
satish
supergroup
Go back to DFS home
Local logs
Any suggestions will be appreciated.
Thanks
Sai
Re: Partition not displaying in the browser
Posted by Sai Sai <sa...@yahoo.in>.
I get this ouput for:
hive> show partitions dividends;
OK
partition
exchange=NASDAQ/symbol=AAPL
exchange=NASDAQ/symbol=INTC
Time taken: 0.133 seconds
But when i navigate to my browser folder the partition INTC is not displayed even after refreshing it a bunch of times, any suggestions will be appreciated:
*************************
Contents of directory /home/satish/data/dividends/input/plain-text/NASDAQ
________________________________
Goto :
________________________________
Go to parent directory
Name
Type
Size
Replication
Block Size
Modification Time
Permission
Owner
Group
AAPL
dir
2013-03-07 08:46
rwxr-xr-x
satish
supergroup
Go back to DFS home
________________________________
Local logs
Any suggestions will be appreciated.
Thanks
Sai
Re: Accessing sub column in hive
Posted by Sai Sai <sa...@yahoo.in>.
I have a table created like this successfully:
CREATE TABLE IF NOT EXISTS employees (name STRING,salary FLOAT,subordinates ARRAY<STRING>,deductions MAP<STRING,FLOAT>,address STRUCT<street:STRING, city:STRING, state:STRING, zip:INT, country:STRING>)
I would like to access/display country column from my address struct.
I have tried this:
select address["country"] from employees;
I get an error.
Please help.
Thanks
Sai
Re: syslog from hive
Posted by Nitin Pawar <ni...@gmail.com>.
for syslog lmgtfy ... http://bit.ly/YeyTwL
no you can not access syslog from hive directly
for that you need to create an external table pointing to log directory and
make sure your log format fits into your table definition
On Thu, Mar 7, 2013 at 4:57 PM, Sai Sai <sa...@yahoo.in> wrote:
> Just wondering what is syslog and if we can access it from hive.
> Any example will help.
> Thanks
> Sai
>
--
Nitin Pawar
Re: syslog from hive
Posted by Sai Sai <sa...@yahoo.in>.
Just wondering what is syslog and if we can access it from hive.
Any example will help.
Thanks
Sai
Re: Where is the location of hive queries
Posted by Sai Sai <sa...@yahoo.in>.
After we run a query in hive shell as:
Select * from myTable;
Are these results getting saved to any file apart from the console/terminal display.
If so where is the location of the results.
Thanks
Sai
Re: Read map value from a table
Posted by Sai Sai <sa...@yahoo.in>.
Here is my data in a file which i have successfully loaded into a table test and successfully get the data for:
Select * from test;
Name ph category
Name1 ph1 {"type":1000,"color":200,"shape":610}
Name2 ph2 {"type":2000,"color":200,"shape":150}
Name3 ph3 {"type":3000,"color":700,"shape":167}
But when i execute this query:
select category["type"] from test;
I get null values;
Please help.
Thanks
Sai
Re: Location of external table in hdfs
Posted by Sai Sai <sa...@yahoo.in>.
Thanks I figured this is in tmp/states
Thanks for your attention.
________________________________
From: Sai Sai <sa...@yahoo.in>
To: "user@hive.apache.org" <us...@hive.apache.org>
Sent: Tuesday, 5 March 2013 8:56 AM
Subject: Re: Location of external table in hdfs
I have created an external table like below and wondering where (folder) in hdfs i can find this:
CREATE EXTERNAL TABLE states(abbreviation string, full_name string) ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t' LOCATION '/tmp/states' ;
Any help is really appreciated.
Thanks
Sai
Re: Location of external table in hdfs
Posted by bharath vissapragada <bh...@gmail.com>.
When you create an external table, original data ('/tmp/states' in
this case) is NOT copied to the warehouse folder (or infact any other
folder for that matter). So you can find it in '/tmp/states' ifself.
On Tue, Mar 5, 2013 at 10:26 PM, Sai Sai <sa...@yahoo.in> wrote:
> I have created an external table like below and wondering where (folder) in
> hdfs i can find this:
>
> CREATE EXTERNAL TABLE states(abbreviation string, full_name string) ROW
> FORMAT DELIMITED FIELDS TERMINATED BY '\t' LOCATION '/tmp/states' ;
>
> Any help is really appreciated.
> Thanks
> Sai
Re: Location of external table in hdfs
Posted by Sai Sai <sa...@yahoo.in>.
I have created an external table like below and wondering where (folder) in hdfs i can find this:
CREATE EXTERNAL TABLE states(abbreviation string, full_name string) ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t' LOCATION '/tmp/states' ;
Any help is really appreciated.
Thanks
Sai
Re: SemanticException Line 1:17 issue
Posted by Nitin Pawar <ni...@gmail.com>.
it exists but where? on your hdfs or local linux filesystem ? so if you
are checking the file with ls -l /tmp/ then add word local
ls can you provide output of $HADOOP_HOME/bin/hadoop dfs -ls /tmp/
LOAD DATA [LOCAL] INPATH 'filepath' [OVERWRITE] INTO TABLE tablename
If the keyword LOCAL is specified, then:
- the load command will look for *filepath* in the local file
system. If a relative path is specified - it will be interpreted
relative to the current directory of the user
On Tue, Mar 5, 2013 at 5:48 PM, Sai Sai <sa...@yahoo.in> wrote:
> Yes Nitin it exists... but still getting the same issue.
>
> ------------------------------
> *From:* Nitin Pawar <ni...@gmail.com>
> *To:* user@hive.apache.org; Sai Sai <sa...@yahoo.in>
> *Sent:* Tuesday, 5 March 2013 4:14 AM
> *Subject:* Re: SemanticException Line 1:17 issue
>
> this file /tmp/o_small.tsv is on your local filesystem or hdfs?
>
>
> On Tue, Mar 5, 2013 at 5:39 PM, Sai Sai <sa...@yahoo.in> wrote:
>
> Hello
>
> I have been stuck on this issue for quite some time and was wondering if
> anyone sees any problem with this that i am not seeing:
>
> I have verified the file exists here and have also manually pasted the
> file into the tmp folder but still running into the same issue.
>
> I am also wondering which folder this maps to in my local drive:
> hdfs://ubuntu:9000/
>
> ***********************************
>
> hive> LOAD DATA INPATH '/tmp/o_small.tsv' OVERWRITE INTO TABLE odata ;
> FAILED: SemanticException Line 1:17 Invalid path ''/tmp/o_small.tsv'': No
> files matching path hdfs://ubuntu:9000/tmp/o_small.tsv
>
> ***********************************
> I have verified the file exists here and have also manually pasted the
> file here but still running into the same issue.
> Please let me know if u have any suggestions will be really appreciated.
> Thanks
> Sai
>
>
>
>
> --
> Nitin Pawar
>
>
>
--
Nitin Pawar
Re: SemanticException Line 1:17 issue
Posted by Sai Sai <sa...@yahoo.in>.
Yes Nitin it exists... but still getting the same issue.
________________________________
From: Nitin Pawar <ni...@gmail.com>
To: user@hive.apache.org; Sai Sai <sa...@yahoo.in>
Sent: Tuesday, 5 March 2013 4:14 AM
Subject: Re: SemanticException Line 1:17 issue
this file /tmp/o_small.tsv is on your local filesystem or hdfs?
On Tue, Mar 5, 2013 at 5:39 PM, Sai Sai <sa...@yahoo.in> wrote:
Hello
>
>
>I have been stuck on this issue for quite some time and was wondering if anyone sees any problem with this that i am not seeing:
>
>
>I have verified the file exists here and have also manually pasted the file into the tmp folder but still running into the same issue.
>
>
>I am also wondering which folder this maps to in my local drive:
>hdfs://ubuntu:9000/
>
>
>***********************************
>
>
>hive> LOAD DATA INPATH '/tmp/o_small.tsv' OVERWRITE INTO TABLE odata ;
>FAILED: SemanticException Line 1:17 Invalid path ''/tmp/o_small.tsv'': No files matching path hdfs://ubuntu:9000/tmp/o_small.tsv
>
>
>***********************************
>I have verified the file exists here and have also manually pasted the file here but still running into the same issue.
>Please let me know if u have any suggestions will be really appreciated.
>ThanksSai
>
--
Nitin Pawar
Re: SemanticException Line 1:17 issue
Posted by Nitin Pawar <ni...@gmail.com>.
this file /tmp/o_small.tsv is on your local filesystem or hdfs?
On Tue, Mar 5, 2013 at 5:39 PM, Sai Sai <sa...@yahoo.in> wrote:
> Hello
>
> I have been stuck on this issue for quite some time and was wondering if
> anyone sees any problem with this that i am not seeing:
>
> I have verified the file exists here and have also manually pasted the
> file into the tmp folder but still running into the same issue.
>
> I am also wondering which folder this maps to in my local drive:
> hdfs://ubuntu:9000/
>
> ***********************************
>
> hive> LOAD DATA INPATH '/tmp/o_small.tsv' OVERWRITE INTO TABLE odata ;
> FAILED: SemanticException Line 1:17 Invalid path ''/tmp/o_small.tsv'': No
> files matching path hdfs://ubuntu:9000/tmp/o_small.tsv
>
> ***********************************
> I have verified the file exists here and have also manually pasted the
> file here but still running into the same issue.
> Please let me know if u have any suggestions will be really appreciated.
> Thanks
> Sai
>
--
Nitin Pawar
Re: SemanticException Line 1:17 issue
Posted by Sai Sai <sa...@yahoo.in>.
Hello
I have been stuck on this issue for quite some time and was wondering if anyone sees any problem with this that i am not seeing:
I have verified the file exists here and have also manually pasted the file into the tmp folder but still running into the same issue.
I am also wondering which folder this maps to in my local drive:
hdfs://ubuntu:9000/
***********************************
hive> LOAD DATA INPATH '/tmp/o_small.tsv' OVERWRITE INTO TABLE odata ;
FAILED: SemanticException Line 1:17 Invalid path ''/tmp/o_small.tsv'': No files matching path hdfs://ubuntu:9000/tmp/o_small.tsv
***********************************
I have verified the file exists here and have also manually pasted the file here but still running into the same issue.
Please let me know if u have any suggestions will be really appreciated.
Thanks
Sai
Re: show tables in bin does not display the tables
Posted by Sai Sai <sa...@yahoo.in>.
Hello
I have noticed when i execute the following command from hive shell in diff folders it behaves in diff ways and was wondering if this is right:
show tables;
from the bin folder under my hive install folder it just shows tab_name:
****************************
myUser@ubuntu:~/work/hive-0.10.0-bin/bin$ ./hive
hive> show tables;
OK
tab_name
Time taken: 5.268 seconds
****************************
But when i excecute the same command from my install folder:
****************************
myUser@ubuntu:~/work/hive-0.10.0-bin/bin$ cd ..
hive> show tables;
OK
tab_name
employees
sample_pages
Time taken: 13.547 seconds
****************************
Please let me know.
Thanks
Sai
Re: hive commands from a file
Posted by Sai Sai <sa...@yahoo.in>.
Thanks Krishna/Nitin.
________________________________
From: Nitin Pawar <ni...@gmail.com>
To: user@hive.apache.org
Sent: Monday, 4 March 2013 2:28 AM
Subject: Re: hive commands from a file
Try hive -f filename
On Mar 4, 2013 3:55 PM, "Sai Sai" <sa...@yahoo.in> wrote:
Just wondering if it is possible to run a bunch of hive commands from a file rather than one a time.
>For ex:
>1. Create external...
>2. Load ...
>3. Select * from ...
>4....
>
>
>Thanks
>Sai
>
Re: hive commands from a file
Posted by Nitin Pawar <ni...@gmail.com>.
Try hive -f filename
On Mar 4, 2013 3:55 PM, "Sai Sai" <sa...@yahoo.in> wrote:
> Just wondering if it is possible to run a bunch of hive commands from a
> file rather than one a time.
> For ex:
> 1. Create external...
> 2. Load ...
> 3. Select * from ...
> 4....
>
> Thanks
> Sai
>