You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hive.apache.org by Sreenath <sr...@gmail.com> on 2014/02/14 11:43:01 UTC
INSERT OVERWRITE LOCAL DIRECTORY
i have two queries
1. select * from tab1 limit 3;
this returns the 3 rows quickly withoput launcing any map reduce jobs;
2. INSERT OVERWRITE LOCAL DIRECTORY "/tmp/query1/" select * from tab1 limit
3;
but the same above query to write to a local directory launches a map
reduce job and scans through all the rows launching map-reduce jobs
why the differenceinexecution of both queries?
--
Sreenath S Kamath
Bangalore
Ph No:+91-9590989106
Re: INSERT OVERWRITE LOCAL DIRECTORY
Posted by Sreenath <sr...@gmail.com>.
sure that will do what is required thanks nitin
On Fri, Feb 14, 2014 at 4:38 PM, Nitin Pawar <ni...@gmail.com>wrote:
> hive -e " select * from table limit 3" > localfile
>
> will above do for what you want to do?
>
>
> On Fri, Feb 14, 2014 at 4:36 PM, Sreenath <sr...@gmail.com> wrote:
>
>> meaning i want the output in a local directory without it scanning
>> through the whole table
>>
>>
>> On Fri, Feb 14, 2014 at 4:27 PM, Nitin Pawar <ni...@gmail.com>wrote:
>>
>>> what do you mean workaround for this ? its not a problem
>>>
>>>
>>> On Fri, Feb 14, 2014 at 4:26 PM, Sreenath <sr...@gmail.com>wrote:
>>>
>>>> hi all,
>>>> really appreciate the responses do you have any means as a work around
>>>> for this ?
>>>>
>>>>
>>>> On Fri, Feb 14, 2014 at 4:22 PM, shashwat shriparv <
>>>> dwivedishashwat@gmail.com> wrote:
>>>>
>>>>> Because when you do select * the query just has to read and return the
>>>>> resultl but when you give query under query suffle sort and comparison
>>>>> happens thats why mapreduce...
>>>>>
>>>>>
>>>>> *Warm Regards_**∞_*
>>>>> * Shashwat Shriparv*
>>>>> [image: http://www.linkedin.com/pub/shashwat-shriparv/19/214/2a9]<http://www.linkedin.com/pub/shashwat-shriparv/19/214/2a9>[image:
>>>>> https://twitter.com/shriparv] <https://twitter.com/shriparv>[image:
>>>>> https://www.facebook.com/shriparv] <https://www.facebook.com/shriparv>[image:
>>>>> http://google.com/+ShashwatShriparv]<http://google.com/+ShashwatShriparv>[image:
>>>>> http://www.youtube.com/user/sShriparv/videos]<http://www.youtube.com/user/sShriparv/videos>[image:
>>>>> http://profile.yahoo.com/SWXSTW3DVSDTF2HHSRM47AV6DI/]<sh...@yahoo.com>
>>>>>
>>>>>
>>>>>
>>>>> On Fri, Feb 14, 2014 at 4:15 PM, Nitin Pawar <ni...@gmail.com>wrote:
>>>>>
>>>>>> when you are writing a file, it will need an application to create
>>>>>> and write the file.
>>>>>> Thats why the mapreduce job to create the file.
>>>>>>
>>>>>> Where as for other way when its displayed on console, its just hdfs
>>>>>> cat operation
>>>>>>
>>>>>>
>>>>>> On Fri, Feb 14, 2014 at 4:13 PM, Sreenath <sr...@gmail.com>wrote:
>>>>>>
>>>>>>> i have two queries
>>>>>>> 1. select * from tab1 limit 3;
>>>>>>> this returns the 3 rows quickly withoput launcing any map reduce
>>>>>>> jobs;
>>>>>>>
>>>>>>> 2. INSERT OVERWRITE LOCAL DIRECTORY "/tmp/query1/" select * from
>>>>>>> tab1 limit 3;
>>>>>>> but the same above query to write to a local directory launches a
>>>>>>> map reduce job and scans through all the rows launching map-reduce jobs
>>>>>>>
>>>>>>> why the differenceinexecution of both queries?
>>>>>>>
>>>>>>> --
>>>>>>> Sreenath S Kamath
>>>>>>> Bangalore
>>>>>>> Ph No:+91-9590989106
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Nitin Pawar
>>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Sreenath S Kamath
>>>> Bangalore
>>>> Ph No:+91-9590989106
>>>>
>>>
>>>
>>>
>>> --
>>> Nitin Pawar
>>>
>>
>>
>>
>> --
>> Sreenath S Kamath
>> Bangalore
>> Ph No:+91-9590989106
>>
>
>
>
> --
> Nitin Pawar
>
--
Sreenath S Kamath
Bangalore
Ph No:+91-9590989106
Re: INSERT OVERWRITE LOCAL DIRECTORY
Posted by Nitin Pawar <ni...@gmail.com>.
hive -e " select * from table limit 3" > localfile
will above do for what you want to do?
On Fri, Feb 14, 2014 at 4:36 PM, Sreenath <sr...@gmail.com> wrote:
> meaning i want the output in a local directory without it scanning through
> the whole table
>
>
> On Fri, Feb 14, 2014 at 4:27 PM, Nitin Pawar <ni...@gmail.com>wrote:
>
>> what do you mean workaround for this ? its not a problem
>>
>>
>> On Fri, Feb 14, 2014 at 4:26 PM, Sreenath <sr...@gmail.com>wrote:
>>
>>> hi all,
>>> really appreciate the responses do you have any means as a work around
>>> for this ?
>>>
>>>
>>> On Fri, Feb 14, 2014 at 4:22 PM, shashwat shriparv <
>>> dwivedishashwat@gmail.com> wrote:
>>>
>>>> Because when you do select * the query just has to read and return the
>>>> resultl but when you give query under query suffle sort and comparison
>>>> happens thats why mapreduce...
>>>>
>>>>
>>>> *Warm Regards_**∞_*
>>>> * Shashwat Shriparv*
>>>> [image: http://www.linkedin.com/pub/shashwat-shriparv/19/214/2a9]<http://www.linkedin.com/pub/shashwat-shriparv/19/214/2a9>[image:
>>>> https://twitter.com/shriparv] <https://twitter.com/shriparv>[image:
>>>> https://www.facebook.com/shriparv] <https://www.facebook.com/shriparv>[image:
>>>> http://google.com/+ShashwatShriparv]<http://google.com/+ShashwatShriparv>[image:
>>>> http://www.youtube.com/user/sShriparv/videos]<http://www.youtube.com/user/sShriparv/videos>[image:
>>>> http://profile.yahoo.com/SWXSTW3DVSDTF2HHSRM47AV6DI/]<sh...@yahoo.com>
>>>>
>>>>
>>>>
>>>> On Fri, Feb 14, 2014 at 4:15 PM, Nitin Pawar <ni...@gmail.com>wrote:
>>>>
>>>>> when you are writing a file, it will need an application to create and
>>>>> write the file.
>>>>> Thats why the mapreduce job to create the file.
>>>>>
>>>>> Where as for other way when its displayed on console, its just hdfs
>>>>> cat operation
>>>>>
>>>>>
>>>>> On Fri, Feb 14, 2014 at 4:13 PM, Sreenath <sr...@gmail.com>wrote:
>>>>>
>>>>>> i have two queries
>>>>>> 1. select * from tab1 limit 3;
>>>>>> this returns the 3 rows quickly withoput launcing any map reduce jobs;
>>>>>>
>>>>>> 2. INSERT OVERWRITE LOCAL DIRECTORY "/tmp/query1/" select * from tab1
>>>>>> limit 3;
>>>>>> but the same above query to write to a local directory launches a map
>>>>>> reduce job and scans through all the rows launching map-reduce jobs
>>>>>>
>>>>>> why the differenceinexecution of both queries?
>>>>>>
>>>>>> --
>>>>>> Sreenath S Kamath
>>>>>> Bangalore
>>>>>> Ph No:+91-9590989106
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Nitin Pawar
>>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> Sreenath S Kamath
>>> Bangalore
>>> Ph No:+91-9590989106
>>>
>>
>>
>>
>> --
>> Nitin Pawar
>>
>
>
>
> --
> Sreenath S Kamath
> Bangalore
> Ph No:+91-9590989106
>
--
Nitin Pawar
Re: INSERT OVERWRITE LOCAL DIRECTORY
Posted by Sreenath <sr...@gmail.com>.
meaning i want the output in a local directory without it scanning through
the whole table
On Fri, Feb 14, 2014 at 4:27 PM, Nitin Pawar <ni...@gmail.com>wrote:
> what do you mean workaround for this ? its not a problem
>
>
> On Fri, Feb 14, 2014 at 4:26 PM, Sreenath <sr...@gmail.com> wrote:
>
>> hi all,
>> really appreciate the responses do you have any means as a work around
>> for this ?
>>
>>
>> On Fri, Feb 14, 2014 at 4:22 PM, shashwat shriparv <
>> dwivedishashwat@gmail.com> wrote:
>>
>>> Because when you do select * the query just has to read and return the
>>> resultl but when you give query under query suffle sort and comparison
>>> happens thats why mapreduce...
>>>
>>>
>>> *Warm Regards_**∞_*
>>> * Shashwat Shriparv*
>>> [image: http://www.linkedin.com/pub/shashwat-shriparv/19/214/2a9]<http://www.linkedin.com/pub/shashwat-shriparv/19/214/2a9>[image:
>>> https://twitter.com/shriparv] <https://twitter.com/shriparv>[image:
>>> https://www.facebook.com/shriparv] <https://www.facebook.com/shriparv>[image:
>>> http://google.com/+ShashwatShriparv]<http://google.com/+ShashwatShriparv>[image:
>>> http://www.youtube.com/user/sShriparv/videos]<http://www.youtube.com/user/sShriparv/videos>[image:
>>> http://profile.yahoo.com/SWXSTW3DVSDTF2HHSRM47AV6DI/]<sh...@yahoo.com>
>>>
>>>
>>>
>>> On Fri, Feb 14, 2014 at 4:15 PM, Nitin Pawar <ni...@gmail.com>wrote:
>>>
>>>> when you are writing a file, it will need an application to create and
>>>> write the file.
>>>> Thats why the mapreduce job to create the file.
>>>>
>>>> Where as for other way when its displayed on console, its just hdfs cat
>>>> operation
>>>>
>>>>
>>>> On Fri, Feb 14, 2014 at 4:13 PM, Sreenath <sr...@gmail.com>wrote:
>>>>
>>>>> i have two queries
>>>>> 1. select * from tab1 limit 3;
>>>>> this returns the 3 rows quickly withoput launcing any map reduce jobs;
>>>>>
>>>>> 2. INSERT OVERWRITE LOCAL DIRECTORY "/tmp/query1/" select * from tab1
>>>>> limit 3;
>>>>> but the same above query to write to a local directory launches a map
>>>>> reduce job and scans through all the rows launching map-reduce jobs
>>>>>
>>>>> why the differenceinexecution of both queries?
>>>>>
>>>>> --
>>>>> Sreenath S Kamath
>>>>> Bangalore
>>>>> Ph No:+91-9590989106
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Nitin Pawar
>>>>
>>>
>>>
>>
>>
>> --
>> Sreenath S Kamath
>> Bangalore
>> Ph No:+91-9590989106
>>
>
>
>
> --
> Nitin Pawar
>
--
Sreenath S Kamath
Bangalore
Ph No:+91-9590989106
Re: INSERT OVERWRITE LOCAL DIRECTORY
Posted by Nitin Pawar <ni...@gmail.com>.
what do you mean workaround for this ? its not a problem
On Fri, Feb 14, 2014 at 4:26 PM, Sreenath <sr...@gmail.com> wrote:
> hi all,
> really appreciate the responses do you have any means as a work around for
> this ?
>
>
> On Fri, Feb 14, 2014 at 4:22 PM, shashwat shriparv <
> dwivedishashwat@gmail.com> wrote:
>
>> Because when you do select * the query just has to read and return the
>> resultl but when you give query under query suffle sort and comparison
>> happens thats why mapreduce...
>>
>>
>> *Warm Regards_**∞_*
>> * Shashwat Shriparv*
>> [image: http://www.linkedin.com/pub/shashwat-shriparv/19/214/2a9]<http://www.linkedin.com/pub/shashwat-shriparv/19/214/2a9>[image:
>> https://twitter.com/shriparv] <https://twitter.com/shriparv>[image:
>> https://www.facebook.com/shriparv] <https://www.facebook.com/shriparv>[image:
>> http://google.com/+ShashwatShriparv]<http://google.com/+ShashwatShriparv>[image:
>> http://www.youtube.com/user/sShriparv/videos]<http://www.youtube.com/user/sShriparv/videos>[image:
>> http://profile.yahoo.com/SWXSTW3DVSDTF2HHSRM47AV6DI/]<sh...@yahoo.com>
>>
>>
>>
>> On Fri, Feb 14, 2014 at 4:15 PM, Nitin Pawar <ni...@gmail.com>wrote:
>>
>>> when you are writing a file, it will need an application to create and
>>> write the file.
>>> Thats why the mapreduce job to create the file.
>>>
>>> Where as for other way when its displayed on console, its just hdfs cat
>>> operation
>>>
>>>
>>> On Fri, Feb 14, 2014 at 4:13 PM, Sreenath <sr...@gmail.com>wrote:
>>>
>>>> i have two queries
>>>> 1. select * from tab1 limit 3;
>>>> this returns the 3 rows quickly withoput launcing any map reduce jobs;
>>>>
>>>> 2. INSERT OVERWRITE LOCAL DIRECTORY "/tmp/query1/" select * from tab1
>>>> limit 3;
>>>> but the same above query to write to a local directory launches a map
>>>> reduce job and scans through all the rows launching map-reduce jobs
>>>>
>>>> why the differenceinexecution of both queries?
>>>>
>>>> --
>>>> Sreenath S Kamath
>>>> Bangalore
>>>> Ph No:+91-9590989106
>>>>
>>>
>>>
>>>
>>> --
>>> Nitin Pawar
>>>
>>
>>
>
>
> --
> Sreenath S Kamath
> Bangalore
> Ph No:+91-9590989106
>
--
Nitin Pawar
Re: INSERT OVERWRITE LOCAL DIRECTORY
Posted by Sreenath <sr...@gmail.com>.
hi all,
really appreciate the responses do you have any means as a work around for
this ?
On Fri, Feb 14, 2014 at 4:22 PM, shashwat shriparv <
dwivedishashwat@gmail.com> wrote:
> Because when you do select * the query just has to read and return the
> resultl but when you give query under query suffle sort and comparison
> happens thats why mapreduce...
>
>
> *Warm Regards_**∞_*
> * Shashwat Shriparv*
> [image: http://www.linkedin.com/pub/shashwat-shriparv/19/214/2a9]<http://www.linkedin.com/pub/shashwat-shriparv/19/214/2a9>[image:
> https://twitter.com/shriparv] <https://twitter.com/shriparv>[image:
> https://www.facebook.com/shriparv] <https://www.facebook.com/shriparv>[image:
> http://google.com/+ShashwatShriparv] <http://google.com/+ShashwatShriparv>[image:
> http://www.youtube.com/user/sShriparv/videos]<http://www.youtube.com/user/sShriparv/videos>[image:
> http://profile.yahoo.com/SWXSTW3DVSDTF2HHSRM47AV6DI/] <sh...@yahoo.com>
>
>
>
> On Fri, Feb 14, 2014 at 4:15 PM, Nitin Pawar <ni...@gmail.com>wrote:
>
>> when you are writing a file, it will need an application to create and
>> write the file.
>> Thats why the mapreduce job to create the file.
>>
>> Where as for other way when its displayed on console, its just hdfs cat
>> operation
>>
>>
>> On Fri, Feb 14, 2014 at 4:13 PM, Sreenath <sr...@gmail.com>wrote:
>>
>>> i have two queries
>>> 1. select * from tab1 limit 3;
>>> this returns the 3 rows quickly withoput launcing any map reduce jobs;
>>>
>>> 2. INSERT OVERWRITE LOCAL DIRECTORY "/tmp/query1/" select * from tab1
>>> limit 3;
>>> but the same above query to write to a local directory launches a map
>>> reduce job and scans through all the rows launching map-reduce jobs
>>>
>>> why the differenceinexecution of both queries?
>>>
>>> --
>>> Sreenath S Kamath
>>> Bangalore
>>> Ph No:+91-9590989106
>>>
>>
>>
>>
>> --
>> Nitin Pawar
>>
>
>
--
Sreenath S Kamath
Bangalore
Ph No:+91-9590989106
Re: INSERT OVERWRITE LOCAL DIRECTORY
Posted by shashwat shriparv <dw...@gmail.com>.
Because when you do select * the query just has to read and return the
resultl but when you give query under query suffle sort and comparison
happens thats why mapreduce...
*Warm Regards_**∞_*
* Shashwat Shriparv*
[image: http://www.linkedin.com/pub/shashwat-shriparv/19/214/2a9]<http://www.linkedin.com/pub/shashwat-shriparv/19/214/2a9>[image:
https://twitter.com/shriparv] <https://twitter.com/shriparv>[image:
https://www.facebook.com/shriparv] <https://www.facebook.com/shriparv>[image:
http://google.com/+ShashwatShriparv]
<http://google.com/+ShashwatShriparv>[image:
http://www.youtube.com/user/sShriparv/videos]<http://www.youtube.com/user/sShriparv/videos>[image:
http://profile.yahoo.com/SWXSTW3DVSDTF2HHSRM47AV6DI/] <sh...@yahoo.com>
On Fri, Feb 14, 2014 at 4:15 PM, Nitin Pawar <ni...@gmail.com>wrote:
> when you are writing a file, it will need an application to create and
> write the file.
> Thats why the mapreduce job to create the file.
>
> Where as for other way when its displayed on console, its just hdfs cat
> operation
>
>
> On Fri, Feb 14, 2014 at 4:13 PM, Sreenath <sr...@gmail.com> wrote:
>
>> i have two queries
>> 1. select * from tab1 limit 3;
>> this returns the 3 rows quickly withoput launcing any map reduce jobs;
>>
>> 2. INSERT OVERWRITE LOCAL DIRECTORY "/tmp/query1/" select * from tab1
>> limit 3;
>> but the same above query to write to a local directory launches a map
>> reduce job and scans through all the rows launching map-reduce jobs
>>
>> why the differenceinexecution of both queries?
>>
>> --
>> Sreenath S Kamath
>> Bangalore
>> Ph No:+91-9590989106
>>
>
>
>
> --
> Nitin Pawar
>
Re: INSERT OVERWRITE LOCAL DIRECTORY
Posted by Nitin Pawar <ni...@gmail.com>.
when you are writing a file, it will need an application to create and
write the file.
Thats why the mapreduce job to create the file.
Where as for other way when its displayed on console, its just hdfs cat
operation
On Fri, Feb 14, 2014 at 4:13 PM, Sreenath <sr...@gmail.com> wrote:
> i have two queries
> 1. select * from tab1 limit 3;
> this returns the 3 rows quickly withoput launcing any map reduce jobs;
>
> 2. INSERT OVERWRITE LOCAL DIRECTORY "/tmp/query1/" select * from tab1
> limit 3;
> but the same above query to write to a local directory launches a map
> reduce job and scans through all the rows launching map-reduce jobs
>
> why the differenceinexecution of both queries?
>
> --
> Sreenath S Kamath
> Bangalore
> Ph No:+91-9590989106
>
--
Nitin Pawar