You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hive.apache.org by Ryan LeCompte <le...@gmail.com> on 2009/10/15 22:27:19 UTC
Compressed output from queries
Hello,
If I exect a Hive query in the command-line shell, the results are displayed
in a human-readable format in the shell window. However, if I write the
query such that it gets redirected to a directory, the output looks like
it's stored in multiple "reduce" files, and the output is in compressed
format (non-human readable) format. Is there a way to get Hive to write the
query results to a file in the same way that it writes it to the console?
Thanks,
Ryan
Re: Compressed output from queries
Posted by Jeff Hammerbacher <ha...@cloudera.com>.
Opened a ticket at https://issues.apache.org/jira/browse/HIVE-886. We should
probably do the same for the map and list separators too.
On Sat, Oct 17, 2009 at 3:26 PM, Ryan LeCompte <le...@gmail.com> wrote:
> Hey Zheng,
>
> I was able to get around this for now, but thanks for pointing me in the
> right direction for changing the Hive source code. I'm still a Hive newbie,
> so I won't venture off and change the source code just yet! :) hopefully
> I'll get there soon enough, though.
>
> Thanks,
> Ryan
>
>
>
> On Sat, Oct 17, 2009 at 5:59 PM, Zheng Shao <zs...@gmail.com> wrote:
>
>> It's not supported yet.
>>
>> It's relatively easy to change the code to do that if you are interested.
>>
>> We can add a configurable parameter "hive.insert.directory.separator" in
>> HiveConf.java, and use that in SemanticAnalyzer.genFileSinkPlan():
>> table_desc = PlanUtils.getDefaultTableDesc(Integer.toString(*
>> Utilities.ctrlaCode*),
>> cols, colTypes,
>> false);
>>
>> It will be great if you can try it out and if it works, open a JIRA and
>> attach a patch.
>>
>> Thanks,
>> Zheng
>>
>>
>> On Sat, Oct 17, 2009 at 10:25 AM, Ryan LeCompte <le...@gmail.com>wrote:
>>
>>> Hello Zheng,
>>>
>>> This worked perfectly, thanks. It looks like Hive is using a strange
>>> control character as the field delimiter when it outputs the results to the
>>> directory on disk. Can I specify a different delimiter, perhaps a comma?
>>>
>>> Thanks,
>>> Ryan
>>>
>>>
>>> On Thu, Oct 15, 2009 at 9:06 PM, Zheng Shao <zs...@gmail.com> wrote:
>>>
>>>> set *hive*.*exec*.*compress*.*output=false;
>>>>
>>>> *This will disable the compression of output.
>>>>
>>>> Zheng
>>>>
>>>>
>>>> On Thu, Oct 15, 2009 at 2:02 PM, Ryan LeCompte <le...@gmail.com>wrote:
>>>>
>>>>> Thanks!
>>>>>
>>>>>
>>>>> On Thu, Oct 15, 2009 at 5:01 PM, Ning Zhang <nz...@facebook.com>wrote:
>>>>>
>>>>>> One way you can do is to create a temp table stored as textfile and
>>>>>> the insert query results into it, then get it from HDFS and unzip it
>>>>>> to your local file. Something like:
>>>>>>
>>>>>> hive> create tabel tmp (...) row format delimited fields terminted by
>>>>>> ',' stored as textfile;
>>>>>> hive> insert overwrite table tmp select ...;
>>>>>> $ hadoop dfs -cat <location of your tmp table> | gunzip -c >
>>>>>> my_result.txt
>>>>>>
>>>>>> Thanks,
>>>>>> Ning
>>>>>>
>>>>>> On Oct 15, 2009, at 1:27 PM, Ryan LeCompte wrote:
>>>>>>
>>>>>> > Hello,
>>>>>> >
>>>>>> > If I exect a Hive query in the command-line shell, the results are
>>>>>> > displayed in a human-readable format in the shell window. However,
>>>>>> > if I write the query such that it gets redirected to a directory,
>>>>>> > the output looks like it's stored in multiple "reduce" files, and
>>>>>> > the output is in compressed format (non-human readable) format. Is
>>>>>> > there a way to get Hive to write the query results to a file in the
>>>>>> > same way that it writes it to the console?
>>>>>> >
>>>>>> > Thanks,
>>>>>> > Ryan
>>>>>> >
>>>>>>
>>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Yours,
>>>> Zheng
>>>>
>>>
>>>
>>
>>
>> --
>> Yours,
>> Zheng
>>
>
>
Re: Compressed output from queries
Posted by Ryan LeCompte <le...@gmail.com>.
Hey Zheng,
I was able to get around this for now, but thanks for pointing me in the
right direction for changing the Hive source code. I'm still a Hive newbie,
so I won't venture off and change the source code just yet! :) hopefully
I'll get there soon enough, though.
Thanks,
Ryan
On Sat, Oct 17, 2009 at 5:59 PM, Zheng Shao <zs...@gmail.com> wrote:
> It's not supported yet.
>
> It's relatively easy to change the code to do that if you are interested.
>
> We can add a configurable parameter "hive.insert.directory.separator" in
> HiveConf.java, and use that in SemanticAnalyzer.genFileSinkPlan():
> table_desc = PlanUtils.getDefaultTableDesc(Integer.toString(*
> Utilities.ctrlaCode*),
> cols, colTypes,
> false);
>
> It will be great if you can try it out and if it works, open a JIRA and
> attach a patch.
>
> Thanks,
> Zheng
>
>
> On Sat, Oct 17, 2009 at 10:25 AM, Ryan LeCompte <le...@gmail.com>wrote:
>
>> Hello Zheng,
>>
>> This worked perfectly, thanks. It looks like Hive is using a strange
>> control character as the field delimiter when it outputs the results to the
>> directory on disk. Can I specify a different delimiter, perhaps a comma?
>>
>> Thanks,
>> Ryan
>>
>>
>> On Thu, Oct 15, 2009 at 9:06 PM, Zheng Shao <zs...@gmail.com> wrote:
>>
>>> set *hive*.*exec*.*compress*.*output=false;
>>>
>>> *This will disable the compression of output.
>>>
>>> Zheng
>>>
>>>
>>> On Thu, Oct 15, 2009 at 2:02 PM, Ryan LeCompte <le...@gmail.com>wrote:
>>>
>>>> Thanks!
>>>>
>>>>
>>>> On Thu, Oct 15, 2009 at 5:01 PM, Ning Zhang <nz...@facebook.com>wrote:
>>>>
>>>>> One way you can do is to create a temp table stored as textfile and
>>>>> the insert query results into it, then get it from HDFS and unzip it
>>>>> to your local file. Something like:
>>>>>
>>>>> hive> create tabel tmp (...) row format delimited fields terminted by
>>>>> ',' stored as textfile;
>>>>> hive> insert overwrite table tmp select ...;
>>>>> $ hadoop dfs -cat <location of your tmp table> | gunzip -c >
>>>>> my_result.txt
>>>>>
>>>>> Thanks,
>>>>> Ning
>>>>>
>>>>> On Oct 15, 2009, at 1:27 PM, Ryan LeCompte wrote:
>>>>>
>>>>> > Hello,
>>>>> >
>>>>> > If I exect a Hive query in the command-line shell, the results are
>>>>> > displayed in a human-readable format in the shell window. However,
>>>>> > if I write the query such that it gets redirected to a directory,
>>>>> > the output looks like it's stored in multiple "reduce" files, and
>>>>> > the output is in compressed format (non-human readable) format. Is
>>>>> > there a way to get Hive to write the query results to a file in the
>>>>> > same way that it writes it to the console?
>>>>> >
>>>>> > Thanks,
>>>>> > Ryan
>>>>> >
>>>>>
>>>>>
>>>>
>>>
>>>
>>> --
>>> Yours,
>>> Zheng
>>>
>>
>>
>
>
> --
> Yours,
> Zheng
>
Re: Compressed output from queries
Posted by Zheng Shao <zs...@gmail.com>.
It's not supported yet.
It's relatively easy to change the code to do that if you are interested.
We can add a configurable parameter "hive.insert.directory.separator" in
HiveConf.java, and use that in SemanticAnalyzer.genFileSinkPlan():
table_desc = PlanUtils.getDefaultTableDesc(Integer.toString(*
Utilities.ctrlaCode*),
cols, colTypes, false);
It will be great if you can try it out and if it works, open a JIRA and
attach a patch.
Thanks,
Zheng
On Sat, Oct 17, 2009 at 10:25 AM, Ryan LeCompte <le...@gmail.com> wrote:
> Hello Zheng,
>
> This worked perfectly, thanks. It looks like Hive is using a strange
> control character as the field delimiter when it outputs the results to the
> directory on disk. Can I specify a different delimiter, perhaps a comma?
>
> Thanks,
> Ryan
>
>
> On Thu, Oct 15, 2009 at 9:06 PM, Zheng Shao <zs...@gmail.com> wrote:
>
>> set *hive*.*exec*.*compress*.*output=false;
>>
>> *This will disable the compression of output.
>>
>> Zheng
>>
>>
>> On Thu, Oct 15, 2009 at 2:02 PM, Ryan LeCompte <le...@gmail.com>wrote:
>>
>>> Thanks!
>>>
>>>
>>> On Thu, Oct 15, 2009 at 5:01 PM, Ning Zhang <nz...@facebook.com> wrote:
>>>
>>>> One way you can do is to create a temp table stored as textfile and
>>>> the insert query results into it, then get it from HDFS and unzip it
>>>> to your local file. Something like:
>>>>
>>>> hive> create tabel tmp (...) row format delimited fields terminted by
>>>> ',' stored as textfile;
>>>> hive> insert overwrite table tmp select ...;
>>>> $ hadoop dfs -cat <location of your tmp table> | gunzip -c >
>>>> my_result.txt
>>>>
>>>> Thanks,
>>>> Ning
>>>>
>>>> On Oct 15, 2009, at 1:27 PM, Ryan LeCompte wrote:
>>>>
>>>> > Hello,
>>>> >
>>>> > If I exect a Hive query in the command-line shell, the results are
>>>> > displayed in a human-readable format in the shell window. However,
>>>> > if I write the query such that it gets redirected to a directory,
>>>> > the output looks like it's stored in multiple "reduce" files, and
>>>> > the output is in compressed format (non-human readable) format. Is
>>>> > there a way to get Hive to write the query results to a file in the
>>>> > same way that it writes it to the console?
>>>> >
>>>> > Thanks,
>>>> > Ryan
>>>> >
>>>>
>>>>
>>>
>>
>>
>> --
>> Yours,
>> Zheng
>>
>
>
--
Yours,
Zheng
Re: Compressed output from queries
Posted by Ryan LeCompte <le...@gmail.com>.
Hello Zheng,
This worked perfectly, thanks. It looks like Hive is using a strange control
character as the field delimiter when it outputs the results to the
directory on disk. Can I specify a different delimiter, perhaps a comma?
Thanks,
Ryan
On Thu, Oct 15, 2009 at 9:06 PM, Zheng Shao <zs...@gmail.com> wrote:
> set *hive*.*exec*.*compress*.*output=false;
>
> *This will disable the compression of output.
>
> Zheng
>
>
> On Thu, Oct 15, 2009 at 2:02 PM, Ryan LeCompte <le...@gmail.com> wrote:
>
>> Thanks!
>>
>>
>> On Thu, Oct 15, 2009 at 5:01 PM, Ning Zhang <nz...@facebook.com> wrote:
>>
>>> One way you can do is to create a temp table stored as textfile and
>>> the insert query results into it, then get it from HDFS and unzip it
>>> to your local file. Something like:
>>>
>>> hive> create tabel tmp (...) row format delimited fields terminted by
>>> ',' stored as textfile;
>>> hive> insert overwrite table tmp select ...;
>>> $ hadoop dfs -cat <location of your tmp table> | gunzip -c >
>>> my_result.txt
>>>
>>> Thanks,
>>> Ning
>>>
>>> On Oct 15, 2009, at 1:27 PM, Ryan LeCompte wrote:
>>>
>>> > Hello,
>>> >
>>> > If I exect a Hive query in the command-line shell, the results are
>>> > displayed in a human-readable format in the shell window. However,
>>> > if I write the query such that it gets redirected to a directory,
>>> > the output looks like it's stored in multiple "reduce" files, and
>>> > the output is in compressed format (non-human readable) format. Is
>>> > there a way to get Hive to write the query results to a file in the
>>> > same way that it writes it to the console?
>>> >
>>> > Thanks,
>>> > Ryan
>>> >
>>>
>>>
>>
>
>
> --
> Yours,
> Zheng
>
Re: Compressed output from queries
Posted by Zheng Shao <zs...@gmail.com>.
set *hive*.*exec*.*compress*.*output=false;
*This will disable the compression of output.
Zheng
On Thu, Oct 15, 2009 at 2:02 PM, Ryan LeCompte <le...@gmail.com> wrote:
> Thanks!
>
>
> On Thu, Oct 15, 2009 at 5:01 PM, Ning Zhang <nz...@facebook.com> wrote:
>
>> One way you can do is to create a temp table stored as textfile and
>> the insert query results into it, then get it from HDFS and unzip it
>> to your local file. Something like:
>>
>> hive> create tabel tmp (...) row format delimited fields terminted by
>> ',' stored as textfile;
>> hive> insert overwrite table tmp select ...;
>> $ hadoop dfs -cat <location of your tmp table> | gunzip -c >
>> my_result.txt
>>
>> Thanks,
>> Ning
>>
>> On Oct 15, 2009, at 1:27 PM, Ryan LeCompte wrote:
>>
>> > Hello,
>> >
>> > If I exect a Hive query in the command-line shell, the results are
>> > displayed in a human-readable format in the shell window. However,
>> > if I write the query such that it gets redirected to a directory,
>> > the output looks like it's stored in multiple "reduce" files, and
>> > the output is in compressed format (non-human readable) format. Is
>> > there a way to get Hive to write the query results to a file in the
>> > same way that it writes it to the console?
>> >
>> > Thanks,
>> > Ryan
>> >
>>
>>
>
--
Yours,
Zheng
Re: Compressed output from queries
Posted by Ryan LeCompte <le...@gmail.com>.
Thanks!
On Thu, Oct 15, 2009 at 5:01 PM, Ning Zhang <nz...@facebook.com> wrote:
> One way you can do is to create a temp table stored as textfile and
> the insert query results into it, then get it from HDFS and unzip it
> to your local file. Something like:
>
> hive> create tabel tmp (...) row format delimited fields terminted by
> ',' stored as textfile;
> hive> insert overwrite table tmp select ...;
> $ hadoop dfs -cat <location of your tmp table> | gunzip -c >
> my_result.txt
>
> Thanks,
> Ning
>
> On Oct 15, 2009, at 1:27 PM, Ryan LeCompte wrote:
>
> > Hello,
> >
> > If I exect a Hive query in the command-line shell, the results are
> > displayed in a human-readable format in the shell window. However,
> > if I write the query such that it gets redirected to a directory,
> > the output looks like it's stored in multiple "reduce" files, and
> > the output is in compressed format (non-human readable) format. Is
> > there a way to get Hive to write the query results to a file in the
> > same way that it writes it to the console?
> >
> > Thanks,
> > Ryan
> >
>
>
Re: Compressed output from queries
Posted by Ning Zhang <nz...@facebook.com>.
One way you can do is to create a temp table stored as textfile and
the insert query results into it, then get it from HDFS and unzip it
to your local file. Something like:
hive> create tabel tmp (...) row format delimited fields terminted by
',' stored as textfile;
hive> insert overwrite table tmp select ...;
$ hadoop dfs -cat <location of your tmp table> | gunzip -c >
my_result.txt
Thanks,
Ning
On Oct 15, 2009, at 1:27 PM, Ryan LeCompte wrote:
> Hello,
>
> If I exect a Hive query in the command-line shell, the results are
> displayed in a human-readable format in the shell window. However,
> if I write the query such that it gets redirected to a directory,
> the output looks like it's stored in multiple "reduce" files, and
> the output is in compressed format (non-human readable) format. Is
> there a way to get Hive to write the query results to a file in the
> same way that it writes it to the console?
>
> Thanks,
> Ryan
>