You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@sqoop.apache.org by Richa Sharma <ma...@gmail.com> on 2014/03/11 15:23:24 UTC
Sqoop import into a Hive table with 2 columns as partition keys
Hi,
I am trying to run sqoop import into a Hive table partitioned on 2 columns
(col1 string, col2 string).
I looked at the documentation for both --hive-import and --hcatalog and
could not find any such option.
It works successfully when table is partitioned on 1 column but not when i
increase number of partition columns.
Has anyone tried it and knows how to achieve it ?
Thanks,
Richa
Re: Sqoop import into a Hive table with 2 columns as partition keys
Posted by Venkat Ranganathan <vr...@hortonworks.com>.
If your partition columns are dynamic, then using hcatalog import will be
enough. The Sqoop HCatalog integration section in the Sqoop user guide
has more information on the HCatalog integration (that I linked earlier).
The restriction is that you can't have more than one static partition key.
Venkat
On Wed, Mar 12, 2014 at 11:28 PM, Richa Sharma
<ma...@gmail.com>wrote:
> Thanks for the inputs.
>
> In my case the partition columns are dynamic.. so I guess it will make
> sense to load it into 2 hops.
>
> Venkat, thanks for sharing the ppt on Hcatalog - I couldn't find much
> online so it is very helpful :-)
>
> Richa
>
>
> On Thu, Mar 13, 2014 at 11:33 AM, Venkat Ranganathan <
> vranganathan@hortonworks.com> wrote:
>
>> In Sqoop 1.4.4 we have Hcatalog Integration. This allows importing data
>> into hive partition tables with multiple partition keys (but only can be a
>> static partitioning key). See
>> http://sqoop.apache.org/docs/1.4.4/SqoopUserGuide.html#_sqoop_hcatalog_integration
>>
>> There is a presentation done at Hadoop World on Hcatalog integration that
>> you can refer to for more information
>>
>>
>> https://cwiki.apache.org/confluence/download/attachments/27361435/SqoopHCatIntegration-HadoopWorld2013.pptx
>>
>> Venkat
>>
>>
>> On Wed, Mar 12, 2014 at 9:19 PM, bejoy ks <be...@gmail.com> wrote:
>>
>>> AFAIK. Sqoop 1x imports supports hive single column partitions.
>>>
>>> If it is multi column partitions you might have to take a two hop
>>> approach
>>> 1. Load the data into a single/non partitioned hive tabe
>>> 2. From the above staging table load to the final one.
>>>
>>>
>>> On Tue, Mar 11, 2014 at 7:23 AM, Richa Sharma <
>>> mailtorichasharma@gmail.com> wrote:
>>>
>>>> Hi,
>>>>
>>>> I am trying to run sqoop import into a Hive table partitioned on 2
>>>> columns (col1 string, col2 string).
>>>>
>>>> I looked at the documentation for both --hive-import and --hcatalog and
>>>> could not find any such option.
>>>>
>>>> It works successfully when table is partitioned on 1 column but not
>>>> when i increase number of partition columns.
>>>>
>>>> Has anyone tried it and knows how to achieve it ?
>>>>
>>>> Thanks,
>>>> Richa
>>>>
>>>
>>>
>>>
>>> --
>>> Regards
>>> Bejoy
>>>
>>
>>
>> CONFIDENTIALITY NOTICE
>> NOTICE: This message is intended for the use of the individual or entity
>> to which it is addressed and may contain information that is confidential,
>> privileged and exempt from disclosure under applicable law. If the reader
>> of this message is not the intended recipient, you are hereby notified that
>> any printing, copying, dissemination, distribution, disclosure or
>> forwarding of this communication is strictly prohibited. If you have
>> received this communication in error, please contact the sender immediately
>> and delete it from your system. Thank You.
>
>
>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Sqoop import into a Hive table with 2 columns as partition keys
Posted by Richa Sharma <ma...@gmail.com>.
Thanks for the inputs.
In my case the partition columns are dynamic.. so I guess it will make
sense to load it into 2 hops.
Venkat, thanks for sharing the ppt on Hcatalog - I couldn't find much
online so it is very helpful :-)
Richa
On Thu, Mar 13, 2014 at 11:33 AM, Venkat Ranganathan <
vranganathan@hortonworks.com> wrote:
> In Sqoop 1.4.4 we have Hcatalog Integration. This allows importing data
> into hive partition tables with multiple partition keys (but only can be a
> static partitioning key). See
> http://sqoop.apache.org/docs/1.4.4/SqoopUserGuide.html#_sqoop_hcatalog_integration
>
> There is a presentation done at Hadoop World on Hcatalog integration that
> you can refer to for more information
>
>
> https://cwiki.apache.org/confluence/download/attachments/27361435/SqoopHCatIntegration-HadoopWorld2013.pptx
>
> Venkat
>
>
> On Wed, Mar 12, 2014 at 9:19 PM, bejoy ks <be...@gmail.com> wrote:
>
>> AFAIK. Sqoop 1x imports supports hive single column partitions.
>>
>> If it is multi column partitions you might have to take a two hop approach
>> 1. Load the data into a single/non partitioned hive tabe
>> 2. From the above staging table load to the final one.
>>
>>
>> On Tue, Mar 11, 2014 at 7:23 AM, Richa Sharma <
>> mailtorichasharma@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> I am trying to run sqoop import into a Hive table partitioned on 2
>>> columns (col1 string, col2 string).
>>>
>>> I looked at the documentation for both --hive-import and --hcatalog and
>>> could not find any such option.
>>>
>>> It works successfully when table is partitioned on 1 column but not when
>>> i increase number of partition columns.
>>>
>>> Has anyone tried it and knows how to achieve it ?
>>>
>>> Thanks,
>>> Richa
>>>
>>
>>
>>
>> --
>> Regards
>> Bejoy
>>
>
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
Re: Sqoop import into a Hive table with 2 columns as partition keys
Posted by Venkat Ranganathan <vr...@hortonworks.com>.
In Sqoop 1.4.4 we have Hcatalog Integration. This allows importing data
into hive partition tables with multiple partition keys (but only can be a
static partitioning key). See
http://sqoop.apache.org/docs/1.4.4/SqoopUserGuide.html#_sqoop_hcatalog_integration
There is a presentation done at Hadoop World on Hcatalog integration that
you can refer to for more information
https://cwiki.apache.org/confluence/download/attachments/27361435/SqoopHCatIntegration-HadoopWorld2013.pptx
Venkat
On Wed, Mar 12, 2014 at 9:19 PM, bejoy ks <be...@gmail.com> wrote:
> AFAIK. Sqoop 1x imports supports hive single column partitions.
>
> If it is multi column partitions you might have to take a two hop approach
> 1. Load the data into a single/non partitioned hive tabe
> 2. From the above staging table load to the final one.
>
>
> On Tue, Mar 11, 2014 at 7:23 AM, Richa Sharma <mailtorichasharma@gmail.com
> > wrote:
>
>> Hi,
>>
>> I am trying to run sqoop import into a Hive table partitioned on 2
>> columns (col1 string, col2 string).
>>
>> I looked at the documentation for both --hive-import and --hcatalog and
>> could not find any such option.
>>
>> It works successfully when table is partitioned on 1 column but not when
>> i increase number of partition columns.
>>
>> Has anyone tried it and knows how to achieve it ?
>>
>> Thanks,
>> Richa
>>
>
>
>
> --
> Regards
> Bejoy
>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Sqoop import into a Hive table with 2 columns as partition keys
Posted by bejoy ks <be...@gmail.com>.
AFAIK. Sqoop 1x imports supports hive single column partitions.
If it is multi column partitions you might have to take a two hop approach
1. Load the data into a single/non partitioned hive tabe
2. From the above staging table load to the final one.
On Tue, Mar 11, 2014 at 7:23 AM, Richa Sharma
<ma...@gmail.com>wrote:
> Hi,
>
> I am trying to run sqoop import into a Hive table partitioned on 2 columns
> (col1 string, col2 string).
>
> I looked at the documentation for both --hive-import and --hcatalog and
> could not find any such option.
>
> It works successfully when table is partitioned on 1 column but not when i
> increase number of partition columns.
>
> Has anyone tried it and knows how to achieve it ?
>
> Thanks,
> Richa
>
--
Regards
Bejoy