You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@sqoop.apache.org by "arvind@cloudera.com" <ar...@cloudera.com> on 2011/08/08 19:00:20 UTC
Re: [sqoop-user] Dynamic Partition & Sqoop
[bcc:sqoop-user@cloudera.org (deprecated), to:
sqoop-user@incubator.apache.org. Please continue the discussion on
Apache mailing lists - for more information see [1]].
Hi,
Sqoop does support populating Hive partitions. The relevant command
line arguments you need to supply are --hive-partition-key <key-name>,
--hive-partition-value <value-string>. Please refer to the user-guide
for more details.
Thanks,
Arvind
[1] https://cwiki.apache.org/confluence/display/SQOOP/Mailing+Lists
On Mon, Aug 8, 2011 at 5:17 AM, Cyberlord <kk...@gmail.com> wrote:
> I’d like to know if Dynamic Partition is supported by Sqoop.
>
> For eg. If I have a database(RDBMS) Informix and want to import data
> into Hive,
> Will it be possible for me to import the data through sqoop into Hive
> after creating dynamic partitions (supported by Hive) on-the-fly ?
>
> I possibly thought of loading the data into a temporary table (through
> Sqoop)
> and then load it into the required table through Hive interface.
>
> I wanted to know if I could migrate the data directly into Hive
> through Sqoop while creating dynamic partitions in Hive without
> using a temporary table.
>
> If it is possible, please let me know how to do it.
>
> Thanks :)
>
> --
> NOTE: The mailing list sqoop-user@cloudera.org is deprecated in favor of Apache Sqoop mailing list sqoop-user@incubator.apache.org. Please subscribe to it by sending an email to incubator-sqoop-user-subscribe@apache.org.
>
Re: [sqoop-user] Dynamic Partition & Sqoop
Posted by Arvind Prabhakar <ar...@apache.org>.
Hi Bejoy,
The support for dynamic partitions is not there out of the box. If
this is required for your use case, I would suggest filing a new
feature request Jira in [2].
[2] https://issues.apache.org/jira/browse/SQOOP
Thanks,
Arvind
On Mon, Aug 8, 2011 at 11:12 AM, <be...@gmail.com> wrote:
> Hi Arvind
> With this option is it feasible to accomplish dynamic partitions with a single SQOOP command?
> It does well for populating a static partition with values. It'd be great if you just brief on this implementation.
>
> ie with the step as mentioned, in Sqoop import we need to provide the where clause to fetch the relevant data from db and load it to some pre partition defined hive table, where the partitioned hive column is defined by '--hive-partition-key' and the value defined by '--hive-partition-value'. With this approach. In order to implement DYNAMIC partitions we need to analyze and identify the different values corresponding to the partition column of hive from db table. If there are n different values identified then n different SQOOP imports with appropriate where clauses need to be fired . Is my understanding right?
>
> Thank You
>
> Regards
> Bejoy K S
>
> -----Original Message-----
> From: "arvind@cloudera.com" <ar...@cloudera.com>
> Date: Mon, 8 Aug 2011 10:00:20
> To: <sq...@incubator.apache.org>
> Reply-To: sqoop-user@cloudera.org
> Subject: Re: [sqoop-user] Dynamic Partition & Sqoop
>
> [bcc:sqoop-user@cloudera.org (deprecated), to:
> sqoop-user@incubator.apache.org. Please continue the discussion on
> Apache mailing lists - for more information see [1]].
>
> Hi,
>
> Sqoop does support populating Hive partitions. The relevant command
> line arguments you need to supply are --hive-partition-key <key-name>,
> --hive-partition-value <value-string>. Please refer to the user-guide
> for more details.
>
> Thanks,
> Arvind
>
> [1] https://cwiki.apache.org/confluence/display/SQOOP/Mailing+Lists
>
> On Mon, Aug 8, 2011 at 5:17 AM, Cyberlord <kk...@gmail.com> wrote:
>> I’d like to know if Dynamic Partition is supported by Sqoop.
>>
>> For eg. If I have a database(RDBMS) Informix and want to import data
>> into Hive,
>> Will it be possible for me to import the data through sqoop into Hive
>> after creating dynamic partitions (supported by Hive) on-the-fly ?
>>
>> I possibly thought of loading the data into a temporary table (through
>> Sqoop)
>> and then load it into the required table through Hive interface.
>>
>> I wanted to know if I could migrate the data directly into Hive
>> through Sqoop while creating dynamic partitions in Hive without
>> using a temporary table.
>>
>> If it is possible, please let me know how to do it.
>>
>> Thanks :)
>>
>> --
>> NOTE: The mailing list sqoop-user@cloudera.org is deprecated in favor of Apache Sqoop mailing list sqoop-user@incubator.apache.org. Please subscribe to it by sending an email to incubator-sqoop-user-subscribe@apache.org.
>>
>
> --
> NOTE: The mailing list sqoop-user@cloudera.org is deprecated in favor of Apache Sqoop mailing list sqoop-user@incubator.apache.org. Please subscribe to it by sending an email to incubator-sqoop-user-subscribe@apache.org.
>
Re: [sqoop-user] Dynamic Partition & Sqoop
Posted by be...@gmail.com.
Hi Arvind
With this option is it feasible to accomplish dynamic partitions with a single SQOOP command?
It does well for populating a static partition with values. It'd be great if you just brief on this implementation.
ie with the step as mentioned, in Sqoop import we need to provide the where clause to fetch the relevant data from db and load it to some pre partition defined hive table, where the partitioned hive column is defined by '--hive-partition-key' and the value defined by '--hive-partition-value'. With this approach. In order to implement DYNAMIC partitions we need to analyze and identify the different values corresponding to the partition column of hive from db table. If there are n different values identified then n different SQOOP imports with appropriate where clauses need to be fired . Is my understanding right?
Thank You
Regards
Bejoy K S
-----Original Message-----
From: "arvind@cloudera.com" <ar...@cloudera.com>
Date: Mon, 8 Aug 2011 10:00:20
To: <sq...@incubator.apache.org>
Reply-To: sqoop-user@cloudera.org
Subject: Re: [sqoop-user] Dynamic Partition & Sqoop
[bcc:sqoop-user@cloudera.org (deprecated), to:
sqoop-user@incubator.apache.org. Please continue the discussion on
Apache mailing lists - for more information see [1]].
Hi,
Sqoop does support populating Hive partitions. The relevant command
line arguments you need to supply are --hive-partition-key <key-name>,
--hive-partition-value <value-string>. Please refer to the user-guide
for more details.
Thanks,
Arvind
[1] https://cwiki.apache.org/confluence/display/SQOOP/Mailing+Lists
On Mon, Aug 8, 2011 at 5:17 AM, Cyberlord <kk...@gmail.com> wrote:
> I’d like to know if Dynamic Partition is supported by Sqoop.
>
> For eg. If I have a database(RDBMS) Informix and want to import data
> into Hive,
> Will it be possible for me to import the data through sqoop into Hive
> after creating dynamic partitions (supported by Hive) on-the-fly ?
>
> I possibly thought of loading the data into a temporary table (through
> Sqoop)
> and then load it into the required table through Hive interface.
>
> I wanted to know if I could migrate the data directly into Hive
> through Sqoop while creating dynamic partitions in Hive without
> using a temporary table.
>
> If it is possible, please let me know how to do it.
>
> Thanks :)
>
> --
> NOTE: The mailing list sqoop-user@cloudera.org is deprecated in favor of Apache Sqoop mailing list sqoop-user@incubator.apache.org. Please subscribe to it by sending an email to incubator-sqoop-user-subscribe@apache.org.
>
--
NOTE: The mailing list sqoop-user@cloudera.org is deprecated in favor of Apache Sqoop mailing list sqoop-user@incubator.apache.org. Please subscribe to it by sending an email to incubator-sqoop-user-subscribe@apache.org.