You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@sqoop.apache.org by Muthu Pandi <mu...@gmail.com> on 2014/07/11 08:05:46 UTC
Export only newly inserted values
Dear All
Am in an analytic project where all the data are fed into hive
warehouse and am in need of exporting those values to mysql since the
analytic on hive is too slow. Data will be coming to hive in timely fashion
so i need to export only the new data not all the data using sqoop export
kindly point me the direction.
*Regards Muthupandi.K*
[image: Picture (Device Independent Bitmap)]
Re: Export only newly inserted values
Posted by Brenden Cobb <Br...@humedica.com>.
Hey- sorry I misunderstood your question.
I don't think you're going to find an out-of-the-box solution for doing this with Sqoop. My suggestion is you'll need to schedule some ETL work in Map reduce, Pig, or Hive in order to separate out only the records your want to export, then call Sqoop to move the new file. I have a similar situation and Sqoop exports are really just a transport mechanism to move static data out of HDFS. Shaping that data is left to other tools.
From: Muthu Pandi <mu...@gmail.com>>
Reply-To: "user@sqoop.apache.org<ma...@sqoop.apache.org>" <us...@sqoop.apache.org>>
Date: Wednesday, July 16, 2014 12:14 AM
To: "user@sqoop.apache.org<ma...@sqoop.apache.org>" <us...@sqoop.apache.org>>
Subject: Re: Export only newly inserted values
Yes i can use Oozie for the schedule but while export the whole data will be exported again and again. let say i had 10000 rows at 4:00 and i do export 10000 data will be exported after 15 Min data will be increased by 10500 so while exporting at 4:15 again 10500 data will be exported i need to export only the new data that is 500 data.
Regards
Muthupandi.K
9894594112
[Picture (Device Independent Bitmap)]
On Tue, Jul 15, 2014 at 10:06 PM, Brenden Cobb <Br...@humedica.com>> wrote:
You might want to look into creating an Oozie job to handle your periodic exports.
From: Muthu Pandi <mu...@gmail.com>>
Reply-To: "user@sqoop.apache.org<ma...@sqoop.apache.org>" <us...@sqoop.apache.org>>
Date: Monday, July 14, 2014 12:25 AM
To: "user@sqoop.apache.org<ma...@sqoop.apache.org>" <us...@sqoop.apache.org>>
Subject: Re: Export only newly inserted values
Yes but i have to export periodically say 15 mins for the same day
Regards
Muthupandi.K
9894594112
[Picture (Device Independent Bitmap)]
On Sun, Jul 13, 2014 at 6:58 AM, Venkat Ranganathan <vr...@hortonworks.com>> wrote:
If your hive table is partitioned by date (say year/month/day), then you can export data from a specific partition using hcatalog
Thanks
Venkat
On Thu, Jul 10, 2014 at 11:05 PM, Muthu Pandi <mu...@gmail.com>> wrote:
Dear All
Am in an analytic project where all the data are fed into hive warehouse and am in need of exporting those values to mysql since the analytic on hive is too slow. Data will be coming to hive in timely fashion so i need to export only the new data not all the data using sqoop export kindly point me the direction.
Regards
Muthupandi.K
[Picture (Device Independent Bitmap)]
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
Re: Export only newly inserted values
Posted by Muthu Pandi <mu...@gmail.com>.
Yes i can use Oozie for the schedule but while export the whole data will
be exported again and again. let say i had 10000 rows at 4:00 and i do
export 10000 data will be exported after 15 Min data will be increased by
10500 so while exporting at 4:15 again 10500 data will be exported i need
to export only the new data that is 500 data.
*Regards Muthupandi.K9894594112*
[image: Picture (Device Independent Bitmap)]
On Tue, Jul 15, 2014 at 10:06 PM, Brenden Cobb <Br...@humedica.com>
wrote:
> You might want to look into creating an Oozie job to handle your
> periodic exports.
>
>
>
> From: Muthu Pandi <mu...@gmail.com>
> Reply-To: "user@sqoop.apache.org" <us...@sqoop.apache.org>
> Date: Monday, July 14, 2014 12:25 AM
> To: "user@sqoop.apache.org" <us...@sqoop.apache.org>
> Subject: Re: Export only newly inserted values
>
> Yes but i have to export periodically say 15 mins for the same day
>
>
>
> *Regards Muthupandi.K 9894594112*
>
> [image: Picture (Device Independent Bitmap)]
>
>
>
> On Sun, Jul 13, 2014 at 6:58 AM, Venkat Ranganathan <
> vranganathan@hortonworks.com> wrote:
>
>> If your hive table is partitioned by date (say year/month/day), then you
>> can export data from a specific partition using hcatalog
>>
>> Thanks
>>
>> Venkat
>>
>>
>> On Thu, Jul 10, 2014 at 11:05 PM, Muthu Pandi <mu...@gmail.com>
>> wrote:
>>
>>> Dear All
>>>
>>> Am in an analytic project where all the data are fed into hive
>>> warehouse and am in need of exporting those values to mysql since the
>>> analytic on hive is too slow. Data will be coming to hive in timely fashion
>>> so i need to export only the new data not all the data using sqoop export
>>> kindly point me the direction.
>>>
>>>
>>>
>>> *Regards Muthupandi.K *
>>>
>>> [image: Picture (Device Independent Bitmap)]
>>>
>>>
>>
>> CONFIDENTIALITY NOTICE
>> NOTICE: This message is intended for the use of the individual or entity
>> to which it is addressed and may contain information that is confidential,
>> privileged and exempt from disclosure under applicable law. If the reader
>> of this message is not the intended recipient, you are hereby notified that
>> any printing, copying, dissemination, distribution, disclosure or
>> forwarding of this communication is strictly prohibited. If you have
>> received this communication in error, please contact the sender immediately
>> and delete it from your system. Thank You.
>
>
>
Re: Export only newly inserted values
Posted by Brenden Cobb <Br...@humedica.com>.
You might want to look into creating an Oozie job to handle your periodic exports.
From: Muthu Pandi <mu...@gmail.com>>
Reply-To: "user@sqoop.apache.org<ma...@sqoop.apache.org>" <us...@sqoop.apache.org>>
Date: Monday, July 14, 2014 12:25 AM
To: "user@sqoop.apache.org<ma...@sqoop.apache.org>" <us...@sqoop.apache.org>>
Subject: Re: Export only newly inserted values
Yes but i have to export periodically say 15 mins for the same day
Regards
Muthupandi.K
9894594112
[Picture (Device Independent Bitmap)]
On Sun, Jul 13, 2014 at 6:58 AM, Venkat Ranganathan <vr...@hortonworks.com>> wrote:
If your hive table is partitioned by date (say year/month/day), then you can export data from a specific partition using hcatalog
Thanks
Venkat
On Thu, Jul 10, 2014 at 11:05 PM, Muthu Pandi <mu...@gmail.com>> wrote:
Dear All
Am in an analytic project where all the data are fed into hive warehouse and am in need of exporting those values to mysql since the analytic on hive is too slow. Data will be coming to hive in timely fashion so i need to export only the new data not all the data using sqoop export kindly point me the direction.
Regards
Muthupandi.K
[Picture (Device Independent Bitmap)]
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
Re: Export only newly inserted values
Posted by Muthu Pandi <mu...@gmail.com>.
Yes but i have to export periodically say 15 mins for the same day
*Regards Muthupandi.K9894594112*
[image: Picture (Device Independent Bitmap)]
On Sun, Jul 13, 2014 at 6:58 AM, Venkat Ranganathan <
vranganathan@hortonworks.com> wrote:
> If your hive table is partitioned by date (say year/month/day), then you
> can export data from a specific partition using hcatalog
>
> Thanks
>
> Venkat
>
>
> On Thu, Jul 10, 2014 at 11:05 PM, Muthu Pandi <mu...@gmail.com> wrote:
>
>> Dear All
>>
>> Am in an analytic project where all the data are fed into hive
>> warehouse and am in need of exporting those values to mysql since the
>> analytic on hive is too slow. Data will be coming to hive in timely fashion
>> so i need to export only the new data not all the data using sqoop export
>> kindly point me the direction.
>>
>>
>>
>> *Regards Muthupandi.K*
>>
>> [image: Picture (Device Independent Bitmap)]
>>
>>
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
Re: Export only newly inserted values
Posted by Venkat Ranganathan <vr...@hortonworks.com>.
If your hive table is partitioned by date (say year/month/day), then you
can export data from a specific partition using hcatalog
Thanks
Venkat
On Thu, Jul 10, 2014 at 11:05 PM, Muthu Pandi <mu...@gmail.com> wrote:
> Dear All
>
> Am in an analytic project where all the data are fed into hive
> warehouse and am in need of exporting those values to mysql since the
> analytic on hive is too slow. Data will be coming to hive in timely fashion
> so i need to export only the new data not all the data using sqoop export
> kindly point me the direction.
>
>
>
> *Regards Muthupandi.K*
>
> [image: Picture (Device Independent Bitmap)]
>
>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.