You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hive.apache.org by Min Zhou <co...@gmail.com> on 2009/07/14 12:18:19 UTC
how to export hive tables into oracle/mysql?
Hi all,
How do you export hive tables into oracle/mysql? through oci(Oracle Call
Interface), jdbc(oci/thin/MysqlDriver) or odbc?
Written such tools in c/c++ through HDFS native library or in java(jdbc)?
How do you translate hive tables' schema into oracle/mysql shema?
Thanks,
Min
--
My research interests are distributed systems, parallel computing and
bytecode based virtual machine.
My profile:
http://www.linkedin.com/in/coderplay
My blog:
http://coderplay.javaeye.com
Re: how to export hive tables into oracle/mysql?
Posted by Peter Skomoroch <pe...@gmail.com>.
new JIRA for the output delimiter issue:
https://issues.apache.org/jira/browse/HIVE-634
On Tue, Jul 14, 2009 at 12:24 PM, Prasad Chakka <pc...@facebook.com>wrote:
> Peter, can you open a JIRA for the output delimiter issue?
>
> Also the method described by Peter is the best (you could even split the
> output into multiple files and use Oracle’s parallel loader if you have a
> multi-core client and server). JDBC/ODBC/OCI will not be able to handle
> large data volumes.
>
>
> ------------------------------
> *From: *Peter Skomoroch <pe...@gmail.com>
> *Reply-To: *<hi...@hadoop.apache.org>
> *Date: *Tue, 14 Jul 2009 07:57:32 -0700
> *To: *<hi...@hadoop.apache.org>
> *Subject: *Re: how to export hive tables into oracle/mysql?
>
>
> I run a cluster on EC2, at the moment I just dump tables to ctrl-A
> delimited flat files with INSERT OVERWRITE DIRECTORY and bulk load those to
> MySQL with LOAD DATA LOCAL INFILE:
>
> something like this:
>
> LOAD DATA LOCAL INFILE '/mnt/pages.txt'
> INTO TABLE new_pages
> FIELDS TERMINATED BY 0x01
> LINES TERMINATED BY '\n'
> (id, url, title, page_latest, total_pageviews, monthly_trend, daily_trend);
>
>
> There seems to be a bug where ctrl-A is the only delimiter used, regardless
> of the Hive table structure (tab delimited etc.)
>
>
> On Tue, Jul 14, 2009 at 10:45 AM, Edward Capriolo <ed...@gmail.com>
> wrote:
>
> Min,
>
> Funny should should ask. I was talking to Aaron about sqoop, and
> un-squoop (hadoop->SQL). I am also started building a dboutput udf
> which I am about to open another thread about. It looks like we may
> open up a hadoop Jira, and possible a Hive Jiira to chat about this.
> Stay tuned!
>
> Edward
>
> On Tue, Jul 14, 2009 at 6:18 AM, Min Zhou<co...@gmail.com> wrote:
> > Hi all,
> >
> > How do you export hive tables into oracle/mysql? through oci(Oracle Call
> > Interface), jdbc(oci/thin/MysqlDriver) or odbc?
> > Written such tools in c/c++ through HDFS native library or in java(jdbc)?
> > How do you translate hive tables' schema into oracle/mysql shema?
> >
> > Thanks,
> > Min
> > --
> > My research interests are distributed systems, parallel computing and
> > bytecode based virtual machine.
> >
> > My profile:
> > http://www.linkedin.com/in/coderplay
> > My blog:
> > http://coderplay.javaeye.com
> >
>
>
>
>
> --
> Peter N. Skomoroch
> 617.285.8348
> http://www.datawrangling.com
> http://delicious.com/pskomoroch
> http://twitter.com/peteskomoroch
>
>
--
Peter N. Skomoroch
617.285.8348
http://www.datawrangling.com
http://delicious.com/pskomoroch
http://twitter.com/peteskomoroch
Re: how to export hive tables into oracle/mysql?
Posted by Prasad Chakka <pc...@facebook.com>.
Peter, can you open a JIRA for the output delimiter issue?
Also the method described by Peter is the best (you could even split the output into multiple files and use Oracle's parallel loader if you have a multi-core client and server). JDBC/ODBC/OCI will not be able to handle large data volumes.
________________________________
From: Peter Skomoroch <pe...@gmail.com>
Reply-To: <hi...@hadoop.apache.org>
Date: Tue, 14 Jul 2009 07:57:32 -0700
To: <hi...@hadoop.apache.org>
Subject: Re: how to export hive tables into oracle/mysql?
I run a cluster on EC2, at the moment I just dump tables to ctrl-A delimited flat files with INSERT OVERWRITE DIRECTORY and bulk load those to MySQL with LOAD DATA LOCAL INFILE:
something like this:
LOAD DATA LOCAL INFILE '/mnt/pages.txt'
INTO TABLE new_pages
FIELDS TERMINATED BY 0x01
LINES TERMINATED BY '\n'
(id, url, title, page_latest, total_pageviews, monthly_trend, daily_trend);
There seems to be a bug where ctrl-A is the only delimiter used, regardless of the Hive table structure (tab delimited etc.)
On Tue, Jul 14, 2009 at 10:45 AM, Edward Capriolo <ed...@gmail.com> wrote:
Min,
Funny should should ask. I was talking to Aaron about sqoop, and
un-squoop (hadoop->SQL). I am also started building a dboutput udf
which I am about to open another thread about. It looks like we may
open up a hadoop Jira, and possible a Hive Jiira to chat about this.
Stay tuned!
Edward
On Tue, Jul 14, 2009 at 6:18 AM, Min Zhou<co...@gmail.com> wrote:
> Hi all,
>
> How do you export hive tables into oracle/mysql? through oci(Oracle Call
> Interface), jdbc(oci/thin/MysqlDriver) or odbc?
> Written such tools in c/c++ through HDFS native library or in java(jdbc)?
> How do you translate hive tables' schema into oracle/mysql shema?
>
> Thanks,
> Min
> --
> My research interests are distributed systems, parallel computing and
> bytecode based virtual machine.
>
> My profile:
> http://www.linkedin.com/in/coderplay
> My blog:
> http://coderplay.javaeye.com
>
--
Peter N. Skomoroch
617.285.8348
http://www.datawrangling.com
http://delicious.com/pskomoroch
http://twitter.com/peteskomoroch
Re: how to export hive tables into oracle/mysql?
Posted by Peter Skomoroch <pe...@gmail.com>.
I run a cluster on EC2, at the moment I just dump tables to ctrl-A delimited
flat files with INSERT OVERWRITE DIRECTORY and bulk load those to MySQL with
LOAD DATA LOCAL INFILE:
something like this:
LOAD DATA LOCAL INFILE '/mnt/pages.txt'
INTO TABLE new_pages
FIELDS TERMINATED BY 0x01
LINES TERMINATED BY '\n'
(id, url, title, page_latest, total_pageviews, monthly_trend, daily_trend);
There seems to be a bug where ctrl-A is the only delimiter used, regardless
of the Hive table structure (tab delimited etc.)
On Tue, Jul 14, 2009 at 10:45 AM, Edward Capriolo <ed...@gmail.com>wrote:
> Min,
>
> Funny should should ask. I was talking to Aaron about sqoop, and
> un-squoop (hadoop->SQL). I am also started building a dboutput udf
> which I am about to open another thread about. It looks like we may
> open up a hadoop Jira, and possible a Hive Jiira to chat about this.
> Stay tuned!
>
> Edward
>
> On Tue, Jul 14, 2009 at 6:18 AM, Min Zhou<co...@gmail.com> wrote:
> > Hi all,
> >
> > How do you export hive tables into oracle/mysql? through oci(Oracle Call
> > Interface), jdbc(oci/thin/MysqlDriver) or odbc?
> > Written such tools in c/c++ through HDFS native library or in java(jdbc)?
> > How do you translate hive tables' schema into oracle/mysql shema?
> >
> > Thanks,
> > Min
> > --
> > My research interests are distributed systems, parallel computing and
> > bytecode based virtual machine.
> >
> > My profile:
> > http://www.linkedin.com/in/coderplay
> > My blog:
> > http://coderplay.javaeye.com
> >
>
--
Peter N. Skomoroch
617.285.8348
http://www.datawrangling.com
http://delicious.com/pskomoroch
http://twitter.com/peteskomoroch
Re: how to export hive tables into oracle/mysql?
Posted by Edward Capriolo <ed...@gmail.com>.
Min,
Funny should should ask. I was talking to Aaron about sqoop, and
un-squoop (hadoop->SQL). I am also started building a dboutput udf
which I am about to open another thread about. It looks like we may
open up a hadoop Jira, and possible a Hive Jiira to chat about this.
Stay tuned!
Edward
On Tue, Jul 14, 2009 at 6:18 AM, Min Zhou<co...@gmail.com> wrote:
> Hi all,
>
> How do you export hive tables into oracle/mysql? through oci(Oracle Call
> Interface), jdbc(oci/thin/MysqlDriver) or odbc?
> Written such tools in c/c++ through HDFS native library or in java(jdbc)?
> How do you translate hive tables' schema into oracle/mysql shema?
>
> Thanks,
> Min
> --
> My research interests are distributed systems, parallel computing and
> bytecode based virtual machine.
>
> My profile:
> http://www.linkedin.com/in/coderplay
> My blog:
> http://coderplay.javaeye.com
>