You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by dealmaker <vi...@gmail.com> on 2009/05/19 21:54:33 UTC
Mysql Load Data Infile with Hadoop?
Hi,
I am want to load data in mysql using a hadoop file similar to following:
LOAD DATA INFILE 'test.txt' INTO TABLE test
FIELDS TERMINATED BY ',' LINES STARTING BY 'xxx';
But how do I load the hdfs file into the mysql comand above? Do I start the
file name with hdfd://test.txt? I am using java.
Thanks.
--
View this message in context: http://www.nabble.com/Mysql-Load-Data-Infile-with-Hadoop--tp23623256p23623256.html
Sent from the Hadoop core-user mailing list archive at Nabble.com.
Re: Mysql Load Data Infile with Hadoop?
Posted by Sheldon Neuberger <ne...@gmail.com>.
You could copy the file to your local filesystem with something like
`hadoop dfs -copyToLocal test.txt local_test.txt`
On Tue, May 19, 2009 at 3:54 PM, dealmaker <vi...@gmail.com> wrote:
>
> Hi,
> I am want to load data in mysql using a hadoop file similar to following:
> LOAD DATA INFILE 'test.txt' INTO TABLE test
> FIELDS TERMINATED BY ',' LINES STARTING BY 'xxx';
>
> But how do I load the hdfs file into the mysql comand above? Do I start the
> file name with hdfd://test.txt? I am using java.
> Thanks.
>
> --
> View this message in context: http://www.nabble.com/Mysql-Load-Data-Infile-with-Hadoop--tp23623256p23623256.html
> Sent from the Hadoop core-user mailing list archive at Nabble.com.
>
>
Re: Mysql Load Data Infile with Hadoop?
Posted by Alex Loddengaard <al...@cloudera.com>.
DBOutputFormat will very likely put significantly more load on your MySQL
server vs. LOAD DATA INFILE. DBOutputFormat will trounce your MySQL server
with at least one connection per reducer. This may be OK if you have a
small number of reducers and a small amount of output data. LOAD DATA
INFILE is a much safer approach, in that it won't potentially axe your MySQL
database.
Alex
On Tue, May 19, 2009 at 2:58 PM, dealmaker <vi...@gmail.com> wrote:
>
> Does DBOutputFormat have similar performance as Load Data Infile?
> Thanks.
>
>
> TimRobertson100 wrote:
> >
> > So you are using a java program to execute a "load data infile"
> > command on mysql through JDBC?
> > If so I *think* you would have to copy it onto the mysql machine from
> > HDFS first, or the machine running the command and then try a 'load
> > data local infile'.
> >
> > Or pehaps use the
> >
> http://hadoop.apache.org/core/docs/current/api/org/apache/hadoop/mapred/lib/db/DBOutputFormat.html
> >
> > Cheers
> >
> > Tim
> >
> > On Tue, May 19, 2009 at 9:54 PM, dealmaker <vi...@gmail.com> wrote:
> >>
> >> Hi,
> >> I am want to load data in mysql using a hadoop file similar to
> >> following:
> >> LOAD DATA INFILE 'test.txt' INTO TABLE test
> >> FIELDS TERMINATED BY ',' LINES STARTING BY 'xxx';
> >>
> >> But how do I load the hdfs file into the mysql comand above? Do I start
> >> the
> >> file name with hdfd://test.txt? I am using java.
> >> Thanks.
> >>
> >> --
> >> View this message in context:
> >>
> http://www.nabble.com/Mysql-Load-Data-Infile-with-Hadoop--tp23623256p23623256.html
> >> Sent from the Hadoop core-user mailing list archive at Nabble.com.
> >>
> >>
> >
> >
>
> --
> View this message in context:
> http://www.nabble.com/Mysql-Load-Data-Infile-with-Hadoop--tp23623256p23625140.html
> Sent from the Hadoop core-user mailing list archive at Nabble.com.
>
>
Re: Mysql Load Data Infile with Hadoop?
Posted by dealmaker <vi...@gmail.com>.
Does DBOutputFormat have similar performance as Load Data Infile?
Thanks.
TimRobertson100 wrote:
>
> So you are using a java program to execute a "load data infile"
> command on mysql through JDBC?
> If so I *think* you would have to copy it onto the mysql machine from
> HDFS first, or the machine running the command and then try a 'load
> data local infile'.
>
> Or pehaps use the
> http://hadoop.apache.org/core/docs/current/api/org/apache/hadoop/mapred/lib/db/DBOutputFormat.html
>
> Cheers
>
> Tim
>
> On Tue, May 19, 2009 at 9:54 PM, dealmaker <vi...@gmail.com> wrote:
>>
>> Hi,
>> I am want to load data in mysql using a hadoop file similar to
>> following:
>> LOAD DATA INFILE 'test.txt' INTO TABLE test
>> FIELDS TERMINATED BY ',' LINES STARTING BY 'xxx';
>>
>> But how do I load the hdfs file into the mysql comand above? Do I start
>> the
>> file name with hdfd://test.txt? I am using java.
>> Thanks.
>>
>> --
>> View this message in context:
>> http://www.nabble.com/Mysql-Load-Data-Infile-with-Hadoop--tp23623256p23623256.html
>> Sent from the Hadoop core-user mailing list archive at Nabble.com.
>>
>>
>
>
--
View this message in context: http://www.nabble.com/Mysql-Load-Data-Infile-with-Hadoop--tp23623256p23625140.html
Sent from the Hadoop core-user mailing list archive at Nabble.com.
RE: Mysql Load Data Infile with Hadoop?
Posted by Marc Limotte <ml...@feeva.com>.
You might also try using something like Fuse-dfs (http://wiki.apache.org/hadoop/MountableHDFS) to "mount" the HDFS file system on the mysql machine. You could then use a standard unix path to specify the loadfile.
Marc
-----Original Message-----
From: tim robertson [mailto:timrobertson100@gmail.com]
Sent: Tuesday, May 19, 2009 1:03 PM
To: core-user@hadoop.apache.org
Subject: Re: Mysql Load Data Infile with Hadoop?
So you are using a java program to execute a "load data infile"
command on mysql through JDBC?
If so I *think* you would have to copy it onto the mysql machine from
HDFS first, or the machine running the command and then try a 'load
data local infile'.
Or pehaps use the
http://hadoop.apache.org/core/docs/current/api/org/apache/hadoop/mapred/lib/db/DBOutputFormat.html
Cheers
Tim
On Tue, May 19, 2009 at 9:54 PM, dealmaker <vi...@gmail.com> wrote:
>
> Hi,
> I am want to load data in mysql using a hadoop file similar to following:
> LOAD DATA INFILE 'test.txt' INTO TABLE test
> FIELDS TERMINATED BY ',' LINES STARTING BY 'xxx';
>
> But how do I load the hdfs file into the mysql comand above? Do I start the
> file name with hdfd://test.txt? I am using java.
> Thanks.
>
> --
> View this message in context: http://www.nabble.com/Mysql-Load-Data-Infile-with-Hadoop--tp23623256p23623256.html
> Sent from the Hadoop core-user mailing list archive at Nabble.com.
>
>
PRIVATE AND CONFIDENTIAL - NOTICE TO RECIPIENT: THIS E-MAIL IS MEANT FOR ONLY THE INTENDED RECIPIENT OF THE TRANSMISSION, AND MAY BE A COMMUNICATION PRIVILEGE BY LAW. IF YOU RECEIVED THIS E-MAIL IN ERROR, ANY REVIEW, USE, DISSEMINATION, DISTRIBUTION, OR COPYING OF THIS EMAIL IS STRICTLY PROHIBITED. PLEASE NOTIFY US IMMEDIATELY OF THE ERROR BY RETURN E-MAIL AND PLEASE DELETE THIS MESSAGE FROM YOUR SYSTEM.
Re: Mysql Load Data Infile with Hadoop?
Posted by tim robertson <ti...@gmail.com>.
So you are using a java program to execute a "load data infile"
command on mysql through JDBC?
If so I *think* you would have to copy it onto the mysql machine from
HDFS first, or the machine running the command and then try a 'load
data local infile'.
Or pehaps use the
http://hadoop.apache.org/core/docs/current/api/org/apache/hadoop/mapred/lib/db/DBOutputFormat.html
Cheers
Tim
On Tue, May 19, 2009 at 9:54 PM, dealmaker <vi...@gmail.com> wrote:
>
> Hi,
> I am want to load data in mysql using a hadoop file similar to following:
> LOAD DATA INFILE 'test.txt' INTO TABLE test
> FIELDS TERMINATED BY ',' LINES STARTING BY 'xxx';
>
> But how do I load the hdfs file into the mysql comand above? Do I start the
> file name with hdfd://test.txt? I am using java.
> Thanks.
>
> --
> View this message in context: http://www.nabble.com/Mysql-Load-Data-Infile-with-Hadoop--tp23623256p23623256.html
> Sent from the Hadoop core-user mailing list archive at Nabble.com.
>
>