You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hive.apache.org by "Omer, Farah" <fo...@microstrategy.com> on 2012/03/01 17:20:41 UTC
How to load a table from external server....
Hello,
Could anybody tell me how can I load data into a Hive table when the flat file is existing on another server and bit locally on Hadoop node.
For example, I am trying to load the table LU_CUSTOMER, and the flat file for this table exists on some other RH linux server: 10.11.12.13. The LU_CUSTOMER flat file is about 30 GB in size, hence if I move it locally to the Hadoop node, that will take a long time. I am trying to avoid this loading onto Hadoop node part.
So I wonder if there is a way to load the table directly from the other server.
The syntax that I know currently is: LOAD DATA LOCAL INPATH '/home/nzdata/CLOUD/SCRIPT/LU_CUSTOMER.txt' OVERWRITE INTO TABLE LU_CUSTOMER;
But in case I want to load from the other server directly, the path won't be local.
Any suggestions? Is that possible....
Thanks.
Farah Omer
Senior DB Engineer, MicroStrategy, Inc.
T: 703 2702230
E: fomer@microstrategy.com<ma...@microstrategy.com>
http://www.microstrategy.com
Re: How to load a table from external server....
Posted by Bejoy Ks <be...@yahoo.com>.
Hi Omar
You'd definitely need to copy the file into hdfs from your remote server. Even if you use LOAD DATA LOCAL ... the data is copied to hdfs . Hive would trigger map reduce jobs for hive QL and for that data is needed in hdfs. The better approach would be
-copy the data into hdfs
-use it with hive
Regards
Bejoy.KS
________________________________
From: "Omer, Farah" <fo...@microstrategy.com>
To: "hive-user@hadoop.apache.org" <hi...@hadoop.apache.org>
Sent: Thursday, March 1, 2012 9:50 PM
Subject: How to load a table from external server....
Hello,
Could anybody tell me how can I load data into a Hive table when the flat file is existing on another server and bit locally on Hadoop node.
For example, I am trying to load the table LU_CUSTOMER, and the flat file for this table exists on some other RH linux server: 10.11.12.13. The LU_CUSTOMER flat file is about 30 GB in size, hence if I move it locally to the Hadoop node, that will take
a long time. I am trying to avoid this loading onto Hadoop node part.
So I wonder if there is a way to load the table directly from the other server.
The syntax that I know currently is: LOAD DATA LOCAL INPATH '/home/nzdata/CLOUD/SCRIPT/LU_CUSTOMER.txt' OVERWRITE INTO TABLE LU_CUSTOMER;
But in case I want to load from the other server directly, the path won’t be local.
Any suggestions? Is that possible….
Thanks.
Farah Omer
Senior DB Engineer, MicroStrategy, Inc.
T:703 2702230
E:fomer@microstrategy.com
http://www.microstrategy.com
Re: How to load a table from external server....
Posted by Alex Kozlov <al...@cloudera.com>.
Hi Farah, If the data are on another server, you still need to move it one
way or another. A bare bone way to do this is to use `hadoop fs -put ...`
command after which you can create external or managed table in Hive. If
the data are in a relational DB you can use sqoop. You can also look into
Flume or Oozie workflow management tools to manage the whole process.
--
Alex K
<http://www.cloudera.com/company/press-center/hadoop-world-nyc/>
On Thu, Mar 1, 2012 at 8:20 AM, Omer, Farah <fo...@microstrategy.com> wrote:
> Hello,
>
> Could anybody tell me how can I load data into a Hive table when the flat
> file is existing on another server and bit locally on Hadoop node.
>
> For example, I am trying to load the table LU_CUSTOMER, and the flat file
> for this table exists on some other RH linux server: 10.11.12.13. The
> LU_CUSTOMER flat file is about 30 GB in size, hence if I move it locally to
> the Hadoop node, that will take a long time. I am trying to avoid this
> loading onto Hadoop node part.
> So I wonder if there is a way to load the table directly from the other
> server.
>
> The syntax that I know currently is: LOAD DATA LOCAL INPATH
> '/home/nzdata/CLOUD/SCRIPT/LU_CUSTOMER.txt' OVERWRITE INTO TABLE
> LU_CUSTOMER;
>
> But in case I want to load from the other server directly, the path won’t
> be local.
>
> Any suggestions? Is that possible….
>
> Thanks.
>
> *Farah Omer*
>
> *Senior DB Engineer, MicroStrategy, Inc.*
> T: 703 2702230
> E: *fomer@microstrategy.com* <fo...@microstrategy.com>
> *http://www.microstrategy.com* <http://www.microstrategy.com>
>
>
>
>
Re: How to load a table from external server....
Posted by Jonathan Seidman <jo...@gmail.com>.
Farah – can you configure the remote server as a client machine? You would
just need to install Hadoop with a configuration pointing to your cluster,
and then install Hive. You'd then be able to execute all Hive commands
against your cluster. Note that you won't run any daemons on this node, so
you want to make sure that none of the Hadoop processes are getting started.
Jonathan
On Thu, Mar 1, 2012 at 10:20 AM, Omer, Farah <fo...@microstrategy.com>wrote:
> Hello,
>
> Could anybody tell me how can I load data into a Hive table when the flat
> file is existing on another server and bit locally on Hadoop node.
>
> For example, I am trying to load the table LU_CUSTOMER, and the flat file
> for this table exists on some other RH linux server: 10.11.12.13. The
> LU_CUSTOMER flat file is about 30 GB in size, hence if I move it locally to
> the Hadoop node, that will take a long time. I am trying to avoid this
> loading onto Hadoop node part.
> So I wonder if there is a way to load the table directly from the other
> server.
>
> The syntax that I know currently is: LOAD DATA LOCAL INPATH
> '/home/nzdata/CLOUD/SCRIPT/LU_CUSTOMER.txt' OVERWRITE INTO TABLE
> LU_CUSTOMER;
>
> But in case I want to load from the other server directly, the path won’t
> be local.
>
> Any suggestions? Is that possible….
>
> Thanks.
>
> *Farah Omer*
>
> *Senior DB Engineer, MicroStrategy, Inc.*
> T: 703 2702230
> E: *fomer@microstrategy.com* <fo...@microstrategy.com>
> *http://www.microstrategy.com* <http://www.microstrategy.com>
>
>
>
>