You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-user@hadoop.apache.org by iwannaplay games <fu...@gmail.com> on 2012/07/19 12:19:46 UTC
Loading data in hdfs
Hi,
I am unable to use sqoop and want to load data in hdfs for testing,
Is there any way by which i can load my csv or text file to hadoop
file system directly without writing code in java
Regards
Prabhjot
Re: Loading data in hdfs
Posted by VIGNESH PRAJAPATI <vi...@gmail.com>.
Hello,
You can also do this by using Hive, But after that you have to deal with
database and table for further processing.
On Thu, Jul 19, 2012 at 6:19 AM, iwannaplay games <
funnlearnforkids@gmail.com> wrote:
> Hi,
>
> I am unable to use sqoop and want to load data in hdfs for testing,
> Is there any way by which i can load my csv or text file to hadoop
> file system directly without writing code in java
>
> Regards
> Prabhjot
>
--
*Vignesh Prajapati*
Tel: 9427415949 |
vignesh2066@gmail.com | www.vipras.com.co.in
MYTK [image: Facebook] <https://www.facebook.com/vigs143> [image:
Twitter]<http://twitter.com/#%21/vigs143> [image:
LinkedIn] <http://www.linkedin.com/pub/vignesh-prajapati/37/756/46a> [image:
about.me] <http://www.way4fun.tk><http://r1.wisestamp.com/r/landing?promo=7&dest=http%3A%2F%2Fwww.wisestamp.com%2Femail-install%3Futm_source%3Dextension%26utm_medium%3Demail%26utm_campaign%3Dpromo_7>
Re: Loading data in hdfs
Posted by Mohammad Tariq <do...@gmail.com>.
There is absolutely no need to be sorry. And once you have the data
inside your Hdfs you can use importtsv to imort the data like this :
$ bin/hbase org.apache.hadoop.hbase.mapreduce.ImportTsv
-Dimporttsv.columns=a,b,c <tablename> <hdfs-inputdir>
Regards,
Mohammad Tariq
On Thu, Jul 19, 2012 at 5:12 PM, iwannaplay games
<fu...@gmail.com> wrote:
> Hi,
>
> I am sorry ,i am troubling you a lot.
>
> Thanks for helping me :)
>
> This file is there in my hdfs IPData ( both in csv and tsv format).
> Can i transfer the records from this file and populate this data using
> importtsv?
> If yes then what source and destination should i write.
>
> Thanks
> Prabhjot
>
>
> On 7/19/12, Mohammad Tariq <do...@gmail.com> wrote:
>> By looking at the csv, I would suggest to create an hbase table, say
>> 'IPINFO' with one column family, say 'cf' having 3 columns for
>> 'startip', 'endip' and 'countryname' respectively.
>>
>> Regards,
>> Mohammad Tariq
>>
>>
>> On Thu, Jul 19, 2012 at 4:44 PM, Mohammad Tariq <do...@gmail.com> wrote:
>>> You have a few options. You can write a java program using Hbase API
>>> to do that. But you won't be able to exploit the parallelism to its
>>> fullest. Another option is to write a MapReduce program to do the
>>> same. You can also use Hive and Pig to serve the purpose. But if you
>>> are just starting with Hbase, then I would suggest to first get
>>> yourself familiar with Hbase API first and then go for other things.
>>>
>>> Regards,
>>> Mohammad Tariq
>>>
>>>
>>> On Thu, Jul 19, 2012 at 4:12 PM, iwannaplay games
>>> <fu...@gmail.com> wrote:
>>>> PFA the csv file
>>>>
>>>> Thanks
>>>> Prabhjot
>>>>
>>>>
>>>> On 7/19/12, Mohammad Tariq <do...@gmail.com> wrote:
>>>>> Could you show me the structure of your csv, if possible??
>>>>>
>>>>> Regards,
>>>>> Mohammad Tariq
>>>>>
>>>>>
>>>>> On Thu, Jul 19, 2012 at 4:03 PM, iwannaplay games
>>>>> <fu...@gmail.com> wrote:
>>>>>> Thanks Tariq
>>>>>>
>>>>>> Now i want to convert this csv file to table.(HBase table with column
>>>>>> families)
>>>>>> How can i do that
>>>>>>
>>>>>> Regards
>>>>>> Prabhjot
>>>>>>
>>>>>>
>>>>>> On 7/19/12, Mohammad Tariq <do...@gmail.com> wrote:
>>>>>>> Hi Prabhjot
>>>>>>>
>>>>>>> You can also use :
>>>>>>> hadoop fs -put <src fs path> <destn hdfs path>
>>>>>>>
>>>>>>> Regards,
>>>>>>> Mohammad Tariq
>>>>>>>
>>>>>>>
>>>>>>> On Thu, Jul 19, 2012 at 3:52 PM, Bejoy Ks <be...@gmail.com>
>>>>>>> wrote:
>>>>>>>> Hi Prabhjot
>>>>>>>>
>>>>>>>> Yes, Just use the filesystem commands
>>>>>>>> hadoop fs -copyFromLocal <src fs path> <destn hdfs path>
>>>>>>>>
>>>>>>>> Regards
>>>>>>>> Bejoy KS
>>>>>>>>
>>>>>>>> On Thu, Jul 19, 2012 at 3:49 PM, iwannaplay games
>>>>>>>> <fu...@gmail.com> wrote:
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> I am unable to use sqoop and want to load data in hdfs for testing,
>>>>>>>>> Is there any way by which i can load my csv or text file to hadoop
>>>>>>>>> file system directly without writing code in java
>>>>>>>>>
>>>>>>>>> Regards
>>>>>>>>> Prabhjot
>>>>>>>
>>>>>
>>
Re: Loading data in hdfs
Posted by iwannaplay games <fu...@gmail.com>.
Hi,
I am sorry ,i am troubling you a lot.
Thanks for helping me :)
This file is there in my hdfs IPData ( both in csv and tsv format).
Can i transfer the records from this file and populate this data using
importtsv?
If yes then what source and destination should i write.
Thanks
Prabhjot
On 7/19/12, Mohammad Tariq <do...@gmail.com> wrote:
> By looking at the csv, I would suggest to create an hbase table, say
> 'IPINFO' with one column family, say 'cf' having 3 columns for
> 'startip', 'endip' and 'countryname' respectively.
>
> Regards,
> Mohammad Tariq
>
>
> On Thu, Jul 19, 2012 at 4:44 PM, Mohammad Tariq <do...@gmail.com> wrote:
>> You have a few options. You can write a java program using Hbase API
>> to do that. But you won't be able to exploit the parallelism to its
>> fullest. Another option is to write a MapReduce program to do the
>> same. You can also use Hive and Pig to serve the purpose. But if you
>> are just starting with Hbase, then I would suggest to first get
>> yourself familiar with Hbase API first and then go for other things.
>>
>> Regards,
>> Mohammad Tariq
>>
>>
>> On Thu, Jul 19, 2012 at 4:12 PM, iwannaplay games
>> <fu...@gmail.com> wrote:
>>> PFA the csv file
>>>
>>> Thanks
>>> Prabhjot
>>>
>>>
>>> On 7/19/12, Mohammad Tariq <do...@gmail.com> wrote:
>>>> Could you show me the structure of your csv, if possible??
>>>>
>>>> Regards,
>>>> Mohammad Tariq
>>>>
>>>>
>>>> On Thu, Jul 19, 2012 at 4:03 PM, iwannaplay games
>>>> <fu...@gmail.com> wrote:
>>>>> Thanks Tariq
>>>>>
>>>>> Now i want to convert this csv file to table.(HBase table with column
>>>>> families)
>>>>> How can i do that
>>>>>
>>>>> Regards
>>>>> Prabhjot
>>>>>
>>>>>
>>>>> On 7/19/12, Mohammad Tariq <do...@gmail.com> wrote:
>>>>>> Hi Prabhjot
>>>>>>
>>>>>> You can also use :
>>>>>> hadoop fs -put <src fs path> <destn hdfs path>
>>>>>>
>>>>>> Regards,
>>>>>> Mohammad Tariq
>>>>>>
>>>>>>
>>>>>> On Thu, Jul 19, 2012 at 3:52 PM, Bejoy Ks <be...@gmail.com>
>>>>>> wrote:
>>>>>>> Hi Prabhjot
>>>>>>>
>>>>>>> Yes, Just use the filesystem commands
>>>>>>> hadoop fs -copyFromLocal <src fs path> <destn hdfs path>
>>>>>>>
>>>>>>> Regards
>>>>>>> Bejoy KS
>>>>>>>
>>>>>>> On Thu, Jul 19, 2012 at 3:49 PM, iwannaplay games
>>>>>>> <fu...@gmail.com> wrote:
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> I am unable to use sqoop and want to load data in hdfs for testing,
>>>>>>>> Is there any way by which i can load my csv or text file to hadoop
>>>>>>>> file system directly without writing code in java
>>>>>>>>
>>>>>>>> Regards
>>>>>>>> Prabhjot
>>>>>>
>>>>
>
Re: Loading data in hdfs
Posted by Mohammad Tariq <do...@gmail.com>.
By looking at the csv, I would suggest to create an hbase table, say
'IPINFO' with one column family, say 'cf' having 3 columns for
'startip', 'endip' and 'countryname' respectively.
Regards,
Mohammad Tariq
On Thu, Jul 19, 2012 at 4:44 PM, Mohammad Tariq <do...@gmail.com> wrote:
> You have a few options. You can write a java program using Hbase API
> to do that. But you won't be able to exploit the parallelism to its
> fullest. Another option is to write a MapReduce program to do the
> same. You can also use Hive and Pig to serve the purpose. But if you
> are just starting with Hbase, then I would suggest to first get
> yourself familiar with Hbase API first and then go for other things.
>
> Regards,
> Mohammad Tariq
>
>
> On Thu, Jul 19, 2012 at 4:12 PM, iwannaplay games
> <fu...@gmail.com> wrote:
>> PFA the csv file
>>
>> Thanks
>> Prabhjot
>>
>>
>> On 7/19/12, Mohammad Tariq <do...@gmail.com> wrote:
>>> Could you show me the structure of your csv, if possible??
>>>
>>> Regards,
>>> Mohammad Tariq
>>>
>>>
>>> On Thu, Jul 19, 2012 at 4:03 PM, iwannaplay games
>>> <fu...@gmail.com> wrote:
>>>> Thanks Tariq
>>>>
>>>> Now i want to convert this csv file to table.(HBase table with column
>>>> families)
>>>> How can i do that
>>>>
>>>> Regards
>>>> Prabhjot
>>>>
>>>>
>>>> On 7/19/12, Mohammad Tariq <do...@gmail.com> wrote:
>>>>> Hi Prabhjot
>>>>>
>>>>> You can also use :
>>>>> hadoop fs -put <src fs path> <destn hdfs path>
>>>>>
>>>>> Regards,
>>>>> Mohammad Tariq
>>>>>
>>>>>
>>>>> On Thu, Jul 19, 2012 at 3:52 PM, Bejoy Ks <be...@gmail.com>
>>>>> wrote:
>>>>>> Hi Prabhjot
>>>>>>
>>>>>> Yes, Just use the filesystem commands
>>>>>> hadoop fs -copyFromLocal <src fs path> <destn hdfs path>
>>>>>>
>>>>>> Regards
>>>>>> Bejoy KS
>>>>>>
>>>>>> On Thu, Jul 19, 2012 at 3:49 PM, iwannaplay games
>>>>>> <fu...@gmail.com> wrote:
>>>>>>> Hi,
>>>>>>>
>>>>>>> I am unable to use sqoop and want to load data in hdfs for testing,
>>>>>>> Is there any way by which i can load my csv or text file to hadoop
>>>>>>> file system directly without writing code in java
>>>>>>>
>>>>>>> Regards
>>>>>>> Prabhjot
>>>>>
>>>
Re: Loading data in hdfs
Posted by Mohammad Tariq <do...@gmail.com>.
You have a few options. You can write a java program using Hbase API
to do that. But you won't be able to exploit the parallelism to its
fullest. Another option is to write a MapReduce program to do the
same. You can also use Hive and Pig to serve the purpose. But if you
are just starting with Hbase, then I would suggest to first get
yourself familiar with Hbase API first and then go for other things.
Regards,
Mohammad Tariq
On Thu, Jul 19, 2012 at 4:12 PM, iwannaplay games
<fu...@gmail.com> wrote:
> PFA the csv file
>
> Thanks
> Prabhjot
>
>
> On 7/19/12, Mohammad Tariq <do...@gmail.com> wrote:
>> Could you show me the structure of your csv, if possible??
>>
>> Regards,
>> Mohammad Tariq
>>
>>
>> On Thu, Jul 19, 2012 at 4:03 PM, iwannaplay games
>> <fu...@gmail.com> wrote:
>>> Thanks Tariq
>>>
>>> Now i want to convert this csv file to table.(HBase table with column
>>> families)
>>> How can i do that
>>>
>>> Regards
>>> Prabhjot
>>>
>>>
>>> On 7/19/12, Mohammad Tariq <do...@gmail.com> wrote:
>>>> Hi Prabhjot
>>>>
>>>> You can also use :
>>>> hadoop fs -put <src fs path> <destn hdfs path>
>>>>
>>>> Regards,
>>>> Mohammad Tariq
>>>>
>>>>
>>>> On Thu, Jul 19, 2012 at 3:52 PM, Bejoy Ks <be...@gmail.com>
>>>> wrote:
>>>>> Hi Prabhjot
>>>>>
>>>>> Yes, Just use the filesystem commands
>>>>> hadoop fs -copyFromLocal <src fs path> <destn hdfs path>
>>>>>
>>>>> Regards
>>>>> Bejoy KS
>>>>>
>>>>> On Thu, Jul 19, 2012 at 3:49 PM, iwannaplay games
>>>>> <fu...@gmail.com> wrote:
>>>>>> Hi,
>>>>>>
>>>>>> I am unable to use sqoop and want to load data in hdfs for testing,
>>>>>> Is there any way by which i can load my csv or text file to hadoop
>>>>>> file system directly without writing code in java
>>>>>>
>>>>>> Regards
>>>>>> Prabhjot
>>>>
>>
Re: Loading data in hdfs
Posted by iwannaplay games <fu...@gmail.com>.
PFA the csv file
Thanks
Prabhjot
On 7/19/12, Mohammad Tariq <do...@gmail.com> wrote:
> Could you show me the structure of your csv, if possible??
>
> Regards,
> Mohammad Tariq
>
>
> On Thu, Jul 19, 2012 at 4:03 PM, iwannaplay games
> <fu...@gmail.com> wrote:
>> Thanks Tariq
>>
>> Now i want to convert this csv file to table.(HBase table with column
>> families)
>> How can i do that
>>
>> Regards
>> Prabhjot
>>
>>
>> On 7/19/12, Mohammad Tariq <do...@gmail.com> wrote:
>>> Hi Prabhjot
>>>
>>> You can also use :
>>> hadoop fs -put <src fs path> <destn hdfs path>
>>>
>>> Regards,
>>> Mohammad Tariq
>>>
>>>
>>> On Thu, Jul 19, 2012 at 3:52 PM, Bejoy Ks <be...@gmail.com>
>>> wrote:
>>>> Hi Prabhjot
>>>>
>>>> Yes, Just use the filesystem commands
>>>> hadoop fs -copyFromLocal <src fs path> <destn hdfs path>
>>>>
>>>> Regards
>>>> Bejoy KS
>>>>
>>>> On Thu, Jul 19, 2012 at 3:49 PM, iwannaplay games
>>>> <fu...@gmail.com> wrote:
>>>>> Hi,
>>>>>
>>>>> I am unable to use sqoop and want to load data in hdfs for testing,
>>>>> Is there any way by which i can load my csv or text file to hadoop
>>>>> file system directly without writing code in java
>>>>>
>>>>> Regards
>>>>> Prabhjot
>>>
>
Re: Loading data in hdfs
Posted by Mohammad Tariq <do...@gmail.com>.
Could you show me the structure of your csv, if possible??
Regards,
Mohammad Tariq
On Thu, Jul 19, 2012 at 4:03 PM, iwannaplay games
<fu...@gmail.com> wrote:
> Thanks Tariq
>
> Now i want to convert this csv file to table.(HBase table with column families)
> How can i do that
>
> Regards
> Prabhjot
>
>
> On 7/19/12, Mohammad Tariq <do...@gmail.com> wrote:
>> Hi Prabhjot
>>
>> You can also use :
>> hadoop fs -put <src fs path> <destn hdfs path>
>>
>> Regards,
>> Mohammad Tariq
>>
>>
>> On Thu, Jul 19, 2012 at 3:52 PM, Bejoy Ks <be...@gmail.com> wrote:
>>> Hi Prabhjot
>>>
>>> Yes, Just use the filesystem commands
>>> hadoop fs -copyFromLocal <src fs path> <destn hdfs path>
>>>
>>> Regards
>>> Bejoy KS
>>>
>>> On Thu, Jul 19, 2012 at 3:49 PM, iwannaplay games
>>> <fu...@gmail.com> wrote:
>>>> Hi,
>>>>
>>>> I am unable to use sqoop and want to load data in hdfs for testing,
>>>> Is there any way by which i can load my csv or text file to hadoop
>>>> file system directly without writing code in java
>>>>
>>>> Regards
>>>> Prabhjot
>>
Re: Loading data in hdfs
Posted by lars hofhansl <lh...@yahoo.com>.
Check out HBase's importtsv. (http://hbase.apache.org/book/ops_mgt.html#importtsv)
-- Lars
----- Original Message -----
From: iwannaplay games <fu...@gmail.com>
To: hdfs-user@hadoop.apache.org
Cc:
Sent: Thursday, July 19, 2012 3:33 AM
Subject: Re: Loading data in hdfs
Thanks Tariq
Now i want to convert this csv file to table.(HBase table with column families)
How can i do that
Regards
Prabhjot
On 7/19/12, Mohammad Tariq <do...@gmail.com> wrote:
> Hi Prabhjot
>
> You can also use :
> hadoop fs -put <src fs path> <destn hdfs path>
>
> Regards,
> Mohammad Tariq
>
>
> On Thu, Jul 19, 2012 at 3:52 PM, Bejoy Ks <be...@gmail.com> wrote:
>> Hi Prabhjot
>>
>> Yes, Just use the filesystem commands
>> hadoop fs -copyFromLocal <src fs path> <destn hdfs path>
>>
>> Regards
>> Bejoy KS
>>
>> On Thu, Jul 19, 2012 at 3:49 PM, iwannaplay games
>> <fu...@gmail.com> wrote:
>>> Hi,
>>>
>>> I am unable to use sqoop and want to load data in hdfs for testing,
>>> Is there any way by which i can load my csv or text file to hadoop
>>> file system directly without writing code in java
>>>
>>> Regards
>>> Prabhjot
>
Re: Loading data in hdfs
Posted by Harsh J <ha...@cloudera.com>.
Hi,
This is a question suited for the HBase user lists (user@hbase.apache.org).
In any case, you want to use the importtsv tool:
http://hbase.apache.org/book.html#importtsv. You do not have to write any
custom code to do it.
On Thu, Jul 19, 2012 at 4:03 PM, iwannaplay games <
funnlearnforkids@gmail.com> wrote:
> Thanks Tariq
>
> Now i want to convert this csv file to table.(HBase table with column
> families)
> How can i do that
>
> Regards
> Prabhjot
>
>
> On 7/19/12, Mohammad Tariq <do...@gmail.com> wrote:
> > Hi Prabhjot
> >
> > You can also use :
> > hadoop fs -put <src fs path> <destn hdfs path>
> >
> > Regards,
> > Mohammad Tariq
> >
> >
> > On Thu, Jul 19, 2012 at 3:52 PM, Bejoy Ks <be...@gmail.com>
> wrote:
> >> Hi Prabhjot
> >>
> >> Yes, Just use the filesystem commands
> >> hadoop fs -copyFromLocal <src fs path> <destn hdfs path>
> >>
> >> Regards
> >> Bejoy KS
> >>
> >> On Thu, Jul 19, 2012 at 3:49 PM, iwannaplay games
> >> <fu...@gmail.com> wrote:
> >>> Hi,
> >>>
> >>> I am unable to use sqoop and want to load data in hdfs for testing,
> >>> Is there any way by which i can load my csv or text file to hadoop
> >>> file system directly without writing code in java
> >>>
> >>> Regards
> >>> Prabhjot
> >
>
--
Harsh J
Re: Loading data in hdfs
Posted by iwannaplay games <fu...@gmail.com>.
Thanks Tariq
Now i want to convert this csv file to table.(HBase table with column families)
How can i do that
Regards
Prabhjot
On 7/19/12, Mohammad Tariq <do...@gmail.com> wrote:
> Hi Prabhjot
>
> You can also use :
> hadoop fs -put <src fs path> <destn hdfs path>
>
> Regards,
> Mohammad Tariq
>
>
> On Thu, Jul 19, 2012 at 3:52 PM, Bejoy Ks <be...@gmail.com> wrote:
>> Hi Prabhjot
>>
>> Yes, Just use the filesystem commands
>> hadoop fs -copyFromLocal <src fs path> <destn hdfs path>
>>
>> Regards
>> Bejoy KS
>>
>> On Thu, Jul 19, 2012 at 3:49 PM, iwannaplay games
>> <fu...@gmail.com> wrote:
>>> Hi,
>>>
>>> I am unable to use sqoop and want to load data in hdfs for testing,
>>> Is there any way by which i can load my csv or text file to hadoop
>>> file system directly without writing code in java
>>>
>>> Regards
>>> Prabhjot
>
Re: Loading data in hdfs
Posted by Mohammad Tariq <do...@gmail.com>.
Hi Prabhjot
You can also use :
hadoop fs -put <src fs path> <destn hdfs path>
Regards,
Mohammad Tariq
On Thu, Jul 19, 2012 at 3:52 PM, Bejoy Ks <be...@gmail.com> wrote:
> Hi Prabhjot
>
> Yes, Just use the filesystem commands
> hadoop fs -copyFromLocal <src fs path> <destn hdfs path>
>
> Regards
> Bejoy KS
>
> On Thu, Jul 19, 2012 at 3:49 PM, iwannaplay games
> <fu...@gmail.com> wrote:
>> Hi,
>>
>> I am unable to use sqoop and want to load data in hdfs for testing,
>> Is there any way by which i can load my csv or text file to hadoop
>> file system directly without writing code in java
>>
>> Regards
>> Prabhjot
Re: Loading data in hdfs
Posted by Bejoy Ks <be...@gmail.com>.
Hi Prabhjot
Yes, Just use the filesystem commands
hadoop fs -copyFromLocal <src fs path> <destn hdfs path>
Regards
Bejoy KS
On Thu, Jul 19, 2012 at 3:49 PM, iwannaplay games
<fu...@gmail.com> wrote:
> Hi,
>
> I am unable to use sqoop and want to load data in hdfs for testing,
> Is there any way by which i can load my csv or text file to hadoop
> file system directly without writing code in java
>
> Regards
> Prabhjot