You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by sakin cali <sa...@gmail.com> on 2012/05/24 08:12:22 UTC
Problem doing bulk load programmatically
Hi,
I am trying generate an HFile and load it to hbase programmatically.
(importtsv does not fit my requirements)
I have modified TestLoadIncrementalHFiles class and try to load an example
hfile.
But I get errors at the last step
--------------------------------------
java.io.IOException: java.io.IOException: Failed rename of *
file:/tmp/hbase/test/myfam/_tmp/mytable,1.top* to *
file:/tmp/hbase-hbase/hbase/mytable/742248ed8cda9fe0dce2f345149fa8d5/myfam/8964877901933124837
*
at org.apache.hadoop.hbase.regionserver.StoreFile.rename(StoreFile.java:512)
----------------------------------------
Although such an error occured, renaming seems succesfully because when I
checked their md5sum values, I saw that both have the same values...
I could not figure out the problem,
any ideas?
I have attached test code and logs.
I'am using cloudera's virtualbox vm image for testing environment
( 64 bit centos, version Cdh3u4 )
Re: Problem doing bulk load programmatically
Posted by sakin cali <sa...@gmail.com>.
thank you,
as said before I have one more question,
have any idea?
I have to get the execution time for bulk load
Simply I get the time before and after the doBulkLoad method,
But doBulkLoad spawns threads inside, Although I get a time difference, it
will not be real execution time...
What do you think?
-------------
start = System.currentTimeMillis();
loader.doBulkLoad(dir,table);
end = System.currentTimeMillis();
diff= end-start;
-------------
On 24 May 2012 15:02, shashwat shriparv <dw...@gmail.com> wrote:
> Super, i m happy that it solve your problem :) Cheer :)
>
> On Thu, May 24, 2012 at 3:10 PM, sakin cali <sa...@gmail.com> wrote:
>
> > Thank you,
> >
> > As said the problem was about hdfs,
> > putting dfs configuration directly into configuration object solved my
> > problem
> >
> > --------------
> > Configuration conf = HBaseConfiguration.create();
> > conf.set("fs.default.name", "hdfs://master:54310");
> >
> > ..
> > ..
> > FileSystem fs = FileSystem.get(conf);
> >
> > -------------
> >
> > I have one more question,
> > I have to get the execution time for bulk load
> > Simply I get the time before and after the doBulkLoad method,
> > But doBulkLoad spawns threads inside, Although I get a time difference,
> it
> > will not be real execution time...
> > What do you think?
> >
> > -------------
> > start = System.currentTimeMillis();
> > loader.doBulkLoad(dir,table);
> > end = System.currentTimeMillis();
> > diff= end-start;
> > -------------
> >
> > On 24 May 2012 11:13, shashwat shriparv <dw...@gmail.com>
> wrote:
> >
> > > your hbase is not able to communicate to hadoop, so before starting
> hbase
> > > put the core-site.xml, hdfs-site.xml and mapred-site.xml from hadoop
> > conf
> > > dir to hbase conf dir and then start hbase and try,, also you can copy
> > > hadoop core from hadoop to hbase library..
> > >
> > > Regards
> > > ∞
> > >
> > > Shashwat Shriparv
> > >
> > >
> > >
> > >
> > > On Thu, May 24, 2012 at 1:41 PM, shashwat shriparv <
> > > dwivedishashwat@gmail.com> wrote:
> > >
> > > > Check if hdfs is running fine and you can access it, and haddop is
> able
> > > to
> > > > access the file try putting that file on hdfs and then try.
> > > >
> > > > Regards
> > > >
> > > > ∞
> > > > Shashwat Shriparv
> > > >
> > > >
> > > >
> > > > On Thu, May 24, 2012 at 12:35 PM, sakin cali <sakinnncali@gmail.com
> > > >wrote:
> > > >
> > > >> Something new...
> > > >>
> > > >> When I start MiniDfsCluster and use its filesystem in my code,
> > > >> bulk load finishes successfully..
> > > >>
> > > >> dfs = new MiniDFSCluster(conf, 2, true, (String[]) null);
> > > >> 41.// set file system to the mini dfs just started up
> > > >> 42.FileSystem fs = dfs.getFileSystem();
> > > >>
> > > >> I think I have a problem accessing running HDFS ..
> > > >>
> > > >>
> > > >> On 24 May 2012 09:12, sakin cali <sa...@gmail.com> wrote:
> > > >>
> > > >> > Hi,
> > > >> >
> > > >> > I am trying generate an HFile and load it to hbase
> programmatically.
> > > >> > (importtsv does not fit my requirements)
> > > >> >
> > > >> > I have modified TestLoadIncrementalHFiles class and try to load an
> > > >> example
> > > >> > hfile.
> > > >> >
> > > >> > But I get errors at the last step
> > > >> >
> > > >> > --------------------------------------
> > > >> > java.io.IOException: java.io.IOException: Failed rename of *
> > > >> > file:/tmp/hbase/test/myfam/_tmp/mytable,1.top* to *
> > > >> >
> > > >>
> > >
> >
> file:/tmp/hbase-hbase/hbase/mytable/742248ed8cda9fe0dce2f345149fa8d5/myfam/8964877901933124837
> > > >> > *
> > > >> > at
> > > >> >
> > > >>
> > >
> org.apache.hadoop.hbase.regionserver.StoreFile.rename(StoreFile.java:512)
> > > >> > ----------------------------------------
> > > >> >
> > > >> > Although such an error occured, renaming seems succesfully because
> > > when
> > > >> I
> > > >> > checked their md5sum values, I saw that both have the same
> values...
> > > >> >
> > > >> > I could not figure out the problem,
> > > >> > any ideas?
> > > >> >
> > > >> > I have attached test code and logs.
> > > >> >
> > > >> > I'am using cloudera's virtualbox vm image for testing environment
> > > >> > ( 64 bit centos, version Cdh3u4 )
> > > >> >
> > > >> >
> > > >> >
> > > >>
> > > >
> > > >
> > > >
> > > > --
> > > >
> > > >
> > > > ∞
> > > > Shashwat Shriparv
> > > >
> > > >
> > > >
> > >
> > >
> > > --
> > >
> > >
> > > ∞
> > > Shashwat Shriparv
> > >
> >
>
>
>
> --
>
>
> ∞
> Shashwat Shriparv
>
Re: Problem doing bulk load programmatically
Posted by shashwat shriparv <dw...@gmail.com>.
Super, i m happy that it solve your problem :) Cheer :)
On Thu, May 24, 2012 at 3:10 PM, sakin cali <sa...@gmail.com> wrote:
> Thank you,
>
> As said the problem was about hdfs,
> putting dfs configuration directly into configuration object solved my
> problem
>
> --------------
> Configuration conf = HBaseConfiguration.create();
> conf.set("fs.default.name", "hdfs://master:54310");
>
> ..
> ..
> FileSystem fs = FileSystem.get(conf);
>
> -------------
>
> I have one more question,
> I have to get the execution time for bulk load
> Simply I get the time before and after the doBulkLoad method,
> But doBulkLoad spawns threads inside, Although I get a time difference, it
> will not be real execution time...
> What do you think?
>
> -------------
> start = System.currentTimeMillis();
> loader.doBulkLoad(dir,table);
> end = System.currentTimeMillis();
> diff= end-start;
> -------------
>
> On 24 May 2012 11:13, shashwat shriparv <dw...@gmail.com> wrote:
>
> > your hbase is not able to communicate to hadoop, so before starting hbase
> > put the core-site.xml, hdfs-site.xml and mapred-site.xml from hadoop
> conf
> > dir to hbase conf dir and then start hbase and try,, also you can copy
> > hadoop core from hadoop to hbase library..
> >
> > Regards
> > ∞
> >
> > Shashwat Shriparv
> >
> >
> >
> >
> > On Thu, May 24, 2012 at 1:41 PM, shashwat shriparv <
> > dwivedishashwat@gmail.com> wrote:
> >
> > > Check if hdfs is running fine and you can access it, and haddop is able
> > to
> > > access the file try putting that file on hdfs and then try.
> > >
> > > Regards
> > >
> > > ∞
> > > Shashwat Shriparv
> > >
> > >
> > >
> > > On Thu, May 24, 2012 at 12:35 PM, sakin cali <sakinnncali@gmail.com
> > >wrote:
> > >
> > >> Something new...
> > >>
> > >> When I start MiniDfsCluster and use its filesystem in my code,
> > >> bulk load finishes successfully..
> > >>
> > >> dfs = new MiniDFSCluster(conf, 2, true, (String[]) null);
> > >> 41.// set file system to the mini dfs just started up
> > >> 42.FileSystem fs = dfs.getFileSystem();
> > >>
> > >> I think I have a problem accessing running HDFS ..
> > >>
> > >>
> > >> On 24 May 2012 09:12, sakin cali <sa...@gmail.com> wrote:
> > >>
> > >> > Hi,
> > >> >
> > >> > I am trying generate an HFile and load it to hbase programmatically.
> > >> > (importtsv does not fit my requirements)
> > >> >
> > >> > I have modified TestLoadIncrementalHFiles class and try to load an
> > >> example
> > >> > hfile.
> > >> >
> > >> > But I get errors at the last step
> > >> >
> > >> > --------------------------------------
> > >> > java.io.IOException: java.io.IOException: Failed rename of *
> > >> > file:/tmp/hbase/test/myfam/_tmp/mytable,1.top* to *
> > >> >
> > >>
> >
> file:/tmp/hbase-hbase/hbase/mytable/742248ed8cda9fe0dce2f345149fa8d5/myfam/8964877901933124837
> > >> > *
> > >> > at
> > >> >
> > >>
> > org.apache.hadoop.hbase.regionserver.StoreFile.rename(StoreFile.java:512)
> > >> > ----------------------------------------
> > >> >
> > >> > Although such an error occured, renaming seems succesfully because
> > when
> > >> I
> > >> > checked their md5sum values, I saw that both have the same values...
> > >> >
> > >> > I could not figure out the problem,
> > >> > any ideas?
> > >> >
> > >> > I have attached test code and logs.
> > >> >
> > >> > I'am using cloudera's virtualbox vm image for testing environment
> > >> > ( 64 bit centos, version Cdh3u4 )
> > >> >
> > >> >
> > >> >
> > >>
> > >
> > >
> > >
> > > --
> > >
> > >
> > > ∞
> > > Shashwat Shriparv
> > >
> > >
> > >
> >
> >
> > --
> >
> >
> > ∞
> > Shashwat Shriparv
> >
>
--
∞
Shashwat Shriparv
Re: Problem doing bulk load programmatically
Posted by sakin cali <sa...@gmail.com>.
Harsh,
I run my test bulk load code from eclipse,
------------------------------------
> start = System.currentTimeMillis();
> loader.doBulkLoad(dir,table);
> end = System.currentTimeMillis();
> diff= end-start;
> System.out.println("result:"+diff);
------------------------------------
I want to get real "bulk load time",
I close my eclipse app when I saw "result" line on the console,
and when I checked table contents from hbase, it seems that all records are
inserted...
But if I do not close eclipse, application seems running (I think because
of the background threads) for some time...
So,
How can I get the real "bulk load time" ,
any idea?
On 24 May 2012 17:26, Harsh J <ha...@cloudera.com> wrote:
> Sakin,
>
> The bulk load method does run "hbase.loadincremental.threads.max"
> threads (configurable) with the default being dynamically equal to the
> number of CPUs available on the machine.
>
> On Thu, May 24, 2012 at 3:10 PM, sakin cali <sa...@gmail.com> wrote:
> > Thank you,
> >
> > As said the problem was about hdfs,
> > putting dfs configuration directly into configuration object solved my
> > problem
> >
> > --------------
> > Configuration conf = HBaseConfiguration.create();
> > conf.set("fs.default.name", "hdfs://master:54310");
> >
> > ..
> > ..
> > FileSystem fs = FileSystem.get(conf);
> >
> > -------------
> >
> > I have one more question,
> > I have to get the execution time for bulk load
> > Simply I get the time before and after the doBulkLoad method,
> > But doBulkLoad spawns threads inside, Although I get a time difference,
> it
> > will not be real execution time...
> > What do you think?
> >
> > -------------
> > start = System.currentTimeMillis();
> > loader.doBulkLoad(dir,table);
> > end = System.currentTimeMillis();
> > diff= end-start;
> > -------------
> >
> > On 24 May 2012 11:13, shashwat shriparv <dw...@gmail.com>
> wrote:
> >
> >> your hbase is not able to communicate to hadoop, so before starting
> hbase
> >> put the core-site.xml, hdfs-site.xml and mapred-site.xml from hadoop
> conf
> >> dir to hbase conf dir and then start hbase and try,, also you can copy
> >> hadoop core from hadoop to hbase library..
> >>
> >> Regards
> >> ∞
> >>
> >> Shashwat Shriparv
> >>
> >>
> >>
> >>
> >> On Thu, May 24, 2012 at 1:41 PM, shashwat shriparv <
> >> dwivedishashwat@gmail.com> wrote:
> >>
> >> > Check if hdfs is running fine and you can access it, and haddop is
> able
> >> to
> >> > access the file try putting that file on hdfs and then try.
> >> >
> >> > Regards
> >> >
> >> > ∞
> >> > Shashwat Shriparv
> >> >
> >> >
> >> >
> >> > On Thu, May 24, 2012 at 12:35 PM, sakin cali <sakinnncali@gmail.com
> >> >wrote:
> >> >
> >> >> Something new...
> >> >>
> >> >> When I start MiniDfsCluster and use its filesystem in my code,
> >> >> bulk load finishes successfully..
> >> >>
> >> >> dfs = new MiniDFSCluster(conf, 2, true, (String[]) null);
> >> >> 41.// set file system to the mini dfs just started up
> >> >> 42.FileSystem fs = dfs.getFileSystem();
> >> >>
> >> >> I think I have a problem accessing running HDFS ..
> >> >>
> >> >>
> >> >> On 24 May 2012 09:12, sakin cali <sa...@gmail.com> wrote:
> >> >>
> >> >> > Hi,
> >> >> >
> >> >> > I am trying generate an HFile and load it to hbase
> programmatically.
> >> >> > (importtsv does not fit my requirements)
> >> >> >
> >> >> > I have modified TestLoadIncrementalHFiles class and try to load an
> >> >> example
> >> >> > hfile.
> >> >> >
> >> >> > But I get errors at the last step
> >> >> >
> >> >> > --------------------------------------
> >> >> > java.io.IOException: java.io.IOException: Failed rename of *
> >> >> > file:/tmp/hbase/test/myfam/_tmp/mytable,1.top* to *
> >> >> >
> >> >>
> >>
> file:/tmp/hbase-hbase/hbase/mytable/742248ed8cda9fe0dce2f345149fa8d5/myfam/8964877901933124837
> >> >> > *
> >> >> > at
> >> >> >
> >> >>
> >>
> org.apache.hadoop.hbase.regionserver.StoreFile.rename(StoreFile.java:512)
> >> >> > ----------------------------------------
> >> >> >
> >> >> > Although such an error occured, renaming seems succesfully because
> >> when
> >> >> I
> >> >> > checked their md5sum values, I saw that both have the same
> values...
> >> >> >
> >> >> > I could not figure out the problem,
> >> >> > any ideas?
> >> >> >
> >> >> > I have attached test code and logs.
> >> >> >
> >> >> > I'am using cloudera's virtualbox vm image for testing environment
> >> >> > ( 64 bit centos, version Cdh3u4 )
> >> >> >
> >> >> >
> >> >> >
> >> >>
> >> >
> >> >
> >> >
> >> > --
> >> >
> >> >
> >> > ∞
> >> > Shashwat Shriparv
> >> >
> >> >
> >> >
> >>
> >>
> >> --
> >>
> >>
> >> ∞
> >> Shashwat Shriparv
> >>
>
>
>
> --
> Harsh J
>
Re: Problem doing bulk load programmatically
Posted by Harsh J <ha...@cloudera.com>.
Sakin,
The bulk load method does run "hbase.loadincremental.threads.max"
threads (configurable) with the default being dynamically equal to the
number of CPUs available on the machine.
On Thu, May 24, 2012 at 3:10 PM, sakin cali <sa...@gmail.com> wrote:
> Thank you,
>
> As said the problem was about hdfs,
> putting dfs configuration directly into configuration object solved my
> problem
>
> --------------
> Configuration conf = HBaseConfiguration.create();
> conf.set("fs.default.name", "hdfs://master:54310");
>
> ..
> ..
> FileSystem fs = FileSystem.get(conf);
>
> -------------
>
> I have one more question,
> I have to get the execution time for bulk load
> Simply I get the time before and after the doBulkLoad method,
> But doBulkLoad spawns threads inside, Although I get a time difference, it
> will not be real execution time...
> What do you think?
>
> -------------
> start = System.currentTimeMillis();
> loader.doBulkLoad(dir,table);
> end = System.currentTimeMillis();
> diff= end-start;
> -------------
>
> On 24 May 2012 11:13, shashwat shriparv <dw...@gmail.com> wrote:
>
>> your hbase is not able to communicate to hadoop, so before starting hbase
>> put the core-site.xml, hdfs-site.xml and mapred-site.xml from hadoop conf
>> dir to hbase conf dir and then start hbase and try,, also you can copy
>> hadoop core from hadoop to hbase library..
>>
>> Regards
>> ∞
>>
>> Shashwat Shriparv
>>
>>
>>
>>
>> On Thu, May 24, 2012 at 1:41 PM, shashwat shriparv <
>> dwivedishashwat@gmail.com> wrote:
>>
>> > Check if hdfs is running fine and you can access it, and haddop is able
>> to
>> > access the file try putting that file on hdfs and then try.
>> >
>> > Regards
>> >
>> > ∞
>> > Shashwat Shriparv
>> >
>> >
>> >
>> > On Thu, May 24, 2012 at 12:35 PM, sakin cali <sakinnncali@gmail.com
>> >wrote:
>> >
>> >> Something new...
>> >>
>> >> When I start MiniDfsCluster and use its filesystem in my code,
>> >> bulk load finishes successfully..
>> >>
>> >> dfs = new MiniDFSCluster(conf, 2, true, (String[]) null);
>> >> 41.// set file system to the mini dfs just started up
>> >> 42.FileSystem fs = dfs.getFileSystem();
>> >>
>> >> I think I have a problem accessing running HDFS ..
>> >>
>> >>
>> >> On 24 May 2012 09:12, sakin cali <sa...@gmail.com> wrote:
>> >>
>> >> > Hi,
>> >> >
>> >> > I am trying generate an HFile and load it to hbase programmatically.
>> >> > (importtsv does not fit my requirements)
>> >> >
>> >> > I have modified TestLoadIncrementalHFiles class and try to load an
>> >> example
>> >> > hfile.
>> >> >
>> >> > But I get errors at the last step
>> >> >
>> >> > --------------------------------------
>> >> > java.io.IOException: java.io.IOException: Failed rename of *
>> >> > file:/tmp/hbase/test/myfam/_tmp/mytable,1.top* to *
>> >> >
>> >>
>> file:/tmp/hbase-hbase/hbase/mytable/742248ed8cda9fe0dce2f345149fa8d5/myfam/8964877901933124837
>> >> > *
>> >> > at
>> >> >
>> >>
>> org.apache.hadoop.hbase.regionserver.StoreFile.rename(StoreFile.java:512)
>> >> > ----------------------------------------
>> >> >
>> >> > Although such an error occured, renaming seems succesfully because
>> when
>> >> I
>> >> > checked their md5sum values, I saw that both have the same values...
>> >> >
>> >> > I could not figure out the problem,
>> >> > any ideas?
>> >> >
>> >> > I have attached test code and logs.
>> >> >
>> >> > I'am using cloudera's virtualbox vm image for testing environment
>> >> > ( 64 bit centos, version Cdh3u4 )
>> >> >
>> >> >
>> >> >
>> >>
>> >
>> >
>> >
>> > --
>> >
>> >
>> > ∞
>> > Shashwat Shriparv
>> >
>> >
>> >
>>
>>
>> --
>>
>>
>> ∞
>> Shashwat Shriparv
>>
--
Harsh J
Re: Problem doing bulk load programmatically
Posted by sakin cali <sa...@gmail.com>.
Thank you,
As said the problem was about hdfs,
putting dfs configuration directly into configuration object solved my
problem
--------------
Configuration conf = HBaseConfiguration.create();
conf.set("fs.default.name", "hdfs://master:54310");
..
..
FileSystem fs = FileSystem.get(conf);
-------------
I have one more question,
I have to get the execution time for bulk load
Simply I get the time before and after the doBulkLoad method,
But doBulkLoad spawns threads inside, Although I get a time difference, it
will not be real execution time...
What do you think?
-------------
start = System.currentTimeMillis();
loader.doBulkLoad(dir,table);
end = System.currentTimeMillis();
diff= end-start;
-------------
On 24 May 2012 11:13, shashwat shriparv <dw...@gmail.com> wrote:
> your hbase is not able to communicate to hadoop, so before starting hbase
> put the core-site.xml, hdfs-site.xml and mapred-site.xml from hadoop conf
> dir to hbase conf dir and then start hbase and try,, also you can copy
> hadoop core from hadoop to hbase library..
>
> Regards
> ∞
>
> Shashwat Shriparv
>
>
>
>
> On Thu, May 24, 2012 at 1:41 PM, shashwat shriparv <
> dwivedishashwat@gmail.com> wrote:
>
> > Check if hdfs is running fine and you can access it, and haddop is able
> to
> > access the file try putting that file on hdfs and then try.
> >
> > Regards
> >
> > ∞
> > Shashwat Shriparv
> >
> >
> >
> > On Thu, May 24, 2012 at 12:35 PM, sakin cali <sakinnncali@gmail.com
> >wrote:
> >
> >> Something new...
> >>
> >> When I start MiniDfsCluster and use its filesystem in my code,
> >> bulk load finishes successfully..
> >>
> >> dfs = new MiniDFSCluster(conf, 2, true, (String[]) null);
> >> 41.// set file system to the mini dfs just started up
> >> 42.FileSystem fs = dfs.getFileSystem();
> >>
> >> I think I have a problem accessing running HDFS ..
> >>
> >>
> >> On 24 May 2012 09:12, sakin cali <sa...@gmail.com> wrote:
> >>
> >> > Hi,
> >> >
> >> > I am trying generate an HFile and load it to hbase programmatically.
> >> > (importtsv does not fit my requirements)
> >> >
> >> > I have modified TestLoadIncrementalHFiles class and try to load an
> >> example
> >> > hfile.
> >> >
> >> > But I get errors at the last step
> >> >
> >> > --------------------------------------
> >> > java.io.IOException: java.io.IOException: Failed rename of *
> >> > file:/tmp/hbase/test/myfam/_tmp/mytable,1.top* to *
> >> >
> >>
> file:/tmp/hbase-hbase/hbase/mytable/742248ed8cda9fe0dce2f345149fa8d5/myfam/8964877901933124837
> >> > *
> >> > at
> >> >
> >>
> org.apache.hadoop.hbase.regionserver.StoreFile.rename(StoreFile.java:512)
> >> > ----------------------------------------
> >> >
> >> > Although such an error occured, renaming seems succesfully because
> when
> >> I
> >> > checked their md5sum values, I saw that both have the same values...
> >> >
> >> > I could not figure out the problem,
> >> > any ideas?
> >> >
> >> > I have attached test code and logs.
> >> >
> >> > I'am using cloudera's virtualbox vm image for testing environment
> >> > ( 64 bit centos, version Cdh3u4 )
> >> >
> >> >
> >> >
> >>
> >
> >
> >
> > --
> >
> >
> > ∞
> > Shashwat Shriparv
> >
> >
> >
>
>
> --
>
>
> ∞
> Shashwat Shriparv
>
Re: Problem doing bulk load programmatically
Posted by shashwat shriparv <dw...@gmail.com>.
your hbase is not able to communicate to hadoop, so before starting hbase
put the core-site.xml, hdfs-site.xml and mapred-site.xml from hadoop conf
dir to hbase conf dir and then start hbase and try,, also you can copy
hadoop core from hadoop to hbase library..
Regards
∞
Shashwat Shriparv
On Thu, May 24, 2012 at 1:41 PM, shashwat shriparv <
dwivedishashwat@gmail.com> wrote:
> Check if hdfs is running fine and you can access it, and haddop is able to
> access the file try putting that file on hdfs and then try.
>
> Regards
>
> ∞
> Shashwat Shriparv
>
>
>
> On Thu, May 24, 2012 at 12:35 PM, sakin cali <sa...@gmail.com>wrote:
>
>> Something new...
>>
>> When I start MiniDfsCluster and use its filesystem in my code,
>> bulk load finishes successfully..
>>
>> dfs = new MiniDFSCluster(conf, 2, true, (String[]) null);
>> 41.// set file system to the mini dfs just started up
>> 42.FileSystem fs = dfs.getFileSystem();
>>
>> I think I have a problem accessing running HDFS ..
>>
>>
>> On 24 May 2012 09:12, sakin cali <sa...@gmail.com> wrote:
>>
>> > Hi,
>> >
>> > I am trying generate an HFile and load it to hbase programmatically.
>> > (importtsv does not fit my requirements)
>> >
>> > I have modified TestLoadIncrementalHFiles class and try to load an
>> example
>> > hfile.
>> >
>> > But I get errors at the last step
>> >
>> > --------------------------------------
>> > java.io.IOException: java.io.IOException: Failed rename of *
>> > file:/tmp/hbase/test/myfam/_tmp/mytable,1.top* to *
>> >
>> file:/tmp/hbase-hbase/hbase/mytable/742248ed8cda9fe0dce2f345149fa8d5/myfam/8964877901933124837
>> > *
>> > at
>> >
>> org.apache.hadoop.hbase.regionserver.StoreFile.rename(StoreFile.java:512)
>> > ----------------------------------------
>> >
>> > Although such an error occured, renaming seems succesfully because when
>> I
>> > checked their md5sum values, I saw that both have the same values...
>> >
>> > I could not figure out the problem,
>> > any ideas?
>> >
>> > I have attached test code and logs.
>> >
>> > I'am using cloudera's virtualbox vm image for testing environment
>> > ( 64 bit centos, version Cdh3u4 )
>> >
>> >
>> >
>>
>
>
>
> --
>
>
> ∞
> Shashwat Shriparv
>
>
>
--
∞
Shashwat Shriparv
Re: Problem doing bulk load programmatically
Posted by shashwat shriparv <dw...@gmail.com>.
Check if hdfs is running fine and you can access it, and haddop is able to
access the file try putting that file on hdfs and then try.
Regards
∞
Shashwat Shriparv
On Thu, May 24, 2012 at 12:35 PM, sakin cali <sa...@gmail.com> wrote:
> Something new...
>
> When I start MiniDfsCluster and use its filesystem in my code,
> bulk load finishes successfully..
>
> dfs = new MiniDFSCluster(conf, 2, true, (String[]) null);
> 41.// set file system to the mini dfs just started up
> 42.FileSystem fs = dfs.getFileSystem();
>
> I think I have a problem accessing running HDFS ..
>
>
> On 24 May 2012 09:12, sakin cali <sa...@gmail.com> wrote:
>
> > Hi,
> >
> > I am trying generate an HFile and load it to hbase programmatically.
> > (importtsv does not fit my requirements)
> >
> > I have modified TestLoadIncrementalHFiles class and try to load an
> example
> > hfile.
> >
> > But I get errors at the last step
> >
> > --------------------------------------
> > java.io.IOException: java.io.IOException: Failed rename of *
> > file:/tmp/hbase/test/myfam/_tmp/mytable,1.top* to *
> >
> file:/tmp/hbase-hbase/hbase/mytable/742248ed8cda9fe0dce2f345149fa8d5/myfam/8964877901933124837
> > *
> > at
> > org.apache.hadoop.hbase.regionserver.StoreFile.rename(StoreFile.java:512)
> > ----------------------------------------
> >
> > Although such an error occured, renaming seems succesfully because when I
> > checked their md5sum values, I saw that both have the same values...
> >
> > I could not figure out the problem,
> > any ideas?
> >
> > I have attached test code and logs.
> >
> > I'am using cloudera's virtualbox vm image for testing environment
> > ( 64 bit centos, version Cdh3u4 )
> >
> >
> >
>
--
∞
Shashwat Shriparv
Re: Problem doing bulk load programmatically
Posted by sakin cali <sa...@gmail.com>.
Something new...
When I start MiniDfsCluster and use its filesystem in my code,
bulk load finishes successfully..
dfs = new MiniDFSCluster(conf, 2, true, (String[]) null);
41.// set file system to the mini dfs just started up
42.FileSystem fs = dfs.getFileSystem();
I think I have a problem accessing running HDFS ..
On 24 May 2012 09:12, sakin cali <sa...@gmail.com> wrote:
> Hi,
>
> I am trying generate an HFile and load it to hbase programmatically.
> (importtsv does not fit my requirements)
>
> I have modified TestLoadIncrementalHFiles class and try to load an example
> hfile.
>
> But I get errors at the last step
>
> --------------------------------------
> java.io.IOException: java.io.IOException: Failed rename of *
> file:/tmp/hbase/test/myfam/_tmp/mytable,1.top* to *
> file:/tmp/hbase-hbase/hbase/mytable/742248ed8cda9fe0dce2f345149fa8d5/myfam/8964877901933124837
> *
> at
> org.apache.hadoop.hbase.regionserver.StoreFile.rename(StoreFile.java:512)
> ----------------------------------------
>
> Although such an error occured, renaming seems succesfully because when I
> checked their md5sum values, I saw that both have the same values...
>
> I could not figure out the problem,
> any ideas?
>
> I have attached test code and logs.
>
> I'am using cloudera's virtualbox vm image for testing environment
> ( 64 bit centos, version Cdh3u4 )
>
>
>