You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Jianshi Huang <ji...@gmail.com> on 2014/07/29 08:39:34 UTC
Completebulkload with namespace option?
I'm using completebulkload to load 500GB of data to a table (presplitted).
However, it reports the following errors:
Looks like completebulkload didn't recognize the namespace part
(namespace:table).
Is there an option to do it? I can't find one in Google...
Exception in thread "main" 14/07/28 23:32:19 INFO
mapreduce.LoadIncrementalHFiles: Trying to load
hfile=hdfs://xxx/vertices/PROP/f5cbf0965ff44cb8bdabd038e66485c3
first=dc595cfe#cust#1812199228741466242
last=dc68cedc#cust#2251647837553603393
java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.hbase.mapreduce.Driver.main(Driver.java:54)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
Caused by: java.lang.IllegalStateException:
java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative
path in absolute URI: grapple:vertices,37.bottom
at
org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.groupOrSplitPhase(LoadIncrementalHFiles.java:421)
at
org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.doBulkLoad(LoadIncrementalHFiles.java:291)
at
org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.run(LoadIncrementalHFiles.java:825)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
at
org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.main(LoadIncrementalHFiles.java:831)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72)
at org.apache.hadoop.util.ProgramDriver.run(ProgramDriver.java:145)
at
org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:153)
... 10 more
Caused by: java.lang.IllegalArgumentException: java.net.URISyntaxException:
Relative path in absolute URI: grapple:vertices,37.bottom
at org.apache.hadoop.fs.Path.initialize(Path.java:206)
at org.apache.hadoop.fs.Path.<init>(Path.java:172)
at org.apache.hadoop.fs.Path.<init>(Path.java:94)
at
org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.splitStoreFile(LoadIncrementalHFiles.java:450)
at
org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.groupOrSplit(LoadIncrementalHFiles.java:516)
at
org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles$2.call(LoadIncrementalHFiles.java:400)
at
org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles$2.call(LoadIncrementalHFiles.java:398)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:724)
Caused by: java.net.URISyntaxException: Relative path in absolute URI:
grapple:vertices,37.bottom
at java.net.URI.checkPath(URI.java:1804)
at java.net.URI.<init>(URI.java:752)
at org.apache.hadoop.fs.Path.initialize(Path.java:203)
... 10 more
--
Jianshi Huang
LinkedIn: jianshi
Twitter: @jshuang
Github & Blog: http://huangjs.github.com/
Re: Completebulkload with namespace option?
Posted by Jianshi Huang <ji...@gmail.com>.
Wow, thanks! :)
On Thu, Jul 31, 2014 at 10:07 AM, Ted Yu <yu...@gmail.com> wrote:
> Matteo acted very fast - this has been fixed by HBASE-11609
>
> Cheers
>
>
> On Wed, Jul 30, 2014 at 7:02 PM, Jianshi Huang <ji...@gmail.com>
> wrote:
>
> > Created a Jira issue.
> >
> > https://issues.apache.org/jira/browse/HBASE-11622
> >
> >
> > On Tue, Jul 29, 2014 at 11:46 PM, Bharath Vissapragada <
> > bharathv@cloudera.com> wrote:
> >
> > > Appears to be a bug. It should be TableName.valueOf(...) or something
> > > similar. Mind filing a jira?
> > >
> > >
> > > On Tue, Jul 29, 2014 at 12:22 PM, Jianshi Huang <
> jianshi.huang@gmail.com
> > >
> > > wrote:
> > >
> > > > I see why, looking at the source code of LoadIncrementalHFiles.java,
> it
> > > > seems the temporary path created for splitting will contain ':',
> > > >
> > > > The error part should be this:
> > > > String uniqueName = getUniqueName(table.getName());
> > > > HColumnDescriptor familyDesc =
> > > > table.getTableDescriptor().getFamily(item.family);
> > > > Path botOut = new Path(tmpDir, uniqueName + ".bottom");
> > > > Path topOut = new Path(tmpDir, uniqueName + ".top");
> > > > splitStoreFile(getConf(), hfilePath, familyDesc, splitKey,
> > > > botOut, topOut);
> > > >
> > > > uniqueName will be "namespce:table" so new Path will fail.
> > > >
> > > > A bug right?
> > > >
> > > > Jianshi
> > > >
> > > >
> > > > On Tue, Jul 29, 2014 at 2:42 PM, Jianshi Huang <
> > jianshi.huang@gmail.com>
> > > > wrote:
> > > >
> > > > > I'm using hbase 0.98 with HDP 2.1.
> > > > >
> > > > >
> > > > > On Tue, Jul 29, 2014 at 2:39 PM, Jianshi Huang <
> > > jianshi.huang@gmail.com>
> > > > > wrote:
> > > > >
> > > > >> I'm using completebulkload to load 500GB of data to a table
> > > > >> (presplitted). However, it reports the following errors:
> > > > >>
> > > > >> Looks like completebulkload didn't recognize the namespace part
> > > > >> (namespace:table).
> > > > >>
> > > > >> Is there an option to do it? I can't find one in Google...
> > > > >>
> > > > >> Exception in thread "main" 14/07/28 23:32:19 INFO
> > > > >> mapreduce.LoadIncrementalHFiles: Trying to load
> > > > >> hfile=hdfs://xxx/vertices/PROP/f5cbf0965ff44cb8bdabd038e66485c3
> > > > >> first=dc595cfe#cust#1812199228741466242
> > > > >> last=dc68cedc#cust#2251647837553603393
> > > > >> java.lang.reflect.InvocationTargetException
> > > > >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> > Method)
> > > > >> at
> > > > >>
> > > >
> > >
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> > > > >> at
> > > > >>
> > > >
> > >
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > > > >> at java.lang.reflect.Method.invoke(Method.java:606)
> > > > >> at
> > > org.apache.hadoop.hbase.mapreduce.Driver.main(Driver.java:54)
> > > > >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> > Method)
> > > > >> at
> > > > >>
> > > >
> > >
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> > > > >> at
> > > > >>
> > > >
> > >
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > > > >> at java.lang.reflect.Method.invoke(Method.java:606)
> > > > >> at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
> > > > >> Caused by: java.lang.IllegalStateException:
> > > > >> java.lang.IllegalArgumentException: java.net.URISyntaxException:
> > > > Relative
> > > > >> path in absolute URI: grapple:vertices,37.bottom
> > > > >> at
> > > > >>
> > > >
> > >
> >
> org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.groupOrSplitPhase(LoadIncrementalHFiles.java:421)
> > > > >> at
> > > > >>
> > > >
> > >
> >
> org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.doBulkLoad(LoadIncrementalHFiles.java:291)
> > > > >> at
> > > > >>
> > > >
> > >
> >
> org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.run(LoadIncrementalHFiles.java:825)
> > > > >> at
> org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> > > > >> at
> org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
> > > > >> at
> > > > >>
> > > >
> > >
> >
> org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.main(LoadIncrementalHFiles.java:831)
> > > > >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> > Method)
> > > > >> at
> > > > >>
> > > >
> > >
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> > > > >> at
> > > > >>
> > > >
> > >
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > > > >> at java.lang.reflect.Method.invoke(Method.java:606)
> > > > >> at
> > > > >>
> > > >
> > >
> >
> org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72)
> > > > >> at
> > > > >> org.apache.hadoop.util.ProgramDriver.run(ProgramDriver.java:145)
> > > > >> at
> > > > >>
> org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:153)
> > > > >> ... 10 more
> > > > >> Caused by: java.lang.IllegalArgumentException:
> > > > >> java.net.URISyntaxException: Relative path in absolute URI:
> > > > >> grapple:vertices,37.bottom
> > > > >> at org.apache.hadoop.fs.Path.initialize(Path.java:206)
> > > > >> at org.apache.hadoop.fs.Path.<init>(Path.java:172)
> > > > >> at org.apache.hadoop.fs.Path.<init>(Path.java:94)
> > > > >> at
> > > > >>
> > > >
> > >
> >
> org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.splitStoreFile(LoadIncrementalHFiles.java:450)
> > > > >> at
> > > > >>
> > > >
> > >
> >
> org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.groupOrSplit(LoadIncrementalHFiles.java:516)
> > > > >> at
> > > > >>
> > > >
> > >
> >
> org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles$2.call(LoadIncrementalHFiles.java:400)
> > > > >> at
> > > > >>
> > > >
> > >
> >
> org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles$2.call(LoadIncrementalHFiles.java:398)
> > > > >> at
> java.util.concurrent.FutureTask.run(FutureTask.java:262)
> > > > >> at
> > > > >>
> > > >
> > >
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> > > > >> at
> > > > >>
> > > >
> > >
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> > > > >> at java.lang.Thread.run(Thread.java:724)
> > > > >> Caused by: java.net.URISyntaxException: Relative path in absolute
> > URI:
> > > > >> grapple:vertices,37.bottom
> > > > >> at java.net.URI.checkPath(URI.java:1804)
> > > > >> at java.net.URI.<init>(URI.java:752)
> > > > >> at org.apache.hadoop.fs.Path.initialize(Path.java:203)
> > > > >> ... 10 more
> > > > >>
> > > > >> --
> > > > >> Jianshi Huang
> > > > >>
> > > > >> LinkedIn: jianshi
> > > > >> Twitter: @jshuang
> > > > >> Github & Blog: http://huangjs.github.com/
> > > > >>
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Jianshi Huang
> > > > >
> > > > > LinkedIn: jianshi
> > > > > Twitter: @jshuang
> > > > > Github & Blog: http://huangjs.github.com/
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Jianshi Huang
> > > >
> > > > LinkedIn: jianshi
> > > > Twitter: @jshuang
> > > > Github & Blog: http://huangjs.github.com/
> > > >
> > >
> > >
> > >
> > > --
> > > Bharath Vissapragada
> > > <http://www.cloudera.com>
> > >
> >
> >
> >
> > --
> > Jianshi Huang
> >
> > LinkedIn: jianshi
> > Twitter: @jshuang
> > Github & Blog: http://huangjs.github.com/
> >
>
--
Jianshi Huang
LinkedIn: jianshi
Twitter: @jshuang
Github & Blog: http://huangjs.github.com/
Re: Completebulkload with namespace option?
Posted by Ted Yu <yu...@gmail.com>.
Matteo acted very fast - this has been fixed by HBASE-11609
Cheers
On Wed, Jul 30, 2014 at 7:02 PM, Jianshi Huang <ji...@gmail.com>
wrote:
> Created a Jira issue.
>
> https://issues.apache.org/jira/browse/HBASE-11622
>
>
> On Tue, Jul 29, 2014 at 11:46 PM, Bharath Vissapragada <
> bharathv@cloudera.com> wrote:
>
> > Appears to be a bug. It should be TableName.valueOf(...) or something
> > similar. Mind filing a jira?
> >
> >
> > On Tue, Jul 29, 2014 at 12:22 PM, Jianshi Huang <jianshi.huang@gmail.com
> >
> > wrote:
> >
> > > I see why, looking at the source code of LoadIncrementalHFiles.java, it
> > > seems the temporary path created for splitting will contain ':',
> > >
> > > The error part should be this:
> > > String uniqueName = getUniqueName(table.getName());
> > > HColumnDescriptor familyDesc =
> > > table.getTableDescriptor().getFamily(item.family);
> > > Path botOut = new Path(tmpDir, uniqueName + ".bottom");
> > > Path topOut = new Path(tmpDir, uniqueName + ".top");
> > > splitStoreFile(getConf(), hfilePath, familyDesc, splitKey,
> > > botOut, topOut);
> > >
> > > uniqueName will be "namespce:table" so new Path will fail.
> > >
> > > A bug right?
> > >
> > > Jianshi
> > >
> > >
> > > On Tue, Jul 29, 2014 at 2:42 PM, Jianshi Huang <
> jianshi.huang@gmail.com>
> > > wrote:
> > >
> > > > I'm using hbase 0.98 with HDP 2.1.
> > > >
> > > >
> > > > On Tue, Jul 29, 2014 at 2:39 PM, Jianshi Huang <
> > jianshi.huang@gmail.com>
> > > > wrote:
> > > >
> > > >> I'm using completebulkload to load 500GB of data to a table
> > > >> (presplitted). However, it reports the following errors:
> > > >>
> > > >> Looks like completebulkload didn't recognize the namespace part
> > > >> (namespace:table).
> > > >>
> > > >> Is there an option to do it? I can't find one in Google...
> > > >>
> > > >> Exception in thread "main" 14/07/28 23:32:19 INFO
> > > >> mapreduce.LoadIncrementalHFiles: Trying to load
> > > >> hfile=hdfs://xxx/vertices/PROP/f5cbf0965ff44cb8bdabd038e66485c3
> > > >> first=dc595cfe#cust#1812199228741466242
> > > >> last=dc68cedc#cust#2251647837553603393
> > > >> java.lang.reflect.InvocationTargetException
> > > >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
> > > >> at
> > > >>
> > >
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> > > >> at
> > > >>
> > >
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > > >> at java.lang.reflect.Method.invoke(Method.java:606)
> > > >> at
> > org.apache.hadoop.hbase.mapreduce.Driver.main(Driver.java:54)
> > > >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
> > > >> at
> > > >>
> > >
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> > > >> at
> > > >>
> > >
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > > >> at java.lang.reflect.Method.invoke(Method.java:606)
> > > >> at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
> > > >> Caused by: java.lang.IllegalStateException:
> > > >> java.lang.IllegalArgumentException: java.net.URISyntaxException:
> > > Relative
> > > >> path in absolute URI: grapple:vertices,37.bottom
> > > >> at
> > > >>
> > >
> >
> org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.groupOrSplitPhase(LoadIncrementalHFiles.java:421)
> > > >> at
> > > >>
> > >
> >
> org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.doBulkLoad(LoadIncrementalHFiles.java:291)
> > > >> at
> > > >>
> > >
> >
> org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.run(LoadIncrementalHFiles.java:825)
> > > >> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> > > >> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
> > > >> at
> > > >>
> > >
> >
> org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.main(LoadIncrementalHFiles.java:831)
> > > >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
> > > >> at
> > > >>
> > >
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> > > >> at
> > > >>
> > >
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > > >> at java.lang.reflect.Method.invoke(Method.java:606)
> > > >> at
> > > >>
> > >
> >
> org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72)
> > > >> at
> > > >> org.apache.hadoop.util.ProgramDriver.run(ProgramDriver.java:145)
> > > >> at
> > > >> org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:153)
> > > >> ... 10 more
> > > >> Caused by: java.lang.IllegalArgumentException:
> > > >> java.net.URISyntaxException: Relative path in absolute URI:
> > > >> grapple:vertices,37.bottom
> > > >> at org.apache.hadoop.fs.Path.initialize(Path.java:206)
> > > >> at org.apache.hadoop.fs.Path.<init>(Path.java:172)
> > > >> at org.apache.hadoop.fs.Path.<init>(Path.java:94)
> > > >> at
> > > >>
> > >
> >
> org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.splitStoreFile(LoadIncrementalHFiles.java:450)
> > > >> at
> > > >>
> > >
> >
> org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.groupOrSplit(LoadIncrementalHFiles.java:516)
> > > >> at
> > > >>
> > >
> >
> org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles$2.call(LoadIncrementalHFiles.java:400)
> > > >> at
> > > >>
> > >
> >
> org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles$2.call(LoadIncrementalHFiles.java:398)
> > > >> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> > > >> at
> > > >>
> > >
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> > > >> at
> > > >>
> > >
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> > > >> at java.lang.Thread.run(Thread.java:724)
> > > >> Caused by: java.net.URISyntaxException: Relative path in absolute
> URI:
> > > >> grapple:vertices,37.bottom
> > > >> at java.net.URI.checkPath(URI.java:1804)
> > > >> at java.net.URI.<init>(URI.java:752)
> > > >> at org.apache.hadoop.fs.Path.initialize(Path.java:203)
> > > >> ... 10 more
> > > >>
> > > >> --
> > > >> Jianshi Huang
> > > >>
> > > >> LinkedIn: jianshi
> > > >> Twitter: @jshuang
> > > >> Github & Blog: http://huangjs.github.com/
> > > >>
> > > >
> > > >
> > > >
> > > > --
> > > > Jianshi Huang
> > > >
> > > > LinkedIn: jianshi
> > > > Twitter: @jshuang
> > > > Github & Blog: http://huangjs.github.com/
> > > >
> > >
> > >
> > >
> > > --
> > > Jianshi Huang
> > >
> > > LinkedIn: jianshi
> > > Twitter: @jshuang
> > > Github & Blog: http://huangjs.github.com/
> > >
> >
> >
> >
> > --
> > Bharath Vissapragada
> > <http://www.cloudera.com>
> >
>
>
>
> --
> Jianshi Huang
>
> LinkedIn: jianshi
> Twitter: @jshuang
> Github & Blog: http://huangjs.github.com/
>
Re: Completebulkload with namespace option?
Posted by Jianshi Huang <ji...@gmail.com>.
Created a Jira issue.
https://issues.apache.org/jira/browse/HBASE-11622
On Tue, Jul 29, 2014 at 11:46 PM, Bharath Vissapragada <
bharathv@cloudera.com> wrote:
> Appears to be a bug. It should be TableName.valueOf(...) or something
> similar. Mind filing a jira?
>
>
> On Tue, Jul 29, 2014 at 12:22 PM, Jianshi Huang <ji...@gmail.com>
> wrote:
>
> > I see why, looking at the source code of LoadIncrementalHFiles.java, it
> > seems the temporary path created for splitting will contain ':',
> >
> > The error part should be this:
> > String uniqueName = getUniqueName(table.getName());
> > HColumnDescriptor familyDesc =
> > table.getTableDescriptor().getFamily(item.family);
> > Path botOut = new Path(tmpDir, uniqueName + ".bottom");
> > Path topOut = new Path(tmpDir, uniqueName + ".top");
> > splitStoreFile(getConf(), hfilePath, familyDesc, splitKey,
> > botOut, topOut);
> >
> > uniqueName will be "namespce:table" so new Path will fail.
> >
> > A bug right?
> >
> > Jianshi
> >
> >
> > On Tue, Jul 29, 2014 at 2:42 PM, Jianshi Huang <ji...@gmail.com>
> > wrote:
> >
> > > I'm using hbase 0.98 with HDP 2.1.
> > >
> > >
> > > On Tue, Jul 29, 2014 at 2:39 PM, Jianshi Huang <
> jianshi.huang@gmail.com>
> > > wrote:
> > >
> > >> I'm using completebulkload to load 500GB of data to a table
> > >> (presplitted). However, it reports the following errors:
> > >>
> > >> Looks like completebulkload didn't recognize the namespace part
> > >> (namespace:table).
> > >>
> > >> Is there an option to do it? I can't find one in Google...
> > >>
> > >> Exception in thread "main" 14/07/28 23:32:19 INFO
> > >> mapreduce.LoadIncrementalHFiles: Trying to load
> > >> hfile=hdfs://xxx/vertices/PROP/f5cbf0965ff44cb8bdabd038e66485c3
> > >> first=dc595cfe#cust#1812199228741466242
> > >> last=dc68cedc#cust#2251647837553603393
> > >> java.lang.reflect.InvocationTargetException
> > >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > >> at
> > >>
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> > >> at
> > >>
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > >> at java.lang.reflect.Method.invoke(Method.java:606)
> > >> at
> org.apache.hadoop.hbase.mapreduce.Driver.main(Driver.java:54)
> > >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > >> at
> > >>
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> > >> at
> > >>
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > >> at java.lang.reflect.Method.invoke(Method.java:606)
> > >> at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
> > >> Caused by: java.lang.IllegalStateException:
> > >> java.lang.IllegalArgumentException: java.net.URISyntaxException:
> > Relative
> > >> path in absolute URI: grapple:vertices,37.bottom
> > >> at
> > >>
> >
> org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.groupOrSplitPhase(LoadIncrementalHFiles.java:421)
> > >> at
> > >>
> >
> org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.doBulkLoad(LoadIncrementalHFiles.java:291)
> > >> at
> > >>
> >
> org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.run(LoadIncrementalHFiles.java:825)
> > >> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> > >> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
> > >> at
> > >>
> >
> org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.main(LoadIncrementalHFiles.java:831)
> > >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > >> at
> > >>
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> > >> at
> > >>
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > >> at java.lang.reflect.Method.invoke(Method.java:606)
> > >> at
> > >>
> >
> org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72)
> > >> at
> > >> org.apache.hadoop.util.ProgramDriver.run(ProgramDriver.java:145)
> > >> at
> > >> org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:153)
> > >> ... 10 more
> > >> Caused by: java.lang.IllegalArgumentException:
> > >> java.net.URISyntaxException: Relative path in absolute URI:
> > >> grapple:vertices,37.bottom
> > >> at org.apache.hadoop.fs.Path.initialize(Path.java:206)
> > >> at org.apache.hadoop.fs.Path.<init>(Path.java:172)
> > >> at org.apache.hadoop.fs.Path.<init>(Path.java:94)
> > >> at
> > >>
> >
> org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.splitStoreFile(LoadIncrementalHFiles.java:450)
> > >> at
> > >>
> >
> org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.groupOrSplit(LoadIncrementalHFiles.java:516)
> > >> at
> > >>
> >
> org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles$2.call(LoadIncrementalHFiles.java:400)
> > >> at
> > >>
> >
> org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles$2.call(LoadIncrementalHFiles.java:398)
> > >> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> > >> at
> > >>
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> > >> at
> > >>
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> > >> at java.lang.Thread.run(Thread.java:724)
> > >> Caused by: java.net.URISyntaxException: Relative path in absolute URI:
> > >> grapple:vertices,37.bottom
> > >> at java.net.URI.checkPath(URI.java:1804)
> > >> at java.net.URI.<init>(URI.java:752)
> > >> at org.apache.hadoop.fs.Path.initialize(Path.java:203)
> > >> ... 10 more
> > >>
> > >> --
> > >> Jianshi Huang
> > >>
> > >> LinkedIn: jianshi
> > >> Twitter: @jshuang
> > >> Github & Blog: http://huangjs.github.com/
> > >>
> > >
> > >
> > >
> > > --
> > > Jianshi Huang
> > >
> > > LinkedIn: jianshi
> > > Twitter: @jshuang
> > > Github & Blog: http://huangjs.github.com/
> > >
> >
> >
> >
> > --
> > Jianshi Huang
> >
> > LinkedIn: jianshi
> > Twitter: @jshuang
> > Github & Blog: http://huangjs.github.com/
> >
>
>
>
> --
> Bharath Vissapragada
> <http://www.cloudera.com>
>
--
Jianshi Huang
LinkedIn: jianshi
Twitter: @jshuang
Github & Blog: http://huangjs.github.com/
Re: Completebulkload with namespace option?
Posted by Matteo Bertozzi <th...@gmail.com>.
we don't need the name to be related to the table name.
We can replace the getUniqueName() using something like this
String getUniqueName(TableName tableName) {
String name = UUID.randomUUID().toString().replaceAll("-", "") +
"," + regionCount.incrementAndGet();
return name;
}
Matteo
On Tue, Jul 29, 2014 at 4:52 PM, Ted Yu <yu...@gmail.com> wrote:
> TableName.valueOf() would return an instance of TableName whose String form
> would contain ':'.
>
> FYI
>
>
> On Tue, Jul 29, 2014 at 8:46 AM, Bharath Vissapragada <
> bharathv@cloudera.com
> > wrote:
>
> > Appears to be a bug. It should be TableName.valueOf(...) or something
> > similar. Mind filing a jira?
> >
> >
> > On Tue, Jul 29, 2014 at 12:22 PM, Jianshi Huang <jianshi.huang@gmail.com
> >
> > wrote:
> >
> > > I see why, looking at the source code of LoadIncrementalHFiles.java, it
> > > seems the temporary path created for splitting will contain ':',
> > >
> > > The error part should be this:
> > > String uniqueName = getUniqueName(table.getName());
> > > HColumnDescriptor familyDesc =
> > > table.getTableDescriptor().getFamily(item.family);
> > > Path botOut = new Path(tmpDir, uniqueName + ".bottom");
> > > Path topOut = new Path(tmpDir, uniqueName + ".top");
> > > splitStoreFile(getConf(), hfilePath, familyDesc, splitKey,
> > > botOut, topOut);
> > >
> > > uniqueName will be "namespce:table" so new Path will fail.
> > >
> > > A bug right?
> > >
> > > Jianshi
> > >
> > >
> > > On Tue, Jul 29, 2014 at 2:42 PM, Jianshi Huang <
> jianshi.huang@gmail.com>
> > > wrote:
> > >
> > > > I'm using hbase 0.98 with HDP 2.1.
> > > >
> > > >
> > > > On Tue, Jul 29, 2014 at 2:39 PM, Jianshi Huang <
> > jianshi.huang@gmail.com>
> > > > wrote:
> > > >
> > > >> I'm using completebulkload to load 500GB of data to a table
> > > >> (presplitted). However, it reports the following errors:
> > > >>
> > > >> Looks like completebulkload didn't recognize the namespace part
> > > >> (namespace:table).
> > > >>
> > > >> Is there an option to do it? I can't find one in Google...
> > > >>
> > > >> Exception in thread "main" 14/07/28 23:32:19 INFO
> > > >> mapreduce.LoadIncrementalHFiles: Trying to load
> > > >> hfile=hdfs://xxx/vertices/PROP/f5cbf0965ff44cb8bdabd038e66485c3
> > > >> first=dc595cfe#cust#1812199228741466242
> > > >> last=dc68cedc#cust#2251647837553603393
> > > >> java.lang.reflect.InvocationTargetException
> > > >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
> > > >> at
> > > >>
> > >
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> > > >> at
> > > >>
> > >
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > > >> at java.lang.reflect.Method.invoke(Method.java:606)
> > > >> at
> > org.apache.hadoop.hbase.mapreduce.Driver.main(Driver.java:54)
> > > >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
> > > >> at
> > > >>
> > >
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> > > >> at
> > > >>
> > >
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > > >> at java.lang.reflect.Method.invoke(Method.java:606)
> > > >> at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
> > > >> Caused by: java.lang.IllegalStateException:
> > > >> java.lang.IllegalArgumentException: java.net.URISyntaxException:
> > > Relative
> > > >> path in absolute URI: grapple:vertices,37.bottom
> > > >> at
> > > >>
> > >
> >
> org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.groupOrSplitPhase(LoadIncrementalHFiles.java:421)
> > > >> at
> > > >>
> > >
> >
> org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.doBulkLoad(LoadIncrementalHFiles.java:291)
> > > >> at
> > > >>
> > >
> >
> org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.run(LoadIncrementalHFiles.java:825)
> > > >> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> > > >> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
> > > >> at
> > > >>
> > >
> >
> org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.main(LoadIncrementalHFiles.java:831)
> > > >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
> > > >> at
> > > >>
> > >
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> > > >> at
> > > >>
> > >
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > > >> at java.lang.reflect.Method.invoke(Method.java:606)
> > > >> at
> > > >>
> > >
> >
> org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72)
> > > >> at
> > > >> org.apache.hadoop.util.ProgramDriver.run(ProgramDriver.java:145)
> > > >> at
> > > >> org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:153)
> > > >> ... 10 more
> > > >> Caused by: java.lang.IllegalArgumentException:
> > > >> java.net.URISyntaxException: Relative path in absolute URI:
> > > >> grapple:vertices,37.bottom
> > > >> at org.apache.hadoop.fs.Path.initialize(Path.java:206)
> > > >> at org.apache.hadoop.fs.Path.<init>(Path.java:172)
> > > >> at org.apache.hadoop.fs.Path.<init>(Path.java:94)
> > > >> at
> > > >>
> > >
> >
> org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.splitStoreFile(LoadIncrementalHFiles.java:450)
> > > >> at
> > > >>
> > >
> >
> org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.groupOrSplit(LoadIncrementalHFiles.java:516)
> > > >> at
> > > >>
> > >
> >
> org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles$2.call(LoadIncrementalHFiles.java:400)
> > > >> at
> > > >>
> > >
> >
> org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles$2.call(LoadIncrementalHFiles.java:398)
> > > >> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> > > >> at
> > > >>
> > >
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> > > >> at
> > > >>
> > >
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> > > >> at java.lang.Thread.run(Thread.java:724)
> > > >> Caused by: java.net.URISyntaxException: Relative path in absolute
> URI:
> > > >> grapple:vertices,37.bottom
> > > >> at java.net.URI.checkPath(URI.java:1804)
> > > >> at java.net.URI.<init>(URI.java:752)
> > > >> at org.apache.hadoop.fs.Path.initialize(Path.java:203)
> > > >> ... 10 more
> > > >>
> > > >> --
> > > >> Jianshi Huang
> > > >>
> > > >> LinkedIn: jianshi
> > > >> Twitter: @jshuang
> > > >> Github & Blog: http://huangjs.github.com/
> > > >>
> > > >
> > > >
> > > >
> > > > --
> > > > Jianshi Huang
> > > >
> > > > LinkedIn: jianshi
> > > > Twitter: @jshuang
> > > > Github & Blog: http://huangjs.github.com/
> > > >
> > >
> > >
> > >
> > > --
> > > Jianshi Huang
> > >
> > > LinkedIn: jianshi
> > > Twitter: @jshuang
> > > Github & Blog: http://huangjs.github.com/
> > >
> >
> >
> >
> > --
> > Bharath Vissapragada
> > <http://www.cloudera.com>
> >
>
Re: Completebulkload with namespace option?
Posted by Ted Yu <yu...@gmail.com>.
TableName.valueOf() would return an instance of TableName whose String form
would contain ':'.
FYI
On Tue, Jul 29, 2014 at 8:46 AM, Bharath Vissapragada <bharathv@cloudera.com
> wrote:
> Appears to be a bug. It should be TableName.valueOf(...) or something
> similar. Mind filing a jira?
>
>
> On Tue, Jul 29, 2014 at 12:22 PM, Jianshi Huang <ji...@gmail.com>
> wrote:
>
> > I see why, looking at the source code of LoadIncrementalHFiles.java, it
> > seems the temporary path created for splitting will contain ':',
> >
> > The error part should be this:
> > String uniqueName = getUniqueName(table.getName());
> > HColumnDescriptor familyDesc =
> > table.getTableDescriptor().getFamily(item.family);
> > Path botOut = new Path(tmpDir, uniqueName + ".bottom");
> > Path topOut = new Path(tmpDir, uniqueName + ".top");
> > splitStoreFile(getConf(), hfilePath, familyDesc, splitKey,
> > botOut, topOut);
> >
> > uniqueName will be "namespce:table" so new Path will fail.
> >
> > A bug right?
> >
> > Jianshi
> >
> >
> > On Tue, Jul 29, 2014 at 2:42 PM, Jianshi Huang <ji...@gmail.com>
> > wrote:
> >
> > > I'm using hbase 0.98 with HDP 2.1.
> > >
> > >
> > > On Tue, Jul 29, 2014 at 2:39 PM, Jianshi Huang <
> jianshi.huang@gmail.com>
> > > wrote:
> > >
> > >> I'm using completebulkload to load 500GB of data to a table
> > >> (presplitted). However, it reports the following errors:
> > >>
> > >> Looks like completebulkload didn't recognize the namespace part
> > >> (namespace:table).
> > >>
> > >> Is there an option to do it? I can't find one in Google...
> > >>
> > >> Exception in thread "main" 14/07/28 23:32:19 INFO
> > >> mapreduce.LoadIncrementalHFiles: Trying to load
> > >> hfile=hdfs://xxx/vertices/PROP/f5cbf0965ff44cb8bdabd038e66485c3
> > >> first=dc595cfe#cust#1812199228741466242
> > >> last=dc68cedc#cust#2251647837553603393
> > >> java.lang.reflect.InvocationTargetException
> > >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > >> at
> > >>
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> > >> at
> > >>
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > >> at java.lang.reflect.Method.invoke(Method.java:606)
> > >> at
> org.apache.hadoop.hbase.mapreduce.Driver.main(Driver.java:54)
> > >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > >> at
> > >>
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> > >> at
> > >>
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > >> at java.lang.reflect.Method.invoke(Method.java:606)
> > >> at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
> > >> Caused by: java.lang.IllegalStateException:
> > >> java.lang.IllegalArgumentException: java.net.URISyntaxException:
> > Relative
> > >> path in absolute URI: grapple:vertices,37.bottom
> > >> at
> > >>
> >
> org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.groupOrSplitPhase(LoadIncrementalHFiles.java:421)
> > >> at
> > >>
> >
> org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.doBulkLoad(LoadIncrementalHFiles.java:291)
> > >> at
> > >>
> >
> org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.run(LoadIncrementalHFiles.java:825)
> > >> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> > >> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
> > >> at
> > >>
> >
> org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.main(LoadIncrementalHFiles.java:831)
> > >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > >> at
> > >>
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> > >> at
> > >>
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > >> at java.lang.reflect.Method.invoke(Method.java:606)
> > >> at
> > >>
> >
> org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72)
> > >> at
> > >> org.apache.hadoop.util.ProgramDriver.run(ProgramDriver.java:145)
> > >> at
> > >> org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:153)
> > >> ... 10 more
> > >> Caused by: java.lang.IllegalArgumentException:
> > >> java.net.URISyntaxException: Relative path in absolute URI:
> > >> grapple:vertices,37.bottom
> > >> at org.apache.hadoop.fs.Path.initialize(Path.java:206)
> > >> at org.apache.hadoop.fs.Path.<init>(Path.java:172)
> > >> at org.apache.hadoop.fs.Path.<init>(Path.java:94)
> > >> at
> > >>
> >
> org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.splitStoreFile(LoadIncrementalHFiles.java:450)
> > >> at
> > >>
> >
> org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.groupOrSplit(LoadIncrementalHFiles.java:516)
> > >> at
> > >>
> >
> org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles$2.call(LoadIncrementalHFiles.java:400)
> > >> at
> > >>
> >
> org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles$2.call(LoadIncrementalHFiles.java:398)
> > >> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> > >> at
> > >>
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> > >> at
> > >>
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> > >> at java.lang.Thread.run(Thread.java:724)
> > >> Caused by: java.net.URISyntaxException: Relative path in absolute URI:
> > >> grapple:vertices,37.bottom
> > >> at java.net.URI.checkPath(URI.java:1804)
> > >> at java.net.URI.<init>(URI.java:752)
> > >> at org.apache.hadoop.fs.Path.initialize(Path.java:203)
> > >> ... 10 more
> > >>
> > >> --
> > >> Jianshi Huang
> > >>
> > >> LinkedIn: jianshi
> > >> Twitter: @jshuang
> > >> Github & Blog: http://huangjs.github.com/
> > >>
> > >
> > >
> > >
> > > --
> > > Jianshi Huang
> > >
> > > LinkedIn: jianshi
> > > Twitter: @jshuang
> > > Github & Blog: http://huangjs.github.com/
> > >
> >
> >
> >
> > --
> > Jianshi Huang
> >
> > LinkedIn: jianshi
> > Twitter: @jshuang
> > Github & Blog: http://huangjs.github.com/
> >
>
>
>
> --
> Bharath Vissapragada
> <http://www.cloudera.com>
>
Re: Completebulkload with namespace option?
Posted by Bharath Vissapragada <bh...@cloudera.com>.
Appears to be a bug. It should be TableName.valueOf(...) or something
similar. Mind filing a jira?
On Tue, Jul 29, 2014 at 12:22 PM, Jianshi Huang <ji...@gmail.com>
wrote:
> I see why, looking at the source code of LoadIncrementalHFiles.java, it
> seems the temporary path created for splitting will contain ':',
>
> The error part should be this:
> String uniqueName = getUniqueName(table.getName());
> HColumnDescriptor familyDesc =
> table.getTableDescriptor().getFamily(item.family);
> Path botOut = new Path(tmpDir, uniqueName + ".bottom");
> Path topOut = new Path(tmpDir, uniqueName + ".top");
> splitStoreFile(getConf(), hfilePath, familyDesc, splitKey,
> botOut, topOut);
>
> uniqueName will be "namespce:table" so new Path will fail.
>
> A bug right?
>
> Jianshi
>
>
> On Tue, Jul 29, 2014 at 2:42 PM, Jianshi Huang <ji...@gmail.com>
> wrote:
>
> > I'm using hbase 0.98 with HDP 2.1.
> >
> >
> > On Tue, Jul 29, 2014 at 2:39 PM, Jianshi Huang <ji...@gmail.com>
> > wrote:
> >
> >> I'm using completebulkload to load 500GB of data to a table
> >> (presplitted). However, it reports the following errors:
> >>
> >> Looks like completebulkload didn't recognize the namespace part
> >> (namespace:table).
> >>
> >> Is there an option to do it? I can't find one in Google...
> >>
> >> Exception in thread "main" 14/07/28 23:32:19 INFO
> >> mapreduce.LoadIncrementalHFiles: Trying to load
> >> hfile=hdfs://xxx/vertices/PROP/f5cbf0965ff44cb8bdabd038e66485c3
> >> first=dc595cfe#cust#1812199228741466242
> >> last=dc68cedc#cust#2251647837553603393
> >> java.lang.reflect.InvocationTargetException
> >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >> at
> >>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> >> at
> >>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> >> at java.lang.reflect.Method.invoke(Method.java:606)
> >> at org.apache.hadoop.hbase.mapreduce.Driver.main(Driver.java:54)
> >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >> at
> >>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> >> at
> >>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> >> at java.lang.reflect.Method.invoke(Method.java:606)
> >> at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
> >> Caused by: java.lang.IllegalStateException:
> >> java.lang.IllegalArgumentException: java.net.URISyntaxException:
> Relative
> >> path in absolute URI: grapple:vertices,37.bottom
> >> at
> >>
> org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.groupOrSplitPhase(LoadIncrementalHFiles.java:421)
> >> at
> >>
> org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.doBulkLoad(LoadIncrementalHFiles.java:291)
> >> at
> >>
> org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.run(LoadIncrementalHFiles.java:825)
> >> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> >> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
> >> at
> >>
> org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.main(LoadIncrementalHFiles.java:831)
> >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >> at
> >>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> >> at
> >>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> >> at java.lang.reflect.Method.invoke(Method.java:606)
> >> at
> >>
> org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72)
> >> at
> >> org.apache.hadoop.util.ProgramDriver.run(ProgramDriver.java:145)
> >> at
> >> org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:153)
> >> ... 10 more
> >> Caused by: java.lang.IllegalArgumentException:
> >> java.net.URISyntaxException: Relative path in absolute URI:
> >> grapple:vertices,37.bottom
> >> at org.apache.hadoop.fs.Path.initialize(Path.java:206)
> >> at org.apache.hadoop.fs.Path.<init>(Path.java:172)
> >> at org.apache.hadoop.fs.Path.<init>(Path.java:94)
> >> at
> >>
> org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.splitStoreFile(LoadIncrementalHFiles.java:450)
> >> at
> >>
> org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.groupOrSplit(LoadIncrementalHFiles.java:516)
> >> at
> >>
> org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles$2.call(LoadIncrementalHFiles.java:400)
> >> at
> >>
> org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles$2.call(LoadIncrementalHFiles.java:398)
> >> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> >> at
> >>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> >> at
> >>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> >> at java.lang.Thread.run(Thread.java:724)
> >> Caused by: java.net.URISyntaxException: Relative path in absolute URI:
> >> grapple:vertices,37.bottom
> >> at java.net.URI.checkPath(URI.java:1804)
> >> at java.net.URI.<init>(URI.java:752)
> >> at org.apache.hadoop.fs.Path.initialize(Path.java:203)
> >> ... 10 more
> >>
> >> --
> >> Jianshi Huang
> >>
> >> LinkedIn: jianshi
> >> Twitter: @jshuang
> >> Github & Blog: http://huangjs.github.com/
> >>
> >
> >
> >
> > --
> > Jianshi Huang
> >
> > LinkedIn: jianshi
> > Twitter: @jshuang
> > Github & Blog: http://huangjs.github.com/
> >
>
>
>
> --
> Jianshi Huang
>
> LinkedIn: jianshi
> Twitter: @jshuang
> Github & Blog: http://huangjs.github.com/
>
--
Bharath Vissapragada
<http://www.cloudera.com>
Re: Completebulkload with namespace option?
Posted by Jianshi Huang <ji...@gmail.com>.
I see why, looking at the source code of LoadIncrementalHFiles.java, it
seems the temporary path created for splitting will contain ':',
The error part should be this:
String uniqueName = getUniqueName(table.getName());
HColumnDescriptor familyDesc =
table.getTableDescriptor().getFamily(item.family);
Path botOut = new Path(tmpDir, uniqueName + ".bottom");
Path topOut = new Path(tmpDir, uniqueName + ".top");
splitStoreFile(getConf(), hfilePath, familyDesc, splitKey,
botOut, topOut);
uniqueName will be "namespce:table" so new Path will fail.
A bug right?
Jianshi
On Tue, Jul 29, 2014 at 2:42 PM, Jianshi Huang <ji...@gmail.com>
wrote:
> I'm using hbase 0.98 with HDP 2.1.
>
>
> On Tue, Jul 29, 2014 at 2:39 PM, Jianshi Huang <ji...@gmail.com>
> wrote:
>
>> I'm using completebulkload to load 500GB of data to a table
>> (presplitted). However, it reports the following errors:
>>
>> Looks like completebulkload didn't recognize the namespace part
>> (namespace:table).
>>
>> Is there an option to do it? I can't find one in Google...
>>
>> Exception in thread "main" 14/07/28 23:32:19 INFO
>> mapreduce.LoadIncrementalHFiles: Trying to load
>> hfile=hdfs://xxx/vertices/PROP/f5cbf0965ff44cb8bdabd038e66485c3
>> first=dc595cfe#cust#1812199228741466242
>> last=dc68cedc#cust#2251647837553603393
>> java.lang.reflect.InvocationTargetException
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:606)
>> at org.apache.hadoop.hbase.mapreduce.Driver.main(Driver.java:54)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:606)
>> at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
>> Caused by: java.lang.IllegalStateException:
>> java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative
>> path in absolute URI: grapple:vertices,37.bottom
>> at
>> org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.groupOrSplitPhase(LoadIncrementalHFiles.java:421)
>> at
>> org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.doBulkLoad(LoadIncrementalHFiles.java:291)
>> at
>> org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.run(LoadIncrementalHFiles.java:825)
>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
>> at
>> org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.main(LoadIncrementalHFiles.java:831)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:606)
>> at
>> org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72)
>> at
>> org.apache.hadoop.util.ProgramDriver.run(ProgramDriver.java:145)
>> at
>> org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:153)
>> ... 10 more
>> Caused by: java.lang.IllegalArgumentException:
>> java.net.URISyntaxException: Relative path in absolute URI:
>> grapple:vertices,37.bottom
>> at org.apache.hadoop.fs.Path.initialize(Path.java:206)
>> at org.apache.hadoop.fs.Path.<init>(Path.java:172)
>> at org.apache.hadoop.fs.Path.<init>(Path.java:94)
>> at
>> org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.splitStoreFile(LoadIncrementalHFiles.java:450)
>> at
>> org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.groupOrSplit(LoadIncrementalHFiles.java:516)
>> at
>> org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles$2.call(LoadIncrementalHFiles.java:400)
>> at
>> org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles$2.call(LoadIncrementalHFiles.java:398)
>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>> at java.lang.Thread.run(Thread.java:724)
>> Caused by: java.net.URISyntaxException: Relative path in absolute URI:
>> grapple:vertices,37.bottom
>> at java.net.URI.checkPath(URI.java:1804)
>> at java.net.URI.<init>(URI.java:752)
>> at org.apache.hadoop.fs.Path.initialize(Path.java:203)
>> ... 10 more
>>
>> --
>> Jianshi Huang
>>
>> LinkedIn: jianshi
>> Twitter: @jshuang
>> Github & Blog: http://huangjs.github.com/
>>
>
>
>
> --
> Jianshi Huang
>
> LinkedIn: jianshi
> Twitter: @jshuang
> Github & Blog: http://huangjs.github.com/
>
--
Jianshi Huang
LinkedIn: jianshi
Twitter: @jshuang
Github & Blog: http://huangjs.github.com/
Re: Completebulkload with namespace option?
Posted by Jianshi Huang <ji...@gmail.com>.
I'm using hbase 0.98 with HDP 2.1.
On Tue, Jul 29, 2014 at 2:39 PM, Jianshi Huang <ji...@gmail.com>
wrote:
> I'm using completebulkload to load 500GB of data to a table (presplitted).
> However, it reports the following errors:
>
> Looks like completebulkload didn't recognize the namespace part
> (namespace:table).
>
> Is there an option to do it? I can't find one in Google...
>
> Exception in thread "main" 14/07/28 23:32:19 INFO
> mapreduce.LoadIncrementalHFiles: Trying to load
> hfile=hdfs://xxx/vertices/PROP/f5cbf0965ff44cb8bdabd038e66485c3
> first=dc595cfe#cust#1812199228741466242
> last=dc68cedc#cust#2251647837553603393
> java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.hbase.mapreduce.Driver.main(Driver.java:54)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
> Caused by: java.lang.IllegalStateException:
> java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative
> path in absolute URI: grapple:vertices,37.bottom
> at
> org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.groupOrSplitPhase(LoadIncrementalHFiles.java:421)
> at
> org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.doBulkLoad(LoadIncrementalHFiles.java:291)
> at
> org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.run(LoadIncrementalHFiles.java:825)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
> at
> org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.main(LoadIncrementalHFiles.java:831)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72)
> at org.apache.hadoop.util.ProgramDriver.run(ProgramDriver.java:145)
> at
> org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:153)
> ... 10 more
> Caused by: java.lang.IllegalArgumentException:
> java.net.URISyntaxException: Relative path in absolute URI:
> grapple:vertices,37.bottom
> at org.apache.hadoop.fs.Path.initialize(Path.java:206)
> at org.apache.hadoop.fs.Path.<init>(Path.java:172)
> at org.apache.hadoop.fs.Path.<init>(Path.java:94)
> at
> org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.splitStoreFile(LoadIncrementalHFiles.java:450)
> at
> org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.groupOrSplit(LoadIncrementalHFiles.java:516)
> at
> org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles$2.call(LoadIncrementalHFiles.java:400)
> at
> org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles$2.call(LoadIncrementalHFiles.java:398)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:724)
> Caused by: java.net.URISyntaxException: Relative path in absolute URI:
> grapple:vertices,37.bottom
> at java.net.URI.checkPath(URI.java:1804)
> at java.net.URI.<init>(URI.java:752)
> at org.apache.hadoop.fs.Path.initialize(Path.java:203)
> ... 10 more
>
> --
> Jianshi Huang
>
> LinkedIn: jianshi
> Twitter: @jshuang
> Github & Blog: http://huangjs.github.com/
>
--
Jianshi Huang
LinkedIn: jianshi
Twitter: @jshuang
Github & Blog: http://huangjs.github.com/