You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Gautam <ga...@gmail.com> on 2014/09/16 01:58:33 UTC
Copying data from 94 to 98 ..
Hello,
I'm trying to copy data between Hbase clusters on different
versions. I am using :
/usr/bin/hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot
-chuser hbase
-chgroup hadoop
-snapshot msg_snapshot
-mappers 50
-copy-from hftp://src-cluster:50070/hbase
-copy-to hdfs:/dest-cluster:8020/hbase
Till now, based on various tips from the mailing list, I have modified the
source cluster data dir paths to mimic the 98 convention (archive, table
data paths, etc). This helped in jumping some roadblocks but not all.
This is what I see now :
14/09/15 23:34:59 ERROR snapshot.ExportSnapshot: Snapshot export failed
java.io.IOException
at
org.apache.hadoop.hbase.snapshot.SnapshotManifestV1.loadRegionManifests(SnapshotManifestV1.java:145)
at
org.apache.hadoop.hbase.snapshot.SnapshotManifest.load(SnapshotManifest.java:265)
at
org.apache.hadoop.hbase.snapshot.SnapshotManifest.open(SnapshotManifest.java:119)
at
org.apache.hadoop.hbase.snapshot.SnapshotReferenceUtil.visitTableStoreFiles(SnapshotReferenceUtil.java:125)
...
..
Caused by: java.io.IOException: read=-1, wanted=4
at org.apache.hadoop.hbase.io.Reference.read(Reference.java:175)
at
org.apache.hadoop.hbase.regionserver.StoreFileInfo.<init>(StoreFileInfo.java:115)
at
org.apache.hadoop.hbase.regionserver.HRegionFileSystem.getStoreFiles(HRegionFileSystem.java
..
Fails while trying to read refernce hfile. Is this something folks have
done before and/or is possible to do? I'd really like to do this without
having to upgrade my source cluster or downgrade my dest cluster.
I'm using 94.6 on source cluster and 98.1 on dest cluster.
Cheers,
-Gautam.
Re: Copying data from 94 to 98 ..
Posted by Gautam <ga...@gmail.com>.
14/09/15 23:34:59 DEBUG snapshot.SnapshotManifestV1: Adding reference for
file (4/4): hftp://
master42.stg.com:50070/hbase/.hbase-snapshot/msg_snapshot/84f60fc2aa7e96df91e6289e6c19dc25/c/afe341e4149649578c5861e32494dbec
14/09/15 23:34:59 ERROR snapshot.ExportSnapshot: Snapshot export failed
java.io.IOException
at
org.apache.hadoop.hbase.snapshot.SnapshotManifestV1.loadRegionManifests(SnapshotManifestV1.java:145)
at
org.apache.hadoop.hbase.snapshot.SnapshotManifest.load(SnapshotManifest.java:265)
at
org.apache.hadoop.hbase.snapshot.SnapshotManifest.open(SnapshotManifest.java:119)
at
org.apache.hadoop.hbase.snapshot.SnapshotReferenceUtil.visitTableStoreFiles(SnapshotReferenceUtil.java:125)
at
org.apache.hadoop.hbase.snapshot.SnapshotReferenceUtil.visitReferencedFiles(SnapshotReferenceUtil.java:108)
at
org.apache.hadoop.hbase.snapshot.ExportSnapshot.getSnapshotFiles(ExportSnapshot.java:479)
at
org.apache.hadoop.hbase.snapshot.ExportSnapshot.access$200(ExportSnapshot.java:89)
at
org.apache.hadoop.hbase.snapshot.ExportSnapshot$ExportSnapshotInputFormat.getSplits(ExportSnapshot.java:600)
at org.apache.hadoop.mapred.JobClient.writeNewSplits(JobClient.java:1107)
at org.apache.hadoop.mapred.JobClient.writeSplits(JobClient.java:1124)
at org.apache.hadoop.mapred.JobClient.access$600(JobClient.java:178)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:1023)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:976)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1554)
at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:976)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:582)
at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:612)
at
org.apache.hadoop.hbase.snapshot.ExportSnapshot.runCopyJob(ExportSnapshot.java:751)
at
org.apache.hadoop.hbase.snapshot.ExportSnapshot.run(ExportSnapshot.java:905)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at
org.apache.hadoop.hbase.snapshot.ExportSnapshot.innerMain(ExportSnapshot.java:975)
at
org.apache.hadoop.hbase.snapshot.ExportSnapshot.main(ExportSnapshot.java:979)
Caused by: java.io.IOException: read=-1, wanted=4
at org.apache.hadoop.hbase.io.Reference.read(Reference.java:175)
at
org.apache.hadoop.hbase.regionserver.StoreFileInfo.<init>(StoreFileInfo.java:115)
at
org.apache.hadoop.hbase.regionserver.HRegionFileSystem.getStoreFiles(HRegionFileSystem.java:204)
at
org.apache.hadoop.hbase.snapshot.SnapshotManifestV1.buildManifestFromDisk(SnapshotManifestV1.java:179)
at
org.apache.hadoop.hbase.snapshot.SnapshotManifestV1$1.call(SnapshotManifestV1.java:131)
at
org.apache.hadoop.hbase.snapshot.SnapshotManifestV1$1.call(SnapshotManifestV1.java:127)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
On Mon, Sep 15, 2014 at 10:18 PM, Matteo Bertozzi <th...@gmail.com>
wrote:
> can you post the full exception and the file path ?
> maybe there is a bug in looking up the reference file.
> It seems to not be able to find enough data in the file...
>
> Matteo
>
>
> On Mon, Sep 15, 2014 at 10:08 PM, Gautam <ga...@gmail.com> wrote:
>
> > Thanks for the reply Matteo.
> >
> > This is exactly what I did. I modified the source cluster's dir structure
> > to mimic that of the 98 cluster. I even got as far as it trying to look
> > through the reference files.
> >
> > I end up with this exception :
> >
> > 14/09/15 23:34:59 ERROR snapshot.ExportSnapshot: Snapshot export failed
> > java.io.IOException
> > at
> >
> >
> org.apache.hadoop.hbase.snapshot.SnapshotManifestV1.loadRegionManifests(SnapshotManifestV1.java:145)
> > at
> >
> >
> org.apache.hadoop.hbase.snapshot.SnapshotManifest.load(SnapshotManifest.java:265)
> > at
> >
> >
> org.apache.hadoop.hbase.snapshot.SnapshotManifest.open(SnapshotManifest.java:119)
> > at
> >
> >
> org.apache.hadoop.hbase.snapshot.SnapshotReferenceUtil.visitTableStoreFiles(SnapshotReferenceUtil.java:125)
> > ...
> > ..
> > Caused by: java.io.IOException: read=-1, wanted=4
> > at org.apache.hadoop.hbase.io.Reference.read(Reference.java:175)
> > at
> >
> >
> org.apache.hadoop.hbase.regionserver.StoreFileInfo.<init>(StoreFileInfo.java:115)
> > at
> >
> >
> org.apache.hadoop.hbase.regionserver.HRegionFileSystem.getStoreFiles(HRegionFileSystem.java
> > ..
> >
> >
> > This and Ted's reply about HBASE-7987 leads me to believe that the export
> > tool from my distro is in capable of working around the regionManifest
> file
> > requirement. I'm now left with the option of downgrading my dest cluster
> to
> > 94, copying data and then upgrading using the upgrade migration tool.
> > Wanted to know if others have tried this or there are other things I can
> > do. If not, i'l just go ahead and do this :-)
> >
> >
> > Cheers,
> > -Gautam.
> >
> >
> >
> > On Mon, Sep 15, 2014 at 8:10 PM, Matteo Bertozzi <
> theo.bertozzi@gmail.com>
> > wrote:
> >
> > > 94 and 98 differs in directory layout
> > > so 98 is not able to read 94 layout unless you run the migration tool
> > > which is basically moving all the data in a "default" namespace
> directory
> > > e.g.
> > > /hbase/table -> /hbase/data/default/table
> > > /hbase/.archive/table -> /hbase/archive/default/table
> > >
> > > Matteo
> > >
> > >
> > > On Mon, Sep 15, 2014 at 6:17 PM, Gautam <ga...@gmail.com>
> wrote:
> > >
> > > > Yep, looks like the CDH distro backports HBASE-7987. Having said
> that,
> > is
> > > > there a transition path for us or are we hosed :-) ? In general,
> what's
> > > the
> > > > recommended way to achieve this, at this point I feel i'm going
> around
> > > the
> > > > system to achieve what I want. If nothing else works with export
> > snapshot
> > > > should I just downgrade to 94, export snapshot and then upgrade to
> 98?
> > Is
> > > > the upgrade migration path different from what export snapshot does
> > (i'd
> > > > imagine yes)?
> > > >
> > > > Cheers,
> > > > -Gautam.
> > > >
> > > >
> > > >
> > > >
> > > > On Mon, Sep 15, 2014 at 5:14 PM, Ted Yu <yu...@gmail.com> wrote:
> > > >
> > > > > bq. 98.1 on dest cluster
> > > > >
> > > > > Looking at the history for SnapshotManifestV1, it came with
> > HBASE-7987
> > > > > which went to 0.99.0
> > > > >
> > > > > Perhaps you're using a distro with HBASE-7987 ?
> > > > >
> > > > > On Mon, Sep 15, 2014 at 4:58 PM, Gautam <ga...@gmail.com>
> > > wrote:
> > > > >
> > > > > > Hello,
> > > > > > I'm trying to copy data between Hbase clusters on
> > different
> > > > > > versions. I am using :
> > > > > >
> > > > > > /usr/bin/hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot
> > > > > > -chuser hbase
> > > > > > -chgroup hadoop
> > > > > > -snapshot msg_snapshot
> > > > > > -mappers 50
> > > > > > -copy-from hftp://src-cluster:50070/hbase
> > > > > > -copy-to hdfs:/dest-cluster:8020/hbase
> > > > > >
> > > > > >
> > > > > > Till now, based on various tips from the mailing list, I have
> > > modified
> > > > > the
> > > > > > source cluster data dir paths to mimic the 98 convention
> (archive,
> > > > table
> > > > > > data paths, etc). This helped in jumping some roadblocks but not
> > all.
> > > > > >
> > > > > > This is what I see now :
> > > > > >
> > > > > >
> > > > > > 14/09/15 23:34:59 ERROR snapshot.ExportSnapshot: Snapshot export
> > > failed
> > > > > > java.io.IOException
> > > > > > at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.snapshot.SnapshotManifestV1.loadRegionManifests(SnapshotManifestV1.java:145)
> > > > > > at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.snapshot.SnapshotManifest.load(SnapshotManifest.java:265)
> > > > > > at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.snapshot.SnapshotManifest.open(SnapshotManifest.java:119)
> > > > > > at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.snapshot.SnapshotReferenceUtil.visitTableStoreFiles(SnapshotReferenceUtil.java:125)
> > > > > > ...
> > > > > > ..
> > > > > > Caused by: java.io.IOException: read=-1, wanted=4
> > > > > > at org.apache.hadoop.hbase.io.Reference.read(Reference.java:175)
> > > > > > at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.regionserver.StoreFileInfo.<init>(StoreFileInfo.java:115)
> > > > > > at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.regionserver.HRegionFileSystem.getStoreFiles(HRegionFileSystem.java
> > > > > > ..
> > > > > >
> > > > > >
> > > > > > Fails while trying to read refernce hfile. Is this something
> folks
> > > have
> > > > > > done before and/or is possible to do? I'd really like to do this
> > > > without
> > > > > > having to upgrade my source cluster or downgrade my dest cluster.
> > > > > >
> > > > > > I'm using 94.6 on source cluster and 98.1 on dest cluster.
> > > > > >
> > > > > >
> > > > > > Cheers,
> > > > > > -Gautam.
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > "If you really want something in this life, you have to work for it.
> > Now,
> > > > quiet! They're about to announce the lottery numbers..."
> > > >
> > >
> >
> >
> >
> > --
> > "If you really want something in this life, you have to work for it. Now,
> > quiet! They're about to announce the lottery numbers..."
> >
>
--
"If you really want something in this life, you have to work for it. Now,
quiet! They're about to announce the lottery numbers..."
Re: Copying data from 94 to 98 ..
Posted by Matteo Bertozzi <th...@gmail.com>.
ok, so you have an empty reference file under .snapshot/SnapshotName/...
what you can do is find all the reference files and replace them with the
version in /hbase/.archive
you can find reference files because are in the form "name.regionName"
(they are the only files with a dot in the middle)
so.. something like
cp /hbase/.archive/tableName/region/myfile.parent
/hbase/.snapshot/snapshotName/region/myfile.parent
Matteo
On Mon, Sep 15, 2014 at 11:00 PM, Gautam <ga...@gmail.com> wrote:
> Jerry,
> Can you elaborate on what you mean by "export table to hdfs"? I
> initially tried running the export on src cluster (-copy-to
> hdfs://dest/hbase ), it complains while trying to write the data to dest
> cluster (due to the hdfs protocol version mismatch). Then I tried running
> export on dest cluster (-copy-from hftp://src/hbase).
>
> On Mon, Sep 15, 2014 at 10:36 PM, Jerry He <je...@gmail.com> wrote:
>
> > While you continue on the snapshot approach, have you tried to Export the
> > table in 0.94 to hdfs, and then Import the data from hdfs to 0.98?
> > On Sep 15, 2014 10:19 PM, "Matteo Bertozzi" <th...@gmail.com>
> > wrote:
> >
> > > can you post the full exception and the file path ?
> > > maybe there is a bug in looking up the reference file.
> > > It seems to not be able to find enough data in the file...
> > >
> > > Matteo
> > >
> > >
> > > On Mon, Sep 15, 2014 at 10:08 PM, Gautam <ga...@gmail.com>
> > wrote:
> > >
> > > > Thanks for the reply Matteo.
> > > >
> > > > This is exactly what I did. I modified the source cluster's dir
> > structure
> > > > to mimic that of the 98 cluster. I even got as far as it trying to
> look
> > > > through the reference files.
> > > >
> > > > I end up with this exception :
> > > >
> > > > 14/09/15 23:34:59 ERROR snapshot.ExportSnapshot: Snapshot export
> failed
> > > > java.io.IOException
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.snapshot.SnapshotManifestV1.loadRegionManifests(SnapshotManifestV1.java:145)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.snapshot.SnapshotManifest.load(SnapshotManifest.java:265)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.snapshot.SnapshotManifest.open(SnapshotManifest.java:119)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.snapshot.SnapshotReferenceUtil.visitTableStoreFiles(SnapshotReferenceUtil.java:125)
> > > > ...
> > > > ..
> > > > Caused by: java.io.IOException: read=-1, wanted=4
> > > > at org.apache.hadoop.hbase.io.Reference.read(Reference.java:175)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.regionserver.StoreFileInfo.<init>(StoreFileInfo.java:115)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.regionserver.HRegionFileSystem.getStoreFiles(HRegionFileSystem.java
> > > > ..
> > > >
> > > >
> > > > This and Ted's reply about HBASE-7987 leads me to believe that the
> > export
> > > > tool from my distro is in capable of working around the
> regionManifest
> > > file
> > > > requirement. I'm now left with the option of downgrading my dest
> > cluster
> > > to
> > > > 94, copying data and then upgrading using the upgrade migration tool.
> > > > Wanted to know if others have tried this or there are other things I
> > can
> > > > do. If not, i'l just go ahead and do this :-)
> > > >
> > > >
> > > > Cheers,
> > > > -Gautam.
> > > >
> > > >
> > > >
> > > > On Mon, Sep 15, 2014 at 8:10 PM, Matteo Bertozzi <
> > > theo.bertozzi@gmail.com>
> > > > wrote:
> > > >
> > > > > 94 and 98 differs in directory layout
> > > > > so 98 is not able to read 94 layout unless you run the migration
> tool
> > > > > which is basically moving all the data in a "default" namespace
> > > directory
> > > > > e.g.
> > > > > /hbase/table -> /hbase/data/default/table
> > > > > /hbase/.archive/table -> /hbase/archive/default/table
> > > > >
> > > > > Matteo
> > > > >
> > > > >
> > > > > On Mon, Sep 15, 2014 at 6:17 PM, Gautam <ga...@gmail.com>
> > > wrote:
> > > > >
> > > > > > Yep, looks like the CDH distro backports HBASE-7987. Having said
> > > that,
> > > > is
> > > > > > there a transition path for us or are we hosed :-) ? In general,
> > > what's
> > > > > the
> > > > > > recommended way to achieve this, at this point I feel i'm going
> > > around
> > > > > the
> > > > > > system to achieve what I want. If nothing else works with export
> > > > snapshot
> > > > > > should I just downgrade to 94, export snapshot and then upgrade
> to
> > > 98?
> > > > Is
> > > > > > the upgrade migration path different from what export snapshot
> does
> > > > (i'd
> > > > > > imagine yes)?
> > > > > >
> > > > > > Cheers,
> > > > > > -Gautam.
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Mon, Sep 15, 2014 at 5:14 PM, Ted Yu <yu...@gmail.com>
> > wrote:
> > > > > >
> > > > > > > bq. 98.1 on dest cluster
> > > > > > >
> > > > > > > Looking at the history for SnapshotManifestV1, it came with
> > > > HBASE-7987
> > > > > > > which went to 0.99.0
> > > > > > >
> > > > > > > Perhaps you're using a distro with HBASE-7987 ?
> > > > > > >
> > > > > > > On Mon, Sep 15, 2014 at 4:58 PM, Gautam <
> gautamkowshik@gmail.com
> > >
> > > > > wrote:
> > > > > > >
> > > > > > > > Hello,
> > > > > > > > I'm trying to copy data between Hbase clusters on
> > > > different
> > > > > > > > versions. I am using :
> > > > > > > >
> > > > > > > > /usr/bin/hbase
> org.apache.hadoop.hbase.snapshot.ExportSnapshot
> > > > > > > > -chuser hbase
> > > > > > > > -chgroup hadoop
> > > > > > > > -snapshot msg_snapshot
> > > > > > > > -mappers 50
> > > > > > > > -copy-from hftp://src-cluster:50070/hbase
> > > > > > > > -copy-to hdfs:/dest-cluster:8020/hbase
> > > > > > > >
> > > > > > > >
> > > > > > > > Till now, based on various tips from the mailing list, I have
> > > > > modified
> > > > > > > the
> > > > > > > > source cluster data dir paths to mimic the 98 convention
> > > (archive,
> > > > > > table
> > > > > > > > data paths, etc). This helped in jumping some roadblocks but
> > not
> > > > all.
> > > > > > > >
> > > > > > > > This is what I see now :
> > > > > > > >
> > > > > > > >
> > > > > > > > 14/09/15 23:34:59 ERROR snapshot.ExportSnapshot: Snapshot
> > export
> > > > > failed
> > > > > > > > java.io.IOException
> > > > > > > > at
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.snapshot.SnapshotManifestV1.loadRegionManifests(SnapshotManifestV1.java:145)
> > > > > > > > at
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.snapshot.SnapshotManifest.load(SnapshotManifest.java:265)
> > > > > > > > at
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.snapshot.SnapshotManifest.open(SnapshotManifest.java:119)
> > > > > > > > at
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.snapshot.SnapshotReferenceUtil.visitTableStoreFiles(SnapshotReferenceUtil.java:125)
> > > > > > > > ...
> > > > > > > > ..
> > > > > > > > Caused by: java.io.IOException: read=-1, wanted=4
> > > > > > > > at
> > org.apache.hadoop.hbase.io.Reference.read(Reference.java:175)
> > > > > > > > at
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.regionserver.StoreFileInfo.<init>(StoreFileInfo.java:115)
> > > > > > > > at
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.regionserver.HRegionFileSystem.getStoreFiles(HRegionFileSystem.java
> > > > > > > > ..
> > > > > > > >
> > > > > > > >
> > > > > > > > Fails while trying to read refernce hfile. Is this something
> > > folks
> > > > > have
> > > > > > > > done before and/or is possible to do? I'd really like to do
> > this
> > > > > > without
> > > > > > > > having to upgrade my source cluster or downgrade my dest
> > cluster.
> > > > > > > >
> > > > > > > > I'm using 94.6 on source cluster and 98.1 on dest cluster.
> > > > > > > >
> > > > > > > >
> > > > > > > > Cheers,
> > > > > > > > -Gautam.
> > > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > "If you really want something in this life, you have to work for
> > it.
> > > > Now,
> > > > > > quiet! They're about to announce the lottery numbers..."
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > "If you really want something in this life, you have to work for it.
> > Now,
> > > > quiet! They're about to announce the lottery numbers..."
> > > >
> > >
> >
>
>
>
> --
> "If you really want something in this life, you have to work for it. Now,
> quiet! They're about to announce the lottery numbers..."
>
Re: Copying data from 94 to 98 ..
Posted by Gautam <ga...@gmail.com>.
Jerry,
Can you elaborate on what you mean by "export table to hdfs"? I
initially tried running the export on src cluster (-copy-to
hdfs://dest/hbase ), it complains while trying to write the data to dest
cluster (due to the hdfs protocol version mismatch). Then I tried running
export on dest cluster (-copy-from hftp://src/hbase).
On Mon, Sep 15, 2014 at 10:36 PM, Jerry He <je...@gmail.com> wrote:
> While you continue on the snapshot approach, have you tried to Export the
> table in 0.94 to hdfs, and then Import the data from hdfs to 0.98?
> On Sep 15, 2014 10:19 PM, "Matteo Bertozzi" <th...@gmail.com>
> wrote:
>
> > can you post the full exception and the file path ?
> > maybe there is a bug in looking up the reference file.
> > It seems to not be able to find enough data in the file...
> >
> > Matteo
> >
> >
> > On Mon, Sep 15, 2014 at 10:08 PM, Gautam <ga...@gmail.com>
> wrote:
> >
> > > Thanks for the reply Matteo.
> > >
> > > This is exactly what I did. I modified the source cluster's dir
> structure
> > > to mimic that of the 98 cluster. I even got as far as it trying to look
> > > through the reference files.
> > >
> > > I end up with this exception :
> > >
> > > 14/09/15 23:34:59 ERROR snapshot.ExportSnapshot: Snapshot export failed
> > > java.io.IOException
> > > at
> > >
> > >
> >
> org.apache.hadoop.hbase.snapshot.SnapshotManifestV1.loadRegionManifests(SnapshotManifestV1.java:145)
> > > at
> > >
> > >
> >
> org.apache.hadoop.hbase.snapshot.SnapshotManifest.load(SnapshotManifest.java:265)
> > > at
> > >
> > >
> >
> org.apache.hadoop.hbase.snapshot.SnapshotManifest.open(SnapshotManifest.java:119)
> > > at
> > >
> > >
> >
> org.apache.hadoop.hbase.snapshot.SnapshotReferenceUtil.visitTableStoreFiles(SnapshotReferenceUtil.java:125)
> > > ...
> > > ..
> > > Caused by: java.io.IOException: read=-1, wanted=4
> > > at org.apache.hadoop.hbase.io.Reference.read(Reference.java:175)
> > > at
> > >
> > >
> >
> org.apache.hadoop.hbase.regionserver.StoreFileInfo.<init>(StoreFileInfo.java:115)
> > > at
> > >
> > >
> >
> org.apache.hadoop.hbase.regionserver.HRegionFileSystem.getStoreFiles(HRegionFileSystem.java
> > > ..
> > >
> > >
> > > This and Ted's reply about HBASE-7987 leads me to believe that the
> export
> > > tool from my distro is in capable of working around the regionManifest
> > file
> > > requirement. I'm now left with the option of downgrading my dest
> cluster
> > to
> > > 94, copying data and then upgrading using the upgrade migration tool.
> > > Wanted to know if others have tried this or there are other things I
> can
> > > do. If not, i'l just go ahead and do this :-)
> > >
> > >
> > > Cheers,
> > > -Gautam.
> > >
> > >
> > >
> > > On Mon, Sep 15, 2014 at 8:10 PM, Matteo Bertozzi <
> > theo.bertozzi@gmail.com>
> > > wrote:
> > >
> > > > 94 and 98 differs in directory layout
> > > > so 98 is not able to read 94 layout unless you run the migration tool
> > > > which is basically moving all the data in a "default" namespace
> > directory
> > > > e.g.
> > > > /hbase/table -> /hbase/data/default/table
> > > > /hbase/.archive/table -> /hbase/archive/default/table
> > > >
> > > > Matteo
> > > >
> > > >
> > > > On Mon, Sep 15, 2014 at 6:17 PM, Gautam <ga...@gmail.com>
> > wrote:
> > > >
> > > > > Yep, looks like the CDH distro backports HBASE-7987. Having said
> > that,
> > > is
> > > > > there a transition path for us or are we hosed :-) ? In general,
> > what's
> > > > the
> > > > > recommended way to achieve this, at this point I feel i'm going
> > around
> > > > the
> > > > > system to achieve what I want. If nothing else works with export
> > > snapshot
> > > > > should I just downgrade to 94, export snapshot and then upgrade to
> > 98?
> > > Is
> > > > > the upgrade migration path different from what export snapshot does
> > > (i'd
> > > > > imagine yes)?
> > > > >
> > > > > Cheers,
> > > > > -Gautam.
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > On Mon, Sep 15, 2014 at 5:14 PM, Ted Yu <yu...@gmail.com>
> wrote:
> > > > >
> > > > > > bq. 98.1 on dest cluster
> > > > > >
> > > > > > Looking at the history for SnapshotManifestV1, it came with
> > > HBASE-7987
> > > > > > which went to 0.99.0
> > > > > >
> > > > > > Perhaps you're using a distro with HBASE-7987 ?
> > > > > >
> > > > > > On Mon, Sep 15, 2014 at 4:58 PM, Gautam <gautamkowshik@gmail.com
> >
> > > > wrote:
> > > > > >
> > > > > > > Hello,
> > > > > > > I'm trying to copy data between Hbase clusters on
> > > different
> > > > > > > versions. I am using :
> > > > > > >
> > > > > > > /usr/bin/hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot
> > > > > > > -chuser hbase
> > > > > > > -chgroup hadoop
> > > > > > > -snapshot msg_snapshot
> > > > > > > -mappers 50
> > > > > > > -copy-from hftp://src-cluster:50070/hbase
> > > > > > > -copy-to hdfs:/dest-cluster:8020/hbase
> > > > > > >
> > > > > > >
> > > > > > > Till now, based on various tips from the mailing list, I have
> > > > modified
> > > > > > the
> > > > > > > source cluster data dir paths to mimic the 98 convention
> > (archive,
> > > > > table
> > > > > > > data paths, etc). This helped in jumping some roadblocks but
> not
> > > all.
> > > > > > >
> > > > > > > This is what I see now :
> > > > > > >
> > > > > > >
> > > > > > > 14/09/15 23:34:59 ERROR snapshot.ExportSnapshot: Snapshot
> export
> > > > failed
> > > > > > > java.io.IOException
> > > > > > > at
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.snapshot.SnapshotManifestV1.loadRegionManifests(SnapshotManifestV1.java:145)
> > > > > > > at
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.snapshot.SnapshotManifest.load(SnapshotManifest.java:265)
> > > > > > > at
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.snapshot.SnapshotManifest.open(SnapshotManifest.java:119)
> > > > > > > at
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.snapshot.SnapshotReferenceUtil.visitTableStoreFiles(SnapshotReferenceUtil.java:125)
> > > > > > > ...
> > > > > > > ..
> > > > > > > Caused by: java.io.IOException: read=-1, wanted=4
> > > > > > > at
> org.apache.hadoop.hbase.io.Reference.read(Reference.java:175)
> > > > > > > at
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.regionserver.StoreFileInfo.<init>(StoreFileInfo.java:115)
> > > > > > > at
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.regionserver.HRegionFileSystem.getStoreFiles(HRegionFileSystem.java
> > > > > > > ..
> > > > > > >
> > > > > > >
> > > > > > > Fails while trying to read refernce hfile. Is this something
> > folks
> > > > have
> > > > > > > done before and/or is possible to do? I'd really like to do
> this
> > > > > without
> > > > > > > having to upgrade my source cluster or downgrade my dest
> cluster.
> > > > > > >
> > > > > > > I'm using 94.6 on source cluster and 98.1 on dest cluster.
> > > > > > >
> > > > > > >
> > > > > > > Cheers,
> > > > > > > -Gautam.
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > "If you really want something in this life, you have to work for
> it.
> > > Now,
> > > > > quiet! They're about to announce the lottery numbers..."
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > "If you really want something in this life, you have to work for it.
> Now,
> > > quiet! They're about to announce the lottery numbers..."
> > >
> >
>
--
"If you really want something in this life, you have to work for it. Now,
quiet! They're about to announce the lottery numbers..."
Re: Copying data from 94 to 98 ..
Posted by Brian Jeltema <br...@digitalenvoy.net>.
I’ve been successfully moving snapshots from 94 to 98 using webhdfs. On the 94 cluster:
hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot -snapshot snappy -copy-to webhdfs://host-on-98-cluster/apps/hbase/data -mappers 12
and then manually fixing the file system layout.
On Sep 16, 2014, at 3:10 AM, Jerry He <je...@gmail.com> wrote:
> The Export and Import table tools in HBase. They are slower than snapshot
> though. Also, you probably need to come up with something to get around
> your hdfs protocol mismatch for this as well.
Re: Copying data from 94 to 98 ..
Posted by Jerry He <je...@gmail.com>.
The Export and Import table tools in HBase. They are slower than snapshot
though. Also, you probably need to come up with something to get around
your hdfs protocol mismatch for this as well.
Re: Copying data from 94 to 98 ..
Posted by Jerry He <je...@gmail.com>.
While you continue on the snapshot approach, have you tried to Export the
table in 0.94 to hdfs, and then Import the data from hdfs to 0.98?
On Sep 15, 2014 10:19 PM, "Matteo Bertozzi" <th...@gmail.com> wrote:
> can you post the full exception and the file path ?
> maybe there is a bug in looking up the reference file.
> It seems to not be able to find enough data in the file...
>
> Matteo
>
>
> On Mon, Sep 15, 2014 at 10:08 PM, Gautam <ga...@gmail.com> wrote:
>
> > Thanks for the reply Matteo.
> >
> > This is exactly what I did. I modified the source cluster's dir structure
> > to mimic that of the 98 cluster. I even got as far as it trying to look
> > through the reference files.
> >
> > I end up with this exception :
> >
> > 14/09/15 23:34:59 ERROR snapshot.ExportSnapshot: Snapshot export failed
> > java.io.IOException
> > at
> >
> >
> org.apache.hadoop.hbase.snapshot.SnapshotManifestV1.loadRegionManifests(SnapshotManifestV1.java:145)
> > at
> >
> >
> org.apache.hadoop.hbase.snapshot.SnapshotManifest.load(SnapshotManifest.java:265)
> > at
> >
> >
> org.apache.hadoop.hbase.snapshot.SnapshotManifest.open(SnapshotManifest.java:119)
> > at
> >
> >
> org.apache.hadoop.hbase.snapshot.SnapshotReferenceUtil.visitTableStoreFiles(SnapshotReferenceUtil.java:125)
> > ...
> > ..
> > Caused by: java.io.IOException: read=-1, wanted=4
> > at org.apache.hadoop.hbase.io.Reference.read(Reference.java:175)
> > at
> >
> >
> org.apache.hadoop.hbase.regionserver.StoreFileInfo.<init>(StoreFileInfo.java:115)
> > at
> >
> >
> org.apache.hadoop.hbase.regionserver.HRegionFileSystem.getStoreFiles(HRegionFileSystem.java
> > ..
> >
> >
> > This and Ted's reply about HBASE-7987 leads me to believe that the export
> > tool from my distro is in capable of working around the regionManifest
> file
> > requirement. I'm now left with the option of downgrading my dest cluster
> to
> > 94, copying data and then upgrading using the upgrade migration tool.
> > Wanted to know if others have tried this or there are other things I can
> > do. If not, i'l just go ahead and do this :-)
> >
> >
> > Cheers,
> > -Gautam.
> >
> >
> >
> > On Mon, Sep 15, 2014 at 8:10 PM, Matteo Bertozzi <
> theo.bertozzi@gmail.com>
> > wrote:
> >
> > > 94 and 98 differs in directory layout
> > > so 98 is not able to read 94 layout unless you run the migration tool
> > > which is basically moving all the data in a "default" namespace
> directory
> > > e.g.
> > > /hbase/table -> /hbase/data/default/table
> > > /hbase/.archive/table -> /hbase/archive/default/table
> > >
> > > Matteo
> > >
> > >
> > > On Mon, Sep 15, 2014 at 6:17 PM, Gautam <ga...@gmail.com>
> wrote:
> > >
> > > > Yep, looks like the CDH distro backports HBASE-7987. Having said
> that,
> > is
> > > > there a transition path for us or are we hosed :-) ? In general,
> what's
> > > the
> > > > recommended way to achieve this, at this point I feel i'm going
> around
> > > the
> > > > system to achieve what I want. If nothing else works with export
> > snapshot
> > > > should I just downgrade to 94, export snapshot and then upgrade to
> 98?
> > Is
> > > > the upgrade migration path different from what export snapshot does
> > (i'd
> > > > imagine yes)?
> > > >
> > > > Cheers,
> > > > -Gautam.
> > > >
> > > >
> > > >
> > > >
> > > > On Mon, Sep 15, 2014 at 5:14 PM, Ted Yu <yu...@gmail.com> wrote:
> > > >
> > > > > bq. 98.1 on dest cluster
> > > > >
> > > > > Looking at the history for SnapshotManifestV1, it came with
> > HBASE-7987
> > > > > which went to 0.99.0
> > > > >
> > > > > Perhaps you're using a distro with HBASE-7987 ?
> > > > >
> > > > > On Mon, Sep 15, 2014 at 4:58 PM, Gautam <ga...@gmail.com>
> > > wrote:
> > > > >
> > > > > > Hello,
> > > > > > I'm trying to copy data between Hbase clusters on
> > different
> > > > > > versions. I am using :
> > > > > >
> > > > > > /usr/bin/hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot
> > > > > > -chuser hbase
> > > > > > -chgroup hadoop
> > > > > > -snapshot msg_snapshot
> > > > > > -mappers 50
> > > > > > -copy-from hftp://src-cluster:50070/hbase
> > > > > > -copy-to hdfs:/dest-cluster:8020/hbase
> > > > > >
> > > > > >
> > > > > > Till now, based on various tips from the mailing list, I have
> > > modified
> > > > > the
> > > > > > source cluster data dir paths to mimic the 98 convention
> (archive,
> > > > table
> > > > > > data paths, etc). This helped in jumping some roadblocks but not
> > all.
> > > > > >
> > > > > > This is what I see now :
> > > > > >
> > > > > >
> > > > > > 14/09/15 23:34:59 ERROR snapshot.ExportSnapshot: Snapshot export
> > > failed
> > > > > > java.io.IOException
> > > > > > at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.snapshot.SnapshotManifestV1.loadRegionManifests(SnapshotManifestV1.java:145)
> > > > > > at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.snapshot.SnapshotManifest.load(SnapshotManifest.java:265)
> > > > > > at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.snapshot.SnapshotManifest.open(SnapshotManifest.java:119)
> > > > > > at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.snapshot.SnapshotReferenceUtil.visitTableStoreFiles(SnapshotReferenceUtil.java:125)
> > > > > > ...
> > > > > > ..
> > > > > > Caused by: java.io.IOException: read=-1, wanted=4
> > > > > > at org.apache.hadoop.hbase.io.Reference.read(Reference.java:175)
> > > > > > at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.regionserver.StoreFileInfo.<init>(StoreFileInfo.java:115)
> > > > > > at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.regionserver.HRegionFileSystem.getStoreFiles(HRegionFileSystem.java
> > > > > > ..
> > > > > >
> > > > > >
> > > > > > Fails while trying to read refernce hfile. Is this something
> folks
> > > have
> > > > > > done before and/or is possible to do? I'd really like to do this
> > > > without
> > > > > > having to upgrade my source cluster or downgrade my dest cluster.
> > > > > >
> > > > > > I'm using 94.6 on source cluster and 98.1 on dest cluster.
> > > > > >
> > > > > >
> > > > > > Cheers,
> > > > > > -Gautam.
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > "If you really want something in this life, you have to work for it.
> > Now,
> > > > quiet! They're about to announce the lottery numbers..."
> > > >
> > >
> >
> >
> >
> > --
> > "If you really want something in this life, you have to work for it. Now,
> > quiet! They're about to announce the lottery numbers..."
> >
>
Re: Copying data from 94 to 98 ..
Posted by Matteo Bertozzi <th...@gmail.com>.
can you post the full exception and the file path ?
maybe there is a bug in looking up the reference file.
It seems to not be able to find enough data in the file...
Matteo
On Mon, Sep 15, 2014 at 10:08 PM, Gautam <ga...@gmail.com> wrote:
> Thanks for the reply Matteo.
>
> This is exactly what I did. I modified the source cluster's dir structure
> to mimic that of the 98 cluster. I even got as far as it trying to look
> through the reference files.
>
> I end up with this exception :
>
> 14/09/15 23:34:59 ERROR snapshot.ExportSnapshot: Snapshot export failed
> java.io.IOException
> at
>
> org.apache.hadoop.hbase.snapshot.SnapshotManifestV1.loadRegionManifests(SnapshotManifestV1.java:145)
> at
>
> org.apache.hadoop.hbase.snapshot.SnapshotManifest.load(SnapshotManifest.java:265)
> at
>
> org.apache.hadoop.hbase.snapshot.SnapshotManifest.open(SnapshotManifest.java:119)
> at
>
> org.apache.hadoop.hbase.snapshot.SnapshotReferenceUtil.visitTableStoreFiles(SnapshotReferenceUtil.java:125)
> ...
> ..
> Caused by: java.io.IOException: read=-1, wanted=4
> at org.apache.hadoop.hbase.io.Reference.read(Reference.java:175)
> at
>
> org.apache.hadoop.hbase.regionserver.StoreFileInfo.<init>(StoreFileInfo.java:115)
> at
>
> org.apache.hadoop.hbase.regionserver.HRegionFileSystem.getStoreFiles(HRegionFileSystem.java
> ..
>
>
> This and Ted's reply about HBASE-7987 leads me to believe that the export
> tool from my distro is in capable of working around the regionManifest file
> requirement. I'm now left with the option of downgrading my dest cluster to
> 94, copying data and then upgrading using the upgrade migration tool.
> Wanted to know if others have tried this or there are other things I can
> do. If not, i'l just go ahead and do this :-)
>
>
> Cheers,
> -Gautam.
>
>
>
> On Mon, Sep 15, 2014 at 8:10 PM, Matteo Bertozzi <th...@gmail.com>
> wrote:
>
> > 94 and 98 differs in directory layout
> > so 98 is not able to read 94 layout unless you run the migration tool
> > which is basically moving all the data in a "default" namespace directory
> > e.g.
> > /hbase/table -> /hbase/data/default/table
> > /hbase/.archive/table -> /hbase/archive/default/table
> >
> > Matteo
> >
> >
> > On Mon, Sep 15, 2014 at 6:17 PM, Gautam <ga...@gmail.com> wrote:
> >
> > > Yep, looks like the CDH distro backports HBASE-7987. Having said that,
> is
> > > there a transition path for us or are we hosed :-) ? In general, what's
> > the
> > > recommended way to achieve this, at this point I feel i'm going around
> > the
> > > system to achieve what I want. If nothing else works with export
> snapshot
> > > should I just downgrade to 94, export snapshot and then upgrade to 98?
> Is
> > > the upgrade migration path different from what export snapshot does
> (i'd
> > > imagine yes)?
> > >
> > > Cheers,
> > > -Gautam.
> > >
> > >
> > >
> > >
> > > On Mon, Sep 15, 2014 at 5:14 PM, Ted Yu <yu...@gmail.com> wrote:
> > >
> > > > bq. 98.1 on dest cluster
> > > >
> > > > Looking at the history for SnapshotManifestV1, it came with
> HBASE-7987
> > > > which went to 0.99.0
> > > >
> > > > Perhaps you're using a distro with HBASE-7987 ?
> > > >
> > > > On Mon, Sep 15, 2014 at 4:58 PM, Gautam <ga...@gmail.com>
> > wrote:
> > > >
> > > > > Hello,
> > > > > I'm trying to copy data between Hbase clusters on
> different
> > > > > versions. I am using :
> > > > >
> > > > > /usr/bin/hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot
> > > > > -chuser hbase
> > > > > -chgroup hadoop
> > > > > -snapshot msg_snapshot
> > > > > -mappers 50
> > > > > -copy-from hftp://src-cluster:50070/hbase
> > > > > -copy-to hdfs:/dest-cluster:8020/hbase
> > > > >
> > > > >
> > > > > Till now, based on various tips from the mailing list, I have
> > modified
> > > > the
> > > > > source cluster data dir paths to mimic the 98 convention (archive,
> > > table
> > > > > data paths, etc). This helped in jumping some roadblocks but not
> all.
> > > > >
> > > > > This is what I see now :
> > > > >
> > > > >
> > > > > 14/09/15 23:34:59 ERROR snapshot.ExportSnapshot: Snapshot export
> > failed
> > > > > java.io.IOException
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.snapshot.SnapshotManifestV1.loadRegionManifests(SnapshotManifestV1.java:145)
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.snapshot.SnapshotManifest.load(SnapshotManifest.java:265)
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.snapshot.SnapshotManifest.open(SnapshotManifest.java:119)
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.snapshot.SnapshotReferenceUtil.visitTableStoreFiles(SnapshotReferenceUtil.java:125)
> > > > > ...
> > > > > ..
> > > > > Caused by: java.io.IOException: read=-1, wanted=4
> > > > > at org.apache.hadoop.hbase.io.Reference.read(Reference.java:175)
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.regionserver.StoreFileInfo.<init>(StoreFileInfo.java:115)
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.regionserver.HRegionFileSystem.getStoreFiles(HRegionFileSystem.java
> > > > > ..
> > > > >
> > > > >
> > > > > Fails while trying to read refernce hfile. Is this something folks
> > have
> > > > > done before and/or is possible to do? I'd really like to do this
> > > without
> > > > > having to upgrade my source cluster or downgrade my dest cluster.
> > > > >
> > > > > I'm using 94.6 on source cluster and 98.1 on dest cluster.
> > > > >
> > > > >
> > > > > Cheers,
> > > > > -Gautam.
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > "If you really want something in this life, you have to work for it.
> Now,
> > > quiet! They're about to announce the lottery numbers..."
> > >
> >
>
>
>
> --
> "If you really want something in this life, you have to work for it. Now,
> quiet! They're about to announce the lottery numbers..."
>
Re: Copying data from 94 to 98 ..
Posted by Gautam <ga...@gmail.com>.
Thanks for the reply Matteo.
This is exactly what I did. I modified the source cluster's dir structure
to mimic that of the 98 cluster. I even got as far as it trying to look
through the reference files.
I end up with this exception :
14/09/15 23:34:59 ERROR snapshot.ExportSnapshot: Snapshot export failed
java.io.IOException
at
org.apache.hadoop.hbase.snapshot.SnapshotManifestV1.loadRegionManifests(SnapshotManifestV1.java:145)
at
org.apache.hadoop.hbase.snapshot.SnapshotManifest.load(SnapshotManifest.java:265)
at
org.apache.hadoop.hbase.snapshot.SnapshotManifest.open(SnapshotManifest.java:119)
at
org.apache.hadoop.hbase.snapshot.SnapshotReferenceUtil.visitTableStoreFiles(SnapshotReferenceUtil.java:125)
...
..
Caused by: java.io.IOException: read=-1, wanted=4
at org.apache.hadoop.hbase.io.Reference.read(Reference.java:175)
at
org.apache.hadoop.hbase.regionserver.StoreFileInfo.<init>(StoreFileInfo.java:115)
at
org.apache.hadoop.hbase.regionserver.HRegionFileSystem.getStoreFiles(HRegionFileSystem.java
..
This and Ted's reply about HBASE-7987 leads me to believe that the export
tool from my distro is in capable of working around the regionManifest file
requirement. I'm now left with the option of downgrading my dest cluster to
94, copying data and then upgrading using the upgrade migration tool.
Wanted to know if others have tried this or there are other things I can
do. If not, i'l just go ahead and do this :-)
Cheers,
-Gautam.
On Mon, Sep 15, 2014 at 8:10 PM, Matteo Bertozzi <th...@gmail.com>
wrote:
> 94 and 98 differs in directory layout
> so 98 is not able to read 94 layout unless you run the migration tool
> which is basically moving all the data in a "default" namespace directory
> e.g.
> /hbase/table -> /hbase/data/default/table
> /hbase/.archive/table -> /hbase/archive/default/table
>
> Matteo
>
>
> On Mon, Sep 15, 2014 at 6:17 PM, Gautam <ga...@gmail.com> wrote:
>
> > Yep, looks like the CDH distro backports HBASE-7987. Having said that, is
> > there a transition path for us or are we hosed :-) ? In general, what's
> the
> > recommended way to achieve this, at this point I feel i'm going around
> the
> > system to achieve what I want. If nothing else works with export snapshot
> > should I just downgrade to 94, export snapshot and then upgrade to 98? Is
> > the upgrade migration path different from what export snapshot does (i'd
> > imagine yes)?
> >
> > Cheers,
> > -Gautam.
> >
> >
> >
> >
> > On Mon, Sep 15, 2014 at 5:14 PM, Ted Yu <yu...@gmail.com> wrote:
> >
> > > bq. 98.1 on dest cluster
> > >
> > > Looking at the history for SnapshotManifestV1, it came with HBASE-7987
> > > which went to 0.99.0
> > >
> > > Perhaps you're using a distro with HBASE-7987 ?
> > >
> > > On Mon, Sep 15, 2014 at 4:58 PM, Gautam <ga...@gmail.com>
> wrote:
> > >
> > > > Hello,
> > > > I'm trying to copy data between Hbase clusters on different
> > > > versions. I am using :
> > > >
> > > > /usr/bin/hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot
> > > > -chuser hbase
> > > > -chgroup hadoop
> > > > -snapshot msg_snapshot
> > > > -mappers 50
> > > > -copy-from hftp://src-cluster:50070/hbase
> > > > -copy-to hdfs:/dest-cluster:8020/hbase
> > > >
> > > >
> > > > Till now, based on various tips from the mailing list, I have
> modified
> > > the
> > > > source cluster data dir paths to mimic the 98 convention (archive,
> > table
> > > > data paths, etc). This helped in jumping some roadblocks but not all.
> > > >
> > > > This is what I see now :
> > > >
> > > >
> > > > 14/09/15 23:34:59 ERROR snapshot.ExportSnapshot: Snapshot export
> failed
> > > > java.io.IOException
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.snapshot.SnapshotManifestV1.loadRegionManifests(SnapshotManifestV1.java:145)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.snapshot.SnapshotManifest.load(SnapshotManifest.java:265)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.snapshot.SnapshotManifest.open(SnapshotManifest.java:119)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.snapshot.SnapshotReferenceUtil.visitTableStoreFiles(SnapshotReferenceUtil.java:125)
> > > > ...
> > > > ..
> > > > Caused by: java.io.IOException: read=-1, wanted=4
> > > > at org.apache.hadoop.hbase.io.Reference.read(Reference.java:175)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.regionserver.StoreFileInfo.<init>(StoreFileInfo.java:115)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.regionserver.HRegionFileSystem.getStoreFiles(HRegionFileSystem.java
> > > > ..
> > > >
> > > >
> > > > Fails while trying to read refernce hfile. Is this something folks
> have
> > > > done before and/or is possible to do? I'd really like to do this
> > without
> > > > having to upgrade my source cluster or downgrade my dest cluster.
> > > >
> > > > I'm using 94.6 on source cluster and 98.1 on dest cluster.
> > > >
> > > >
> > > > Cheers,
> > > > -Gautam.
> > > >
> > >
> >
> >
> >
> > --
> > "If you really want something in this life, you have to work for it. Now,
> > quiet! They're about to announce the lottery numbers..."
> >
>
--
"If you really want something in this life, you have to work for it. Now,
quiet! They're about to announce the lottery numbers..."
Re: Copying data from 94 to 98 ..
Posted by Matteo Bertozzi <th...@gmail.com>.
94 and 98 differs in directory layout
so 98 is not able to read 94 layout unless you run the migration tool
which is basically moving all the data in a "default" namespace directory
e.g.
/hbase/table -> /hbase/data/default/table
/hbase/.archive/table -> /hbase/archive/default/table
Matteo
On Mon, Sep 15, 2014 at 6:17 PM, Gautam <ga...@gmail.com> wrote:
> Yep, looks like the CDH distro backports HBASE-7987. Having said that, is
> there a transition path for us or are we hosed :-) ? In general, what's the
> recommended way to achieve this, at this point I feel i'm going around the
> system to achieve what I want. If nothing else works with export snapshot
> should I just downgrade to 94, export snapshot and then upgrade to 98? Is
> the upgrade migration path different from what export snapshot does (i'd
> imagine yes)?
>
> Cheers,
> -Gautam.
>
>
>
>
> On Mon, Sep 15, 2014 at 5:14 PM, Ted Yu <yu...@gmail.com> wrote:
>
> > bq. 98.1 on dest cluster
> >
> > Looking at the history for SnapshotManifestV1, it came with HBASE-7987
> > which went to 0.99.0
> >
> > Perhaps you're using a distro with HBASE-7987 ?
> >
> > On Mon, Sep 15, 2014 at 4:58 PM, Gautam <ga...@gmail.com> wrote:
> >
> > > Hello,
> > > I'm trying to copy data between Hbase clusters on different
> > > versions. I am using :
> > >
> > > /usr/bin/hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot
> > > -chuser hbase
> > > -chgroup hadoop
> > > -snapshot msg_snapshot
> > > -mappers 50
> > > -copy-from hftp://src-cluster:50070/hbase
> > > -copy-to hdfs:/dest-cluster:8020/hbase
> > >
> > >
> > > Till now, based on various tips from the mailing list, I have modified
> > the
> > > source cluster data dir paths to mimic the 98 convention (archive,
> table
> > > data paths, etc). This helped in jumping some roadblocks but not all.
> > >
> > > This is what I see now :
> > >
> > >
> > > 14/09/15 23:34:59 ERROR snapshot.ExportSnapshot: Snapshot export failed
> > > java.io.IOException
> > > at
> > >
> > >
> >
> org.apache.hadoop.hbase.snapshot.SnapshotManifestV1.loadRegionManifests(SnapshotManifestV1.java:145)
> > > at
> > >
> > >
> >
> org.apache.hadoop.hbase.snapshot.SnapshotManifest.load(SnapshotManifest.java:265)
> > > at
> > >
> > >
> >
> org.apache.hadoop.hbase.snapshot.SnapshotManifest.open(SnapshotManifest.java:119)
> > > at
> > >
> > >
> >
> org.apache.hadoop.hbase.snapshot.SnapshotReferenceUtil.visitTableStoreFiles(SnapshotReferenceUtil.java:125)
> > > ...
> > > ..
> > > Caused by: java.io.IOException: read=-1, wanted=4
> > > at org.apache.hadoop.hbase.io.Reference.read(Reference.java:175)
> > > at
> > >
> > >
> >
> org.apache.hadoop.hbase.regionserver.StoreFileInfo.<init>(StoreFileInfo.java:115)
> > > at
> > >
> > >
> >
> org.apache.hadoop.hbase.regionserver.HRegionFileSystem.getStoreFiles(HRegionFileSystem.java
> > > ..
> > >
> > >
> > > Fails while trying to read refernce hfile. Is this something folks have
> > > done before and/or is possible to do? I'd really like to do this
> without
> > > having to upgrade my source cluster or downgrade my dest cluster.
> > >
> > > I'm using 94.6 on source cluster and 98.1 on dest cluster.
> > >
> > >
> > > Cheers,
> > > -Gautam.
> > >
> >
>
>
>
> --
> "If you really want something in this life, you have to work for it. Now,
> quiet! They're about to announce the lottery numbers..."
>
Re: Copying data from 94 to 98 ..
Posted by Gautam <ga...@gmail.com>.
Yep, looks like the CDH distro backports HBASE-7987. Having said that, is
there a transition path for us or are we hosed :-) ? In general, what's the
recommended way to achieve this, at this point I feel i'm going around the
system to achieve what I want. If nothing else works with export snapshot
should I just downgrade to 94, export snapshot and then upgrade to 98? Is
the upgrade migration path different from what export snapshot does (i'd
imagine yes)?
Cheers,
-Gautam.
On Mon, Sep 15, 2014 at 5:14 PM, Ted Yu <yu...@gmail.com> wrote:
> bq. 98.1 on dest cluster
>
> Looking at the history for SnapshotManifestV1, it came with HBASE-7987
> which went to 0.99.0
>
> Perhaps you're using a distro with HBASE-7987 ?
>
> On Mon, Sep 15, 2014 at 4:58 PM, Gautam <ga...@gmail.com> wrote:
>
> > Hello,
> > I'm trying to copy data between Hbase clusters on different
> > versions. I am using :
> >
> > /usr/bin/hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot
> > -chuser hbase
> > -chgroup hadoop
> > -snapshot msg_snapshot
> > -mappers 50
> > -copy-from hftp://src-cluster:50070/hbase
> > -copy-to hdfs:/dest-cluster:8020/hbase
> >
> >
> > Till now, based on various tips from the mailing list, I have modified
> the
> > source cluster data dir paths to mimic the 98 convention (archive, table
> > data paths, etc). This helped in jumping some roadblocks but not all.
> >
> > This is what I see now :
> >
> >
> > 14/09/15 23:34:59 ERROR snapshot.ExportSnapshot: Snapshot export failed
> > java.io.IOException
> > at
> >
> >
> org.apache.hadoop.hbase.snapshot.SnapshotManifestV1.loadRegionManifests(SnapshotManifestV1.java:145)
> > at
> >
> >
> org.apache.hadoop.hbase.snapshot.SnapshotManifest.load(SnapshotManifest.java:265)
> > at
> >
> >
> org.apache.hadoop.hbase.snapshot.SnapshotManifest.open(SnapshotManifest.java:119)
> > at
> >
> >
> org.apache.hadoop.hbase.snapshot.SnapshotReferenceUtil.visitTableStoreFiles(SnapshotReferenceUtil.java:125)
> > ...
> > ..
> > Caused by: java.io.IOException: read=-1, wanted=4
> > at org.apache.hadoop.hbase.io.Reference.read(Reference.java:175)
> > at
> >
> >
> org.apache.hadoop.hbase.regionserver.StoreFileInfo.<init>(StoreFileInfo.java:115)
> > at
> >
> >
> org.apache.hadoop.hbase.regionserver.HRegionFileSystem.getStoreFiles(HRegionFileSystem.java
> > ..
> >
> >
> > Fails while trying to read refernce hfile. Is this something folks have
> > done before and/or is possible to do? I'd really like to do this without
> > having to upgrade my source cluster or downgrade my dest cluster.
> >
> > I'm using 94.6 on source cluster and 98.1 on dest cluster.
> >
> >
> > Cheers,
> > -Gautam.
> >
>
--
"If you really want something in this life, you have to work for it. Now,
quiet! They're about to announce the lottery numbers..."
Re: Copying data from 94 to 98 ..
Posted by Ted Yu <yu...@gmail.com>.
bq. 98.1 on dest cluster
Looking at the history for SnapshotManifestV1, it came with HBASE-7987
which went to 0.99.0
Perhaps you're using a distro with HBASE-7987 ?
On Mon, Sep 15, 2014 at 4:58 PM, Gautam <ga...@gmail.com> wrote:
> Hello,
> I'm trying to copy data between Hbase clusters on different
> versions. I am using :
>
> /usr/bin/hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot
> -chuser hbase
> -chgroup hadoop
> -snapshot msg_snapshot
> -mappers 50
> -copy-from hftp://src-cluster:50070/hbase
> -copy-to hdfs:/dest-cluster:8020/hbase
>
>
> Till now, based on various tips from the mailing list, I have modified the
> source cluster data dir paths to mimic the 98 convention (archive, table
> data paths, etc). This helped in jumping some roadblocks but not all.
>
> This is what I see now :
>
>
> 14/09/15 23:34:59 ERROR snapshot.ExportSnapshot: Snapshot export failed
> java.io.IOException
> at
>
> org.apache.hadoop.hbase.snapshot.SnapshotManifestV1.loadRegionManifests(SnapshotManifestV1.java:145)
> at
>
> org.apache.hadoop.hbase.snapshot.SnapshotManifest.load(SnapshotManifest.java:265)
> at
>
> org.apache.hadoop.hbase.snapshot.SnapshotManifest.open(SnapshotManifest.java:119)
> at
>
> org.apache.hadoop.hbase.snapshot.SnapshotReferenceUtil.visitTableStoreFiles(SnapshotReferenceUtil.java:125)
> ...
> ..
> Caused by: java.io.IOException: read=-1, wanted=4
> at org.apache.hadoop.hbase.io.Reference.read(Reference.java:175)
> at
>
> org.apache.hadoop.hbase.regionserver.StoreFileInfo.<init>(StoreFileInfo.java:115)
> at
>
> org.apache.hadoop.hbase.regionserver.HRegionFileSystem.getStoreFiles(HRegionFileSystem.java
> ..
>
>
> Fails while trying to read refernce hfile. Is this something folks have
> done before and/or is possible to do? I'd really like to do this without
> having to upgrade my source cluster or downgrade my dest cluster.
>
> I'm using 94.6 on source cluster and 98.1 on dest cluster.
>
>
> Cheers,
> -Gautam.
>
Re: Copying data from 94 to 98 ..
Posted by Gautam <ga...@gmail.com>.
To be clear I am running this command on the dest cluster.
-G.
On Mon, Sep 15, 2014 at 4:58 PM, Gautam <ga...@gmail.com> wrote:
> Hello,
> I'm trying to copy data between Hbase clusters on different
> versions. I am using :
>
> /usr/bin/hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot
> -chuser hbase
> -chgroup hadoop
> -snapshot msg_snapshot
> -mappers 50
> -copy-from hftp://src-cluster:50070/hbase
> -copy-to hdfs:/dest-cluster:8020/hbase
>
>
> Till now, based on various tips from the mailing list, I have modified the
> source cluster data dir paths to mimic the 98 convention (archive, table
> data paths, etc). This helped in jumping some roadblocks but not all.
>
> This is what I see now :
>
>
> 14/09/15 23:34:59 ERROR snapshot.ExportSnapshot: Snapshot export failed
> java.io.IOException
> at
> org.apache.hadoop.hbase.snapshot.SnapshotManifestV1.loadRegionManifests(SnapshotManifestV1.java:145)
> at
> org.apache.hadoop.hbase.snapshot.SnapshotManifest.load(SnapshotManifest.java:265)
> at
> org.apache.hadoop.hbase.snapshot.SnapshotManifest.open(SnapshotManifest.java:119)
> at
> org.apache.hadoop.hbase.snapshot.SnapshotReferenceUtil.visitTableStoreFiles(SnapshotReferenceUtil.java:125)
> ...
> ..
> Caused by: java.io.IOException: read=-1, wanted=4
> at org.apache.hadoop.hbase.io.Reference.read(Reference.java:175)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileInfo.<init>(StoreFileInfo.java:115)
> at
> org.apache.hadoop.hbase.regionserver.HRegionFileSystem.getStoreFiles(HRegionFileSystem.java
> ..
>
>
> Fails while trying to read refernce hfile. Is this something folks have
> done before and/or is possible to do? I'd really like to do this without
> having to upgrade my source cluster or downgrade my dest cluster.
>
> I'm using 94.6 on source cluster and 98.1 on dest cluster.
>
>
> Cheers,
> -Gautam.
>
--
"If you really want something in this life, you have to work for it. Now,
quiet! They're about to announce the lottery numbers..."