You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Cooper Bethea <co...@siftscience.com> on 2013/10/29 23:06:18 UTC
snapshot export failure
Hi everybody,
I've been attempting to export a snapshot from one of my HBase clusters to
the other. Both clusters are running 0.94.6-cdh4.4.0. I am able to
successfully export a very small collection (few MB), but when I attempt to
export a larger collection something odd happens. The Export MR finishes
successfully, but leaves files in /hbase/.snapshot/.tmp/SNAPSHOT_NAME
instead of /hbase/.snapshot/SNAPSHOT_NAME. There are no errors in the logs.
Is this familiar to anyone?
Re: snapshot export failure
Posted by Cooper Bethea <co...@siftscience.com>.
I re-ran ExportSnapshot; it completed successfully and I am now able to
clone the snapshot to a new table. No errors reported. Here's the output.
¯\_(ツ)_/¯
hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot -snapshot
grains_1383068263 -copy-to hdfs://experiment-b-name-1d:8020/hbase -mappers 6
2013-10-29/22:47:58.986 UTC [main] org.apache.hadoop.conf.Configuration
[WARN] - hadoop.native.lib is deprecated. Instead, use
io.native.lib.available
2013-10-29/22:48:09.157 UTC [main] org.apache.hadoop.hbase.util.FSVisitor
[INFO] - No logs under
directory:hdfs://production-b/hbase/.snapshot/grains_1383068263/.logs
2013-10-29/22:49:37.642 UTC [main]
org.apache.hadoop.io.compress.zlib.ZlibFactory [INFO] - Successfully loaded
& initialized native-zlib library
2013-10-29/22:49:37.644 UTC [main] org.apache.hadoop.io.compress.CodecPool
[INFO] - Got brand-new compressor [.deflate]
2013-10-29/22:49:38.210 UTC [main] org.apache.hadoop.mapred.JobClient
[WARN] - Use GenericOptionsParser for parsing the arguments. Applications
should implement Tool for the same.
2013-10-29/22:49:38.432 UTC [main]
org.apache.hadoop.mapreduce.lib.input.FileInputFormat [INFO] - Total input
paths to process : 6
2013-10-29/22:49:38.805 UTC [main] org.apache.hadoop.mapred.JobClient
[INFO] - Running job: job_201310241813_0018
2013-10-29/22:49:39.811 UTC [main] org.apache.hadoop.mapred.JobClient
[INFO] - map 0% reduce 0%
2013-10-29/22:50:18.891 UTC [main] org.apache.hadoop.mapred.JobClient
[INFO] - map 1% reduce 0%
2013-10-29/22:50:48.937 UTC [main] org.apache.hadoop.mapred.JobClient
[INFO] - map 2% reduce 0%
2013-10-29/22:51:21.986 UTC [main] org.apache.hadoop.mapred.JobClient
[INFO] - map 3% reduce 0%
2013-10-29/22:51:59.037 UTC [main] org.apache.hadoop.mapred.JobClient
[INFO] - map 4% reduce 0%
2013-10-29/22:52:34.088 UTC [main] org.apache.hadoop.mapred.JobClient
[INFO] - map 5% reduce 0%
2013-10-29/22:53:04.128 UTC [main] org.apache.hadoop.mapred.JobClient
[INFO] - map 6% reduce 0%
2013-10-29/22:53:30.164 UTC [main] org.apache.hadoop.mapred.JobClient
[INFO] - map 7% reduce 0%
2013-10-29/22:53:46.185 UTC [main] org.apache.hadoop.mapred.JobClient
[INFO] - map 8% reduce 0%
2013-10-29/22:53:56.201 UTC [main] org.apache.hadoop.mapred.JobClient
[INFO] - map 9% reduce 0%
2013-10-29/22:54:05.215 UTC [main] org.apache.hadoop.mapred.JobClient
[INFO] - map 10% reduce 0%
2013-10-29/22:54:14.228 UTC [main] org.apache.hadoop.mapred.JobClient
[INFO] - map 11% reduce 0%
2013-10-29/22:54:23.242 UTC [main] org.apache.hadoop.mapred.JobClient
[INFO] - map 12% reduce 0%
2013-10-29/22:54:29.250 UTC [main] org.apache.hadoop.mapred.JobClient
[INFO] - map 13% reduce 0%
2013-10-29/22:54:32.254 UTC [main] org.apache.hadoop.mapred.JobClient
[INFO] - map 14% reduce 0%
2013-10-29/22:54:38.264 UTC [main] org.apache.hadoop.mapred.JobClient
[INFO] - map 15% reduce 0%
2013-10-29/22:54:41.268 UTC [main] org.apache.hadoop.mapred.JobClient
[INFO] - map 16% reduce 0%
2013-10-29/22:54:44.276 UTC [main] org.apache.hadoop.mapred.JobClient
[INFO] - map 17% reduce 0%
2013-10-29/22:54:47.280 UTC [main] org.apache.hadoop.mapred.JobClient
[INFO] - map 19% reduce 0%
2013-10-29/22:54:50.284 UTC [main] org.apache.hadoop.mapred.JobClient
[INFO] - map 20% reduce 0%
2013-10-29/22:54:53.289 UTC [main] org.apache.hadoop.mapred.JobClient
[INFO] - map 21% reduce 0%
2013-10-29/22:54:56.294 UTC [main] org.apache.hadoop.mapred.JobClient
[INFO] - map 23% reduce 0%
2013-10-29/22:54:59.298 UTC [main] org.apache.hadoop.mapred.JobClient
[INFO] - map 25% reduce 0%
2013-10-29/22:55:02.303 UTC [main] org.apache.hadoop.mapred.JobClient
[INFO] - map 27% reduce 0%
2013-10-29/22:55:05.308 UTC [main] org.apache.hadoop.mapred.JobClient
[INFO] - map 29% reduce 0%
2013-10-29/22:55:08.313 UTC [main] org.apache.hadoop.mapred.JobClient
[INFO] - map 31% reduce 0%
2013-10-29/22:55:11.318 UTC [main] org.apache.hadoop.mapred.JobClient
[INFO] - map 34% reduce 0%
2013-10-29/22:55:14.322 UTC [main] org.apache.hadoop.mapred.JobClient
[INFO] - map 36% reduce 0%
2013-10-29/22:55:16.326 UTC [main] org.apache.hadoop.mapred.JobClient
[INFO] - map 37% reduce 0%
2013-10-29/22:55:17.328 UTC [main] org.apache.hadoop.mapred.JobClient
[INFO] - map 39% reduce 0%
2013-10-29/22:55:20.333 UTC [main] org.apache.hadoop.mapred.JobClient
[INFO] - map 41% reduce 0%
2013-10-29/22:55:23.337 UTC [main] org.apache.hadoop.mapred.JobClient
[INFO] - map 44% reduce 0%
2013-10-29/22:55:26.342 UTC [main] org.apache.hadoop.mapred.JobClient
[INFO] - map 46% reduce 0%
2013-10-29/22:55:29.347 UTC [main] org.apache.hadoop.mapred.JobClient
[INFO] - map 49% reduce 0%
2013-10-29/22:55:32.352 UTC [main] org.apache.hadoop.mapred.JobClient
[INFO] - map 52% reduce 0%
2013-10-29/22:55:35.357 UTC [main] org.apache.hadoop.mapred.JobClient
[INFO] - map 54% reduce 0%
2013-10-29/22:55:37.360 UTC [main] org.apache.hadoop.mapred.JobClient
[INFO] - map 55% reduce 0%
2013-10-29/22:55:38.362 UTC [main] org.apache.hadoop.mapred.JobClient
[INFO] - map 56% reduce 0%
2013-10-29/22:55:40.365 UTC [main] org.apache.hadoop.mapred.JobClient
[INFO] - map 57% reduce 0%
2013-10-29/22:55:41.367 UTC [main] org.apache.hadoop.mapred.JobClient
[INFO] - map 59% reduce 0%
2013-10-29/22:55:44.372 UTC [main] org.apache.hadoop.mapred.JobClient
[INFO] - map 62% reduce 0%
2013-10-29/22:55:47.377 UTC [main] org.apache.hadoop.mapred.JobClient
[INFO] - map 65% reduce 0%
2013-10-29/22:55:50.381 UTC [main] org.apache.hadoop.mapred.JobClient
[INFO] - map 68% reduce 0%
2013-10-29/22:55:53.386 UTC [main] org.apache.hadoop.mapred.JobClient
[INFO] - map 71% reduce 0%
2013-10-29/22:55:55.390 UTC [main] org.apache.hadoop.mapred.JobClient
[INFO] - map 72% reduce 0%
2013-10-29/22:55:58.394 UTC [main] org.apache.hadoop.mapred.JobClient
[INFO] - map 73% reduce 0%
2013-10-29/22:56:08.408 UTC [main] org.apache.hadoop.mapred.JobClient
[INFO] - map 74% reduce 0%
2013-10-29/22:56:11.412 UTC [main] org.apache.hadoop.mapred.JobClient
[INFO] - map 75% reduce 0%
2013-10-29/22:56:14.417 UTC [main] org.apache.hadoop.mapred.JobClient
[INFO] - map 76% reduce 0%
2013-10-29/22:56:16.420 UTC [main] org.apache.hadoop.mapred.JobClient
[INFO] - map 77% reduce 0%
2013-10-29/22:56:19.425 UTC [main] org.apache.hadoop.mapred.JobClient
[INFO] - map 79% reduce 0%
2013-10-29/22:56:23.434 UTC [main] org.apache.hadoop.mapred.JobClient
[INFO] - map 81% reduce 0%
2013-10-29/22:56:26.439 UTC [main] org.apache.hadoop.mapred.JobClient
[INFO] - map 84% reduce 0%
2013-10-29/22:56:29.444 UTC [main] org.apache.hadoop.mapred.JobClient
[INFO] - map 86% reduce 0%
2013-10-29/22:56:32.449 UTC [main] org.apache.hadoop.mapred.JobClient
[INFO] - map 89% reduce 0%
2013-10-29/22:56:35.454 UTC [main] org.apache.hadoop.mapred.JobClient
[INFO] - map 92% reduce 0%
2013-10-29/22:56:38.459 UTC [main] org.apache.hadoop.mapred.JobClient
[INFO] - map 94% reduce 0%
2013-10-29/22:56:41.464 UTC [main] org.apache.hadoop.mapred.JobClient
[INFO] - map 97% reduce 0%
2013-10-29/22:56:44.468 UTC [main] org.apache.hadoop.mapred.JobClient
[INFO] - map 99% reduce 0%
2013-10-29/22:56:47.474 UTC [main] org.apache.hadoop.mapred.JobClient
[INFO] - map 100% reduce 0%
2013-10-29/22:56:48.477 UTC [main] org.apache.hadoop.mapred.JobClient
[INFO] - Job complete: job_201310241813_0018
2013-10-29/22:56:48.494 UTC [main] org.apache.hadoop.mapred.Counters [INFO]
- Counters: 25
2013-10-29/22:56:48.498 UTC [main] org.apache.hadoop.mapred.Counters [INFO]
- File System Counters
2013-10-29/22:56:48.501 UTC [main] org.apache.hadoop.mapred.Counters [INFO]
- FILE: Number of bytes read=0
2013-10-29/22:56:48.501 UTC [main] org.apache.hadoop.mapred.Counters [INFO]
- FILE: Number of bytes written=1307244
2013-10-29/22:56:48.502 UTC [main] org.apache.hadoop.mapred.Counters [INFO]
- FILE: Number of read operations=0
2013-10-29/22:56:48.502 UTC [main] org.apache.hadoop.mapred.Counters [INFO]
- FILE: Number of large read operations=0
2013-10-29/22:56:48.502 UTC [main] org.apache.hadoop.mapred.Counters [INFO]
- FILE: Number of write operations=0
2013-10-29/22:56:48.502 UTC [main] org.apache.hadoop.mapred.Counters [INFO]
- HDFS: Number of bytes read=466490
2013-10-29/22:56:48.503 UTC [main] org.apache.hadoop.mapred.Counters [INFO]
- HDFS: Number of bytes written=0
2013-10-29/22:56:48.503 UTC [main] org.apache.hadoop.mapred.Counters [INFO]
- HDFS: Number of read operations=43906
2013-10-29/22:56:48.503 UTC [main] org.apache.hadoop.mapred.Counters [INFO]
- HDFS: Number of large read operations=0
2013-10-29/22:56:48.503 UTC [main] org.apache.hadoop.mapred.Counters [INFO]
- HDFS: Number of write operations=0
2013-10-29/22:56:48.506 UTC [main] org.apache.hadoop.mapred.Counters [INFO]
- Job Counters
2013-10-29/22:56:48.507 UTC [main] org.apache.hadoop.mapred.Counters [INFO]
- Launched map tasks=6
2013-10-29/22:56:48.508 UTC [main] org.apache.hadoop.mapred.Counters [INFO]
- Data-local map tasks=4
2013-10-29/22:56:48.508 UTC [main] org.apache.hadoop.mapred.Counters [INFO]
- Rack-local map tasks=2
2013-10-29/22:56:48.508 UTC [main] org.apache.hadoop.mapred.Counters [INFO]
- Total time spent by all maps in occupied slots (ms)=2218597
2013-10-29/22:56:48.508 UTC [main] org.apache.hadoop.mapred.Counters [INFO]
- Total time spent by all reduces in occupied slots (ms)=0
2013-10-29/22:56:48.509 UTC [main] org.apache.hadoop.mapred.Counters [INFO]
- Total time spent by all maps waiting after reserving slots (ms)=0
2013-10-29/22:56:48.509 UTC [main] org.apache.hadoop.mapred.Counters [INFO]
- Total time spent by all reduces waiting after reserving slots (ms)=0
2013-10-29/22:56:48.512 UTC [main] org.apache.hadoop.mapred.Counters [INFO]
- Map-Reduce Framework
2013-10-29/22:56:48.512 UTC [main] org.apache.hadoop.mapred.Counters [INFO]
- Map input records=5010
2013-10-29/22:56:48.512 UTC [main] org.apache.hadoop.mapred.Counters [INFO]
- Map output records=0
2013-10-29/22:56:48.512 UTC [main] org.apache.hadoop.mapred.Counters [INFO]
- Input split bytes=990
2013-10-29/22:56:48.513 UTC [main] org.apache.hadoop.mapred.Counters [INFO]
- Spilled Records=0
2013-10-29/22:56:48.513 UTC [main] org.apache.hadoop.mapred.Counters [INFO]
- CPU time spent (ms)=134140
2013-10-29/22:56:48.513 UTC [main] org.apache.hadoop.mapred.Counters [INFO]
- Physical memory (bytes) snapshot=1306460160
2013-10-29/22:56:48.513 UTC [main] org.apache.hadoop.mapred.Counters [INFO]
- Virtual memory (bytes) snapshot=9209536512
2013-10-29/22:56:48.514 UTC [main] org.apache.hadoop.mapred.Counters [INFO]
- Total committed heap usage (bytes)=2658074624
On Tue, Oct 29, 2013 at 3:35 PM, Matteo Bertozzi <th...@gmail.com>wrote:
> could you re-run the ExportSnapshot and post the output?
> also could you check the exit code? echo $?
>
>
> Matteo
>
>
>
> On Tue, Oct 29, 2013 at 10:29 PM, Cooper Bethea <coops@siftscience.com
> >wrote:
>
> > Ted, thanks for responding.
> >
> > I've checked the namenode log. There are no messages at greater than INFO
> > level emitted by the namenode during the period when the copy ran.
> >
> > The Export job tasks likewise emitted no errors.
> >
> >
> > On Tue, Oct 29, 2013 at 3:09 PM, Ted Yu <yu...@gmail.com> wrote:
> >
> > > bq. There are no errors in the logs.
> > >
> > > I assume you have checked master log / region server logs.
> > > How about namenode log ?
> > >
> > >
> > > On Tue, Oct 29, 2013 at 3:06 PM, Cooper Bethea <coops@siftscience.com
> > > >wrote:
> > >
> > > > Hi everybody,
> > > >
> > > > I've been attempting to export a snapshot from one of my HBase
> clusters
> > > to
> > > > the other. Both clusters are running 0.94.6-cdh4.4.0. I am able to
> > > > successfully export a very small collection (few MB), but when I
> > attempt
> > > to
> > > > export a larger collection something odd happens. The Export MR
> > finishes
> > > > successfully, but leaves files in /hbase/.snapshot/.tmp/SNAPSHOT_NAME
> > > > instead of /hbase/.snapshot/SNAPSHOT_NAME. There are no errors in the
> > > logs.
> > > >
> > > > Is this familiar to anyone?
> > > >
> > >
> >
>
Re: snapshot export failure
Posted by Matteo Bertozzi <th...@gmail.com>.
could you re-run the ExportSnapshot and post the output?
also could you check the exit code? echo $?
Matteo
On Tue, Oct 29, 2013 at 10:29 PM, Cooper Bethea <co...@siftscience.com>wrote:
> Ted, thanks for responding.
>
> I've checked the namenode log. There are no messages at greater than INFO
> level emitted by the namenode during the period when the copy ran.
>
> The Export job tasks likewise emitted no errors.
>
>
> On Tue, Oct 29, 2013 at 3:09 PM, Ted Yu <yu...@gmail.com> wrote:
>
> > bq. There are no errors in the logs.
> >
> > I assume you have checked master log / region server logs.
> > How about namenode log ?
> >
> >
> > On Tue, Oct 29, 2013 at 3:06 PM, Cooper Bethea <coops@siftscience.com
> > >wrote:
> >
> > > Hi everybody,
> > >
> > > I've been attempting to export a snapshot from one of my HBase clusters
> > to
> > > the other. Both clusters are running 0.94.6-cdh4.4.0. I am able to
> > > successfully export a very small collection (few MB), but when I
> attempt
> > to
> > > export a larger collection something odd happens. The Export MR
> finishes
> > > successfully, but leaves files in /hbase/.snapshot/.tmp/SNAPSHOT_NAME
> > > instead of /hbase/.snapshot/SNAPSHOT_NAME. There are no errors in the
> > logs.
> > >
> > > Is this familiar to anyone?
> > >
> >
>
Re: snapshot export failure
Posted by Cooper Bethea <co...@siftscience.com>.
Ted, thanks for responding.
I've checked the namenode log. There are no messages at greater than INFO
level emitted by the namenode during the period when the copy ran.
The Export job tasks likewise emitted no errors.
On Tue, Oct 29, 2013 at 3:09 PM, Ted Yu <yu...@gmail.com> wrote:
> bq. There are no errors in the logs.
>
> I assume you have checked master log / region server logs.
> How about namenode log ?
>
>
> On Tue, Oct 29, 2013 at 3:06 PM, Cooper Bethea <coops@siftscience.com
> >wrote:
>
> > Hi everybody,
> >
> > I've been attempting to export a snapshot from one of my HBase clusters
> to
> > the other. Both clusters are running 0.94.6-cdh4.4.0. I am able to
> > successfully export a very small collection (few MB), but when I attempt
> to
> > export a larger collection something odd happens. The Export MR finishes
> > successfully, but leaves files in /hbase/.snapshot/.tmp/SNAPSHOT_NAME
> > instead of /hbase/.snapshot/SNAPSHOT_NAME. There are no errors in the
> logs.
> >
> > Is this familiar to anyone?
> >
>
Re: snapshot export failure
Posted by Ted Yu <yu...@gmail.com>.
bq. There are no errors in the logs.
I assume you have checked master log / region server logs.
How about namenode log ?
On Tue, Oct 29, 2013 at 3:06 PM, Cooper Bethea <co...@siftscience.com>wrote:
> Hi everybody,
>
> I've been attempting to export a snapshot from one of my HBase clusters to
> the other. Both clusters are running 0.94.6-cdh4.4.0. I am able to
> successfully export a very small collection (few MB), but when I attempt to
> export a larger collection something odd happens. The Export MR finishes
> successfully, but leaves files in /hbase/.snapshot/.tmp/SNAPSHOT_NAME
> instead of /hbase/.snapshot/SNAPSHOT_NAME. There are no errors in the logs.
>
> Is this familiar to anyone?
>