You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Hansi Klose <ha...@web.de> on 2014/04/14 12:06:12 UTC
replication verifyrep
Hi,
I wrote a little script which should control the running replication.
The script is triggered by cron and executes the following command with the actual time stamp in "endtime" and
a time stamp => endtime - 10800000 milli seconds. So the time frame is 3 hours.
hadoop jar /usr/lib/hbase/hbase.jar verifyrep --starttime=1397217601927 --endtime=1397228401927 --families=t 1 tablename 2>&1
After some running's the script found some BADROWS.
14/04/11 17:04:05 INFO mapred.JobClient: BADROWS=176
14/04/11 17:04:05 INFO mapred.JobClient: GOODROWS=2
I executed the same command 20 Minutes later in the shell and got :
hadoop jar /usr/lib/hbase/hbase.jar verifyrep --starttime=1397217601927 --endtime=1397228401927 --families=t 1 tablename 2>&1
14/04/11 17:21:03 INFO mapred.JobClient: BADROWS=178
After that I run the command with the same start time and the actual timestamp an end time, so the time frame is greater
but with the same start time. And now I got :
hadoop jar /usr/lib/hbase/hbase.jar verifyrep --starttime=1397217601927 --endtime=1397230074876 --families=t 1 tablename 2>&1
14/04/11 17:28:28 INFO mapred.JobClient: GOODROWS=184
Is there something wrong with the command?
In our metrics i could not see that three is an Issue at that time.
We are a little bit confused about the endtime. In all documents they talk about stoptime.
But we found that in the job configuration there is no parameter called stoptime.
We found the "verifyrep.startTime" which hold the value of the starttime in our command and
"verifyrep.endTime" which is alway 0 when we use stoptime in the command.
So we decided to use endtime
Even in the code http://hbase.apache.org/xref/org/apache/hadoop/hbase/mapreduce/replication/VerifyReplication.html
they use: "static long endTime = Long.MAX_VALUE;"
Which name is the right on? endtime or stoptime?
We use cdh 4.2.0.
Regards Hansi
Aw: Re: Re: replication verifyrep
Posted by Hansi Klose <ha...@web.de>.
Thank you Jean-Daniel,
i found the logs.
Regards Hansi
> Gesendet: Dienstag, 15. April 2014 um 18:34 Uhr
> Von: "Jean-Daniel Cryans" <jd...@apache.org>
> An: "user@hbase.apache.org" <us...@hbase.apache.org>
> Betreff: Re: Re: replication verifyrep
>
> On Tue, Apr 15, 2014 at 12:17 AM, Hansi Klose <ha...@web.de> wrote:
>
> > Hi Jean-Daniel,
> >
> > thank you for your answer and bring some light into the darkness.
> >
>
> You're welcome!
>
> >
> > > You can see the bad rows listed in the user logs for your MR job.
> >
> > What log do you mean. The output from the command line?
> > I only see the count of GOOD or BAD rows.
> > Are the bad rows listed in that log which are not replicated?
> >
>
> You started VerifyReplication via "hadoop jar", so it's a MapReduce job. Go
> to your JobTracker's web UI, you should see your jobs there, then checkout
> one of them and click on one of the completed maps then look for the log.
> The bad rows are listed in that output.
>
> J-D
>
Re: Re: replication verifyrep
Posted by Jean-Daniel Cryans <jd...@apache.org>.
On Tue, Apr 15, 2014 at 12:17 AM, Hansi Klose <ha...@web.de> wrote:
> Hi Jean-Daniel,
>
> thank you for your answer and bring some light into the darkness.
>
You're welcome!
>
> > You can see the bad rows listed in the user logs for your MR job.
>
> What log do you mean. The output from the command line?
> I only see the count of GOOD or BAD rows.
> Are the bad rows listed in that log which are not replicated?
>
You started VerifyReplication via "hadoop jar", so it's a MapReduce job. Go
to your JobTracker's web UI, you should see your jobs there, then checkout
one of them and click on one of the completed maps then look for the log.
The bad rows are listed in that output.
J-D
Aw: Re: replication verifyrep
Posted by Hansi Klose <ha...@web.de>.
Hi Jean-Daniel,
thank you for your answer and bring some light into the darkness.
> You can see the bad rows listed in the user logs for your MR job.
What log do you mean. The output from the command line?
I only see the count of GOOD or BAD rows.
Are the bad rows listed in that log which are not replicated?
Regards Hansi
> Gesendet: Montag, 14. April 2014 um 19:25 Uhr
> Von: "Jean-Daniel Cryans" <jd...@apache.org>
> An: "user@hbase.apache.org" <us...@hbase.apache.org>
> Betreff: Re: replication verifyrep
>
> Yeah you should use endtime, it was fixed as part of
> https://issues.apache.org/jira/browse/HBASE-10395.
>
> You can see the bad rows listed in the user logs for your MR job.
>
> J-D
>
>
> On Mon, Apr 14, 2014 at 3:06 AM, Hansi Klose <ha...@web.de> wrote:
>
> > Hi,
> >
> > I wrote a little script which should control the running replication.
> >
> > The script is triggered by cron and executes the following command with
> > the actual time stamp in "endtime" and
> > a time stamp => endtime - 10800000 milli seconds. So the time frame is 3
> > hours.
> >
> > hadoop jar /usr/lib/hbase/hbase.jar verifyrep --starttime=1397217601927
> > --endtime=1397228401927 --families=t 1 tablename 2>&1
> >
> > After some running's the script found some BADROWS.
> >
> > 14/04/11 17:04:05 INFO mapred.JobClient: BADROWS=176
> > 14/04/11 17:04:05 INFO mapred.JobClient: GOODROWS=2
> >
> > I executed the same command 20 Minutes later in the shell and got :
> >
> > hadoop jar /usr/lib/hbase/hbase.jar verifyrep --starttime=1397217601927
> > --endtime=1397228401927 --families=t 1 tablename 2>&1
> > 14/04/11 17:21:03 INFO mapred.JobClient: BADROWS=178
> >
> > After that I run the command with the same start time and the actual
> > timestamp an end time, so the time frame is greater
> > but with the same start time. And now I got :
> >
> > hadoop jar /usr/lib/hbase/hbase.jar verifyrep --starttime=1397217601927
> > --endtime=1397230074876 --families=t 1 tablename 2>&1
> > 14/04/11 17:28:28 INFO mapred.JobClient: GOODROWS=184
> >
> > Is there something wrong with the command?
> > In our metrics i could not see that three is an Issue at that time.
> >
> > We are a little bit confused about the endtime. In all documents they talk
> > about stoptime.
> > But we found that in the job configuration there is no parameter called
> > stoptime.
> > We found the "verifyrep.startTime" which hold the value of the starttime
> > in our command and
> > "verifyrep.endTime" which is alway 0 when we use stoptime in the command.
> > So we decided to use endtime
> >
> > Even in the code
> > http://hbase.apache.org/xref/org/apache/hadoop/hbase/mapreduce/replication/VerifyReplication.html
> > they use: "static long endTime = Long.MAX_VALUE;"
> >
> > Which name is the right on? endtime or stoptime?
> >
> > We use cdh 4.2.0.
> >
> > Regards Hansi
> >
>
Re: replication verifyrep
Posted by Jean-Daniel Cryans <jd...@apache.org>.
Yeah you should use endtime, it was fixed as part of
https://issues.apache.org/jira/browse/HBASE-10395.
You can see the bad rows listed in the user logs for your MR job.
J-D
On Mon, Apr 14, 2014 at 3:06 AM, Hansi Klose <ha...@web.de> wrote:
> Hi,
>
> I wrote a little script which should control the running replication.
>
> The script is triggered by cron and executes the following command with
> the actual time stamp in "endtime" and
> a time stamp => endtime - 10800000 milli seconds. So the time frame is 3
> hours.
>
> hadoop jar /usr/lib/hbase/hbase.jar verifyrep --starttime=1397217601927
> --endtime=1397228401927 --families=t 1 tablename 2>&1
>
> After some running's the script found some BADROWS.
>
> 14/04/11 17:04:05 INFO mapred.JobClient: BADROWS=176
> 14/04/11 17:04:05 INFO mapred.JobClient: GOODROWS=2
>
> I executed the same command 20 Minutes later in the shell and got :
>
> hadoop jar /usr/lib/hbase/hbase.jar verifyrep --starttime=1397217601927
> --endtime=1397228401927 --families=t 1 tablename 2>&1
> 14/04/11 17:21:03 INFO mapred.JobClient: BADROWS=178
>
> After that I run the command with the same start time and the actual
> timestamp an end time, so the time frame is greater
> but with the same start time. And now I got :
>
> hadoop jar /usr/lib/hbase/hbase.jar verifyrep --starttime=1397217601927
> --endtime=1397230074876 --families=t 1 tablename 2>&1
> 14/04/11 17:28:28 INFO mapred.JobClient: GOODROWS=184
>
> Is there something wrong with the command?
> In our metrics i could not see that three is an Issue at that time.
>
> We are a little bit confused about the endtime. In all documents they talk
> about stoptime.
> But we found that in the job configuration there is no parameter called
> stoptime.
> We found the "verifyrep.startTime" which hold the value of the starttime
> in our command and
> "verifyrep.endTime" which is alway 0 when we use stoptime in the command.
> So we decided to use endtime
>
> Even in the code
> http://hbase.apache.org/xref/org/apache/hadoop/hbase/mapreduce/replication/VerifyReplication.html
> they use: "static long endTime = Long.MAX_VALUE;"
>
> Which name is the right on? endtime or stoptime?
>
> We use cdh 4.2.0.
>
> Regards Hansi
>