You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Antonio Si <an...@gmail.com> on 2018/08/26 21:34:05 UTC
question on reducing number of versions
Hello,
I have a hbase table whose definition has a max number of versions set to
36000.
I have verified that there are rows which have more than 20000 versions
saved.
Now, I change the definition of the table and reduce the max number of
versions to 18000. Will I see the size of the table being reduced as I am
not seeing that?
Also, after I reduce the max number of versions, I try to create a
snapshot, but I am getting a
com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.services.s3.mo
del.AmazonS3Exception: Not Found (Service: Amazon S3; Status Code: 404;
Error Code: 404 Not Found;
What may be the cause of that?
I am using s3 as my storage.
Thanks in advance for your suggestions.
Antonio.
Re: question on reducing number of versions
Posted by Anil Gupta <an...@gmail.com>.
You should see a smaller t2 after major compaction if your table actually had versions over 18k.(as Ted mentioned)
Sent from my iPhone
> On Aug 26, 2018, at 5:20 PM, Ted Yu <yu...@gmail.com> wrote:
>
> This depends on how far down you revise the max versions for table t2.
> If your data normally only reaches 15000 versions and you lower max
> versions to ~15000, there wouldn't be much saving.
>
> FYI
>
>> On Sun, Aug 26, 2018 at 3:52 PM Antonio Si <an...@gmail.com> wrote:
>>
>> Thanks Anil.
>>
>> We are using hbase on s3. Yes, I understand 18000 is very high. We are in
>> the process of reducing it.
>>
>> If I have a snapshot and I restore the table from this snapshot. Let's call
>> this table t1.
>> I then clone another table from the same snapshot, call it t2.
>>
>> If I reduce the max versions of t2 and run a major compaction on t2, will I
>> see the decrease in table size for t2? If I compare the size of t2 and t1,
>> I should see a smaller size for t2?
>>
>> Thanks.
>>
>> Antonio.
>>
>>> On Sun, Aug 26, 2018 at 3:33 PM Anil Gupta <an...@gmail.com> wrote:
>>>
>>> You will need to do major compaction on table for the table to
>>> clean/delete up extra version.
>>> Btw, 18000 max version is a unusually high value.
>>>
>>> Are you using hbase on s3 or hbase on hdfs?
>>>
>>> Sent from my iPhone
>>>
>>>> On Aug 26, 2018, at 2:34 PM, Antonio Si <an...@gmail.com> wrote:
>>>>
>>>> Hello,
>>>>
>>>> I have a hbase table whose definition has a max number of versions set
>> to
>>>> 36000.
>>>> I have verified that there are rows which have more than 20000 versions
>>>> saved.
>>>>
>>>> Now, I change the definition of the table and reduce the max number of
>>>> versions to 18000. Will I see the size of the table being reduced as I
>> am
>>>> not seeing that?
>>>>
>>>> Also, after I reduce the max number of versions, I try to create a
>>>> snapshot, but I am getting a
>>>> com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.services.s3.mo
>>>>
>>>> del.AmazonS3Exception: Not Found (Service: Amazon S3; Status Code: 404;
>>>> Error Code: 404 Not Found;
>>>>
>>>>
>>>> What may be the cause of that?
>>>>
>>>> I am using s3 as my storage.
>>>>
>>>>
>>>> Thanks in advance for your suggestions.
>>>>
>>>>
>>>> Antonio.
>>>
>>
Re: question on reducing number of versions
Posted by Ted Yu <yu...@gmail.com>.
This depends on how far down you revise the max versions for table t2.
If your data normally only reaches 15000 versions and you lower max
versions to ~15000, there wouldn't be much saving.
FYI
On Sun, Aug 26, 2018 at 3:52 PM Antonio Si <an...@gmail.com> wrote:
> Thanks Anil.
>
> We are using hbase on s3. Yes, I understand 18000 is very high. We are in
> the process of reducing it.
>
> If I have a snapshot and I restore the table from this snapshot. Let's call
> this table t1.
> I then clone another table from the same snapshot, call it t2.
>
> If I reduce the max versions of t2 and run a major compaction on t2, will I
> see the decrease in table size for t2? If I compare the size of t2 and t1,
> I should see a smaller size for t2?
>
> Thanks.
>
> Antonio.
>
> On Sun, Aug 26, 2018 at 3:33 PM Anil Gupta <an...@gmail.com> wrote:
>
> > You will need to do major compaction on table for the table to
> > clean/delete up extra version.
> > Btw, 18000 max version is a unusually high value.
> >
> > Are you using hbase on s3 or hbase on hdfs?
> >
> > Sent from my iPhone
> >
> > > On Aug 26, 2018, at 2:34 PM, Antonio Si <an...@gmail.com> wrote:
> > >
> > > Hello,
> > >
> > > I have a hbase table whose definition has a max number of versions set
> to
> > > 36000.
> > > I have verified that there are rows which have more than 20000 versions
> > > saved.
> > >
> > > Now, I change the definition of the table and reduce the max number of
> > > versions to 18000. Will I see the size of the table being reduced as I
> am
> > > not seeing that?
> > >
> > > Also, after I reduce the max number of versions, I try to create a
> > > snapshot, but I am getting a
> > > com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.services.s3.mo
> > >
> > > del.AmazonS3Exception: Not Found (Service: Amazon S3; Status Code: 404;
> > > Error Code: 404 Not Found;
> > >
> > >
> > > What may be the cause of that?
> > >
> > > I am using s3 as my storage.
> > >
> > >
> > > Thanks in advance for your suggestions.
> > >
> > >
> > > Antonio.
> >
>
Re: question on reducing number of versions
Posted by Antonio Si <an...@gmail.com>.
Thanks Anil.
We are using hbase on s3. Yes, I understand 18000 is very high. We are in
the process of reducing it.
If I have a snapshot and I restore the table from this snapshot. Let's call
this table t1.
I then clone another table from the same snapshot, call it t2.
If I reduce the max versions of t2 and run a major compaction on t2, will I
see the decrease in table size for t2? If I compare the size of t2 and t1,
I should see a smaller size for t2?
Thanks.
Antonio.
On Sun, Aug 26, 2018 at 3:33 PM Anil Gupta <an...@gmail.com> wrote:
> You will need to do major compaction on table for the table to
> clean/delete up extra version.
> Btw, 18000 max version is a unusually high value.
>
> Are you using hbase on s3 or hbase on hdfs?
>
> Sent from my iPhone
>
> > On Aug 26, 2018, at 2:34 PM, Antonio Si <an...@gmail.com> wrote:
> >
> > Hello,
> >
> > I have a hbase table whose definition has a max number of versions set to
> > 36000.
> > I have verified that there are rows which have more than 20000 versions
> > saved.
> >
> > Now, I change the definition of the table and reduce the max number of
> > versions to 18000. Will I see the size of the table being reduced as I am
> > not seeing that?
> >
> > Also, after I reduce the max number of versions, I try to create a
> > snapshot, but I am getting a
> > com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.services.s3.mo
> >
> > del.AmazonS3Exception: Not Found (Service: Amazon S3; Status Code: 404;
> > Error Code: 404 Not Found;
> >
> >
> > What may be the cause of that?
> >
> > I am using s3 as my storage.
> >
> >
> > Thanks in advance for your suggestions.
> >
> >
> > Antonio.
>
Re: question on reducing number of versions
Posted by Anil Gupta <an...@gmail.com>.
You will need to do major compaction on table for the table to clean/delete up extra version.
Btw, 18000 max version is a unusually high value.
Are you using hbase on s3 or hbase on hdfs?
Sent from my iPhone
> On Aug 26, 2018, at 2:34 PM, Antonio Si <an...@gmail.com> wrote:
>
> Hello,
>
> I have a hbase table whose definition has a max number of versions set to
> 36000.
> I have verified that there are rows which have more than 20000 versions
> saved.
>
> Now, I change the definition of the table and reduce the max number of
> versions to 18000. Will I see the size of the table being reduced as I am
> not seeing that?
>
> Also, after I reduce the max number of versions, I try to create a
> snapshot, but I am getting a
> com.amazon.ws.emr.hadoop.fs.shaded.com.amazonaws.services.s3.mo
>
> del.AmazonS3Exception: Not Found (Service: Amazon S3; Status Code: 404;
> Error Code: 404 Not Found;
>
>
> What may be the cause of that?
>
> I am using s3 as my storage.
>
>
> Thanks in advance for your suggestions.
>
>
> Antonio.