You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Serega Sheypak <se...@gmail.com> on 2014/09/02 20:40:39 UTC
ttl problem, cells are not deleted
Hi, I'm trying to use HBase TTL feature
here is table DDL:
hbase(main):004:0> describe 'item_to_item_similar_rec'
DESCRIPTION
ENABLED
{NAME => 'my_table', FAMILIES => [{NAME => 'd', DATA_BLOCK_ENCODING =>
'NONE', BLOOMFILTER => 'ROWCOL', REPLI true
CATION_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'NONE', MIN_VERSIONS
=> '1', TTL => '172800', KEEP_DELETED_CELLS => 'fal
se', BLOCKSIZE => '65536', IN_MEMORY => 'true', ENCODE_ON_DISK => 'true',
BLOCKCACHE => 'true'}]}
1 row(s) in 0.1730 seconds
It has ttl set to 172800 seconds = 2 days.
I see a cells that are much older than two days:
column=d:099, timestamp=1409049061509, value=\x01\
current ts = 1409679511000
1409679511000 - 1409049061509 = 630449491
Re: ttl problem, cells are not deleted
Posted by Jean-Marc Spaggiari <je...@spaggiari.org>.
No. The oposite. You will need to setup MIN_VERSIONS to 0 if you don't want
to keep any version if older than 2 days.
2014-09-02 15:13 GMT-04:00 Serega Sheypak <se...@gmail.com>:
> Thanks :) So if I set MIN_VERSIONS>1, then TTL feature would completely
> remove experied row?
> My Idea is to automatically remove experied data. It's ok to see sometimes
> expired data but not more than for +-2 days (TTL-1 second + 1 day waiting
> for next major compation).
>
>
> 2014-09-02 23:03 GMT+04:00 lars hofhansl <la...@apache.org>:
>
> > Are these the last versions of a cell? You have MIN_VERSIONS set to 1,
> > meaning HBase will keep at least one version around regardless of whether
> > it is expired by TTL or not.
> >
> > -- Lars
> >
> >
> >
> > ________________________________
> > From: Serega Sheypak <se...@gmail.com>
> > To: user <us...@hbase.apache.org>
> > Sent: Tuesday, September 2, 2014 11:40 AM
> > Subject: ttl problem, cells are not deleted
> >
> >
> > Hi, I'm trying to use HBase TTL feature
> > here is table DDL:
> >
> > hbase(main):004:0> describe 'item_to_item_similar_rec'
> > DESCRIPTION
> > ENABLED
> >
> > {NAME => 'my_table', FAMILIES => [{NAME => 'd', DATA_BLOCK_ENCODING =>
> > 'NONE', BLOOMFILTER => 'ROWCOL', REPLI true
> >
> > CATION_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'NONE', MIN_VERSIONS
> > => '1', TTL => '172800', KEEP_DELETED_CELLS => 'fal
> >
> > se', BLOCKSIZE => '65536', IN_MEMORY => 'true', ENCODE_ON_DISK => 'true',
> > BLOCKCACHE => 'true'}]}
> >
> > 1 row(s) in 0.1730 seconds
> >
> > It has ttl set to 172800 seconds = 2 days.
> > I see a cells that are much older than two days:
> >
> > column=d:099, timestamp=1409049061509, value=\x01\
> > current ts = 1409679511000
> >
> > 1409679511000 - 1409049061509 = 630449491
> >
>
Re: ttl problem, cells are not deleted
Posted by Serega Sheypak <se...@gmail.com>.
Thanks :) So if I set MIN_VERSIONS>1, then TTL feature would completely
remove experied row?
My Idea is to automatically remove experied data. It's ok to see sometimes
expired data but not more than for +-2 days (TTL-1 second + 1 day waiting
for next major compation).
2014-09-02 23:03 GMT+04:00 lars hofhansl <la...@apache.org>:
> Are these the last versions of a cell? You have MIN_VERSIONS set to 1,
> meaning HBase will keep at least one version around regardless of whether
> it is expired by TTL or not.
>
> -- Lars
>
>
>
> ________________________________
> From: Serega Sheypak <se...@gmail.com>
> To: user <us...@hbase.apache.org>
> Sent: Tuesday, September 2, 2014 11:40 AM
> Subject: ttl problem, cells are not deleted
>
>
> Hi, I'm trying to use HBase TTL feature
> here is table DDL:
>
> hbase(main):004:0> describe 'item_to_item_similar_rec'
> DESCRIPTION
> ENABLED
>
> {NAME => 'my_table', FAMILIES => [{NAME => 'd', DATA_BLOCK_ENCODING =>
> 'NONE', BLOOMFILTER => 'ROWCOL', REPLI true
>
> CATION_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'NONE', MIN_VERSIONS
> => '1', TTL => '172800', KEEP_DELETED_CELLS => 'fal
>
> se', BLOCKSIZE => '65536', IN_MEMORY => 'true', ENCODE_ON_DISK => 'true',
> BLOCKCACHE => 'true'}]}
>
> 1 row(s) in 0.1730 seconds
>
> It has ttl set to 172800 seconds = 2 days.
> I see a cells that are much older than two days:
>
> column=d:099, timestamp=1409049061509, value=\x01\
> current ts = 1409679511000
>
> 1409679511000 - 1409049061509 = 630449491
>
Re: ttl problem, cells are not deleted
Posted by lars hofhansl <la...@apache.org>.
Are these the last versions of a cell? You have MIN_VERSIONS set to 1, meaning HBase will keep at least one version around regardless of whether it is expired by TTL or not.
-- Lars
________________________________
From: Serega Sheypak <se...@gmail.com>
To: user <us...@hbase.apache.org>
Sent: Tuesday, September 2, 2014 11:40 AM
Subject: ttl problem, cells are not deleted
Hi, I'm trying to use HBase TTL feature
here is table DDL:
hbase(main):004:0> describe 'item_to_item_similar_rec'
DESCRIPTION
ENABLED
{NAME => 'my_table', FAMILIES => [{NAME => 'd', DATA_BLOCK_ENCODING =>
'NONE', BLOOMFILTER => 'ROWCOL', REPLI true
CATION_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'NONE', MIN_VERSIONS
=> '1', TTL => '172800', KEEP_DELETED_CELLS => 'fal
se', BLOCKSIZE => '65536', IN_MEMORY => 'true', ENCODE_ON_DISK => 'true',
BLOCKCACHE => 'true'}]}
1 row(s) in 0.1730 seconds
It has ttl set to 172800 seconds = 2 days.
I see a cells that are much older than two days:
column=d:099, timestamp=1409049061509, value=\x01\
current ts = 1409679511000
1409679511000 - 1409049061509 = 630449491
Re: ttl problem, cells are not deleted
Posted by Serega Sheypak <se...@gmail.com>.
Yes, It was my misunderstanding. I supposed that HBase would keep at least
1 version. It forbids to add more versions since I don't need them.
TTL should remove all expired versions during major compation and table row
count would be decreased.
So I got a little more knowledge. Thanks!
2014-09-02 23:18 GMT+04:00 Jean-Marc Spaggiari <je...@spaggiari.org>:
> Correct.
>
> Default value is 0. So you have added this parameter in your table creation
> script?
>
>
> 2014-09-02 15:15 GMT-04:00 Serega Sheypak <se...@gmail.com>:
>
> > Fix, sorry, should set to MIN_VERSIONS=0, correct?
> >
> >
> > 2014-09-02 23:01 GMT+04:00 Jean-Marc Spaggiari <jean-marc@spaggiari.org
> >:
> >
> > > Oh! I know why!
> > >
> > > MIN_VERSIONS is setup to 1! Soyou will always have at least one
> version,
> > > even if TTL is passed.
> > >
> > > If you really want all to be removed, change this to 0.
> > >
> > > JM
> > >
> > >
> > > 2014-09-02 14:58 GMT-04:00 Ted Yu <yu...@gmail.com>:
> > >
> > > > What release of hbase are you using ?
> > > >
> > > > Have you checked region server log for compaction related logs ?
> > > >
> > > > Cheers
> > > >
> > > > On Sep 2, 2014, at 11:40 AM, Serega Sheypak <
> serega.sheypak@gmail.com>
> > > > wrote:
> > > >
> > > > > Hi, I'm trying to use HBase TTL feature
> > > > > here is table DDL:
> > > > >
> > > > > hbase(main):004:0> describe 'item_to_item_similar_rec'
> > > > > DESCRIPTION
> > > > > ENABLED
> > > > >
> > > > > {NAME => 'my_table', FAMILIES => [{NAME => 'd', DATA_BLOCK_ENCODING
> > =>
> > > > > 'NONE', BLOOMFILTER => 'ROWCOL', REPLI true
> > > > >
> > > > > CATION_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'NONE',
> > > MIN_VERSIONS
> > > > > => '1', TTL => '172800', KEEP_DELETED_CELLS => 'fal
> > > > >
> > > > > se', BLOCKSIZE => '65536', IN_MEMORY => 'true', ENCODE_ON_DISK =>
> > > 'true',
> > > > > BLOCKCACHE => 'true'}]}
> > > > >
> > > > > 1 row(s) in 0.1730 seconds
> > > > >
> > > > > It has ttl set to 172800 seconds = 2 days.
> > > > > I see a cells that are much older than two days:
> > > > >
> > > > > column=d:099, timestamp=1409049061509, value=\x01\
> > > > > current ts = 1409679511000
> > > > >
> > > > > 1409679511000 - 1409049061509 = 630449491
> > > >
> > >
> >
>
Re: ttl problem, cells are not deleted
Posted by Jean-Marc Spaggiari <je...@spaggiari.org>.
Correct.
Default value is 0. So you have added this parameter in your table creation
script?
2014-09-02 15:15 GMT-04:00 Serega Sheypak <se...@gmail.com>:
> Fix, sorry, should set to MIN_VERSIONS=0, correct?
>
>
> 2014-09-02 23:01 GMT+04:00 Jean-Marc Spaggiari <je...@spaggiari.org>:
>
> > Oh! I know why!
> >
> > MIN_VERSIONS is setup to 1! Soyou will always have at least one version,
> > even if TTL is passed.
> >
> > If you really want all to be removed, change this to 0.
> >
> > JM
> >
> >
> > 2014-09-02 14:58 GMT-04:00 Ted Yu <yu...@gmail.com>:
> >
> > > What release of hbase are you using ?
> > >
> > > Have you checked region server log for compaction related logs ?
> > >
> > > Cheers
> > >
> > > On Sep 2, 2014, at 11:40 AM, Serega Sheypak <se...@gmail.com>
> > > wrote:
> > >
> > > > Hi, I'm trying to use HBase TTL feature
> > > > here is table DDL:
> > > >
> > > > hbase(main):004:0> describe 'item_to_item_similar_rec'
> > > > DESCRIPTION
> > > > ENABLED
> > > >
> > > > {NAME => 'my_table', FAMILIES => [{NAME => 'd', DATA_BLOCK_ENCODING
> =>
> > > > 'NONE', BLOOMFILTER => 'ROWCOL', REPLI true
> > > >
> > > > CATION_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'NONE',
> > MIN_VERSIONS
> > > > => '1', TTL => '172800', KEEP_DELETED_CELLS => 'fal
> > > >
> > > > se', BLOCKSIZE => '65536', IN_MEMORY => 'true', ENCODE_ON_DISK =>
> > 'true',
> > > > BLOCKCACHE => 'true'}]}
> > > >
> > > > 1 row(s) in 0.1730 seconds
> > > >
> > > > It has ttl set to 172800 seconds = 2 days.
> > > > I see a cells that are much older than two days:
> > > >
> > > > column=d:099, timestamp=1409049061509, value=\x01\
> > > > current ts = 1409679511000
> > > >
> > > > 1409679511000 - 1409049061509 = 630449491
> > >
> >
>
Re: ttl problem, cells are not deleted
Posted by Serega Sheypak <se...@gmail.com>.
Fix, sorry, should set to MIN_VERSIONS=0, correct?
2014-09-02 23:01 GMT+04:00 Jean-Marc Spaggiari <je...@spaggiari.org>:
> Oh! I know why!
>
> MIN_VERSIONS is setup to 1! Soyou will always have at least one version,
> even if TTL is passed.
>
> If you really want all to be removed, change this to 0.
>
> JM
>
>
> 2014-09-02 14:58 GMT-04:00 Ted Yu <yu...@gmail.com>:
>
> > What release of hbase are you using ?
> >
> > Have you checked region server log for compaction related logs ?
> >
> > Cheers
> >
> > On Sep 2, 2014, at 11:40 AM, Serega Sheypak <se...@gmail.com>
> > wrote:
> >
> > > Hi, I'm trying to use HBase TTL feature
> > > here is table DDL:
> > >
> > > hbase(main):004:0> describe 'item_to_item_similar_rec'
> > > DESCRIPTION
> > > ENABLED
> > >
> > > {NAME => 'my_table', FAMILIES => [{NAME => 'd', DATA_BLOCK_ENCODING =>
> > > 'NONE', BLOOMFILTER => 'ROWCOL', REPLI true
> > >
> > > CATION_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'NONE',
> MIN_VERSIONS
> > > => '1', TTL => '172800', KEEP_DELETED_CELLS => 'fal
> > >
> > > se', BLOCKSIZE => '65536', IN_MEMORY => 'true', ENCODE_ON_DISK =>
> 'true',
> > > BLOCKCACHE => 'true'}]}
> > >
> > > 1 row(s) in 0.1730 seconds
> > >
> > > It has ttl set to 172800 seconds = 2 days.
> > > I see a cells that are much older than two days:
> > >
> > > column=d:099, timestamp=1409049061509, value=\x01\
> > > current ts = 1409679511000
> > >
> > > 1409679511000 - 1409049061509 = 630449491
> >
>
Re: ttl problem, cells are not deleted
Posted by Jean-Marc Spaggiari <je...@spaggiari.org>.
Oh! I know why!
MIN_VERSIONS is setup to 1! Soyou will always have at least one version,
even if TTL is passed.
If you really want all to be removed, change this to 0.
JM
2014-09-02 14:58 GMT-04:00 Ted Yu <yu...@gmail.com>:
> What release of hbase are you using ?
>
> Have you checked region server log for compaction related logs ?
>
> Cheers
>
> On Sep 2, 2014, at 11:40 AM, Serega Sheypak <se...@gmail.com>
> wrote:
>
> > Hi, I'm trying to use HBase TTL feature
> > here is table DDL:
> >
> > hbase(main):004:0> describe 'item_to_item_similar_rec'
> > DESCRIPTION
> > ENABLED
> >
> > {NAME => 'my_table', FAMILIES => [{NAME => 'd', DATA_BLOCK_ENCODING =>
> > 'NONE', BLOOMFILTER => 'ROWCOL', REPLI true
> >
> > CATION_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'NONE', MIN_VERSIONS
> > => '1', TTL => '172800', KEEP_DELETED_CELLS => 'fal
> >
> > se', BLOCKSIZE => '65536', IN_MEMORY => 'true', ENCODE_ON_DISK => 'true',
> > BLOCKCACHE => 'true'}]}
> >
> > 1 row(s) in 0.1730 seconds
> >
> > It has ttl set to 172800 seconds = 2 days.
> > I see a cells that are much older than two days:
> >
> > column=d:099, timestamp=1409049061509, value=\x01\
> > current ts = 1409679511000
> >
> > 1409679511000 - 1409049061509 = 630449491
>
Re: ttl problem, cells are not deleted
Posted by Ted Yu <yu...@gmail.com>.
What release of hbase are you using ?
Have you checked region server log for compaction related logs ?
Cheers
On Sep 2, 2014, at 11:40 AM, Serega Sheypak <se...@gmail.com> wrote:
> Hi, I'm trying to use HBase TTL feature
> here is table DDL:
>
> hbase(main):004:0> describe 'item_to_item_similar_rec'
> DESCRIPTION
> ENABLED
>
> {NAME => 'my_table', FAMILIES => [{NAME => 'd', DATA_BLOCK_ENCODING =>
> 'NONE', BLOOMFILTER => 'ROWCOL', REPLI true
>
> CATION_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'NONE', MIN_VERSIONS
> => '1', TTL => '172800', KEEP_DELETED_CELLS => 'fal
>
> se', BLOCKSIZE => '65536', IN_MEMORY => 'true', ENCODE_ON_DISK => 'true',
> BLOCKCACHE => 'true'}]}
>
> 1 row(s) in 0.1730 seconds
>
> It has ttl set to 172800 seconds = 2 days.
> I see a cells that are much older than two days:
>
> column=d:099, timestamp=1409049061509, value=\x01\
> current ts = 1409679511000
>
> 1409679511000 - 1409049061509 = 630449491
Re: ttl problem, cells are not deleted
Posted by Esteban Gutierrez <es...@cloudera.com>.
Hi Serega,
Can you try to find the HFile with that has this row? do you see any issue
in the region server if this is trying to compact this HFile?
cheers,
esteban.
--
Cloudera, Inc.
On Tue, Sep 2, 2014 at 11:46 AM, Serega Sheypak <se...@gmail.com>
wrote:
> major comactions are not disabled.
> hbase.hregion.majorcompaction=1day... a lot of days passed, what can I also
> check?
> table has ~30*10^6 records, it's a tiny table.
>
>
> 2014-09-02 22:43 GMT+04:00 Jean-Marc Spaggiari <je...@spaggiari.org>:
>
> > Is it a big table? A test cluster? Can you try to compact the table?
> >
> >
> > 2014-09-02 14:40 GMT-04:00 Serega Sheypak <se...@gmail.com>:
> >
> > > Hi, I'm trying to use HBase TTL feature
> > > here is table DDL:
> > >
> > > hbase(main):004:0> describe 'item_to_item_similar_rec'
> > > DESCRIPTION
> > > ENABLED
> > >
> > > {NAME => 'my_table', FAMILIES => [{NAME => 'd', DATA_BLOCK_ENCODING =>
> > > 'NONE', BLOOMFILTER => 'ROWCOL', REPLI true
> > >
> > > CATION_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'NONE',
> > MIN_VERSIONS
> > > => '1', TTL => '172800', KEEP_DELETED_CELLS => 'fal
> > >
> > > se', BLOCKSIZE => '65536', IN_MEMORY => 'true', ENCODE_ON_DISK =>
> > 'true',
> > > BLOCKCACHE => 'true'}]}
> > >
> > > 1 row(s) in 0.1730 seconds
> > >
> > > It has ttl set to 172800 seconds = 2 days.
> > > I see a cells that are much older than two days:
> > >
> > > column=d:099, timestamp=1409049061509, value=\x01\
> > > current ts = 1409679511000
> > >
> > > 1409679511000 - 1409049061509 = 630449491
> > >
> >
>
Re: ttl problem, cells are not deleted
Posted by Serega Sheypak <se...@gmail.com>.
major comactions are not disabled.
hbase.hregion.majorcompaction=1day... a lot of days passed, what can I also
check?
table has ~30*10^6 records, it's a tiny table.
2014-09-02 22:43 GMT+04:00 Jean-Marc Spaggiari <je...@spaggiari.org>:
> Is it a big table? A test cluster? Can you try to compact the table?
>
>
> 2014-09-02 14:40 GMT-04:00 Serega Sheypak <se...@gmail.com>:
>
> > Hi, I'm trying to use HBase TTL feature
> > here is table DDL:
> >
> > hbase(main):004:0> describe 'item_to_item_similar_rec'
> > DESCRIPTION
> > ENABLED
> >
> > {NAME => 'my_table', FAMILIES => [{NAME => 'd', DATA_BLOCK_ENCODING =>
> > 'NONE', BLOOMFILTER => 'ROWCOL', REPLI true
> >
> > CATION_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'NONE',
> MIN_VERSIONS
> > => '1', TTL => '172800', KEEP_DELETED_CELLS => 'fal
> >
> > se', BLOCKSIZE => '65536', IN_MEMORY => 'true', ENCODE_ON_DISK =>
> 'true',
> > BLOCKCACHE => 'true'}]}
> >
> > 1 row(s) in 0.1730 seconds
> >
> > It has ttl set to 172800 seconds = 2 days.
> > I see a cells that are much older than two days:
> >
> > column=d:099, timestamp=1409049061509, value=\x01\
> > current ts = 1409679511000
> >
> > 1409679511000 - 1409049061509 = 630449491
> >
>
Re: ttl problem, cells are not deleted
Posted by Jean-Marc Spaggiari <je...@spaggiari.org>.
Is it a big table? A test cluster? Can you try to compact the table?
2014-09-02 14:40 GMT-04:00 Serega Sheypak <se...@gmail.com>:
> Hi, I'm trying to use HBase TTL feature
> here is table DDL:
>
> hbase(main):004:0> describe 'item_to_item_similar_rec'
> DESCRIPTION
> ENABLED
>
> {NAME => 'my_table', FAMILIES => [{NAME => 'd', DATA_BLOCK_ENCODING =>
> 'NONE', BLOOMFILTER => 'ROWCOL', REPLI true
>
> CATION_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'NONE', MIN_VERSIONS
> => '1', TTL => '172800', KEEP_DELETED_CELLS => 'fal
>
> se', BLOCKSIZE => '65536', IN_MEMORY => 'true', ENCODE_ON_DISK => 'true',
> BLOCKCACHE => 'true'}]}
>
> 1 row(s) in 0.1730 seconds
>
> It has ttl set to 172800 seconds = 2 days.
> I see a cells that are much older than two days:
>
> column=d:099, timestamp=1409049061509, value=\x01\
> current ts = 1409679511000
>
> 1409679511000 - 1409049061509 = 630449491
>