You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Jean-Marc Spaggiari <je...@spaggiari.org> on 2015/08/05 19:15:18 UTC
Flushs and compactions
Quick question here.
Compactions seems to be triggered when we flush automatically, but not when
we flush manually.
for 1 to 10
put into a table
flush the table
end of for
Will create 10 files, and will not trigger any compaction.
Should we also compact after manual flushs?
JM
Re: Flushs and compactions
Posted by Jean-Marc Spaggiari <je...@spaggiari.org>.
With DEBUG level: http://pastebin.com/arvD95bA
2015-08-06 12:14 GMT-04:00 Ted Yu <yu...@gmail.com>:
> JMS:
> Having log level at DEBUG would give us more clue.
>
> FYI
>
> On Thu, Aug 6, 2015 at 9:12 AM, Esteban Gutierrez <es...@cloudera.com>
> wrote:
>
> > From the logs it looks like the flushes requested the compactions as
> > expected once the min number of files for a minor compaction was reached,
> > but I don't see any compaction running afterwards:
> >
> > 2015-08-06 09:21:57,465 INFO
> > [StoreOpener-3d02eb59261771da84824c44e6d2ca40-1]
> > compactions.CompactionConfiguration: size [134217728,
> 9223372036854775807);
> > files [3, 10); ratio 1,200000; off-peak ratio 5,000000; throttle point
> > 2684354560; major period 604800000, major jitter 0,500000, min locality
> to
> > compact 0,000000.
> > ...
> > 2015-08-06 09:21:57,854 INFO
> > [rs(localhost,33882,1438867297154)-flush-proc-pool5-thread-1]
> > regionserver.HRegion: Finished memstore flush of ~160 B/160,
> currentsize=0
> > B/0 for region testflush,,1438867317389.3d02eb59261771da84824c44e6d2ca40.
> > in 77ms, sequenceid=6, compaction requested=false
> > ...
> > 2015-08-06 09:21:58,097 INFO
> > [rs(localhost,33882,1438867297154)-flush-proc-pool6-thread-1]
> > regionserver.HRegion: Finished memstore flush of ~160 B/160,
> currentsize=0
> > B/0 for region testflush,,1438867317389.3d02eb59261771da84824c44e6d2ca40.
> > in 12ms, sequenceid=11, compaction requested=false
> > ...
> > 2015-08-06 09:21:58,318 INFO
> > [rs(localhost,33882,1438867297154)-flush-proc-pool7-thread-1]
> > regionserver.HRegion: Finished memstore flush of ~160 B/160,
> currentsize=0
> > B/0 for region testflush,,1438867317389.3d02eb59261771da84824c44e6d2ca40.
> > in 9ms, sequenceid=16, compaction requested=true
> > ... no compactions here ...
> > 2015-08-06 10:31:37,420 INFO [LruBlockCacheStatsExecutor]
> > hfile.LruBlockCache: totalSize=1.61 MB, freeSize=1.53 GB, max=1.53 GB,
> > blockCount=1, accesses=1, hits=0, hitRatio=0, cachingAccesses=1,
> > cachingHits=0, cachingHitsRatio=0,evictions=419, evicted=0,
> > evictedPerRun=0.0
> >
> > Are you sure you have the correct log4j config? I will try to reproduce
> > this and I let you know JMS.
> >
> > esteban.
> >
> >
> >
> > --
> > Cloudera, Inc.
> >
> >
> > On Thu, Aug 6, 2015 at 7:35 AM, Jean-Marc Spaggiari <
> > jean-marc@spaggiari.org
> > > wrote:
> >
> > > Nothing on the logs...
> > >
> > > http://pastebin.com/4U6Fmxt9
> > >
> > > Running standalone. Kept the server running for 1h after I tried, still
> > no
> > > compactions.
> > >
> > > 2015-08-06 0:52 GMT-04:00 ramkrishna vasudevan <
> > > ramkrishna.s.vasudevan@gmail.com>:
> > >
> > > > Any log saying skipping the compaction? I need to check the code
> here.
> > > > Is the compaction thread picking up at a later point of time?
> > > >
> > > > Regards
> > > > Ram
> > > >
> > > > On Wed, Aug 5, 2015 at 10:55 PM, Jean-Marc Spaggiari <
> > > > jean-marc@spaggiari.org> wrote:
> > > >
> > > > > I tried that with 1.0.1 but I think it's the same with 1.1.0.
> > > > >
> > > > > How to reproduce:
> > > > >
> > > > > create 'testflush', 'f1'
> > > > > put 'testflush', 'row1', 'f1:c1', 'val1'
> > > > > flush 'testflush'
> > > > > put 'testflush', 'row1', 'f1:c1', 'val1'
> > > > > flush 'testflush'
> > > > > put 'testflush', 'row1', 'f1:c1', 'val1'
> > > > > flush 'testflush'
> > > > > put 'testflush', 'row1', 'f1:c1', 'val1'
> > > > > flush 'testflush'
> > > > > put 'testflush', 'row1', 'f1:c1', 'val1'
> > > > > flush 'testflush'
> > > > > put 'testflush', 'row1', 'f1:c1', 'val1'
> > > > > flush 'testflush'
> > > > > put 'testflush', 'row1', 'f1:c1', 'val1'
> > > > > flush 'testflush'
> > > > > put 'testflush', 'row1', 'f1:c1', 'val1'
> > > > > flush 'testflush'
> > > > > put 'testflush', 'row1', 'f1:c1', 'val1'
> > > > > flush 'testflush'
> > > > > put 'testflush', 'row1', 'f1:c1', 'val1'
> > > > > flush 'testflush'
> > > > > put 'testflush', 'row1', 'f1:c1', 'val1'
> > > > > flush 'testflush'
> > > > >
> > > > > Result:
> > > > > jmspaggiari@t430s:~/hbase-1.0.0$ tree
> > > > > /tmp/hbase-jmspaggiari/hbase/data/default/
> > > > > /tmp/hbase-jmspaggiari/hbase/data/default/
> > > > > └── testflush
> > > > > └── 6546495c8df1e1b0a5a5a11d5df23e44
> > > > > ├── f1
> > > > > │ ├── 195e2b5af8e842f89440927a12c38c25
> > > > > │ ├── 266ef04cefe944249d72791a5741a64e
> > > > > │ ├── 33b01c2bf3c34d98a01bba95ffa58d0c
> > > > > │ ├── 365dbdc33f324f67bf466965cc6ad9e8
> > > > > │ ├── 47a47d405bce4bf0a261a16a1ba5694f
> > > > > │ ├── 54b7a073289b459292d0eff447d413a5
> > > > > │ ├── 55fafb3c30fa4067a3bab54823478cdd
> > > > > │ ├── 923ddff172a542b6a5ba6f3467983efa
> > > > > │ ├── c69127daa4d34fe8b832e1fafea42713
> > > > > │ ├── da4fe34334824f6ea9f5a12dfb93cab9
> > > > > │ └── f3a15c326535420cb68d505ce9798d40
> > > > > └── recovered.edits
> > > > > └── 2.seqid
> > > > >
> > > > > 4 directories, 12 files
> > > > >
> > > > > I can test the same on 1.1.0 if you want, just let me know.
> > > > >
> > > > > 2015-08-05 13:18 GMT-04:00 Ted Yu <yu...@gmail.com>:
> > > > >
> > > > > > Can you provide some more detail please ?
> > > > > >
> > > > > > Release of hbase
> > > > > > region server log (snippet) showing that compaction is skipped
> > > > > >
> > > > > > Thanks
> > > > > >
> > > > > > On Wed, Aug 5, 2015 at 10:15 AM, Jean-Marc Spaggiari <
> > > > > > jean-marc@spaggiari.org> wrote:
> > > > > >
> > > > > > > Quick question here.
> > > > > > >
> > > > > > > Compactions seems to be triggered when we flush automatically,
> > but
> > > > not
> > > > > > when
> > > > > > > we flush manually.
> > > > > > >
> > > > > > > for 1 to 10
> > > > > > > put into a table
> > > > > > > flush the table
> > > > > > > end of for
> > > > > > >
> > > > > > > Will create 10 files, and will not trigger any compaction.
> > > > > > >
> > > > > > > Should we also compact after manual flushs?
> > > > > > >
> > > > > > > JM
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Flushs and compactions
Posted by Ted Yu <yu...@gmail.com>.
JMS:
Having log level at DEBUG would give us more clue.
FYI
On Thu, Aug 6, 2015 at 9:12 AM, Esteban Gutierrez <es...@cloudera.com>
wrote:
> From the logs it looks like the flushes requested the compactions as
> expected once the min number of files for a minor compaction was reached,
> but I don't see any compaction running afterwards:
>
> 2015-08-06 09:21:57,465 INFO
> [StoreOpener-3d02eb59261771da84824c44e6d2ca40-1]
> compactions.CompactionConfiguration: size [134217728, 9223372036854775807);
> files [3, 10); ratio 1,200000; off-peak ratio 5,000000; throttle point
> 2684354560; major period 604800000, major jitter 0,500000, min locality to
> compact 0,000000.
> ...
> 2015-08-06 09:21:57,854 INFO
> [rs(localhost,33882,1438867297154)-flush-proc-pool5-thread-1]
> regionserver.HRegion: Finished memstore flush of ~160 B/160, currentsize=0
> B/0 for region testflush,,1438867317389.3d02eb59261771da84824c44e6d2ca40.
> in 77ms, sequenceid=6, compaction requested=false
> ...
> 2015-08-06 09:21:58,097 INFO
> [rs(localhost,33882,1438867297154)-flush-proc-pool6-thread-1]
> regionserver.HRegion: Finished memstore flush of ~160 B/160, currentsize=0
> B/0 for region testflush,,1438867317389.3d02eb59261771da84824c44e6d2ca40.
> in 12ms, sequenceid=11, compaction requested=false
> ...
> 2015-08-06 09:21:58,318 INFO
> [rs(localhost,33882,1438867297154)-flush-proc-pool7-thread-1]
> regionserver.HRegion: Finished memstore flush of ~160 B/160, currentsize=0
> B/0 for region testflush,,1438867317389.3d02eb59261771da84824c44e6d2ca40.
> in 9ms, sequenceid=16, compaction requested=true
> ... no compactions here ...
> 2015-08-06 10:31:37,420 INFO [LruBlockCacheStatsExecutor]
> hfile.LruBlockCache: totalSize=1.61 MB, freeSize=1.53 GB, max=1.53 GB,
> blockCount=1, accesses=1, hits=0, hitRatio=0, cachingAccesses=1,
> cachingHits=0, cachingHitsRatio=0,evictions=419, evicted=0,
> evictedPerRun=0.0
>
> Are you sure you have the correct log4j config? I will try to reproduce
> this and I let you know JMS.
>
> esteban.
>
>
>
> --
> Cloudera, Inc.
>
>
> On Thu, Aug 6, 2015 at 7:35 AM, Jean-Marc Spaggiari <
> jean-marc@spaggiari.org
> > wrote:
>
> > Nothing on the logs...
> >
> > http://pastebin.com/4U6Fmxt9
> >
> > Running standalone. Kept the server running for 1h after I tried, still
> no
> > compactions.
> >
> > 2015-08-06 0:52 GMT-04:00 ramkrishna vasudevan <
> > ramkrishna.s.vasudevan@gmail.com>:
> >
> > > Any log saying skipping the compaction? I need to check the code here.
> > > Is the compaction thread picking up at a later point of time?
> > >
> > > Regards
> > > Ram
> > >
> > > On Wed, Aug 5, 2015 at 10:55 PM, Jean-Marc Spaggiari <
> > > jean-marc@spaggiari.org> wrote:
> > >
> > > > I tried that with 1.0.1 but I think it's the same with 1.1.0.
> > > >
> > > > How to reproduce:
> > > >
> > > > create 'testflush', 'f1'
> > > > put 'testflush', 'row1', 'f1:c1', 'val1'
> > > > flush 'testflush'
> > > > put 'testflush', 'row1', 'f1:c1', 'val1'
> > > > flush 'testflush'
> > > > put 'testflush', 'row1', 'f1:c1', 'val1'
> > > > flush 'testflush'
> > > > put 'testflush', 'row1', 'f1:c1', 'val1'
> > > > flush 'testflush'
> > > > put 'testflush', 'row1', 'f1:c1', 'val1'
> > > > flush 'testflush'
> > > > put 'testflush', 'row1', 'f1:c1', 'val1'
> > > > flush 'testflush'
> > > > put 'testflush', 'row1', 'f1:c1', 'val1'
> > > > flush 'testflush'
> > > > put 'testflush', 'row1', 'f1:c1', 'val1'
> > > > flush 'testflush'
> > > > put 'testflush', 'row1', 'f1:c1', 'val1'
> > > > flush 'testflush'
> > > > put 'testflush', 'row1', 'f1:c1', 'val1'
> > > > flush 'testflush'
> > > > put 'testflush', 'row1', 'f1:c1', 'val1'
> > > > flush 'testflush'
> > > >
> > > > Result:
> > > > jmspaggiari@t430s:~/hbase-1.0.0$ tree
> > > > /tmp/hbase-jmspaggiari/hbase/data/default/
> > > > /tmp/hbase-jmspaggiari/hbase/data/default/
> > > > └── testflush
> > > > └── 6546495c8df1e1b0a5a5a11d5df23e44
> > > > ├── f1
> > > > │ ├── 195e2b5af8e842f89440927a12c38c25
> > > > │ ├── 266ef04cefe944249d72791a5741a64e
> > > > │ ├── 33b01c2bf3c34d98a01bba95ffa58d0c
> > > > │ ├── 365dbdc33f324f67bf466965cc6ad9e8
> > > > │ ├── 47a47d405bce4bf0a261a16a1ba5694f
> > > > │ ├── 54b7a073289b459292d0eff447d413a5
> > > > │ ├── 55fafb3c30fa4067a3bab54823478cdd
> > > > │ ├── 923ddff172a542b6a5ba6f3467983efa
> > > > │ ├── c69127daa4d34fe8b832e1fafea42713
> > > > │ ├── da4fe34334824f6ea9f5a12dfb93cab9
> > > > │ └── f3a15c326535420cb68d505ce9798d40
> > > > └── recovered.edits
> > > > └── 2.seqid
> > > >
> > > > 4 directories, 12 files
> > > >
> > > > I can test the same on 1.1.0 if you want, just let me know.
> > > >
> > > > 2015-08-05 13:18 GMT-04:00 Ted Yu <yu...@gmail.com>:
> > > >
> > > > > Can you provide some more detail please ?
> > > > >
> > > > > Release of hbase
> > > > > region server log (snippet) showing that compaction is skipped
> > > > >
> > > > > Thanks
> > > > >
> > > > > On Wed, Aug 5, 2015 at 10:15 AM, Jean-Marc Spaggiari <
> > > > > jean-marc@spaggiari.org> wrote:
> > > > >
> > > > > > Quick question here.
> > > > > >
> > > > > > Compactions seems to be triggered when we flush automatically,
> but
> > > not
> > > > > when
> > > > > > we flush manually.
> > > > > >
> > > > > > for 1 to 10
> > > > > > put into a table
> > > > > > flush the table
> > > > > > end of for
> > > > > >
> > > > > > Will create 10 files, and will not trigger any compaction.
> > > > > >
> > > > > > Should we also compact after manual flushs?
> > > > > >
> > > > > > JM
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Flushs and compactions
Posted by Esteban Gutierrez <es...@cloudera.com>.
>From the logs it looks like the flushes requested the compactions as
expected once the min number of files for a minor compaction was reached,
but I don't see any compaction running afterwards:
2015-08-06 09:21:57,465 INFO
[StoreOpener-3d02eb59261771da84824c44e6d2ca40-1]
compactions.CompactionConfiguration: size [134217728, 9223372036854775807);
files [3, 10); ratio 1,200000; off-peak ratio 5,000000; throttle point
2684354560; major period 604800000, major jitter 0,500000, min locality to
compact 0,000000.
...
2015-08-06 09:21:57,854 INFO
[rs(localhost,33882,1438867297154)-flush-proc-pool5-thread-1]
regionserver.HRegion: Finished memstore flush of ~160 B/160, currentsize=0
B/0 for region testflush,,1438867317389.3d02eb59261771da84824c44e6d2ca40.
in 77ms, sequenceid=6, compaction requested=false
...
2015-08-06 09:21:58,097 INFO
[rs(localhost,33882,1438867297154)-flush-proc-pool6-thread-1]
regionserver.HRegion: Finished memstore flush of ~160 B/160, currentsize=0
B/0 for region testflush,,1438867317389.3d02eb59261771da84824c44e6d2ca40.
in 12ms, sequenceid=11, compaction requested=false
...
2015-08-06 09:21:58,318 INFO
[rs(localhost,33882,1438867297154)-flush-proc-pool7-thread-1]
regionserver.HRegion: Finished memstore flush of ~160 B/160, currentsize=0
B/0 for region testflush,,1438867317389.3d02eb59261771da84824c44e6d2ca40.
in 9ms, sequenceid=16, compaction requested=true
... no compactions here ...
2015-08-06 10:31:37,420 INFO [LruBlockCacheStatsExecutor]
hfile.LruBlockCache: totalSize=1.61 MB, freeSize=1.53 GB, max=1.53 GB,
blockCount=1, accesses=1, hits=0, hitRatio=0, cachingAccesses=1,
cachingHits=0, cachingHitsRatio=0,evictions=419, evicted=0,
evictedPerRun=0.0
Are you sure you have the correct log4j config? I will try to reproduce
this and I let you know JMS.
esteban.
--
Cloudera, Inc.
On Thu, Aug 6, 2015 at 7:35 AM, Jean-Marc Spaggiari <jean-marc@spaggiari.org
> wrote:
> Nothing on the logs...
>
> http://pastebin.com/4U6Fmxt9
>
> Running standalone. Kept the server running for 1h after I tried, still no
> compactions.
>
> 2015-08-06 0:52 GMT-04:00 ramkrishna vasudevan <
> ramkrishna.s.vasudevan@gmail.com>:
>
> > Any log saying skipping the compaction? I need to check the code here.
> > Is the compaction thread picking up at a later point of time?
> >
> > Regards
> > Ram
> >
> > On Wed, Aug 5, 2015 at 10:55 PM, Jean-Marc Spaggiari <
> > jean-marc@spaggiari.org> wrote:
> >
> > > I tried that with 1.0.1 but I think it's the same with 1.1.0.
> > >
> > > How to reproduce:
> > >
> > > create 'testflush', 'f1'
> > > put 'testflush', 'row1', 'f1:c1', 'val1'
> > > flush 'testflush'
> > > put 'testflush', 'row1', 'f1:c1', 'val1'
> > > flush 'testflush'
> > > put 'testflush', 'row1', 'f1:c1', 'val1'
> > > flush 'testflush'
> > > put 'testflush', 'row1', 'f1:c1', 'val1'
> > > flush 'testflush'
> > > put 'testflush', 'row1', 'f1:c1', 'val1'
> > > flush 'testflush'
> > > put 'testflush', 'row1', 'f1:c1', 'val1'
> > > flush 'testflush'
> > > put 'testflush', 'row1', 'f1:c1', 'val1'
> > > flush 'testflush'
> > > put 'testflush', 'row1', 'f1:c1', 'val1'
> > > flush 'testflush'
> > > put 'testflush', 'row1', 'f1:c1', 'val1'
> > > flush 'testflush'
> > > put 'testflush', 'row1', 'f1:c1', 'val1'
> > > flush 'testflush'
> > > put 'testflush', 'row1', 'f1:c1', 'val1'
> > > flush 'testflush'
> > >
> > > Result:
> > > jmspaggiari@t430s:~/hbase-1.0.0$ tree
> > > /tmp/hbase-jmspaggiari/hbase/data/default/
> > > /tmp/hbase-jmspaggiari/hbase/data/default/
> > > └── testflush
> > > └── 6546495c8df1e1b0a5a5a11d5df23e44
> > > ├── f1
> > > │ ├── 195e2b5af8e842f89440927a12c38c25
> > > │ ├── 266ef04cefe944249d72791a5741a64e
> > > │ ├── 33b01c2bf3c34d98a01bba95ffa58d0c
> > > │ ├── 365dbdc33f324f67bf466965cc6ad9e8
> > > │ ├── 47a47d405bce4bf0a261a16a1ba5694f
> > > │ ├── 54b7a073289b459292d0eff447d413a5
> > > │ ├── 55fafb3c30fa4067a3bab54823478cdd
> > > │ ├── 923ddff172a542b6a5ba6f3467983efa
> > > │ ├── c69127daa4d34fe8b832e1fafea42713
> > > │ ├── da4fe34334824f6ea9f5a12dfb93cab9
> > > │ └── f3a15c326535420cb68d505ce9798d40
> > > └── recovered.edits
> > > └── 2.seqid
> > >
> > > 4 directories, 12 files
> > >
> > > I can test the same on 1.1.0 if you want, just let me know.
> > >
> > > 2015-08-05 13:18 GMT-04:00 Ted Yu <yu...@gmail.com>:
> > >
> > > > Can you provide some more detail please ?
> > > >
> > > > Release of hbase
> > > > region server log (snippet) showing that compaction is skipped
> > > >
> > > > Thanks
> > > >
> > > > On Wed, Aug 5, 2015 at 10:15 AM, Jean-Marc Spaggiari <
> > > > jean-marc@spaggiari.org> wrote:
> > > >
> > > > > Quick question here.
> > > > >
> > > > > Compactions seems to be triggered when we flush automatically, but
> > not
> > > > when
> > > > > we flush manually.
> > > > >
> > > > > for 1 to 10
> > > > > put into a table
> > > > > flush the table
> > > > > end of for
> > > > >
> > > > > Will create 10 files, and will not trigger any compaction.
> > > > >
> > > > > Should we also compact after manual flushs?
> > > > >
> > > > > JM
> > > > >
> > > >
> > >
> >
>
Re: Flushs and compactions
Posted by Jean-Marc Spaggiari <je...@spaggiari.org>.
Nothing on the logs...
http://pastebin.com/4U6Fmxt9
Running standalone. Kept the server running for 1h after I tried, still no
compactions.
2015-08-06 0:52 GMT-04:00 ramkrishna vasudevan <
ramkrishna.s.vasudevan@gmail.com>:
> Any log saying skipping the compaction? I need to check the code here.
> Is the compaction thread picking up at a later point of time?
>
> Regards
> Ram
>
> On Wed, Aug 5, 2015 at 10:55 PM, Jean-Marc Spaggiari <
> jean-marc@spaggiari.org> wrote:
>
> > I tried that with 1.0.1 but I think it's the same with 1.1.0.
> >
> > How to reproduce:
> >
> > create 'testflush', 'f1'
> > put 'testflush', 'row1', 'f1:c1', 'val1'
> > flush 'testflush'
> > put 'testflush', 'row1', 'f1:c1', 'val1'
> > flush 'testflush'
> > put 'testflush', 'row1', 'f1:c1', 'val1'
> > flush 'testflush'
> > put 'testflush', 'row1', 'f1:c1', 'val1'
> > flush 'testflush'
> > put 'testflush', 'row1', 'f1:c1', 'val1'
> > flush 'testflush'
> > put 'testflush', 'row1', 'f1:c1', 'val1'
> > flush 'testflush'
> > put 'testflush', 'row1', 'f1:c1', 'val1'
> > flush 'testflush'
> > put 'testflush', 'row1', 'f1:c1', 'val1'
> > flush 'testflush'
> > put 'testflush', 'row1', 'f1:c1', 'val1'
> > flush 'testflush'
> > put 'testflush', 'row1', 'f1:c1', 'val1'
> > flush 'testflush'
> > put 'testflush', 'row1', 'f1:c1', 'val1'
> > flush 'testflush'
> >
> > Result:
> > jmspaggiari@t430s:~/hbase-1.0.0$ tree
> > /tmp/hbase-jmspaggiari/hbase/data/default/
> > /tmp/hbase-jmspaggiari/hbase/data/default/
> > └── testflush
> > └── 6546495c8df1e1b0a5a5a11d5df23e44
> > ├── f1
> > │ ├── 195e2b5af8e842f89440927a12c38c25
> > │ ├── 266ef04cefe944249d72791a5741a64e
> > │ ├── 33b01c2bf3c34d98a01bba95ffa58d0c
> > │ ├── 365dbdc33f324f67bf466965cc6ad9e8
> > │ ├── 47a47d405bce4bf0a261a16a1ba5694f
> > │ ├── 54b7a073289b459292d0eff447d413a5
> > │ ├── 55fafb3c30fa4067a3bab54823478cdd
> > │ ├── 923ddff172a542b6a5ba6f3467983efa
> > │ ├── c69127daa4d34fe8b832e1fafea42713
> > │ ├── da4fe34334824f6ea9f5a12dfb93cab9
> > │ └── f3a15c326535420cb68d505ce9798d40
> > └── recovered.edits
> > └── 2.seqid
> >
> > 4 directories, 12 files
> >
> > I can test the same on 1.1.0 if you want, just let me know.
> >
> > 2015-08-05 13:18 GMT-04:00 Ted Yu <yu...@gmail.com>:
> >
> > > Can you provide some more detail please ?
> > >
> > > Release of hbase
> > > region server log (snippet) showing that compaction is skipped
> > >
> > > Thanks
> > >
> > > On Wed, Aug 5, 2015 at 10:15 AM, Jean-Marc Spaggiari <
> > > jean-marc@spaggiari.org> wrote:
> > >
> > > > Quick question here.
> > > >
> > > > Compactions seems to be triggered when we flush automatically, but
> not
> > > when
> > > > we flush manually.
> > > >
> > > > for 1 to 10
> > > > put into a table
> > > > flush the table
> > > > end of for
> > > >
> > > > Will create 10 files, and will not trigger any compaction.
> > > >
> > > > Should we also compact after manual flushs?
> > > >
> > > > JM
> > > >
> > >
> >
>
Re: Flushs and compactions
Posted by ramkrishna vasudevan <ra...@gmail.com>.
Any log saying skipping the compaction? I need to check the code here.
Is the compaction thread picking up at a later point of time?
Regards
Ram
On Wed, Aug 5, 2015 at 10:55 PM, Jean-Marc Spaggiari <
jean-marc@spaggiari.org> wrote:
> I tried that with 1.0.1 but I think it's the same with 1.1.0.
>
> How to reproduce:
>
> create 'testflush', 'f1'
> put 'testflush', 'row1', 'f1:c1', 'val1'
> flush 'testflush'
> put 'testflush', 'row1', 'f1:c1', 'val1'
> flush 'testflush'
> put 'testflush', 'row1', 'f1:c1', 'val1'
> flush 'testflush'
> put 'testflush', 'row1', 'f1:c1', 'val1'
> flush 'testflush'
> put 'testflush', 'row1', 'f1:c1', 'val1'
> flush 'testflush'
> put 'testflush', 'row1', 'f1:c1', 'val1'
> flush 'testflush'
> put 'testflush', 'row1', 'f1:c1', 'val1'
> flush 'testflush'
> put 'testflush', 'row1', 'f1:c1', 'val1'
> flush 'testflush'
> put 'testflush', 'row1', 'f1:c1', 'val1'
> flush 'testflush'
> put 'testflush', 'row1', 'f1:c1', 'val1'
> flush 'testflush'
> put 'testflush', 'row1', 'f1:c1', 'val1'
> flush 'testflush'
>
> Result:
> jmspaggiari@t430s:~/hbase-1.0.0$ tree
> /tmp/hbase-jmspaggiari/hbase/data/default/
> /tmp/hbase-jmspaggiari/hbase/data/default/
> └── testflush
> └── 6546495c8df1e1b0a5a5a11d5df23e44
> ├── f1
> │ ├── 195e2b5af8e842f89440927a12c38c25
> │ ├── 266ef04cefe944249d72791a5741a64e
> │ ├── 33b01c2bf3c34d98a01bba95ffa58d0c
> │ ├── 365dbdc33f324f67bf466965cc6ad9e8
> │ ├── 47a47d405bce4bf0a261a16a1ba5694f
> │ ├── 54b7a073289b459292d0eff447d413a5
> │ ├── 55fafb3c30fa4067a3bab54823478cdd
> │ ├── 923ddff172a542b6a5ba6f3467983efa
> │ ├── c69127daa4d34fe8b832e1fafea42713
> │ ├── da4fe34334824f6ea9f5a12dfb93cab9
> │ └── f3a15c326535420cb68d505ce9798d40
> └── recovered.edits
> └── 2.seqid
>
> 4 directories, 12 files
>
> I can test the same on 1.1.0 if you want, just let me know.
>
> 2015-08-05 13:18 GMT-04:00 Ted Yu <yu...@gmail.com>:
>
> > Can you provide some more detail please ?
> >
> > Release of hbase
> > region server log (snippet) showing that compaction is skipped
> >
> > Thanks
> >
> > On Wed, Aug 5, 2015 at 10:15 AM, Jean-Marc Spaggiari <
> > jean-marc@spaggiari.org> wrote:
> >
> > > Quick question here.
> > >
> > > Compactions seems to be triggered when we flush automatically, but not
> > when
> > > we flush manually.
> > >
> > > for 1 to 10
> > > put into a table
> > > flush the table
> > > end of for
> > >
> > > Will create 10 files, and will not trigger any compaction.
> > >
> > > Should we also compact after manual flushs?
> > >
> > > JM
> > >
> >
>
Re: Flushs and compactions
Posted by Jean-Marc Spaggiari <je...@spaggiari.org>.
I tried that with 1.0.1 but I think it's the same with 1.1.0.
How to reproduce:
create 'testflush', 'f1'
put 'testflush', 'row1', 'f1:c1', 'val1'
flush 'testflush'
put 'testflush', 'row1', 'f1:c1', 'val1'
flush 'testflush'
put 'testflush', 'row1', 'f1:c1', 'val1'
flush 'testflush'
put 'testflush', 'row1', 'f1:c1', 'val1'
flush 'testflush'
put 'testflush', 'row1', 'f1:c1', 'val1'
flush 'testflush'
put 'testflush', 'row1', 'f1:c1', 'val1'
flush 'testflush'
put 'testflush', 'row1', 'f1:c1', 'val1'
flush 'testflush'
put 'testflush', 'row1', 'f1:c1', 'val1'
flush 'testflush'
put 'testflush', 'row1', 'f1:c1', 'val1'
flush 'testflush'
put 'testflush', 'row1', 'f1:c1', 'val1'
flush 'testflush'
put 'testflush', 'row1', 'f1:c1', 'val1'
flush 'testflush'
Result:
jmspaggiari@t430s:~/hbase-1.0.0$ tree
/tmp/hbase-jmspaggiari/hbase/data/default/
/tmp/hbase-jmspaggiari/hbase/data/default/
└── testflush
└── 6546495c8df1e1b0a5a5a11d5df23e44
├── f1
│ ├── 195e2b5af8e842f89440927a12c38c25
│ ├── 266ef04cefe944249d72791a5741a64e
│ ├── 33b01c2bf3c34d98a01bba95ffa58d0c
│ ├── 365dbdc33f324f67bf466965cc6ad9e8
│ ├── 47a47d405bce4bf0a261a16a1ba5694f
│ ├── 54b7a073289b459292d0eff447d413a5
│ ├── 55fafb3c30fa4067a3bab54823478cdd
│ ├── 923ddff172a542b6a5ba6f3467983efa
│ ├── c69127daa4d34fe8b832e1fafea42713
│ ├── da4fe34334824f6ea9f5a12dfb93cab9
│ └── f3a15c326535420cb68d505ce9798d40
└── recovered.edits
└── 2.seqid
4 directories, 12 files
I can test the same on 1.1.0 if you want, just let me know.
2015-08-05 13:18 GMT-04:00 Ted Yu <yu...@gmail.com>:
> Can you provide some more detail please ?
>
> Release of hbase
> region server log (snippet) showing that compaction is skipped
>
> Thanks
>
> On Wed, Aug 5, 2015 at 10:15 AM, Jean-Marc Spaggiari <
> jean-marc@spaggiari.org> wrote:
>
> > Quick question here.
> >
> > Compactions seems to be triggered when we flush automatically, but not
> when
> > we flush manually.
> >
> > for 1 to 10
> > put into a table
> > flush the table
> > end of for
> >
> > Will create 10 files, and will not trigger any compaction.
> >
> > Should we also compact after manual flushs?
> >
> > JM
> >
>
Re: Flushs and compactions
Posted by Ted Yu <yu...@gmail.com>.
Can you provide some more detail please ?
Release of hbase
region server log (snippet) showing that compaction is skipped
Thanks
On Wed, Aug 5, 2015 at 10:15 AM, Jean-Marc Spaggiari <
jean-marc@spaggiari.org> wrote:
> Quick question here.
>
> Compactions seems to be triggered when we flush automatically, but not when
> we flush manually.
>
> for 1 to 10
> put into a table
> flush the table
> end of for
>
> Will create 10 files, and will not trigger any compaction.
>
> Should we also compact after manual flushs?
>
> JM
>