You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Li Yong <la...@foxmail.com> on 2014/07/23 10:22:15 UTC
Too many delete delay
Hi all,
I move my cluster to another cluster with CopyTable, then I found too many timeout, Most of them are Delete! I found the table has too many storefiels, so i do a major_compact, then things be better. After an hour, timeout again! I don't know why so many delay in delete, not put, not scan. Can anyone tell me, when will delete operation use too much time? Thanks!
[ HBase version : 0.94.9 Hadoop version : 1.0.4 ]
回复： Too many delete delay
Posted by Li Yong <la...@foxmail.com>.
Logs here, all periodicFlusher and cacheFlusher, I don't see any other logs.
2014-07-23 18:20:37,227 INFO [regionserver60020.periodicFlusher] regionserver.HRegionServer: regionserver60020.periodicFlusher requesting flush for region contacts,5c,1405328195955.804e9746463e04db5184c898e8a2324e. after a delay of 18996
2014-07-23 18:20:37,327 INFO [regionserver60020.periodicFlusher] regionserver.HRegionServer: regionserver60020.periodicFlusher requesting flush for region gzone_feed,47,1405320650874.866d48ae4c774f1c22a289409a0ac4e4. after a delay of 21268
2014-07-23 18:20:37,327 INFO [regionserver60020.periodicFlusher] regionserver.HRegionServer: regionserver60020.periodicFlusher requesting flush for region contacts,5c,1405328195955.804e9746463e04db5184c898e8a2324e. after a delay of 13740
2014-07-23 18:20:37,427 INFO [regionserver60020.periodicFlusher] regionserver.HRegionServer: regionserver60020.periodicFlusher requesting flush for region gzone_feed,47,1405320650874.866d48ae4c774f1c22a289409a0ac4e4. after a delay of 3561
2014-07-23 18:20:37,427 INFO [regionserver60020.periodicFlusher] regionserver.HRegionServer: regionserver60020.periodicFlusher requesting flush for region contacts,5c,1405328195955.804e9746463e04db5184c898e8a2324e. after a delay of 7662
2014-07-23 18:20:37,527 INFO [regionserver60020.periodicFlusher] regionserver.HRegionServer: regionserver60020.periodicFlusher requesting flush for region gzone_feed,47,1405320650874.866d48ae4c774f1c22a289409a0ac4e4. after a delay of 5499
2014-07-23 18:20:37,527 INFO [regionserver60020.periodicFlusher] regionserver.HRegionServer: regionserver60020.periodicFlusher requesting flush for region contacts,5c,1405328195955.804e9746463e04db5184c898e8a2324e. after a delay of 5585
2014-07-23 18:20:37,627 INFO [regionserver60020.periodicFlusher] regionserver.HRegionServer: regionserver60020.periodicFlusher requesting flush for region gzone_feed,47,1405320650874.866d48ae4c774f1c22a289409a0ac4e4. after a delay of 9535
2014-07-23 18:20:37,627 INFO [regionserver60020.periodicFlusher] regionserver.HRegionServer: regionserver60020.periodicFlusher requesting flush for region contacts,5c,1405328195955.804e9746463e04db5184c898e8a2324e. after a delay of 10826
2014-07-23 18:20:37,727 INFO [regionserver60020.periodicFlusher] regionserver.HRegionServer: regionserver60020.periodicFlusher requesting flush for region gzone_feed,47,1405320650874.866d48ae4c774f1c22a289409a0ac4e4. after a delay of 9893
2014-07-23 18:20:37,727 INFO [regionserver60020.periodicFlusher] regionserver.HRegionServer: regionserver60020.periodicFlusher requesting flush for region contacts,5c,1405328195955.804e9746463e04db5184c898e8a2324e. after a delay of 6856
2014-07-23 18:20:37,754 DEBUG [regionserver60020.cacheFlusher] regionserver.HRegion: Started memstore flush for gzone_feed,47,1405320650874.866d48ae4c774f1c22a289409a0ac4e4., current region memstore size 52.0k
2014-07-23 18:20:37,754 DEBUG [regionserver60020.cacheFlusher] regionserver.HRegion: Finished snapshotting gzone_feed,47,1405320650874.866d48ae4c774f1c22a289409a0ac4e4., commencing wait for mvcc, flushsize=53272
2014-07-23 18:20:37,754 DEBUG [regionserver60020.cacheFlusher] regionserver.HRegion: Finished snapshotting, commencing flushing stores
2014-07-23 18:20:37,760 DEBUG [regionserver60020.cacheFlusher] util.FSUtils: Creating file=hdfs://namenode001.m6:8020/hbase/gzone_feed/866d48ae4c774f1c22a289409a0ac4e4/.tmp/733507fa34c54c3c9bd0ac69d6a7dcd4 with permission=rwxrwxrwx
2014-07-23 18:20:37,760 INFO [regionserver60020.cacheFlusher] util.FSUtils: FileSystem doesn't support getDefaultReplication
2014-07-23 18:20:37,760 INFO [regionserver60020.cacheFlusher] util.FSUtils: FileSystem doesn't support getDefaultBlockSize
2014-07-23 18:20:37,761 DEBUG [regionserver60020.cacheFlusher] hfile.HFileWriterV2: Initialized with CacheConfig:enabled [cacheDataOnRead=true] [cacheDataOnWrite=false] [cacheIndexesOnWrite=false] [cacheBloomsOnWrite=false] [cacheEvictOnClose=false] [cacheCompressed=false]
2014-07-23 18:20:37,761 INFO [regionserver60020.cacheFlusher] regionserver.StoreFile: Bloom filter type for hdfs://namenode001.m6:8020/hbase/gzone_feed/866d48ae4c774f1c22a289409a0ac4e4/.tmp/733507fa34c54c3c9bd0ac69d6a7dcd4: ROW, CompoundBloomFilterWriter
2014-07-23 18:20:37,761 INFO [regionserver60020.cacheFlusher] regionserver.StoreFile: Delete Family Bloom filter type for hdfs://namenode001.m6:8020/hbase/gzone_feed/866d48ae4c774f1c22a289409a0ac4e4/.tmp/733507fa34c54c3c9bd0ac69d6a7dcd4: CompoundBloomFilterWriter
2014-07-23 18:20:37,762 DEBUG [regionserver60020.cacheFlusher] util.CompoundBloomFilterWriter: Compacted Bloom chunk #0 from [109306 max keys, 131072 bytes] to [106 max keys, 128 bytes]
2014-07-23 18:20:37,772 INFO [regionserver60020.cacheFlusher] regionserver.StoreFile: General Bloom and NO DeleteFamily was added to HFile (hdfs://namenode001.m6:8020/hbase/gzone_feed/866d48ae4c774f1c22a289409a0ac4e4/.tmp/733507fa34c54c3c9bd0ac69d6a7dcd4)
2014-07-23 18:20:37,772 INFO [regionserver60020.cacheFlusher] regionserver.Store: Flushed , sequenceid=1643049017, memsize=51.5k, into tmp file hdfs://namenode001.m6:8020/hbase/gzone_feed/866d48ae4c774f1c22a289409a0ac4e4/.tmp/733507fa34c54c3c9bd0ac69d6a7dcd4
2014-07-23 18:20:37,775 INFO [regionserver60020.cacheFlusher] regionserver.StoreFile$Reader: Loaded ROW (CompoundBloomFilter) metadata for 733507fa34c54c3c9bd0ac69d6a7dcd4
2014-07-23 18:20:37,775 DEBUG [regionserver60020.cacheFlusher] regionserver.Store: Renaming flushed file at hdfs://namenode001.m6:8020/hbase/gzone_feed/866d48ae4c774f1c22a289409a0ac4e4/.tmp/733507fa34c54c3c9bd0ac69d6a7dcd4 to hdfs://namenode001.m6:8020/hbase/gzone_feed/866d48ae4c774f1c22a289409a0ac4e4/gfd/733507fa34c54c3c9bd0ac69d6a7dcd4
2014-07-23 18:20:37,778 INFO [regionserver60020.cacheFlusher] regionserver.StoreFile$Reader: Loaded ROW (CompoundBloomFilter) metadata for 733507fa34c54c3c9bd0ac69d6a7dcd4
2014-07-23 18:20:37,778 INFO [regionserver60020.cacheFlusher] regionserver.Store: Added hdfs://namenode001.m6:8020/hbase/gzone_feed/866d48ae4c774f1c22a289409a0ac4e4/gfd/733507fa34c54c3c9bd0ac69d6a7dcd4, entries=59, sequenceid=1643049017, filesize=22.5k
2014-07-23 18:20:37,779 INFO [regionserver60020.cacheFlusher] regionserver.HRegion: Finished memstore flush of ~52.0k/53272, currentsize=0.0/0 for region gzone_feed,47,1405320650874.866d48ae4c774f1c22a289409a0ac4e4. in 25ms, sequenceid=1643049017, compaction requested=false
2014-07-23 18:20:37,827 INFO [regionserver60020.periodicFlusher] regionserver.HRegionServer: regionserver60020.periodicFlusher requesting flush for region contacts,5c,1405328195955.804e9746463e04db5184c898e8a2324e. after a delay of 4168
2014-07-23 18:20:37,927 INFO [regionserver60020.periodicFlusher] regionserver.HRegionServer: regionserver60020.periodicFlusher requesting flush for region contacts,5c,1405328195955.804e9746463e04db5184c898e8a2324e. after a delay of 7616
2014-07-23 18:20:38,090 INFO [regionserver60020.periodicFlusher] regionserver.HRegionServer: regionserver60020.periodicFlusher requesting flush for region contacts,5c,1405328195955.804e9746463e04db5184c898e8a2324e. after a delay of 8700
2014-07-23 18:20:38,190 INFO [regionserver60020.periodicFlusher] regionserver.HRegionServer: regionserver60020.periodicFlusher requesting flush for region contacts,5c,1405328195955.804e9746463e04db5184c898e8a2324e. after a delay of 13115
2014-07-23 18:20:38,290 INFO [regionserver60020.periodicFlusher] regionserver.HRegionServer: regionserver60020.periodicFlusher requesting flush for region contacts,5c,1405328195955.804e9746463e04db5184c898e8a2324e. after a delay of 9066
2014-07-23 18:20:38,390 INFO [regionserver60020.periodicFlusher] regionserver.HRegionServer: regionserver60020.periodicFlusher requesting flush for region contacts,5c,1405328195955.804e9746463e04db5184c898e8a2324e. after a delay of 19966
2014-07-23 18:20:38,490 INFO [regionserver60020.periodicFlusher] regionserver.HRegionServer: regionserver60020.periodicFlusher requesting flush for region contacts,5c,1405328195955.804e9746463e04db5184c898e8a2324e. after a delay of 15650
2014-07-23 18:20:38,591 INFO [regionserver60020.periodicFlusher] regionserver.HRegionServer: regionserver60020.periodicFlusher requesting flush for region contacts,5c,1405328195955.804e9746463e04db5184c898e8a2324e. after a delay of 3669
2014-07-23 18:20:38,691 INFO [regionserver60020.periodicFlusher] regionserver.HRegionServer: regionserver60020.periodicFlusher requesting flush for region contacts,5c,1405328195955.804e9746463e04db5184c898e8a2324e. after a delay of 6606
2014-07-23 18:20:38,791 INFO [regionserver60020.periodicFlusher] regionserver.HRegionServer: regionserver60020.periodicFlusher requesting flush for region contacts,5c,1405328195955.804e9746463e04db5184c898e8a2324e. after a delay of 11460
2014-07-23 18:20:38,891 INFO [regionserver60020.periodicFlusher] regionserver.HRegionServer: regionserver60020.periodicFlusher requesting flush for region contacts,5c,1405328195955.804e9746463e04db5184c898e8a2324e. after a delay of 17366
2014-07-23 18:20:38,991 INFO [regionserver60020.periodicFlusher] regionserver.HRegionServer: regionserver60020.periodicFlusher requesting flush for region contacts,5c,1405328195955.804e9746463e04db5184c898e8a2324e. after a delay of 21608
2014-07-23 18:20:39,091 INFO [regionserver60020.periodicFlusher] regionserver.HRegionServer: regionserver60020.periodicFlusher requesting flush for region contacts,5c,1405328195955.804e9746463e04db5184c898e8a2324e. after a delay of 22710
2014-07-23 18:20:39,191 INFO [regionserver60020.periodicFlusher] regionserver.HRegionServer: regionserver60020.periodicFlusher requesting flush for region contacts,5c,1405328195955.804e9746463e04db5184c898e8a2324e. after a delay of 10969
2014-07-23 18:20:39,291 INFO [regionserver60020.periodicFlusher] regionserver.HRegionServer: regionserver60020.periodicFlusher requesting flush for region contacts,5c,1405328195955.804e9746463e04db5184c898e8a2324e. after a delay of 20242
2014-07-23 18:20:39,391 INFO [regionserver60020.periodicFlusher] regionserver.HRegionServer: regionserver60020.periodicFlusher requesting flush for region contacts,5c,1405328195955.804e9746463e04db5184c898e8a2324e. after a delay of 18010
2014-07-23 18:20:39,491 INFO [regionserver60020.periodicFlusher] regionserver.HRegionServer: regionserver60020.periodicFlusher requesting flush for region contacts,5c,1405328195955.804e9746463e04db5184c898e8a2324e. after a delay of 20988
2014-07-23 18:20:39,591 INFO [regionserver60020.periodicFlusher] regionserver.HRegionServer: regionserver60020.periodicFlusher requesting flush for region contacts,5c,1405328195955.804e9746463e04db5184c898e8a2324e. after a delay of 11394
2014-07-23 18:20:39,692 INFO [regionserver60020.periodicFlusher] regionserver.HRegionServer: regionserver60020.periodicFlusher requesting flush for region contacts,5c,1405328195955.804e9746463e04db5184c898e8a2324e. after a delay of 7605
2014-07-23 18:20:39,791 INFO [regionserver60020.periodicFlusher] regionserver.HRegionServer: regionserver60020.periodicFlusher requesting flush for region contacts,5c,1405328195955.804e9746463e04db5184c898e8a2324e. after a delay of 7173
2014-07-23 18:20:39,891 INFO [regionserver60020.periodicFlusher] regionserver.HRegionServer: regionserver60020.periodicFlusher requesting flush for region contacts,5c,1405328195955.804e9746463e04db5184c898e8a2324e. after a delay of 21445
2014-07-23 18:20:39,991 INFO [regionserver60020.periodicFlusher] regionserver.HRegionServer: regionserver60020.periodicFlusher requesting flush for region contacts,5c,1405328195955.804e9746463e04db5184c898e8a2324e. after a delay of 13041
2014-07-23 18:20:40,091 INFO [regionserver60020.periodicFlusher] regionserver.HRegionServer: regionserver60020.periodicFlusher requesting flush for region contacts,5c,1405328195955.804e9746463e04db5184c898e8a2324e. after a delay of 19472
2014-07-23 18:20:40,191 INFO [regionserver60020.periodicFlusher] regionserver.HRegionServer: regionserver60020.periodicFlusher requesting flush for region contacts,5c,1405328195955.804e9746463e04db5184c898e8a2324e. after a delay of 14905
2014-07-23 18:20:40,292 INFO [regionserver60020.periodicFlusher] regionserver.HRegionServer: regionserver60020.periodicFlusher requesting flush for region contacts,5c,1405328195955.804e9746463e04db5184c898e8a2324e. after a delay of 19938
2014-07-23 18:20:40,393 INFO [regionserver60020.periodicFlusher] regionserver.HRegionServer: regionserver60020.periodicFlusher requesting flush for region contacts,5c,1405328195955.804e9746463e04db5184c898e8a2324e. after a delay of 6972
2014-07-23 18:20:40,493 INFO [regionserver60020.periodicFlusher] regionserver.HRegionServer: regionserver60020.periodicFlusher requesting flush for region contacts,5c,1405328195955.804e9746463e04db5184c898e8a2324e. after a delay of 14143
2014-07-23 18:20:40,594 INFO [regionserver60020.periodicFlusher] regionserver.HRegionServer: regionserver60020.periodicFlusher requesting flush for region contacts,5c,1405328195955.804e9746463e04db5184c898e8a2324e. after a delay of 17582
2014-07-23 18:20:40,694 INFO [regionserver60020.periodicFlusher] regionserver.HRegionServer: regionserver60020.periodicFlusher requesting flush for region contacts,5c,1405328195955.804e9746463e04db5184c898e8a2324e. after a delay of 12537
2014-07-23 18:20:40,794 INFO [regionserver60020.periodicFlusher] regionserver.HRegionServer: regionserver60020.periodicFlusher requesting flush for region contacts,5c,1405328195955.804e9746463e04db5184c898e8a2324e. after a delay of 19323
2014-07-23 18:20:40,894 INFO [regionserver60020.periodicFlusher] regionserver.HRegionServer: regionserver60020.periodicFlusher requesting flush for region contacts,5c,1405328195955.804e9746463e04db5184c898e8a2324e. after a delay of 12464
2014-07-23 18:20:40,994 INFO [regionserver60020.periodicFlusher] regionserver.HRegionServer: regionserver60020.periodicFlusher requesting flush for region contacts,5c,1405328195955.804e9746463e04db5184c898e8a2324e. after a delay of 4562
2014-07-23 18:20:41,094 INFO [regionserver60020.periodicFlusher] regionserver.HRegionServer: regionserver60020.periodicFlusher requesting flush for region contacts,5c,1405328195955.804e9746463e04db5184c898e8a2324e. after a delay of 18853
2014-07-23 18:20:41,194 INFO [regionserver60020.periodicFlusher] regionserver.HRegionServer: regionserver60020.periodicFlusher requesting flush for region contacts,5c,1405328195955.804e9746463e04db5184c898e8a2324e. after a delay of 16893
2014-07-23 18:20:41,294 INFO [regionserver60020.periodicFlusher] regionserver.HRegionServer: regionserver60020.periodicFlusher requesting flush for region contacts,5c,1405328195955.804e9746463e04db5184c898e8a2324e. after a delay of 8540
2014-07-23 18:20:41,394 INFO [regionserver60020.periodicFlusher] regionserver.HRegionServer: regionserver60020.periodicFlusher requesting flush for region contacts,5c,1405328195955.804e9746463e04db5184c898e8a2324e. after a delay of 13745
2014-07-23 18:20:41,494 INFO [regionserver60020.periodicFlusher] regionserver.HRegionServer: regionserver60020.periodicFlusher requesting flush for region contacts,5c,1405328195955.804e9746463e04db5184c898e8a2324e. after a delay of 11255
2014-07-23 18:20:41,594 INFO [regionserver60020.periodicFlusher] regionserver.HRegionServer: regionserver60020.periodicFlusher requesting flush for region contacts,5c,1405328195955.804e9746463e04db5184c898e8a2324e. after a delay of 22645
2014-07-23 18:20:41,695 INFO [regionserver60020.periodicFlusher] regionserver.HRegionServer: regionserver60020.periodicFlusher requesting flush for region contacts,5c,1405328195955.804e9746463e04db5184c898e8a2324e. after a delay of 15900
2014-07-23 18:20:41,795 INFO [regionserver60020.periodicFlusher] regionserver.HRegionServer: regionserver60020.periodicFlusher requesting flush for region contacts,5c,1405328195955.804e9746463e04db5184c898e8a2324e. after a delay of 11104
2014-07-23 18:20:41,895 INFO [regionserver60020.periodicFlusher] regionserver.HRegionServer: regionserver60020.periodicFlusher requesting flush for region contacts,5c,1405328195955.804e9746463e04db5184c898e8a2324e. after a delay of 8389
2014-07-23 18:20:41,995 INFO [regionserver60020.periodicFlusher] regionserver.HRegionServer: regionserver60020.periodicFlusher requesting flush for region contacts,5c,1405328195955.804e9746463e04db5184c898e8a2324e. after a delay of 13591
2014-07-23 18:20:42,095 INFO [regionserver60020.periodicFlusher] regionserver.HRegionServer: regionserver60020.periodicFlusher requesting flush for region contacts,5c,1405328195955.804e9746463e04db5184c898e8a2324e. after a delay of 17391
2014-07-23 18:20:42,195 INFO [regionserver60020.periodicFlusher] regionserver.HRegionServer: regionserver60020.periodicFlusher requesting flush for region contacts,5c,1405328195955.804e9746463e04db5184c898e8a2324e. after a delay of 13307
2014-07-23 18:20:42,296 INFO [regionserver60020.periodicFlusher] regionserver.HRegionServer: regionserver60020.periodicFlusher requesting flush for region contacts,5c,1405328195955.804e9746463e04db5184c898e8a2324e. after a delay of 21090
2014-07-23 18:20:42,396 INFO [regionserver60020.periodicFlusher] regionserver.HRegionServer: regionserver60020.periodicFlusher requesting flush for region contacts,5c,1405328195955.804e9746463e04db5184c898e8a2324e. after a delay of 5536
2014-07-23 18:20:42,436 DEBUG [regionserver60020.cacheFlusher] regionserver.HRegion: Started memstore flush for contacts,5c,1405328195955.804e9746463e04db5184c898e8a2324e., current region memstore size 3.6m
2014-07-23 18:20:42,436 DEBUG [regionserver60020.cacheFlusher] regionserver.HRegion: Finished snapshotting contacts,5c,1405328195955.804e9746463e04db5184c898e8a2324e., commencing wait for mvcc, flushsize=3788720
2014-07-23 18:20:42,436 DEBUG [regionserver60020.cacheFlusher] regionserver.HRegion: Finished snapshotting, commencing flushing stores
2014-07-23 18:20:42,437 DEBUG [regionserver60020.cacheFlusher] util.FSUtils: Creating file=hdfs://namenode001.m6:8020/hbase/contacts/804e9746463e04db5184c898e8a2324e/.tmp/ddf1eb1f101f4917a0b65e8b71088a61 with permission=rwxrwxrwx
2014-07-23 18:20:42,437 INFO [regionserver60020.cacheFlusher] util.FSUtils: FileSystem doesn't support getDefaultReplication
2014-07-23 18:20:42,437 INFO [regionserver60020.cacheFlusher] util.FSUtils: FileSystem doesn't support getDefaultBlockSize
2014-07-23 18:20:42,438 DEBUG [regionserver60020.cacheFlusher] hfile.HFileWriterV2: Initialized with CacheConfig:enabled [cacheDataOnRead=true] [cacheDataOnWrite=false] [cacheIndexesOnWrite=false] [cacheBloomsOnWrite=false] [cacheEvictOnClose=false] [cacheCompressed=false]
2014-07-23 18:20:42,438 INFO [regionserver60020.cacheFlusher] regionserver.StoreFile: Bloom filter type for hdfs://namenode001.m6:8020/hbase/contacts/804e9746463e04db5184c898e8a2324e/.tmp/ddf1eb1f101f4917a0b65e8b71088a61: ROW, CompoundBloomFilterWriter
2014-07-23 18:20:42,438 INFO [regionserver60020.cacheFlusher] regionserver.StoreFile: Delete Family Bloom filter type for hdfs://namenode001.m6:8020/hbase/contacts/804e9746463e04db5184c898e8a2324e/.tmp/ddf1eb1f101f4917a0b65e8b71088a61: CompoundBloomFilterWriter
2014-07-23 18:20:42,487 DEBUG [regionserver60020.cacheFlusher] util.CompoundBloomFilterWriter: Compacted Bloom chunk #0 from [109306 max keys, 131072 bytes] to [1707 max keys, 2048 bytes]
2014-07-23 18:20:42,490 INFO [regionserver60020.cacheFlusher] regionserver.StoreFile: General Bloom and NO DeleteFamily was added to HFile (hdfs://namenode001.m6:8020/hbase/contacts/804e9746463e04db5184c898e8a2324e/.tmp/ddf1eb1f101f4917a0b65e8b71088a61)
2014-07-23 18:20:42,490 INFO [regionserver60020.cacheFlusher] regionserver.Store: Flushed , sequenceid=1643051384, memsize=3.6m, into tmp file hdfs://namenode001.m6:8020/hbase/contacts/804e9746463e04db5184c898e8a2324e/.tmp/ddf1eb1f101f4917a0b65e8b71088a61
2014-07-23 18:20:42,493 INFO [regionserver60020.cacheFlusher] regionserver.StoreFile$Reader: Loaded ROW (CompoundBloomFilter) metadata for ddf1eb1f101f4917a0b65e8b71088a61
2014-07-23 18:20:42,494 DEBUG [regionserver60020.cacheFlusher] regionserver.Store: Renaming flushed file at hdfs://namenode001.m6:8020/hbase/contacts/804e9746463e04db5184c898e8a2324e/.tmp/ddf1eb1f101f4917a0b65e8b71088a61 to hdfs://namenode001.m6:8020/hbase/contacts/804e9746463e04db5184c898e8a2324e/pd/ddf1eb1f101f4917a0b65e8b71088a61
2014-07-23 18:20:42,497 INFO [regionserver60020.cacheFlusher] regionserver.StoreFile$Reader: Loaded ROW (CompoundBloomFilter) metadata for ddf1eb1f101f4917a0b65e8b71088a61
2014-07-23 18:20:42,497 INFO [regionserver60020.cacheFlusher] regionserver.Store: Added hdfs://namenode001.m6:8020/hbase/contacts/804e9746463e04db5184c898e8a2324e/pd/ddf1eb1f101f4917a0b65e8b71088a61, entries=928, sequenceid=1643051384, filesize=3.4m
2014-07-23 18:20:42,497 INFO [regionserver60020.cacheFlusher] regionserver.HRegion: Finished memstore flush of ~3.6m/3788720, currentsize=0.0/0 for region contacts,5c,1405328195955.804e9746463e04db5184c898e8a2324e. in 61ms, sequenceid=1643051384, compaction requested=false
------------------ 原始邮件 ------------------
发件人: "Ted Yu";<yu...@gmail.com>;
发送时间: 2014年7月23日(星期三) 晚上6:17
收件人: "user@hbase.apache.org"<us...@hbase.apache.org>;
抄送: "user"<us...@hbase.apache.org>;
主题: Re: Too many delete delay
If flush interval is set to 1 hr, that wouldn't explain why you frequently encountered slow performance for deletes, right ?
Mind pastebin'ing portion of region server log ?
Cheers
On Jul 23, 2014, at 1:57 AM, "Li Yong" <la...@foxmail.com> wrote:
> Thanks, Ted.
> I see thousands upon thousands 'periodicFlusher requesting flush for region' logs. I guess the conf 'hbase.regionserver.optionalcacheflushinterval' effects this! We use the default 1h.
>
>
> Is this the reason delete delay too much? And how can I repair this without shutdown cluster?
>
>
> ------------------ Original ------------------
> From: "Ted Yu";<yu...@gmail.com>;
> Date: Wed, Jul 23, 2014 05:41 PM
> To: "user@hbase.apache.org"<us...@hbase.apache.org>;
> Cc: "user"<us...@hbase.apache.org>;
> Subject: Re: Too many delete delay
>
>
>
> Have you checked region server logs to see if there was some clue ?
>
> If so, mind pastebin'ing some snippet ?
>
> Cheers
>
> On Jul 23, 2014, at 12:22 AM, "Li Yong" <la...@foxmail.com> wrote:
>
>> Hi all,
>> I move my cluster to another cluster with CopyTable, then I found too many timeout, Most of them are Delete! I found the table has too many storefiels, so i do a major_compact, then things be better. After an hour, timeout again! I don't know why so many delay in delete, not put, not scan. Can anyone tell me, when will delete operation use too much time? Thanks!
>> [ HBase version : 0.94.9 Hadoop version : 1.0.4 ]
Re: Too many delete delay
Posted by Ted Yu <yu...@gmail.com>.
If flush interval is set to 1 hr, that wouldn't explain why you frequently encountered slow performance for deletes, right ?
Mind pastebin'ing portion of region server log ?
Cheers
On Jul 23, 2014, at 1:57 AM, "Li Yong" <la...@foxmail.com> wrote:
> Thanks, Ted.
> I see thousands upon thousands 'periodicFlusher requesting flush for region' logs. I guess the conf 'hbase.regionserver.optionalcacheflushinterval' effects this! We use the default 1h.
>
>
> Is this the reason delete delay too much? And how can I repair this without shutdown cluster?
>
>
> ------------------ Original ------------------
> From: "Ted Yu";<yu...@gmail.com>;
> Date: Wed, Jul 23, 2014 05:41 PM
> To: "user@hbase.apache.org"<us...@hbase.apache.org>;
> Cc: "user"<us...@hbase.apache.org>;
> Subject: Re: Too many delete delay
>
>
>
> Have you checked region server logs to see if there was some clue ?
>
> If so, mind pastebin'ing some snippet ?
>
> Cheers
>
> On Jul 23, 2014, at 12:22 AM, "Li Yong" <la...@foxmail.com> wrote:
>
>> Hi all,
>> I move my cluster to another cluster with CopyTable, then I found too many timeout, Most of them are Delete! I found the table has too many storefiels, so i do a major_compact, then things be better. After an hour, timeout again! I don't know why so many delay in delete, not put, not scan. Can anyone tell me, when will delete operation use too much time? Thanks!
>> [ HBase version : 0.94.9 Hadoop version : 1.0.4 ]
Re: Too many delete delay
Posted by Li Yong <la...@foxmail.com>.
Thanks, Ted.
I see thousands upon thousands 'periodicFlusher requesting flush for region' logs. I guess the conf 'hbase.regionserver.optionalcacheflushinterval' effects this! We use the default 1h.
Is this the reason delete delay too much? And how can I repair this without shutdown cluster?
------------------ Original ------------------
From: "Ted Yu";<yu...@gmail.com>;
Date: Wed, Jul 23, 2014 05:41 PM
To: "user@hbase.apache.org"<us...@hbase.apache.org>;
Cc: "user"<us...@hbase.apache.org>;
Subject: Re: Too many delete delay
Have you checked region server logs to see if there was some clue ?
If so, mind pastebin'ing some snippet ?
Cheers
On Jul 23, 2014, at 12:22 AM, "Li Yong" <la...@foxmail.com> wrote:
> Hi all,
> I move my cluster to another cluster with CopyTable, then I found too many timeout, Most of them are Delete! I found the table has too many storefiels, so i do a major_compact, then things be better. After an hour, timeout again! I don't know why so many delay in delete, not put, not scan. Can anyone tell me, when will delete operation use too much time? Thanks!
> [ HBase version : 0.94.9 Hadoop version : 1.0.4 ]
Re: Too many delete delay
Posted by Ted Yu <yu...@gmail.com>.
Have you checked region server logs to see if there was some clue ?
If so, mind pastebin'ing some snippet ?
Cheers
On Jul 23, 2014, at 12:22 AM, "Li Yong" <la...@foxmail.com> wrote:
> Hi all,
> I move my cluster to another cluster with CopyTable, then I found too many timeout, Most of them are Delete! I found the table has too many storefiels, so i do a major_compact, then things be better. After an hour, timeout again! I don't know why so many delay in delete, not put, not scan. Can anyone tell me, when will delete operation use too much time? Thanks!
> [ HBase version : 0.94.9 Hadoop version : 1.0.4 ]