You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@kudu.apache.org by 一米阳光 <71...@qq.com> on 2018/05/13 07:56:04 UTC
scan performance super bad
hi, i have faced a difficult problem when using kudu 1.6.
my kudu table schema is generally like this:
column name:key, type:string, prefix encoding, lz4 compression, primary key
column name:value, type:string, lz4 compression
the primary key is built from several parts:
001320_201803220420_00000001
the first part is a unique id,
the second part is time format string,
the third part is incremental integer(for a unique id and an fixed time, there may exist multi value, so i used this part to distinguish)
the table range partition use the first part, split it like below
range<005000
005000<= range <010000
010000<= range <015000
015000<= range <020000
.....
.....
995000<= range
when i want to scan data for a unique id and range of time, the lower bound like 001320_201803220420_00000001 and the higher bound like 001320_201803230420_99999999, it takes about 500ms to call kuduScanner.nextRows() and the number of rows it returns is between 20~50. All size of data between the bound is about 8000, so i should call hundreds times nextRows() to fetch all data, and it finally cost several minutes.
i don't know why this happened and how to resolve it....maybe the final solution is that i should giving up kudu, using hbase instead...
Re: scan performance super bad
Posted by Todd Lipcon <to...@cloudera.com>.
On Sun, May 13, 2018 at 9:35 PM, 一米阳光 <71...@qq.com> wrote:
> hi, super thanks for reply.
> the table schema and partitioning info is:
> Schema
> ColumnIDTypeEncodingCompressionRead defaultWrite default
> *key* 0 string NOT NULL PREFIX_ENCODING LZ4 - -
> metric_value 1 string NOT NULL AUTO_ENCODING LZ4 - -
> dimension_00 2 string NOT NULL AUTO_ENCODING LZ4 - -
> dimension_01 3 string NOT NULL AUTO_ENCODING LZ4 - -
> dimension_02 4 string NOT NULL AUTO_ENCODING LZ4 - -
> dimension_03 5 string NOT NULL AUTO_ENCODING LZ4 - -
> dimension_04 6 string NOT NULL AUTO_ENCODING LZ4 - -Partition Schema
>
> RANGE (key) (
> PARTITION VALUES < "005000",
> PARTITION "005000" <= VALUES < "010000",
> PARTITION "010000" <= VALUES < "015000",
> PARTITION "015000" <= VALUES < "020000",
> PARTITION "020000" <= VALUES < "025000",
> PARTITION "025000" <= VALUES < "030000",
> PARTITION "030000" <= VALUES < "035000",
> PARTITION "035000" <= VALUES < "040000",
> PARTITION "040000" <= VALUES < "045000",
> PARTITION "045000" <= VALUES < "050000",
> PARTITION "050000" <= VALUES < "055000",
> PARTITION "055000" <= VALUES < "060000",
> PARTITION "060000" <= VALUES < "065000",
> PARTITION "065000" <= VALUES < "070000",
> PARTITION "070000" <= VALUES < "075000",
> PARTITION "075000" <= VALUES < "080000",
> PARTITION "080000" <= VALUES < "085000",
> PARTITION "085000" <= VALUES < "090000",
> PARTITION "090000" <= VALUES < "095000",
> PARTITION "095000" <= VALUES < "100000",
> PARTITION "100000" <= VALUES < "1000000",
> PARTITION "1000000" <= VALUES < "105000",
> PARTITION "105000" <= VALUES < "110000",
> PARTITION "110000" <= VALUES < "115000",
> PARTITION "115000" <= VALUES < "120000",
> PARTITION "120000" <= VALUES < "125000",
> PARTITION "125000" <= VALUES < "130000",
> PARTITION "130000" <= VALUES < "135000",
> PARTITION "135000" <= VALUES < "140000",
> PARTITION "140000" <= VALUES < "145000",
> PARTITION "145000" <= VALUES < "150000",
> PARTITION "150000" <= VALUES < "155000",
> PARTITION "155000" <= VALUES < "160000",
> PARTITION "160000" <= VALUES < "165000",
> PARTITION "165000" <= VALUES < "170000",
> PARTITION "170000" <= VALUES < "175000",
> PARTITION "175000" <= VALUES < "180000",
> PARTITION "180000" <= VALUES < "185000",
> PARTITION "185000" <= VALUES < "190000",
> PARTITION "190000" <= VALUES < "195000",
> PARTITION "195000" <= VALUES < "200000",
> PARTITION "200000" <= VALUES < "205000",
> PARTITION "205000" <= VALUES < "210000",
> PARTITION "210000" <= VALUES < "215000",
> PARTITION "215000" <= VALUES < "220000",
> PARTITION "220000" <= VALUES < "225000",
> PARTITION "225000" <= VALUES < "230000",
> PARTITION "230000" <= VALUES < "235000",
> PARTITION "235000" <= VALUES < "240000",
> PARTITION "240000" <= VALUES < "245000",
> PARTITION "245000" <= VALUES < "250000",
> PARTITION "250000" <= VALUES < "255000",
> PARTITION "255000" <= VALUES < "260000",
> PARTITION "260000" <= VALUES < "265000",
> PARTITION "265000" <= VALUES < "270000",
> PARTITION "270000" <= VALUES < "275000",
> PARTITION "275000" <= VALUES < "280000",
> PARTITION "280000" <= VALUES < "285000",
> PARTITION "285000" <= VALUES < "290000",
> PARTITION "290000" <= VALUES < "295000",
> PARTITION "295000" <= VALUES < "300000",
> PARTITION "300000" <= VALUES < "305000",
> PARTITION "305000" <= VALUES < "310000",
> PARTITION "310000" <= VALUES < "315000",
> PARTITION "315000" <= VALUES < "320000",
> PARTITION "320000" <= VALUES < "325000",
> PARTITION "325000" <= VALUES < "330000",
> PARTITION "330000" <= VALUES < "335000",
> PARTITION "335000" <= VALUES < "340000",
> PARTITION "340000" <= VALUES < "345000",
> PARTITION "345000" <= VALUES < "350000",
> PARTITION "350000" <= VALUES < "355000",
> PARTITION "355000" <= VALUES < "360000",
> PARTITION "360000" <= VALUES < "365000",
> PARTITION "365000" <= VALUES < "370000",
> PARTITION "370000" <= VALUES < "375000",
> PARTITION "375000" <= VALUES < "380000",
> PARTITION "380000" <= VALUES < "385000",
> PARTITION "385000" <= VALUES < "390000",
> PARTITION "390000" <= VALUES < "395000",
> PARTITION "395000" <= VALUES < "400000",
> PARTITION "400000" <= VALUES < "405000",
> PARTITION "405000" <= VALUES < "410000",
> PARTITION "410000" <= VALUES < "415000",
> PARTITION "415000" <= VALUES < "420000",
> PARTITION "420000" <= VALUES < "425000",
> PARTITION "425000" <= VALUES < "430000",
> PARTITION "430000" <= VALUES < "435000",
> PARTITION "435000" <= VALUES < "440000",
> PARTITION "440000" <= VALUES < "445000",
> PARTITION "445000" <= VALUES < "450000",
> PARTITION "450000" <= VALUES < "455000",
> PARTITION "455000" <= VALUES < "460000",
> PARTITION "460000" <= VALUES < "465000",
> PARTITION "465000" <= VALUES < "470000",
> PARTITION "470000" <= VALUES < "475000",
> PARTITION "475000" <= VALUES < "480000",
> PARTITION "480000" <= VALUES < "485000",
> PARTITION "485000" <= VALUES < "490000",
> PARTITION "490000" <= VALUES < "495000",
> PARTITION "495000" <= VALUES < "500000",
> PARTITION "500000" <= VALUES < "505000",
> PARTITION "505000" <= VALUES < "510000",
> PARTITION "510000" <= VALUES < "515000",
> PARTITION "515000" <= VALUES < "520000",
> PARTITION "520000" <= VALUES < "525000",
> PARTITION "525000" <= VALUES < "530000",
> PARTITION "530000" <= VALUES < "535000",
> PARTITION "535000" <= VALUES < "540000",
> PARTITION "540000" <= VALUES < "545000",
> PARTITION "545000" <= VALUES < "550000",
> PARTITION "550000" <= VALUES < "555000",
> PARTITION "555000" <= VALUES < "560000",
> PARTITION "560000" <= VALUES < "565000",
> PARTITION "565000" <= VALUES < "570000",
> PARTITION "570000" <= VALUES < "575000",
> PARTITION "575000" <= VALUES < "580000",
> PARTITION "580000" <= VALUES < "585000",
> PARTITION "585000" <= VALUES < "590000",
> PARTITION "590000" <= VALUES < "595000",
> PARTITION "595000" <= VALUES < "600000",
> PARTITION "600000" <= VALUES < "605000",
> PARTITION "605000" <= VALUES < "610000",
> PARTITION "610000" <= VALUES < "615000",
> PARTITION "615000" <= VALUES < "620000",
> PARTITION "620000" <= VALUES < "625000",
> PARTITION "625000" <= VALUES < "630000",
> PARTITION "630000" <= VALUES < "635000",
> PARTITION "635000" <= VALUES < "640000",
> PARTITION "640000" <= VALUES < "645000",
> PARTITION "645000" <= VALUES < "650000",
> PARTITION "650000" <= VALUES < "655000",
> PARTITION "655000" <= VALUES < "660000",
> PARTITION "660000" <= VALUES < "665000",
> PARTITION "665000" <= VALUES < "670000",
> PARTITION "670000" <= VALUES < "675000",
> PARTITION "675000" <= VALUES < "680000",
> PARTITION "680000" <= VALUES < "685000",
> PARTITION "685000" <= VALUES < "690000",
> PARTITION "690000" <= VALUES < "695000",
> PARTITION "695000" <= VALUES < "700000",
> PARTITION "700000" <= VALUES < "705000",
> PARTITION "705000" <= VALUES < "710000",
> PARTITION "710000" <= VALUES < "715000",
> PARTITION "715000" <= VALUES < "720000",
> PARTITION "720000" <= VALUES < "725000",
> PARTITION "725000" <= VALUES < "730000",
> PARTITION "730000" <= VALUES < "735000",
> PARTITION "735000" <= VALUES < "740000",
> PARTITION "740000" <= VALUES < "745000",
> PARTITION "745000" <= VALUES < "750000",
> PARTITION "750000" <= VALUES < "755000",
> PARTITION "755000" <= VALUES < "760000",
> PARTITION "760000" <= VALUES < "765000",
> PARTITION "765000" <= VALUES < "770000",
> PARTITION "770000" <= VALUES < "775000",
> PARTITION "775000" <= VALUES < "780000",
> PARTITION "780000" <= VALUES < "785000",
> PARTITION "785000" <= VALUES < "790000",
> PARTITION "790000" <= VALUES < "795000",
> PARTITION "795000" <= VALUES < "800000",
> PARTITION "800000" <= VALUES < "805000",
> PARTITION "805000" <= VALUES < "810000",
> PARTITION "810000" <= VALUES < "815000",
> PARTITION "815000" <= VALUES < "820000",
> PARTITION "820000" <= VALUES < "825000",
> PARTITION "825000" <= VALUES < "830000",
> PARTITION "830000" <= VALUES < "835000",
> PARTITION "835000" <= VALUES < "840000",
> PARTITION "840000" <= VALUES < "845000",
> PARTITION "845000" <= VALUES < "850000",
> PARTITION "850000" <= VALUES < "855000",
> PARTITION "855000" <= VALUES < "860000",
> PARTITION "860000" <= VALUES < "865000",
> PARTITION "865000" <= VALUES < "870000",
> PARTITION "870000" <= VALUES < "875000",
> PARTITION "875000" <= VALUES < "880000",
> PARTITION "880000" <= VALUES < "885000",
> PARTITION "885000" <= VALUES < "890000",
> PARTITION "890000" <= VALUES < "895000",
> PARTITION "895000" <= VALUES < "900000",
> PARTITION "900000" <= VALUES < "905000",
> PARTITION "905000" <= VALUES < "910000",
> PARTITION "910000" <= VALUES < "915000",
> PARTITION "915000" <= VALUES < "920000",
> PARTITION "920000" <= VALUES < "925000",
> PARTITION "925000" <= VALUES < "930000",
> PARTITION "930000" <= VALUES < "935000",
> PARTITION "935000" <= VALUES < "940000",
> PARTITION "940000" <= VALUES < "945000",
> PARTITION "945000" <= VALUES < "950000",
> PARTITION "950000" <= VALUES < "955000",
> PARTITION "955000" <= VALUES < "960000",
> PARTITION "960000" <= VALUES < "965000",
> PARTITION "965000" <= VALUES < "970000",
> PARTITION "970000" <= VALUES < "975000",
> PARTITION "975000" <= VALUES < "980000",
> PARTITION "980000" <= VALUES < "985000",
> PARTITION "985000" <= VALUES < "990000",
> PARTITION "990000" <= VALUES < "995000",
> PARTITION VALUES >= "995000"
> )
>
>
>
So it looks like you have a numeric value being stored here in the string
column. Are you sure that you are properly zero-padding when creating your
key? For example if you accidentally scan from "50_..." to "80_..." you
will end up scanning a huge portion of your table.
> i did not delete rows in this table ever.
>
> my scanner code is below:
> buildKey method will build the lower bound and the upper bound, the unique
> id is same, the startRow offset(third part) is 0, and the endRow offset is
> 99999999, startRow and endRow only differs from time.
> though the max offset is big(9999999), generally it is less than 100.
>
> private KuduScanner buildScanner(Metric startRow, Metric endRow, List<Integer> dimensionIds, List<DimensionFilter> dimensionFilterList) {
> KuduTable kuduTable = kuduService.getKuduTable(BizConfig.parseFrom(startRow.getBizId()));
>
> PartialRow lower = kuduTable.getSchema().newPartialRow();
> lower.addString("key", buildKey(startRow));
> PartialRow upper = kuduTable.getSchema().newPartialRow();
> upper.addString("key", buildKey(endRow));
>
> LOG.info("build scanner. lower = {}, upper = {}", buildKey(startRow), buildKey(endRow));
>
> KuduScanner.KuduScannerBuilder builder = kuduService.getKuduClient().newScannerBuilder(kuduTable);
> builder.setProjectedColumnNames(COLUMNS);
> builder.lowerBound(lower);
> builder.exclusiveUpperBound(upper);
> builder.prefetching(true);
> builder.batchSizeBytes(MAX_BATCH_SIZE);
>
> if (CollectionUtils.isNotEmpty(dimensionFilterList)) {
> for (int i = 0; i < dimensionIds.size() && i < MAX_DIMENSION_NUM; i++) {
> for (DimensionFilter dimensionFilter : dimensionFilterList) {
> if (!Objects.equals(dimensionFilter.getDimensionId(), dimensionIds.get(i))) {
> continue;
> }
> ColumnSchema columnSchema = kuduTable.getSchema().getColumn(String.format("dimension_%02d", i));
> KuduPredicate predicate = buildKuduPredicate(columnSchema, dimensionFilter);
> if (predicate != null) {
> builder.addPredicate(predicate);
> LOG.info("add predicate. predicate = {}", predicate.toString());
> }
> }
> }
> }
> return builder.build();
> }
>
>
What client version are you using? 1.7.0?
> i checked the metrics, only get content below, it seems no relationship
> with my table.
>
Looks like you got the metrics from the kudu master, not a tablet server.
You need to figure out which tablet server you are scanning and grab the
metrics from that one.
-Todd
--
Todd Lipcon
Software Engineer, Cloudera
re：scan performance super bad
Posted by 一米阳光 <71...@qq.com>.
>So it looks like you have a numeric value being stored here in the string >column. Are you sure that you are properly zero-padding when creating your >key? For example if you accidentally scan from "50_..." to "80_..." you >will end up scanning a huge portion of your table.i'm sure they are properly zero-padding.
>What client version are you using? 1.7.0?i used 1.6.0
>Looks like you got the metrics from the kudu master, not a tablet server. >You need to figure out which tablet server you are scanning and grab the >metrics from that one.oh i have checked the metrics and find that rows_scanned is not much more than rows_returned.let me describe the situation. data is like below, i add a column filter on dimension00 with value 'iphone',and the rows_scanned is about 5 times more than the rows_returned, so i think it is reasonable.key dimension00000001_time_0 iphone000001_time_1 android000001_time_2 ipad000001_time_3 windows_phone000001_time_4 unkown
in addition, the leader of the tablet which store the target partition was killed by mistake, and then a new leader comes up, i found that scan performance was much better. all server config is same, the slow leader's workload is lower than the new one, but it's performance is bad...do you have some config suggestions?
re：scan performance super bad
Posted by 一米阳光 <71...@qq.com>.
hi, super thanks for reply.the table schema and partitioning info is:
Schema
ColumnIDTypeEncodingCompressionRead defaultWrite default
key0string NOT NULLPREFIX_ENCODINGLZ4--
metric_value1string NOT NULLAUTO_ENCODINGLZ4--
dimension_002string NOT NULLAUTO_ENCODINGLZ4--
dimension_013string NOT NULLAUTO_ENCODINGLZ4--
dimension_024string NOT NULLAUTO_ENCODINGLZ4--
dimension_035string NOT NULLAUTO_ENCODINGLZ4--
dimension_046string NOT NULLAUTO_ENCODINGLZ4--
Partition Schema
RANGE (key) ( PARTITION VALUES < "005000", PARTITION "005000" <= VALUES < "010000", PARTITION "010000" <= VALUES < "015000", PARTITION "015000" <= VALUES < "020000", PARTITION "020000" <= VALUES < "025000", PARTITION "025000" <= VALUES < "030000", PARTITION "030000" <= VALUES < "035000", PARTITION "035000" <= VALUES < "040000", PARTITION "040000" <= VALUES < "045000", PARTITION "045000" <= VALUES < "050000", PARTITION "050000" <= VALUES < "055000", PARTITION "055000" <= VALUES < "060000", PARTITION "060000" <= VALUES < "065000", PARTITION "065000" <= VALUES < "070000", PARTITION "070000" <= VALUES < "075000", PARTITION "075000" <= VALUES < "080000", PARTITION "080000" <= VALUES < "085000", PARTITION "085000" <= VALUES < "090000", PARTITION "090000" <= VALUES < "095000", PARTITION "095000" <= VALUES < "100000", PARTITION "100000" <= VALUES < "1000000", PARTITION "1000000" <= VALUES < "105000", PARTITION "105000" <= VALUES < "110000", PARTITION "110000" <= VALUES < "115000", PARTITION "115000" <= VALUES < "120000", PARTITION "120000" <= VALUES < "125000", PARTITION "125000" <= VALUES < "130000", PARTITION "130000" <= VALUES < "135000", PARTITION "135000" <= VALUES < "140000", PARTITION "140000" <= VALUES < "145000", PARTITION "145000" <= VALUES < "150000", PARTITION "150000" <= VALUES < "155000", PARTITION "155000" <= VALUES < "160000", PARTITION "160000" <= VALUES < "165000", PARTITION "165000" <= VALUES < "170000", PARTITION "170000" <= VALUES < "175000", PARTITION "175000" <= VALUES < "180000", PARTITION "180000" <= VALUES < "185000", PARTITION "185000" <= VALUES < "190000", PARTITION "190000" <= VALUES < "195000", PARTITION "195000" <= VALUES < "200000", PARTITION "200000" <= VALUES < "205000", PARTITION "205000" <= VALUES < "210000", PARTITION "210000" <= VALUES < "215000", PARTITION "215000" <= VALUES < "220000", PARTITION "220000" <= VALUES < "225000", PARTITION "225000" <= VALUES < "230000", PARTITION "230000" <= VALUES < "235000", PARTITION "235000" <= VALUES < "240000", PARTITION "240000" <= VALUES < "245000", PARTITION "245000" <= VALUES < "250000", PARTITION "250000" <= VALUES < "255000", PARTITION "255000" <= VALUES < "260000", PARTITION "260000" <= VALUES < "265000", PARTITION "265000" <= VALUES < "270000", PARTITION "270000" <= VALUES < "275000", PARTITION "275000" <= VALUES < "280000", PARTITION "280000" <= VALUES < "285000", PARTITION "285000" <= VALUES < "290000", PARTITION "290000" <= VALUES < "295000", PARTITION "295000" <= VALUES < "300000", PARTITION "300000" <= VALUES < "305000", PARTITION "305000" <= VALUES < "310000", PARTITION "310000" <= VALUES < "315000", PARTITION "315000" <= VALUES < "320000", PARTITION "320000" <= VALUES < "325000", PARTITION "325000" <= VALUES < "330000", PARTITION "330000" <= VALUES < "335000", PARTITION "335000" <= VALUES < "340000", PARTITION "340000" <= VALUES < "345000", PARTITION "345000" <= VALUES < "350000", PARTITION "350000" <= VALUES < "355000", PARTITION "355000" <= VALUES < "360000", PARTITION "360000" <= VALUES < "365000", PARTITION "365000" <= VALUES < "370000", PARTITION "370000" <= VALUES < "375000", PARTITION "375000" <= VALUES < "380000", PARTITION "380000" <= VALUES < "385000", PARTITION "385000" <= VALUES < "390000", PARTITION "390000" <= VALUES < "395000", PARTITION "395000" <= VALUES < "400000", PARTITION "400000" <= VALUES < "405000", PARTITION "405000" <= VALUES < "410000", PARTITION "410000" <= VALUES < "415000", PARTITION "415000" <= VALUES < "420000", PARTITION "420000" <= VALUES < "425000", PARTITION "425000" <= VALUES < "430000", PARTITION "430000" <= VALUES < "435000", PARTITION "435000" <= VALUES < "440000", PARTITION "440000" <= VALUES < "445000", PARTITION "445000" <= VALUES < "450000", PARTITION "450000" <= VALUES < "455000", PARTITION "455000" <= VALUES < "460000", PARTITION "460000" <= VALUES < "465000", PARTITION "465000" <= VALUES < "470000", PARTITION "470000" <= VALUES < "475000", PARTITION "475000" <= VALUES < "480000", PARTITION "480000" <= VALUES < "485000", PARTITION "485000" <= VALUES < "490000", PARTITION "490000" <= VALUES < "495000", PARTITION "495000" <= VALUES < "500000", PARTITION "500000" <= VALUES < "505000", PARTITION "505000" <= VALUES < "510000", PARTITION "510000" <= VALUES < "515000", PARTITION "515000" <= VALUES < "520000", PARTITION "520000" <= VALUES < "525000", PARTITION "525000" <= VALUES < "530000", PARTITION "530000" <= VALUES < "535000", PARTITION "535000" <= VALUES < "540000", PARTITION "540000" <= VALUES < "545000", PARTITION "545000" <= VALUES < "550000", PARTITION "550000" <= VALUES < "555000", PARTITION "555000" <= VALUES < "560000", PARTITION "560000" <= VALUES < "565000", PARTITION "565000" <= VALUES < "570000", PARTITION "570000" <= VALUES < "575000", PARTITION "575000" <= VALUES < "580000", PARTITION "580000" <= VALUES < "585000", PARTITION "585000" <= VALUES < "590000", PARTITION "590000" <= VALUES < "595000", PARTITION "595000" <= VALUES < "600000", PARTITION "600000" <= VALUES < "605000", PARTITION "605000" <= VALUES < "610000", PARTITION "610000" <= VALUES < "615000", PARTITION "615000" <= VALUES < "620000", PARTITION "620000" <= VALUES < "625000", PARTITION "625000" <= VALUES < "630000", PARTITION "630000" <= VALUES < "635000", PARTITION "635000" <= VALUES < "640000", PARTITION "640000" <= VALUES < "645000", PARTITION "645000" <= VALUES < "650000", PARTITION "650000" <= VALUES < "655000", PARTITION "655000" <= VALUES < "660000", PARTITION "660000" <= VALUES < "665000", PARTITION "665000" <= VALUES < "670000", PARTITION "670000" <= VALUES < "675000", PARTITION "675000" <= VALUES < "680000", PARTITION "680000" <= VALUES < "685000", PARTITION "685000" <= VALUES < "690000", PARTITION "690000" <= VALUES < "695000", PARTITION "695000" <= VALUES < "700000", PARTITION "700000" <= VALUES < "705000", PARTITION "705000" <= VALUES < "710000", PARTITION "710000" <= VALUES < "715000", PARTITION "715000" <= VALUES < "720000", PARTITION "720000" <= VALUES < "725000", PARTITION "725000" <= VALUES < "730000", PARTITION "730000" <= VALUES < "735000", PARTITION "735000" <= VALUES < "740000", PARTITION "740000" <= VALUES < "745000", PARTITION "745000" <= VALUES < "750000", PARTITION "750000" <= VALUES < "755000", PARTITION "755000" <= VALUES < "760000", PARTITION "760000" <= VALUES < "765000", PARTITION "765000" <= VALUES < "770000", PARTITION "770000" <= VALUES < "775000", PARTITION "775000" <= VALUES < "780000", PARTITION "780000" <= VALUES < "785000", PARTITION "785000" <= VALUES < "790000", PARTITION "790000" <= VALUES < "795000", PARTITION "795000" <= VALUES < "800000", PARTITION "800000" <= VALUES < "805000", PARTITION "805000" <= VALUES < "810000", PARTITION "810000" <= VALUES < "815000", PARTITION "815000" <= VALUES < "820000", PARTITION "820000" <= VALUES < "825000", PARTITION "825000" <= VALUES < "830000", PARTITION "830000" <= VALUES < "835000", PARTITION "835000" <= VALUES < "840000", PARTITION "840000" <= VALUES < "845000", PARTITION "845000" <= VALUES < "850000", PARTITION "850000" <= VALUES < "855000", PARTITION "855000" <= VALUES < "860000", PARTITION "860000" <= VALUES < "865000", PARTITION "865000" <= VALUES < "870000", PARTITION "870000" <= VALUES < "875000", PARTITION "875000" <= VALUES < "880000", PARTITION "880000" <= VALUES < "885000", PARTITION "885000" <= VALUES < "890000", PARTITION "890000" <= VALUES < "895000", PARTITION "895000" <= VALUES < "900000", PARTITION "900000" <= VALUES < "905000", PARTITION "905000" <= VALUES < "910000", PARTITION "910000" <= VALUES < "915000", PARTITION "915000" <= VALUES < "920000", PARTITION "920000" <= VALUES < "925000", PARTITION "925000" <= VALUES < "930000", PARTITION "930000" <= VALUES < "935000", PARTITION "935000" <= VALUES < "940000", PARTITION "940000" <= VALUES < "945000", PARTITION "945000" <= VALUES < "950000", PARTITION "950000" <= VALUES < "955000", PARTITION "955000" <= VALUES < "960000", PARTITION "960000" <= VALUES < "965000", PARTITION "965000" <= VALUES < "970000", PARTITION "970000" <= VALUES < "975000", PARTITION "975000" <= VALUES < "980000", PARTITION "980000" <= VALUES < "985000", PARTITION "985000" <= VALUES < "990000", PARTITION "990000" <= VALUES < "995000", PARTITION VALUES >= "995000" )
i did not delete rows in this table ever.
my scanner code is below:
buildKey method will build the lower bound and the upper bound, the unique id is same, the startRow offset(third part) is 0, and the endRow offset is 99999999, startRow and endRow only differs from time.
though the max offset is big(9999999), generally it is less than 100.
private KuduScanner buildScanner(Metric startRow, Metric endRow, List<Integer> dimensionIds, List<DimensionFilter> dimensionFilterList) {
KuduTable kuduTable = kuduService.getKuduTable(BizConfig.parseFrom(startRow.getBizId()));
PartialRow lower = kuduTable.getSchema().newPartialRow();
lower.addString("key", buildKey(startRow));
PartialRow upper = kuduTable.getSchema().newPartialRow();
upper.addString("key", buildKey(endRow));
LOG.info("build scanner. lower = {}, upper = {}", buildKey(startRow), buildKey(endRow));
KuduScanner.KuduScannerBuilder builder = kuduService.getKuduClient().newScannerBuilder(kuduTable);
builder.setProjectedColumnNames(COLUMNS);
builder.lowerBound(lower);
builder.exclusiveUpperBound(upper);
builder.prefetching(true);
builder.batchSizeBytes(MAX_BATCH_SIZE);
if (CollectionUtils.isNotEmpty(dimensionFilterList)) {
for (int i = 0; i < dimensionIds.size() && i < MAX_DIMENSION_NUM; i++) {
for (DimensionFilter dimensionFilter : dimensionFilterList) {
if (!Objects.equals(dimensionFilter.getDimensionId(), dimensionIds.get(i))) {
continue;
}
ColumnSchema columnSchema = kuduTable.getSchema().getColumn(String.format("dimension_%02d", i));
KuduPredicate predicate = buildKuduPredicate(columnSchema, dimensionFilter);
if (predicate != null) {
builder.addPredicate(predicate);
LOG.info("add predicate. predicate = {}", predicate.toString());
}
}
}
}
return builder.build();
}
i checked the metrics, only get content below, it seems no relationship with my table.
[ { "type": "tablet", "id": "00000000000000000000000000000000", "attributes": { "partition": "RANGE (entry_type, entry_id) PARTITION UNBOUNDED", "table_name": "sys.catalog", "table_id": "sys.catalog.id" }, "metrics": [ { "name": "scanner_rows_returned", "value": 0 }, { "name": "scanner_rows_scanned", "value": 0 } ] } ]
Re: scan performance super bad
Posted by Todd Lipcon <to...@cloudera.com>.
Can you share the code you are using to create the scanner and call
nextRows?
Can you also copy-paste the info provided on the web UI of the kudu master
for this table? It will show the schema and partitioning information.
Is it possible that your table includes a lot of deleted rows? i.e did you
load the table, then delete all the rows, then load again? This can cause
some performance issues in current versions of Kudu as the scanner needs to
"skip over" the deleted rows before it finds any to return.
Based on your description I would expect this to be doing a simple range
scan for the returned rows, and return in just a few milliseconds. The fact
that it is taking 500ms implies that the server is scanning a lot of
non-matching rows before finding a few that match. You can also check the
metrics:
http://my-tablet-server:8050/metrics?metrics=scanner_rows
and compare the 'rows scanned' vs 'rows returned' metric. Capture the
values both before and after you run the query, and you should see if
'rows_scanned' is much larger than 'rows_returned'.
-Todd
On Sun, May 13, 2018 at 12:56 AM, 一米阳光 <71...@qq.com> wrote:
> hi, i have faced a difficult problem when using kudu 1.6.
>
> my kudu table schema is generally like this:
> column name:key, type:string, prefix encoding, lz4 compression, primary key
> column name:value, type:string, lz4 compression
>
> the primary key is built from several parts:
> 001320_201803220420_00000001
> the first part is a unique id,
> the second part is time format string,
> the third part is incremental integer(for a unique id and an fixed time,
> there may exist multi value, so i used this part to distinguish
> <http://dict.youdao.com/w/distinguish/#keyfrom=E2Ctranslation>)
>
> the table range partition use the first part, split it like below
> range<005000
> 005000<= range <010000
> 010000<= range <015000
> 015000<= range <020000
> .....
> .....
> 995000<= range
>
> when i want to scan data for a unique id and range of time, the lower
> bound like 001320_201803220420_00000001 and the higher bound like
> 001320_201803230420_99999999, it takes about 500ms to call
> kuduScanner.nextRows() and the number of rows it returns is between 20~50.
> All size of data between the bound is about 8000, so i should call hundreds
> times nextRows() to fetch all data, and it finally cost several minutes.
>
> i don't know why this happened and how to resolve it....maybe the final
> solution is that i should giving up kudu, using hbase instead...
>
--
Todd Lipcon
Software Engineer, Cloudera