You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "ZHA_Moonlight (JIRA)" <ji...@apache.org> on 2017/09/26 06:03:00 UTC
[jira] [Updated] (HBASE-18879) Hbase FilterList cause KeyOnlyFilter
not work
[ https://issues.apache.org/jira/browse/HBASE-18879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
ZHA_Moonlight updated HBASE-18879:
----------------------------------
Description:
when use FilterList and KeyOnlyFilter together, if we put KeyOnlyFilter before FilterList, the KeyOnlyFilter may not work, means it will also grab the cell values:
{code:java}
List<Filter> filters = new ArrayList<Filter>();
Filter filter1 = new SingleColumnValueFilter(Bytes.toBytes("cf"), Bytes.toBytes("column1"),
CompareOp.EQUAL, Bytes.toBytes("value1"));
Filter filter2 = new SingleColumnValueFilter(Bytes.toBytes("cf"), Bytes.toBytes("column1"),
CompareOp.EQUAL, Bytes.toBytes("value2"));
filters.add(filter1);
filters.add(filter2);
FilterList filterListAll = new FilterList(Operator.MUST_PASS_ALL,
new KeyOnlyFilter(),
new FilterList(Operator.MUST_PASS_ONE, filters));
{code}
use the above code as filter to scan a table, it will return the cells with value instead of only return the key, if we put KeyOnlyFilter after FilterList as following, it works well.
{code:java}
FilterList filterListAll = new FilterList(Operator.MUST_PASS_ALL,
new FilterList(Operator.MUST_PASS_ONE, filters),
new KeyOnlyFilter());
{code}
the cause should due to the following code at hbase-client FilterList.java
{code:java}
@Override
@edu.umd.cs.findbugs.annotations.SuppressWarnings(value="SF_SWITCH_FALLTHROUGH",
justification="Intentional")
public ReturnCode filterKeyValue(Cell v) throws IOException {
this.referenceKV = v;
// Accumulates successive transformation of every filter that includes the Cell:
Cell transformed = v;
ReturnCode rc = operator == Operator.MUST_PASS_ONE?
ReturnCode.SKIP: ReturnCode.INCLUDE;
int listize = filters.size();
for (int i = 0; i < listize; i++) {
Filter filter = filters.get(i);
if (operator == Operator.MUST_PASS_ALL) {
if (filter.filterAllRemaining()) {
return ReturnCode.NEXT_ROW;
}
{color:red} ReturnCode code = filter.filterKeyValue(v);{color}
switch (code) {
// Override INCLUDE and continue to evaluate.
case INCLUDE_AND_NEXT_COL:
rc = ReturnCode.INCLUDE_AND_NEXT_COL; // FindBugs SF_SWITCH_FALLTHROUGH
case INCLUDE:
{color:red} transformed = filter.transformCell(transformed);{color}
continue;
case SEEK_NEXT_USING_HINT:
seekHintFilter = filter;
return code;
default:
return code;
}
}
{code}
notice the “{color:red}” label line, first line is a recursive invocation, it will assign a Cell results to the FilterList.transformedKV(we call it A), the results is from the FilterList with 2 SingleColumnValueFilter, so A with contains the cell value, while the second line with return A to the var transformed.
back to the following loop, we can see the FilterList return results is var "transformed " which will override in each loop, so the value is determined by the last filter, so the order of KeyOnlyFilter will impact the results.
{code:java}
Cell transformed = v;
ReturnCode rc = operator == Operator.MUST_PASS_ONE?
ReturnCode.SKIP: ReturnCode.INCLUDE;
int listize = filters.size();
for (int i = 0; i < listize; i++) {
Filter filter = filters.get(i);
if (operator == Operator.MUST_PASS_ALL) {
if (filter.filterAllRemaining()) {
return ReturnCode.NEXT_ROW;
}
ReturnCode code = filter.filterKeyValue(v);
switch (code) {
// Override INCLUDE and continue to evaluate.
case INCLUDE_AND_NEXT_COL:
rc = ReturnCode.INCLUDE_AND_NEXT_COL; // FindBugs SF_SWITCH_FALLTHROUGH
case INCLUDE:
transformed = filter.transformCell(transformed);
continue;
case SEEK_NEXT_USING_HINT:
seekHintFilter = filter;
return code;
default:
return code;
}
{code}
was:
when use FilterList and KeyOnlyFilter together, if we put KeyOnlyFilter before FilterList, the KeyOnlyFilter may not work, means it will also grab the cell values:
{code:java}
List<Filter> filters = new ArrayList<Filter>();
Filter filter1 = new SingleColumnValueFilter(Bytes.toBytes("cf"), Bytes.toBytes("column1"),
CompareOp.EQUAL, Bytes.toBytes("value1"));
Filter filter2 = new SingleColumnValueFilter(Bytes.toBytes("cf"), Bytes.toBytes("column1"),
CompareOp.EQUAL, Bytes.toBytes("value2"));
filters.add(filter1);
filters.add(filter2);
FilterList filterListAll = new FilterList(Operator.MUST_PASS_ALL,
new KeyOnlyFilter(),
new FilterList(Operator.MUST_PASS_ONE, filters));
{code}
use the above code as filter to scan a table, it will return the cells with value instead of only return the key, if we put KeyOnlyFilter after FilterList as following, it works well.
{code:java}
FilterList filterListAll = new FilterList(Operator.MUST_PASS_ALL,
new FilterList(Operator.MUST_PASS_ONE, filters),
new KeyOnlyFilter());
{code}
the cause should due to the following code at hbase-client FilterList.java
{code:java}
@Override
@edu.umd.cs.findbugs.annotations.SuppressWarnings(value="SF_SWITCH_FALLTHROUGH",
justification="Intentional")
public ReturnCode filterKeyValue(Cell v) throws IOException {
this.referenceKV = v;
// Accumulates successive transformation of every filter that includes the Cell:
Cell transformed = v;
ReturnCode rc = operator == Operator.MUST_PASS_ONE?
ReturnCode.SKIP: ReturnCode.INCLUDE;
int listize = filters.size();
for (int i = 0; i < listize; i++) {
Filter filter = filters.get(i);
if (operator == Operator.MUST_PASS_ALL) {
if (filter.filterAllRemaining()) {
return ReturnCode.NEXT_ROW;
}
{color:red} ReturnCode code = filter.filterKeyValue(v);
{color} switch (code) {
// Override INCLUDE and continue to evaluate.
case INCLUDE_AND_NEXT_COL:
rc = ReturnCode.INCLUDE_AND_NEXT_COL; // FindBugs SF_SWITCH_FALLTHROUGH
case INCLUDE:
{color:red} transformed = filter.transformCell(transformed);{color}
continue;
case SEEK_NEXT_USING_HINT:
seekHintFilter = filter;
return code;
default:
return code;
}
}
{code}
notice the {color:red} label line, first line is a recursive invocation, it will assign a Cell results to the FilterList.transformedKV(we call it A), the results is from the FilterList with 2 SingleColumnValueFilter, so A with contains the cell value, while the second line with return A to the var transformed.
back to the following loop, we can see the FilterList return results is var "transformed " which will override in each loop, so the value is determined by the last filter, so the order of KeyOnlyFilter will impact the results.
{code:java}
Cell transformed = v;
ReturnCode rc = operator == Operator.MUST_PASS_ONE?
ReturnCode.SKIP: ReturnCode.INCLUDE;
int listize = filters.size();
for (int i = 0; i < listize; i++) {
Filter filter = filters.get(i);
if (operator == Operator.MUST_PASS_ALL) {
if (filter.filterAllRemaining()) {
return ReturnCode.NEXT_ROW;
}
ReturnCode code = filter.filterKeyValue(v);
switch (code) {
// Override INCLUDE and continue to evaluate.
case INCLUDE_AND_NEXT_COL:
rc = ReturnCode.INCLUDE_AND_NEXT_COL; // FindBugs SF_SWITCH_FALLTHROUGH
case INCLUDE:
transformed = filter.transformCell(transformed);
continue;
case SEEK_NEXT_USING_HINT:
seekHintFilter = filter;
return code;
default:
return code;
}
{code}
> Hbase FilterList cause KeyOnlyFilter not work
> ---------------------------------------------
>
> Key: HBASE-18879
> URL: https://issues.apache.org/jira/browse/HBASE-18879
> Project: HBase
> Issue Type: Bug
> Components: Filters
> Affects Versions: 1.2.4
> Environment: OS： Red Hat 4.4.7-11
> Hadoop: 2.6.4
> Hbase: 1.2.4
> Reporter: ZHA_Moonlight
>
> when use FilterList and KeyOnlyFilter together, if we put KeyOnlyFilter before FilterList, the KeyOnlyFilter may not work, means it will also grab the cell values:
> {code:java}
> List<Filter> filters = new ArrayList<Filter>();
> Filter filter1 = new SingleColumnValueFilter(Bytes.toBytes("cf"), Bytes.toBytes("column1"),
> CompareOp.EQUAL, Bytes.toBytes("value1"));
> Filter filter2 = new SingleColumnValueFilter(Bytes.toBytes("cf"), Bytes.toBytes("column1"),
> CompareOp.EQUAL, Bytes.toBytes("value2"));
> filters.add(filter1);
> filters.add(filter2);
> FilterList filterListAll = new FilterList(Operator.MUST_PASS_ALL,
> new KeyOnlyFilter(),
> new FilterList(Operator.MUST_PASS_ONE, filters));
> {code}
> use the above code as filter to scan a table, it will return the cells with value instead of only return the key, if we put KeyOnlyFilter after FilterList as following, it works well.
>
> {code:java}
> FilterList filterListAll = new FilterList(Operator.MUST_PASS_ALL,
> new FilterList(Operator.MUST_PASS_ONE, filters),
> new KeyOnlyFilter());
> {code}
> the cause should due to the following code at hbase-client FilterList.java
> {code:java}
> @Override
> @edu.umd.cs.findbugs.annotations.SuppressWarnings(value="SF_SWITCH_FALLTHROUGH",
> justification="Intentional")
> public ReturnCode filterKeyValue(Cell v) throws IOException {
> this.referenceKV = v;
> // Accumulates successive transformation of every filter that includes the Cell:
> Cell transformed = v;
> ReturnCode rc = operator == Operator.MUST_PASS_ONE?
> ReturnCode.SKIP: ReturnCode.INCLUDE;
> int listize = filters.size();
> for (int i = 0; i < listize; i++) {
> Filter filter = filters.get(i);
> if (operator == Operator.MUST_PASS_ALL) {
> if (filter.filterAllRemaining()) {
> return ReturnCode.NEXT_ROW;
> }
> {color:red} ReturnCode code = filter.filterKeyValue(v);{color}
> switch (code) {
> // Override INCLUDE and continue to evaluate.
> case INCLUDE_AND_NEXT_COL:
> rc = ReturnCode.INCLUDE_AND_NEXT_COL; // FindBugs SF_SWITCH_FALLTHROUGH
> case INCLUDE:
> {color:red} transformed = filter.transformCell(transformed);{color}
> continue;
> case SEEK_NEXT_USING_HINT:
> seekHintFilter = filter;
> return code;
> default:
> return code;
> }
> }
> {code}
> notice the “{color:red}” label line, first line is a recursive invocation, it will assign a Cell results to the FilterList.transformedKV(we call it A), the results is from the FilterList with 2 SingleColumnValueFilter, so A with contains the cell value, while the second line with return A to the var transformed.
> back to the following loop, we can see the FilterList return results is var "transformed " which will override in each loop, so the value is determined by the last filter, so the order of KeyOnlyFilter will impact the results.
> {code:java}
> Cell transformed = v;
> ReturnCode rc = operator == Operator.MUST_PASS_ONE?
> ReturnCode.SKIP: ReturnCode.INCLUDE;
> int listize = filters.size();
> for (int i = 0; i < listize; i++) {
> Filter filter = filters.get(i);
> if (operator == Operator.MUST_PASS_ALL) {
> if (filter.filterAllRemaining()) {
> return ReturnCode.NEXT_ROW;
> }
> ReturnCode code = filter.filterKeyValue(v);
> switch (code) {
> // Override INCLUDE and continue to evaluate.
> case INCLUDE_AND_NEXT_COL:
> rc = ReturnCode.INCLUDE_AND_NEXT_COL; // FindBugs SF_SWITCH_FALLTHROUGH
> case INCLUDE:
> transformed = filter.transformCell(transformed);
> continue;
> case SEEK_NEXT_USING_HINT:
> seekHintFilter = filter;
> return code;
> default:
> return code;
> }
>
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)