Adding property.key support?

[Ceki Gülcü <ce...@qos.ch>]

Scott,

Instead of discussing the how easy/hard it is to extend the lbel parser, I 
propose that we go through a small exercise together.

Goal
----

Adding support for expressions of type (property.someKey = 'some value')

Summary of required steps
-------------------------

1) o.a.l.lbel.TokenStream needs to return a new token of type "PROPERTY" and
value "someKey".

2) Test that TokenStream does this.

3) Modify the bsubfactor() method in o.a.l.lbel.Parser to handle
PROPERTY tokens. It should create a new leaf node corresponding to the
PROPERTY token.

bsubfactor() is the only part of the parser that deals directly with 
*leaves* of the syntax tree. Unless the basic form of boolean expressions 
change, that is never, bsubfactor() is the only method that needs to be 
modified for adding support for a new keyword.

4) Leaf nodes in LBEL are always contain objects implementing the 
Comparator interface. Thus, we meed to add PropertyComparator in 
o.a.l.lbel.comparator package in order to compare a given property of 
LoggingEvent instances with the key specified in the boolean expression.

5) Test that expressions involving the property keyword work as expected.

Here we go.

Step 1: PROPERTY keyword support in TokenStream
------------------------------------------------

Normally this consists of adding the keyword to an internal map maintained 
by TokenStream.  This is done by a one line statement similar to the following.

   keywordMap.put("property", new Token(Token.PROPERTY, "property"));

However, "property" is special because it is followed by the key of the 
property. (We don't write
"property" but "property.someKey".)

The o.a.l.lbel.TokenStream class heavily relies on java.io.StreamTokenizer 
which has the capability of recognizing identifiers. The next() method does 
much of the work in TokenStream.  TokenStream considers tokens marked as 
StreamTokenizer.TT_WORD as either an identifier or keyword. Keyword 
recognition is a special case for processing words.

Here is the relevant code in TokenStream:

  case StreamTokenizer.TT_WORD:
    String key = tokenizer.sval;
    String lowerCaseKey = key.toLowerCase();
    Token result = (Token) keywordMap.get(lowerCaseKey);
    if(result != null) {
      current = result;
    } else {
      current = new Token(Token.LITERAL, tokenizer.sval);
    }
    break;

In the above code, if the returned token is a keyword, then we return the 
associated Token in the map of keywords maintained by TokenStream. 
Otherwise, a LITERAL token is returned.

We need to change the above code in order to add support for the 
"property." followed by a key. This is easy enough to do.

  case StreamTokenizer.TT_WORD:
    String txt = tokenizer.sval;
    String lowerCaseTxt = txt.toLowerCase();

     if(txt.startsWith("property.")) {
       current = extractPropertyToken(txt);
     } else {
       Token result = (Token) keywordMap.get(lowerCaseTxt);
       if(result != null) {
         current = result;
       } else {
         current = new Token(Token.LITERAL, tokenizer.sval);
       }
     }
     break;

Here is extractPropertyToken(String).

   Token extractPropertyToken(String txt) throws ScanError {
     int point = txt.indexOf('.');
     String key = txt.substring(point+1);
     // Is the key empty? (An empty key is the only thing that can go wrong at
     // this stage.
     if(key == null || key.length() == 0) {
       throw new ScanError("["+txt+"] has zero-legnth key.");
     } else {
       return new Token(Token.PROPERTY, key);
     }
   }

Step 2: Check that TokenStream handles "property.key"

In TokenStreamTest we add a new test.

   public void testProperty() throws IOException, ScanError {
     StringReader sr = new StringReader(" property.x property.xyz property.");
     TokenStream ts = new TokenStream(sr);

     ts.next(); t = ts.getCurrent();
     assertEquals(Token.PROPERTY, t.getType());
     assertEquals("x", t.getValue());

     ts.next(); t = ts.getCurrent();
     assertEquals(Token.PROPERTY, t.getType());
     assertEquals("xyz", t.getValue());

     try {
       ts.next();
       fail("A ScanError should have been thrown");
     } catch(ScanError e) {
     }
   }

If you run this test, the bar should be green.

Step 3: Handle Token.PROPERTY in bsubfactor.
--------------------------------------------

This can be achieved by adding the following few lines of code in the 
switch of bsubfactor.

     case Token.PROPERTY:
       String key = (String) token.getValue();
       ts.next();
       operator = getOperator();
       ts.next();
       literal = getLiteral();
       return new Node(Node.COMPARATOR, new PropertyComparator(operator, 
key, literal));

Case 4: Add PropertyComparator in o.a.l.lbel.comparator.

PropertyComparator can be adapted from  the existing MessageComparator class.

public class PropertyComparator implements Comparator {
   Operator operator;
   String key;
   String rightSide;
   Pattern rightSidePattern;
   Perl5Matcher matcher;

   public PropertyComparator(final Operator operator, String key, String 
rightSide)
     throws ScanError {
     this.operator = operator;
     this.key = key;
     this.rightSide = rightSide;

     if (operator.isRegex()) {
       Perl5Compiler compiler = new Perl5Compiler();
       matcher = new Perl5Matcher();

       try {
         rightSidePattern = compiler.compile(rightSide);
       } catch (MalformedPatternException mfpe) {
         throw new ScanError("Malformed pattern [" + rightSide + "]", mfpe);
       }
     }
   }

   public boolean compare(LoggingEvent event) {
     String leftSide = event.getProperty(key);

     if (operator.isRegex()) {
       boolean match =
         matcher.contains(leftSide, rightSidePattern);
       if (operator.getCode() == Operator.REGEX_MATCH) {
         return match;
       } else {
         return !match;
       }
     }
     int compResult;

     if(leftSide == null) {
       compResult = -1;
     } else {
       compResult = leftSide.compareTo(rightSide);
     }

     switch (operator.getCode()) {
     case Operator.EQUAL:
       return compResult == 0;
     case Operator.NOT_EQUAL:
       return compResult != 0;
     case Operator.GREATER:
       return compResult > 0;
     case Operator.GREATER_OR_EQUAL:
       return compResult >= 0;
     case Operator.LESS:
       return compResult < 0;
     case Operator.LESS_OR_EQUAL:
       return compResult <= 0;
     }

     throw new IllegalStateException(
       "Unreachable state reached, operator " + operator);
   }

   public String toString() {
     return "PropertyComparator("+operator+", "+rightSide+")";
   }
}

The above is almost a straight copy and paste of MessageComparator. At some 
point, we should factor out the common parts in comprators and consolidate 
the code.

Step 5: Test that expressions involving the property keyword work as expected

public class EventEvaluationTest extends TestCase {
   LoggingEvent event;
   EventEvaluator evaluator;

   protected void setUp() throws Exception {
     super.setUp();
     event = new LoggingEvent();
     event.setLevel(Level.INFO);
     event.setMessage("hello world");
     event.setLoggerName("org.wombat");
     event.setProperty("x", "y follows x");
   }

   public void testProperty() throws ScanError {
     evaluator = new LBELEventEvaluator("property.x = 'y follows x'");
     assertTrue(evaluator.evaluate(event));
     evaluator = new LBELEventEvaluator("property.x >= 'y follows x'");
     assertTrue(evaluator.evaluate(event));
     evaluator = new LBELEventEvaluator("property.x <= 'y follows x'");
     assertTrue(evaluator.evaluate(event));

     evaluator = new LBELEventEvaluator("property.x != 'y'");
     assertTrue(evaluator.evaluate(event));
     evaluator = new LBELEventEvaluator("property.x > 'y'");
     assertTrue(evaluator.evaluate(event));
     evaluator = new LBELEventEvaluator("property.x >= 'y'");
     assertTrue(evaluator.evaluate(event));

     evaluator = new LBELEventEvaluator("property.y = 'toto'");
     assertTrue(!evaluator.evaluate(event));
   }
}

I hope this shows how easy is to grow LBEL *even* for exceptional cases 
such as PROPERTY.

-- 
Ceki Gülcü

   The complete log4j manual: http://www.qos.ch/log4j/



---------------------------------------------------------------------
To unsubscribe, e-mail: log4j-dev-unsubscribe@logging.apache.org
For additional commands, e-mail: log4j-dev-help@logging.apache.org