You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@skywalking.apache.org by wu...@apache.org on 2021/10/11 15:26:23 UTC
[skywalking] 01/01: Support `(str->long)` cast statement in OAL
core engine.
This is an automated email from the ASF dual-hosted git repository.
wusheng pushed a commit to branch oal2
in repository https://gitbox.apache.org/repos/asf/skywalking.git
commit 38cba46a3317f8a4f3552ee60efa603139c27bcc
Author: Wu Sheng <wu...@foxmail.com>
AuthorDate: Mon Oct 11 23:26:07 2021 +0800
Support `(str->long)` cast statement in OAL core engine.
---
CHANGES.md | 3 ++
docs/en/concepts-and-designs/oal.md | 22 +++++++++++++--
.../apache/skywalking/oal/rt/grammar/OALLexer.g4 | 3 ++
.../apache/skywalking/oal/rt/grammar/OALParser.g4 | 28 +++++++++++++++----
.../skywalking/oal/rt/parser/AnalysisResult.java | 6 ++++
.../apache/skywalking/oal/rt/parser/Argument.java | 4 +++
.../oal/rt/parser/ConditionExpression.java | 1 +
.../skywalking/oal/rt/parser/DeepAnalysis.java | 12 ++++++--
.../skywalking/oal/rt/parser/EntryMethod.java | 5 ++--
.../skywalking/oal/rt/parser/OALListener.java | 15 ++++++++++
.../Argument.java => util/TypeCastUtil.java} | 32 ++++++++++++----------
.../skywalking/oal/rt/parser/ScriptParserTest.java | 22 ++++++++++++++-
12 files changed, 123 insertions(+), 30 deletions(-)
diff --git a/CHANGES.md b/CHANGES.md
index d671d8a..a6eff8e 100644
--- a/CHANGES.md
+++ b/CHANGES.md
@@ -12,6 +12,9 @@ Release Notes.
* Upgrade Armeria to 1.12, upgrade OpenSearch test version to 1.1.0.
* Add component definition for `Apache-Kylin`.
* Enhance `get` generation mechanism of OAL engine, support map type of source's field.
+* Fix `funcParamExpression` and `literalExpression` can't be used in the same aggregation function.
+* Support cast statement in the OAL core engine.
+* Support `(str->long)` cast statement.
#### UI
* Optimize endpoint dependency.
diff --git a/docs/en/concepts-and-designs/oal.md b/docs/en/concepts-and-designs/oal.md
index 6b9ca92..e152396 100644
--- a/docs/en/concepts-and-designs/oal.md
+++ b/docs/en/concepts-and-designs/oal.md
@@ -15,15 +15,17 @@ You can open set `SW_OAL_ENGINE_DEBUG=Y` at system env to see which classes are
Scripts should be named `*.oal`
```
// Declare the metrics.
-METRICS_NAME = from(SCOPE.(* | [FIELD][,FIELD ...]))
-[.filter(FIELD OP [INT | STRING])]
+METRICS_NAME = from(SCOPE.(* | [FIELD][,FIELD ...]) CAST)
+[.filter(FIELD CAST OP [INT | STRING])]
.FUNCTION([PARAM][, PARAM ...])
// Disable hard code
disable(METRICS_NAME);
```
-## Scope
+## From
+The **from** statement defines the data source of this OAL expression.
+
Primary **SCOPE**s are `All`, `Service`, `ServiceInstance`, `Endpoint`, `ServiceRelation`, `ServiceInstanceRelation`, and `EndpointRelation`.
There are also some secondary scopes which belong to a primary scope.
@@ -95,6 +97,20 @@ All metrics data will be grouped by Scope.ID and min-level TimeBucket.
- In the `Endpoint` scope, the Scope.ID is same as the Endpoint ID (i.e. the unique ID based on service and its endpoint).
+## Cast
+Fields of source are static type. In some cases, the type required the filter expression and aggregation function doesn't
+match the type in the source, such as tag value in the source is String type, most aggregation calculation requires numeric.
+
+Cast expression is provided to do so.
+- `(str->long)`, cast string type into long.
+
+> mq_consume_latency = from(Service.tag["transmission.latency"](str->long)).longAvg(); // the value of tag is string type.
+
+Cast statement is supported in
+1. **From statement**. `from(source.attre(cast))`.
+2. **Filter expression**. `.filter(tag["transmission.latency"](str->long) > 0)`
+3. **Aggregation function parameter**. `.longAvg(strField1(str->long)== 1, strField2(str->long))`
+
## Disable
`Disable` is an advanced statement in OAL, which is only used in certain cases.
Some of the aggregation and metrics are defined through core hard codes. Examples include `segment` and `top_n_database_statement`.
diff --git a/oap-server/oal-grammar/src/main/antlr4/org/apache/skywalking/oal/rt/grammar/OALLexer.g4 b/oap-server/oal-grammar/src/main/antlr4/org/apache/skywalking/oal/rt/grammar/OALLexer.g4
index b881a98..dba80fe 100644
--- a/oap-server/oal-grammar/src/main/antlr4/org/apache/skywalking/oal/rt/grammar/OALLexer.g4
+++ b/oap-server/oal-grammar/src/main/antlr4/org/apache/skywalking/oal/rt/grammar/OALLexer.g4
@@ -135,3 +135,6 @@ fragment Letter
| ~[\u0000-\u007F\uD800-\uDBFF] // covers all characters above 0x7F which are not a surrogate
| [\uD800-\uDBFF] [\uDC00-\uDFFF] // covers UTF-16 surrogate pairs encodings for U+10000 to U+10FFFF
;
+
+// Type cast rule
+STRING_TO_LONG: '(str->long)';
\ No newline at end of file
diff --git a/oap-server/oal-grammar/src/main/antlr4/org/apache/skywalking/oal/rt/grammar/OALParser.g4 b/oap-server/oal-grammar/src/main/antlr4/org/apache/skywalking/oal/rt/grammar/OALParser.g4
index 5d82076..584d485 100644
--- a/oap-server/oal-grammar/src/main/antlr4/org/apache/skywalking/oal/rt/grammar/OALParser.g4
+++ b/oap-server/oal-grammar/src/main/antlr4/org/apache/skywalking/oal/rt/grammar/OALParser.g4
@@ -38,7 +38,7 @@ disableStatement
;
metricStatement
- : FROM LR_BRACKET source (sourceAttributeStmt+) RR_BRACKET (filterStatement+)? DOT aggregateFunction
+ : FROM LR_BRACKET source (sourceAttributeStmt+) (sourceAttrCast)? RR_BRACKET (filterStatement+)? DOT aggregateFunction
;
filterStatement
@@ -69,7 +69,7 @@ sourceAttributeStmt
;
sourceAttribute
- : IDENTIFIER | ALL
+ : IDENTIFIER | ALL | mapAttribute
;
variable
@@ -77,7 +77,7 @@ variable
;
aggregateFunction
- : functionName LR_BRACKET ((funcParamExpression (COMMA funcParamExpression)?) | (literalExpression (COMMA literalExpression)?))? RR_BRACKET
+ : functionName LR_BRACKET ((funcParamExpression|literalExpression) (COMMA (funcParamExpression | literalExpression))?)? RR_BRACKET
;
functionName
@@ -89,7 +89,7 @@ funcParamExpression
;
literalExpression
- : BOOL_LITERAL | NUMBER_LITERAL | IDENTIFIER
+ : BOOL_LITERAL | NUMBER_LITERAL | IDENTIFIER functionArgCast?
;
expression
@@ -153,11 +153,11 @@ multiConditionValue
;
conditionAttributeStmt
- : conditionAttribute ((DOT conditionAttribute)*)
+ : conditionAttribute ((DOT conditionAttribute)*) (expressionAttrCast)?
;
conditionAttribute
- : IDENTIFIER | mapAttribute
+ : (IDENTIFIER | mapAttribute)
;
mapAttribute
@@ -179,3 +179,19 @@ enumConditionValue
numberConditionValue
: NUMBER_LITERAL
;
+
+sourceAttrCast
+ : castStmt
+ ;
+
+expressionAttrCast
+ : castStmt
+ ;
+
+functionArgCast
+ : castStmt
+ ;
+
+castStmt
+ : STRING_TO_LONG
+ ;
\ No newline at end of file
diff --git a/oap-server/oal-rt/src/main/java/org/apache/skywalking/oal/rt/parser/AnalysisResult.java b/oap-server/oal-rt/src/main/java/org/apache/skywalking/oal/rt/parser/AnalysisResult.java
index e954c78..36a22d7 100644
--- a/oap-server/oal-rt/src/main/java/org/apache/skywalking/oal/rt/parser/AnalysisResult.java
+++ b/oap-server/oal-rt/src/main/java/org/apache/skywalking/oal/rt/parser/AnalysisResult.java
@@ -46,6 +46,8 @@ public class AnalysisResult {
private List<String> sourceAttribute = new ArrayList<>();
+ private String sourceCastType;
+
private String aggregationFunctionName;
private String metricsClassName;
@@ -110,6 +112,10 @@ public class AnalysisResult {
funcArgs.add(argument);
}
+ public void addCastForLatestArg(String castType) {
+ funcArgs.get(funcArgs.size() - 1).setCastType(castType);
+ }
+
public Argument getNextFuncArg() {
return funcArgs.get(argGetIdx++);
}
diff --git a/oap-server/oal-rt/src/main/java/org/apache/skywalking/oal/rt/parser/Argument.java b/oap-server/oal-rt/src/main/java/org/apache/skywalking/oal/rt/parser/Argument.java
index f84f6db..8766ce1 100644
--- a/oap-server/oal-rt/src/main/java/org/apache/skywalking/oal/rt/parser/Argument.java
+++ b/oap-server/oal-rt/src/main/java/org/apache/skywalking/oal/rt/parser/Argument.java
@@ -21,6 +21,7 @@ package org.apache.skywalking.oal.rt.parser;
import java.util.List;
import lombok.Getter;
import lombok.RequiredArgsConstructor;
+import lombok.Setter;
/**
* Function argument.
@@ -32,4 +33,7 @@ public class Argument {
private final int type;
private final List<String> text;
+
+ @Setter
+ private String castType;
}
diff --git a/oap-server/oal-rt/src/main/java/org/apache/skywalking/oal/rt/parser/ConditionExpression.java b/oap-server/oal-rt/src/main/java/org/apache/skywalking/oal/rt/parser/ConditionExpression.java
index 8a1efaf..3e611ba 100644
--- a/oap-server/oal-rt/src/main/java/org/apache/skywalking/oal/rt/parser/ConditionExpression.java
+++ b/oap-server/oal-rt/src/main/java/org/apache/skywalking/oal/rt/parser/ConditionExpression.java
@@ -36,6 +36,7 @@ public class ConditionExpression {
private String value;
private List<String> values;
private boolean number;
+ private String castType;
public ConditionExpression(final String expressionType, final String attributes, final String value) {
this.expressionType = expressionType;
diff --git a/oap-server/oal-rt/src/main/java/org/apache/skywalking/oal/rt/parser/DeepAnalysis.java b/oap-server/oal-rt/src/main/java/org/apache/skywalking/oal/rt/parser/DeepAnalysis.java
index 13977f9..e7a87cc 100644
--- a/oap-server/oal-rt/src/main/java/org/apache/skywalking/oal/rt/parser/DeepAnalysis.java
+++ b/oap-server/oal-rt/src/main/java/org/apache/skywalking/oal/rt/parser/DeepAnalysis.java
@@ -24,6 +24,7 @@ import java.lang.reflect.Method;
import java.lang.reflect.Parameter;
import java.util.List;
import org.apache.skywalking.oal.rt.util.ClassMethodUtil;
+import org.apache.skywalking.oal.rt.util.TypeCastUtil;
import org.apache.skywalking.oap.server.core.analysis.metrics.Metrics;
import org.apache.skywalking.oap.server.core.analysis.metrics.annotation.Arg;
import org.apache.skywalking.oap.server.core.analysis.metrics.annotation.ConstOne;
@@ -56,7 +57,7 @@ public class DeepAnalysis {
final Expression filterExpression = new Expression();
filterExpression.setExpressionObject(matcherInfo.getMatcher().getName());
- filterExpression.setLeft("source." + getter);
+ filterExpression.setLeft(TypeCastUtil.withCast(expression.getCastType(), "source." + getter));
filterExpression.setRight(expression.getValue());
result.addFilterExpressions(filterExpression);
}
@@ -94,7 +95,12 @@ public class DeepAnalysis {
Annotation annotation = parameterAnnotations[0];
if (annotation instanceof SourceFrom) {
entryMethod.addArg(
- parameterType, "source." + ClassMethodUtil.toGetMethod(result.getSourceAttribute()));
+ parameterType,
+ TypeCastUtil.withCast(
+ result.getSourceCastType(),
+ "source." + ClassMethodUtil.toGetMethod(result.getSourceAttribute())
+ )
+ );
} else if (annotation instanceof ConstOne) {
entryMethod.addArg(parameterType, "1");
} else if (annotation instanceof org.apache.skywalking.oap.server.core.analysis.metrics.annotation.Expression) {
@@ -113,7 +119,7 @@ public class DeepAnalysis {
final Expression argExpression = new Expression();
argExpression.setRight(expression.getValue());
argExpression.setExpressionObject(matcherInfo.getMatcher().getName());
- argExpression.setLeft("source." + getter);
+ argExpression.setLeft(TypeCastUtil.withCast(expression.getCastType(), "source." + getter));
entryMethod.addArg(argExpression);
}
diff --git a/oap-server/oal-rt/src/main/java/org/apache/skywalking/oal/rt/parser/EntryMethod.java b/oap-server/oal-rt/src/main/java/org/apache/skywalking/oal/rt/parser/EntryMethod.java
index 6f002a3..251f8e9 100644
--- a/oap-server/oal-rt/src/main/java/org/apache/skywalking/oal/rt/parser/EntryMethod.java
+++ b/oap-server/oal-rt/src/main/java/org/apache/skywalking/oal/rt/parser/EntryMethod.java
@@ -23,6 +23,7 @@ import java.util.List;
import lombok.Getter;
import lombok.Setter;
import org.apache.skywalking.oal.rt.util.ClassMethodUtil;
+import org.apache.skywalking.oal.rt.util.TypeCastUtil;
@Getter
@Setter
@@ -42,9 +43,9 @@ public class EntryMethod {
return;
}
addArg(parameterType, arg.getType(), parameterType.equals(boolean.class) ?
- "source." + ClassMethodUtil.toIsMethod(arg.getText())
+ TypeCastUtil.withCast(arg.getCastType(), "source." + ClassMethodUtil.toIsMethod(arg.getText()))
:
- "source." + ClassMethodUtil.toGetMethod(arg.getText()));
+ TypeCastUtil.withCast(arg.getCastType(), "source." + ClassMethodUtil.toGetMethod(arg.getText())));
}
void addArg(Class<?> parameterType, String expression) {
diff --git a/oap-server/oal-rt/src/main/java/org/apache/skywalking/oal/rt/parser/OALListener.java b/oap-server/oal-rt/src/main/java/org/apache/skywalking/oal/rt/parser/OALListener.java
index 38cc775..a84e8ad 100644
--- a/oap-server/oal-rt/src/main/java/org/apache/skywalking/oal/rt/parser/OALListener.java
+++ b/oap-server/oal-rt/src/main/java/org/apache/skywalking/oal/rt/parser/OALListener.java
@@ -64,6 +64,11 @@ public class OALListener extends OALParserBaseListener {
}
@Override
+ public void enterSourceAttrCast(OALParser.SourceAttrCastContext ctx) {
+ current.setSourceCastType(ctx.getText());
+ }
+
+ @Override
public void enterVariable(OALParser.VariableContext ctx) {
}
@@ -205,6 +210,11 @@ public class OALListener extends OALParserBaseListener {
enterConditionValue(ctx.getText());
}
+ @Override
+ public void enterExpressionAttrCast(final OALParser.ExpressionAttrCastContext ctx) {
+ conditionExpression.setCastType(ctx.getText());
+ }
+
private void enterConditionValue(String value) {
if (value.split("\\.").length == 2 && !value.startsWith("\"")) {
// Value is an enum.
@@ -226,6 +236,11 @@ public class OALListener extends OALParserBaseListener {
current.addFuncArg(new Argument(EntryMethod.IDENTIFIER_TYPE, Arrays.asList(ctx.getText().split("\\."))));
}
+ @Override
+ public void enterFunctionArgCast(final OALParser.FunctionArgCastContext ctx) {
+ current.addCastForLatestArg(ctx.getText());
+ }
+
private String metricsNameFormat(String source) {
source = firstLetterUpper(source);
int idx;
diff --git a/oap-server/oal-rt/src/main/java/org/apache/skywalking/oal/rt/parser/Argument.java b/oap-server/oal-rt/src/main/java/org/apache/skywalking/oal/rt/util/TypeCastUtil.java
similarity index 53%
copy from oap-server/oal-rt/src/main/java/org/apache/skywalking/oal/rt/parser/Argument.java
copy to oap-server/oal-rt/src/main/java/org/apache/skywalking/oal/rt/util/TypeCastUtil.java
index f84f6db..973dd06 100644
--- a/oap-server/oal-rt/src/main/java/org/apache/skywalking/oal/rt/parser/Argument.java
+++ b/oap-server/oal-rt/src/main/java/org/apache/skywalking/oal/rt/util/TypeCastUtil.java
@@ -16,20 +16,22 @@
*
*/
-package org.apache.skywalking.oal.rt.parser;
+package org.apache.skywalking.oal.rt.util;
-import java.util.List;
-import lombok.Getter;
-import lombok.RequiredArgsConstructor;
-
-/**
- * Function argument.
- */
-@Getter
-@RequiredArgsConstructor
-public class Argument {
-
- private final int type;
-
- private final List<String> text;
+public class TypeCastUtil {
+ /**
+ * @param castType to change the value of given original expression.
+ * @param originalExpression to read the value
+ * @return cast expression if cast type exists and is legal.
+ */
+ public static String withCast(String castType, String originalExpression) {
+ if (castType == null) {
+ return originalExpression;
+ }
+ if ("(str->long)".equals(castType)) {
+ return "Long.parseLong(" + originalExpression + ")";
+ }
+ throw new IllegalArgumentException(
+ "castType:" + castType + " is legal, context expression:" + originalExpression);
+ }
}
diff --git a/oap-server/oal-rt/src/test/java/org/apache/skywalking/oal/rt/parser/ScriptParserTest.java b/oap-server/oal-rt/src/test/java/org/apache/skywalking/oal/rt/parser/ScriptParserTest.java
index c39b361..af076f1 100644
--- a/oap-server/oal-rt/src/test/java/org/apache/skywalking/oal/rt/parser/ScriptParserTest.java
+++ b/oap-server/oal-rt/src/test/java/org/apache/skywalking/oal/rt/parser/ScriptParserTest.java
@@ -253,7 +253,9 @@ public class ScriptParserTest {
@Test
public void testParse9() throws IOException {
ScriptParser parser = ScriptParser.createFromScriptText(
- "ServicePercent = from(Service.sidecar.internalError).filter(sidecar.internalError == \"abc\").percent(sidecar.internalError != \"\");", TEST_SOURCE_PACKAGE);
+ "ServicePercent = from(Service.sidecar.internalError).filter(sidecar.internalError == \"abc\").percent(sidecar.internalError != \"\");",
+ TEST_SOURCE_PACKAGE
+ );
List<AnalysisResult> results = parser.parse().getMetricsStmts();
AnalysisResult servicePercent = results.get(0);
@@ -299,6 +301,24 @@ public class ScriptParserTest {
}
@Test
+ public void testParse12() throws IOException {
+ ScriptParser parser = ScriptParser.createFromScriptText(
+ "cast_metrics = from(Service.tag[\"transmission.latency\"](str->long)).filter(tag[\"transmission.latency\"](str->long) > 0).longAvg(strField1(str->long)== 1, strField2(str->long));",
+ TEST_SOURCE_PACKAGE
+ );
+ List<AnalysisResult> results = parser.parse().getMetricsStmts();
+ AnalysisResult castExp = results.get(0);
+ Assert.assertEquals("(str->long)", castExp.getSourceCastType());
+ final List<Expression> filterExpressions = castExp.getFilterExpressions();
+ Assert.assertEquals(1, filterExpressions.size());
+ Assert.assertEquals(
+ "Long.parseLong(source.getTag(\"transmission.latency\"))", filterExpressions.get(0).getLeft());
+ Assert.assertEquals("(str->long)", castExp.getFuncConditionExpressions().get(0).getCastType());
+ Assert.assertEquals(EntryMethod.IDENTIFIER_TYPE, castExp.getFuncArgs().get(0).getType());
+ Assert.assertEquals("(str->long)", castExp.getFuncArgs().get(0).getCastType());
+ }
+
+ @Test
public void testDisable() throws IOException {
ScriptParser parser = ScriptParser.createFromScriptText("disable(segment);", TEST_SOURCE_PACKAGE);
DisableCollection collection = parser.parse().getDisableCollection();