You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@skywalking.apache.org by wu...@apache.org on 2021/10/11 15:26:22 UTC
[skywalking] branch oal2 created (now 38cba46)
This is an automated email from the ASF dual-hosted git repository.
wusheng pushed a change to branch oal2
in repository https://gitbox.apache.org/repos/asf/skywalking.git.
at 38cba46 Support `(str->long)` cast statement in OAL core engine.
This branch includes the following new commits:
new 38cba46 Support `(str->long)` cast statement in OAL core engine.
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
[skywalking] 01/01: Support `(str->long)` cast statement in OAL
core engine.
Posted by wu...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
wusheng pushed a commit to branch oal2
in repository https://gitbox.apache.org/repos/asf/skywalking.git
commit 38cba46a3317f8a4f3552ee60efa603139c27bcc
Author: Wu Sheng <wu...@foxmail.com>
AuthorDate: Mon Oct 11 23:26:07 2021 +0800
Support `(str->long)` cast statement in OAL core engine.
---
CHANGES.md | 3 ++
docs/en/concepts-and-designs/oal.md | 22 +++++++++++++--
.../apache/skywalking/oal/rt/grammar/OALLexer.g4 | 3 ++
.../apache/skywalking/oal/rt/grammar/OALParser.g4 | 28 +++++++++++++++----
.../skywalking/oal/rt/parser/AnalysisResult.java | 6 ++++
.../apache/skywalking/oal/rt/parser/Argument.java | 4 +++
.../oal/rt/parser/ConditionExpression.java | 1 +
.../skywalking/oal/rt/parser/DeepAnalysis.java | 12 ++++++--
.../skywalking/oal/rt/parser/EntryMethod.java | 5 ++--
.../skywalking/oal/rt/parser/OALListener.java | 15 ++++++++++
.../Argument.java => util/TypeCastUtil.java} | 32 ++++++++++++----------
.../skywalking/oal/rt/parser/ScriptParserTest.java | 22 ++++++++++++++-
12 files changed, 123 insertions(+), 30 deletions(-)
diff --git a/CHANGES.md b/CHANGES.md
index d671d8a..a6eff8e 100644
--- a/CHANGES.md
+++ b/CHANGES.md
@@ -12,6 +12,9 @@ Release Notes.
* Upgrade Armeria to 1.12, upgrade OpenSearch test version to 1.1.0.
* Add component definition for `Apache-Kylin`.
* Enhance `get` generation mechanism of OAL engine, support map type of source's field.
+* Fix `funcParamExpression` and `literalExpression` can't be used in the same aggregation function.
+* Support cast statement in the OAL core engine.
+* Support `(str->long)` cast statement.
#### UI
* Optimize endpoint dependency.
diff --git a/docs/en/concepts-and-designs/oal.md b/docs/en/concepts-and-designs/oal.md
index 6b9ca92..e152396 100644
--- a/docs/en/concepts-and-designs/oal.md
+++ b/docs/en/concepts-and-designs/oal.md
@@ -15,15 +15,17 @@ You can open set `SW_OAL_ENGINE_DEBUG=Y` at system env to see which classes are
Scripts should be named `*.oal`
```
// Declare the metrics.
-METRICS_NAME = from(SCOPE.(* | [FIELD][,FIELD ...]))
-[.filter(FIELD OP [INT | STRING])]
+METRICS_NAME = from(SCOPE.(* | [FIELD][,FIELD ...]) CAST)
+[.filter(FIELD CAST OP [INT | STRING])]
.FUNCTION([PARAM][, PARAM ...])
// Disable hard code
disable(METRICS_NAME);
```
-## Scope
+## From
+The **from** statement defines the data source of this OAL expression.
+
Primary **SCOPE**s are `All`, `Service`, `ServiceInstance`, `Endpoint`, `ServiceRelation`, `ServiceInstanceRelation`, and `EndpointRelation`.
There are also some secondary scopes which belong to a primary scope.
@@ -95,6 +97,20 @@ All metrics data will be grouped by Scope.ID and min-level TimeBucket.
- In the `Endpoint` scope, the Scope.ID is same as the Endpoint ID (i.e. the unique ID based on service and its endpoint).
+## Cast
+Fields of source are static type. In some cases, the type required the filter expression and aggregation function doesn't
+match the type in the source, such as tag value in the source is String type, most aggregation calculation requires numeric.
+
+Cast expression is provided to do so.
+- `(str->long)`, cast string type into long.
+
+> mq_consume_latency = from(Service.tag["transmission.latency"](str->long)).longAvg(); // the value of tag is string type.
+
+Cast statement is supported in
+1. **From statement**. `from(source.attre(cast))`.
+2. **Filter expression**. `.filter(tag["transmission.latency"](str->long) > 0)`
+3. **Aggregation function parameter**. `.longAvg(strField1(str->long)== 1, strField2(str->long))`
+
## Disable
`Disable` is an advanced statement in OAL, which is only used in certain cases.
Some of the aggregation and metrics are defined through core hard codes. Examples include `segment` and `top_n_database_statement`.
diff --git a/oap-server/oal-grammar/src/main/antlr4/org/apache/skywalking/oal/rt/grammar/OALLexer.g4 b/oap-server/oal-grammar/src/main/antlr4/org/apache/skywalking/oal/rt/grammar/OALLexer.g4
index b881a98..dba80fe 100644
--- a/oap-server/oal-grammar/src/main/antlr4/org/apache/skywalking/oal/rt/grammar/OALLexer.g4
+++ b/oap-server/oal-grammar/src/main/antlr4/org/apache/skywalking/oal/rt/grammar/OALLexer.g4
@@ -135,3 +135,6 @@ fragment Letter
| ~[\u0000-\u007F\uD800-\uDBFF] // covers all characters above 0x7F which are not a surrogate
| [\uD800-\uDBFF] [\uDC00-\uDFFF] // covers UTF-16 surrogate pairs encodings for U+10000 to U+10FFFF
;
+
+// Type cast rule
+STRING_TO_LONG: '(str->long)';
\ No newline at end of file
diff --git a/oap-server/oal-grammar/src/main/antlr4/org/apache/skywalking/oal/rt/grammar/OALParser.g4 b/oap-server/oal-grammar/src/main/antlr4/org/apache/skywalking/oal/rt/grammar/OALParser.g4
index 5d82076..584d485 100644
--- a/oap-server/oal-grammar/src/main/antlr4/org/apache/skywalking/oal/rt/grammar/OALParser.g4
+++ b/oap-server/oal-grammar/src/main/antlr4/org/apache/skywalking/oal/rt/grammar/OALParser.g4
@@ -38,7 +38,7 @@ disableStatement
;
metricStatement
- : FROM LR_BRACKET source (sourceAttributeStmt+) RR_BRACKET (filterStatement+)? DOT aggregateFunction
+ : FROM LR_BRACKET source (sourceAttributeStmt+) (sourceAttrCast)? RR_BRACKET (filterStatement+)? DOT aggregateFunction
;
filterStatement
@@ -69,7 +69,7 @@ sourceAttributeStmt
;
sourceAttribute
- : IDENTIFIER | ALL
+ : IDENTIFIER | ALL | mapAttribute
;
variable
@@ -77,7 +77,7 @@ variable
;
aggregateFunction
- : functionName LR_BRACKET ((funcParamExpression (COMMA funcParamExpression)?) | (literalExpression (COMMA literalExpression)?))? RR_BRACKET
+ : functionName LR_BRACKET ((funcParamExpression|literalExpression) (COMMA (funcParamExpression | literalExpression))?)? RR_BRACKET
;
functionName
@@ -89,7 +89,7 @@ funcParamExpression
;
literalExpression
- : BOOL_LITERAL | NUMBER_LITERAL | IDENTIFIER
+ : BOOL_LITERAL | NUMBER_LITERAL | IDENTIFIER functionArgCast?
;
expression
@@ -153,11 +153,11 @@ multiConditionValue
;
conditionAttributeStmt
- : conditionAttribute ((DOT conditionAttribute)*)
+ : conditionAttribute ((DOT conditionAttribute)*) (expressionAttrCast)?
;
conditionAttribute
- : IDENTIFIER | mapAttribute
+ : (IDENTIFIER | mapAttribute)
;
mapAttribute
@@ -179,3 +179,19 @@ enumConditionValue
numberConditionValue
: NUMBER_LITERAL
;
+
+sourceAttrCast
+ : castStmt
+ ;
+
+expressionAttrCast
+ : castStmt
+ ;
+
+functionArgCast
+ : castStmt
+ ;
+
+castStmt
+ : STRING_TO_LONG
+ ;
\ No newline at end of file
diff --git a/oap-server/oal-rt/src/main/java/org/apache/skywalking/oal/rt/parser/AnalysisResult.java b/oap-server/oal-rt/src/main/java/org/apache/skywalking/oal/rt/parser/AnalysisResult.java
index e954c78..36a22d7 100644
--- a/oap-server/oal-rt/src/main/java/org/apache/skywalking/oal/rt/parser/AnalysisResult.java
+++ b/oap-server/oal-rt/src/main/java/org/apache/skywalking/oal/rt/parser/AnalysisResult.java
@@ -46,6 +46,8 @@ public class AnalysisResult {
private List<String> sourceAttribute = new ArrayList<>();
+ private String sourceCastType;
+
private String aggregationFunctionName;
private String metricsClassName;
@@ -110,6 +112,10 @@ public class AnalysisResult {
funcArgs.add(argument);
}
+ public void addCastForLatestArg(String castType) {
+ funcArgs.get(funcArgs.size() - 1).setCastType(castType);
+ }
+
public Argument getNextFuncArg() {
return funcArgs.get(argGetIdx++);
}
diff --git a/oap-server/oal-rt/src/main/java/org/apache/skywalking/oal/rt/parser/Argument.java b/oap-server/oal-rt/src/main/java/org/apache/skywalking/oal/rt/parser/Argument.java
index f84f6db..8766ce1 100644
--- a/oap-server/oal-rt/src/main/java/org/apache/skywalking/oal/rt/parser/Argument.java
+++ b/oap-server/oal-rt/src/main/java/org/apache/skywalking/oal/rt/parser/Argument.java
@@ -21,6 +21,7 @@ package org.apache.skywalking.oal.rt.parser;
import java.util.List;
import lombok.Getter;
import lombok.RequiredArgsConstructor;
+import lombok.Setter;
/**
* Function argument.
@@ -32,4 +33,7 @@ public class Argument {
private final int type;
private final List<String> text;
+
+ @Setter
+ private String castType;
}
diff --git a/oap-server/oal-rt/src/main/java/org/apache/skywalking/oal/rt/parser/ConditionExpression.java b/oap-server/oal-rt/src/main/java/org/apache/skywalking/oal/rt/parser/ConditionExpression.java
index 8a1efaf..3e611ba 100644
--- a/oap-server/oal-rt/src/main/java/org/apache/skywalking/oal/rt/parser/ConditionExpression.java
+++ b/oap-server/oal-rt/src/main/java/org/apache/skywalking/oal/rt/parser/ConditionExpression.java
@@ -36,6 +36,7 @@ public class ConditionExpression {
private String value;
private List<String> values;
private boolean number;
+ private String castType;
public ConditionExpression(final String expressionType, final String attributes, final String value) {
this.expressionType = expressionType;
diff --git a/oap-server/oal-rt/src/main/java/org/apache/skywalking/oal/rt/parser/DeepAnalysis.java b/oap-server/oal-rt/src/main/java/org/apache/skywalking/oal/rt/parser/DeepAnalysis.java
index 13977f9..e7a87cc 100644
--- a/oap-server/oal-rt/src/main/java/org/apache/skywalking/oal/rt/parser/DeepAnalysis.java
+++ b/oap-server/oal-rt/src/main/java/org/apache/skywalking/oal/rt/parser/DeepAnalysis.java
@@ -24,6 +24,7 @@ import java.lang.reflect.Method;
import java.lang.reflect.Parameter;
import java.util.List;
import org.apache.skywalking.oal.rt.util.ClassMethodUtil;
+import org.apache.skywalking.oal.rt.util.TypeCastUtil;
import org.apache.skywalking.oap.server.core.analysis.metrics.Metrics;
import org.apache.skywalking.oap.server.core.analysis.metrics.annotation.Arg;
import org.apache.skywalking.oap.server.core.analysis.metrics.annotation.ConstOne;
@@ -56,7 +57,7 @@ public class DeepAnalysis {
final Expression filterExpression = new Expression();
filterExpression.setExpressionObject(matcherInfo.getMatcher().getName());
- filterExpression.setLeft("source." + getter);
+ filterExpression.setLeft(TypeCastUtil.withCast(expression.getCastType(), "source." + getter));
filterExpression.setRight(expression.getValue());
result.addFilterExpressions(filterExpression);
}
@@ -94,7 +95,12 @@ public class DeepAnalysis {
Annotation annotation = parameterAnnotations[0];
if (annotation instanceof SourceFrom) {
entryMethod.addArg(
- parameterType, "source." + ClassMethodUtil.toGetMethod(result.getSourceAttribute()));
+ parameterType,
+ TypeCastUtil.withCast(
+ result.getSourceCastType(),
+ "source." + ClassMethodUtil.toGetMethod(result.getSourceAttribute())
+ )
+ );
} else if (annotation instanceof ConstOne) {
entryMethod.addArg(parameterType, "1");
} else if (annotation instanceof org.apache.skywalking.oap.server.core.analysis.metrics.annotation.Expression) {
@@ -113,7 +119,7 @@ public class DeepAnalysis {
final Expression argExpression = new Expression();
argExpression.setRight(expression.getValue());
argExpression.setExpressionObject(matcherInfo.getMatcher().getName());
- argExpression.setLeft("source." + getter);
+ argExpression.setLeft(TypeCastUtil.withCast(expression.getCastType(), "source." + getter));
entryMethod.addArg(argExpression);
}
diff --git a/oap-server/oal-rt/src/main/java/org/apache/skywalking/oal/rt/parser/EntryMethod.java b/oap-server/oal-rt/src/main/java/org/apache/skywalking/oal/rt/parser/EntryMethod.java
index 6f002a3..251f8e9 100644
--- a/oap-server/oal-rt/src/main/java/org/apache/skywalking/oal/rt/parser/EntryMethod.java
+++ b/oap-server/oal-rt/src/main/java/org/apache/skywalking/oal/rt/parser/EntryMethod.java
@@ -23,6 +23,7 @@ import java.util.List;
import lombok.Getter;
import lombok.Setter;
import org.apache.skywalking.oal.rt.util.ClassMethodUtil;
+import org.apache.skywalking.oal.rt.util.TypeCastUtil;
@Getter
@Setter
@@ -42,9 +43,9 @@ public class EntryMethod {
return;
}
addArg(parameterType, arg.getType(), parameterType.equals(boolean.class) ?
- "source." + ClassMethodUtil.toIsMethod(arg.getText())
+ TypeCastUtil.withCast(arg.getCastType(), "source." + ClassMethodUtil.toIsMethod(arg.getText()))
:
- "source." + ClassMethodUtil.toGetMethod(arg.getText()));
+ TypeCastUtil.withCast(arg.getCastType(), "source." + ClassMethodUtil.toGetMethod(arg.getText())));
}
void addArg(Class<?> parameterType, String expression) {
diff --git a/oap-server/oal-rt/src/main/java/org/apache/skywalking/oal/rt/parser/OALListener.java b/oap-server/oal-rt/src/main/java/org/apache/skywalking/oal/rt/parser/OALListener.java
index 38cc775..a84e8ad 100644
--- a/oap-server/oal-rt/src/main/java/org/apache/skywalking/oal/rt/parser/OALListener.java
+++ b/oap-server/oal-rt/src/main/java/org/apache/skywalking/oal/rt/parser/OALListener.java
@@ -64,6 +64,11 @@ public class OALListener extends OALParserBaseListener {
}
@Override
+ public void enterSourceAttrCast(OALParser.SourceAttrCastContext ctx) {
+ current.setSourceCastType(ctx.getText());
+ }
+
+ @Override
public void enterVariable(OALParser.VariableContext ctx) {
}
@@ -205,6 +210,11 @@ public class OALListener extends OALParserBaseListener {
enterConditionValue(ctx.getText());
}
+ @Override
+ public void enterExpressionAttrCast(final OALParser.ExpressionAttrCastContext ctx) {
+ conditionExpression.setCastType(ctx.getText());
+ }
+
private void enterConditionValue(String value) {
if (value.split("\\.").length == 2 && !value.startsWith("\"")) {
// Value is an enum.
@@ -226,6 +236,11 @@ public class OALListener extends OALParserBaseListener {
current.addFuncArg(new Argument(EntryMethod.IDENTIFIER_TYPE, Arrays.asList(ctx.getText().split("\\."))));
}
+ @Override
+ public void enterFunctionArgCast(final OALParser.FunctionArgCastContext ctx) {
+ current.addCastForLatestArg(ctx.getText());
+ }
+
private String metricsNameFormat(String source) {
source = firstLetterUpper(source);
int idx;
diff --git a/oap-server/oal-rt/src/main/java/org/apache/skywalking/oal/rt/parser/Argument.java b/oap-server/oal-rt/src/main/java/org/apache/skywalking/oal/rt/util/TypeCastUtil.java
similarity index 53%
copy from oap-server/oal-rt/src/main/java/org/apache/skywalking/oal/rt/parser/Argument.java
copy to oap-server/oal-rt/src/main/java/org/apache/skywalking/oal/rt/util/TypeCastUtil.java
index f84f6db..973dd06 100644
--- a/oap-server/oal-rt/src/main/java/org/apache/skywalking/oal/rt/parser/Argument.java
+++ b/oap-server/oal-rt/src/main/java/org/apache/skywalking/oal/rt/util/TypeCastUtil.java
@@ -16,20 +16,22 @@
*
*/
-package org.apache.skywalking.oal.rt.parser;
+package org.apache.skywalking.oal.rt.util;
-import java.util.List;
-import lombok.Getter;
-import lombok.RequiredArgsConstructor;
-
-/**
- * Function argument.
- */
-@Getter
-@RequiredArgsConstructor
-public class Argument {
-
- private final int type;
-
- private final List<String> text;
+public class TypeCastUtil {
+ /**
+ * @param castType to change the value of given original expression.
+ * @param originalExpression to read the value
+ * @return cast expression if cast type exists and is legal.
+ */
+ public static String withCast(String castType, String originalExpression) {
+ if (castType == null) {
+ return originalExpression;
+ }
+ if ("(str->long)".equals(castType)) {
+ return "Long.parseLong(" + originalExpression + ")";
+ }
+ throw new IllegalArgumentException(
+ "castType:" + castType + " is legal, context expression:" + originalExpression);
+ }
}
diff --git a/oap-server/oal-rt/src/test/java/org/apache/skywalking/oal/rt/parser/ScriptParserTest.java b/oap-server/oal-rt/src/test/java/org/apache/skywalking/oal/rt/parser/ScriptParserTest.java
index c39b361..af076f1 100644
--- a/oap-server/oal-rt/src/test/java/org/apache/skywalking/oal/rt/parser/ScriptParserTest.java
+++ b/oap-server/oal-rt/src/test/java/org/apache/skywalking/oal/rt/parser/ScriptParserTest.java
@@ -253,7 +253,9 @@ public class ScriptParserTest {
@Test
public void testParse9() throws IOException {
ScriptParser parser = ScriptParser.createFromScriptText(
- "ServicePercent = from(Service.sidecar.internalError).filter(sidecar.internalError == \"abc\").percent(sidecar.internalError != \"\");", TEST_SOURCE_PACKAGE);
+ "ServicePercent = from(Service.sidecar.internalError).filter(sidecar.internalError == \"abc\").percent(sidecar.internalError != \"\");",
+ TEST_SOURCE_PACKAGE
+ );
List<AnalysisResult> results = parser.parse().getMetricsStmts();
AnalysisResult servicePercent = results.get(0);
@@ -299,6 +301,24 @@ public class ScriptParserTest {
}
@Test
+ public void testParse12() throws IOException {
+ ScriptParser parser = ScriptParser.createFromScriptText(
+ "cast_metrics = from(Service.tag[\"transmission.latency\"](str->long)).filter(tag[\"transmission.latency\"](str->long) > 0).longAvg(strField1(str->long)== 1, strField2(str->long));",
+ TEST_SOURCE_PACKAGE
+ );
+ List<AnalysisResult> results = parser.parse().getMetricsStmts();
+ AnalysisResult castExp = results.get(0);
+ Assert.assertEquals("(str->long)", castExp.getSourceCastType());
+ final List<Expression> filterExpressions = castExp.getFilterExpressions();
+ Assert.assertEquals(1, filterExpressions.size());
+ Assert.assertEquals(
+ "Long.parseLong(source.getTag(\"transmission.latency\"))", filterExpressions.get(0).getLeft());
+ Assert.assertEquals("(str->long)", castExp.getFuncConditionExpressions().get(0).getCastType());
+ Assert.assertEquals(EntryMethod.IDENTIFIER_TYPE, castExp.getFuncArgs().get(0).getType());
+ Assert.assertEquals("(str->long)", castExp.getFuncArgs().get(0).getCastType());
+ }
+
+ @Test
public void testDisable() throws IOException {
ScriptParser parser = ScriptParser.createFromScriptText("disable(segment);", TEST_SOURCE_PACKAGE);
DisableCollection collection = parser.parse().getDisableCollection();