You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@skywalking.apache.org by "yswdqz (via GitHub)" <gi...@apache.org> on 2023/09/24 15:30:00 UTC
[GitHub] [skywalking] yswdqz opened a new pull request, #11354: [Feature] Substitute classes to prepare for native-image.
yswdqz opened a new pull request, #11354:
URL: https://github.com/apache/skywalking/pull/11354
related #11279
- [ ] If this is non-trivial feature, paste the links/URLs to the design doc.
- [x] Update the documentation to include this new feature.
- [x] Tests(including UT, IT, E2E) are added to verify the new feature.
- [ ] If it's UI related, attach the screenshots below.
- [ ] If this pull request closes/resolves/fixes an existing issue, replace the issue number. Closes #<issue number>.
- [x] Update the [`CHANGES` log](https://github.com/apache/skywalking/blob/master/docs/en/changes/changes.md).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on a diff in pull request #11354: [Feature] Substitute classes to prepare for native-image.
Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on code in PR #11354:
URL: https://github.com/apache/skywalking/pull/11354#discussion_r1337920004
##########
oap-server/server-starter/src/main/resources/application.yml:
##########
@@ -319,6 +322,12 @@ envoy-metric:
# Be careful, when using environment variables to pass this configuration, use single quotes(`''`) to avoid it being evaluated by the shell.
k8sServiceNameRule: ${K8S_SERVICE_NAME_RULE:"${pod.metadata.labels.(service.istio.io/canonical-name)}"}
istioServiceNameRule: ${ISTIO_SERVICE_NAME_RULE:"${serviceEntry.metadata.name}"}
+ default-graalvm:
+ acceptMetricsService: ${SW_ENVOY_METRIC_SERVICE:true}
+ alsHTTPAnalysis: ${SW_ENVOY_METRIC_ALS_HTTP_ANALYSIS:""}
+ alsTCPAnalysis: ${SW_ENVOY_METRIC_ALS_TCP_ANALYSIS:""}
+ k8sServiceNameRule: ${K8S_SERVICE_NAME_RULE:"${pod.metadata.labels.(service.istio.io/canonical-name)}"}
+ istioServiceNameRule: ${ISTIO_SERVICE_NAME_RULE:"${serviceEntry.metadata.name}"}
Review Comment:
One more, let's call this `graalvm` and add a new application.yml for graal rather than changing the default one. The end user of our main release can't use the provider, so, let's avoid the confusion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on pull request #11354: [Feature] Substitute classes to prepare for native-image.
Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on PR #11354:
URL: https://github.com/apache/skywalking/pull/11354#issuecomment-1732616163
I will be on flight tomorrow, Would try my best to have a round review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on a diff in pull request #11354: [Feature] Substitute classes to prepare for native-image.
Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on code in PR #11354:
URL: https://github.com/apache/skywalking/pull/11354#discussion_r1336507986
##########
.github/workflows/native-image.yaml:
##########
@@ -0,0 +1,41 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+name: GraalVM Community Edition build
+on:
+ pull_request:
+jobs:
+ build:
+ runs-on: ubuntu-latest
+ steps:
+ - uses: actions/checkout@v3
+ with:
+ submodules: true
+ - uses: graalvm/setup-graalvm@v1
+ with:
+ java-version: '17.0.8'
+ distribution: 'graalvm-community'
+ components: 'native-image'
+ github-token: ${{ secrets.GITHUB_TOKEN }}
+ - name: Native Compile
+ run: |
+ ./mvnw -Dmaven.test.skip install
+ ./mvnw -Pnative clean package -Dmaven.test.skip
Review Comment:
Do we have to run the local install first, then run native profile?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on a diff in pull request #11354: [Feature] Substitute classes to prepare for native-image.
Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on code in PR #11354:
URL: https://github.com/apache/skywalking/pull/11354#discussion_r1337133170
##########
docs/en/setup/backend/backend-graalvm.md:
##########
@@ -0,0 +1,48 @@
+# GraalVM native-image for SkyWalking (Experimental)
+
+## Native-Image
+Native Image is a technology to ahead-of-time compile Java code to a standalone executable, called a native image.
+This executable includes the application classes, classes from its dependencies, runtime library classes,
+and statically linked native code from JDK. It does not run on the Java VM, but includes necessary components like memory management, thread scheduling, and so on from a different runtime system, called “Substrate VM”.
+Substrate VM is the name for the runtime components (like the deoptimizer, garbage collector, thread scheduling etc.).
+The resulting program has faster startup time and lower runtime memory overhead compared to a JVM.
+
+SkyWalking currently offers support for OAP servers running as native-image. However, please note that the OAP started in this manner does not have the same functionality as the regular OAP, and some features are not yet supported.
+
+## Compile Guide
+Notice: If you are not familiar with the compilation process, please read [How-to-build](https://skywalking.apache.org/docs/main/next/en/guides/how-to-build/) first.
+
+The native-image compilation is not enabled by default. To enable it, we need to activate `native` profile during compilation, such as:
+
+```shell
+
+./mvnw -Pnative,backend clean package -Dmaven.test.skip
+
+```
+
+Then, 2 packages are in `distribution/graal/dist`, The package named `apache-skywalking-apm-native-pre-bin.tar.gz` is unnecessary for most users. It is just an intermediate product for generating the final native-image program, used for testing purposes.
+The real outcome is the package named `apache-skywalking-apm-native-bin.tar.gz`, and followings are the introduction to its package structure.
Review Comment:
```suggestion
The package named `apache-skywalking-apm-native-bin.tar.gz` includes the final compiled native binary, relative configuration files, and booting shells. Read `Package Structure` doc for more details.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] yswdqz commented on pull request #11354: [Feature] Substitute classes to prepare for native-image.
Posted by "yswdqz (via GitHub)" <gi...@apache.org>.
yswdqz commented on PR #11354:
URL: https://github.com/apache/skywalking/pull/11354#issuecomment-1732839364
> It depends on whether the changes are needed at build time or runtime, i.e., @yswdqz without these changes in this PR, do we have errors at build time or runtime? If it's runtime errors, we can replace all the `*Provider` with a GraalVM-specific implementation without needing to use `unpack` and rewrite, for example, regarding the `EnvoyMetricReceiverProvider`, we can simply create an implementation `EnvoyMetricReceiverGraalProvider` that extends `EnvoyMetricReceiverProvider`, and change the name to `default-graalvm`, then modify `application.yaml`
>
> ```yaml
> envoy-metric:
> selector: ${SW_ENVOY_METRIC:default-graalvm}
> ```
>
> Although @wu-sheng had mentioned in elsewhere, I'd want to emphasize that, bear in mind if we want to make the native binaries first-class support, we can refactor the existing codes in many aspects to suit the native builds, as long as they are reasonable, instead of hacking the codes as if they are from third-party.
The error will occur at runtime. I will try to add a new provider and test it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] kezhenxu94 commented on a diff in pull request #11354: [Feature] Substitute classes to prepare for native-image.
Posted by "kezhenxu94 (via GitHub)" <gi...@apache.org>.
kezhenxu94 commented on code in PR #11354:
URL: https://github.com/apache/skywalking/pull/11354#discussion_r1337917700
##########
graal/skywalking-on-graal-compiling-plugin/src/main/java/org/apache/skywalking/graal/EnvoyGeneratorMojo.java:
##########
@@ -0,0 +1,66 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.graal;
+
+import org.apache.maven.plugin.AbstractMojo;
+import org.apache.maven.plugins.annotations.LifecyclePhase;
+import org.apache.maven.plugins.annotations.Mojo;
+
+import org.apache.maven.plugins.annotations.Parameter;
+import org.apache.maven.project.MavenProject;
+import org.apache.skywalking.oap.server.library.module.ModuleStartException;
+import org.apache.skywalking.oap.server.receiver.envoy.als.ServiceMetaInfo;
+import org.apache.skywalking.oap.server.receiver.envoy.als.mx.FieldsHelper;
+
+import java.io.File;
+import java.nio.file.Files;
+import java.nio.file.Paths;
+
+@Mojo(name = "envoy-generate", defaultPhase = LifecyclePhase.PROCESS_CLASSES)
Review Comment:
This step is optional as it adds nothing at build time, `FieldsHelper#init` is to populate the fields in memory, it doesn't generate any file and after the maven plugin exits all states are lost
##########
graal/dist-material/bin/oapService-native-jar.sh:
##########
@@ -0,0 +1,45 @@
+#!/usr/bin/env sh
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+
+PRG="$0"
Review Comment:
I believe this file will be totally different when the native binary is built
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on pull request #11354: [Feature] Substitute classes to prepare for native-image.
Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on PR #11354:
URL: https://github.com/apache/skywalking/pull/11354#issuecomment-1732828556
Yes, for provider, we could simply use a new provider.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on a diff in pull request #11354: [Feature] Substitute classes to prepare for native-image.
Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on code in PR #11354:
URL: https://github.com/apache/skywalking/pull/11354#discussion_r1335570425
##########
distribution/graal/oal-rt-graal-native/src/main/java/org/apache/skywalking/oal/rt/OALRuntime.java:
##########
@@ -0,0 +1,673 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oal.rt;
+
+import freemarker.template.Configuration;
+import freemarker.template.Version;
+import javassist.CannotCompileException;
+import javassist.ClassPool;
+import javassist.CtClass;
+import javassist.CtConstructor;
+import javassist.CtField;
+import javassist.CtNewConstructor;
+import javassist.CtNewMethod;
+import javassist.NotFoundException;
+import javassist.bytecode.AnnotationsAttribute;
+import javassist.bytecode.ClassFile;
+import javassist.bytecode.ConstPool;
+import javassist.bytecode.SignatureAttribute;
+import javassist.bytecode.annotation.Annotation;
+import javassist.bytecode.annotation.ClassMemberValue;
+import javassist.bytecode.annotation.IntegerMemberValue;
+import javassist.bytecode.annotation.StringMemberValue;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.commons.lang3.JavaVersion;
+import org.apache.commons.lang3.SystemUtils;
+import org.apache.skywalking.oal.rt.output.AllDispatcherContext;
+import org.apache.skywalking.oal.rt.output.DispatcherContext;
+import org.apache.skywalking.oal.rt.parser.AnalysisResult;
+import org.apache.skywalking.oal.rt.parser.OALScripts;
+import org.apache.skywalking.oal.rt.parser.ScriptParser;
+import org.apache.skywalking.oal.rt.parser.SourceColumn;
+import org.apache.skywalking.oap.server.core.analysis.DisableRegister;
+import org.apache.skywalking.oap.server.core.analysis.DispatcherDetectorListener;
+import org.apache.skywalking.oap.server.core.analysis.SourceDispatcher;
+import org.apache.skywalking.oap.server.core.analysis.Stream;
+import org.apache.skywalking.oap.server.core.analysis.StreamAnnotationListener;
+import org.apache.skywalking.oap.server.core.oal.rt.OALCompileException;
+import org.apache.skywalking.oap.server.core.oal.rt.OALDefine;
+import org.apache.skywalking.oap.server.core.oal.rt.OALEngine;
+import org.apache.skywalking.oap.server.core.source.oal.rt.dispatcher.DispatcherClassPackageHolder;
+import org.apache.skywalking.oap.server.core.source.oal.rt.metrics.MetricClassPackageHolder;
+import org.apache.skywalking.oap.server.core.source.oal.rt.metrics.builder.MetricBuilderClassPackageHolder;
+import org.apache.skywalking.oap.server.core.storage.StorageBuilderFactory;
+import org.apache.skywalking.oap.server.core.storage.StorageException;
+import org.apache.skywalking.oap.server.core.storage.annotation.BanyanDB;
+import org.apache.skywalking.oap.server.core.storage.annotation.Column;
+import org.apache.skywalking.oap.server.library.module.ModuleStartException;
+
+import java.io.DataOutputStream;
+import java.io.File;
+import java.io.FileNotFoundException;
+import java.io.FileOutputStream;
+import java.io.FileReader;
+import java.io.IOException;
+import java.io.Reader;
+import java.io.StringWriter;
+import java.net.URI;
+import java.net.URL;
+import java.nio.file.FileSystem;
+import java.nio.file.FileSystems;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.ArrayList;
+import java.util.Enumeration;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.Objects;
+
+/**
+ * The old logic of this class used runtime class generation, which is not supported by native-image.
+ * Therefore, in the new logic, class files are generated during the compilation process, see (@link org.apache.skywalking.graal.Generator) and loaded at runtime.
+ */
+@Slf4j
+public class OALRuntime implements OALEngine {
Review Comment:
For the implementation, could you share the workflow about when OAL runtime would generate the classes, and package into the jar files, then when how to use them in the runtime?
This part should be important for this PR. We should document this in the comment as well.
Also, this class seems still mixed the existing OAL runtime logic and part of changes, let's discuss about how to narrow down the case and focus the required change, polish the codes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] yswdqz commented on a diff in pull request #11354: [Feature] Substitute classes to prepare for native-image.
Posted by "yswdqz (via GitHub)" <gi...@apache.org>.
yswdqz commented on code in PR #11354:
URL: https://github.com/apache/skywalking/pull/11354#discussion_r1335692186
##########
distribution/graal/skywalking-generate-plugin/pom.xml:
##########
@@ -0,0 +1,120 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one or more
+ ~ contributor license agreements. See the NOTICE file distributed with
+ ~ this work for additional information regarding copyright ownership.
+ ~ The ASF licenses this file to You under the Apache License, Version 2.0
+ ~ (the "License"); you may not use this file except in compliance with
+ ~ the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing, software
+ ~ distributed under the License is distributed on an "AS IS" BASIS,
+ ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ ~ See the License for the specific language governing permissions and
+ ~ limitations under the License.
+ ~
+ -->
+
+<project xmlns="http://maven.apache.org/POM/4.0.0"
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <modelVersion>4.0.0</modelVersion>
+ <parent>
+ <groupId>org.apache.skywalking</groupId>
+ <artifactId>graal</artifactId>
+ <version>9.7.0-SNAPSHOT</version>
+ </parent>
+
+ <packaging>maven-plugin</packaging>
+
+ <artifactId>skywalking-generate-plugin</artifactId>
Review Comment:
I think perhaps it shouldn't be changed, because there might be more things that need to be generated during construction in the future, and we shouldn't create many plugins to generate different content.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on a diff in pull request #11354: [Feature] Substitute classes to prepare for native-image.
Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on code in PR #11354:
URL: https://github.com/apache/skywalking/pull/11354#discussion_r1336535893
##########
distribution/graal/apache-skywalking-native-apm/pom.xml:
##########
@@ -0,0 +1,107 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one or more
+ ~ contributor license agreements. See the NOTICE file distributed with
+ ~ this work for additional information regarding copyright ownership.
+ ~ The ASF licenses this file to You under the Apache License, Version 2.0
+ ~ (the "License"); you may not use this file except in compliance with
+ ~ the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing, software
+ ~ distributed under the License is distributed on an "AS IS" BASIS,
+ ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ ~ See the License for the specific language governing permissions and
+ ~ limitations under the License.
+ ~
+ -->
+
+<project xmlns="http://maven.apache.org/POM/4.0.0"
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <modelVersion>4.0.0</modelVersion>
+ <parent>
+ <groupId>org.apache.skywalking</groupId>
+ <artifactId>graal</artifactId>
+ <version>9.7.0-SNAPSHOT</version>
+ </parent>
+
+ <artifactId>apache-skywalking-native-apm</artifactId>
+
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.skywalking</groupId>
+ <artifactId>main</artifactId>
+ <version>${project.version}</version>
+ </dependency>
+ </dependencies>
+
+ <profiles>
+ <profile>
+ <id>native</id>
+ <build>
+ <plugins>
+ <plugin>
+ <groupId>org.apache.maven.plugins</groupId>
+ <artifactId>maven-dependency-plugin</artifactId>
+ <executions>
+ <execution>
+ <id>copy-dependencies</id>
+ <phase>package</phase>
+ <goals>
+ <goal>copy-dependencies</goal>
+ </goals>
+ <configuration>
+ <includeScope>runtime</includeScope>
+ <outputDirectory>${project.build.directory}/oap-libs</outputDirectory>
+ </configuration>
+ </execution>
+ </executions>
+ </plugin>
+ <plugin>
+ <artifactId>maven-assembly-plugin</artifactId>
+ <executions>
+ <execution>
+ <id>dist</id>
+ <phase>package</phase>
+ <goals>
+ <goal>single</goal>
+ </goals>
+ <configuration>
+ <finalName>apache-skywalking-apm-native-bin</finalName>
Review Comment:
```suggestion
<finalName>apache-skywalking-apm-pre-native-bin</finalName>
```
Let's use pre-native, the final tar ball should be called native. This is only a tar file including jars for preparation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on a diff in pull request #11354: [Feature] Substitute classes to prepare for native-image.
Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on code in PR #11354:
URL: https://github.com/apache/skywalking/pull/11354#discussion_r1335567385
##########
distribution/graal/oal-rt-graal-native/src/main/java/org/apache/skywalking/oal/rt/OALRuntime.java:
##########
@@ -0,0 +1,673 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oal.rt;
+
+import freemarker.template.Configuration;
+import freemarker.template.Version;
+import javassist.CannotCompileException;
+import javassist.ClassPool;
+import javassist.CtClass;
+import javassist.CtConstructor;
+import javassist.CtField;
+import javassist.CtNewConstructor;
+import javassist.CtNewMethod;
+import javassist.NotFoundException;
+import javassist.bytecode.AnnotationsAttribute;
+import javassist.bytecode.ClassFile;
+import javassist.bytecode.ConstPool;
+import javassist.bytecode.SignatureAttribute;
+import javassist.bytecode.annotation.Annotation;
+import javassist.bytecode.annotation.ClassMemberValue;
+import javassist.bytecode.annotation.IntegerMemberValue;
+import javassist.bytecode.annotation.StringMemberValue;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.commons.lang3.JavaVersion;
+import org.apache.commons.lang3.SystemUtils;
+import org.apache.skywalking.oal.rt.output.AllDispatcherContext;
+import org.apache.skywalking.oal.rt.output.DispatcherContext;
+import org.apache.skywalking.oal.rt.parser.AnalysisResult;
+import org.apache.skywalking.oal.rt.parser.OALScripts;
+import org.apache.skywalking.oal.rt.parser.ScriptParser;
+import org.apache.skywalking.oal.rt.parser.SourceColumn;
+import org.apache.skywalking.oap.server.core.analysis.DisableRegister;
+import org.apache.skywalking.oap.server.core.analysis.DispatcherDetectorListener;
+import org.apache.skywalking.oap.server.core.analysis.SourceDispatcher;
+import org.apache.skywalking.oap.server.core.analysis.Stream;
+import org.apache.skywalking.oap.server.core.analysis.StreamAnnotationListener;
+import org.apache.skywalking.oap.server.core.oal.rt.OALCompileException;
+import org.apache.skywalking.oap.server.core.oal.rt.OALDefine;
+import org.apache.skywalking.oap.server.core.oal.rt.OALEngine;
+import org.apache.skywalking.oap.server.core.source.oal.rt.dispatcher.DispatcherClassPackageHolder;
+import org.apache.skywalking.oap.server.core.source.oal.rt.metrics.MetricClassPackageHolder;
+import org.apache.skywalking.oap.server.core.source.oal.rt.metrics.builder.MetricBuilderClassPackageHolder;
+import org.apache.skywalking.oap.server.core.storage.StorageBuilderFactory;
+import org.apache.skywalking.oap.server.core.storage.StorageException;
+import org.apache.skywalking.oap.server.core.storage.annotation.BanyanDB;
+import org.apache.skywalking.oap.server.core.storage.annotation.Column;
+import org.apache.skywalking.oap.server.library.module.ModuleStartException;
+
+import java.io.DataOutputStream;
+import java.io.File;
+import java.io.FileNotFoundException;
+import java.io.FileOutputStream;
+import java.io.FileReader;
+import java.io.IOException;
+import java.io.Reader;
+import java.io.StringWriter;
+import java.net.URI;
+import java.net.URL;
+import java.nio.file.FileSystem;
+import java.nio.file.FileSystems;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.ArrayList;
+import java.util.Enumeration;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.Objects;
+
+/**
+ * The old logic of this class used runtime class generation, which is not supported by native-image.
+ * Therefore, in the new logic, class files are generated during the compilation process, see (@link org.apache.skywalking.graal.Generator) and loaded at runtime.
Review Comment:
Same suggestion here, please reform the comments.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on a diff in pull request #11354: [Feature] Substitute classes to prepare for native-image.
Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on code in PR #11354:
URL: https://github.com/apache/skywalking/pull/11354#discussion_r1335566754
##########
distribution/graal/meter-analyzer-graal-native/src/main/java/org/apache/skywalking/oap/meter/analyzer/Analyzer.java:
##########
@@ -0,0 +1,47 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.meter.analyzer;
+
+import com.google.common.collect.ImmutableMap;
+import io.vavr.Tuple;
+import io.vavr.Tuple2;
+import org.apache.skywalking.oap.meter.analyzer.dsl.SampleFamily;
+import org.apache.skywalking.oap.server.core.analysis.meter.MeterSystem;
+
+/**
+ * The old logic of this class used Groovy;
+ * however, GraalVM does not support it well, so it has been replaced with an empty implementation.
+ * In the future, other methods will be used to implement this class.
Review Comment:
```suggestion
* TODO, disabled MAL kernel due to the Graal limitation on Groovy stack.
* SkyWalking will provide an alternative solution to replace the groovy stack based solution in the near future.
```
When you try to put a comment from an official perspective, please be certain way out.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on a diff in pull request #11354: [Feature] Substitute classes to prepare for native-image.
Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on code in PR #11354:
URL: https://github.com/apache/skywalking/pull/11354#discussion_r1335585809
##########
distribution/graal/oal-rt-graal-native/src/main/java/org/apache/skywalking/oal/rt/OALRuntime.java:
##########
@@ -0,0 +1,673 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oal.rt;
+
+import freemarker.template.Configuration;
+import freemarker.template.Version;
+import javassist.CannotCompileException;
+import javassist.ClassPool;
+import javassist.CtClass;
+import javassist.CtConstructor;
+import javassist.CtField;
+import javassist.CtNewConstructor;
+import javassist.CtNewMethod;
+import javassist.NotFoundException;
+import javassist.bytecode.AnnotationsAttribute;
+import javassist.bytecode.ClassFile;
+import javassist.bytecode.ConstPool;
+import javassist.bytecode.SignatureAttribute;
+import javassist.bytecode.annotation.Annotation;
+import javassist.bytecode.annotation.ClassMemberValue;
+import javassist.bytecode.annotation.IntegerMemberValue;
+import javassist.bytecode.annotation.StringMemberValue;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.commons.lang3.JavaVersion;
+import org.apache.commons.lang3.SystemUtils;
+import org.apache.skywalking.oal.rt.output.AllDispatcherContext;
+import org.apache.skywalking.oal.rt.output.DispatcherContext;
+import org.apache.skywalking.oal.rt.parser.AnalysisResult;
+import org.apache.skywalking.oal.rt.parser.OALScripts;
+import org.apache.skywalking.oal.rt.parser.ScriptParser;
+import org.apache.skywalking.oal.rt.parser.SourceColumn;
+import org.apache.skywalking.oap.server.core.analysis.DisableRegister;
+import org.apache.skywalking.oap.server.core.analysis.DispatcherDetectorListener;
+import org.apache.skywalking.oap.server.core.analysis.SourceDispatcher;
+import org.apache.skywalking.oap.server.core.analysis.Stream;
+import org.apache.skywalking.oap.server.core.analysis.StreamAnnotationListener;
+import org.apache.skywalking.oap.server.core.oal.rt.OALCompileException;
+import org.apache.skywalking.oap.server.core.oal.rt.OALDefine;
+import org.apache.skywalking.oap.server.core.oal.rt.OALEngine;
+import org.apache.skywalking.oap.server.core.source.oal.rt.dispatcher.DispatcherClassPackageHolder;
+import org.apache.skywalking.oap.server.core.source.oal.rt.metrics.MetricClassPackageHolder;
+import org.apache.skywalking.oap.server.core.source.oal.rt.metrics.builder.MetricBuilderClassPackageHolder;
+import org.apache.skywalking.oap.server.core.storage.StorageBuilderFactory;
+import org.apache.skywalking.oap.server.core.storage.StorageException;
+import org.apache.skywalking.oap.server.core.storage.annotation.BanyanDB;
+import org.apache.skywalking.oap.server.core.storage.annotation.Column;
+import org.apache.skywalking.oap.server.library.module.ModuleStartException;
+
+import java.io.DataOutputStream;
+import java.io.File;
+import java.io.FileNotFoundException;
+import java.io.FileOutputStream;
+import java.io.FileReader;
+import java.io.IOException;
+import java.io.Reader;
+import java.io.StringWriter;
+import java.net.URI;
+import java.net.URL;
+import java.nio.file.FileSystem;
+import java.nio.file.FileSystems;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.ArrayList;
+import java.util.Enumeration;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.Objects;
+
+/**
+ * The old logic of this class used runtime class generation, which is not supported by native-image.
+ * Therefore, in the new logic, class files are generated during the compilation process, see (@link org.apache.skywalking.graal.Generator) and loaded at runtime.
+ */
+@Slf4j
+public class OALRuntime implements OALEngine {
Review Comment:
> I will try departing generateOALClassFIles later, I previously placed it in OALRuntime to reuse the existing API, but it seems indeed unnecessary, and indeed, the logic has become less clear.
We could and should reuse the logic, but I don't want to copy the existing codes in the patch codes. You could consider this as a guidance principle for all my given comments.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng merged pull request #11354: [Feature] Substitute classes to prepare for native-image.
Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng merged PR #11354:
URL: https://github.com/apache/skywalking/pull/11354
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] yswdqz commented on a diff in pull request #11354: [Feature] Substitute classes to prepare for native-image.
Posted by "yswdqz (via GitHub)" <gi...@apache.org>.
yswdqz commented on code in PR #11354:
URL: https://github.com/apache/skywalking/pull/11354#discussion_r1335627167
##########
distribution/graal/skywalking-generate-plugin/pom.xml:
##########
@@ -0,0 +1,120 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one or more
+ ~ contributor license agreements. See the NOTICE file distributed with
+ ~ this work for additional information regarding copyright ownership.
+ ~ The ASF licenses this file to You under the Apache License, Version 2.0
+ ~ (the "License"); you may not use this file except in compliance with
+ ~ the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing, software
+ ~ distributed under the License is distributed on an "AS IS" BASIS,
+ ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ ~ See the License for the specific language governing permissions and
+ ~ limitations under the License.
+ ~
+ -->
+
+<project xmlns="http://maven.apache.org/POM/4.0.0"
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <modelVersion>4.0.0</modelVersion>
+ <parent>
+ <groupId>org.apache.skywalking</groupId>
+ <artifactId>graal</artifactId>
+ <version>9.7.0-SNAPSHOT</version>
+ </parent>
+
+ <packaging>maven-plugin</packaging>
+
+ <artifactId>skywalking-generate-plugin</artifactId>
Review Comment:
Sorry,
```
FieldsHelper.SINGLETON.init("metadata-service-mapping.yaml", ServiceMetaInfo.class);
```
is also executed by this plugin.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on a diff in pull request #11354: [Feature] Substitute classes to prepare for native-image.
Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on code in PR #11354:
URL: https://github.com/apache/skywalking/pull/11354#discussion_r1336513350
##########
docs/en/setup/backend/backend-graalvm.md:
##########
@@ -0,0 +1,45 @@
+# How to use SkyWalking with GraalVM native-image (Experimental)
+
+## Native-Image
+Native Image is a technology to ahead-of-time compile Java code to a standalone executable, called a native image.
+This executable includes the application classes, classes from its dependencies, runtime library classes,
+and statically linked native code from JDK. It does not run on the Java VM, but includes necessary components like memory management, thread scheduling, and so on from a different runtime system, called “Substrate VM”.
+Substrate VM is the name for the runtime components (like the deoptimizer, garbage collector, thread scheduling etc.).
+The resulting program has faster startup time and lower runtime memory overhead compared to a JVM.
+
+SkyWalking currently offers support for native-image. However, please note that the OAP started in this manner does not have the same functionality as the regular OAP, and some features are not yet supported.
Review Comment:
```suggestion
SkyWalking currently offers support for OAP servers running as native-image. However, please note that the OAP started in this manner does not have the same functionality as the regular OAP, and some features are not yet supported.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] yswdqz commented on a diff in pull request #11354: [Feature] Substitute classes to prepare for native-image.
Posted by "yswdqz (via GitHub)" <gi...@apache.org>.
yswdqz commented on code in PR #11354:
URL: https://github.com/apache/skywalking/pull/11354#discussion_r1336612604
##########
distribution/graal/dist-material/bin/exportEnv.sh:
##########
@@ -19,4 +19,5 @@
export SW_CONFIG_PATHS=${OAP_HOME}/config
export SW_ENVOY_METRIC=default-graalvm
-export SW_LOG_ANALYZER=default-graalvm
\ No newline at end of file
+export SW_LOG_ANALYZER=default-graalvm
+export SW_OAL_ENGINE_DEBUG=true
Review Comment:
Should be runtime.
My fault, we do not need to set SW_OAL_ENGINE_DEBUG
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] yswdqz commented on a diff in pull request #11354: [Feature] Substitute classes to prepare for native-image.
Posted by "yswdqz (via GitHub)" <gi...@apache.org>.
yswdqz commented on code in PR #11354:
URL: https://github.com/apache/skywalking/pull/11354#discussion_r1336613841
##########
pom.xml:
##########
@@ -33,6 +33,7 @@
<modules>
<module>apm-protocol</module>
+ <module>distribution</module>
Review Comment:
I included the distribution module because I think that perhaps one day we may need more than just one packaging way like native-image. We might have many other packaging ways, and they should be parallel to the graal module.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] yswdqz commented on a diff in pull request #11354: [Feature] Substitute classes to prepare for native-image.
Posted by "yswdqz (via GitHub)" <gi...@apache.org>.
yswdqz commented on code in PR #11354:
URL: https://github.com/apache/skywalking/pull/11354#discussion_r1335621199
##########
distribution/graal/skywalking-generate-plugin/pom.xml:
##########
@@ -0,0 +1,120 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one or more
+ ~ contributor license agreements. See the NOTICE file distributed with
+ ~ this work for additional information regarding copyright ownership.
+ ~ The ASF licenses this file to You under the Apache License, Version 2.0
+ ~ (the "License"); you may not use this file except in compliance with
+ ~ the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing, software
+ ~ distributed under the License is distributed on an "AS IS" BASIS,
+ ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ ~ See the License for the specific language governing permissions and
+ ~ limitations under the License.
+ ~
+ -->
+
+<project xmlns="http://maven.apache.org/POM/4.0.0"
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <modelVersion>4.0.0</modelVersion>
+ <parent>
+ <groupId>org.apache.skywalking</groupId>
+ <artifactId>graal</artifactId>
+ <version>9.7.0-SNAPSHOT</version>
+ </parent>
+
+ <packaging>maven-plugin</packaging>
+
+ <artifactId>skywalking-generate-plugin</artifactId>
Review Comment:
Yes, but not only oal code, also envoy-metrics.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on a diff in pull request #11354: [Feature] Substitute classes to prepare for native-image.
Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on code in PR #11354:
URL: https://github.com/apache/skywalking/pull/11354#discussion_r1335619237
##########
distribution/graal/skywalking-generate-plugin/pom.xml:
##########
@@ -0,0 +1,120 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one or more
+ ~ contributor license agreements. See the NOTICE file distributed with
+ ~ this work for additional information regarding copyright ownership.
+ ~ The ASF licenses this file to You under the Apache License, Version 2.0
+ ~ (the "License"); you may not use this file except in compliance with
+ ~ the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing, software
+ ~ distributed under the License is distributed on an "AS IS" BASIS,
+ ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ ~ See the License for the specific language governing permissions and
+ ~ limitations under the License.
+ ~
+ -->
+
+<project xmlns="http://maven.apache.org/POM/4.0.0"
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <modelVersion>4.0.0</modelVersion>
+ <parent>
+ <groupId>org.apache.skywalking</groupId>
+ <artifactId>graal</artifactId>
+ <version>9.7.0-SNAPSHOT</version>
+ </parent>
+
+ <packaging>maven-plugin</packaging>
+
+ <artifactId>skywalking-generate-plugin</artifactId>
Review Comment:
This should be renamed to `oal-on-graal-compiling-plugin`. And all existing codes in this are not runtime relative, right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on a diff in pull request #11354: [Feature] Substitute classes to prepare for native-image.
Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on code in PR #11354:
URL: https://github.com/apache/skywalking/pull/11354#discussion_r1336508552
##########
docs/en/setup/backend/backend-graalvm.md:
##########
@@ -0,0 +1,45 @@
+# How to use SkyWalking with GraalVM native-image (Experimental)
+
+## Native-Image
+Native Image is a technology to ahead-of-time compile Java code to a standalone executable, called a native image.
+This executable includes the application classes, classes from its dependencies, runtime library classes,
+and statically linked native code from JDK. It does not run on the Java VM, but includes necessary components like memory management, thread scheduling, and so on from a different runtime system, called “Substrate VM”.
+Substrate VM is the name for the runtime components (like the deoptimizer, garbage collector, thread scheduling etc.).
+The resulting program has faster startup time and lower runtime memory overhead compared to a JVM.
+
+SkyWalking currently offers support for native-image. However, please note that the OAP started in this manner does not have the same functionality as the regular OAP, and some features are not yet supported.
+
+## Compile Guide
+Notice: If you are not familiar with the compilation process, please read [How-to-build](https://skywalking.apache.org/docs/main/next/en/guides/how-to-build/) first.
+
+The native-image compilation is not enabled by default. To enable it, we need to specify `-p native` during compilation, such as:
+
+```shell
+./mvnw -P native clean package -Dmaven.test.skip
+```
Review Comment:
```suggestion
The native-image compilation is not enabled by default. To enable it, we need to activate `native` profile during compilation, such as:
```shell
./mvnw -Pnative clean package -Dmaven.test.skip
```
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on a diff in pull request #11354: [Feature] Substitute classes to prepare for native-image.
Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on code in PR #11354:
URL: https://github.com/apache/skywalking/pull/11354#discussion_r1337135332
##########
graal/dist-material/bin/oapService-native-jar.sh:
##########
@@ -0,0 +1,45 @@
+#!/usr/bin/env sh
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+
+PRG="$0"
Review Comment:
@kezhenxu94 Could you review the shell?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on pull request #11354: [Feature] Substitute classes to prepare for native-image.
Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on PR #11354:
URL: https://github.com/apache/skywalking/pull/11354#issuecomment-1734700051
<img width="325" alt="image" src="https://github.com/apache/skywalking/assets/5441976/590ee553-e5a1-4431-80d7-2c92896acc11">
I am not sure what is the root cause yet, but all modules are not detected as a maven module. So you wouldn't be able to search classes in the patch. @yswdqz Please fix this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on a diff in pull request #11354: [Feature] Substitute classes to prepare for native-image.
Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on code in PR #11354:
URL: https://github.com/apache/skywalking/pull/11354#discussion_r1335266005
##########
distribution/graal/envoy-metrics-receiver-plugin-graal-native/src/main/java/org/apache/skywalking/oap/server/receiver/envoy/EnvoyMetricReceiverProvider.java:
##########
@@ -0,0 +1,109 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.server.receiver.envoy;
+
+import org.apache.skywalking.aop.server.receiver.mesh.MeshReceiverModule;
+import org.apache.skywalking.oap.server.core.CoreModule;
+import org.apache.skywalking.oap.server.core.oal.rt.OALEngineLoaderService;
+import org.apache.skywalking.oap.server.core.server.GRPCHandlerRegister;
+import org.apache.skywalking.oap.server.library.module.ModuleDefine;
+import org.apache.skywalking.oap.server.library.module.ModuleProvider;
+import org.apache.skywalking.oap.server.library.module.ModuleStartException;
+import org.apache.skywalking.oap.server.library.module.ServiceNotProvidedException;
+import org.apache.skywalking.oap.server.receiver.sharing.server.SharingServerModule;
+import org.apache.skywalking.oap.server.telemetry.TelemetryModule;
+
+public class EnvoyMetricReceiverProvider extends ModuleProvider {
+ private EnvoyMetricReceiverConfig config;
+
+ protected String fieldMappingFile = "metadata-service-mapping.yaml";
+
+ @Override
+ public String name() {
+ return "default";
+ }
+
+ @Override
+ public Class<? extends ModuleDefine> module() {
+ return EnvoyMetricReceiverModule.class;
+ }
+
+ @Override
+ public ConfigCreator newConfigCreator() {
+ return new ConfigCreator<EnvoyMetricReceiverConfig>() {
+ @Override
+ public Class type() {
+ return EnvoyMetricReceiverConfig.class;
+ }
+
+ @Override
+ public void onInitialized(final EnvoyMetricReceiverConfig initialized) {
+ config = initialized;
+ }
+ };
+ }
+
+ // change
+ /**
+ * do nothing for now.
+ */
+ @Override
+ public void prepare() throws ServiceNotProvidedException, ModuleStartException {
+
+ }
+ // change end
Review Comment:
Suggestions.
1. Put all changes at the end of the file, which would be more clear.
2. As you had removed the old logic, where are they now?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on a diff in pull request #11354: [Feature] Substitute classes to prepare for native-image.
Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on code in PR #11354:
URL: https://github.com/apache/skywalking/pull/11354#discussion_r1336509291
##########
docs/en/setup/backend/backend-graalvm.md:
##########
@@ -0,0 +1,45 @@
+# How to use SkyWalking with GraalVM native-image (Experimental)
Review Comment:
```suggestion
# GraalVM native-image for SkyWalking (Experimental)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] yswdqz commented on pull request #11354: [Feature] Substitute classes to prepare for native-image.
Posted by "yswdqz (via GitHub)" <gi...@apache.org>.
yswdqz commented on PR #11354:
URL: https://github.com/apache/skywalking/pull/11354#issuecomment-1732881107
My test was successful, I will submit the related commit shortly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on a diff in pull request #11354: [Feature] Substitute classes to prepare for native-image.
Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on code in PR #11354:
URL: https://github.com/apache/skywalking/pull/11354#discussion_r1335625267
##########
distribution/graal/skywalking-generate-plugin/pom.xml:
##########
@@ -0,0 +1,120 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one or more
+ ~ contributor license agreements. See the NOTICE file distributed with
+ ~ this work for additional information regarding copyright ownership.
+ ~ The ASF licenses this file to You under the Apache License, Version 2.0
+ ~ (the "License"); you may not use this file except in compliance with
+ ~ the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing, software
+ ~ distributed under the License is distributed on an "AS IS" BASIS,
+ ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ ~ See the License for the specific language governing permissions and
+ ~ limitations under the License.
+ ~
+ -->
+
+<project xmlns="http://maven.apache.org/POM/4.0.0"
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <modelVersion>4.0.0</modelVersion>
+ <parent>
+ <groupId>org.apache.skywalking</groupId>
+ <artifactId>graal</artifactId>
+ <version>9.7.0-SNAPSHOT</version>
+ </parent>
+
+ <packaging>maven-plugin</packaging>
+
+ <artifactId>skywalking-generate-plugin</artifactId>
Review Comment:
What do you mean envoy-metrics? I think you misuse terms. In the SkyWalking OAP, metrics are only generated by OAL or MAL, regardless of the original data sources.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on a diff in pull request #11354: [Feature] Substitute classes to prepare for native-image.
Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on code in PR #11354:
URL: https://github.com/apache/skywalking/pull/11354#discussion_r1336997435
##########
pom.xml:
##########
@@ -33,6 +33,7 @@
<modules>
<module>apm-protocol</module>
+ <module>distribution</module>
Review Comment:
There is no other package mode. Let's not design things unexisting, OK?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on a diff in pull request #11354: [Feature] Substitute classes to prepare for native-image.
Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on code in PR #11354:
URL: https://github.com/apache/skywalking/pull/11354#discussion_r1337053652
##########
docs/en/setup/backend/backend-graalvm.md:
##########
@@ -0,0 +1,45 @@
+# How to use SkyWalking with GraalVM native-image (Experimental)
+
+## Native-Image
+Native Image is a technology to ahead-of-time compile Java code to a standalone executable, called a native image.
+This executable includes the application classes, classes from its dependencies, runtime library classes,
+and statically linked native code from JDK. It does not run on the Java VM, but includes necessary components like memory management, thread scheduling, and so on from a different runtime system, called “Substrate VM”.
+Substrate VM is the name for the runtime components (like the deoptimizer, garbage collector, thread scheduling etc.).
+The resulting program has faster startup time and lower runtime memory overhead compared to a JVM.
+
+SkyWalking currently offers support for native-image. However, please note that the OAP started in this manner does not have the same functionality as the regular OAP, and some features are not yet supported.
+
+## Compile Guide
+Notice: If you are not familiar with the compilation process, please read [How-to-build](https://skywalking.apache.org/docs/main/next/en/guides/how-to-build/) first.
+
+The native-image compilation is not enabled by default. To enable it, we need to specify `-p native` during compilation, such as:
+
+```shell
+./mvnw -P native clean package -Dmaven.test.skip
+```
+
+Then, All packages are in `distribution/graal/dist` (.tar.gz for Linux and .zip for Windows).
+
+## Package Structure
+
+SkyWalking’s native-image distribution package consists of the following parts:
+
+* bin/cmd scripts: Located in the /bin folder. Includes startup Linux shell and Windows cmd scripts for the backend server.
+
+* Backend config: Located in the /config folder. Includes settings files of the backend. Running native-image does not require additional configuration, so you can refer to [backend-setup](https://skywalking.apache.org/docs/main/next/en/setup/backend/backend-setup/) to learn how to configure it.
+
+* Native image executable: Located in /image folder. It can be run directly, but it is not recommended to do so, as the absence of some environment variables can lead to startup failure.
+
+## How To Use
+By executing following:
+
+```shell
+./bin/oapService-native.sh
+```
+we can successfully start SkyWalking-oap.
+
+## Problems
+With native-image, some features are not yet supported.
+
+1. [LAL](https://skywalking.apache.org/docs/main/next/en/concepts-and-designs/lal/), [MAL](https://skywalking.apache.org/docs/main/next/en/concepts-and-designs/mal/), and some other features related to them are not supported at the moment.
+2. The configuration file for [OAL](https://skywalking.apache.org/docs/main/next/en/concepts-and-designs/oal/) can only be provided during the compilation period, meaning that it cannot be added or modified during runtime.
Review Comment:
```suggestion
2. The [OAL](https://skywalking.apache.org/docs/main/next/en/concepts-and-designs/oal/) files are used in the compiling stage, which means that users would not see these files inside the native image package, and can't change it. Consider recompiling and packaging from the source codes including your OAL file changes.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] yswdqz commented on a diff in pull request #11354: [Feature] Substitute classes to prepare for native-image.
Posted by "yswdqz (via GitHub)" <gi...@apache.org>.
yswdqz commented on code in PR #11354:
URL: https://github.com/apache/skywalking/pull/11354#discussion_r1337227242
##########
graal/graal-server-starter/pom.xml:
##########
@@ -0,0 +1,133 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one or more
+ ~ contributor license agreements. See the NOTICE file distributed with
+ ~ this work for additional information regarding copyright ownership.
+ ~ The ASF licenses this file to You under the Apache License, Version 2.0
+ ~ (the "License"); you may not use this file except in compliance with
+ ~ the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing, software
+ ~ distributed under the License is distributed on an "AS IS" BASIS,
+ ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ ~ See the License for the specific language governing permissions and
+ ~ limitations under the License.
+ ~
+ -->
+
+<project xmlns="http://maven.apache.org/POM/4.0.0"
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <modelVersion>4.0.0</modelVersion>
+ <parent>
+ <groupId>org.apache.skywalking</groupId>
+ <artifactId>graal</artifactId>
+ <version>9.7.0-SNAPSHOT</version>
+ </parent>
+ <artifactId>graal-server-starter</artifactId>
+
+ <properties>
+ <image.name>skywalking-oap</image.name>
+ </properties>
+
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.skywalking</groupId>
+ <artifactId>server-starter</artifactId>
+ <version>${project.version}</version>
+ <exclusions>
+ <exclusion>
+ <groupId>org.apache.skywalking</groupId>
+ <artifactId>meter-analyzer</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>org.apache.skywalking</groupId>
+ <artifactId>log-analyzer</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>org.apache.skywalking</groupId>
+ <artifactId>server-core</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>org.apache.skywalking</groupId>
+ <artifactId>oal-rt</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>org.apache.skywalking</groupId>
+ <artifactId>agent-analyzer</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>org.apache.skywalking</groupId>
+ <artifactId>library-util</artifactId>
+ </exclusion>
+ </exclusions>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.skywalking</groupId>
+ <artifactId>envoy-metrics-receiver-plugin-graal-native</artifactId>
+ <version>${project.version}</version>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.skywalking</groupId>
+ <artifactId>library-util-graal-native</artifactId>
+ <version>${project.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.skywalking</groupId>
+ <artifactId>meter-analyzer-graal-native</artifactId>
+ <version>${project.version}</version>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.skywalking</groupId>
+ <artifactId>server-core-graal-native</artifactId>
+ <version>${project.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.skywalking</groupId>
+ <artifactId>log-analyzer-graal-native</artifactId>
+ <version>${project.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.skywalking</groupId>
+ <artifactId>oal-rt-graal-native</artifactId>
+ <version>${project.version}</version>
+ </dependency>
+
+ <dependency>
+ <groupId>org.graalvm.nativeimage</groupId>
+ <artifactId>svm</artifactId>
+ <scope>provided</scope>
+ <version>22.3.1</version>
+ </dependency>
Review Comment:
My fault, I should remove it when I am not using GraalVM substitutions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on pull request #11354: [Feature] Substitute classes to prepare for native-image.
Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on PR #11354:
URL: https://github.com/apache/skywalking/pull/11354#issuecomment-1732714258
As reading the codes of this PR, most codes are copied from the existing codes due to the maven `unpack` solution. This is practical, meanwhile, a little annoying, and this harms the maintainability. We will never know if a patch for the codes would be misaligned.
So, I want to ask whether it is possible to use `inherit` + `shade` + `unpack` to add a little complexity to the maven pom.xml.
For example, `EnvoyMetricReceiverProvider` should be unpacked and changed, so we can do
1. Write a new class called `EnvoyMetricReceiverProviderGraal` to inherit `EnvoyMetricReceiverProvider` class and override the method. We also could change the original class to create new methods for overriding simpler.
2. Use `shade` plugin to rename `EnvoyMetricReceiverProvider` to `EnvoyMetricReceiverProviderOrigin`, and `EnvoyMetricReceiverProviderGraal` to `EnvoyMetricReceiverProvider` with unpack as well.
@yswdqz Is this possible? We would expect you could reduce 2k LOCs to less than 500, I hope. And mostly, there would be no many duplicate codes. cc @kezhenxu94
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on a diff in pull request #11354: [Feature] Substitute classes to prepare for native-image.
Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on code in PR #11354:
URL: https://github.com/apache/skywalking/pull/11354#discussion_r1335899439
##########
distribution/graal/skywalking-generate-plugin/pom.xml:
##########
@@ -0,0 +1,120 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one or more
+ ~ contributor license agreements. See the NOTICE file distributed with
+ ~ this work for additional information regarding copyright ownership.
+ ~ The ASF licenses this file to You under the Apache License, Version 2.0
+ ~ (the "License"); you may not use this file except in compliance with
+ ~ the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing, software
+ ~ distributed under the License is distributed on an "AS IS" BASIS,
+ ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ ~ See the License for the specific language governing permissions and
+ ~ limitations under the License.
+ ~
+ -->
+
+<project xmlns="http://maven.apache.org/POM/4.0.0"
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <modelVersion>4.0.0</modelVersion>
+ <parent>
+ <groupId>org.apache.skywalking</groupId>
+ <artifactId>graal</artifactId>
+ <version>9.7.0-SNAPSHOT</version>
+ </parent>
+
+ <packaging>maven-plugin</packaging>
+
+ <artifactId>skywalking-generate-plugin</artifactId>
Review Comment:
This has not relationship with the OAL. So it should be in two places.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on a diff in pull request #11354: [Feature] Substitute classes to prepare for native-image.
Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on code in PR #11354:
URL: https://github.com/apache/skywalking/pull/11354#discussion_r1336537770
##########
docs/en/setup/backend/backend-graalvm.md:
##########
@@ -0,0 +1,45 @@
+# How to use SkyWalking with GraalVM native-image (Experimental)
+
+## Native-Image
+Native Image is a technology to ahead-of-time compile Java code to a standalone executable, called a native image.
+This executable includes the application classes, classes from its dependencies, runtime library classes,
+and statically linked native code from JDK. It does not run on the Java VM, but includes necessary components like memory management, thread scheduling, and so on from a different runtime system, called “Substrate VM”.
+Substrate VM is the name for the runtime components (like the deoptimizer, garbage collector, thread scheduling etc.).
+The resulting program has faster startup time and lower runtime memory overhead compared to a JVM.
+
+SkyWalking currently offers support for native-image. However, please note that the OAP started in this manner does not have the same functionality as the regular OAP, and some features are not yet supported.
+
+## Compile Guide
+Notice: If you are not familiar with the compilation process, please read [How-to-build](https://skywalking.apache.org/docs/main/next/en/guides/how-to-build/) first.
+
+The native-image compilation is not enabled by default. To enable it, we need to specify `-p native` during compilation, such as:
+
+```shell
+./mvnw -P native clean package -Dmaven.test.skip
+```
+
+Then, All packages are in `distribution/graal/dist` (.tar.gz for Linux and .zip for Windows).
Review Comment:
I think we should be clear about pre-native and native tarballs
And, I don't see anything about .zip and windows. Please be careful about what you are writing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] kezhenxu94 commented on pull request #11354: [Feature] Substitute classes to prepare for native-image.
Posted by "kezhenxu94 (via GitHub)" <gi...@apache.org>.
kezhenxu94 commented on PR #11354:
URL: https://github.com/apache/skywalking/pull/11354#issuecomment-1732808488
> As reading the codes of this PR, most codes are copied from the existing codes due to the maven `unpack` solution. This is practical, meanwhile, a little annoying, and this harms the maintainability. We will never know if a patch for the codes would be misaligned.
>
> So, I want to ask whether it is possible to use `inherit` + `shade` + `unpack` to add a little complexity to the maven pom.xml.
>
> For example, `EnvoyMetricReceiverProvider` should be unpacked and changed, so we can do
>
> 1. Write a new class called `EnvoyMetricReceiverProviderGraal` to inherit `EnvoyMetricReceiverProvider` class and override the method. We also could change the original class to create new methods for overriding simpler.
> 2. Use `shade` plugin to rename `EnvoyMetricReceiverProvider` to `EnvoyMetricReceiverProviderOrigin`, and `EnvoyMetricReceiverProviderGraal` to `EnvoyMetricReceiverProvider` with unpack as well.
It depends on whether the changes are needed at build time or runtime, i.e., @yswdqz without these changes in this PR, do we have errors at build time or runtime? If it's runtime errors, we can replace all the `*Provider` with a GraalVM-specific implementation without needing to use `unpack` and rewrite, for example, regarding the `EnvoyMetricReceiverProvider`, we can simply create an implementation `EnvoyMetricReceiverGraalProvider` that extends `EnvoyMetricReceiverProvider`, and change the name to `default-graalvm`, then modify `application.yaml`
```yaml
envoy-metric:
selector: ${SW_ENVOY_METRIC:default-graalvm}
```
Although @wu-sheng had mentioned in elsewhere, I'd want to emphasize that, bear in mind if we want to make the native binaries first-class support, we can refactor the existing codes in many aspects to suit the native builds, as long as they are reasonable, instead of hacking the codes as if they are from third-party.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on a diff in pull request #11354: [Feature] Substitute classes to prepare for native-image.
Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on code in PR #11354:
URL: https://github.com/apache/skywalking/pull/11354#discussion_r1336508917
##########
docs/en/setup/backend/backend-graalvm.md:
##########
@@ -0,0 +1,45 @@
+# How to use SkyWalking with GraalVM native-image (Experimental)
+
+## Native-Image
+Native Image is a technology to ahead-of-time compile Java code to a standalone executable, called a native image.
+This executable includes the application classes, classes from its dependencies, runtime library classes,
+and statically linked native code from JDK. It does not run on the Java VM, but includes necessary components like memory management, thread scheduling, and so on from a different runtime system, called “Substrate VM”.
+Substrate VM is the name for the runtime components (like the deoptimizer, garbage collector, thread scheduling etc.).
+The resulting program has faster startup time and lower runtime memory overhead compared to a JVM.
+
+SkyWalking currently offers support for native-image. However, please note that the OAP started in this manner does not have the same functionality as the regular OAP, and some features are not yet supported.
+
+## Compile Guide
+Notice: If you are not familiar with the compilation process, please read [How-to-build](https://skywalking.apache.org/docs/main/next/en/guides/how-to-build/) first.
+
+The native-image compilation is not enabled by default. To enable it, we need to specify `-p native` during compilation, such as:
+
+```shell
+./mvnw -P native clean package -Dmaven.test.skip
+```
+
+Then, All packages are in `distribution/graal/dist` (.tar.gz for Linux and .zip for Windows).
+
+## Package Structure
+
+SkyWalking’s native-image distribution package consists of the following parts:
+
+* bin/cmd scripts: Located in the /bin folder. Includes startup Linux shell and Windows cmd scripts for the backend server.
+
+* Backend config: Located in the /config folder. Includes settings files of the backend. Running native-image does not require additional configuration, so you can refer to [backend-setup](https://skywalking.apache.org/docs/main/next/en/setup/backend/backend-setup/) to learn how to configure it.
+
+* Native image executable: Located in /image folder. It can be run directly, but it is not recommended to do so, as the absence of some environment variables can lead to startup failure.
+
+## How To Use
+By executing following:
+
+```shell
+./bin/oapService-native.sh
+```
+we can successfully start SkyWalking-oap.
+
+## Problems
Review Comment:
```suggestion
## Differences and TODO
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on a diff in pull request #11354: [Feature] Substitute classes to prepare for native-image.
Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on code in PR #11354:
URL: https://github.com/apache/skywalking/pull/11354#discussion_r1336509137
##########
docs/en/setup/backend/backend-graalvm.md:
##########
@@ -0,0 +1,45 @@
+# How to use SkyWalking with GraalVM native-image (Experimental)
+
+## Native-Image
+Native Image is a technology to ahead-of-time compile Java code to a standalone executable, called a native image.
+This executable includes the application classes, classes from its dependencies, runtime library classes,
+and statically linked native code from JDK. It does not run on the Java VM, but includes necessary components like memory management, thread scheduling, and so on from a different runtime system, called “Substrate VM”.
+Substrate VM is the name for the runtime components (like the deoptimizer, garbage collector, thread scheduling etc.).
+The resulting program has faster startup time and lower runtime memory overhead compared to a JVM.
+
+SkyWalking currently offers support for native-image. However, please note that the OAP started in this manner does not have the same functionality as the regular OAP, and some features are not yet supported.
+
+## Compile Guide
+Notice: If you are not familiar with the compilation process, please read [How-to-build](https://skywalking.apache.org/docs/main/next/en/guides/how-to-build/) first.
+
+The native-image compilation is not enabled by default. To enable it, we need to specify `-p native` during compilation, such as:
+
+```shell
+./mvnw -P native clean package -Dmaven.test.skip
+```
+
+Then, All packages are in `distribution/graal/dist` (.tar.gz for Linux and .zip for Windows).
+
+## Package Structure
+
+SkyWalking’s native-image distribution package consists of the following parts:
+
+* bin/cmd scripts: Located in the /bin folder. Includes startup Linux shell and Windows cmd scripts for the backend server.
+
+* Backend config: Located in the /config folder. Includes settings files of the backend. Running native-image does not require additional configuration, so you can refer to [backend-setup](https://skywalking.apache.org/docs/main/next/en/setup/backend/backend-setup/) to learn how to configure it.
+
+* Native image executable: Located in /image folder. It can be run directly, but it is not recommended to do so, as the absence of some environment variables can lead to startup failure.
+
+## How To Use
+By executing following:
+
+```shell
+./bin/oapService-native.sh
+```
+we can successfully start SkyWalking-oap.
+
+## Problems
+With native-image, some features are not yet supported.
+
+1. [LAL](https://skywalking.apache.org/docs/main/next/en/concepts-and-designs/lal/), [MAL](https://skywalking.apache.org/docs/main/next/en/concepts-and-designs/mal/), and some other features related to them are not supported at the moment.
+2. The configuration file for [OAL](https://skywalking.apache.org/docs/main/next/en/concepts-and-designs/oal/) can only be provided during the compilation period, meaning that it cannot be added or modified during runtime.
Review Comment:
I think this is only the difference, we would never support OAL change in the runtime.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] yswdqz commented on a diff in pull request #11354: [Feature] Substitute classes to prepare for native-image.
Posted by "yswdqz (via GitHub)" <gi...@apache.org>.
yswdqz commented on code in PR #11354:
URL: https://github.com/apache/skywalking/pull/11354#discussion_r1335580434
##########
distribution/graal/oal-rt-graal-native/src/main/java/org/apache/skywalking/oal/rt/OALRuntime.java:
##########
@@ -0,0 +1,673 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oal.rt;
+
+import freemarker.template.Configuration;
+import freemarker.template.Version;
+import javassist.CannotCompileException;
+import javassist.ClassPool;
+import javassist.CtClass;
+import javassist.CtConstructor;
+import javassist.CtField;
+import javassist.CtNewConstructor;
+import javassist.CtNewMethod;
+import javassist.NotFoundException;
+import javassist.bytecode.AnnotationsAttribute;
+import javassist.bytecode.ClassFile;
+import javassist.bytecode.ConstPool;
+import javassist.bytecode.SignatureAttribute;
+import javassist.bytecode.annotation.Annotation;
+import javassist.bytecode.annotation.ClassMemberValue;
+import javassist.bytecode.annotation.IntegerMemberValue;
+import javassist.bytecode.annotation.StringMemberValue;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.commons.lang3.JavaVersion;
+import org.apache.commons.lang3.SystemUtils;
+import org.apache.skywalking.oal.rt.output.AllDispatcherContext;
+import org.apache.skywalking.oal.rt.output.DispatcherContext;
+import org.apache.skywalking.oal.rt.parser.AnalysisResult;
+import org.apache.skywalking.oal.rt.parser.OALScripts;
+import org.apache.skywalking.oal.rt.parser.ScriptParser;
+import org.apache.skywalking.oal.rt.parser.SourceColumn;
+import org.apache.skywalking.oap.server.core.analysis.DisableRegister;
+import org.apache.skywalking.oap.server.core.analysis.DispatcherDetectorListener;
+import org.apache.skywalking.oap.server.core.analysis.SourceDispatcher;
+import org.apache.skywalking.oap.server.core.analysis.Stream;
+import org.apache.skywalking.oap.server.core.analysis.StreamAnnotationListener;
+import org.apache.skywalking.oap.server.core.oal.rt.OALCompileException;
+import org.apache.skywalking.oap.server.core.oal.rt.OALDefine;
+import org.apache.skywalking.oap.server.core.oal.rt.OALEngine;
+import org.apache.skywalking.oap.server.core.source.oal.rt.dispatcher.DispatcherClassPackageHolder;
+import org.apache.skywalking.oap.server.core.source.oal.rt.metrics.MetricClassPackageHolder;
+import org.apache.skywalking.oap.server.core.source.oal.rt.metrics.builder.MetricBuilderClassPackageHolder;
+import org.apache.skywalking.oap.server.core.storage.StorageBuilderFactory;
+import org.apache.skywalking.oap.server.core.storage.StorageException;
+import org.apache.skywalking.oap.server.core.storage.annotation.BanyanDB;
+import org.apache.skywalking.oap.server.core.storage.annotation.Column;
+import org.apache.skywalking.oap.server.library.module.ModuleStartException;
+
+import java.io.DataOutputStream;
+import java.io.File;
+import java.io.FileNotFoundException;
+import java.io.FileOutputStream;
+import java.io.FileReader;
+import java.io.IOException;
+import java.io.Reader;
+import java.io.StringWriter;
+import java.net.URI;
+import java.net.URL;
+import java.nio.file.FileSystem;
+import java.nio.file.FileSystems;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.ArrayList;
+import java.util.Enumeration;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.Objects;
+
+/**
+ * The old logic of this class used runtime class generation, which is not supported by native-image.
+ * Therefore, in the new logic, class files are generated during the compilation process, see (@link org.apache.skywalking.graal.Generator) and loaded at runtime.
+ */
+@Slf4j
+public class OALRuntime implements OALEngine {
Review Comment:
At build time, `skywalking-generate-plugin` call `OALRuntime#generateOALClassFiles`(this method added by me). Based on what you said, I think this part should depart from the OALRuntime.
At runtime, `OALRuntime#start` will be called, and it should be substituted by load classes which we generate before.(old logic is generate class at runtime).
I will try departing `generateOALClassFIles` later, I previously placed it in `OALRuntime` to reuse the existing API, but it seems indeed unnecessary, and indeed, the logic has become less clear.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on a diff in pull request #11354: [Feature] Substitute classes to prepare for native-image.
Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on code in PR #11354:
URL: https://github.com/apache/skywalking/pull/11354#discussion_r1335641394
##########
distribution/graal/oal-rt-graal-native/src/main/java/org/apache/skywalking/oal/rt/OALRuntime.java:
##########
@@ -0,0 +1,673 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oal.rt;
+
+import freemarker.template.Configuration;
+import freemarker.template.Version;
+import javassist.CannotCompileException;
+import javassist.ClassPool;
+import javassist.CtClass;
+import javassist.CtConstructor;
+import javassist.CtField;
+import javassist.CtNewConstructor;
+import javassist.CtNewMethod;
+import javassist.NotFoundException;
+import javassist.bytecode.AnnotationsAttribute;
+import javassist.bytecode.ClassFile;
+import javassist.bytecode.ConstPool;
+import javassist.bytecode.SignatureAttribute;
+import javassist.bytecode.annotation.Annotation;
+import javassist.bytecode.annotation.ClassMemberValue;
+import javassist.bytecode.annotation.IntegerMemberValue;
+import javassist.bytecode.annotation.StringMemberValue;
+import lombok.extern.slf4j.Slf4j;
+import org.apache.commons.lang3.JavaVersion;
+import org.apache.commons.lang3.SystemUtils;
+import org.apache.skywalking.oal.rt.output.AllDispatcherContext;
+import org.apache.skywalking.oal.rt.output.DispatcherContext;
+import org.apache.skywalking.oal.rt.parser.AnalysisResult;
+import org.apache.skywalking.oal.rt.parser.OALScripts;
+import org.apache.skywalking.oal.rt.parser.ScriptParser;
+import org.apache.skywalking.oal.rt.parser.SourceColumn;
+import org.apache.skywalking.oap.server.core.analysis.DisableRegister;
+import org.apache.skywalking.oap.server.core.analysis.DispatcherDetectorListener;
+import org.apache.skywalking.oap.server.core.analysis.SourceDispatcher;
+import org.apache.skywalking.oap.server.core.analysis.Stream;
+import org.apache.skywalking.oap.server.core.analysis.StreamAnnotationListener;
+import org.apache.skywalking.oap.server.core.oal.rt.OALCompileException;
+import org.apache.skywalking.oap.server.core.oal.rt.OALDefine;
+import org.apache.skywalking.oap.server.core.oal.rt.OALEngine;
+import org.apache.skywalking.oap.server.core.source.oal.rt.dispatcher.DispatcherClassPackageHolder;
+import org.apache.skywalking.oap.server.core.source.oal.rt.metrics.MetricClassPackageHolder;
+import org.apache.skywalking.oap.server.core.source.oal.rt.metrics.builder.MetricBuilderClassPackageHolder;
+import org.apache.skywalking.oap.server.core.storage.StorageBuilderFactory;
+import org.apache.skywalking.oap.server.core.storage.StorageException;
+import org.apache.skywalking.oap.server.core.storage.annotation.BanyanDB;
+import org.apache.skywalking.oap.server.core.storage.annotation.Column;
+import org.apache.skywalking.oap.server.library.module.ModuleStartException;
+
+import java.io.DataOutputStream;
+import java.io.File;
+import java.io.FileNotFoundException;
+import java.io.FileOutputStream;
+import java.io.FileReader;
+import java.io.IOException;
+import java.io.Reader;
+import java.io.StringWriter;
+import java.net.URI;
+import java.net.URL;
+import java.nio.file.FileSystem;
+import java.nio.file.FileSystems;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.ArrayList;
+import java.util.Enumeration;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.Objects;
+
+/**
+ * The old logic of this class used runtime class generation, which is not supported by native-image.
+ * Therefore, in the new logic, class files are generated during the compilation process, see (@link org.apache.skywalking.graal.Generator) and loaded at runtime.
+ */
+@Slf4j
+public class OALRuntime implements OALEngine {
+
+ private static final String CLASS_FILE_CHARSET = "UTF-8";
+ private static final String METRICS_FUNCTION_PACKAGE = "org.apache.skywalking.oap.server.core.analysis.metrics.";
+ private static final String WITH_METADATA_INTERFACE = "org.apache.skywalking.oap.server.core.analysis.metrics.WithMetadata";
+ private static final String DISPATCHER_INTERFACE = "org.apache.skywalking.oap.server.core.analysis.SourceDispatcher";
+ private static final String METRICS_STREAM_PROCESSOR = "org.apache.skywalking.oap.server.core.analysis.worker.MetricsStreamProcessor";
+ private static final String[] METRICS_CLASS_METHODS = {
+ "id",
+ "hashCode",
+ "remoteHashCode",
+ "equals",
+ "serialize",
+ "deserialize",
+ "getMeta",
+ "toHour",
+ "toDay"
+ };
+ private static final String[] METRICS_BUILDER_CLASS_METHODS = {
+ "entity2Storage",
+ "storage2Entity"
+ };
+ private final OALDefine oalDefine;
+ private final ClassPool classPool;
+ private ClassLoader currentClassLoader;
+ private Configuration configuration;
+ private AllDispatcherContext allDispatcherContext;
+ private StreamAnnotationListener streamAnnotationListener;
+ private DispatcherDetectorListener dispatcherDetectorListener;
+ private StorageBuilderFactory storageBuilderFactory;
+ private final List<Class> metricsClasses;
+ private final List<Class> dispatcherClasses;
+ private final boolean openEngineDebug;
+ // change
+ private static boolean INITIALED = false;
+ //change end
+
+ public OALRuntime(OALDefine define) {
+ oalDefine = define;
+ classPool = ClassPool.getDefault();
+ configuration = new Configuration(new Version("2.3.28"));
+ configuration.setEncoding(Locale.ENGLISH, CLASS_FILE_CHARSET);
+ configuration.setClassLoaderForTemplateLoading(OALRuntime.class.getClassLoader(), "/code-templates");
+ allDispatcherContext = new AllDispatcherContext();
+ metricsClasses = new ArrayList<>();
+ dispatcherClasses = new ArrayList<>();
+ // change
+ openEngineDebug = true;
+ // change end
+ }
+
+ @Override
+ public void setStreamListener(StreamAnnotationListener listener) throws ModuleStartException {
+ this.streamAnnotationListener = listener;
+ }
+
+ @Override
+ public void setDispatcherListener(DispatcherDetectorListener listener) throws ModuleStartException {
+ dispatcherDetectorListener = listener;
+ }
+
+ @Override
+ public void setStorageBuilderFactory(final StorageBuilderFactory factory) {
+ storageBuilderFactory = factory;
+ }
+
+ @Override
+ public void notifyAllListeners() throws ModuleStartException {
+ for (Class metricsClass : metricsClasses) {
+ try {
+ streamAnnotationListener.notify(metricsClass);
+ } catch (StorageException e) {
+ throw new ModuleStartException(e.getMessage(), e);
+ }
+ }
+ for (Class dispatcherClass : dispatcherClasses) {
+ try {
+ dispatcherDetectorListener.addIfAsSourceDispatcher(dispatcherClass);
+ } catch (Exception e) {
+ throw new ModuleStartException(e.getMessage(), e);
+ }
+ }
+ }
+
+ private void generateClassAtRuntime(OALScripts oalScripts) throws OALCompileException {
+ List<AnalysisResult> metricsStmts = oalScripts.getMetricsStmts();
+ metricsStmts.forEach(this::buildDispatcherContext);
+
+ for (AnalysisResult metricsStmt : metricsStmts) {
+ metricsClasses.add(generateMetricsClass(metricsStmt));
+ generateMetricsBuilderClass(metricsStmt);
+ }
+
+ for (Map.Entry<String, DispatcherContext> entry : allDispatcherContext.getAllContext().entrySet()) {
+ dispatcherClasses.add(generateDispatcherClass(entry.getKey(), entry.getValue()));
+ }
+
+ oalScripts.getDisableCollection().getAllDisableSources().forEach(disable -> {
+ DisableRegister.INSTANCE.add(disable);
+ });
+ }
+
+ /**
+ * Generate metrics class, and inject it to classloader
+ */
+ private Class generateMetricsClass(AnalysisResult metricsStmt) throws OALCompileException {
+ String className = metricsClassName(metricsStmt, false);
+ CtClass parentMetricsClass = null;
+ try {
+ parentMetricsClass = classPool.get(METRICS_FUNCTION_PACKAGE + metricsStmt.getMetricsClassName());
+ } catch (NotFoundException e) {
+ log.error("Can't find parent class for " + className + ".", e);
+ throw new OALCompileException(e.getMessage(), e);
+ }
+ CtClass metricsClass = classPool.makeClass(metricsClassName(metricsStmt, true), parentMetricsClass);
+ try {
+ metricsClass.addInterface(classPool.get(WITH_METADATA_INTERFACE));
+ } catch (NotFoundException e) {
+ log.error("Can't find WithMetadata interface for " + className + ".", e);
+ throw new OALCompileException(e.getMessage(), e);
+ }
+
+ ClassFile metricsClassClassFile = metricsClass.getClassFile();
+ ConstPool constPool = metricsClassClassFile.getConstPool();
+
+ /**
+ * Create empty construct
+ */
+ try {
+ CtConstructor defaultConstructor = CtNewConstructor.make("public " + className + "() {}", metricsClass);
+ metricsClass.addConstructor(defaultConstructor);
+ } catch (CannotCompileException e) {
+ log.error("Can't add empty constructor in " + className + ".", e);
+ throw new OALCompileException(e.getMessage(), e);
+ }
+
+ /**
+ * Add fields with annotations.
+ *
+ * private ${sourceField.typeName} ${sourceField.fieldName};
+ */
+ for (SourceColumn field : metricsStmt.getFieldsFromSource()) {
+ try {
+ CtField newField = CtField.make(
+ "private " + field.getType()
+ .getName() + " " + field.getFieldName() + ";", metricsClass);
+
+ metricsClass.addField(newField);
+
+ metricsClass.addMethod(CtNewMethod.getter(field.getFieldGetter(), newField));
+ metricsClass.addMethod(CtNewMethod.setter(field.getFieldSetter(), newField));
+
+ AnnotationsAttribute annotationsAttribute = new AnnotationsAttribute(
+ constPool, AnnotationsAttribute.visibleTag);
+ /**
+ * Add @Column(name = "${sourceField.columnName}")
+ */
+ Annotation columnAnnotation = new Annotation(Column.class.getName(), constPool);
+ columnAnnotation.addMemberValue("name", new StringMemberValue(field.getColumnName(), constPool));
+ if (field.getType().equals(String.class)) {
+ columnAnnotation.addMemberValue("length", new IntegerMemberValue(constPool, field.getLength()));
+ }
+ annotationsAttribute.addAnnotation(columnAnnotation);
+ if (field.isID()) {
+ // Add SeriesID = 0 annotation to ID field.
+ Annotation banyanShardingKeyAnnotation = new Annotation(BanyanDB.SeriesID.class.getName(), constPool);
+ banyanShardingKeyAnnotation.addMemberValue("index", new IntegerMemberValue(constPool, 0));
+ annotationsAttribute.addAnnotation(banyanShardingKeyAnnotation);
+ }
+
+ if (field.isGroupByCondInTopN()) {
+ Annotation banyanTopNAggregationAnnotation = new Annotation(BanyanDB.TopNAggregation.class.getName(), constPool);
+ annotationsAttribute.addAnnotation(banyanTopNAggregationAnnotation);
+ }
+
+ newField.getFieldInfo().addAttribute(annotationsAttribute);
+ } catch (CannotCompileException e) {
+ log.error(
+ "Can't add field(including set/get) " + field.getFieldName() + " in " + className + ".", e);
+ throw new OALCompileException(e.getMessage(), e);
+ }
+ }
+
+ /**
+ * Generate methods
+ */
+ for (String method : METRICS_CLASS_METHODS) {
+ StringWriter methodEntity = new StringWriter();
+ try {
+ configuration.getTemplate("metrics/" + method + ".ftl").process(metricsStmt, methodEntity);
+ metricsClass.addMethod(CtNewMethod.make(methodEntity.toString(), metricsClass));
+ } catch (Exception e) {
+ log.error("Can't generate method " + method + " for " + className + ".", e);
+ throw new OALCompileException(e.getMessage(), e);
+ }
+ }
+
+ /**
+ * Add following annotation to the metrics class
+ *
+ * at Stream(name = "${tableName}", scopeId = ${sourceScopeId}, builder = ${metricsName}Metrics.Builder.class, processor = MetricsStreamProcessor.class)
+ */
+ AnnotationsAttribute annotationsAttribute = new AnnotationsAttribute(
+ constPool, AnnotationsAttribute.visibleTag);
+ Annotation streamAnnotation = new Annotation(Stream.class.getName(), constPool);
+ streamAnnotation.addMemberValue("name", new StringMemberValue(metricsStmt.getTableName(), constPool));
+ streamAnnotation.addMemberValue(
+ "scopeId", new IntegerMemberValue(constPool, metricsStmt.getFrom().getSourceScopeId()));
+ streamAnnotation.addMemberValue(
+ "builder", new ClassMemberValue(metricsBuilderClassName(metricsStmt, true), constPool));
+ streamAnnotation.addMemberValue("processor", new ClassMemberValue(METRICS_STREAM_PROCESSOR, constPool));
+
+ annotationsAttribute.addAnnotation(streamAnnotation);
+ metricsClassClassFile.addAttribute(annotationsAttribute);
+
+ Class targetClass;
+ try {
+ if (SystemUtils.isJavaVersionAtMost(JavaVersion.JAVA_1_8)) {
+ targetClass = metricsClass.toClass(currentClassLoader, null);
+ } else {
+ targetClass = metricsClass.toClass(MetricClassPackageHolder.class);
+ }
+ } catch (CannotCompileException e) {
+ log.error("Can't compile/load " + className + ".", e);
+ throw new OALCompileException(e.getMessage(), e);
+ }
+
+ log.debug("Generate metrics class, " + metricsClass.getName());
+ writeGeneratedFile(metricsClass, "metrics");
+
+ return targetClass;
+ }
+
+ /**
+ * Generate metrics class builder and inject it to classloader
+ */
+ private void generateMetricsBuilderClass(AnalysisResult metricsStmt) throws OALCompileException {
+ String className = metricsBuilderClassName(metricsStmt, false);
+ CtClass metricsBuilderClass = classPool.makeClass(metricsBuilderClassName(metricsStmt, true));
+ try {
+ metricsBuilderClass.addInterface(classPool.get(storageBuilderFactory.builderTemplate().getSuperClass()));
+ } catch (NotFoundException e) {
+ log.error("Can't find StorageBuilder interface for " + className + ".", e);
+ throw new OALCompileException(e.getMessage(), e);
+ }
+
+ /**
+ * Create empty construct
+ */
+ try {
+ CtConstructor defaultConstructor = CtNewConstructor.make(
+ "public " + className + "() {}", metricsBuilderClass);
+ metricsBuilderClass.addConstructor(defaultConstructor);
+ } catch (CannotCompileException e) {
+ log.error("Can't add empty constructor in " + className + ".", e);
+ throw new OALCompileException(e.getMessage(), e);
+ }
+
+ /**
+ * Generate methods
+ */
+ for (String method : METRICS_BUILDER_CLASS_METHODS) {
+ StringWriter methodEntity = new StringWriter();
+ try {
+ configuration
+ .getTemplate(storageBuilderFactory.builderTemplate().getTemplatePath() + "/" + method + ".ftl")
+ .process(metricsStmt, methodEntity);
+ metricsBuilderClass.addMethod(CtNewMethod.make(methodEntity.toString(), metricsBuilderClass));
+ } catch (Exception e) {
+ log.error("Can't generate method " + method + " for " + className + ".", e);
+ throw new OALCompileException(e.getMessage(), e);
+ }
+ }
+
+ try {
+ if (SystemUtils.isJavaVersionAtMost(JavaVersion.JAVA_1_8)) {
+ metricsBuilderClass.toClass(currentClassLoader, null);
+ } else {
+ metricsBuilderClass.toClass(MetricBuilderClassPackageHolder.class);
+ }
+ } catch (CannotCompileException e) {
+ log.error("Can't compile/load " + className + ".", e);
+ throw new OALCompileException(e.getMessage(), e);
+ }
+
+ writeGeneratedFile(metricsBuilderClass, "metrics/builder");
+ }
+
+ /**
+ * Generate SourceDispatcher class and inject it to classloader
+ */
+ private Class generateDispatcherClass(String scopeName,
+ DispatcherContext dispatcherContext) throws OALCompileException {
+
+ String className = dispatcherClassName(scopeName, false);
+ CtClass dispatcherClass = classPool.makeClass(dispatcherClassName(scopeName, true));
+ try {
+ CtClass dispatcherInterface = classPool.get(DISPATCHER_INTERFACE);
+
+ dispatcherClass.addInterface(dispatcherInterface);
+
+ /**
+ * Set generic signature
+ */
+ String sourceClassName = oalDefine.getSourcePackage() + dispatcherContext.getSource();
+ SignatureAttribute.ClassSignature dispatcherSignature =
+ new SignatureAttribute.ClassSignature(
+ null, null,
+ // Set interface and its generic params
+ new SignatureAttribute.ClassType[] {
+ new SignatureAttribute.ClassType(
+ SourceDispatcher.class
+ .getCanonicalName(),
+ new SignatureAttribute.TypeArgument[] {
+ new SignatureAttribute.TypeArgument(
+ new SignatureAttribute.ClassType(
+ sourceClassName))
+ }
+ )
+ }
+ );
+
+ dispatcherClass.setGenericSignature(dispatcherSignature.encode());
+ } catch (NotFoundException e) {
+ log.error("Can't find Dispatcher interface for " + className + ".", e);
+ throw new OALCompileException(e.getMessage(), e);
+ }
+
+ /**
+ * Generate methods
+ */
+ for (AnalysisResult dispatcherContextMetric : dispatcherContext.getMetrics()) {
+ StringWriter methodEntity = new StringWriter();
+ try {
+ configuration.getTemplate("dispatcher/doMetrics.ftl").process(dispatcherContextMetric, methodEntity);
+ dispatcherClass.addMethod(CtNewMethod.make(methodEntity.toString(), dispatcherClass));
+ } catch (Exception e) {
+ log.error(
+ "Can't generate method do" + dispatcherContextMetric.getMetricsName() + " for " + className + ".",
+ e
+ );
+ log.error("Method body as following" + System.lineSeparator() + "{}", methodEntity);
+ throw new OALCompileException(e.getMessage(), e);
+ }
+ }
+
+ try {
+ StringWriter methodEntity = new StringWriter();
+ configuration.getTemplate("dispatcher/dispatch.ftl").process(dispatcherContext, methodEntity);
+ dispatcherClass.addMethod(CtNewMethod.make(methodEntity.toString(), dispatcherClass));
+ } catch (Exception e) {
+ log.error("Can't generate method dispatch for " + className + ".", e);
+ throw new OALCompileException(e.getMessage(), e);
+ }
+
+ Class targetClass;
+ try {
+ if (SystemUtils.isJavaVersionAtMost(JavaVersion.JAVA_1_8)) {
+ targetClass = dispatcherClass.toClass(currentClassLoader, null);
+ } else {
+ targetClass = dispatcherClass.toClass(DispatcherClassPackageHolder.class);
+ }
+ } catch (CannotCompileException e) {
+ log.error("Can't compile/load " + className + ".", e);
+ throw new OALCompileException(e.getMessage(), e);
+ }
+
+ writeGeneratedFile(dispatcherClass, "dispatcher");
+ return targetClass;
+ }
+
+ private String metricsClassName(AnalysisResult metricsStmt, boolean fullName) {
+ return (fullName ? oalDefine.getDynamicMetricsClassPackage() : "") + metricsStmt.getMetricsName() + "Metrics";
+ }
+
+ private String metricsBuilderClassName(AnalysisResult metricsStmt, boolean fullName) {
+ return (fullName ? oalDefine.getDynamicMetricsBuilderClassPackage() : "") + metricsStmt.getMetricsName() + "MetricsBuilder";
+ }
+
+ private String dispatcherClassName(String scopeName, boolean fullName) {
+ return (fullName ? oalDefine.getDynamicDispatcherClassPackage() : "") + scopeName + "Dispatcher";
+ }
+
+ private void buildDispatcherContext(AnalysisResult metricsStmt) {
+ String sourceName = metricsStmt.getFrom().getSourceName();
+
+ DispatcherContext context = allDispatcherContext.getAllContext().computeIfAbsent(sourceName, name -> {
+ DispatcherContext absent = new DispatcherContext();
+ absent.setSourcePackage(oalDefine.getSourcePackage());
+ absent.setSource(name);
+ absent.setPackageName(name.toLowerCase());
+ return absent;
+ });
+ metricsStmt.setMetricsClassPackage(oalDefine.getDynamicMetricsClassPackage());
+ metricsStmt.setSourcePackage(oalDefine.getSourcePackage());
+ context.getMetrics().add(metricsStmt);
+ }
+
+ private void writeGeneratedFile(CtClass metricsClass, String type) throws OALCompileException {
+ if (openEngineDebug) {
+ String className = metricsClass.getSimpleName();
+ DataOutputStream printWriter = null;
+ try {
+ File currentFolder = new File(GeneratedFileConfiguration.getGeneratedFilePath());
+ File folder = new File(currentFolder, type);
+ if (!folder.exists()) {
+ folder.mkdirs();
+ }
+ File file = new File(folder, className + ".class");
+ if (file.exists()) {
+ file.delete();
+ }
+ file.createNewFile();
+
+ printWriter = new DataOutputStream(new FileOutputStream(file));
+ metricsClass.toBytecode(printWriter);
+ printWriter.flush();
+ } catch (IOException e) {
+ log.warn("Can't create " + className + ".txt, ignore.", e);
+ return;
+ } catch (CannotCompileException e) {
+ log.warn("Can't compile " + className + ".class(should not happen), ignore.", e);
+ return;
+ } finally {
+ if (printWriter != null) {
+ try {
+ printWriter.close();
+ } catch (IOException e) {
+
+ }
+ }
+ }
+ }
+ }
+
+ // ------------------------------------ substituted method ------------------------------------
+ public void generateOALClassFiles(ClassLoader currentClassLoader, String rootPath) throws ModuleStartException, OALCompileException {
+ this.currentClassLoader = currentClassLoader;
+ Reader read;
+
+ try {
+ String root = rootPath + File.separator + ".." + File.separator +
+ ".." + File.separator + ".." + File.separator + ".." + File.separator +
+ "oap-server" + File.separator + "server-starter" + File.separator +
+ "target" + File.separator + "classes" + File.separator;
+ String configFile = oalDefine.getConfigFile();
+ read = new FileReader(root + configFile);
+ } catch (FileNotFoundException e) {
+ throw new ModuleStartException("Can't locate " + oalDefine.getConfigFile(), e);
+ }
+
+ OALScripts oalScripts;
+ try {
+ ScriptParser scriptParser = ScriptParser.createFromFile(read, oalDefine.getSourcePackage());
+ oalScripts = scriptParser.parse();
+ } catch (IOException e) {
+ throw new ModuleStartException("OAL script parse analysis failure.", e);
+ }
+
+ this.generateClassAtRuntime(oalScripts);
+ }
+
+ @Override
+ public void start(ClassLoader currentClassLoader) {
+ if (INITIALED) {
+ return;
+ }
+ if (Objects.equals(System.getProperty("org.graalvm.nativeimage.imagecode"), "runtime")) {
+ startByNativeImage(currentClassLoader);
+ } else {
+ startByJar(currentClassLoader);
+ }
+ INITIALED = true;
+ }
+
+ private void startByNativeImage(ClassLoader currentClassLoader) {
+ try (FileSystem fileSystem = FileSystems.newFileSystem(URI.create("resource:/"), Map.of(), currentClassLoader)) {
+ Path metrics = fileSystem.getPath("org/apache/skywalking/oap/server/core/source/oal/rt/metrics");
+ Path dispatcher = fileSystem.getPath("org/apache/skywalking/oap/server/core/source/oal/rt/dispatcher");
+
+ try (java.util.stream.Stream<Path> files = Files.walk(metrics)) {
+ files.forEach(file -> {
+ if (!file.toString().endsWith(".class")) {
+ return;
+ }
+ String name = file.toString().replace(File.separator, ".");
+
+ name = name.substring(0, name.length() - ".class".length());
+ if (name.startsWith("org.apache.skywalking.oap.server.core.source.oal.rt.metrics.builder")) {
+ return;
+ }
+ try {
+ Class<?> aClass = Class.forName(name);
+ if (!aClass.isAnnotationPresent(Stream.class)) {
+ return;
+ }
+ metricsClasses.add(aClass);
+ } catch (ClassNotFoundException e) {
+ // should not reach here
+ log.error(e.getMessage());
+ }
+ });
+ }
+ catch (IOException e) {
+ log.error("Failed to walk the path: " + metrics, e);
+ }
+ try (java.util.stream.Stream<Path> files = Files.walk(dispatcher)) {
+ files.forEach(file -> {
+ if (!file.toString().endsWith(".class")) {
+ return;
+ }
+ String name = file.toString().replace(File.separator, ".");
+ name = name.substring(0, name.length() - ".class".length());
+ try {
+ dispatcherClasses.add(Class.forName(name));
+ } catch (ClassNotFoundException e) {
+ // should not reach here
+ log.error(e.getMessage());
+ }
+ });
+ }
+ catch (IOException e) {
+ log.error("Failed to walk the path: " + dispatcher, e);
+ }
+ } catch (IOException e) {
+ // should not reach here
+ log.error("Failed to create FileSystem", e);
+ }
+ }
+
+ private void startByJar(ClassLoader currentClassLoader) {
+ String metricsPath = "org/apache/skywalking/oap/server/core/source/oal/rt/metrics";
+ String dispatcherPath = "org/apache/skywalking/oap/server/core/source/oal/rt/dispatcher";
Review Comment:
These could be converted from `METRICS_FUNCTION_PACKAGE`... by codes. Don't hard code things in two places.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on a diff in pull request #11354: [Feature] Substitute classes to prepare for native-image.
Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on code in PR #11354:
URL: https://github.com/apache/skywalking/pull/11354#discussion_r1335644799
##########
distribution/graal/skywalking-generate-plugin/pom.xml:
##########
@@ -0,0 +1,120 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one or more
+ ~ contributor license agreements. See the NOTICE file distributed with
+ ~ this work for additional information regarding copyright ownership.
+ ~ The ASF licenses this file to You under the Apache License, Version 2.0
+ ~ (the "License"); you may not use this file except in compliance with
+ ~ the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing, software
+ ~ distributed under the License is distributed on an "AS IS" BASIS,
+ ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ ~ See the License for the specific language governing permissions and
+ ~ limitations under the License.
+ ~
+ -->
+
+<project xmlns="http://maven.apache.org/POM/4.0.0"
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <modelVersion>4.0.0</modelVersion>
+ <parent>
+ <groupId>org.apache.skywalking</groupId>
+ <artifactId>graal</artifactId>
+ <version>9.7.0-SNAPSHOT</version>
+ </parent>
+
+ <packaging>maven-plugin</packaging>
+
+ <artifactId>skywalking-generate-plugin</artifactId>
Review Comment:
That is Envoy ALS relative only. Please separate them in the right place. Same as suggested here, https://github.com/apache/skywalking/pull/11354#discussion_r1335581945
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on a diff in pull request #11354: [Feature] Substitute classes to prepare for native-image.
Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on code in PR #11354:
URL: https://github.com/apache/skywalking/pull/11354#discussion_r1337131697
##########
docs/en/setup/backend/backend-graalvm.md:
##########
@@ -0,0 +1,48 @@
+# GraalVM native-image for SkyWalking (Experimental)
+
+## Native-Image
+Native Image is a technology to ahead-of-time compile Java code to a standalone executable, called a native image.
+This executable includes the application classes, classes from its dependencies, runtime library classes,
+and statically linked native code from JDK. It does not run on the Java VM, but includes necessary components like memory management, thread scheduling, and so on from a different runtime system, called “Substrate VM”.
+Substrate VM is the name for the runtime components (like the deoptimizer, garbage collector, thread scheduling etc.).
+The resulting program has faster startup time and lower runtime memory overhead compared to a JVM.
+
+SkyWalking currently offers support for OAP servers running as native-image. However, please note that the OAP started in this manner does not have the same functionality as the regular OAP, and some features are not yet supported.
+
+## Compile Guide
+Notice: If you are not familiar with the compilation process, please read [How-to-build](https://skywalking.apache.org/docs/main/next/en/guides/how-to-build/) first.
+
+The native-image compilation is not enabled by default. To enable it, we need to activate `native` profile during compilation, such as:
+
+```shell
+
+./mvnw -Pnative,backend clean package -Dmaven.test.skip
+
+```
+
+Then, 2 packages are in `distribution/graal/dist`, The package named `apache-skywalking-apm-native-pre-bin.tar.gz` is unnecessary for most users. It is just an intermediate product for generating the final native-image program, used for testing purposes.
Review Comment:
```suggestion
Then, 2 packages are in `distribution/graal/dist`,
As this is an experimental feature, a package named `apache-skywalking-apm-native-pre-bin.tar.gz` is a tarball for GraalVMization friendly, including original classes to be compiled by GraalVM.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on a diff in pull request #11354: [Feature] Substitute classes to prepare for native-image.
Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on code in PR #11354:
URL: https://github.com/apache/skywalking/pull/11354#discussion_r1336499532
##########
distribution/graal/dist-material/bin/exportEnv.sh:
##########
@@ -19,4 +19,5 @@
export SW_CONFIG_PATHS=${OAP_HOME}/config
export SW_ENVOY_METRIC=default-graalvm
-export SW_LOG_ANALYZER=default-graalvm
\ No newline at end of file
+export SW_LOG_ANALYZER=default-graalvm
+export SW_OAL_ENGINE_DEBUG=true
Review Comment:
Is this file controlling compiling stage or runtime stage? I feel these are mixed somehow.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on a diff in pull request #11354: [Feature] Substitute classes to prepare for native-image.
Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on code in PR #11354:
URL: https://github.com/apache/skywalking/pull/11354#discussion_r1337134638
##########
.github/workflows/native-image.yaml:
##########
@@ -0,0 +1,40 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+name: GraalVM Community Edition build
+on:
+ pull_request:
+jobs:
+ build:
+ runs-on: ubuntu-latest
+ steps:
+ - uses: actions/checkout@v3
+ with:
+ submodules: true
+ - uses: graalvm/setup-graalvm@v1
+ with:
+ java-version: '17.0.8'
+ distribution: 'graalvm-community'
+ components: 'native-image'
+ github-token: ${{ secrets.GITHUB_TOKEN }}
+ - name: Native Compile
+ run: |
+ ./mvnw -Pnative,backend clean package -Dmaven.test.skip
Review Comment:
```suggestion
./mvnw -Pbackend,native clean package -Dmaven.test.skip
```
Put backend before native seems more consistent with intuition, although this doesn't change the final result.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] kezhenxu94 commented on a diff in pull request #11354: [Feature] Substitute classes to prepare for native-image.
Posted by "kezhenxu94 (via GitHub)" <gi...@apache.org>.
kezhenxu94 commented on code in PR #11354:
URL: https://github.com/apache/skywalking/pull/11354#discussion_r1335303605
##########
.github/workflows/native-image.yaml:
##########
@@ -0,0 +1,43 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+name: GraalVM Community Edition build
+on:
+ pull_request:
+jobs:
+ build:
+ runs-on: ubuntu-latest
+ steps:
+ - uses: actions/checkout@v3
Review Comment:
```suggestion
- uses: actions/checkout@v3
with:
submodules: true
```
##########
.github/workflows/native-image.yaml:
##########
@@ -0,0 +1,43 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+name: GraalVM Community Edition build
+on:
+ pull_request:
+jobs:
+ build:
+ runs-on: ubuntu-latest
+ steps:
+ - uses: actions/checkout@v3
+ - uses: graalvm/setup-graalvm@v1
+ with:
+ java-version: '17.0.8'
+ distribution: 'graalvm-community'
+ components: 'native-image'
+ github-token: ${{ secrets.GITHUB_TOKEN }}
+ - name: Init And Update Submodule
+ run: |
+ git submodule init
+ git submodule update
Review Comment:
```suggestion
```
##########
distribution/graal/library-util-graal-native/src/main/java/org/apache/skywalking/oap/server/library/util/ResourceUtils.java:
##########
@@ -0,0 +1,80 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.oap.server.library.util;
+
+import java.io.File;
+import java.io.FileNotFoundException;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.io.Reader;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.util.ArrayList;
+
+import java.util.List;
+
+public class ResourceUtils {
+
+ private static final String CONFIG_PATH_ENV = "SW_CONFIG_PATHS";
+
+ private static Path PATH = null;
+
+ static {
+ if (System.getenv(CONFIG_PATH_ENV) != null) {
+ String skywalkingConfigPaths = System.getenv(CONFIG_PATH_ENV);
+ PATH = Paths.get(skywalkingConfigPaths);
+ }
Review Comment:
Perhaps set a default value for `PATH` or throw an exception if it's null
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] yswdqz commented on a diff in pull request #11354: [Feature] Substitute classes to prepare for native-image.
Posted by "yswdqz (via GitHub)" <gi...@apache.org>.
yswdqz commented on code in PR #11354:
URL: https://github.com/apache/skywalking/pull/11354#discussion_r1335627167
##########
distribution/graal/skywalking-generate-plugin/pom.xml:
##########
@@ -0,0 +1,120 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one or more
+ ~ contributor license agreements. See the NOTICE file distributed with
+ ~ this work for additional information regarding copyright ownership.
+ ~ The ASF licenses this file to You under the Apache License, Version 2.0
+ ~ (the "License"); you may not use this file except in compliance with
+ ~ the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing, software
+ ~ distributed under the License is distributed on an "AS IS" BASIS,
+ ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ ~ See the License for the specific language governing permissions and
+ ~ limitations under the License.
+ ~
+ -->
+
+<project xmlns="http://maven.apache.org/POM/4.0.0"
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <modelVersion>4.0.0</modelVersion>
+ <parent>
+ <groupId>org.apache.skywalking</groupId>
+ <artifactId>graal</artifactId>
+ <version>9.7.0-SNAPSHOT</version>
+ </parent>
+
+ <packaging>maven-plugin</packaging>
+
+ <artifactId>skywalking-generate-plugin</artifactId>
Review Comment:
Sorry, I means
```
FieldsHelper.SINGLETON.init("metadata-service-mapping.yaml", ServiceMetaInfo.class);
```
is also executed by this plugin.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on a diff in pull request #11354: [Feature] Substitute classes to prepare for native-image.
Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on code in PR #11354:
URL: https://github.com/apache/skywalking/pull/11354#discussion_r1337136814
##########
graal/graal-server-starter/pom.xml:
##########
@@ -0,0 +1,133 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one or more
+ ~ contributor license agreements. See the NOTICE file distributed with
+ ~ this work for additional information regarding copyright ownership.
+ ~ The ASF licenses this file to You under the Apache License, Version 2.0
+ ~ (the "License"); you may not use this file except in compliance with
+ ~ the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing, software
+ ~ distributed under the License is distributed on an "AS IS" BASIS,
+ ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ ~ See the License for the specific language governing permissions and
+ ~ limitations under the License.
+ ~
+ -->
+
+<project xmlns="http://maven.apache.org/POM/4.0.0"
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <modelVersion>4.0.0</modelVersion>
+ <parent>
+ <groupId>org.apache.skywalking</groupId>
+ <artifactId>graal</artifactId>
+ <version>9.7.0-SNAPSHOT</version>
+ </parent>
+ <artifactId>graal-server-starter</artifactId>
+
+ <properties>
+ <image.name>skywalking-oap</image.name>
+ </properties>
+
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.skywalking</groupId>
+ <artifactId>server-starter</artifactId>
+ <version>${project.version}</version>
+ <exclusions>
+ <exclusion>
+ <groupId>org.apache.skywalking</groupId>
+ <artifactId>meter-analyzer</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>org.apache.skywalking</groupId>
+ <artifactId>log-analyzer</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>org.apache.skywalking</groupId>
+ <artifactId>server-core</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>org.apache.skywalking</groupId>
+ <artifactId>oal-rt</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>org.apache.skywalking</groupId>
+ <artifactId>agent-analyzer</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>org.apache.skywalking</groupId>
+ <artifactId>library-util</artifactId>
+ </exclusion>
+ </exclusions>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.skywalking</groupId>
+ <artifactId>envoy-metrics-receiver-plugin-graal-native</artifactId>
+ <version>${project.version}</version>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.skywalking</groupId>
+ <artifactId>library-util-graal-native</artifactId>
+ <version>${project.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.skywalking</groupId>
+ <artifactId>meter-analyzer-graal-native</artifactId>
+ <version>${project.version}</version>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.skywalking</groupId>
+ <artifactId>server-core-graal-native</artifactId>
+ <version>${project.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.skywalking</groupId>
+ <artifactId>log-analyzer-graal-native</artifactId>
+ <version>${project.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.skywalking</groupId>
+ <artifactId>oal-rt-graal-native</artifactId>
+ <version>${project.version}</version>
+ </dependency>
+
+ <dependency>
+ <groupId>org.graalvm.nativeimage</groupId>
+ <artifactId>svm</artifactId>
+ <scope>provided</scope>
+ <version>22.3.1</version>
+ </dependency>
Review Comment:
Is this required for now? I believe this is for the next stage?
Meanwhile, please manage the dependency version from the bom too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on a diff in pull request #11354: [Feature] Substitute classes to prepare for native-image.
Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on code in PR #11354:
URL: https://github.com/apache/skywalking/pull/11354#discussion_r1336507597
##########
pom.xml:
##########
@@ -33,6 +33,7 @@
<modules>
<module>apm-protocol</module>
+ <module>distribution</module>
Review Comment:
```suggestion
<module>native-dist</module>
```
This seems a better name. And I want to ask why this module is not controlled by the profile, instead, you added the profile inside the pom.xml of this new module?
AFAIK, if I don't want the native image, this module is not necessary too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] yswdqz commented on a diff in pull request #11354: [Feature] Substitute classes to prepare for native-image.
Posted by "yswdqz (via GitHub)" <gi...@apache.org>.
yswdqz commented on code in PR #11354:
URL: https://github.com/apache/skywalking/pull/11354#discussion_r1336614405
##########
.github/workflows/native-image.yaml:
##########
@@ -0,0 +1,41 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+name: GraalVM Community Edition build
+on:
+ pull_request:
+jobs:
+ build:
+ runs-on: ubuntu-latest
+ steps:
+ - uses: actions/checkout@v3
+ with:
+ submodules: true
+ - uses: graalvm/setup-graalvm@v1
+ with:
+ java-version: '17.0.8'
+ distribution: 'graalvm-community'
+ components: 'native-image'
+ github-token: ${{ secrets.GITHUB_TOKEN }}
+ - name: Native Compile
+ run: |
+ ./mvnw -Dmaven.test.skip install
+ ./mvnw -Pnative clean package -Dmaven.test.skip
Review Comment:
Yes, I have tested it, otherwise, it will not be able to find the non-native modules.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on a diff in pull request #11354: [Feature] Substitute classes to prepare for native-image.
Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on code in PR #11354:
URL: https://github.com/apache/skywalking/pull/11354#discussion_r1335581945
##########
distribution/graal/skywalking-providers-graal-native/pom.xml:
##########
@@ -0,0 +1,40 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one or more
+ ~ contributor license agreements. See the NOTICE file distributed with
+ ~ this work for additional information regarding copyright ownership.
+ ~ The ASF licenses this file to You under the Apache License, Version 2.0
+ ~ (the "License"); you may not use this file except in compliance with
+ ~ the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing, software
+ ~ distributed under the License is distributed on an "AS IS" BASIS,
+ ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ ~ See the License for the specific language governing permissions and
+ ~ limitations under the License.
+ ~
+ -->
+
+<project xmlns="http://maven.apache.org/POM/4.0.0"
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <modelVersion>4.0.0</modelVersion>
+ <parent>
+ <groupId>org.apache.skywalking</groupId>
+ <artifactId>graal</artifactId>
+ <version>9.7.0-SNAPSHOT</version>
+ </parent>
+
+ <artifactId>skywalking-providers-graal-native</artifactId>
Review Comment:
This seems strange. Should we keep the style that makes patch modules match the existing ones?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] yswdqz commented on a diff in pull request #11354: [Feature] Substitute classes to prepare for native-image.
Posted by "yswdqz (via GitHub)" <gi...@apache.org>.
yswdqz commented on code in PR #11354:
URL: https://github.com/apache/skywalking/pull/11354#discussion_r1335583319
##########
distribution/graal/skywalking-providers-graal-native/pom.xml:
##########
@@ -0,0 +1,40 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one or more
+ ~ contributor license agreements. See the NOTICE file distributed with
+ ~ this work for additional information regarding copyright ownership.
+ ~ The ASF licenses this file to You under the Apache License, Version 2.0
+ ~ (the "License"); you may not use this file except in compliance with
+ ~ the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing, software
+ ~ distributed under the License is distributed on an "AS IS" BASIS,
+ ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ ~ See the License for the specific language governing permissions and
+ ~ limitations under the License.
+ ~
+ -->
+
+<project xmlns="http://maven.apache.org/POM/4.0.0"
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <modelVersion>4.0.0</modelVersion>
+ <parent>
+ <groupId>org.apache.skywalking</groupId>
+ <artifactId>graal</artifactId>
+ <version>9.7.0-SNAPSHOT</version>
+ </parent>
+
+ <artifactId>skywalking-providers-graal-native</artifactId>
Review Comment:
OK.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking] wu-sheng commented on a diff in pull request #11354: [Feature] Substitute classes to prepare for native-image.
Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on code in PR #11354:
URL: https://github.com/apache/skywalking/pull/11354#discussion_r1337921548
##########
graal/skywalking-on-graal-compiling-plugin/src/main/java/org/apache/skywalking/graal/EnvoyGeneratorMojo.java:
##########
@@ -0,0 +1,66 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.skywalking.graal;
+
+import org.apache.maven.plugin.AbstractMojo;
+import org.apache.maven.plugins.annotations.LifecyclePhase;
+import org.apache.maven.plugins.annotations.Mojo;
+
+import org.apache.maven.plugins.annotations.Parameter;
+import org.apache.maven.project.MavenProject;
+import org.apache.skywalking.oap.server.library.module.ModuleStartException;
+import org.apache.skywalking.oap.server.receiver.envoy.als.ServiceMetaInfo;
+import org.apache.skywalking.oap.server.receiver.envoy.als.mx.FieldsHelper;
+
+import java.io.File;
+import java.nio.file.Files;
+import java.nio.file.Paths;
+
+@Mojo(name = "envoy-generate", defaultPhase = LifecyclePhase.PROCESS_CLASSES)
Review Comment:
@yswdqz Yes, this seems not necessary. This is just an initialization, and doesn't generate anything for the runtime. Could you explain why do we need this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org