You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@skywalking.apache.org by GitBox <gi...@apache.org> on 2020/12/01 02:32:44 UTC
[GitHub] [skywalking-data-collect-protocol] wu-sheng opened a new pull request #35: Add logging collect protocol
wu-sheng opened a new pull request #35:
URL: https://github.com/apache/skywalking-data-collect-protocol/pull/35
Due to prepare for the Satellite log collection, I am adding the collect service for logs.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking-data-collect-protocol] EvanLjp commented on a change in pull request #35: Add logging collect protocol
Posted by GitBox <gi...@apache.org>.
EvanLjp commented on a change in pull request #35:
URL: https://github.com/apache/skywalking-data-collect-protocol/pull/35#discussion_r533111029
##########
File path: logging/Logging.proto
##########
@@ -0,0 +1,107 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+syntax = "proto3";
+
+option java_multiple_files = true;
+option java_package = "org.apache.skywalking.apm.network.logging.v3";
+option csharp_namespace = "SkyWalking.NetworkProtocol.V3";
+option go_package = "skywalking/network/language/agent/v3";
+
+import "common/Common.proto";
+
+// Report collected logs into the OAP backend
+service LogReportService {
+ // Recommend to report log data in a stream mode.
+ // The service/instance/endpoint of the log could share the previous value if they are not set.
+ // Reporting the logs of same service in the batch mode could reduce the network cost.
+ rpc collect (stream LogData) returns (Commands) {
+ }
+}
+
+// Log data is collected through file scratcher of agent.
+// Natively, Satellite provides various ways to collect logs.
+message LogData {
+ // [Optional] The timestamp of the log, in millisecond.
+ // If not set, OAP server would use the received timestamp as log's timestamp, or relies on the OAP server analyzer.
+ int64 timestamp = 1;
+ // [Required] **Service**. Represents a set/group of workloads which provide the same behaviours for incoming requests.
+ //
+ // The logic name represents the service. This would show as a separate node in the topology.
+ // The metrics analyzed from the spans, would be aggregated for this entity as the service level.
+ //
+ // If this is not the first element of the streaming, use the previous not-null name as the service name.
+ string service = 2;
+ // [Optional] **Service Instance**. Each individual workload in the Service group is known as an instance. Like `pods` in Kubernetes, it
+ // doesn't need to be a single OS process, however, if you are using instrument agents, an instance is actually a real OS process.
+ //
+ // The logic name represents the service instance. This would show as a separate node in the instance relationship.
+ // The metrics analyzed from the spans, would be aggregated for this entity as the service instance level.
+ string serviceInstance = 3;
+ // [Optional] **Endpoint**. A path in a service for incoming requests, such as an HTTP URI path or a gRPC service class + method signature.
+ //
+ // The logic name represents the endpoint, which logs belong.
+ string endpoint = 4;
+ // [Required] The content of the log.
+ LogDataBody body = 5;
+ // [Optional] Logs with trace context
+ TraceContext traceContext = 6;
+}
+
+// The content of the log data
+message LogDataBody {
+ // A type to match analyzer(s) at the OAP server.
+ // The data could be analysis at the client side, but could be partial
+ string type = 1;
+ // Content with extendable format.
+ oneof content {
+ TextLog text = 2;
+ JSONLog json = 3;
+ YAMLLog yaml = 4;
+ }
+}
+
+// Literal text log, typically requires regex or split mechanism to filter meaningful info.
+message TextLog {
+ string text = 1;
+}
+
+// JSON formatted log. The json field represents the string could be formatted as a JSON object.
+message JSONLog {
+ string json = 1;
+}
+
+// YAML formatted log. The yaml field represents the string could be formatted as a YAML map.
+message YAMLLog {
+ string yaml = 1;
+}
+
+// Logs with trace context, represent agent system has injects context(IDs) into log text.
+message TraceContext {
+ // [Optional] The available tags. OAP server could provide search/analysis capabilities base on these.
+ repeated KeyStringValuePair tags = 7;
Review comment:
Maybe tags should belong to the LogData rather than TraceContext. Log maybe also have some tags, such as threadID, loggerName
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking-data-collect-protocol] wu-sheng commented on a change in pull request #35: Add logging collect protocol
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on a change in pull request #35:
URL: https://github.com/apache/skywalking-data-collect-protocol/pull/35#discussion_r533070881
##########
File path: logging/Logging.proto
##########
@@ -0,0 +1,79 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+syntax = "proto3";
+
+option java_multiple_files = true;
+option java_package = "org.apache.skywalking.apm.network.logging.v3";
+option csharp_namespace = "SkyWalking.NetworkProtocol.V3";
+option go_package = "skywalking/network/language/agent/v3";
+
+import "common/Common.proto";
+
+// Report collected logs into the OAP backend
+service LogReportService {
+ // Recommend to report log data in a stream mode.
+ // The service/instance/endpoint of the log could share the previous value if they are not set.
+ // Reporting the logs of same service in the batch mode could reduce the network cost.
+ rpc collect (stream LogData) returns (Commands) {
+ }
+}
+
+// Log data is collected through file scratcher of agent.
+// Natively, Satellite provides various ways to collect logs.
+message LogData {
Review comment:
This is the time analyzed from the log content if possible. This is another tradeoff between client analysis and server analysis.
I am polishing this, let me think a little more.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking-data-collect-protocol] wu-sheng commented on a change in pull request #35: Add logging collect protocol
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on a change in pull request #35:
URL: https://github.com/apache/skywalking-data-collect-protocol/pull/35#discussion_r533035752
##########
File path: logging/Logging.proto
##########
@@ -0,0 +1,79 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+syntax = "proto3";
+
+option java_multiple_files = true;
+option java_package = "org.apache.skywalking.apm.network.logging.v3";
+option csharp_namespace = "SkyWalking.NetworkProtocol.V3";
+option go_package = "skywalking/network/language/agent/v3";
+
+import "common/Common.proto";
+
+// Report collected logs into the OAP backend
+service LogReportService {
+ // Recommend to report log data in a stream mode.
+ // The service/instance/endpoint of the log could share the previous value if they are not set.
+ // Reporting the logs of same service in the batch mode could reduce the network cost.
+ rpc collect (stream LogData) returns (Commands) {
+ }
+}
+
+// Log data is collected through file scratcher of agent.
+// Natively, Satellite provides various ways to collect logs.
+message LogData {
+ // [Required] **Service**. Represents a set/group of workloads which provide the same behaviours for incoming requests.
+ //
+ // The logic name represents the service. This would show as a separate node in the topology.
+ // The metrics analyzed from the spans, would be aggregated for this entity as the service level.
+ //
+ // If this is not the first element of the streaming, use the previous not-null name as the service name.
+ string service = 1;
+ // [Optional] **Service Instance**. Each individual workload in the Service group is known as an instance. Like `pods` in Kubernetes, it
+ // doesn't need to be a single OS process, however, if you are using instrument agents, an instance is actually a real OS process.
+ //
+ // The logic name represents the service instance. This would show as a separate node in the instance relationship.
+ // The metrics analyzed from the spans, would be aggregated for this entity as the service instance level.
+ string serviceInstance = 2;
+ // [Optional] **Endpoint**. A path in a service for incoming requests, such as an HTTP URI path or a gRPC service class + method signature.
+ //
+ // The logic name represents the endpoint, which logs belong.
+ string endpoint = 3;
Review comment:
I have tagged it as `[Optional]`, as in many cases, it is not used. The good thing in protobuf is null field costs very limited resources.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking-data-collect-protocol] hanahmily commented on a change in pull request #35: Add logging collect protocol
Posted by GitBox <gi...@apache.org>.
hanahmily commented on a change in pull request #35:
URL: https://github.com/apache/skywalking-data-collect-protocol/pull/35#discussion_r533042107
##########
File path: logging/Logging.proto
##########
@@ -0,0 +1,82 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+syntax = "proto3";
+
+option java_multiple_files = true;
+option java_package = "org.apache.skywalking.apm.network.logging.v3";
+option csharp_namespace = "SkyWalking.NetworkProtocol.V3";
+option go_package = "skywalking/network/language/agent/v3";
+
+import "common/Common.proto";
+
+// Report collected logs into the OAP backend
+service LogReportService {
+ // Recommend to report log data in a stream mode.
+ // The service/instance/endpoint of the log could share the previous value if they are not set.
+ // Reporting the logs of same service in the batch mode could reduce the network cost.
+ rpc collect (stream LogData) returns (Commands) {
+ }
+}
+
+// Log data is collected through file scratcher of agent.
+// Natively, Satellite provides various ways to collect logs.
+message LogData {
+ // [Optional] The timestamp of the log.
+ // If not set, OAP server would use the received timestamp as log's timestamp.
Review comment:
we should comment it's in `nanosecond` which could be parsed easily by Java, Go, and others.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking-data-collect-protocol] wu-sheng commented on a change in pull request #35: Add logging collect protocol
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on a change in pull request #35:
URL: https://github.com/apache/skywalking-data-collect-protocol/pull/35#discussion_r533040008
##########
File path: logging/Logging.proto
##########
@@ -0,0 +1,79 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+syntax = "proto3";
+
+option java_multiple_files = true;
+option java_package = "org.apache.skywalking.apm.network.logging.v3";
+option csharp_namespace = "SkyWalking.NetworkProtocol.V3";
+option go_package = "skywalking/network/language/agent/v3";
+
+import "common/Common.proto";
+
+// Report collected logs into the OAP backend
+service LogReportService {
+ // Recommend to report log data in a stream mode.
+ // The service/instance/endpoint of the log could share the previous value if they are not set.
+ // Reporting the logs of same service in the batch mode could reduce the network cost.
+ rpc collect (stream LogData) returns (Commands) {
+ }
+}
+
+// Log data is collected through file scratcher of agent.
+// Natively, Satellite provides various ways to collect logs.
+message LogData {
Review comment:
Let me add it. A client-side timestamp makes more sense.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking-data-collect-protocol] hanahmily commented on a change in pull request #35: Add logging collect protocol
Posted by GitBox <gi...@apache.org>.
hanahmily commented on a change in pull request #35:
URL: https://github.com/apache/skywalking-data-collect-protocol/pull/35#discussion_r533044022
##########
File path: logging/Logging.proto
##########
@@ -0,0 +1,82 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+syntax = "proto3";
+
+option java_multiple_files = true;
+option java_package = "org.apache.skywalking.apm.network.logging.v3";
+option csharp_namespace = "SkyWalking.NetworkProtocol.V3";
+option go_package = "skywalking/network/language/agent/v3";
+
+import "common/Common.proto";
+
+// Report collected logs into the OAP backend
+service LogReportService {
+ // Recommend to report log data in a stream mode.
+ // The service/instance/endpoint of the log could share the previous value if they are not set.
+ // Reporting the logs of same service in the batch mode could reduce the network cost.
+ rpc collect (stream LogData) returns (Commands) {
+ }
+}
+
+// Log data is collected through file scratcher of agent.
+// Natively, Satellite provides various ways to collect logs.
+message LogData {
+ // [Optional] The timestamp of the log.
+ // If not set, OAP server would use the received timestamp as log's timestamp.
Review comment:
all right, let's keep it as is. but let's comment it's in `millisecond`. I encountered several GO developers who treated it as `nanosecond` by mistake since the first project adopts this protocol is satellite.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking-data-collect-protocol] EvanLjp commented on pull request #35: Add logging collect protocol
Posted by GitBox <gi...@apache.org>.
EvanLjp commented on pull request #35:
URL: https://github.com/apache/skywalking-data-collect-protocol/pull/35#issuecomment-736178538
Whether the endpoint name is required or not, in many cases the log does not know which endpoint it belongs to.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking-data-collect-protocol] wu-sheng commented on pull request #35: Add logging collect protocol
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on pull request #35:
URL: https://github.com/apache/skywalking-data-collect-protocol/pull/35#issuecomment-736263851
@hanahmily @kezhenxu94 @EvanLjp I submit another version, from my understanding, should resolve your concerns.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking-data-collect-protocol] hanahmily commented on a change in pull request #35: Add logging collect protocol
Posted by GitBox <gi...@apache.org>.
hanahmily commented on a change in pull request #35:
URL: https://github.com/apache/skywalking-data-collect-protocol/pull/35#discussion_r533039710
##########
File path: logging/Logging.proto
##########
@@ -0,0 +1,79 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+syntax = "proto3";
+
+option java_multiple_files = true;
+option java_package = "org.apache.skywalking.apm.network.logging.v3";
+option csharp_namespace = "SkyWalking.NetworkProtocol.V3";
+option go_package = "skywalking/network/language/agent/v3";
+
+import "common/Common.proto";
+
+// Report collected logs into the OAP backend
+service LogReportService {
+ // Recommend to report log data in a stream mode.
+ // The service/instance/endpoint of the log could share the previous value if they are not set.
+ // Reporting the logs of same service in the batch mode could reduce the network cost.
+ rpc collect (stream LogData) returns (Commands) {
+ }
+}
+
+// Log data is collected through file scratcher of agent.
+// Natively, Satellite provides various ways to collect logs.
+message LogData {
Review comment:
Do we need to add `timestamp` here? if its value is 0, oap would decide it by itself.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking-data-collect-protocol] wu-sheng commented on a change in pull request #35: Add logging collect protocol
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on a change in pull request #35:
URL: https://github.com/apache/skywalking-data-collect-protocol/pull/35#discussion_r533042011
##########
File path: logging/Logging.proto
##########
@@ -0,0 +1,79 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+syntax = "proto3";
+
+option java_multiple_files = true;
+option java_package = "org.apache.skywalking.apm.network.logging.v3";
+option csharp_namespace = "SkyWalking.NetworkProtocol.V3";
+option go_package = "skywalking/network/language/agent/v3";
+
+import "common/Common.proto";
+
+// Report collected logs into the OAP backend
+service LogReportService {
+ // Recommend to report log data in a stream mode.
+ // The service/instance/endpoint of the log could share the previous value if they are not set.
+ // Reporting the logs of same service in the batch mode could reduce the network cost.
+ rpc collect (stream LogData) returns (Commands) {
+ }
+}
+
+// Log data is collected through file scratcher of agent.
+// Natively, Satellite provides various ways to collect logs.
+message LogData {
+ // [Required] **Service**. Represents a set/group of workloads which provide the same behaviours for incoming requests.
+ //
+ // The logic name represents the service. This would show as a separate node in the topology.
+ // The metrics analyzed from the spans, would be aggregated for this entity as the service level.
+ //
+ // If this is not the first element of the streaming, use the previous not-null name as the service name.
+ string service = 1;
Review comment:
It could exist, if not, the backend is going to use the previous non-null name to override. I know, at this point, this seems not a required field, but I doubt whether client-side, such as satellite, whether use this in this way.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking-data-collect-protocol] kezhenxu94 commented on a change in pull request #35: Add logging collect protocol
Posted by GitBox <gi...@apache.org>.
kezhenxu94 commented on a change in pull request #35:
URL: https://github.com/apache/skywalking-data-collect-protocol/pull/35#discussion_r533063867
##########
File path: logging/Logging.proto
##########
@@ -0,0 +1,79 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+syntax = "proto3";
+
+option java_multiple_files = true;
+option java_package = "org.apache.skywalking.apm.network.logging.v3";
+option csharp_namespace = "SkyWalking.NetworkProtocol.V3";
+option go_package = "skywalking/network/language/agent/v3";
+
+import "common/Common.proto";
+
+// Report collected logs into the OAP backend
+service LogReportService {
+ // Recommend to report log data in a stream mode.
+ // The service/instance/endpoint of the log could share the previous value if they are not set.
+ // Reporting the logs of same service in the batch mode could reduce the network cost.
+ rpc collect (stream LogData) returns (Commands) {
+ }
+}
+
+// Log data is collected through file scratcher of agent.
+// Natively, Satellite provides various ways to collect logs.
+message LogData {
Review comment:
Is it necessary? IMO most logs in reality already contain a timestamp. Especially in JSON/YAML format, it’s rather easy to get the timestamp
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking-data-collect-protocol] wu-sheng commented on a change in pull request #35: Add logging collect protocol
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on a change in pull request #35:
URL: https://github.com/apache/skywalking-data-collect-protocol/pull/35#discussion_r533042635
##########
File path: logging/Logging.proto
##########
@@ -0,0 +1,82 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+syntax = "proto3";
+
+option java_multiple_files = true;
+option java_package = "org.apache.skywalking.apm.network.logging.v3";
+option csharp_namespace = "SkyWalking.NetworkProtocol.V3";
+option go_package = "skywalking/network/language/agent/v3";
+
+import "common/Common.proto";
+
+// Report collected logs into the OAP backend
+service LogReportService {
+ // Recommend to report log data in a stream mode.
+ // The service/instance/endpoint of the log could share the previous value if they are not set.
+ // Reporting the logs of same service in the batch mode could reduce the network cost.
+ rpc collect (stream LogData) returns (Commands) {
+ }
+}
+
+// Log data is collected through file scratcher of agent.
+// Natively, Satellite provides various ways to collect logs.
+message LogData {
+ // [Optional] The timestamp of the log.
+ // If not set, OAP server would use the received timestamp as log's timestamp.
Review comment:
But this will be different from all other protocols, I don't think it is a good idea, providing two kinds of time formats in the same project.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking-data-collect-protocol] wu-sheng commented on a change in pull request #35: Add logging collect protocol
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on a change in pull request #35:
URL: https://github.com/apache/skywalking-data-collect-protocol/pull/35#discussion_r533112598
##########
File path: logging/Logging.proto
##########
@@ -0,0 +1,107 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+syntax = "proto3";
+
+option java_multiple_files = true;
+option java_package = "org.apache.skywalking.apm.network.logging.v3";
+option csharp_namespace = "SkyWalking.NetworkProtocol.V3";
+option go_package = "skywalking/network/language/agent/v3";
+
+import "common/Common.proto";
+
+// Report collected logs into the OAP backend
+service LogReportService {
+ // Recommend to report log data in a stream mode.
+ // The service/instance/endpoint of the log could share the previous value if they are not set.
+ // Reporting the logs of same service in the batch mode could reduce the network cost.
+ rpc collect (stream LogData) returns (Commands) {
+ }
+}
+
+// Log data is collected through file scratcher of agent.
+// Natively, Satellite provides various ways to collect logs.
+message LogData {
+ // [Optional] The timestamp of the log, in millisecond.
+ // If not set, OAP server would use the received timestamp as log's timestamp, or relies on the OAP server analyzer.
+ int64 timestamp = 1;
+ // [Required] **Service**. Represents a set/group of workloads which provide the same behaviours for incoming requests.
+ //
+ // The logic name represents the service. This would show as a separate node in the topology.
+ // The metrics analyzed from the spans, would be aggregated for this entity as the service level.
+ //
+ // If this is not the first element of the streaming, use the previous not-null name as the service name.
+ string service = 2;
+ // [Optional] **Service Instance**. Each individual workload in the Service group is known as an instance. Like `pods` in Kubernetes, it
+ // doesn't need to be a single OS process, however, if you are using instrument agents, an instance is actually a real OS process.
+ //
+ // The logic name represents the service instance. This would show as a separate node in the instance relationship.
+ // The metrics analyzed from the spans, would be aggregated for this entity as the service instance level.
+ string serviceInstance = 3;
+ // [Optional] **Endpoint**. A path in a service for incoming requests, such as an HTTP URI path or a gRPC service class + method signature.
+ //
+ // The logic name represents the endpoint, which logs belong.
+ string endpoint = 4;
+ // [Required] The content of the log.
+ LogDataBody body = 5;
+ // [Optional] Logs with trace context
+ TraceContext traceContext = 6;
+}
+
+// The content of the log data
+message LogDataBody {
+ // A type to match analyzer(s) at the OAP server.
+ // The data could be analysis at the client side, but could be partial
+ string type = 1;
+ // Content with extendable format.
+ oneof content {
+ TextLog text = 2;
+ JSONLog json = 3;
+ YAMLLog yaml = 4;
+ }
+}
+
+// Literal text log, typically requires regex or split mechanism to filter meaningful info.
+message TextLog {
+ string text = 1;
+}
+
+// JSON formatted log. The json field represents the string could be formatted as a JSON object.
+message JSONLog {
+ string json = 1;
+}
+
+// YAML formatted log. The yaml field represents the string could be formatted as a YAML map.
+message YAMLLog {
+ string yaml = 1;
+}
+
+// Logs with trace context, represent agent system has injects context(IDs) into log text.
+message TraceContext {
+ // [Optional] The available tags. OAP server could provide search/analysis capabilities base on these.
+ repeated KeyStringValuePair tags = 7;
Review comment:
Sorry, my copy/paste issue, fixing.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking-data-collect-protocol] wu-sheng merged pull request #35: Add logging collect protocol
Posted by GitBox <gi...@apache.org>.
wu-sheng merged pull request #35:
URL: https://github.com/apache/skywalking-data-collect-protocol/pull/35
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking-data-collect-protocol] hanahmily commented on a change in pull request #35: Add logging collect protocol
Posted by GitBox <gi...@apache.org>.
hanahmily commented on a change in pull request #35:
URL: https://github.com/apache/skywalking-data-collect-protocol/pull/35#discussion_r533046051
##########
File path: logging/Logging.proto
##########
@@ -0,0 +1,82 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+syntax = "proto3";
+
+option java_multiple_files = true;
+option java_package = "org.apache.skywalking.apm.network.logging.v3";
+option csharp_namespace = "SkyWalking.NetworkProtocol.V3";
+option go_package = "skywalking/network/language/agent/v3";
+
+import "common/Common.proto";
+
+// Report collected logs into the OAP backend
+service LogReportService {
+ // Recommend to report log data in a stream mode.
+ // The service/instance/endpoint of the log could share the previous value if they are not set.
+ // Reporting the logs of same service in the batch mode could reduce the network cost.
+ rpc collect (stream LogData) returns (Commands) {
+ }
+}
+
+// Log data is collected through file scratcher of agent.
+// Natively, Satellite provides various ways to collect logs.
+message LogData {
+ // [Optional] The timestamp of the log.
+ // If not set, OAP server would use the received timestamp as log's timestamp.
+ int64 timestamp = 1;
+ // [Required] **Service**. Represents a set/group of workloads which provide the same behaviours for incoming requests.
+ //
+ // The logic name represents the service. This would show as a separate node in the topology.
+ // The metrics analyzed from the spans, would be aggregated for this entity as the service level.
+ //
+ // If this is not the first element of the streaming, use the previous not-null name as the service name.
+ string service = 2;
+ // [Optional] **Service Instance**. Each individual workload in the Service group is known as an instance. Like `pods` in Kubernetes, it
+ // doesn't need to be a single OS process, however, if you are using instrument agents, an instance is actually a real OS process.
+ //
+ // The logic name represents the service instance. This would show as a separate node in the instance relationship.
+ // The metrics analyzed from the spans, would be aggregated for this entity as the service instance level.
+ string serviceInstance =3;
+ // [Optional] **Endpoint**. A path in a service for incoming requests, such as an HTTP URI path or a gRPC service class + method signature.
+ //
+ // The logic name represents the endpoint, which logs belong.
+ string endpoint = 4;
+ // [Required] The literal text of the logs.
+ string log = 5;
Review comment:
How about using [this type](https://github.com/open-telemetry/opentelemetry-proto/blob/master/opentelemetry/proto/logs/v1/logs.proto#L107) or other similar paths.
The benefit is to extend the log raw format without panic. and some optional fields could be embedded in it to make LogData clearer.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking-data-collect-protocol] EvanLjp commented on pull request #35: Add logging collect protocol
Posted by GitBox <gi...@apache.org>.
EvanLjp commented on pull request #35:
URL: https://github.com/apache/skywalking-data-collect-protocol/pull/35#issuecomment-736507268
LGTM
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking-data-collect-protocol] kezhenxu94 commented on a change in pull request #35: Add logging collect protocol
Posted by GitBox <gi...@apache.org>.
kezhenxu94 commented on a change in pull request #35:
URL: https://github.com/apache/skywalking-data-collect-protocol/pull/35#discussion_r533036195
##########
File path: logging/Logging.proto
##########
@@ -0,0 +1,79 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+syntax = "proto3";
+
+option java_multiple_files = true;
+option java_package = "org.apache.skywalking.apm.network.logging.v3";
+option csharp_namespace = "SkyWalking.NetworkProtocol.V3";
+option go_package = "skywalking/network/language/agent/v3";
+
+import "common/Common.proto";
+
+// Report collected logs into the OAP backend
+service LogReportService {
+ // Recommend to report log data in a stream mode.
+ // The service/instance/endpoint of the log could share the previous value if they are not set.
+ // Reporting the logs of same service in the batch mode could reduce the network cost.
+ rpc collect (stream LogData) returns (Commands) {
+ }
+}
+
+// Log data is collected through file scratcher of agent.
+// Natively, Satellite provides various ways to collect logs.
+message LogData {
+ // [Required] **Service**. Represents a set/group of workloads which provide the same behaviours for incoming requests.
+ //
+ // The logic name represents the service. This would show as a separate node in the topology.
+ // The metrics analyzed from the spans, would be aggregated for this entity as the service level.
+ //
+ // If this is not the first element of the streaming, use the previous not-null name as the service name.
+ string service = 1;
Review comment:
So that means `service` is always present? What's the point of "If this is not the first element of the streaming, use the previous not-null name as the service name.",
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking-data-collect-protocol] EvanLjp commented on a change in pull request #35: Add logging collect protocol
Posted by GitBox <gi...@apache.org>.
EvanLjp commented on a change in pull request #35:
URL: https://github.com/apache/skywalking-data-collect-protocol/pull/35#discussion_r533035067
##########
File path: logging/Logging.proto
##########
@@ -0,0 +1,79 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+syntax = "proto3";
+
+option java_multiple_files = true;
+option java_package = "org.apache.skywalking.apm.network.logging.v3";
+option csharp_namespace = "SkyWalking.NetworkProtocol.V3";
+option go_package = "skywalking/network/language/agent/v3";
+
+import "common/Common.proto";
+
+// Report collected logs into the OAP backend
+service LogReportService {
+ // Recommend to report log data in a stream mode.
+ // The service/instance/endpoint of the log could share the previous value if they are not set.
+ // Reporting the logs of same service in the batch mode could reduce the network cost.
+ rpc collect (stream LogData) returns (Commands) {
+ }
+}
+
+// Log data is collected through file scratcher of agent.
+// Natively, Satellite provides various ways to collect logs.
+message LogData {
+ // [Required] **Service**. Represents a set/group of workloads which provide the same behaviours for incoming requests.
+ //
+ // The logic name represents the service. This would show as a separate node in the topology.
+ // The metrics analyzed from the spans, would be aggregated for this entity as the service level.
+ //
+ // If this is not the first element of the streaming, use the previous not-null name as the service name.
+ string service = 1;
+ // [Optional] **Service Instance**. Each individual workload in the Service group is known as an instance. Like `pods` in Kubernetes, it
+ // doesn't need to be a single OS process, however, if you are using instrument agents, an instance is actually a real OS process.
+ //
+ // The logic name represents the service instance. This would show as a separate node in the instance relationship.
+ // The metrics analyzed from the spans, would be aggregated for this entity as the service instance level.
+ string serviceInstance = 2;
+ // [Optional] **Endpoint**. A path in a service for incoming requests, such as an HTTP URI path or a gRPC service class + method signature.
+ //
+ // The logic name represents the endpoint, which logs belong.
+ string endpoint = 3;
Review comment:
Whether the endpoint name is required or not, in many cases the log does not know which endpoint it belongs to.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking-data-collect-protocol] wu-sheng commented on a change in pull request #35: Add logging collect protocol
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on a change in pull request #35:
URL: https://github.com/apache/skywalking-data-collect-protocol/pull/35#discussion_r533035432
##########
File path: logging/Logging.proto
##########
@@ -0,0 +1,79 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+syntax = "proto3";
+
+option java_multiple_files = true;
+option java_package = "org.apache.skywalking.apm.network.logging.v3";
+option csharp_namespace = "SkyWalking.NetworkProtocol.V3";
+option go_package = "skywalking/network/language/agent/v3";
+
+import "common/Common.proto";
+
+// Report collected logs into the OAP backend
+service LogReportService {
+ // Recommend to report log data in a stream mode.
+ // The service/instance/endpoint of the log could share the previous value if they are not set.
+ // Reporting the logs of same service in the batch mode could reduce the network cost.
+ rpc collect (stream LogData) returns (Commands) {
+ }
+}
+
+// Log data is collected through file scratcher of agent.
+// Natively, Satellite provides various ways to collect logs.
+message LogData {
+ // [Required] **Service**. Represents a set/group of workloads which provide the same behaviours for incoming requests.
+ //
+ // The logic name represents the service. This would show as a separate node in the topology.
+ // The metrics analyzed from the spans, would be aggregated for this entity as the service level.
+ //
+ // If this is not the first element of the streaming, use the previous not-null name as the service name.
+ string service = 1;
Review comment:
You could submit logs of different services in one stream if you want. That is why I put `Required` here. The idea is that is only an optimization.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking-data-collect-protocol] kezhenxu94 commented on a change in pull request #35: Add logging collect protocol
Posted by GitBox <gi...@apache.org>.
kezhenxu94 commented on a change in pull request #35:
URL: https://github.com/apache/skywalking-data-collect-protocol/pull/35#discussion_r533064756
##########
File path: logging/Logging.proto
##########
@@ -0,0 +1,79 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+syntax = "proto3";
+
+option java_multiple_files = true;
+option java_package = "org.apache.skywalking.apm.network.logging.v3";
+option csharp_namespace = "SkyWalking.NetworkProtocol.V3";
+option go_package = "skywalking/network/language/agent/v3";
+
+import "common/Common.proto";
+
+// Report collected logs into the OAP backend
+service LogReportService {
+ // Recommend to report log data in a stream mode.
+ // The service/instance/endpoint of the log could share the previous value if they are not set.
+ // Reporting the logs of same service in the batch mode could reduce the network cost.
+ rpc collect (stream LogData) returns (Commands) {
+ }
+}
+
+// Log data is collected through file scratcher of agent.
+// Natively, Satellite provides various ways to collect logs.
+message LogData {
Review comment:
And, this timestamp is not the timestamp when they were generated, instead, it’s when the logs are sent, which I don’t think is useful
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking-data-collect-protocol] wu-sheng commented on a change in pull request #35: Add logging collect protocol
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on a change in pull request #35:
URL: https://github.com/apache/skywalking-data-collect-protocol/pull/35#discussion_r533119539
##########
File path: logging/Logging.proto
##########
@@ -0,0 +1,107 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+syntax = "proto3";
+
+option java_multiple_files = true;
+option java_package = "org.apache.skywalking.apm.network.logging.v3";
+option csharp_namespace = "SkyWalking.NetworkProtocol.V3";
+option go_package = "skywalking/network/language/agent/v3";
+
+import "common/Common.proto";
+
+// Report collected logs into the OAP backend
+service LogReportService {
+ // Recommend to report log data in a stream mode.
+ // The service/instance/endpoint of the log could share the previous value if they are not set.
+ // Reporting the logs of same service in the batch mode could reduce the network cost.
+ rpc collect (stream LogData) returns (Commands) {
+ }
+}
+
+// Log data is collected through file scratcher of agent.
+// Natively, Satellite provides various ways to collect logs.
+message LogData {
+ // [Optional] The timestamp of the log, in millisecond.
+ // If not set, OAP server would use the received timestamp as log's timestamp, or relies on the OAP server analyzer.
+ int64 timestamp = 1;
+ // [Required] **Service**. Represents a set/group of workloads which provide the same behaviours for incoming requests.
+ //
+ // The logic name represents the service. This would show as a separate node in the topology.
+ // The metrics analyzed from the spans, would be aggregated for this entity as the service level.
+ //
+ // If this is not the first element of the streaming, use the previous not-null name as the service name.
+ string service = 2;
+ // [Optional] **Service Instance**. Each individual workload in the Service group is known as an instance. Like `pods` in Kubernetes, it
+ // doesn't need to be a single OS process, however, if you are using instrument agents, an instance is actually a real OS process.
+ //
+ // The logic name represents the service instance. This would show as a separate node in the instance relationship.
+ // The metrics analyzed from the spans, would be aggregated for this entity as the service instance level.
+ string serviceInstance = 3;
+ // [Optional] **Endpoint**. A path in a service for incoming requests, such as an HTTP URI path or a gRPC service class + method signature.
+ //
+ // The logic name represents the endpoint, which logs belong.
+ string endpoint = 4;
+ // [Required] The content of the log.
+ LogDataBody body = 5;
+ // [Optional] Logs with trace context
+ TraceContext traceContext = 6;
+}
+
+// The content of the log data
+message LogDataBody {
+ // A type to match analyzer(s) at the OAP server.
+ // The data could be analysis at the client side, but could be partial
+ string type = 1;
+ // Content with extendable format.
+ oneof content {
+ TextLog text = 2;
+ JSONLog json = 3;
+ YAMLLog yaml = 4;
+ }
+}
+
+// Literal text log, typically requires regex or split mechanism to filter meaningful info.
+message TextLog {
+ string text = 1;
+}
+
+// JSON formatted log. The json field represents the string could be formatted as a JSON object.
+message JSONLog {
+ string json = 1;
+}
+
+// YAML formatted log. The yaml field represents the string could be formatted as a YAML map.
+message YAMLLog {
+ string yaml = 1;
+}
+
+// Logs with trace context, represent agent system has injects context(IDs) into log text.
+message TraceContext {
+ // [Optional] The available tags. OAP server could provide search/analysis capabilities base on these.
+ repeated KeyStringValuePair tags = 7;
Review comment:
Fixed. Please recheck
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking-data-collect-protocol] kezhenxu94 commented on a change in pull request #35: Add logging collect protocol
Posted by GitBox <gi...@apache.org>.
kezhenxu94 commented on a change in pull request #35:
URL: https://github.com/apache/skywalking-data-collect-protocol/pull/35#discussion_r533034878
##########
File path: logging/Logging.proto
##########
@@ -0,0 +1,79 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+syntax = "proto3";
+
+option java_multiple_files = true;
+option java_package = "org.apache.skywalking.apm.network.logging.v3";
+option csharp_namespace = "SkyWalking.NetworkProtocol.V3";
+option go_package = "skywalking/network/language/agent/v3";
+
+import "common/Common.proto";
+
+// Report collected logs into the OAP backend
+service LogReportService {
+ // Recommend to report log data in a stream mode.
+ // The service/instance/endpoint of the log could share the previous value if they are not set.
+ // Reporting the logs of same service in the batch mode could reduce the network cost.
+ rpc collect (stream LogData) returns (Commands) {
+ }
+}
+
+// Log data is collected through file scratcher of agent.
+// Natively, Satellite provides various ways to collect logs.
+message LogData {
+ // [Required] **Service**. Represents a set/group of workloads which provide the same behaviours for incoming requests.
+ //
+ // The logic name represents the service. This would show as a separate node in the topology.
+ // The metrics analyzed from the spans, would be aggregated for this entity as the service level.
+ //
+ // If this is not the first element of the streaming, use the previous not-null name as the service name.
+ string service = 1;
Review comment:
Is this required? I think you meant only the first element of the stream is non-null, otherwise it's null?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking-data-collect-protocol] EvanLjp removed a comment on pull request #35: Add logging collect protocol
Posted by GitBox <gi...@apache.org>.
EvanLjp removed a comment on pull request #35:
URL: https://github.com/apache/skywalking-data-collect-protocol/pull/35#issuecomment-736178538
Whether the endpoint name is required or not, in many cases the log does not know which endpoint it belongs to.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking-data-collect-protocol] kezhenxu94 commented on a change in pull request #35: Add logging collect protocol
Posted by GitBox <gi...@apache.org>.
kezhenxu94 commented on a change in pull request #35:
URL: https://github.com/apache/skywalking-data-collect-protocol/pull/35#discussion_r533064756
##########
File path: logging/Logging.proto
##########
@@ -0,0 +1,79 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+syntax = "proto3";
+
+option java_multiple_files = true;
+option java_package = "org.apache.skywalking.apm.network.logging.v3";
+option csharp_namespace = "SkyWalking.NetworkProtocol.V3";
+option go_package = "skywalking/network/language/agent/v3";
+
+import "common/Common.proto";
+
+// Report collected logs into the OAP backend
+service LogReportService {
+ // Recommend to report log data in a stream mode.
+ // The service/instance/endpoint of the log could share the previous value if they are not set.
+ // Reporting the logs of same service in the batch mode could reduce the network cost.
+ rpc collect (stream LogData) returns (Commands) {
+ }
+}
+
+// Log data is collected through file scratcher of agent.
+// Natively, Satellite provides various ways to collect logs.
+message LogData {
Review comment:
And, this timestamps is not the timestamp when they were generated, instead, it’s when the logs are sent, which I don’t think is useless
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking-data-collect-protocol] kezhenxu94 commented on a change in pull request #35: Add logging collect protocol
Posted by GitBox <gi...@apache.org>.
kezhenxu94 commented on a change in pull request #35:
URL: https://github.com/apache/skywalking-data-collect-protocol/pull/35#discussion_r533063867
##########
File path: logging/Logging.proto
##########
@@ -0,0 +1,79 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+syntax = "proto3";
+
+option java_multiple_files = true;
+option java_package = "org.apache.skywalking.apm.network.logging.v3";
+option csharp_namespace = "SkyWalking.NetworkProtocol.V3";
+option go_package = "skywalking/network/language/agent/v3";
+
+import "common/Common.proto";
+
+// Report collected logs into the OAP backend
+service LogReportService {
+ // Recommend to report log data in a stream mode.
+ // The service/instance/endpoint of the log could share the previous value if they are not set.
+ // Reporting the logs of same service in the batch mode could reduce the network cost.
+ rpc collect (stream LogData) returns (Commands) {
+ }
+}
+
+// Log data is collected through file scratcher of agent.
+// Natively, Satellite provides various ways to collect logs.
+message LogData {
Review comment:
Is it necessary? IMO most logs in reality already contain a timestamp
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking-data-collect-protocol] wu-sheng commented on a change in pull request #35: Add logging collect protocol
Posted by GitBox <gi...@apache.org>.
wu-sheng commented on a change in pull request #35:
URL: https://github.com/apache/skywalking-data-collect-protocol/pull/35#discussion_r533041164
##########
File path: logging/Logging.proto
##########
@@ -0,0 +1,79 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+syntax = "proto3";
+
+option java_multiple_files = true;
+option java_package = "org.apache.skywalking.apm.network.logging.v3";
+option csharp_namespace = "SkyWalking.NetworkProtocol.V3";
+option go_package = "skywalking/network/language/agent/v3";
+
+import "common/Common.proto";
+
+// Report collected logs into the OAP backend
+service LogReportService {
+ // Recommend to report log data in a stream mode.
+ // The service/instance/endpoint of the log could share the previous value if they are not set.
+ // Reporting the logs of same service in the batch mode could reduce the network cost.
+ rpc collect (stream LogData) returns (Commands) {
+ }
+}
+
+// Log data is collected through file scratcher of agent.
+// Natively, Satellite provides various ways to collect logs.
+message LogData {
Review comment:
Added.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [skywalking-data-collect-protocol] kezhenxu94 commented on a change in pull request #35: Add logging collect protocol
Posted by GitBox <gi...@apache.org>.
kezhenxu94 commented on a change in pull request #35:
URL: https://github.com/apache/skywalking-data-collect-protocol/pull/35#discussion_r533064756
##########
File path: logging/Logging.proto
##########
@@ -0,0 +1,79 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+syntax = "proto3";
+
+option java_multiple_files = true;
+option java_package = "org.apache.skywalking.apm.network.logging.v3";
+option csharp_namespace = "SkyWalking.NetworkProtocol.V3";
+option go_package = "skywalking/network/language/agent/v3";
+
+import "common/Common.proto";
+
+// Report collected logs into the OAP backend
+service LogReportService {
+ // Recommend to report log data in a stream mode.
+ // The service/instance/endpoint of the log could share the previous value if they are not set.
+ // Reporting the logs of same service in the batch mode could reduce the network cost.
+ rpc collect (stream LogData) returns (Commands) {
+ }
+}
+
+// Log data is collected through file scratcher of agent.
+// Natively, Satellite provides various ways to collect logs.
+message LogData {
Review comment:
And, this timestamp is not the timestamp when they were generated, instead, it’s when the logs are sent, which I don’t think is useless
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org