You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@apisix.apache.org by GitBox <gi...@apache.org> on 2021/11/17 09:15:12 UTC
[GitHub] [apisix] shuaijinchao opened a new pull request #5538: feat(plugin): support google cloud logging service
shuaijinchao opened a new pull request #5538:
URL: https://github.com/apache/apisix/pull/5538
### What this PR does / why we need it:
FIX #5474
### Pre-submission checklist:
<!--
Please follow the requirements:
1. Use Draft if the PR is not ready to be reviewed
2. Test is required for the feat/fix PR, unless you have a good reason
3. Doc is required for the feat PR
4. Use a new commit to resolve review instead of `push -f`
5. Use "request review" to notify the reviewer once you have resolved the review
-->
* [x] Did you explain what problem does this PR solve? Or what new features have been added?
* [ ] Have you added corresponding test cases?
* [ ] Have you modified the corresponding document?
* [x] Is this PR backward compatible? **If it is not backward compatible, please discuss on the [mailing list](https://github.com/apache/apisix/tree/master#community) first**
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] shuaijinchao commented on a change in pull request #5538: feat(plugin): support google cloud logging service
Posted by GitBox <gi...@apache.org>.
shuaijinchao commented on a change in pull request #5538:
URL: https://github.com/apache/apisix/pull/5538#discussion_r753890294
##########
File path: t/lib/server.lua
##########
@@ -426,6 +426,83 @@ function _M._well_known_openid_configuration()
ngx.say(openid_data)
end
+function _M.google_logging_token()
+ ngx.req.read_body()
+ local data = ngx.decode_args(ngx.req.get_body_data())
+ local jwt = require("resty.jwt")
+ local rsa_public_key = "-----BEGIN PUBLIC KEY-----\nMFwwDQYJKoZIhvcNAQEBBQADSwAwSAJBAKebDxlvQMGyEesAL1r1nIJBkSdqu3Hr\n7noq/0ukiZqVQLSJPMOv0oxQSutvvK3hoibwGakDOza+xRITB7cs2cECAwEAAQ==\n-----END PUBLIC KEY-----"
Review comment:
OK, updated
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] spacewander merged pull request #5538: feat(plugin): support google cloud logging service
Posted by GitBox <gi...@apache.org>.
spacewander merged pull request #5538:
URL: https://github.com/apache/apisix/pull/5538
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] shuaijinchao commented on a change in pull request #5538: feat(plugin): support google cloud logging service
Posted by GitBox <gi...@apache.org>.
shuaijinchao commented on a change in pull request #5538:
URL: https://github.com/apache/apisix/pull/5538#discussion_r754068936
##########
File path: apisix/utils/log-util.lua
##########
@@ -94,6 +94,7 @@ local function get_full_log(ngx, conf)
local log = {
request = {
+ id = var.request_id,
Review comment:
OK, deleted
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] shuaijinchao commented on a change in pull request #5538: feat(plugin): support google cloud logging service
Posted by GitBox <gi...@apache.org>.
shuaijinchao commented on a change in pull request #5538:
URL: https://github.com/apache/apisix/pull/5538#discussion_r754071035
##########
File path: t/plugin/google-logging.t
##########
@@ -0,0 +1,380 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+use t::APISIX 'no_plan';
+
+repeat_each(1);
+no_long_string();
+no_root_location();
+run_tests;
+
+
+add_block_preprocessor(sub {
+ my ($block) = @_;
+
+ if ((!defined $block->error_log) && (!defined $block->no_error_log)) {
+ $block->set_value("no_error_log", "[error]");
+ }
+});
+
+__DATA__
+
+=== TEST 1: Full configuration verification (Auth File)
+--- config
+ location /t {
+ content_by_lua_block {
+ local plugin = require("apisix.plugins.google-logging")
+ local ok, err = plugin.check_schema({
+ auth_file = "/path/to/apache/apisix/auth.json",
+ resource = {
+ type = "global"
+ },
+ scopes = {
+ "https://www.googleapis.com/auth/logging.admin"
+ },
+ log_id = "syslog",
+ max_retry_count = 0,
+ retry_delay = 1,
+ buffer_duration = 60,
+ inactive_timeout = 10,
+ batch_max_size = 100,
+ })
+
+ if not ok then
+ ngx.say(err)
+ else
+ ngx.say("passed")
+ end
+ }
+ }
+--- request
+GET /t
+--- response_body
+passed
+--- no_error_log
Review comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] shuaijinchao commented on a change in pull request #5538: feat(plugin): support google cloud logging service
Posted by GitBox <gi...@apache.org>.
shuaijinchao commented on a change in pull request #5538:
URL: https://github.com/apache/apisix/pull/5538#discussion_r754073143
##########
File path: docs/en/latest/plugins/google-logging.md
##########
@@ -0,0 +1,155 @@
+---
+title: google-logging
Review comment:
@juzhiyuan I think `google-cloud-logging` is more appropriate, what do you think?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] shuaijinchao commented on a change in pull request #5538: feat(plugin): support google cloud logging service
Posted by GitBox <gi...@apache.org>.
shuaijinchao commented on a change in pull request #5538:
URL: https://github.com/apache/apisix/pull/5538#discussion_r754770975
##########
File path: apisix/plugins/google-cloud-logging.lua
##########
@@ -0,0 +1,326 @@
+--
+-- Licensed to the Apache Software Foundation (ASF) under one or more
+-- contributor license agreements. See the NOTICE file distributed with
+-- this work for additional information regarding copyright ownership.
+-- The ASF licenses this file to You under the Apache License, Version 2.0
+-- (the "License"); you may not use this file except in compliance with
+-- the License. You may obtain a copy of the License at
+--
+-- http://www.apache.org/licenses/LICENSE-2.0
+--
+-- Unless required by applicable law or agreed to in writing, software
+-- distributed under the License is distributed on an "AS IS" BASIS,
+-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+-- See the License for the specific language governing permissions and
+-- limitations under the License.
+--
+
+local core = require("apisix.core")
+local ngx = ngx
+local tostring = tostring
+local ipairs = ipairs
+local os_date = os.date
+local math_floor = math.floor
+local ngx_now = ngx.now
+local ngx_timer_at = ngx.timer.at
+local ngx_update_time = ngx.update_time
+local http = require("resty.http")
+local log_util = require("apisix.utils.log-util")
+local batch_processor = require("apisix.utils.batch-processor")
+local google_oauth = require("apisix.plugins.google-cloud-logging.oauth")
+
+
+local buffers = {}
+local auth_config_cache
+local stale_timer_running
+
+
+local plugin_name = "google-cloud-logging"
+local schema = {
+ type = "object",
+ properties = {
+ auth_config = {
+ type = "object",
+ properties = {
+ private_key = { type = "string" },
+ project_id = { type = "string" },
+ token_uri = {
+ type = "string",
+ default = "https://oauth2.googleapis.com/token"
+ },
+ -- https://developers.google.com/identity/protocols/oauth2/scopes#logging
+ scopes = {
+ type = "array",
+ items = {
+ description = "Google OAuth2 Authorization Scopes",
+ type = "string",
+ },
+ minItems = 1,
+ uniqueItems = true,
+ default = {
+ "https://www.googleapis.com/auth/logging.read",
+ "https://www.googleapis.com/auth/logging.write",
+ "https://www.googleapis.com/auth/logging.admin",
+ "https://www.googleapis.com/auth/cloud-platform"
+ }
+ },
+ entries_uri = {
+ type = "string",
+ default = "https://logging.googleapis.com/v2/entries:write"
+ },
+ ssl_verify = {
+ type = "boolean",
+ default = true
+ },
+ },
+ required = { "private_key", "project_id", "token_uri" }
+ },
+ auth_file = { type = "string" },
+ -- https://cloud.google.com/logging/docs/reference/v2/rest/v2/MonitoredResource
+ resource = {
+ type = "object",
+ properties = {
+ type = { type = "string" },
+ labels = { type = "object" }
+ },
+ default = {
+ type = "global"
+ },
+ required = { "type" }
+ },
+ -- https://cloud.google.com/logging/docs/reference/v2/rest/v2/LogEntry
+ log_id = {
+ type = "string",
+ default = "apisix.apache.org%2Flogs"
+ },
+ max_retry_count = {
+ type = "integer",
+ minimum = 0,
+ default = 0
+ },
+ retry_delay = {
+ type = "integer",
+ minimum = 0,
+ default = 1
+ },
+ buffer_duration = {
+ type = "integer",
+ minimum = 1,
+ default = 60
+ },
+ inactive_timeout = {
+ type = "integer",
+ minimum = 1,
+ default = 10
+ },
+ batch_max_size = {
+ type = "integer",
+ minimum = 1,
+ default = 100
+ },
+ },
+ oneOf = {
+ { required = { "auth_config" } },
+ { required = { "auth_file" } },
+ },
+}
+
+
+-- remove stale objects from the memory after timer expires
+local function remove_stale_objects(premature)
+ if premature then
+ return
+ end
+
+ for key, batch in ipairs(buffers) do
+ if #batch.entry_buffer.entries == 0 and #batch.batch_to_process == 0 then
+ core.log.warn("removing batch processor stale object, route id:", tostring(key))
+ buffers[key] = nil
+ end
+ end
+
+ stale_timer_running = false
+end
+
+
+local function send_to_google(oauth, entries)
+ local http_new = http.new()
+ local access_token = oauth:generate_access_token()
+ if not access_token then
+ return nil, "failed to get google oauth token"
+ end
+
+ local res, err = http_new:request_uri(oauth.entries_uri, {
+ ssl_verify = oauth.ssl_verify,
+ method = "POST",
+ body = core.json.encode({
+ entries = entries,
+ partialSuccess = false,
+ }),
+ headers = {
+ ["Content-Type"] = "application/json",
+ ["Authorization"] = "Bearer " .. access_token,
+ },
+ })
+
+ if err then
+ return nil, "failed to write log to google, " .. err
+ end
+
+ if res.status ~= 200 then
+ return nil, res.body
+ end
+
+ return res.body
+end
+
+
+local function get_auth_config(config)
+ if auth_config_cache then
+ return auth_config_cache
+ end
+
+ if config.auth_config then
+ auth_config_cache = config.auth_config
+ return auth_config_cache
+ end
+
+ if not config.auth_file then
+ return nil, "configuration is not defined"
+ end
+
+ local file_content, err = core.io.get_file(config.auth_file)
+ if not file_content then
+ return nil, "failed to read configuration, file: " .. config.auth_file .. " err:" .. err
+ end
+
+ local config_data
+ config_data, err = core.json.decode(file_content)
+ if not config_data then
+ return nil, "config parse failure, data: " .. file_content .. " , err: " .. err
+ end
+
+ auth_config_cache = config.auth_config
+ return auth_config_cache
+end
+
+
+local function get_logger_buffer(conf, ctx)
+ local oauth_client = google_oauth:new(auth_config_cache)
+
+ local process = function(entries)
+ return send_to_google(oauth_client, entries)
+ end
+
+ local config = {
+ name = conf.name or plugin_name,
+ retry_delay = conf.retry_delay,
+ batch_max_size = conf.batch_max_size,
+ max_retry_count = conf.max_retry_count,
+ buffer_duration = conf.buffer_duration,
+ inactive_timeout = conf.inactive_timeout,
+ route_id = ctx.var.route_id,
+ server_addr = ctx.var.server_addr,
+ }
+
+ local buffer, err = batch_processor:new(process, config)
+
+ if not buffer then
+ return nil, "error when creating the batch processor: " .. err
+ end
+
+ return buffer
+end
+
+
+local function get_utc_timestamp()
+ ngx_update_time()
+ local now = ngx_now()
+ local second = math_floor(now)
+ local millisecond = math_floor((now - second) * 1000)
+ return os_date("!%Y-%m-%dT%T.", second) .. core.string.format("%03dZ", millisecond)
+end
Review comment:
the time function has been moved to log_util
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] tokers commented on a change in pull request #5538: feat(plugin): support google cloud logging service
Posted by GitBox <gi...@apache.org>.
tokers commented on a change in pull request #5538:
URL: https://github.com/apache/apisix/pull/5538#discussion_r752983588
##########
File path: docs/en/latest/plugins/google-logging.md
##########
@@ -0,0 +1,156 @@
+---
+title: google-logging
+---
+
+<!--
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+-->
+
+## Summary
+
+- [**Name**](#name)
+- [**Attributes**](#attributes)
+- [**How To Enable**](#how-to-enable)
+- [**Test Plugin**](#test-plugin)
+- [**Disable Plugin**](#disable-plugin)
+
+## Name
+
+`google-logging` plugin is used to send the request log of `Apache APISIX` to the [Google Cloud Logging Service](https://cloud.google.com/logging/).
+
+This plugin provides the ability to push Log data as a batch to Google Cloud logging Service.
+
+For more info on Batch-Processor in Apache APISIX please refer:
+[Batch-Processor](../batch-processor.md)
+
+## Attributes
+
+| Name | Requirement | Default | Description |
+| ----------------------- | ------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| auth_config | Semi-optional | | one of `auth_config` or `auth_file` must be configured |
+| auth_config.private_key | Required | | the private key parameters of the Google service account |
+| auth_config.project_id | Required | | the project id parameters of the Google service account |
+| auth_config.token_uri | Optional | https://oauth2.googleapis.com/token | the token uri parameters of the Google service account |
+| auth_config.entries_uri | Optional | https://logging.googleapis.com/v2/entries:write | google logging service API |
+| auth_config.scopes | Optional | ["https://www.googleapis.com/auth/logging.read","https://www.googleapis.com/auth/logging.write","https://www.googleapis.com/auth/logging.admin","https://www.googleapis.com/auth/cloud-platform"] | the access scopes parameters of the Google service account, refer to: [OAuth 2.0 Scopes for Google APIs](https://developers.google.com/identity/protocols/oauth2/scopes#logging) |
+| auth_file | Semi-optional | | path to the google service account json file（Semi-optional, one of auth_config or auth_file must be configured） |
+| resource | Optional | {"type": "global"} | the Google monitor resource, refer to: [MonitoredResource](https://cloud.google.com/logging/docs/reference/v2/rest/v2/MonitoredResource) |
+| log_id | Optional | apisix.apache.org%2Flogs | google logging id, refer to: [LogEntry](https://cloud.google.com/logging/docs/reference/v2/rest/v2/LogEntry) |
+| max_retry_count | Optional | 0 | max number of retries before removing from the processing pipe line |
+| retry_delay | Optional | 1 | number of seconds the process execution should be delayed if the execution fails |
+| buffer_duration | Optional | 60 | max age in seconds of the oldest entry in a batch before the batch must be processed |
+| inactive_timeout | Optional | 10 | max age in seconds when the buffer will be flushed if inactive |
+| batch_max_size | Optional | 100 | max size of each batch |
+| | | | |
Review comment:
Remove these redundant lines.
##########
File path: docs/en/latest/plugins/google-logging.md
##########
@@ -0,0 +1,156 @@
+---
+title: google-logging
+---
+
+<!--
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+-->
+
+## Summary
+
+- [**Name**](#name)
+- [**Attributes**](#attributes)
+- [**How To Enable**](#how-to-enable)
+- [**Test Plugin**](#test-plugin)
+- [**Disable Plugin**](#disable-plugin)
+
+## Name
+
+`google-logging` plugin is used to send the request log of `Apache APISIX` to the [Google Cloud Logging Service](https://cloud.google.com/logging/).
Review comment:
```suggestion
`google-logging` plugin is used to send the access logs of `Apache APISIX` to the [Google Cloud Logging Service](https://cloud.google.com/logging/).
```
##########
File path: t/lib/server.lua
##########
@@ -426,6 +426,83 @@ function _M._well_known_openid_configuration()
ngx.say(openid_data)
end
+function _M.google_logging_token()
+ ngx.req.read_body()
+ local data = ngx.decode_args(ngx.req.get_body_data())
+ local jwt = require("resty.jwt")
+ local rsa_public_key = "-----BEGIN PUBLIC KEY-----\nMFwwDQYJKoZIhvcNAQEBBQADSwAwSAJBAKebDxlvQMGyEesAL1r1nIJBkSdqu3Hr\n7noq/0ukiZqVQLSJPMOv0oxQSutvvK3hoibwGakDOza+xRITB7cs2cECAwEAAQ==\n-----END PUBLIC KEY-----"
Review comment:
Can we declare two variables outside of this method? I noticed that the same pub key is used in other methods.
##########
File path: apisix/plugins/google-logging/oauth.lua
##########
@@ -0,0 +1,128 @@
+--
+-- Licensed to the Apache Software Foundation (ASF) under one or more
+-- contributor license agreements. See the NOTICE file distributed with
+-- this work for additional information regarding copyright ownership.
+-- The ASF licenses this file to You under the Apache License, Version 2.0
+-- (the "License"); you may not use this file except in compliance with
+-- the License. You may obtain a copy of the License at
+--
+-- http://www.apache.org/licenses/LICENSE-2.0
+--
+-- Unless required by applicable law or agreed to in writing, software
+-- distributed under the License is distributed on an "AS IS" BASIS,
+-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+-- See the License for the specific language governing permissions and
+-- limitations under the License.
+--
+
+local type = type
+local setmetatable = setmetatable
+
+local ngx_update_time = ngx.update_time
+local ngx_time = ngx.time
+local ngx_encode_args = ngx.encode_args
+
+local core = require("apisix.core")
Review comment:
Put `require` block on the top of this file (after the comments)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] shuaijinchao commented on a change in pull request #5538: feat(plugin): support google cloud logging service
Posted by GitBox <gi...@apache.org>.
shuaijinchao commented on a change in pull request #5538:
URL: https://github.com/apache/apisix/pull/5538#discussion_r753027281
##########
File path: t/lib/server.lua
##########
@@ -426,6 +426,83 @@ function _M._well_known_openid_configuration()
ngx.say(openid_data)
end
+function _M.google_logging_token()
+ ngx.req.read_body()
+ local data = ngx.decode_args(ngx.req.get_body_data())
+ local jwt = require("resty.jwt")
+ local rsa_public_key = "-----BEGIN PUBLIC KEY-----\nMFwwDQYJKoZIhvcNAQEBBQADSwAwSAJBAKebDxlvQMGyEesAL1r1nIJBkSdqu3Hr\n7noq/0ukiZqVQLSJPMOv0oxQSutvvK3hoibwGakDOza+xRITB7cs2cECAwEAAQ==\n-----END PUBLIC KEY-----"
Review comment:
I noticed this problem during the development process. But this public_key is only used by Google Logging related functions? Is it necessary to define it in a public variable in the file?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] spacewander commented on a change in pull request #5538: feat(plugin): support google cloud logging service
Posted by GitBox <gi...@apache.org>.
spacewander commented on a change in pull request #5538:
URL: https://github.com/apache/apisix/pull/5538#discussion_r753787079
##########
File path: apisix/utils/log-util.lua
##########
@@ -94,6 +94,7 @@ local function get_full_log(ngx, conf)
local log = {
request = {
+ id = var.request_id,
Review comment:
Better not add a new field to the common library without full discussion.
##########
File path: apisix/core/utils.lua
##########
@@ -260,6 +260,18 @@ function _M.gethostname()
end
+function _M.get_file(file_name)
Review comment:
Could we put it into core/io.lua and let core.request refer it?
##########
File path: apisix/plugins/google-logging.lua
##########
@@ -0,0 +1,298 @@
+--
+-- Licensed to the Apache Software Foundation (ASF) under one or more
+-- contributor license agreements. See the NOTICE file distributed with
+-- this work for additional information regarding copyright ownership.
+-- The ASF licenses this file to You under the Apache License, Version 2.0
+-- (the "License"); you may not use this file except in compliance with
+-- the License. You may obtain a copy of the License at
+--
+-- http://www.apache.org/licenses/LICENSE-2.0
+--
+-- Unless required by applicable law or agreed to in writing, software
+-- distributed under the License is distributed on an "AS IS" BASIS,
+-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+-- See the License for the specific language governing permissions and
+-- limitations under the License.
+--
+
+local core = require("apisix.core")
+local ngx = ngx
+local tostring = tostring
+local ipairs = ipairs
+local sub_str = string.sub
+local os_date = os.date
+
+local ngx_now = ngx.now
+local ngx_timer_at = ngx.timer.at
+local ngx_update_time = ngx.update_time
+
+local http = require("resty.http")
+local log_util = require("apisix.utils.log-util")
+local batch_processor = require("apisix.utils.batch-processor")
+local google_oauth = require("apisix.plugins.google-logging.oauth")
+
+
+local buffers = {}
+local auth_config_cache
+local stale_timer_running
+
+
+local plugin_name = "google-logging"
+local schema = {
+ type = "object",
+ properties = {
+ auth_config = {
+ type = "object",
+ properties = {
+ private_key = { type = "string" },
+ project_id = { type = "string" },
+ token_uri = {
+ type = "string",
+ default = "https://oauth2.googleapis.com/token"
+ },
+ -- https://developers.google.com/identity/protocols/oauth2/scopes#logging
+ scopes = {
+ type = "array",
+ items = {
+ description = "Google OAuth2 Authorization Scopes",
+ type = "string",
+ },
+ minItems = 1,
+ uniqueItems = true,
+ default = {
+ "https://www.googleapis.com/auth/logging.read",
+ "https://www.googleapis.com/auth/logging.write",
+ "https://www.googleapis.com/auth/logging.admin",
+ "https://www.googleapis.com/auth/cloud-platform"
+ }
+ },
+ entries_uri = {
+ type = "string",
+ default = "https://logging.googleapis.com/v2/entries:write"
+ },
+ },
+ required = { "private_key", "project_id", "token_uri" }
+ },
+ auth_file = { type = "string" },
+ -- https://cloud.google.com/logging/docs/reference/v2/rest/v2/MonitoredResource
+ resource = {
+ type = "object",
+ properties = {
+ type = { type = "string" },
+ labels = { type = "object" }
+ },
+ default = {
+ type = "global"
+ },
+ required = { "type" }
+ },
+ -- https://cloud.google.com/logging/docs/reference/v2/rest/v2/LogEntry
+ log_id = { type = "string", default = "apisix.apache.org%2Flogs" },
+ max_retry_count = { type = "integer", minimum = 0, default = 0 },
+ retry_delay = { type = "integer", minimum = 0, default = 1 },
+ buffer_duration = { type = "integer", minimum = 1, default = 60 },
+ inactive_timeout = { type = "integer", minimum = 1, default = 10 },
+ batch_max_size = { type = "integer", minimum = 1, default = 100 },
+ },
+ oneOf = {
+ { required = { "auth_config" } },
+ { required = { "auth_file" } },
+ },
+}
+
+
+-- remove stale objects from the memory after timer expires
+local function remove_stale_objects(premature)
+ if premature then
+ return
+ end
+
+ for key, batch in ipairs(buffers) do
+ if #batch.entry_buffer.entries == 0 and #batch.batch_to_process == 0 then
+ core.log.warn("removing batch processor stale object, route id:", tostring(key))
+ buffers[key] = nil
+ end
+ end
+
+ stale_timer_running = false
+end
+
+
+local function send_to_google(oauth, entries)
+ local http_new = http.new()
+ local res, err = http_new:request_uri(oauth.entries_uri, {
+ ssl_verify = false,
Review comment:
We need to make ssl_verify configurable and default to true.
##########
File path: docs/en/latest/plugins/google-logging.md
##########
@@ -0,0 +1,155 @@
+---
+title: google-logging
+---
+
+<!--
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+-->
+
+## Summary
+
+- [**Name**](#name)
+- [**Attributes**](#attributes)
+- [**How To Enable**](#how-to-enable)
+- [**Test Plugin**](#test-plugin)
+- [**Disable Plugin**](#disable-plugin)
+
+## Name
+
+`google-logging` plugin is used to send the access log of `Apache APISIX` to the [Google Cloud Logging Service](https://cloud.google.com/logging/).
+
+This plugin provides the ability to push Log data as a batch to Google Cloud logging Service.
+
+For more info on Batch-Processor in Apache APISIX please refer:
+[Batch-Processor](../batch-processor.md)
+
+## Attributes
+
+| Name | Requirement | Default | Description |
+| ----------------------- | ------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| auth_config | Semi-optional | | one of `auth_config` or `auth_file` must be configured |
+| auth_config.private_key | Required | | the private key parameters of the Google service account |
+| auth_config.project_id | Required | | the project id parameters of the Google service account |
+| auth_config.token_uri | Optional | https://oauth2.googleapis.com/token | the token uri parameters of the Google service account |
+| auth_config.entries_uri | Optional | https://logging.googleapis.com/v2/entries:write | google logging service API |
+| auth_config.scopes | Optional | ["https://www.googleapis.com/auth/logging.read","https://www.googleapis.com/auth/logging.write","https://www.googleapis.com/auth/logging.admin","https://www.googleapis.com/auth/cloud-platform"] | the access scopes parameters of the Google service account, refer to: [OAuth 2.0 Scopes for Google APIs](https://developers.google.com/identity/protocols/oauth2/scopes#logging) |
+| auth_file | Semi-optional | | path to the google service account json file（Semi-optional, one of auth_config or auth_file must be configured） |
+| resource | Optional | {"type": "global"} | the Google monitor resource, refer to: [MonitoredResource](https://cloud.google.com/logging/docs/reference/v2/rest/v2/MonitoredResource) |
+| log_id | Optional | apisix.apache.org%2Flogs | google logging id, refer to: [LogEntry](https://cloud.google.com/logging/docs/reference/v2/rest/v2/LogEntry) |
+| max_retry_count | Optional | 0 | max number of retries before removing from the processing pipe line |
+| retry_delay | Optional | 1 | number of seconds the process execution should be delayed if the execution fails |
+| buffer_duration | Optional | 60 | max age in seconds of the oldest entry in a batch before the batch must be processed |
+| inactive_timeout | Optional | 10 | max age in seconds when the buffer will be flushed if inactive |
+| batch_max_size | Optional | 100 | max size of each batch |
+
+## How To Enable
+
+The following is an example on how to enable the `google-logging` for a specific route.
+
+### Full configuration
+
+```shell
+curl http://127.0.0.1:9080/apisix/admin/routes/1 -H 'X-API-KEY: edd1c9f034335f136f87ad84b625c8f1' -X PUT -d '
+{
+ "plugins": {
+ "google-logging": {
+ "auth_config":{
+ "project_id":"apisix",
+ "private_key":"-----BEGIN RSA PRIVATE KEY-----KEY-----END RSA PRIVATE KEY-----",
Review comment:
Could we use a valid pkey?
##########
File path: apisix/plugins/google-logging.lua
##########
@@ -0,0 +1,298 @@
+--
+-- Licensed to the Apache Software Foundation (ASF) under one or more
+-- contributor license agreements. See the NOTICE file distributed with
+-- this work for additional information regarding copyright ownership.
+-- The ASF licenses this file to You under the Apache License, Version 2.0
+-- (the "License"); you may not use this file except in compliance with
+-- the License. You may obtain a copy of the License at
+--
+-- http://www.apache.org/licenses/LICENSE-2.0
+--
+-- Unless required by applicable law or agreed to in writing, software
+-- distributed under the License is distributed on an "AS IS" BASIS,
+-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+-- See the License for the specific language governing permissions and
+-- limitations under the License.
+--
+
+local core = require("apisix.core")
+local ngx = ngx
+local tostring = tostring
+local ipairs = ipairs
+local sub_str = string.sub
+local os_date = os.date
+
+local ngx_now = ngx.now
+local ngx_timer_at = ngx.timer.at
+local ngx_update_time = ngx.update_time
+
+local http = require("resty.http")
+local log_util = require("apisix.utils.log-util")
+local batch_processor = require("apisix.utils.batch-processor")
+local google_oauth = require("apisix.plugins.google-logging.oauth")
+
+
+local buffers = {}
+local auth_config_cache
+local stale_timer_running
+
+
+local plugin_name = "google-logging"
+local schema = {
+ type = "object",
+ properties = {
+ auth_config = {
+ type = "object",
+ properties = {
+ private_key = { type = "string" },
+ project_id = { type = "string" },
+ token_uri = {
+ type = "string",
+ default = "https://oauth2.googleapis.com/token"
+ },
+ -- https://developers.google.com/identity/protocols/oauth2/scopes#logging
+ scopes = {
+ type = "array",
+ items = {
+ description = "Google OAuth2 Authorization Scopes",
+ type = "string",
+ },
+ minItems = 1,
+ uniqueItems = true,
+ default = {
+ "https://www.googleapis.com/auth/logging.read",
+ "https://www.googleapis.com/auth/logging.write",
+ "https://www.googleapis.com/auth/logging.admin",
+ "https://www.googleapis.com/auth/cloud-platform"
+ }
+ },
+ entries_uri = {
+ type = "string",
+ default = "https://logging.googleapis.com/v2/entries:write"
+ },
+ },
+ required = { "private_key", "project_id", "token_uri" }
+ },
+ auth_file = { type = "string" },
+ -- https://cloud.google.com/logging/docs/reference/v2/rest/v2/MonitoredResource
+ resource = {
+ type = "object",
+ properties = {
+ type = { type = "string" },
+ labels = { type = "object" }
+ },
+ default = {
+ type = "global"
+ },
+ required = { "type" }
+ },
+ -- https://cloud.google.com/logging/docs/reference/v2/rest/v2/LogEntry
+ log_id = { type = "string", default = "apisix.apache.org%2Flogs" },
+ max_retry_count = { type = "integer", minimum = 0, default = 0 },
+ retry_delay = { type = "integer", minimum = 0, default = 1 },
+ buffer_duration = { type = "integer", minimum = 1, default = 60 },
+ inactive_timeout = { type = "integer", minimum = 1, default = 10 },
+ batch_max_size = { type = "integer", minimum = 1, default = 100 },
+ },
+ oneOf = {
+ { required = { "auth_config" } },
+ { required = { "auth_file" } },
+ },
+}
+
+
+-- remove stale objects from the memory after timer expires
+local function remove_stale_objects(premature)
+ if premature then
+ return
+ end
+
+ for key, batch in ipairs(buffers) do
+ if #batch.entry_buffer.entries == 0 and #batch.batch_to_process == 0 then
+ core.log.warn("removing batch processor stale object, route id:", tostring(key))
+ buffers[key] = nil
+ end
+ end
+
+ stale_timer_running = false
+end
+
+
+local function send_to_google(oauth, entries)
+ local http_new = http.new()
+ local res, err = http_new:request_uri(oauth.entries_uri, {
+ ssl_verify = false,
+ method = "POST",
+ body = core.json.encode({
+ entries = entries,
+ partialSuccess = false,
+ }),
+ headers = {
+ ["Content-Type"] = "application/json",
+ ["Authorization"] = "Bearer " .. oauth:get_access_token(),
+ },
+ })
+
+ if err then
+ return nil, "failed to write log to google, ", err
+ end
+
+ if res.status ~= 200 then
+ return nil, res.body
+ end
+
+ return res.body
+end
+
+
+local function get_auth_config(config)
+ local err
+ local auth_config = {}
+ if config.auth_config then
+ auth_config = config.auth_config
+ end
+
+ if config.auth_file then
+ local file_content
+ file_content, err = core.utils.get_file(config.auth_file)
+ if file_content then
+ auth_config = core.json.decode(file_content)
+ end
+ end
+
+ if err then
+ return nil, err
+ end
+
+ return auth_config
+end
+
+
+local function get_logger_buffer(conf, ctx)
+ local oauth_client = google_oauth:new(auth_config_cache)
+
+ local process = function(entries)
+ return send_to_google(oauth_client, entries)
+ end
+
+ local config = {
+ name = conf.name or plugin_name,
+ retry_delay = conf.retry_delay,
+ batch_max_size = conf.batch_max_size,
+ max_retry_count = conf.max_retry_count,
+ buffer_duration = conf.buffer_duration,
+ inactive_timeout = conf.inactive_timeout,
+ route_id = ctx.var.route_id,
+ server_addr = ctx.var.server_addr,
+ }
+
+ local buffer, err = batch_processor:new(process, config)
+
+ if not buffer then
+ return nil, "error when creating the batch processor: " .. err
+ end
+
+ return buffer
+end
+
+
+local function get_utc_timestamp()
+ ngx_update_time()
+ local now = tostring(ngx_now())
+ local pos = core.string.rfind_char(now, ".", #now - 1)
+ local second = now
+ local millisecond = 0
+ if pos then
+ second = sub_str(now, 1, pos - 1)
+ millisecond = sub_str(now, pos + 1)
+ end
+ return os_date("!%Y-%m-%dT%T.", second) .. core.string.format("%03dZ", millisecond)
+end
+
+
+local function get_logger_entry(conf)
+ if not auth_config_cache then
Review comment:
It would be better to associate the auth config cache into get_auth_config.
##########
File path: Makefile
##########
@@ -303,6 +303,9 @@ install: runtime
$(ENV_INSTALL) -d $(ENV_INST_LUADIR)/apisix/plugins/limit-count
$(ENV_INSTALL) apisix/plugins/limit-count/*.lua $(ENV_INST_LUADIR)/apisix/plugins/limit-count/
+ $(ENV_INSTALL) -d $(ENV_INST_LUADIR)/apisix/plugins/google-logging
+ $(ENV_INSTALL) apisix/plugins/google-logging/*.lua $(ENV_INST_LUADIR)/apisix/plugins/google-logging/
Review comment:
We should use the name `google-cloud-logging` instead of google-logging? It is confusing if we don't mention which google product we are working with.
##########
File path: apisix/plugins/google-logging.lua
##########
@@ -0,0 +1,298 @@
+--
+-- Licensed to the Apache Software Foundation (ASF) under one or more
+-- contributor license agreements. See the NOTICE file distributed with
+-- this work for additional information regarding copyright ownership.
+-- The ASF licenses this file to You under the Apache License, Version 2.0
+-- (the "License"); you may not use this file except in compliance with
+-- the License. You may obtain a copy of the License at
+--
+-- http://www.apache.org/licenses/LICENSE-2.0
+--
+-- Unless required by applicable law or agreed to in writing, software
+-- distributed under the License is distributed on an "AS IS" BASIS,
+-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+-- See the License for the specific language governing permissions and
+-- limitations under the License.
+--
+
+local core = require("apisix.core")
+local ngx = ngx
+local tostring = tostring
+local ipairs = ipairs
+local sub_str = string.sub
+local os_date = os.date
+
+local ngx_now = ngx.now
+local ngx_timer_at = ngx.timer.at
+local ngx_update_time = ngx.update_time
+
+local http = require("resty.http")
+local log_util = require("apisix.utils.log-util")
+local batch_processor = require("apisix.utils.batch-processor")
+local google_oauth = require("apisix.plugins.google-logging.oauth")
+
+
+local buffers = {}
+local auth_config_cache
+local stale_timer_running
+
+
+local plugin_name = "google-logging"
+local schema = {
+ type = "object",
+ properties = {
+ auth_config = {
+ type = "object",
+ properties = {
+ private_key = { type = "string" },
+ project_id = { type = "string" },
+ token_uri = {
+ type = "string",
+ default = "https://oauth2.googleapis.com/token"
+ },
+ -- https://developers.google.com/identity/protocols/oauth2/scopes#logging
+ scopes = {
+ type = "array",
+ items = {
+ description = "Google OAuth2 Authorization Scopes",
+ type = "string",
+ },
+ minItems = 1,
+ uniqueItems = true,
+ default = {
+ "https://www.googleapis.com/auth/logging.read",
+ "https://www.googleapis.com/auth/logging.write",
+ "https://www.googleapis.com/auth/logging.admin",
+ "https://www.googleapis.com/auth/cloud-platform"
+ }
+ },
+ entries_uri = {
+ type = "string",
+ default = "https://logging.googleapis.com/v2/entries:write"
+ },
+ },
+ required = { "private_key", "project_id", "token_uri" }
+ },
+ auth_file = { type = "string" },
+ -- https://cloud.google.com/logging/docs/reference/v2/rest/v2/MonitoredResource
+ resource = {
+ type = "object",
+ properties = {
+ type = { type = "string" },
+ labels = { type = "object" }
+ },
+ default = {
+ type = "global"
+ },
+ required = { "type" }
+ },
+ -- https://cloud.google.com/logging/docs/reference/v2/rest/v2/LogEntry
+ log_id = { type = "string", default = "apisix.apache.org%2Flogs" },
+ max_retry_count = { type = "integer", minimum = 0, default = 0 },
+ retry_delay = { type = "integer", minimum = 0, default = 1 },
+ buffer_duration = { type = "integer", minimum = 1, default = 60 },
+ inactive_timeout = { type = "integer", minimum = 1, default = 10 },
+ batch_max_size = { type = "integer", minimum = 1, default = 100 },
+ },
+ oneOf = {
+ { required = { "auth_config" } },
+ { required = { "auth_file" } },
+ },
+}
+
+
+-- remove stale objects from the memory after timer expires
+local function remove_stale_objects(premature)
+ if premature then
+ return
+ end
+
+ for key, batch in ipairs(buffers) do
+ if #batch.entry_buffer.entries == 0 and #batch.batch_to_process == 0 then
+ core.log.warn("removing batch processor stale object, route id:", tostring(key))
+ buffers[key] = nil
+ end
+ end
+
+ stale_timer_running = false
+end
+
+
+local function send_to_google(oauth, entries)
+ local http_new = http.new()
+ local res, err = http_new:request_uri(oauth.entries_uri, {
+ ssl_verify = false,
+ method = "POST",
+ body = core.json.encode({
+ entries = entries,
+ partialSuccess = false,
+ }),
+ headers = {
+ ["Content-Type"] = "application/json",
+ ["Authorization"] = "Bearer " .. oauth:get_access_token(),
+ },
+ })
+
+ if err then
+ return nil, "failed to write log to google, ", err
+ end
+
+ if res.status ~= 200 then
+ return nil, res.body
+ end
+
+ return res.body
+end
+
+
+local function get_auth_config(config)
+ local err
+ local auth_config = {}
+ if config.auth_config then
+ auth_config = config.auth_config
+ end
+
+ if config.auth_file then
+ local file_content
+ file_content, err = core.utils.get_file(config.auth_file)
+ if file_content then
+ auth_config = core.json.decode(file_content)
+ end
+ end
+
+ if err then
+ return nil, err
+ end
+
+ return auth_config
+end
+
+
+local function get_logger_buffer(conf, ctx)
+ local oauth_client = google_oauth:new(auth_config_cache)
+
+ local process = function(entries)
+ return send_to_google(oauth_client, entries)
+ end
+
+ local config = {
+ name = conf.name or plugin_name,
+ retry_delay = conf.retry_delay,
+ batch_max_size = conf.batch_max_size,
+ max_retry_count = conf.max_retry_count,
+ buffer_duration = conf.buffer_duration,
+ inactive_timeout = conf.inactive_timeout,
+ route_id = ctx.var.route_id,
+ server_addr = ctx.var.server_addr,
+ }
+
+ local buffer, err = batch_processor:new(process, config)
+
+ if not buffer then
+ return nil, "error when creating the batch processor: " .. err
+ end
+
+ return buffer
+end
+
+
+local function get_utc_timestamp()
+ ngx_update_time()
+ local now = tostring(ngx_now())
+ local pos = core.string.rfind_char(now, ".", #now - 1)
+ local second = now
+ local millisecond = 0
+ if pos then
+ second = sub_str(now, 1, pos - 1)
+ millisecond = sub_str(now, pos + 1)
+ end
+ return os_date("!%Y-%m-%dT%T.", second) .. core.string.format("%03dZ", millisecond)
+end
+
+
+local function get_logger_entry(conf)
+ if not auth_config_cache then
+ local auth_config, err = get_auth_config(conf)
+ if err or not auth_config.project_id or not auth_config.private_key then
+ return nil, "failed to get google authentication configuration" .. err
+ end
+
+ auth_config_cache = auth_config
+ end
+
+ local entry = log_util.get_full_log(ngx, conf)
+ local google_entry = {
+ httpRequest = {
+ requestMethod = entry.request.method,
+ requestUrl = entry.request.url,
+ requestSize = entry.request.size,
+ status = entry.response.status,
+ responseSize = entry.response.size,
+ userAgent = entry.request.headers and entry.request.headers["user-agent"],
+ remoteIp = entry.client_ip,
+ serverIp = entry.upstream,
+ latency = tostring(core.string.format("%0.3f", entry.latency / 1000)) .. "s"
+ },
+ jsonPayload = {
+ route_id = entry.route_id,
+ service_id = entry.service_id,
+ },
+ labels = {
+ source = "apache-apisix-google-logging"
+ },
+ timestamp = get_utc_timestamp(),
+ resource = conf.resource,
+ insertId = entry.request.id,
Review comment:
Can we add the var.request_id directly?
##########
File path: docs/en/latest/plugins/google-logging.md
##########
@@ -0,0 +1,155 @@
+---
+title: google-logging
+---
+
+<!--
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+-->
+
+## Summary
+
+- [**Name**](#name)
+- [**Attributes**](#attributes)
+- [**How To Enable**](#how-to-enable)
+- [**Test Plugin**](#test-plugin)
+- [**Disable Plugin**](#disable-plugin)
+
+## Name
+
+`google-logging` plugin is used to send the access log of `Apache APISIX` to the [Google Cloud Logging Service](https://cloud.google.com/logging/).
+
+This plugin provides the ability to push Log data as a batch to Google Cloud logging Service.
Review comment:
```suggestion
This plugin provides the ability to push log data as a batch to Google Cloud logging Service.
```
##########
File path: apisix/plugins/google-logging.lua
##########
@@ -0,0 +1,298 @@
+--
+-- Licensed to the Apache Software Foundation (ASF) under one or more
+-- contributor license agreements. See the NOTICE file distributed with
+-- this work for additional information regarding copyright ownership.
+-- The ASF licenses this file to You under the Apache License, Version 2.0
+-- (the "License"); you may not use this file except in compliance with
+-- the License. You may obtain a copy of the License at
+--
+-- http://www.apache.org/licenses/LICENSE-2.0
+--
+-- Unless required by applicable law or agreed to in writing, software
+-- distributed under the License is distributed on an "AS IS" BASIS,
+-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+-- See the License for the specific language governing permissions and
+-- limitations under the License.
+--
+
+local core = require("apisix.core")
+local ngx = ngx
+local tostring = tostring
+local ipairs = ipairs
+local sub_str = string.sub
+local os_date = os.date
+
+local ngx_now = ngx.now
+local ngx_timer_at = ngx.timer.at
+local ngx_update_time = ngx.update_time
+
+local http = require("resty.http")
+local log_util = require("apisix.utils.log-util")
+local batch_processor = require("apisix.utils.batch-processor")
+local google_oauth = require("apisix.plugins.google-logging.oauth")
+
+
+local buffers = {}
+local auth_config_cache
+local stale_timer_running
+
+
+local plugin_name = "google-logging"
+local schema = {
+ type = "object",
+ properties = {
+ auth_config = {
+ type = "object",
+ properties = {
+ private_key = { type = "string" },
+ project_id = { type = "string" },
+ token_uri = {
+ type = "string",
+ default = "https://oauth2.googleapis.com/token"
+ },
+ -- https://developers.google.com/identity/protocols/oauth2/scopes#logging
+ scopes = {
+ type = "array",
+ items = {
+ description = "Google OAuth2 Authorization Scopes",
+ type = "string",
+ },
+ minItems = 1,
+ uniqueItems = true,
+ default = {
+ "https://www.googleapis.com/auth/logging.read",
+ "https://www.googleapis.com/auth/logging.write",
+ "https://www.googleapis.com/auth/logging.admin",
+ "https://www.googleapis.com/auth/cloud-platform"
+ }
+ },
+ entries_uri = {
+ type = "string",
+ default = "https://logging.googleapis.com/v2/entries:write"
+ },
+ },
+ required = { "private_key", "project_id", "token_uri" }
+ },
+ auth_file = { type = "string" },
+ -- https://cloud.google.com/logging/docs/reference/v2/rest/v2/MonitoredResource
+ resource = {
+ type = "object",
+ properties = {
+ type = { type = "string" },
+ labels = { type = "object" }
+ },
+ default = {
+ type = "global"
+ },
+ required = { "type" }
+ },
+ -- https://cloud.google.com/logging/docs/reference/v2/rest/v2/LogEntry
+ log_id = { type = "string", default = "apisix.apache.org%2Flogs" },
+ max_retry_count = { type = "integer", minimum = 0, default = 0 },
+ retry_delay = { type = "integer", minimum = 0, default = 1 },
+ buffer_duration = { type = "integer", minimum = 1, default = 60 },
+ inactive_timeout = { type = "integer", minimum = 1, default = 10 },
+ batch_max_size = { type = "integer", minimum = 1, default = 100 },
+ },
+ oneOf = {
+ { required = { "auth_config" } },
+ { required = { "auth_file" } },
+ },
+}
+
+
+-- remove stale objects from the memory after timer expires
+local function remove_stale_objects(premature)
+ if premature then
+ return
+ end
+
+ for key, batch in ipairs(buffers) do
+ if #batch.entry_buffer.entries == 0 and #batch.batch_to_process == 0 then
+ core.log.warn("removing batch processor stale object, route id:", tostring(key))
+ buffers[key] = nil
+ end
+ end
+
+ stale_timer_running = false
+end
+
+
+local function send_to_google(oauth, entries)
+ local http_new = http.new()
+ local res, err = http_new:request_uri(oauth.entries_uri, {
+ ssl_verify = false,
+ method = "POST",
+ body = core.json.encode({
+ entries = entries,
+ partialSuccess = false,
+ }),
+ headers = {
+ ["Content-Type"] = "application/json",
+ ["Authorization"] = "Bearer " .. oauth:get_access_token(),
+ },
+ })
+
+ if err then
+ return nil, "failed to write log to google, ", err
+ end
+
+ if res.status ~= 200 then
+ return nil, res.body
+ end
+
+ return res.body
+end
+
+
+local function get_auth_config(config)
+ local err
+ local auth_config = {}
+ if config.auth_config then
+ auth_config = config.auth_config
+ end
+
+ if config.auth_file then
+ local file_content
+ file_content, err = core.utils.get_file(config.auth_file)
+ if file_content then
+ auth_config = core.json.decode(file_content)
Review comment:
Better to check the decode err.
##########
File path: docs/en/latest/plugins/google-logging.md
##########
@@ -0,0 +1,155 @@
+---
+title: google-logging
+---
+
+<!--
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+-->
+
+## Summary
+
+- [**Name**](#name)
+- [**Attributes**](#attributes)
+- [**How To Enable**](#how-to-enable)
+- [**Test Plugin**](#test-plugin)
+- [**Disable Plugin**](#disable-plugin)
+
+## Name
+
+`google-logging` plugin is used to send the access log of `Apache APISIX` to the [Google Cloud Logging Service](https://cloud.google.com/logging/).
+
+This plugin provides the ability to push Log data as a batch to Google Cloud logging Service.
+
+For more info on Batch-Processor in Apache APISIX please refer:
+[Batch-Processor](../batch-processor.md)
+
+## Attributes
+
+| Name | Requirement | Default | Description |
+| ----------------------- | ------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| auth_config | Semi-optional | | one of `auth_config` or `auth_file` must be configured |
+| auth_config.private_key | Required | | the private key parameters of the Google service account |
+| auth_config.project_id | Required | | the project id parameters of the Google service account |
+| auth_config.token_uri | Optional | https://oauth2.googleapis.com/token | the token uri parameters of the Google service account |
+| auth_config.entries_uri | Optional | https://logging.googleapis.com/v2/entries:write | google logging service API |
+| auth_config.scopes | Optional | ["https://www.googleapis.com/auth/logging.read","https://www.googleapis.com/auth/logging.write","https://www.googleapis.com/auth/logging.admin","https://www.googleapis.com/auth/cloud-platform"] | the access scopes parameters of the Google service account, refer to: [OAuth 2.0 Scopes for Google APIs](https://developers.google.com/identity/protocols/oauth2/scopes#logging) |
+| auth_file | Semi-optional | | path to the google service account json file（Semi-optional, one of auth_config or auth_file must be configured） |
+| resource | Optional | {"type": "global"} | the Google monitor resource, refer to: [MonitoredResource](https://cloud.google.com/logging/docs/reference/v2/rest/v2/MonitoredResource) |
+| log_id | Optional | apisix.apache.org%2Flogs | google logging id, refer to: [LogEntry](https://cloud.google.com/logging/docs/reference/v2/rest/v2/LogEntry) |
+| max_retry_count | Optional | 0 | max number of retries before removing from the processing pipe line |
+| retry_delay | Optional | 1 | number of seconds the process execution should be delayed if the execution fails |
+| buffer_duration | Optional | 60 | max age in seconds of the oldest entry in a batch before the batch must be processed |
+| inactive_timeout | Optional | 10 | max age in seconds when the buffer will be flushed if inactive |
+| batch_max_size | Optional | 100 | max size of each batch |
+
+## How To Enable
+
+The following is an example on how to enable the `google-logging` for a specific route.
+
+### Full configuration
+
+```shell
+curl http://127.0.0.1:9080/apisix/admin/routes/1 -H 'X-API-KEY: edd1c9f034335f136f87ad84b625c8f1' -X PUT -d '
+{
+ "plugins": {
+ "google-logging": {
+ "auth_config":{
+ "project_id":"apisix",
+ "private_key":"-----BEGIN RSA PRIVATE KEY-----KEY-----END RSA PRIVATE KEY-----",
+ "token_uri":"https://oauth2.googleapis.com/token",
+ "scopes":[
+ "https://www.googleapis.com/auth/logging.admin"
+ ],
+ "entries_uri":"https://logging.googleapis.com/v2/entries:write"
+ },
+ "resource":{
+ "type":"global"
+ },
+ "log_id":"syslog",
Review comment:
The log_id is incorrect?
##########
File path: docs/en/latest/plugins/google-logging.md
##########
@@ -0,0 +1,155 @@
+---
+title: google-logging
+---
+
+<!--
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+-->
+
+## Summary
+
+- [**Name**](#name)
+- [**Attributes**](#attributes)
+- [**How To Enable**](#how-to-enable)
+- [**Test Plugin**](#test-plugin)
+- [**Disable Plugin**](#disable-plugin)
+
+## Name
+
+`google-logging` plugin is used to send the access log of `Apache APISIX` to the [Google Cloud Logging Service](https://cloud.google.com/logging/).
+
+This plugin provides the ability to push Log data as a batch to Google Cloud logging Service.
+
+For more info on Batch-Processor in Apache APISIX please refer:
+[Batch-Processor](../batch-processor.md)
+
+## Attributes
+
+| Name | Requirement | Default | Description |
+| ----------------------- | ------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| auth_config | Semi-optional | | one of `auth_config` or `auth_file` must be configured |
+| auth_config.private_key | Required | | the private key parameters of the Google service account |
+| auth_config.project_id | Required | | the project id parameters of the Google service account |
+| auth_config.token_uri | Optional | https://oauth2.googleapis.com/token | the token uri parameters of the Google service account |
+| auth_config.entries_uri | Optional | https://logging.googleapis.com/v2/entries:write | google logging service API |
+| auth_config.scopes | Optional | ["https://www.googleapis.com/auth/logging.read","https://www.googleapis.com/auth/logging.write","https://www.googleapis.com/auth/logging.admin","https://www.googleapis.com/auth/cloud-platform"] | the access scopes parameters of the Google service account, refer to: [OAuth 2.0 Scopes for Google APIs](https://developers.google.com/identity/protocols/oauth2/scopes#logging) |
+| auth_file | Semi-optional | | path to the google service account json file（Semi-optional, one of auth_config or auth_file must be configured） |
+| resource | Optional | {"type": "global"} | the Google monitor resource, refer to: [MonitoredResource](https://cloud.google.com/logging/docs/reference/v2/rest/v2/MonitoredResource) |
+| log_id | Optional | apisix.apache.org%2Flogs | google logging id, refer to: [LogEntry](https://cloud.google.com/logging/docs/reference/v2/rest/v2/LogEntry) |
+| max_retry_count | Optional | 0 | max number of retries before removing from the processing pipe line |
+| retry_delay | Optional | 1 | number of seconds the process execution should be delayed if the execution fails |
+| buffer_duration | Optional | 60 | max age in seconds of the oldest entry in a batch before the batch must be processed |
+| inactive_timeout | Optional | 10 | max age in seconds when the buffer will be flushed if inactive |
+| batch_max_size | Optional | 100 | max size of each batch |
+
+## How To Enable
+
+The following is an example on how to enable the `google-logging` for a specific route.
Review comment:
```suggestion
The following is an example of how to enable the `google-logging` for a specific route.
```
##########
File path: apisix/plugins/google-logging.lua
##########
@@ -0,0 +1,298 @@
+--
+-- Licensed to the Apache Software Foundation (ASF) under one or more
+-- contributor license agreements. See the NOTICE file distributed with
+-- this work for additional information regarding copyright ownership.
+-- The ASF licenses this file to You under the Apache License, Version 2.0
+-- (the "License"); you may not use this file except in compliance with
+-- the License. You may obtain a copy of the License at
+--
+-- http://www.apache.org/licenses/LICENSE-2.0
+--
+-- Unless required by applicable law or agreed to in writing, software
+-- distributed under the License is distributed on an "AS IS" BASIS,
+-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+-- See the License for the specific language governing permissions and
+-- limitations under the License.
+--
+
+local core = require("apisix.core")
+local ngx = ngx
+local tostring = tostring
+local ipairs = ipairs
+local sub_str = string.sub
+local os_date = os.date
+
+local ngx_now = ngx.now
+local ngx_timer_at = ngx.timer.at
+local ngx_update_time = ngx.update_time
+
+local http = require("resty.http")
+local log_util = require("apisix.utils.log-util")
+local batch_processor = require("apisix.utils.batch-processor")
+local google_oauth = require("apisix.plugins.google-logging.oauth")
+
+
+local buffers = {}
+local auth_config_cache
+local stale_timer_running
+
+
+local plugin_name = "google-logging"
+local schema = {
+ type = "object",
+ properties = {
+ auth_config = {
+ type = "object",
+ properties = {
+ private_key = { type = "string" },
+ project_id = { type = "string" },
+ token_uri = {
+ type = "string",
+ default = "https://oauth2.googleapis.com/token"
+ },
+ -- https://developers.google.com/identity/protocols/oauth2/scopes#logging
+ scopes = {
+ type = "array",
+ items = {
+ description = "Google OAuth2 Authorization Scopes",
+ type = "string",
+ },
+ minItems = 1,
+ uniqueItems = true,
+ default = {
+ "https://www.googleapis.com/auth/logging.read",
+ "https://www.googleapis.com/auth/logging.write",
+ "https://www.googleapis.com/auth/logging.admin",
+ "https://www.googleapis.com/auth/cloud-platform"
+ }
+ },
+ entries_uri = {
+ type = "string",
+ default = "https://logging.googleapis.com/v2/entries:write"
+ },
+ },
+ required = { "private_key", "project_id", "token_uri" }
+ },
+ auth_file = { type = "string" },
+ -- https://cloud.google.com/logging/docs/reference/v2/rest/v2/MonitoredResource
+ resource = {
+ type = "object",
+ properties = {
+ type = { type = "string" },
+ labels = { type = "object" }
+ },
+ default = {
+ type = "global"
+ },
+ required = { "type" }
+ },
+ -- https://cloud.google.com/logging/docs/reference/v2/rest/v2/LogEntry
+ log_id = { type = "string", default = "apisix.apache.org%2Flogs" },
+ max_retry_count = { type = "integer", minimum = 0, default = 0 },
+ retry_delay = { type = "integer", minimum = 0, default = 1 },
+ buffer_duration = { type = "integer", minimum = 1, default = 60 },
+ inactive_timeout = { type = "integer", minimum = 1, default = 10 },
+ batch_max_size = { type = "integer", minimum = 1, default = 100 },
+ },
+ oneOf = {
+ { required = { "auth_config" } },
+ { required = { "auth_file" } },
+ },
+}
+
+
+-- remove stale objects from the memory after timer expires
+local function remove_stale_objects(premature)
+ if premature then
+ return
+ end
+
+ for key, batch in ipairs(buffers) do
+ if #batch.entry_buffer.entries == 0 and #batch.batch_to_process == 0 then
+ core.log.warn("removing batch processor stale object, route id:", tostring(key))
+ buffers[key] = nil
+ end
+ end
+
+ stale_timer_running = false
+end
+
+
+local function send_to_google(oauth, entries)
+ local http_new = http.new()
+ local res, err = http_new:request_uri(oauth.entries_uri, {
+ ssl_verify = false,
+ method = "POST",
+ body = core.json.encode({
+ entries = entries,
+ partialSuccess = false,
+ }),
+ headers = {
+ ["Content-Type"] = "application/json",
+ ["Authorization"] = "Bearer " .. oauth:get_access_token(),
+ },
+ })
+
+ if err then
+ return nil, "failed to write log to google, ", err
+ end
+
+ if res.status ~= 200 then
+ return nil, res.body
+ end
+
+ return res.body
+end
+
+
+local function get_auth_config(config)
+ local err
+ local auth_config = {}
+ if config.auth_config then
+ auth_config = config.auth_config
+ end
+
+ if config.auth_file then
+ local file_content
+ file_content, err = core.utils.get_file(config.auth_file)
+ if file_content then
+ auth_config = core.json.decode(file_content)
+ end
+ end
+
+ if err then
+ return nil, err
+ end
+
+ return auth_config
+end
+
+
+local function get_logger_buffer(conf, ctx)
+ local oauth_client = google_oauth:new(auth_config_cache)
+
+ local process = function(entries)
+ return send_to_google(oauth_client, entries)
+ end
+
+ local config = {
+ name = conf.name or plugin_name,
+ retry_delay = conf.retry_delay,
+ batch_max_size = conf.batch_max_size,
+ max_retry_count = conf.max_retry_count,
+ buffer_duration = conf.buffer_duration,
+ inactive_timeout = conf.inactive_timeout,
+ route_id = ctx.var.route_id,
+ server_addr = ctx.var.server_addr,
+ }
+
+ local buffer, err = batch_processor:new(process, config)
+
+ if not buffer then
+ return nil, "error when creating the batch processor: " .. err
+ end
+
+ return buffer
+end
+
+
+local function get_utc_timestamp()
+ ngx_update_time()
+ local now = tostring(ngx_now())
Review comment:
We can use math.ceil to get the second & millisecond?
##########
File path: apisix/plugins/google-logging.lua
##########
@@ -0,0 +1,298 @@
+--
+-- Licensed to the Apache Software Foundation (ASF) under one or more
+-- contributor license agreements. See the NOTICE file distributed with
+-- this work for additional information regarding copyright ownership.
+-- The ASF licenses this file to You under the Apache License, Version 2.0
+-- (the "License"); you may not use this file except in compliance with
+-- the License. You may obtain a copy of the License at
+--
+-- http://www.apache.org/licenses/LICENSE-2.0
+--
+-- Unless required by applicable law or agreed to in writing, software
+-- distributed under the License is distributed on an "AS IS" BASIS,
+-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+-- See the License for the specific language governing permissions and
+-- limitations under the License.
+--
+
+local core = require("apisix.core")
+local ngx = ngx
+local tostring = tostring
+local ipairs = ipairs
+local sub_str = string.sub
+local os_date = os.date
+
+local ngx_now = ngx.now
+local ngx_timer_at = ngx.timer.at
+local ngx_update_time = ngx.update_time
+
+local http = require("resty.http")
+local log_util = require("apisix.utils.log-util")
+local batch_processor = require("apisix.utils.batch-processor")
+local google_oauth = require("apisix.plugins.google-logging.oauth")
+
+
+local buffers = {}
+local auth_config_cache
+local stale_timer_running
+
+
+local plugin_name = "google-logging"
+local schema = {
+ type = "object",
+ properties = {
+ auth_config = {
+ type = "object",
+ properties = {
+ private_key = { type = "string" },
+ project_id = { type = "string" },
+ token_uri = {
+ type = "string",
+ default = "https://oauth2.googleapis.com/token"
+ },
+ -- https://developers.google.com/identity/protocols/oauth2/scopes#logging
+ scopes = {
+ type = "array",
+ items = {
+ description = "Google OAuth2 Authorization Scopes",
+ type = "string",
+ },
+ minItems = 1,
+ uniqueItems = true,
+ default = {
+ "https://www.googleapis.com/auth/logging.read",
+ "https://www.googleapis.com/auth/logging.write",
+ "https://www.googleapis.com/auth/logging.admin",
+ "https://www.googleapis.com/auth/cloud-platform"
+ }
+ },
+ entries_uri = {
+ type = "string",
+ default = "https://logging.googleapis.com/v2/entries:write"
+ },
+ },
+ required = { "private_key", "project_id", "token_uri" }
+ },
+ auth_file = { type = "string" },
+ -- https://cloud.google.com/logging/docs/reference/v2/rest/v2/MonitoredResource
+ resource = {
+ type = "object",
+ properties = {
+ type = { type = "string" },
+ labels = { type = "object" }
+ },
+ default = {
+ type = "global"
+ },
+ required = { "type" }
+ },
+ -- https://cloud.google.com/logging/docs/reference/v2/rest/v2/LogEntry
+ log_id = { type = "string", default = "apisix.apache.org%2Flogs" },
+ max_retry_count = { type = "integer", minimum = 0, default = 0 },
+ retry_delay = { type = "integer", minimum = 0, default = 1 },
+ buffer_duration = { type = "integer", minimum = 1, default = 60 },
+ inactive_timeout = { type = "integer", minimum = 1, default = 10 },
+ batch_max_size = { type = "integer", minimum = 1, default = 100 },
+ },
+ oneOf = {
+ { required = { "auth_config" } },
+ { required = { "auth_file" } },
+ },
+}
+
+
+-- remove stale objects from the memory after timer expires
+local function remove_stale_objects(premature)
+ if premature then
+ return
+ end
+
+ for key, batch in ipairs(buffers) do
+ if #batch.entry_buffer.entries == 0 and #batch.batch_to_process == 0 then
+ core.log.warn("removing batch processor stale object, route id:", tostring(key))
+ buffers[key] = nil
+ end
+ end
+
+ stale_timer_running = false
+end
+
+
+local function send_to_google(oauth, entries)
+ local http_new = http.new()
+ local res, err = http_new:request_uri(oauth.entries_uri, {
+ ssl_verify = false,
+ method = "POST",
+ body = core.json.encode({
+ entries = entries,
+ partialSuccess = false,
+ }),
+ headers = {
+ ["Content-Type"] = "application/json",
+ ["Authorization"] = "Bearer " .. oauth:get_access_token(),
+ },
+ })
+
+ if err then
+ return nil, "failed to write log to google, ", err
Review comment:
```suggestion
return nil, "failed to write log to google: " .. err
```
##########
File path: apisix/plugins/google-logging/oauth.lua
##########
@@ -0,0 +1,128 @@
+--
+-- Licensed to the Apache Software Foundation (ASF) under one or more
+-- contributor license agreements. See the NOTICE file distributed with
+-- this work for additional information regarding copyright ownership.
+-- The ASF licenses this file to You under the Apache License, Version 2.0
+-- (the "License"); you may not use this file except in compliance with
+-- the License. You may obtain a copy of the License at
+--
+-- http://www.apache.org/licenses/LICENSE-2.0
+--
+-- Unless required by applicable law or agreed to in writing, software
+-- distributed under the License is distributed on an "AS IS" BASIS,
+-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+-- See the License for the specific language governing permissions and
+-- limitations under the License.
+--
+
+local core = require("apisix.core")
+local type = type
+local setmetatable = setmetatable
+
+local ngx_update_time = ngx.update_time
+local ngx_time = ngx.time
+local ngx_encode_args = ngx.encode_args
+
+local http = require("resty.http")
+local jwt = require("resty.jwt")
+
+
+local function get_timestamp()
+ ngx_update_time()
+ return ngx_time()
+end
+
+
+local _M = {}
+
+
+function _M:get_access_token()
+ if not self.access_token or get_timestamp() > self.access_token_expire_time - 60 then
+ self:refresh_access_token()
+ end
+ return self.access_token
+end
+
+
+function _M:refresh_access_token()
+ local http_new = http.new()
+ local res, err = http_new:request_uri(self.token_uri, {
+ ssl_verify = false,
+ method = "POST",
+ body = ngx_encode_args({
+ grant_type = "urn:ietf:params:oauth:grant-type:jwt-bearer",
+ assertion = self:generate_jwt_token()
+ }),
+ headers = {
+ ["Content-Type"] = "application/x-www-form-urlencoded",
+ },
+ })
+
+ if not res then
+ core.log.error("failed to refresh google oauth access token, ", err)
+ return
+ end
+
+ if res.status ~= 200 then
+ core.log.error("failed to refresh google oauth access token: ", res.body)
+ return
+ end
+
+ res = core.json.decode(res.body)
Review comment:
Need to check the decoded result.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] spacewander commented on a change in pull request #5538: feat(plugin): support google cloud logging service
Posted by GitBox <gi...@apache.org>.
spacewander commented on a change in pull request #5538:
URL: https://github.com/apache/apisix/pull/5538#discussion_r753791053
##########
File path: t/plugin/google-logging.t
##########
@@ -0,0 +1,380 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+use t::APISIX 'no_plan';
+
+repeat_each(1);
+no_long_string();
+no_root_location();
+run_tests;
+
+
+add_block_preprocessor(sub {
+ my ($block) = @_;
+
+ if ((!defined $block->error_log) && (!defined $block->no_error_log)) {
+ $block->set_value("no_error_log", "[error]");
+ }
Review comment:
We also need to set the `request` by default too.
##########
File path: t/plugin/google-logging.t
##########
@@ -0,0 +1,380 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+use t::APISIX 'no_plan';
+
+repeat_each(1);
+no_long_string();
+no_root_location();
+run_tests;
+
+
+add_block_preprocessor(sub {
+ my ($block) = @_;
+
+ if ((!defined $block->error_log) && (!defined $block->no_error_log)) {
+ $block->set_value("no_error_log", "[error]");
+ }
+});
+
+__DATA__
+
+=== TEST 1: Full configuration verification (Auth File)
+--- config
+ location /t {
+ content_by_lua_block {
+ local plugin = require("apisix.plugins.google-logging")
+ local ok, err = plugin.check_schema({
+ auth_file = "/path/to/apache/apisix/auth.json",
+ resource = {
+ type = "global"
+ },
+ scopes = {
+ "https://www.googleapis.com/auth/logging.admin"
+ },
+ log_id = "syslog",
+ max_retry_count = 0,
+ retry_delay = 1,
+ buffer_duration = 60,
+ inactive_timeout = 10,
+ batch_max_size = 100,
+ })
+
+ if not ok then
+ ngx.say(err)
+ else
+ ngx.say("passed")
+ end
+ }
+ }
+--- request
+GET /t
+--- response_body
+passed
+--- no_error_log
Review comment:
Please remove the duplicate sections.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] spacewander commented on a change in pull request #5538: feat(plugin): support google cloud logging service
Posted by GitBox <gi...@apache.org>.
spacewander commented on a change in pull request #5538:
URL: https://github.com/apache/apisix/pull/5538#discussion_r754044327
##########
File path: Makefile
##########
@@ -303,6 +303,9 @@ install: runtime
$(ENV_INSTALL) -d $(ENV_INST_LUADIR)/apisix/plugins/limit-count
$(ENV_INSTALL) apisix/plugins/limit-count/*.lua $(ENV_INST_LUADIR)/apisix/plugins/limit-count/
+ $(ENV_INSTALL) -d $(ENV_INST_LUADIR)/apisix/plugins/google-logging
+ $(ENV_INSTALL) apisix/plugins/google-logging/*.lua $(ENV_INST_LUADIR)/apisix/plugins/google-logging/
Review comment:
Both.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] spacewander commented on a change in pull request #5538: feat(plugin): support google cloud logging service
Posted by GitBox <gi...@apache.org>.
spacewander commented on a change in pull request #5538:
URL: https://github.com/apache/apisix/pull/5538#discussion_r754044676
##########
File path: docs/en/latest/plugins/google-logging.md
##########
@@ -0,0 +1,155 @@
+---
+title: google-logging
Review comment:
Maybe `google-cloud-logging` is enough to identify the product?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] shuaijinchao commented on a change in pull request #5538: feat(plugin): support google cloud logging service
Posted by GitBox <gi...@apache.org>.
shuaijinchao commented on a change in pull request #5538:
URL: https://github.com/apache/apisix/pull/5538#discussion_r754067471
##########
File path: apisix/plugins/google-logging.lua
##########
@@ -0,0 +1,298 @@
+--
+-- Licensed to the Apache Software Foundation (ASF) under one or more
+-- contributor license agreements. See the NOTICE file distributed with
+-- this work for additional information regarding copyright ownership.
+-- The ASF licenses this file to You under the Apache License, Version 2.0
+-- (the "License"); you may not use this file except in compliance with
+-- the License. You may obtain a copy of the License at
+--
+-- http://www.apache.org/licenses/LICENSE-2.0
+--
+-- Unless required by applicable law or agreed to in writing, software
+-- distributed under the License is distributed on an "AS IS" BASIS,
+-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+-- See the License for the specific language governing permissions and
+-- limitations under the License.
+--
+
+local core = require("apisix.core")
+local ngx = ngx
+local tostring = tostring
+local ipairs = ipairs
+local sub_str = string.sub
+local os_date = os.date
+
+local ngx_now = ngx.now
+local ngx_timer_at = ngx.timer.at
+local ngx_update_time = ngx.update_time
+
+local http = require("resty.http")
+local log_util = require("apisix.utils.log-util")
+local batch_processor = require("apisix.utils.batch-processor")
+local google_oauth = require("apisix.plugins.google-logging.oauth")
+
+
+local buffers = {}
+local auth_config_cache
+local stale_timer_running
+
+
+local plugin_name = "google-logging"
+local schema = {
+ type = "object",
+ properties = {
+ auth_config = {
+ type = "object",
+ properties = {
+ private_key = { type = "string" },
+ project_id = { type = "string" },
+ token_uri = {
+ type = "string",
+ default = "https://oauth2.googleapis.com/token"
+ },
+ -- https://developers.google.com/identity/protocols/oauth2/scopes#logging
+ scopes = {
+ type = "array",
+ items = {
+ description = "Google OAuth2 Authorization Scopes",
+ type = "string",
+ },
+ minItems = 1,
+ uniqueItems = true,
+ default = {
+ "https://www.googleapis.com/auth/logging.read",
+ "https://www.googleapis.com/auth/logging.write",
+ "https://www.googleapis.com/auth/logging.admin",
+ "https://www.googleapis.com/auth/cloud-platform"
+ }
+ },
+ entries_uri = {
+ type = "string",
+ default = "https://logging.googleapis.com/v2/entries:write"
+ },
+ },
+ required = { "private_key", "project_id", "token_uri" }
+ },
+ auth_file = { type = "string" },
+ -- https://cloud.google.com/logging/docs/reference/v2/rest/v2/MonitoredResource
+ resource = {
+ type = "object",
+ properties = {
+ type = { type = "string" },
+ labels = { type = "object" }
+ },
+ default = {
+ type = "global"
+ },
+ required = { "type" }
+ },
+ -- https://cloud.google.com/logging/docs/reference/v2/rest/v2/LogEntry
+ log_id = { type = "string", default = "apisix.apache.org%2Flogs" },
+ max_retry_count = { type = "integer", minimum = 0, default = 0 },
+ retry_delay = { type = "integer", minimum = 0, default = 1 },
+ buffer_duration = { type = "integer", minimum = 1, default = 60 },
+ inactive_timeout = { type = "integer", minimum = 1, default = 10 },
+ batch_max_size = { type = "integer", minimum = 1, default = 100 },
+ },
+ oneOf = {
+ { required = { "auth_config" } },
+ { required = { "auth_file" } },
+ },
+}
+
+
+-- remove stale objects from the memory after timer expires
+local function remove_stale_objects(premature)
+ if premature then
+ return
+ end
+
+ for key, batch in ipairs(buffers) do
+ if #batch.entry_buffer.entries == 0 and #batch.batch_to_process == 0 then
+ core.log.warn("removing batch processor stale object, route id:", tostring(key))
+ buffers[key] = nil
+ end
+ end
+
+ stale_timer_running = false
+end
+
+
+local function send_to_google(oauth, entries)
+ local http_new = http.new()
+ local res, err = http_new:request_uri(oauth.entries_uri, {
+ ssl_verify = false,
Review comment:
updated
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] shuaijinchao commented on a change in pull request #5538: feat(plugin): support google cloud logging service
Posted by GitBox <gi...@apache.org>.
shuaijinchao commented on a change in pull request #5538:
URL: https://github.com/apache/apisix/pull/5538#discussion_r754069655
##########
File path: apisix/plugins/google-logging.lua
##########
@@ -0,0 +1,298 @@
+--
+-- Licensed to the Apache Software Foundation (ASF) under one or more
+-- contributor license agreements. See the NOTICE file distributed with
+-- this work for additional information regarding copyright ownership.
+-- The ASF licenses this file to You under the Apache License, Version 2.0
+-- (the "License"); you may not use this file except in compliance with
+-- the License. You may obtain a copy of the License at
+--
+-- http://www.apache.org/licenses/LICENSE-2.0
+--
+-- Unless required by applicable law or agreed to in writing, software
+-- distributed under the License is distributed on an "AS IS" BASIS,
+-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+-- See the License for the specific language governing permissions and
+-- limitations under the License.
+--
+
+local core = require("apisix.core")
+local ngx = ngx
+local tostring = tostring
+local ipairs = ipairs
+local sub_str = string.sub
+local os_date = os.date
+
+local ngx_now = ngx.now
+local ngx_timer_at = ngx.timer.at
+local ngx_update_time = ngx.update_time
+
+local http = require("resty.http")
+local log_util = require("apisix.utils.log-util")
+local batch_processor = require("apisix.utils.batch-processor")
+local google_oauth = require("apisix.plugins.google-logging.oauth")
+
+
+local buffers = {}
+local auth_config_cache
+local stale_timer_running
+
+
+local plugin_name = "google-logging"
+local schema = {
+ type = "object",
+ properties = {
+ auth_config = {
+ type = "object",
+ properties = {
+ private_key = { type = "string" },
+ project_id = { type = "string" },
+ token_uri = {
+ type = "string",
+ default = "https://oauth2.googleapis.com/token"
+ },
+ -- https://developers.google.com/identity/protocols/oauth2/scopes#logging
+ scopes = {
+ type = "array",
+ items = {
+ description = "Google OAuth2 Authorization Scopes",
+ type = "string",
+ },
+ minItems = 1,
+ uniqueItems = true,
+ default = {
+ "https://www.googleapis.com/auth/logging.read",
+ "https://www.googleapis.com/auth/logging.write",
+ "https://www.googleapis.com/auth/logging.admin",
+ "https://www.googleapis.com/auth/cloud-platform"
+ }
+ },
+ entries_uri = {
+ type = "string",
+ default = "https://logging.googleapis.com/v2/entries:write"
+ },
+ },
+ required = { "private_key", "project_id", "token_uri" }
+ },
+ auth_file = { type = "string" },
+ -- https://cloud.google.com/logging/docs/reference/v2/rest/v2/MonitoredResource
+ resource = {
+ type = "object",
+ properties = {
+ type = { type = "string" },
+ labels = { type = "object" }
+ },
+ default = {
+ type = "global"
+ },
+ required = { "type" }
+ },
+ -- https://cloud.google.com/logging/docs/reference/v2/rest/v2/LogEntry
+ log_id = { type = "string", default = "apisix.apache.org%2Flogs" },
+ max_retry_count = { type = "integer", minimum = 0, default = 0 },
+ retry_delay = { type = "integer", minimum = 0, default = 1 },
+ buffer_duration = { type = "integer", minimum = 1, default = 60 },
+ inactive_timeout = { type = "integer", minimum = 1, default = 10 },
+ batch_max_size = { type = "integer", minimum = 1, default = 100 },
+ },
+ oneOf = {
+ { required = { "auth_config" } },
+ { required = { "auth_file" } },
+ },
+}
+
+
+-- remove stale objects from the memory after timer expires
+local function remove_stale_objects(premature)
+ if premature then
+ return
+ end
+
+ for key, batch in ipairs(buffers) do
+ if #batch.entry_buffer.entries == 0 and #batch.batch_to_process == 0 then
+ core.log.warn("removing batch processor stale object, route id:", tostring(key))
+ buffers[key] = nil
+ end
+ end
+
+ stale_timer_running = false
+end
+
+
+local function send_to_google(oauth, entries)
+ local http_new = http.new()
+ local res, err = http_new:request_uri(oauth.entries_uri, {
+ ssl_verify = false,
+ method = "POST",
+ body = core.json.encode({
+ entries = entries,
+ partialSuccess = false,
+ }),
+ headers = {
+ ["Content-Type"] = "application/json",
+ ["Authorization"] = "Bearer " .. oauth:get_access_token(),
+ },
+ })
+
+ if err then
+ return nil, "failed to write log to google, ", err
+ end
+
+ if res.status ~= 200 then
+ return nil, res.body
+ end
+
+ return res.body
+end
+
+
+local function get_auth_config(config)
+ local err
+ local auth_config = {}
+ if config.auth_config then
+ auth_config = config.auth_config
+ end
+
+ if config.auth_file then
+ local file_content
+ file_content, err = core.utils.get_file(config.auth_file)
+ if file_content then
+ auth_config = core.json.decode(file_content)
+ end
+ end
+
+ if err then
+ return nil, err
+ end
+
+ return auth_config
+end
+
+
+local function get_logger_buffer(conf, ctx)
+ local oauth_client = google_oauth:new(auth_config_cache)
+
+ local process = function(entries)
+ return send_to_google(oauth_client, entries)
+ end
+
+ local config = {
+ name = conf.name or plugin_name,
+ retry_delay = conf.retry_delay,
+ batch_max_size = conf.batch_max_size,
+ max_retry_count = conf.max_retry_count,
+ buffer_duration = conf.buffer_duration,
+ inactive_timeout = conf.inactive_timeout,
+ route_id = ctx.var.route_id,
+ server_addr = ctx.var.server_addr,
+ }
+
+ local buffer, err = batch_processor:new(process, config)
+
+ if not buffer then
+ return nil, "error when creating the batch processor: " .. err
+ end
+
+ return buffer
+end
+
+
+local function get_utc_timestamp()
+ ngx_update_time()
+ local now = tostring(ngx_now())
+ local pos = core.string.rfind_char(now, ".", #now - 1)
+ local second = now
+ local millisecond = 0
+ if pos then
+ second = sub_str(now, 1, pos - 1)
+ millisecond = sub_str(now, pos + 1)
+ end
+ return os_date("!%Y-%m-%dT%T.", second) .. core.string.format("%03dZ", millisecond)
+end
+
+
+local function get_logger_entry(conf)
+ if not auth_config_cache then
Review comment:
updated
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] shuaijinchao commented on a change in pull request #5538: feat(plugin): support google cloud logging service
Posted by GitBox <gi...@apache.org>.
shuaijinchao commented on a change in pull request #5538:
URL: https://github.com/apache/apisix/pull/5538#discussion_r754070456
##########
File path: docs/en/latest/plugins/google-logging.md
##########
@@ -0,0 +1,155 @@
+---
+title: google-logging
+---
+
+<!--
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+-->
+
+## Summary
+
+- [**Name**](#name)
+- [**Attributes**](#attributes)
+- [**How To Enable**](#how-to-enable)
+- [**Test Plugin**](#test-plugin)
+- [**Disable Plugin**](#disable-plugin)
+
+## Name
+
+`google-logging` plugin is used to send the access log of `Apache APISIX` to the [Google Cloud Logging Service](https://cloud.google.com/logging/).
+
+This plugin provides the ability to push Log data as a batch to Google Cloud logging Service.
+
+For more info on Batch-Processor in Apache APISIX please refer:
+[Batch-Processor](../batch-processor.md)
+
+## Attributes
+
+| Name | Requirement | Default | Description |
+| ----------------------- | ------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| auth_config | Semi-optional | | one of `auth_config` or `auth_file` must be configured |
+| auth_config.private_key | Required | | the private key parameters of the Google service account |
+| auth_config.project_id | Required | | the project id parameters of the Google service account |
+| auth_config.token_uri | Optional | https://oauth2.googleapis.com/token | the token uri parameters of the Google service account |
+| auth_config.entries_uri | Optional | https://logging.googleapis.com/v2/entries:write | google logging service API |
+| auth_config.scopes | Optional | ["https://www.googleapis.com/auth/logging.read","https://www.googleapis.com/auth/logging.write","https://www.googleapis.com/auth/logging.admin","https://www.googleapis.com/auth/cloud-platform"] | the access scopes parameters of the Google service account, refer to: [OAuth 2.0 Scopes for Google APIs](https://developers.google.com/identity/protocols/oauth2/scopes#logging) |
+| auth_file | Semi-optional | | path to the google service account json file（Semi-optional, one of auth_config or auth_file must be configured） |
+| resource | Optional | {"type": "global"} | the Google monitor resource, refer to: [MonitoredResource](https://cloud.google.com/logging/docs/reference/v2/rest/v2/MonitoredResource) |
+| log_id | Optional | apisix.apache.org%2Flogs | google logging id, refer to: [LogEntry](https://cloud.google.com/logging/docs/reference/v2/rest/v2/LogEntry) |
+| max_retry_count | Optional | 0 | max number of retries before removing from the processing pipe line |
+| retry_delay | Optional | 1 | number of seconds the process execution should be delayed if the execution fails |
+| buffer_duration | Optional | 60 | max age in seconds of the oldest entry in a batch before the batch must be processed |
+| inactive_timeout | Optional | 10 | max age in seconds when the buffer will be flushed if inactive |
+| batch_max_size | Optional | 100 | max size of each batch |
+
+## How To Enable
+
+The following is an example on how to enable the `google-logging` for a specific route.
+
+### Full configuration
+
+```shell
+curl http://127.0.0.1:9080/apisix/admin/routes/1 -H 'X-API-KEY: edd1c9f034335f136f87ad84b625c8f1' -X PUT -d '
+{
+ "plugins": {
+ "google-logging": {
+ "auth_config":{
+ "project_id":"apisix",
+ "private_key":"-----BEGIN RSA PRIVATE KEY-----KEY-----END RSA PRIVATE KEY-----",
+ "token_uri":"https://oauth2.googleapis.com/token",
+ "scopes":[
+ "https://www.googleapis.com/auth/logging.admin"
+ ],
+ "entries_uri":"https://logging.googleapis.com/v2/entries:write"
+ },
+ "resource":{
+ "type":"global"
+ },
+ "log_id":"syslog",
Review comment:
updated
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] shuaijinchao commented on a change in pull request #5538: feat(plugin): support google cloud logging service
Posted by GitBox <gi...@apache.org>.
shuaijinchao commented on a change in pull request #5538:
URL: https://github.com/apache/apisix/pull/5538#discussion_r754067309
##########
File path: apisix/core/utils.lua
##########
@@ -260,6 +260,18 @@ function _M.gethostname()
end
+function _M.get_file(file_name)
Review comment:
updated
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] tzssangglass commented on a change in pull request #5538: feat(plugin): support google cloud logging service
Posted by GitBox <gi...@apache.org>.
tzssangglass commented on a change in pull request #5538:
URL: https://github.com/apache/apisix/pull/5538#discussion_r754359053
##########
File path: apisix/plugins/google-cloud-logging.lua
##########
@@ -0,0 +1,326 @@
+--
+-- Licensed to the Apache Software Foundation (ASF) under one or more
+-- contributor license agreements. See the NOTICE file distributed with
+-- this work for additional information regarding copyright ownership.
+-- The ASF licenses this file to You under the Apache License, Version 2.0
+-- (the "License"); you may not use this file except in compliance with
+-- the License. You may obtain a copy of the License at
+--
+-- http://www.apache.org/licenses/LICENSE-2.0
+--
+-- Unless required by applicable law or agreed to in writing, software
+-- distributed under the License is distributed on an "AS IS" BASIS,
+-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+-- See the License for the specific language governing permissions and
+-- limitations under the License.
+--
+
+local core = require("apisix.core")
+local ngx = ngx
+local tostring = tostring
+local ipairs = ipairs
+local os_date = os.date
+local math_floor = math.floor
+local ngx_now = ngx.now
+local ngx_timer_at = ngx.timer.at
+local ngx_update_time = ngx.update_time
+local http = require("resty.http")
+local log_util = require("apisix.utils.log-util")
+local batch_processor = require("apisix.utils.batch-processor")
+local google_oauth = require("apisix.plugins.google-cloud-logging.oauth")
+
+
+local buffers = {}
+local auth_config_cache
+local stale_timer_running
+
+
+local plugin_name = "google-cloud-logging"
+local schema = {
+ type = "object",
+ properties = {
+ auth_config = {
+ type = "object",
+ properties = {
+ private_key = { type = "string" },
+ project_id = { type = "string" },
+ token_uri = {
+ type = "string",
+ default = "https://oauth2.googleapis.com/token"
+ },
+ -- https://developers.google.com/identity/protocols/oauth2/scopes#logging
+ scopes = {
+ type = "array",
+ items = {
+ description = "Google OAuth2 Authorization Scopes",
+ type = "string",
+ },
+ minItems = 1,
+ uniqueItems = true,
+ default = {
+ "https://www.googleapis.com/auth/logging.read",
+ "https://www.googleapis.com/auth/logging.write",
+ "https://www.googleapis.com/auth/logging.admin",
+ "https://www.googleapis.com/auth/cloud-platform"
+ }
+ },
+ entries_uri = {
+ type = "string",
+ default = "https://logging.googleapis.com/v2/entries:write"
+ },
+ ssl_verify = {
+ type = "boolean",
+ default = true
+ },
+ },
+ required = { "private_key", "project_id", "token_uri" }
+ },
+ auth_file = { type = "string" },
+ -- https://cloud.google.com/logging/docs/reference/v2/rest/v2/MonitoredResource
+ resource = {
+ type = "object",
+ properties = {
+ type = { type = "string" },
+ labels = { type = "object" }
+ },
+ default = {
+ type = "global"
+ },
+ required = { "type" }
+ },
+ -- https://cloud.google.com/logging/docs/reference/v2/rest/v2/LogEntry
+ log_id = {
+ type = "string",
+ default = "apisix.apache.org%2Flogs"
+ },
+ max_retry_count = {
+ type = "integer",
+ minimum = 0,
+ default = 0
+ },
+ retry_delay = {
+ type = "integer",
+ minimum = 0,
+ default = 1
+ },
+ buffer_duration = {
+ type = "integer",
+ minimum = 1,
+ default = 60
+ },
+ inactive_timeout = {
+ type = "integer",
+ minimum = 1,
+ default = 10
+ },
+ batch_max_size = {
+ type = "integer",
+ minimum = 1,
+ default = 100
+ },
+ },
+ oneOf = {
+ { required = { "auth_config" } },
+ { required = { "auth_file" } },
+ },
+}
+
+
+-- remove stale objects from the memory after timer expires
+local function remove_stale_objects(premature)
+ if premature then
+ return
+ end
+
+ for key, batch in ipairs(buffers) do
+ if #batch.entry_buffer.entries == 0 and #batch.batch_to_process == 0 then
+ core.log.warn("removing batch processor stale object, route id:", tostring(key))
+ buffers[key] = nil
+ end
+ end
+
+ stale_timer_running = false
+end
+
+
+local function send_to_google(oauth, entries)
+ local http_new = http.new()
+ local access_token = oauth:generate_access_token()
+ if not access_token then
+ return nil, "failed to get google oauth token"
+ end
+
+ local res, err = http_new:request_uri(oauth.entries_uri, {
+ ssl_verify = oauth.ssl_verify,
+ method = "POST",
+ body = core.json.encode({
+ entries = entries,
+ partialSuccess = false,
+ }),
+ headers = {
+ ["Content-Type"] = "application/json",
+ ["Authorization"] = "Bearer " .. access_token,
+ },
+ })
+
+ if err then
+ return nil, "failed to write log to google, " .. err
+ end
+
+ if res.status ~= 200 then
+ return nil, res.body
+ end
+
+ return res.body
+end
+
+
+local function get_auth_config(config)
+ if auth_config_cache then
+ return auth_config_cache
+ end
+
+ if config.auth_config then
+ auth_config_cache = config.auth_config
+ return auth_config_cache
+ end
+
+ if not config.auth_file then
+ return nil, "configuration is not defined"
+ end
+
+ local file_content, err = core.io.get_file(config.auth_file)
+ if not file_content then
+ return nil, "failed to read configuration, file: " .. config.auth_file .. " err:" .. err
+ end
+
+ local config_data
+ config_data, err = core.json.decode(file_content)
+ if not config_data then
+ return nil, "config parse failure, data: " .. file_content .. " , err: " .. err
+ end
+
+ auth_config_cache = config.auth_config
+ return auth_config_cache
+end
+
+
+local function get_logger_buffer(conf, ctx)
+ local oauth_client = google_oauth:new(auth_config_cache)
+
+ local process = function(entries)
+ return send_to_google(oauth_client, entries)
+ end
+
+ local config = {
+ name = conf.name or plugin_name,
+ retry_delay = conf.retry_delay,
+ batch_max_size = conf.batch_max_size,
+ max_retry_count = conf.max_retry_count,
+ buffer_duration = conf.buffer_duration,
+ inactive_timeout = conf.inactive_timeout,
+ route_id = ctx.var.route_id,
+ server_addr = ctx.var.server_addr,
+ }
+
+ local buffer, err = batch_processor:new(process, config)
+
+ if not buffer then
+ return nil, "error when creating the batch processor: " .. err
+ end
+
+ return buffer
+end
+
+
+local function get_utc_timestamp()
+ ngx_update_time()
+ local now = ngx_now()
+ local second = math_floor(now)
+ local millisecond = math_floor((now - second) * 1000)
+ return os_date("!%Y-%m-%dT%T.", second) .. core.string.format("%03dZ", millisecond)
+end
Review comment:
same as: https://github.com/apache/apisix/blob/master/apisix/plugins/slslog/rfc5424.lua#L83? if true, I think it's better to move this function to common utils.
##########
File path: apisix/plugins/google-cloud-logging.lua
##########
@@ -0,0 +1,326 @@
+--
+-- Licensed to the Apache Software Foundation (ASF) under one or more
+-- contributor license agreements. See the NOTICE file distributed with
+-- this work for additional information regarding copyright ownership.
+-- The ASF licenses this file to You under the Apache License, Version 2.0
+-- (the "License"); you may not use this file except in compliance with
+-- the License. You may obtain a copy of the License at
+--
+-- http://www.apache.org/licenses/LICENSE-2.0
+--
+-- Unless required by applicable law or agreed to in writing, software
+-- distributed under the License is distributed on an "AS IS" BASIS,
+-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+-- See the License for the specific language governing permissions and
+-- limitations under the License.
+--
+
+local core = require("apisix.core")
+local ngx = ngx
+local tostring = tostring
+local ipairs = ipairs
+local os_date = os.date
+local math_floor = math.floor
+local ngx_now = ngx.now
+local ngx_timer_at = ngx.timer.at
+local ngx_update_time = ngx.update_time
+local http = require("resty.http")
+local log_util = require("apisix.utils.log-util")
+local batch_processor = require("apisix.utils.batch-processor")
+local google_oauth = require("apisix.plugins.google-cloud-logging.oauth")
+
+
+local buffers = {}
+local auth_config_cache
+local stale_timer_running
+
+
+local plugin_name = "google-cloud-logging"
+local schema = {
+ type = "object",
+ properties = {
+ auth_config = {
+ type = "object",
+ properties = {
+ private_key = { type = "string" },
+ project_id = { type = "string" },
+ token_uri = {
+ type = "string",
+ default = "https://oauth2.googleapis.com/token"
+ },
+ -- https://developers.google.com/identity/protocols/oauth2/scopes#logging
+ scopes = {
+ type = "array",
+ items = {
+ description = "Google OAuth2 Authorization Scopes",
+ type = "string",
+ },
+ minItems = 1,
+ uniqueItems = true,
+ default = {
+ "https://www.googleapis.com/auth/logging.read",
+ "https://www.googleapis.com/auth/logging.write",
+ "https://www.googleapis.com/auth/logging.admin",
+ "https://www.googleapis.com/auth/cloud-platform"
+ }
+ },
+ entries_uri = {
+ type = "string",
+ default = "https://logging.googleapis.com/v2/entries:write"
+ },
+ ssl_verify = {
+ type = "boolean",
+ default = true
+ },
+ },
+ required = { "private_key", "project_id", "token_uri" }
+ },
+ auth_file = { type = "string" },
+ -- https://cloud.google.com/logging/docs/reference/v2/rest/v2/MonitoredResource
+ resource = {
+ type = "object",
+ properties = {
+ type = { type = "string" },
+ labels = { type = "object" }
+ },
+ default = {
+ type = "global"
+ },
+ required = { "type" }
+ },
+ -- https://cloud.google.com/logging/docs/reference/v2/rest/v2/LogEntry
+ log_id = {
+ type = "string",
+ default = "apisix.apache.org%2Flogs"
+ },
+ max_retry_count = {
+ type = "integer",
+ minimum = 0,
+ default = 0
+ },
+ retry_delay = {
+ type = "integer",
+ minimum = 0,
+ default = 1
+ },
+ buffer_duration = {
+ type = "integer",
+ minimum = 1,
+ default = 60
+ },
+ inactive_timeout = {
+ type = "integer",
+ minimum = 1,
+ default = 10
+ },
+ batch_max_size = {
+ type = "integer",
+ minimum = 1,
+ default = 100
+ },
+ },
+ oneOf = {
+ { required = { "auth_config" } },
+ { required = { "auth_file" } },
+ },
+}
+
+
+-- remove stale objects from the memory after timer expires
+local function remove_stale_objects(premature)
+ if premature then
+ return
+ end
+
+ for key, batch in ipairs(buffers) do
+ if #batch.entry_buffer.entries == 0 and #batch.batch_to_process == 0 then
+ core.log.warn("removing batch processor stale object, route id:", tostring(key))
+ buffers[key] = nil
+ end
+ end
+
+ stale_timer_running = false
+end
+
+
+local function send_to_google(oauth, entries)
+ local http_new = http.new()
+ local access_token = oauth:generate_access_token()
+ if not access_token then
+ return nil, "failed to get google oauth token"
+ end
+
+ local res, err = http_new:request_uri(oauth.entries_uri, {
+ ssl_verify = oauth.ssl_verify,
+ method = "POST",
+ body = core.json.encode({
+ entries = entries,
+ partialSuccess = false,
+ }),
+ headers = {
+ ["Content-Type"] = "application/json",
+ ["Authorization"] = "Bearer " .. access_token,
+ },
+ })
+
+ if err then
+ return nil, "failed to write log to google, " .. err
+ end
+
+ if res.status ~= 200 then
+ return nil, res.body
+ end
+
+ return res.body
+end
+
+
+local function get_auth_config(config)
+ if auth_config_cache then
+ return auth_config_cache
+ end
+
+ if config.auth_config then
+ auth_config_cache = config.auth_config
+ return auth_config_cache
+ end
+
+ if not config.auth_file then
+ return nil, "configuration is not defined"
+ end
+
+ local file_content, err = core.io.get_file(config.auth_file)
+ if not file_content then
+ return nil, "failed to read configuration, file: " .. config.auth_file .. " err:" .. err
+ end
+
+ local config_data
+ config_data, err = core.json.decode(file_content)
+ if not config_data then
+ return nil, "config parse failure, data: " .. file_content .. " , err: " .. err
+ end
+
+ auth_config_cache = config.auth_config
+ return auth_config_cache
+end
+
+
+local function get_logger_buffer(conf, ctx)
+ local oauth_client = google_oauth:new(auth_config_cache)
+
+ local process = function(entries)
+ return send_to_google(oauth_client, entries)
+ end
+
+ local config = {
+ name = conf.name or plugin_name,
+ retry_delay = conf.retry_delay,
+ batch_max_size = conf.batch_max_size,
+ max_retry_count = conf.max_retry_count,
+ buffer_duration = conf.buffer_duration,
+ inactive_timeout = conf.inactive_timeout,
+ route_id = ctx.var.route_id,
+ server_addr = ctx.var.server_addr,
+ }
+
+ local buffer, err = batch_processor:new(process, config)
+
+ if not buffer then
+ return nil, "error when creating the batch processor: " .. err
+ end
+
+ return buffer
+end
+
+
+local function get_utc_timestamp()
+ ngx_update_time()
+ local now = ngx_now()
+ local second = math_floor(now)
+ local millisecond = math_floor((now - second) * 1000)
+ return os_date("!%Y-%m-%dT%T.", second) .. core.string.format("%03dZ", millisecond)
+end
+
+
+local function get_logger_entry(conf, ctx)
+ local auth_config, err = get_auth_config(conf)
+ if err or not auth_config.project_id or not auth_config.private_key then
+ return nil, "failed to get google authentication configuration, " .. err
+ end
+
+ local entry = log_util.get_full_log(ngx, conf)
+ local google_entry = {
+ httpRequest = {
+ requestMethod = entry.request.method,
+ requestUrl = entry.request.url,
+ requestSize = entry.request.size,
+ status = entry.response.status,
+ responseSize = entry.response.size,
+ userAgent = entry.request.headers and entry.request.headers["user-agent"],
+ remoteIp = entry.client_ip,
+ serverIp = entry.upstream,
+ latency = tostring(core.string.format("%0.3f", entry.latency / 1000)) .. "s"
Review comment:
Looks like this is more accurate: https://github.com/apache/apisix/blob/5394dce6fb9dde973baee41a31107d99ab2ca1f3/apisix/utils/log-util.lua#L180?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] spacewander commented on a change in pull request #5538: feat(plugin): support google cloud logging service
Posted by GitBox <gi...@apache.org>.
spacewander commented on a change in pull request #5538:
URL: https://github.com/apache/apisix/pull/5538#discussion_r754763467
##########
File path: docs/en/latest/plugins/google-cloud-logging.md
##########
@@ -0,0 +1,156 @@
+---
+title: google-cloud-logging
+---
+
+<!--
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+-->
+
+## Summary
+
+- [**Name**](#name)
+- [**Attributes**](#attributes)
+- [**How To Enable**](#how-to-enable)
+- [**Test Plugin**](#test-plugin)
+- [**Disable Plugin**](#disable-plugin)
+
+## Name
+
+`google-cloud-logging` plugin is used to send the access log of `Apache APISIX` to the [Google Cloud Logging Service](https://cloud.google.com/logging/).
+
+This plugin provides the ability to push log data as a batch to Google Cloud logging Service.
+
+For more info on Batch-Processor in Apache APISIX please refer:
+[Batch-Processor](../batch-processor.md)
+
+## Attributes
+
+| Name | Requirement | Default | Description |
+| ----------------------- | ------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| auth_config | Semi-optional | | one of `auth_config` or `auth_file` must be configured |
+| auth_config.private_key | Required | | the private key parameters of the Google service account |
Review comment:
Please use `required` like other plugins.
##########
File path: docs/en/latest/plugins/google-cloud-logging.md
##########
@@ -0,0 +1,156 @@
+---
+title: google-cloud-logging
+---
+
+<!--
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+-->
+
+## Summary
+
+- [**Name**](#name)
+- [**Attributes**](#attributes)
+- [**How To Enable**](#how-to-enable)
+- [**Test Plugin**](#test-plugin)
+- [**Disable Plugin**](#disable-plugin)
+
+## Name
+
+`google-cloud-logging` plugin is used to send the access log of `Apache APISIX` to the [Google Cloud Logging Service](https://cloud.google.com/logging/).
+
+This plugin provides the ability to push log data as a batch to Google Cloud logging Service.
+
+For more info on Batch-Processor in Apache APISIX please refer:
+[Batch-Processor](../batch-processor.md)
+
+## Attributes
+
+| Name | Requirement | Default | Description |
+| ----------------------- | ------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| auth_config | Semi-optional | | one of `auth_config` or `auth_file` must be configured |
+| auth_config.private_key | Required | | the private key parameters of the Google service account |
+| auth_config.project_id | Required | | the project id parameters of the Google service account |
+| auth_config.token_uri | Optional | https://oauth2.googleapis.com/token | the token uri parameters of the Google service account |
+| auth_config.entries_uri | Optional | https://logging.googleapis.com/v2/entries:write | google cloud logging service API |
+| auth_config.scopes | Optional | ["https://www.googleapis.com/auth/logging.read","https://www.googleapis.com/auth/logging.write","https://www.googleapis.com/auth/logging.admin","https://www.googleapis.com/auth/cloud-platform"] | the access scopes parameters of the Google service account, refer to: [OAuth 2.0 Scopes for Google APIs](https://developers.google.com/identity/protocols/oauth2/scopes#logging) |
+| auth_config.ssl_verify | Optional | true | enable `SSL` verification, when set to `true`, the server certificate will be verified according to the CA certificate specified by the `ssl.lua_ssl_trusted_certificate` directive in the APISIX main configuration file.|
Review comment:
Should we remove the `lua_ssl_trusted_certificate` stuff? It is used to trust self-signed certificates.
##########
File path: docs/en/latest/plugins/google-cloud-logging.md
##########
@@ -0,0 +1,156 @@
+---
+title: google-cloud-logging
+---
+
+<!--
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+-->
+
+## Summary
+
+- [**Name**](#name)
+- [**Attributes**](#attributes)
+- [**How To Enable**](#how-to-enable)
+- [**Test Plugin**](#test-plugin)
+- [**Disable Plugin**](#disable-plugin)
+
+## Name
+
+`google-cloud-logging` plugin is used to send the access log of `Apache APISIX` to the [Google Cloud Logging Service](https://cloud.google.com/logging/).
+
+This plugin provides the ability to push log data as a batch to Google Cloud logging Service.
+
+For more info on Batch-Processor in Apache APISIX please refer:
+[Batch-Processor](../batch-processor.md)
+
+## Attributes
+
+| Name | Requirement | Default | Description |
+| ----------------------- | ------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| auth_config | Semi-optional | | one of `auth_config` or `auth_file` must be configured |
+| auth_config.private_key | Required | | the private key parameters of the Google service account |
+| auth_config.project_id | Required | | the project id parameters of the Google service account |
+| auth_config.token_uri | Optional | https://oauth2.googleapis.com/token | the token uri parameters of the Google service account |
Review comment:
Ditto
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] shuaijinchao commented on a change in pull request #5538: feat(plugin): support google cloud logging service
Posted by GitBox <gi...@apache.org>.
shuaijinchao commented on a change in pull request #5538:
URL: https://github.com/apache/apisix/pull/5538#discussion_r754069846
##########
File path: apisix/plugins/google-logging/oauth.lua
##########
@@ -0,0 +1,128 @@
+--
+-- Licensed to the Apache Software Foundation (ASF) under one or more
+-- contributor license agreements. See the NOTICE file distributed with
+-- this work for additional information regarding copyright ownership.
+-- The ASF licenses this file to You under the Apache License, Version 2.0
+-- (the "License"); you may not use this file except in compliance with
+-- the License. You may obtain a copy of the License at
+--
+-- http://www.apache.org/licenses/LICENSE-2.0
+--
+-- Unless required by applicable law or agreed to in writing, software
+-- distributed under the License is distributed on an "AS IS" BASIS,
+-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+-- See the License for the specific language governing permissions and
+-- limitations under the License.
+--
+
+local core = require("apisix.core")
+local type = type
+local setmetatable = setmetatable
+
+local ngx_update_time = ngx.update_time
+local ngx_time = ngx.time
+local ngx_encode_args = ngx.encode_args
+
+local http = require("resty.http")
+local jwt = require("resty.jwt")
+
+
+local function get_timestamp()
+ ngx_update_time()
+ return ngx_time()
+end
+
+
+local _M = {}
+
+
+function _M:get_access_token()
+ if not self.access_token or get_timestamp() > self.access_token_expire_time - 60 then
+ self:refresh_access_token()
+ end
+ return self.access_token
+end
+
+
+function _M:refresh_access_token()
+ local http_new = http.new()
+ local res, err = http_new:request_uri(self.token_uri, {
+ ssl_verify = false,
+ method = "POST",
+ body = ngx_encode_args({
+ grant_type = "urn:ietf:params:oauth:grant-type:jwt-bearer",
+ assertion = self:generate_jwt_token()
+ }),
+ headers = {
+ ["Content-Type"] = "application/x-www-form-urlencoded",
+ },
+ })
+
+ if not res then
+ core.log.error("failed to refresh google oauth access token, ", err)
+ return
+ end
+
+ if res.status ~= 200 then
+ core.log.error("failed to refresh google oauth access token: ", res.body)
+ return
+ end
+
+ res = core.json.decode(res.body)
Review comment:
updated
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] shuaijinchao commented on a change in pull request #5538: feat(plugin): support google cloud logging service
Posted by GitBox <gi...@apache.org>.
shuaijinchao commented on a change in pull request #5538:
URL: https://github.com/apache/apisix/pull/5538#discussion_r754073564
##########
File path: apisix/plugins/google-logging.lua
##########
@@ -0,0 +1,298 @@
+--
+-- Licensed to the Apache Software Foundation (ASF) under one or more
+-- contributor license agreements. See the NOTICE file distributed with
+-- this work for additional information regarding copyright ownership.
+-- The ASF licenses this file to You under the Apache License, Version 2.0
+-- (the "License"); you may not use this file except in compliance with
+-- the License. You may obtain a copy of the License at
+--
+-- http://www.apache.org/licenses/LICENSE-2.0
+--
+-- Unless required by applicable law or agreed to in writing, software
+-- distributed under the License is distributed on an "AS IS" BASIS,
+-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+-- See the License for the specific language governing permissions and
+-- limitations under the License.
+--
+
+local core = require("apisix.core")
+local ngx = ngx
+local tostring = tostring
+local ipairs = ipairs
+local sub_str = string.sub
+local os_date = os.date
+
+local ngx_now = ngx.now
+local ngx_timer_at = ngx.timer.at
+local ngx_update_time = ngx.update_time
+
+local http = require("resty.http")
+local log_util = require("apisix.utils.log-util")
+local batch_processor = require("apisix.utils.batch-processor")
+local google_oauth = require("apisix.plugins.google-logging.oauth")
+
+
+local buffers = {}
+local auth_config_cache
+local stale_timer_running
+
+
+local plugin_name = "google-logging"
+local schema = {
+ type = "object",
+ properties = {
+ auth_config = {
+ type = "object",
+ properties = {
+ private_key = { type = "string" },
+ project_id = { type = "string" },
+ token_uri = {
+ type = "string",
+ default = "https://oauth2.googleapis.com/token"
+ },
+ -- https://developers.google.com/identity/protocols/oauth2/scopes#logging
+ scopes = {
+ type = "array",
+ items = {
+ description = "Google OAuth2 Authorization Scopes",
+ type = "string",
+ },
+ minItems = 1,
+ uniqueItems = true,
+ default = {
+ "https://www.googleapis.com/auth/logging.read",
+ "https://www.googleapis.com/auth/logging.write",
+ "https://www.googleapis.com/auth/logging.admin",
+ "https://www.googleapis.com/auth/cloud-platform"
+ }
+ },
+ entries_uri = {
+ type = "string",
+ default = "https://logging.googleapis.com/v2/entries:write"
+ },
+ },
+ required = { "private_key", "project_id", "token_uri" }
+ },
+ auth_file = { type = "string" },
+ -- https://cloud.google.com/logging/docs/reference/v2/rest/v2/MonitoredResource
+ resource = {
+ type = "object",
+ properties = {
+ type = { type = "string" },
+ labels = { type = "object" }
+ },
+ default = {
+ type = "global"
+ },
+ required = { "type" }
+ },
+ -- https://cloud.google.com/logging/docs/reference/v2/rest/v2/LogEntry
+ log_id = { type = "string", default = "apisix.apache.org%2Flogs" },
+ max_retry_count = { type = "integer", minimum = 0, default = 0 },
+ retry_delay = { type = "integer", minimum = 0, default = 1 },
+ buffer_duration = { type = "integer", minimum = 1, default = 60 },
+ inactive_timeout = { type = "integer", minimum = 1, default = 10 },
+ batch_max_size = { type = "integer", minimum = 1, default = 100 },
+ },
+ oneOf = {
+ { required = { "auth_config" } },
+ { required = { "auth_file" } },
+ },
+}
+
+
+-- remove stale objects from the memory after timer expires
+local function remove_stale_objects(premature)
+ if premature then
+ return
+ end
+
+ for key, batch in ipairs(buffers) do
+ if #batch.entry_buffer.entries == 0 and #batch.batch_to_process == 0 then
+ core.log.warn("removing batch processor stale object, route id:", tostring(key))
+ buffers[key] = nil
+ end
+ end
+
+ stale_timer_running = false
+end
+
+
+local function send_to_google(oauth, entries)
+ local http_new = http.new()
+ local res, err = http_new:request_uri(oauth.entries_uri, {
+ ssl_verify = false,
+ method = "POST",
+ body = core.json.encode({
+ entries = entries,
+ partialSuccess = false,
+ }),
+ headers = {
+ ["Content-Type"] = "application/json",
+ ["Authorization"] = "Bearer " .. oauth:get_access_token(),
+ },
+ })
+
+ if err then
+ return nil, "failed to write log to google, ", err
+ end
+
+ if res.status ~= 200 then
+ return nil, res.body
+ end
+
+ return res.body
+end
+
+
+local function get_auth_config(config)
+ local err
+ local auth_config = {}
+ if config.auth_config then
+ auth_config = config.auth_config
+ end
+
+ if config.auth_file then
+ local file_content
+ file_content, err = core.utils.get_file(config.auth_file)
+ if file_content then
+ auth_config = core.json.decode(file_content)
+ end
+ end
+
+ if err then
+ return nil, err
+ end
+
+ return auth_config
+end
+
+
+local function get_logger_buffer(conf, ctx)
+ local oauth_client = google_oauth:new(auth_config_cache)
+
+ local process = function(entries)
+ return send_to_google(oauth_client, entries)
+ end
+
+ local config = {
+ name = conf.name or plugin_name,
+ retry_delay = conf.retry_delay,
+ batch_max_size = conf.batch_max_size,
+ max_retry_count = conf.max_retry_count,
+ buffer_duration = conf.buffer_duration,
+ inactive_timeout = conf.inactive_timeout,
+ route_id = ctx.var.route_id,
+ server_addr = ctx.var.server_addr,
+ }
+
+ local buffer, err = batch_processor:new(process, config)
+
+ if not buffer then
+ return nil, "error when creating the batch processor: " .. err
+ end
+
+ return buffer
+end
+
+
+local function get_utc_timestamp()
+ ngx_update_time()
+ local now = tostring(ngx_now())
Review comment:
yes, updated
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] juzhiyuan commented on a change in pull request #5538: feat(plugin): support google cloud logging service
Posted by GitBox <gi...@apache.org>.
juzhiyuan commented on a change in pull request #5538:
URL: https://github.com/apache/apisix/pull/5538#discussion_r754297322
##########
File path: docs/en/latest/plugins/google-logging.md
##########
@@ -0,0 +1,155 @@
+---
+title: google-logging
Review comment:
good!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] shuaijinchao commented on a change in pull request #5538: feat(plugin): support google cloud logging service
Posted by GitBox <gi...@apache.org>.
shuaijinchao commented on a change in pull request #5538:
URL: https://github.com/apache/apisix/pull/5538#discussion_r754771215
##########
File path: docs/en/latest/plugins/google-cloud-logging.md
##########
@@ -0,0 +1,156 @@
+---
+title: google-cloud-logging
+---
+
+<!--
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+-->
+
+## Summary
+
+- [**Name**](#name)
+- [**Attributes**](#attributes)
+- [**How To Enable**](#how-to-enable)
+- [**Test Plugin**](#test-plugin)
+- [**Disable Plugin**](#disable-plugin)
+
+## Name
+
+`google-cloud-logging` plugin is used to send the access log of `Apache APISIX` to the [Google Cloud Logging Service](https://cloud.google.com/logging/).
+
+This plugin provides the ability to push log data as a batch to Google Cloud logging Service.
+
+For more info on Batch-Processor in Apache APISIX please refer:
+[Batch-Processor](../batch-processor.md)
+
+## Attributes
+
+| Name | Requirement | Default | Description |
+| ----------------------- | ------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| auth_config | Semi-optional | | one of `auth_config` or `auth_file` must be configured |
+| auth_config.private_key | Required | | the private key parameters of the Google service account |
+| auth_config.project_id | Required | | the project id parameters of the Google service account |
+| auth_config.token_uri | Optional | https://oauth2.googleapis.com/token | the token uri parameters of the Google service account |
+| auth_config.entries_uri | Optional | https://logging.googleapis.com/v2/entries:write | google cloud logging service API |
+| auth_config.scopes | Optional | ["https://www.googleapis.com/auth/logging.read","https://www.googleapis.com/auth/logging.write","https://www.googleapis.com/auth/logging.admin","https://www.googleapis.com/auth/cloud-platform"] | the access scopes parameters of the Google service account, refer to: [OAuth 2.0 Scopes for Google APIs](https://developers.google.com/identity/protocols/oauth2/scopes#logging) |
+| auth_config.ssl_verify | Optional | true | enable `SSL` verification, when set to `true`, the server certificate will be verified according to the CA certificate specified by the `ssl.lua_ssl_trusted_certificate` directive in the APISIX main configuration file.|
Review comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] shuaijinchao commented on a change in pull request #5538: feat(plugin): support google cloud logging service
Posted by GitBox <gi...@apache.org>.
shuaijinchao commented on a change in pull request #5538:
URL: https://github.com/apache/apisix/pull/5538#discussion_r753926924
##########
File path: Makefile
##########
@@ -303,6 +303,9 @@ install: runtime
$(ENV_INSTALL) -d $(ENV_INST_LUADIR)/apisix/plugins/limit-count
$(ENV_INSTALL) apisix/plugins/limit-count/*.lua $(ENV_INST_LUADIR)/apisix/plugins/limit-count/
+ $(ENV_INSTALL) -d $(ENV_INST_LUADIR)/apisix/plugins/google-logging
+ $(ENV_INSTALL) apisix/plugins/google-logging/*.lua $(ENV_INST_LUADIR)/apisix/plugins/google-logging/
Review comment:
Are you referring to the name of the directory or the name of the plugin?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] shuaijinchao commented on a change in pull request #5538: feat(plugin): support google cloud logging service
Posted by GitBox <gi...@apache.org>.
shuaijinchao commented on a change in pull request #5538:
URL: https://github.com/apache/apisix/pull/5538#discussion_r754067820
##########
File path: apisix/plugins/google-logging.lua
##########
@@ -0,0 +1,298 @@
+--
+-- Licensed to the Apache Software Foundation (ASF) under one or more
+-- contributor license agreements. See the NOTICE file distributed with
+-- this work for additional information regarding copyright ownership.
+-- The ASF licenses this file to You under the Apache License, Version 2.0
+-- (the "License"); you may not use this file except in compliance with
+-- the License. You may obtain a copy of the License at
+--
+-- http://www.apache.org/licenses/LICENSE-2.0
+--
+-- Unless required by applicable law or agreed to in writing, software
+-- distributed under the License is distributed on an "AS IS" BASIS,
+-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+-- See the License for the specific language governing permissions and
+-- limitations under the License.
+--
+
+local core = require("apisix.core")
+local ngx = ngx
+local tostring = tostring
+local ipairs = ipairs
+local sub_str = string.sub
+local os_date = os.date
+
+local ngx_now = ngx.now
+local ngx_timer_at = ngx.timer.at
+local ngx_update_time = ngx.update_time
+
+local http = require("resty.http")
+local log_util = require("apisix.utils.log-util")
+local batch_processor = require("apisix.utils.batch-processor")
+local google_oauth = require("apisix.plugins.google-logging.oauth")
+
+
+local buffers = {}
+local auth_config_cache
+local stale_timer_running
+
+
+local plugin_name = "google-logging"
+local schema = {
+ type = "object",
+ properties = {
+ auth_config = {
+ type = "object",
+ properties = {
+ private_key = { type = "string" },
+ project_id = { type = "string" },
+ token_uri = {
+ type = "string",
+ default = "https://oauth2.googleapis.com/token"
+ },
+ -- https://developers.google.com/identity/protocols/oauth2/scopes#logging
+ scopes = {
+ type = "array",
+ items = {
+ description = "Google OAuth2 Authorization Scopes",
+ type = "string",
+ },
+ minItems = 1,
+ uniqueItems = true,
+ default = {
+ "https://www.googleapis.com/auth/logging.read",
+ "https://www.googleapis.com/auth/logging.write",
+ "https://www.googleapis.com/auth/logging.admin",
+ "https://www.googleapis.com/auth/cloud-platform"
+ }
+ },
+ entries_uri = {
+ type = "string",
+ default = "https://logging.googleapis.com/v2/entries:write"
+ },
+ },
+ required = { "private_key", "project_id", "token_uri" }
+ },
+ auth_file = { type = "string" },
+ -- https://cloud.google.com/logging/docs/reference/v2/rest/v2/MonitoredResource
+ resource = {
+ type = "object",
+ properties = {
+ type = { type = "string" },
+ labels = { type = "object" }
+ },
+ default = {
+ type = "global"
+ },
+ required = { "type" }
+ },
+ -- https://cloud.google.com/logging/docs/reference/v2/rest/v2/LogEntry
+ log_id = { type = "string", default = "apisix.apache.org%2Flogs" },
+ max_retry_count = { type = "integer", minimum = 0, default = 0 },
+ retry_delay = { type = "integer", minimum = 0, default = 1 },
+ buffer_duration = { type = "integer", minimum = 1, default = 60 },
+ inactive_timeout = { type = "integer", minimum = 1, default = 10 },
+ batch_max_size = { type = "integer", minimum = 1, default = 100 },
+ },
+ oneOf = {
+ { required = { "auth_config" } },
+ { required = { "auth_file" } },
+ },
+}
+
+
+-- remove stale objects from the memory after timer expires
+local function remove_stale_objects(premature)
+ if premature then
+ return
+ end
+
+ for key, batch in ipairs(buffers) do
+ if #batch.entry_buffer.entries == 0 and #batch.batch_to_process == 0 then
+ core.log.warn("removing batch processor stale object, route id:", tostring(key))
+ buffers[key] = nil
+ end
+ end
+
+ stale_timer_running = false
+end
+
+
+local function send_to_google(oauth, entries)
+ local http_new = http.new()
+ local res, err = http_new:request_uri(oauth.entries_uri, {
+ ssl_verify = false,
+ method = "POST",
+ body = core.json.encode({
+ entries = entries,
+ partialSuccess = false,
+ }),
+ headers = {
+ ["Content-Type"] = "application/json",
+ ["Authorization"] = "Bearer " .. oauth:get_access_token(),
+ },
+ })
+
+ if err then
+ return nil, "failed to write log to google, ", err
Review comment:
updated
##########
File path: Makefile
##########
@@ -303,6 +303,9 @@ install: runtime
$(ENV_INSTALL) -d $(ENV_INST_LUADIR)/apisix/plugins/limit-count
$(ENV_INSTALL) apisix/plugins/limit-count/*.lua $(ENV_INST_LUADIR)/apisix/plugins/limit-count/
+ $(ENV_INSTALL) -d $(ENV_INST_LUADIR)/apisix/plugins/google-logging
+ $(ENV_INSTALL) apisix/plugins/google-logging/*.lua $(ENV_INST_LUADIR)/apisix/plugins/google-logging/
Review comment:
updated
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] shuaijinchao commented on a change in pull request #5538: feat(plugin): support google cloud logging service
Posted by GitBox <gi...@apache.org>.
shuaijinchao commented on a change in pull request #5538:
URL: https://github.com/apache/apisix/pull/5538#discussion_r754070885
##########
File path: docs/en/latest/plugins/google-logging.md
##########
@@ -0,0 +1,155 @@
+---
+title: google-logging
+---
+
+<!--
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+-->
+
+## Summary
+
+- [**Name**](#name)
+- [**Attributes**](#attributes)
+- [**How To Enable**](#how-to-enable)
+- [**Test Plugin**](#test-plugin)
+- [**Disable Plugin**](#disable-plugin)
+
+## Name
+
+`google-logging` plugin is used to send the access log of `Apache APISIX` to the [Google Cloud Logging Service](https://cloud.google.com/logging/).
+
+This plugin provides the ability to push Log data as a batch to Google Cloud logging Service.
+
+For more info on Batch-Processor in Apache APISIX please refer:
+[Batch-Processor](../batch-processor.md)
+
+## Attributes
+
+| Name | Requirement | Default | Description |
+| ----------------------- | ------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| auth_config | Semi-optional | | one of `auth_config` or `auth_file` must be configured |
+| auth_config.private_key | Required | | the private key parameters of the Google service account |
+| auth_config.project_id | Required | | the project id parameters of the Google service account |
+| auth_config.token_uri | Optional | https://oauth2.googleapis.com/token | the token uri parameters of the Google service account |
+| auth_config.entries_uri | Optional | https://logging.googleapis.com/v2/entries:write | google logging service API |
+| auth_config.scopes | Optional | ["https://www.googleapis.com/auth/logging.read","https://www.googleapis.com/auth/logging.write","https://www.googleapis.com/auth/logging.admin","https://www.googleapis.com/auth/cloud-platform"] | the access scopes parameters of the Google service account, refer to: [OAuth 2.0 Scopes for Google APIs](https://developers.google.com/identity/protocols/oauth2/scopes#logging) |
+| auth_file | Semi-optional | | path to the google service account json file（Semi-optional, one of auth_config or auth_file must be configured） |
+| resource | Optional | {"type": "global"} | the Google monitor resource, refer to: [MonitoredResource](https://cloud.google.com/logging/docs/reference/v2/rest/v2/MonitoredResource) |
+| log_id | Optional | apisix.apache.org%2Flogs | google logging id, refer to: [LogEntry](https://cloud.google.com/logging/docs/reference/v2/rest/v2/LogEntry) |
+| max_retry_count | Optional | 0 | max number of retries before removing from the processing pipe line |
+| retry_delay | Optional | 1 | number of seconds the process execution should be delayed if the execution fails |
+| buffer_duration | Optional | 60 | max age in seconds of the oldest entry in a batch before the batch must be processed |
+| inactive_timeout | Optional | 10 | max age in seconds when the buffer will be flushed if inactive |
+| batch_max_size | Optional | 100 | max size of each batch |
+
+## How To Enable
+
+The following is an example on how to enable the `google-logging` for a specific route.
Review comment:
updated
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] spacewander commented on a change in pull request #5538: feat(plugin): support google cloud logging service
Posted by GitBox <gi...@apache.org>.
spacewander commented on a change in pull request #5538:
URL: https://github.com/apache/apisix/pull/5538#discussion_r753787079
##########
File path: apisix/utils/log-util.lua
##########
@@ -94,6 +94,7 @@ local function get_full_log(ngx, conf)
local log = {
request = {
+ id = var.request_id,
Review comment:
Better not add a new field to the common library without full discussion.
##########
File path: apisix/core/utils.lua
##########
@@ -260,6 +260,18 @@ function _M.gethostname()
end
+function _M.get_file(file_name)
Review comment:
Could we put it into core/io.lua and let core.request refer it?
##########
File path: apisix/plugins/google-logging.lua
##########
@@ -0,0 +1,298 @@
+--
+-- Licensed to the Apache Software Foundation (ASF) under one or more
+-- contributor license agreements. See the NOTICE file distributed with
+-- this work for additional information regarding copyright ownership.
+-- The ASF licenses this file to You under the Apache License, Version 2.0
+-- (the "License"); you may not use this file except in compliance with
+-- the License. You may obtain a copy of the License at
+--
+-- http://www.apache.org/licenses/LICENSE-2.0
+--
+-- Unless required by applicable law or agreed to in writing, software
+-- distributed under the License is distributed on an "AS IS" BASIS,
+-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+-- See the License for the specific language governing permissions and
+-- limitations under the License.
+--
+
+local core = require("apisix.core")
+local ngx = ngx
+local tostring = tostring
+local ipairs = ipairs
+local sub_str = string.sub
+local os_date = os.date
+
+local ngx_now = ngx.now
+local ngx_timer_at = ngx.timer.at
+local ngx_update_time = ngx.update_time
+
+local http = require("resty.http")
+local log_util = require("apisix.utils.log-util")
+local batch_processor = require("apisix.utils.batch-processor")
+local google_oauth = require("apisix.plugins.google-logging.oauth")
+
+
+local buffers = {}
+local auth_config_cache
+local stale_timer_running
+
+
+local plugin_name = "google-logging"
+local schema = {
+ type = "object",
+ properties = {
+ auth_config = {
+ type = "object",
+ properties = {
+ private_key = { type = "string" },
+ project_id = { type = "string" },
+ token_uri = {
+ type = "string",
+ default = "https://oauth2.googleapis.com/token"
+ },
+ -- https://developers.google.com/identity/protocols/oauth2/scopes#logging
+ scopes = {
+ type = "array",
+ items = {
+ description = "Google OAuth2 Authorization Scopes",
+ type = "string",
+ },
+ minItems = 1,
+ uniqueItems = true,
+ default = {
+ "https://www.googleapis.com/auth/logging.read",
+ "https://www.googleapis.com/auth/logging.write",
+ "https://www.googleapis.com/auth/logging.admin",
+ "https://www.googleapis.com/auth/cloud-platform"
+ }
+ },
+ entries_uri = {
+ type = "string",
+ default = "https://logging.googleapis.com/v2/entries:write"
+ },
+ },
+ required = { "private_key", "project_id", "token_uri" }
+ },
+ auth_file = { type = "string" },
+ -- https://cloud.google.com/logging/docs/reference/v2/rest/v2/MonitoredResource
+ resource = {
+ type = "object",
+ properties = {
+ type = { type = "string" },
+ labels = { type = "object" }
+ },
+ default = {
+ type = "global"
+ },
+ required = { "type" }
+ },
+ -- https://cloud.google.com/logging/docs/reference/v2/rest/v2/LogEntry
+ log_id = { type = "string", default = "apisix.apache.org%2Flogs" },
+ max_retry_count = { type = "integer", minimum = 0, default = 0 },
+ retry_delay = { type = "integer", minimum = 0, default = 1 },
+ buffer_duration = { type = "integer", minimum = 1, default = 60 },
+ inactive_timeout = { type = "integer", minimum = 1, default = 10 },
+ batch_max_size = { type = "integer", minimum = 1, default = 100 },
+ },
+ oneOf = {
+ { required = { "auth_config" } },
+ { required = { "auth_file" } },
+ },
+}
+
+
+-- remove stale objects from the memory after timer expires
+local function remove_stale_objects(premature)
+ if premature then
+ return
+ end
+
+ for key, batch in ipairs(buffers) do
+ if #batch.entry_buffer.entries == 0 and #batch.batch_to_process == 0 then
+ core.log.warn("removing batch processor stale object, route id:", tostring(key))
+ buffers[key] = nil
+ end
+ end
+
+ stale_timer_running = false
+end
+
+
+local function send_to_google(oauth, entries)
+ local http_new = http.new()
+ local res, err = http_new:request_uri(oauth.entries_uri, {
+ ssl_verify = false,
Review comment:
We need to make ssl_verify configurable and default to true.
##########
File path: docs/en/latest/plugins/google-logging.md
##########
@@ -0,0 +1,155 @@
+---
+title: google-logging
+---
+
+<!--
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+-->
+
+## Summary
+
+- [**Name**](#name)
+- [**Attributes**](#attributes)
+- [**How To Enable**](#how-to-enable)
+- [**Test Plugin**](#test-plugin)
+- [**Disable Plugin**](#disable-plugin)
+
+## Name
+
+`google-logging` plugin is used to send the access log of `Apache APISIX` to the [Google Cloud Logging Service](https://cloud.google.com/logging/).
+
+This plugin provides the ability to push Log data as a batch to Google Cloud logging Service.
+
+For more info on Batch-Processor in Apache APISIX please refer:
+[Batch-Processor](../batch-processor.md)
+
+## Attributes
+
+| Name | Requirement | Default | Description |
+| ----------------------- | ------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| auth_config | Semi-optional | | one of `auth_config` or `auth_file` must be configured |
+| auth_config.private_key | Required | | the private key parameters of the Google service account |
+| auth_config.project_id | Required | | the project id parameters of the Google service account |
+| auth_config.token_uri | Optional | https://oauth2.googleapis.com/token | the token uri parameters of the Google service account |
+| auth_config.entries_uri | Optional | https://logging.googleapis.com/v2/entries:write | google logging service API |
+| auth_config.scopes | Optional | ["https://www.googleapis.com/auth/logging.read","https://www.googleapis.com/auth/logging.write","https://www.googleapis.com/auth/logging.admin","https://www.googleapis.com/auth/cloud-platform"] | the access scopes parameters of the Google service account, refer to: [OAuth 2.0 Scopes for Google APIs](https://developers.google.com/identity/protocols/oauth2/scopes#logging) |
+| auth_file | Semi-optional | | path to the google service account json file（Semi-optional, one of auth_config or auth_file must be configured） |
+| resource | Optional | {"type": "global"} | the Google monitor resource, refer to: [MonitoredResource](https://cloud.google.com/logging/docs/reference/v2/rest/v2/MonitoredResource) |
+| log_id | Optional | apisix.apache.org%2Flogs | google logging id, refer to: [LogEntry](https://cloud.google.com/logging/docs/reference/v2/rest/v2/LogEntry) |
+| max_retry_count | Optional | 0 | max number of retries before removing from the processing pipe line |
+| retry_delay | Optional | 1 | number of seconds the process execution should be delayed if the execution fails |
+| buffer_duration | Optional | 60 | max age in seconds of the oldest entry in a batch before the batch must be processed |
+| inactive_timeout | Optional | 10 | max age in seconds when the buffer will be flushed if inactive |
+| batch_max_size | Optional | 100 | max size of each batch |
+
+## How To Enable
+
+The following is an example on how to enable the `google-logging` for a specific route.
+
+### Full configuration
+
+```shell
+curl http://127.0.0.1:9080/apisix/admin/routes/1 -H 'X-API-KEY: edd1c9f034335f136f87ad84b625c8f1' -X PUT -d '
+{
+ "plugins": {
+ "google-logging": {
+ "auth_config":{
+ "project_id":"apisix",
+ "private_key":"-----BEGIN RSA PRIVATE KEY-----KEY-----END RSA PRIVATE KEY-----",
Review comment:
Could we use a valid pkey?
##########
File path: apisix/plugins/google-logging.lua
##########
@@ -0,0 +1,298 @@
+--
+-- Licensed to the Apache Software Foundation (ASF) under one or more
+-- contributor license agreements. See the NOTICE file distributed with
+-- this work for additional information regarding copyright ownership.
+-- The ASF licenses this file to You under the Apache License, Version 2.0
+-- (the "License"); you may not use this file except in compliance with
+-- the License. You may obtain a copy of the License at
+--
+-- http://www.apache.org/licenses/LICENSE-2.0
+--
+-- Unless required by applicable law or agreed to in writing, software
+-- distributed under the License is distributed on an "AS IS" BASIS,
+-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+-- See the License for the specific language governing permissions and
+-- limitations under the License.
+--
+
+local core = require("apisix.core")
+local ngx = ngx
+local tostring = tostring
+local ipairs = ipairs
+local sub_str = string.sub
+local os_date = os.date
+
+local ngx_now = ngx.now
+local ngx_timer_at = ngx.timer.at
+local ngx_update_time = ngx.update_time
+
+local http = require("resty.http")
+local log_util = require("apisix.utils.log-util")
+local batch_processor = require("apisix.utils.batch-processor")
+local google_oauth = require("apisix.plugins.google-logging.oauth")
+
+
+local buffers = {}
+local auth_config_cache
+local stale_timer_running
+
+
+local plugin_name = "google-logging"
+local schema = {
+ type = "object",
+ properties = {
+ auth_config = {
+ type = "object",
+ properties = {
+ private_key = { type = "string" },
+ project_id = { type = "string" },
+ token_uri = {
+ type = "string",
+ default = "https://oauth2.googleapis.com/token"
+ },
+ -- https://developers.google.com/identity/protocols/oauth2/scopes#logging
+ scopes = {
+ type = "array",
+ items = {
+ description = "Google OAuth2 Authorization Scopes",
+ type = "string",
+ },
+ minItems = 1,
+ uniqueItems = true,
+ default = {
+ "https://www.googleapis.com/auth/logging.read",
+ "https://www.googleapis.com/auth/logging.write",
+ "https://www.googleapis.com/auth/logging.admin",
+ "https://www.googleapis.com/auth/cloud-platform"
+ }
+ },
+ entries_uri = {
+ type = "string",
+ default = "https://logging.googleapis.com/v2/entries:write"
+ },
+ },
+ required = { "private_key", "project_id", "token_uri" }
+ },
+ auth_file = { type = "string" },
+ -- https://cloud.google.com/logging/docs/reference/v2/rest/v2/MonitoredResource
+ resource = {
+ type = "object",
+ properties = {
+ type = { type = "string" },
+ labels = { type = "object" }
+ },
+ default = {
+ type = "global"
+ },
+ required = { "type" }
+ },
+ -- https://cloud.google.com/logging/docs/reference/v2/rest/v2/LogEntry
+ log_id = { type = "string", default = "apisix.apache.org%2Flogs" },
+ max_retry_count = { type = "integer", minimum = 0, default = 0 },
+ retry_delay = { type = "integer", minimum = 0, default = 1 },
+ buffer_duration = { type = "integer", minimum = 1, default = 60 },
+ inactive_timeout = { type = "integer", minimum = 1, default = 10 },
+ batch_max_size = { type = "integer", minimum = 1, default = 100 },
+ },
+ oneOf = {
+ { required = { "auth_config" } },
+ { required = { "auth_file" } },
+ },
+}
+
+
+-- remove stale objects from the memory after timer expires
+local function remove_stale_objects(premature)
+ if premature then
+ return
+ end
+
+ for key, batch in ipairs(buffers) do
+ if #batch.entry_buffer.entries == 0 and #batch.batch_to_process == 0 then
+ core.log.warn("removing batch processor stale object, route id:", tostring(key))
+ buffers[key] = nil
+ end
+ end
+
+ stale_timer_running = false
+end
+
+
+local function send_to_google(oauth, entries)
+ local http_new = http.new()
+ local res, err = http_new:request_uri(oauth.entries_uri, {
+ ssl_verify = false,
+ method = "POST",
+ body = core.json.encode({
+ entries = entries,
+ partialSuccess = false,
+ }),
+ headers = {
+ ["Content-Type"] = "application/json",
+ ["Authorization"] = "Bearer " .. oauth:get_access_token(),
+ },
+ })
+
+ if err then
+ return nil, "failed to write log to google, ", err
+ end
+
+ if res.status ~= 200 then
+ return nil, res.body
+ end
+
+ return res.body
+end
+
+
+local function get_auth_config(config)
+ local err
+ local auth_config = {}
+ if config.auth_config then
+ auth_config = config.auth_config
+ end
+
+ if config.auth_file then
+ local file_content
+ file_content, err = core.utils.get_file(config.auth_file)
+ if file_content then
+ auth_config = core.json.decode(file_content)
+ end
+ end
+
+ if err then
+ return nil, err
+ end
+
+ return auth_config
+end
+
+
+local function get_logger_buffer(conf, ctx)
+ local oauth_client = google_oauth:new(auth_config_cache)
+
+ local process = function(entries)
+ return send_to_google(oauth_client, entries)
+ end
+
+ local config = {
+ name = conf.name or plugin_name,
+ retry_delay = conf.retry_delay,
+ batch_max_size = conf.batch_max_size,
+ max_retry_count = conf.max_retry_count,
+ buffer_duration = conf.buffer_duration,
+ inactive_timeout = conf.inactive_timeout,
+ route_id = ctx.var.route_id,
+ server_addr = ctx.var.server_addr,
+ }
+
+ local buffer, err = batch_processor:new(process, config)
+
+ if not buffer then
+ return nil, "error when creating the batch processor: " .. err
+ end
+
+ return buffer
+end
+
+
+local function get_utc_timestamp()
+ ngx_update_time()
+ local now = tostring(ngx_now())
+ local pos = core.string.rfind_char(now, ".", #now - 1)
+ local second = now
+ local millisecond = 0
+ if pos then
+ second = sub_str(now, 1, pos - 1)
+ millisecond = sub_str(now, pos + 1)
+ end
+ return os_date("!%Y-%m-%dT%T.", second) .. core.string.format("%03dZ", millisecond)
+end
+
+
+local function get_logger_entry(conf)
+ if not auth_config_cache then
Review comment:
It would be better to associate the auth config cache into get_auth_config.
##########
File path: Makefile
##########
@@ -303,6 +303,9 @@ install: runtime
$(ENV_INSTALL) -d $(ENV_INST_LUADIR)/apisix/plugins/limit-count
$(ENV_INSTALL) apisix/plugins/limit-count/*.lua $(ENV_INST_LUADIR)/apisix/plugins/limit-count/
+ $(ENV_INSTALL) -d $(ENV_INST_LUADIR)/apisix/plugins/google-logging
+ $(ENV_INSTALL) apisix/plugins/google-logging/*.lua $(ENV_INST_LUADIR)/apisix/plugins/google-logging/
Review comment:
We should use the name `google-cloud-logging` instead of google-logging? It is confusing if we don't mention which google product we are working with.
##########
File path: apisix/plugins/google-logging.lua
##########
@@ -0,0 +1,298 @@
+--
+-- Licensed to the Apache Software Foundation (ASF) under one or more
+-- contributor license agreements. See the NOTICE file distributed with
+-- this work for additional information regarding copyright ownership.
+-- The ASF licenses this file to You under the Apache License, Version 2.0
+-- (the "License"); you may not use this file except in compliance with
+-- the License. You may obtain a copy of the License at
+--
+-- http://www.apache.org/licenses/LICENSE-2.0
+--
+-- Unless required by applicable law or agreed to in writing, software
+-- distributed under the License is distributed on an "AS IS" BASIS,
+-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+-- See the License for the specific language governing permissions and
+-- limitations under the License.
+--
+
+local core = require("apisix.core")
+local ngx = ngx
+local tostring = tostring
+local ipairs = ipairs
+local sub_str = string.sub
+local os_date = os.date
+
+local ngx_now = ngx.now
+local ngx_timer_at = ngx.timer.at
+local ngx_update_time = ngx.update_time
+
+local http = require("resty.http")
+local log_util = require("apisix.utils.log-util")
+local batch_processor = require("apisix.utils.batch-processor")
+local google_oauth = require("apisix.plugins.google-logging.oauth")
+
+
+local buffers = {}
+local auth_config_cache
+local stale_timer_running
+
+
+local plugin_name = "google-logging"
+local schema = {
+ type = "object",
+ properties = {
+ auth_config = {
+ type = "object",
+ properties = {
+ private_key = { type = "string" },
+ project_id = { type = "string" },
+ token_uri = {
+ type = "string",
+ default = "https://oauth2.googleapis.com/token"
+ },
+ -- https://developers.google.com/identity/protocols/oauth2/scopes#logging
+ scopes = {
+ type = "array",
+ items = {
+ description = "Google OAuth2 Authorization Scopes",
+ type = "string",
+ },
+ minItems = 1,
+ uniqueItems = true,
+ default = {
+ "https://www.googleapis.com/auth/logging.read",
+ "https://www.googleapis.com/auth/logging.write",
+ "https://www.googleapis.com/auth/logging.admin",
+ "https://www.googleapis.com/auth/cloud-platform"
+ }
+ },
+ entries_uri = {
+ type = "string",
+ default = "https://logging.googleapis.com/v2/entries:write"
+ },
+ },
+ required = { "private_key", "project_id", "token_uri" }
+ },
+ auth_file = { type = "string" },
+ -- https://cloud.google.com/logging/docs/reference/v2/rest/v2/MonitoredResource
+ resource = {
+ type = "object",
+ properties = {
+ type = { type = "string" },
+ labels = { type = "object" }
+ },
+ default = {
+ type = "global"
+ },
+ required = { "type" }
+ },
+ -- https://cloud.google.com/logging/docs/reference/v2/rest/v2/LogEntry
+ log_id = { type = "string", default = "apisix.apache.org%2Flogs" },
+ max_retry_count = { type = "integer", minimum = 0, default = 0 },
+ retry_delay = { type = "integer", minimum = 0, default = 1 },
+ buffer_duration = { type = "integer", minimum = 1, default = 60 },
+ inactive_timeout = { type = "integer", minimum = 1, default = 10 },
+ batch_max_size = { type = "integer", minimum = 1, default = 100 },
+ },
+ oneOf = {
+ { required = { "auth_config" } },
+ { required = { "auth_file" } },
+ },
+}
+
+
+-- remove stale objects from the memory after timer expires
+local function remove_stale_objects(premature)
+ if premature then
+ return
+ end
+
+ for key, batch in ipairs(buffers) do
+ if #batch.entry_buffer.entries == 0 and #batch.batch_to_process == 0 then
+ core.log.warn("removing batch processor stale object, route id:", tostring(key))
+ buffers[key] = nil
+ end
+ end
+
+ stale_timer_running = false
+end
+
+
+local function send_to_google(oauth, entries)
+ local http_new = http.new()
+ local res, err = http_new:request_uri(oauth.entries_uri, {
+ ssl_verify = false,
+ method = "POST",
+ body = core.json.encode({
+ entries = entries,
+ partialSuccess = false,
+ }),
+ headers = {
+ ["Content-Type"] = "application/json",
+ ["Authorization"] = "Bearer " .. oauth:get_access_token(),
+ },
+ })
+
+ if err then
+ return nil, "failed to write log to google, ", err
+ end
+
+ if res.status ~= 200 then
+ return nil, res.body
+ end
+
+ return res.body
+end
+
+
+local function get_auth_config(config)
+ local err
+ local auth_config = {}
+ if config.auth_config then
+ auth_config = config.auth_config
+ end
+
+ if config.auth_file then
+ local file_content
+ file_content, err = core.utils.get_file(config.auth_file)
+ if file_content then
+ auth_config = core.json.decode(file_content)
+ end
+ end
+
+ if err then
+ return nil, err
+ end
+
+ return auth_config
+end
+
+
+local function get_logger_buffer(conf, ctx)
+ local oauth_client = google_oauth:new(auth_config_cache)
+
+ local process = function(entries)
+ return send_to_google(oauth_client, entries)
+ end
+
+ local config = {
+ name = conf.name or plugin_name,
+ retry_delay = conf.retry_delay,
+ batch_max_size = conf.batch_max_size,
+ max_retry_count = conf.max_retry_count,
+ buffer_duration = conf.buffer_duration,
+ inactive_timeout = conf.inactive_timeout,
+ route_id = ctx.var.route_id,
+ server_addr = ctx.var.server_addr,
+ }
+
+ local buffer, err = batch_processor:new(process, config)
+
+ if not buffer then
+ return nil, "error when creating the batch processor: " .. err
+ end
+
+ return buffer
+end
+
+
+local function get_utc_timestamp()
+ ngx_update_time()
+ local now = tostring(ngx_now())
+ local pos = core.string.rfind_char(now, ".", #now - 1)
+ local second = now
+ local millisecond = 0
+ if pos then
+ second = sub_str(now, 1, pos - 1)
+ millisecond = sub_str(now, pos + 1)
+ end
+ return os_date("!%Y-%m-%dT%T.", second) .. core.string.format("%03dZ", millisecond)
+end
+
+
+local function get_logger_entry(conf)
+ if not auth_config_cache then
+ local auth_config, err = get_auth_config(conf)
+ if err or not auth_config.project_id or not auth_config.private_key then
+ return nil, "failed to get google authentication configuration" .. err
+ end
+
+ auth_config_cache = auth_config
+ end
+
+ local entry = log_util.get_full_log(ngx, conf)
+ local google_entry = {
+ httpRequest = {
+ requestMethod = entry.request.method,
+ requestUrl = entry.request.url,
+ requestSize = entry.request.size,
+ status = entry.response.status,
+ responseSize = entry.response.size,
+ userAgent = entry.request.headers and entry.request.headers["user-agent"],
+ remoteIp = entry.client_ip,
+ serverIp = entry.upstream,
+ latency = tostring(core.string.format("%0.3f", entry.latency / 1000)) .. "s"
+ },
+ jsonPayload = {
+ route_id = entry.route_id,
+ service_id = entry.service_id,
+ },
+ labels = {
+ source = "apache-apisix-google-logging"
+ },
+ timestamp = get_utc_timestamp(),
+ resource = conf.resource,
+ insertId = entry.request.id,
Review comment:
Can we add the var.request_id directly?
##########
File path: docs/en/latest/plugins/google-logging.md
##########
@@ -0,0 +1,155 @@
+---
+title: google-logging
+---
+
+<!--
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+-->
+
+## Summary
+
+- [**Name**](#name)
+- [**Attributes**](#attributes)
+- [**How To Enable**](#how-to-enable)
+- [**Test Plugin**](#test-plugin)
+- [**Disable Plugin**](#disable-plugin)
+
+## Name
+
+`google-logging` plugin is used to send the access log of `Apache APISIX` to the [Google Cloud Logging Service](https://cloud.google.com/logging/).
+
+This plugin provides the ability to push Log data as a batch to Google Cloud logging Service.
Review comment:
```suggestion
This plugin provides the ability to push log data as a batch to Google Cloud logging Service.
```
##########
File path: apisix/plugins/google-logging.lua
##########
@@ -0,0 +1,298 @@
+--
+-- Licensed to the Apache Software Foundation (ASF) under one or more
+-- contributor license agreements. See the NOTICE file distributed with
+-- this work for additional information regarding copyright ownership.
+-- The ASF licenses this file to You under the Apache License, Version 2.0
+-- (the "License"); you may not use this file except in compliance with
+-- the License. You may obtain a copy of the License at
+--
+-- http://www.apache.org/licenses/LICENSE-2.0
+--
+-- Unless required by applicable law or agreed to in writing, software
+-- distributed under the License is distributed on an "AS IS" BASIS,
+-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+-- See the License for the specific language governing permissions and
+-- limitations under the License.
+--
+
+local core = require("apisix.core")
+local ngx = ngx
+local tostring = tostring
+local ipairs = ipairs
+local sub_str = string.sub
+local os_date = os.date
+
+local ngx_now = ngx.now
+local ngx_timer_at = ngx.timer.at
+local ngx_update_time = ngx.update_time
+
+local http = require("resty.http")
+local log_util = require("apisix.utils.log-util")
+local batch_processor = require("apisix.utils.batch-processor")
+local google_oauth = require("apisix.plugins.google-logging.oauth")
+
+
+local buffers = {}
+local auth_config_cache
+local stale_timer_running
+
+
+local plugin_name = "google-logging"
+local schema = {
+ type = "object",
+ properties = {
+ auth_config = {
+ type = "object",
+ properties = {
+ private_key = { type = "string" },
+ project_id = { type = "string" },
+ token_uri = {
+ type = "string",
+ default = "https://oauth2.googleapis.com/token"
+ },
+ -- https://developers.google.com/identity/protocols/oauth2/scopes#logging
+ scopes = {
+ type = "array",
+ items = {
+ description = "Google OAuth2 Authorization Scopes",
+ type = "string",
+ },
+ minItems = 1,
+ uniqueItems = true,
+ default = {
+ "https://www.googleapis.com/auth/logging.read",
+ "https://www.googleapis.com/auth/logging.write",
+ "https://www.googleapis.com/auth/logging.admin",
+ "https://www.googleapis.com/auth/cloud-platform"
+ }
+ },
+ entries_uri = {
+ type = "string",
+ default = "https://logging.googleapis.com/v2/entries:write"
+ },
+ },
+ required = { "private_key", "project_id", "token_uri" }
+ },
+ auth_file = { type = "string" },
+ -- https://cloud.google.com/logging/docs/reference/v2/rest/v2/MonitoredResource
+ resource = {
+ type = "object",
+ properties = {
+ type = { type = "string" },
+ labels = { type = "object" }
+ },
+ default = {
+ type = "global"
+ },
+ required = { "type" }
+ },
+ -- https://cloud.google.com/logging/docs/reference/v2/rest/v2/LogEntry
+ log_id = { type = "string", default = "apisix.apache.org%2Flogs" },
+ max_retry_count = { type = "integer", minimum = 0, default = 0 },
+ retry_delay = { type = "integer", minimum = 0, default = 1 },
+ buffer_duration = { type = "integer", minimum = 1, default = 60 },
+ inactive_timeout = { type = "integer", minimum = 1, default = 10 },
+ batch_max_size = { type = "integer", minimum = 1, default = 100 },
+ },
+ oneOf = {
+ { required = { "auth_config" } },
+ { required = { "auth_file" } },
+ },
+}
+
+
+-- remove stale objects from the memory after timer expires
+local function remove_stale_objects(premature)
+ if premature then
+ return
+ end
+
+ for key, batch in ipairs(buffers) do
+ if #batch.entry_buffer.entries == 0 and #batch.batch_to_process == 0 then
+ core.log.warn("removing batch processor stale object, route id:", tostring(key))
+ buffers[key] = nil
+ end
+ end
+
+ stale_timer_running = false
+end
+
+
+local function send_to_google(oauth, entries)
+ local http_new = http.new()
+ local res, err = http_new:request_uri(oauth.entries_uri, {
+ ssl_verify = false,
+ method = "POST",
+ body = core.json.encode({
+ entries = entries,
+ partialSuccess = false,
+ }),
+ headers = {
+ ["Content-Type"] = "application/json",
+ ["Authorization"] = "Bearer " .. oauth:get_access_token(),
+ },
+ })
+
+ if err then
+ return nil, "failed to write log to google, ", err
+ end
+
+ if res.status ~= 200 then
+ return nil, res.body
+ end
+
+ return res.body
+end
+
+
+local function get_auth_config(config)
+ local err
+ local auth_config = {}
+ if config.auth_config then
+ auth_config = config.auth_config
+ end
+
+ if config.auth_file then
+ local file_content
+ file_content, err = core.utils.get_file(config.auth_file)
+ if file_content then
+ auth_config = core.json.decode(file_content)
Review comment:
Better to check the decode err.
##########
File path: docs/en/latest/plugins/google-logging.md
##########
@@ -0,0 +1,155 @@
+---
+title: google-logging
+---
+
+<!--
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+-->
+
+## Summary
+
+- [**Name**](#name)
+- [**Attributes**](#attributes)
+- [**How To Enable**](#how-to-enable)
+- [**Test Plugin**](#test-plugin)
+- [**Disable Plugin**](#disable-plugin)
+
+## Name
+
+`google-logging` plugin is used to send the access log of `Apache APISIX` to the [Google Cloud Logging Service](https://cloud.google.com/logging/).
+
+This plugin provides the ability to push Log data as a batch to Google Cloud logging Service.
+
+For more info on Batch-Processor in Apache APISIX please refer:
+[Batch-Processor](../batch-processor.md)
+
+## Attributes
+
+| Name | Requirement | Default | Description |
+| ----------------------- | ------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| auth_config | Semi-optional | | one of `auth_config` or `auth_file` must be configured |
+| auth_config.private_key | Required | | the private key parameters of the Google service account |
+| auth_config.project_id | Required | | the project id parameters of the Google service account |
+| auth_config.token_uri | Optional | https://oauth2.googleapis.com/token | the token uri parameters of the Google service account |
+| auth_config.entries_uri | Optional | https://logging.googleapis.com/v2/entries:write | google logging service API |
+| auth_config.scopes | Optional | ["https://www.googleapis.com/auth/logging.read","https://www.googleapis.com/auth/logging.write","https://www.googleapis.com/auth/logging.admin","https://www.googleapis.com/auth/cloud-platform"] | the access scopes parameters of the Google service account, refer to: [OAuth 2.0 Scopes for Google APIs](https://developers.google.com/identity/protocols/oauth2/scopes#logging) |
+| auth_file | Semi-optional | | path to the google service account json file（Semi-optional, one of auth_config or auth_file must be configured） |
+| resource | Optional | {"type": "global"} | the Google monitor resource, refer to: [MonitoredResource](https://cloud.google.com/logging/docs/reference/v2/rest/v2/MonitoredResource) |
+| log_id | Optional | apisix.apache.org%2Flogs | google logging id, refer to: [LogEntry](https://cloud.google.com/logging/docs/reference/v2/rest/v2/LogEntry) |
+| max_retry_count | Optional | 0 | max number of retries before removing from the processing pipe line |
+| retry_delay | Optional | 1 | number of seconds the process execution should be delayed if the execution fails |
+| buffer_duration | Optional | 60 | max age in seconds of the oldest entry in a batch before the batch must be processed |
+| inactive_timeout | Optional | 10 | max age in seconds when the buffer will be flushed if inactive |
+| batch_max_size | Optional | 100 | max size of each batch |
+
+## How To Enable
+
+The following is an example on how to enable the `google-logging` for a specific route.
+
+### Full configuration
+
+```shell
+curl http://127.0.0.1:9080/apisix/admin/routes/1 -H 'X-API-KEY: edd1c9f034335f136f87ad84b625c8f1' -X PUT -d '
+{
+ "plugins": {
+ "google-logging": {
+ "auth_config":{
+ "project_id":"apisix",
+ "private_key":"-----BEGIN RSA PRIVATE KEY-----KEY-----END RSA PRIVATE KEY-----",
+ "token_uri":"https://oauth2.googleapis.com/token",
+ "scopes":[
+ "https://www.googleapis.com/auth/logging.admin"
+ ],
+ "entries_uri":"https://logging.googleapis.com/v2/entries:write"
+ },
+ "resource":{
+ "type":"global"
+ },
+ "log_id":"syslog",
Review comment:
The log_id is incorrect?
##########
File path: docs/en/latest/plugins/google-logging.md
##########
@@ -0,0 +1,155 @@
+---
+title: google-logging
+---
+
+<!--
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+-->
+
+## Summary
+
+- [**Name**](#name)
+- [**Attributes**](#attributes)
+- [**How To Enable**](#how-to-enable)
+- [**Test Plugin**](#test-plugin)
+- [**Disable Plugin**](#disable-plugin)
+
+## Name
+
+`google-logging` plugin is used to send the access log of `Apache APISIX` to the [Google Cloud Logging Service](https://cloud.google.com/logging/).
+
+This plugin provides the ability to push Log data as a batch to Google Cloud logging Service.
+
+For more info on Batch-Processor in Apache APISIX please refer:
+[Batch-Processor](../batch-processor.md)
+
+## Attributes
+
+| Name | Requirement | Default | Description |
+| ----------------------- | ------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| auth_config | Semi-optional | | one of `auth_config` or `auth_file` must be configured |
+| auth_config.private_key | Required | | the private key parameters of the Google service account |
+| auth_config.project_id | Required | | the project id parameters of the Google service account |
+| auth_config.token_uri | Optional | https://oauth2.googleapis.com/token | the token uri parameters of the Google service account |
+| auth_config.entries_uri | Optional | https://logging.googleapis.com/v2/entries:write | google logging service API |
+| auth_config.scopes | Optional | ["https://www.googleapis.com/auth/logging.read","https://www.googleapis.com/auth/logging.write","https://www.googleapis.com/auth/logging.admin","https://www.googleapis.com/auth/cloud-platform"] | the access scopes parameters of the Google service account, refer to: [OAuth 2.0 Scopes for Google APIs](https://developers.google.com/identity/protocols/oauth2/scopes#logging) |
+| auth_file | Semi-optional | | path to the google service account json file（Semi-optional, one of auth_config or auth_file must be configured） |
+| resource | Optional | {"type": "global"} | the Google monitor resource, refer to: [MonitoredResource](https://cloud.google.com/logging/docs/reference/v2/rest/v2/MonitoredResource) |
+| log_id | Optional | apisix.apache.org%2Flogs | google logging id, refer to: [LogEntry](https://cloud.google.com/logging/docs/reference/v2/rest/v2/LogEntry) |
+| max_retry_count | Optional | 0 | max number of retries before removing from the processing pipe line |
+| retry_delay | Optional | 1 | number of seconds the process execution should be delayed if the execution fails |
+| buffer_duration | Optional | 60 | max age in seconds of the oldest entry in a batch before the batch must be processed |
+| inactive_timeout | Optional | 10 | max age in seconds when the buffer will be flushed if inactive |
+| batch_max_size | Optional | 100 | max size of each batch |
+
+## How To Enable
+
+The following is an example on how to enable the `google-logging` for a specific route.
Review comment:
```suggestion
The following is an example of how to enable the `google-logging` for a specific route.
```
##########
File path: apisix/plugins/google-logging.lua
##########
@@ -0,0 +1,298 @@
+--
+-- Licensed to the Apache Software Foundation (ASF) under one or more
+-- contributor license agreements. See the NOTICE file distributed with
+-- this work for additional information regarding copyright ownership.
+-- The ASF licenses this file to You under the Apache License, Version 2.0
+-- (the "License"); you may not use this file except in compliance with
+-- the License. You may obtain a copy of the License at
+--
+-- http://www.apache.org/licenses/LICENSE-2.0
+--
+-- Unless required by applicable law or agreed to in writing, software
+-- distributed under the License is distributed on an "AS IS" BASIS,
+-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+-- See the License for the specific language governing permissions and
+-- limitations under the License.
+--
+
+local core = require("apisix.core")
+local ngx = ngx
+local tostring = tostring
+local ipairs = ipairs
+local sub_str = string.sub
+local os_date = os.date
+
+local ngx_now = ngx.now
+local ngx_timer_at = ngx.timer.at
+local ngx_update_time = ngx.update_time
+
+local http = require("resty.http")
+local log_util = require("apisix.utils.log-util")
+local batch_processor = require("apisix.utils.batch-processor")
+local google_oauth = require("apisix.plugins.google-logging.oauth")
+
+
+local buffers = {}
+local auth_config_cache
+local stale_timer_running
+
+
+local plugin_name = "google-logging"
+local schema = {
+ type = "object",
+ properties = {
+ auth_config = {
+ type = "object",
+ properties = {
+ private_key = { type = "string" },
+ project_id = { type = "string" },
+ token_uri = {
+ type = "string",
+ default = "https://oauth2.googleapis.com/token"
+ },
+ -- https://developers.google.com/identity/protocols/oauth2/scopes#logging
+ scopes = {
+ type = "array",
+ items = {
+ description = "Google OAuth2 Authorization Scopes",
+ type = "string",
+ },
+ minItems = 1,
+ uniqueItems = true,
+ default = {
+ "https://www.googleapis.com/auth/logging.read",
+ "https://www.googleapis.com/auth/logging.write",
+ "https://www.googleapis.com/auth/logging.admin",
+ "https://www.googleapis.com/auth/cloud-platform"
+ }
+ },
+ entries_uri = {
+ type = "string",
+ default = "https://logging.googleapis.com/v2/entries:write"
+ },
+ },
+ required = { "private_key", "project_id", "token_uri" }
+ },
+ auth_file = { type = "string" },
+ -- https://cloud.google.com/logging/docs/reference/v2/rest/v2/MonitoredResource
+ resource = {
+ type = "object",
+ properties = {
+ type = { type = "string" },
+ labels = { type = "object" }
+ },
+ default = {
+ type = "global"
+ },
+ required = { "type" }
+ },
+ -- https://cloud.google.com/logging/docs/reference/v2/rest/v2/LogEntry
+ log_id = { type = "string", default = "apisix.apache.org%2Flogs" },
+ max_retry_count = { type = "integer", minimum = 0, default = 0 },
+ retry_delay = { type = "integer", minimum = 0, default = 1 },
+ buffer_duration = { type = "integer", minimum = 1, default = 60 },
+ inactive_timeout = { type = "integer", minimum = 1, default = 10 },
+ batch_max_size = { type = "integer", minimum = 1, default = 100 },
+ },
+ oneOf = {
+ { required = { "auth_config" } },
+ { required = { "auth_file" } },
+ },
+}
+
+
+-- remove stale objects from the memory after timer expires
+local function remove_stale_objects(premature)
+ if premature then
+ return
+ end
+
+ for key, batch in ipairs(buffers) do
+ if #batch.entry_buffer.entries == 0 and #batch.batch_to_process == 0 then
+ core.log.warn("removing batch processor stale object, route id:", tostring(key))
+ buffers[key] = nil
+ end
+ end
+
+ stale_timer_running = false
+end
+
+
+local function send_to_google(oauth, entries)
+ local http_new = http.new()
+ local res, err = http_new:request_uri(oauth.entries_uri, {
+ ssl_verify = false,
+ method = "POST",
+ body = core.json.encode({
+ entries = entries,
+ partialSuccess = false,
+ }),
+ headers = {
+ ["Content-Type"] = "application/json",
+ ["Authorization"] = "Bearer " .. oauth:get_access_token(),
+ },
+ })
+
+ if err then
+ return nil, "failed to write log to google, ", err
+ end
+
+ if res.status ~= 200 then
+ return nil, res.body
+ end
+
+ return res.body
+end
+
+
+local function get_auth_config(config)
+ local err
+ local auth_config = {}
+ if config.auth_config then
+ auth_config = config.auth_config
+ end
+
+ if config.auth_file then
+ local file_content
+ file_content, err = core.utils.get_file(config.auth_file)
+ if file_content then
+ auth_config = core.json.decode(file_content)
+ end
+ end
+
+ if err then
+ return nil, err
+ end
+
+ return auth_config
+end
+
+
+local function get_logger_buffer(conf, ctx)
+ local oauth_client = google_oauth:new(auth_config_cache)
+
+ local process = function(entries)
+ return send_to_google(oauth_client, entries)
+ end
+
+ local config = {
+ name = conf.name or plugin_name,
+ retry_delay = conf.retry_delay,
+ batch_max_size = conf.batch_max_size,
+ max_retry_count = conf.max_retry_count,
+ buffer_duration = conf.buffer_duration,
+ inactive_timeout = conf.inactive_timeout,
+ route_id = ctx.var.route_id,
+ server_addr = ctx.var.server_addr,
+ }
+
+ local buffer, err = batch_processor:new(process, config)
+
+ if not buffer then
+ return nil, "error when creating the batch processor: " .. err
+ end
+
+ return buffer
+end
+
+
+local function get_utc_timestamp()
+ ngx_update_time()
+ local now = tostring(ngx_now())
Review comment:
We can use math.ceil to get the second & millisecond?
##########
File path: apisix/plugins/google-logging.lua
##########
@@ -0,0 +1,298 @@
+--
+-- Licensed to the Apache Software Foundation (ASF) under one or more
+-- contributor license agreements. See the NOTICE file distributed with
+-- this work for additional information regarding copyright ownership.
+-- The ASF licenses this file to You under the Apache License, Version 2.0
+-- (the "License"); you may not use this file except in compliance with
+-- the License. You may obtain a copy of the License at
+--
+-- http://www.apache.org/licenses/LICENSE-2.0
+--
+-- Unless required by applicable law or agreed to in writing, software
+-- distributed under the License is distributed on an "AS IS" BASIS,
+-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+-- See the License for the specific language governing permissions and
+-- limitations under the License.
+--
+
+local core = require("apisix.core")
+local ngx = ngx
+local tostring = tostring
+local ipairs = ipairs
+local sub_str = string.sub
+local os_date = os.date
+
+local ngx_now = ngx.now
+local ngx_timer_at = ngx.timer.at
+local ngx_update_time = ngx.update_time
+
+local http = require("resty.http")
+local log_util = require("apisix.utils.log-util")
+local batch_processor = require("apisix.utils.batch-processor")
+local google_oauth = require("apisix.plugins.google-logging.oauth")
+
+
+local buffers = {}
+local auth_config_cache
+local stale_timer_running
+
+
+local plugin_name = "google-logging"
+local schema = {
+ type = "object",
+ properties = {
+ auth_config = {
+ type = "object",
+ properties = {
+ private_key = { type = "string" },
+ project_id = { type = "string" },
+ token_uri = {
+ type = "string",
+ default = "https://oauth2.googleapis.com/token"
+ },
+ -- https://developers.google.com/identity/protocols/oauth2/scopes#logging
+ scopes = {
+ type = "array",
+ items = {
+ description = "Google OAuth2 Authorization Scopes",
+ type = "string",
+ },
+ minItems = 1,
+ uniqueItems = true,
+ default = {
+ "https://www.googleapis.com/auth/logging.read",
+ "https://www.googleapis.com/auth/logging.write",
+ "https://www.googleapis.com/auth/logging.admin",
+ "https://www.googleapis.com/auth/cloud-platform"
+ }
+ },
+ entries_uri = {
+ type = "string",
+ default = "https://logging.googleapis.com/v2/entries:write"
+ },
+ },
+ required = { "private_key", "project_id", "token_uri" }
+ },
+ auth_file = { type = "string" },
+ -- https://cloud.google.com/logging/docs/reference/v2/rest/v2/MonitoredResource
+ resource = {
+ type = "object",
+ properties = {
+ type = { type = "string" },
+ labels = { type = "object" }
+ },
+ default = {
+ type = "global"
+ },
+ required = { "type" }
+ },
+ -- https://cloud.google.com/logging/docs/reference/v2/rest/v2/LogEntry
+ log_id = { type = "string", default = "apisix.apache.org%2Flogs" },
+ max_retry_count = { type = "integer", minimum = 0, default = 0 },
+ retry_delay = { type = "integer", minimum = 0, default = 1 },
+ buffer_duration = { type = "integer", minimum = 1, default = 60 },
+ inactive_timeout = { type = "integer", minimum = 1, default = 10 },
+ batch_max_size = { type = "integer", minimum = 1, default = 100 },
+ },
+ oneOf = {
+ { required = { "auth_config" } },
+ { required = { "auth_file" } },
+ },
+}
+
+
+-- remove stale objects from the memory after timer expires
+local function remove_stale_objects(premature)
+ if premature then
+ return
+ end
+
+ for key, batch in ipairs(buffers) do
+ if #batch.entry_buffer.entries == 0 and #batch.batch_to_process == 0 then
+ core.log.warn("removing batch processor stale object, route id:", tostring(key))
+ buffers[key] = nil
+ end
+ end
+
+ stale_timer_running = false
+end
+
+
+local function send_to_google(oauth, entries)
+ local http_new = http.new()
+ local res, err = http_new:request_uri(oauth.entries_uri, {
+ ssl_verify = false,
+ method = "POST",
+ body = core.json.encode({
+ entries = entries,
+ partialSuccess = false,
+ }),
+ headers = {
+ ["Content-Type"] = "application/json",
+ ["Authorization"] = "Bearer " .. oauth:get_access_token(),
+ },
+ })
+
+ if err then
+ return nil, "failed to write log to google, ", err
Review comment:
```suggestion
return nil, "failed to write log to google: " .. err
```
##########
File path: apisix/plugins/google-logging/oauth.lua
##########
@@ -0,0 +1,128 @@
+--
+-- Licensed to the Apache Software Foundation (ASF) under one or more
+-- contributor license agreements. See the NOTICE file distributed with
+-- this work for additional information regarding copyright ownership.
+-- The ASF licenses this file to You under the Apache License, Version 2.0
+-- (the "License"); you may not use this file except in compliance with
+-- the License. You may obtain a copy of the License at
+--
+-- http://www.apache.org/licenses/LICENSE-2.0
+--
+-- Unless required by applicable law or agreed to in writing, software
+-- distributed under the License is distributed on an "AS IS" BASIS,
+-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+-- See the License for the specific language governing permissions and
+-- limitations under the License.
+--
+
+local core = require("apisix.core")
+local type = type
+local setmetatable = setmetatable
+
+local ngx_update_time = ngx.update_time
+local ngx_time = ngx.time
+local ngx_encode_args = ngx.encode_args
+
+local http = require("resty.http")
+local jwt = require("resty.jwt")
+
+
+local function get_timestamp()
+ ngx_update_time()
+ return ngx_time()
+end
+
+
+local _M = {}
+
+
+function _M:get_access_token()
+ if not self.access_token or get_timestamp() > self.access_token_expire_time - 60 then
+ self:refresh_access_token()
+ end
+ return self.access_token
+end
+
+
+function _M:refresh_access_token()
+ local http_new = http.new()
+ local res, err = http_new:request_uri(self.token_uri, {
+ ssl_verify = false,
+ method = "POST",
+ body = ngx_encode_args({
+ grant_type = "urn:ietf:params:oauth:grant-type:jwt-bearer",
+ assertion = self:generate_jwt_token()
+ }),
+ headers = {
+ ["Content-Type"] = "application/x-www-form-urlencoded",
+ },
+ })
+
+ if not res then
+ core.log.error("failed to refresh google oauth access token, ", err)
+ return
+ end
+
+ if res.status ~= 200 then
+ core.log.error("failed to refresh google oauth access token: ", res.body)
+ return
+ end
+
+ res = core.json.decode(res.body)
Review comment:
Need to check the decoded result.
##########
File path: t/plugin/google-logging.t
##########
@@ -0,0 +1,380 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+use t::APISIX 'no_plan';
+
+repeat_each(1);
+no_long_string();
+no_root_location();
+run_tests;
+
+
+add_block_preprocessor(sub {
+ my ($block) = @_;
+
+ if ((!defined $block->error_log) && (!defined $block->no_error_log)) {
+ $block->set_value("no_error_log", "[error]");
+ }
Review comment:
We also need to set the `request` by default too.
##########
File path: t/plugin/google-logging.t
##########
@@ -0,0 +1,380 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+use t::APISIX 'no_plan';
+
+repeat_each(1);
+no_long_string();
+no_root_location();
+run_tests;
+
+
+add_block_preprocessor(sub {
+ my ($block) = @_;
+
+ if ((!defined $block->error_log) && (!defined $block->no_error_log)) {
+ $block->set_value("no_error_log", "[error]");
+ }
+});
+
+__DATA__
+
+=== TEST 1: Full configuration verification (Auth File)
+--- config
+ location /t {
+ content_by_lua_block {
+ local plugin = require("apisix.plugins.google-logging")
+ local ok, err = plugin.check_schema({
+ auth_file = "/path/to/apache/apisix/auth.json",
+ resource = {
+ type = "global"
+ },
+ scopes = {
+ "https://www.googleapis.com/auth/logging.admin"
+ },
+ log_id = "syslog",
+ max_retry_count = 0,
+ retry_delay = 1,
+ buffer_duration = 60,
+ inactive_timeout = 10,
+ batch_max_size = 100,
+ })
+
+ if not ok then
+ ngx.say(err)
+ else
+ ngx.say("passed")
+ end
+ }
+ }
+--- request
+GET /t
+--- response_body
+passed
+--- no_error_log
Review comment:
Please remove the duplicate sections.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] shuaijinchao commented on a change in pull request #5538: feat(plugin): support google cloud logging service
Posted by GitBox <gi...@apache.org>.
shuaijinchao commented on a change in pull request #5538:
URL: https://github.com/apache/apisix/pull/5538#discussion_r754751439
##########
File path: apisix/plugins/google-cloud-logging.lua
##########
@@ -0,0 +1,326 @@
+--
+-- Licensed to the Apache Software Foundation (ASF) under one or more
+-- contributor license agreements. See the NOTICE file distributed with
+-- this work for additional information regarding copyright ownership.
+-- The ASF licenses this file to You under the Apache License, Version 2.0
+-- (the "License"); you may not use this file except in compliance with
+-- the License. You may obtain a copy of the License at
+--
+-- http://www.apache.org/licenses/LICENSE-2.0
+--
+-- Unless required by applicable law or agreed to in writing, software
+-- distributed under the License is distributed on an "AS IS" BASIS,
+-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+-- See the License for the specific language governing permissions and
+-- limitations under the License.
+--
+
+local core = require("apisix.core")
+local ngx = ngx
+local tostring = tostring
+local ipairs = ipairs
+local os_date = os.date
+local math_floor = math.floor
+local ngx_now = ngx.now
+local ngx_timer_at = ngx.timer.at
+local ngx_update_time = ngx.update_time
+local http = require("resty.http")
+local log_util = require("apisix.utils.log-util")
+local batch_processor = require("apisix.utils.batch-processor")
+local google_oauth = require("apisix.plugins.google-cloud-logging.oauth")
+
+
+local buffers = {}
+local auth_config_cache
+local stale_timer_running
+
+
+local plugin_name = "google-cloud-logging"
+local schema = {
+ type = "object",
+ properties = {
+ auth_config = {
+ type = "object",
+ properties = {
+ private_key = { type = "string" },
+ project_id = { type = "string" },
+ token_uri = {
+ type = "string",
+ default = "https://oauth2.googleapis.com/token"
+ },
+ -- https://developers.google.com/identity/protocols/oauth2/scopes#logging
+ scopes = {
+ type = "array",
+ items = {
+ description = "Google OAuth2 Authorization Scopes",
+ type = "string",
+ },
+ minItems = 1,
+ uniqueItems = true,
+ default = {
+ "https://www.googleapis.com/auth/logging.read",
+ "https://www.googleapis.com/auth/logging.write",
+ "https://www.googleapis.com/auth/logging.admin",
+ "https://www.googleapis.com/auth/cloud-platform"
+ }
+ },
+ entries_uri = {
+ type = "string",
+ default = "https://logging.googleapis.com/v2/entries:write"
+ },
+ ssl_verify = {
+ type = "boolean",
+ default = true
+ },
+ },
+ required = { "private_key", "project_id", "token_uri" }
+ },
+ auth_file = { type = "string" },
+ -- https://cloud.google.com/logging/docs/reference/v2/rest/v2/MonitoredResource
+ resource = {
+ type = "object",
+ properties = {
+ type = { type = "string" },
+ labels = { type = "object" }
+ },
+ default = {
+ type = "global"
+ },
+ required = { "type" }
+ },
+ -- https://cloud.google.com/logging/docs/reference/v2/rest/v2/LogEntry
+ log_id = {
+ type = "string",
+ default = "apisix.apache.org%2Flogs"
+ },
+ max_retry_count = {
+ type = "integer",
+ minimum = 0,
+ default = 0
+ },
+ retry_delay = {
+ type = "integer",
+ minimum = 0,
+ default = 1
+ },
+ buffer_duration = {
+ type = "integer",
+ minimum = 1,
+ default = 60
+ },
+ inactive_timeout = {
+ type = "integer",
+ minimum = 1,
+ default = 10
+ },
+ batch_max_size = {
+ type = "integer",
+ minimum = 1,
+ default = 100
+ },
+ },
+ oneOf = {
+ { required = { "auth_config" } },
+ { required = { "auth_file" } },
+ },
+}
+
+
+-- remove stale objects from the memory after timer expires
+local function remove_stale_objects(premature)
+ if premature then
+ return
+ end
+
+ for key, batch in ipairs(buffers) do
+ if #batch.entry_buffer.entries == 0 and #batch.batch_to_process == 0 then
+ core.log.warn("removing batch processor stale object, route id:", tostring(key))
+ buffers[key] = nil
+ end
+ end
+
+ stale_timer_running = false
+end
+
+
+local function send_to_google(oauth, entries)
+ local http_new = http.new()
+ local access_token = oauth:generate_access_token()
+ if not access_token then
+ return nil, "failed to get google oauth token"
+ end
+
+ local res, err = http_new:request_uri(oauth.entries_uri, {
+ ssl_verify = oauth.ssl_verify,
+ method = "POST",
+ body = core.json.encode({
+ entries = entries,
+ partialSuccess = false,
+ }),
+ headers = {
+ ["Content-Type"] = "application/json",
+ ["Authorization"] = "Bearer " .. access_token,
+ },
+ })
+
+ if err then
+ return nil, "failed to write log to google, " .. err
+ end
+
+ if res.status ~= 200 then
+ return nil, res.body
+ end
+
+ return res.body
+end
+
+
+local function get_auth_config(config)
+ if auth_config_cache then
+ return auth_config_cache
+ end
+
+ if config.auth_config then
+ auth_config_cache = config.auth_config
+ return auth_config_cache
+ end
+
+ if not config.auth_file then
+ return nil, "configuration is not defined"
+ end
+
+ local file_content, err = core.io.get_file(config.auth_file)
+ if not file_content then
+ return nil, "failed to read configuration, file: " .. config.auth_file .. " err:" .. err
+ end
+
+ local config_data
+ config_data, err = core.json.decode(file_content)
+ if not config_data then
+ return nil, "config parse failure, data: " .. file_content .. " , err: " .. err
+ end
+
+ auth_config_cache = config.auth_config
+ return auth_config_cache
+end
+
+
+local function get_logger_buffer(conf, ctx)
+ local oauth_client = google_oauth:new(auth_config_cache)
+
+ local process = function(entries)
+ return send_to_google(oauth_client, entries)
+ end
+
+ local config = {
+ name = conf.name or plugin_name,
+ retry_delay = conf.retry_delay,
+ batch_max_size = conf.batch_max_size,
+ max_retry_count = conf.max_retry_count,
+ buffer_duration = conf.buffer_duration,
+ inactive_timeout = conf.inactive_timeout,
+ route_id = ctx.var.route_id,
+ server_addr = ctx.var.server_addr,
+ }
+
+ local buffer, err = batch_processor:new(process, config)
+
+ if not buffer then
+ return nil, "error when creating the batch processor: " .. err
+ end
+
+ return buffer
+end
+
+
+local function get_utc_timestamp()
+ ngx_update_time()
+ local now = ngx_now()
+ local second = math_floor(now)
+ local millisecond = math_floor((now - second) * 1000)
+ return os_date("!%Y-%m-%dT%T.", second) .. core.string.format("%03dZ", millisecond)
+end
+
+
+local function get_logger_entry(conf, ctx)
+ local auth_config, err = get_auth_config(conf)
+ if err or not auth_config.project_id or not auth_config.private_key then
+ return nil, "failed to get google authentication configuration, " .. err
+ end
+
+ local entry = log_util.get_full_log(ngx, conf)
+ local google_entry = {
+ httpRequest = {
+ requestMethod = entry.request.method,
+ requestUrl = entry.request.url,
+ requestSize = entry.request.size,
+ status = entry.response.status,
+ responseSize = entry.response.size,
+ userAgent = entry.request.headers and entry.request.headers["user-agent"],
+ remoteIp = entry.client_ip,
+ serverIp = entry.upstream,
+ latency = tostring(core.string.format("%0.3f", entry.latency / 1000)) .. "s"
Review comment:
The format is google requirements: https://cloud.google.com/logging/docs/reference/v2/rest/v2/LogEntry
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] shuaijinchao commented on a change in pull request #5538: feat(plugin): support google cloud logging service
Posted by GitBox <gi...@apache.org>.
shuaijinchao commented on a change in pull request #5538:
URL: https://github.com/apache/apisix/pull/5538#discussion_r753059781
##########
File path: apisix/plugins/google-logging/oauth.lua
##########
@@ -0,0 +1,128 @@
+--
+-- Licensed to the Apache Software Foundation (ASF) under one or more
+-- contributor license agreements. See the NOTICE file distributed with
+-- this work for additional information regarding copyright ownership.
+-- The ASF licenses this file to You under the Apache License, Version 2.0
+-- (the "License"); you may not use this file except in compliance with
+-- the License. You may obtain a copy of the License at
+--
+-- http://www.apache.org/licenses/LICENSE-2.0
+--
+-- Unless required by applicable law or agreed to in writing, software
+-- distributed under the License is distributed on an "AS IS" BASIS,
+-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+-- See the License for the specific language governing permissions and
+-- limitations under the License.
+--
+
+local type = type
+local setmetatable = setmetatable
+
+local ngx_update_time = ngx.update_time
+local ngx_time = ngx.time
+local ngx_encode_args = ngx.encode_args
+
+local core = require("apisix.core")
Review comment:
done
##########
File path: docs/en/latest/plugins/google-logging.md
##########
@@ -0,0 +1,156 @@
+---
+title: google-logging
+---
+
+<!--
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+-->
+
+## Summary
+
+- [**Name**](#name)
+- [**Attributes**](#attributes)
+- [**How To Enable**](#how-to-enable)
+- [**Test Plugin**](#test-plugin)
+- [**Disable Plugin**](#disable-plugin)
+
+## Name
+
+`google-logging` plugin is used to send the request log of `Apache APISIX` to the [Google Cloud Logging Service](https://cloud.google.com/logging/).
Review comment:
done
##########
File path: docs/en/latest/plugins/google-logging.md
##########
@@ -0,0 +1,156 @@
+---
+title: google-logging
+---
+
+<!--
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+-->
+
+## Summary
+
+- [**Name**](#name)
+- [**Attributes**](#attributes)
+- [**How To Enable**](#how-to-enable)
+- [**Test Plugin**](#test-plugin)
+- [**Disable Plugin**](#disable-plugin)
+
+## Name
+
+`google-logging` plugin is used to send the request log of `Apache APISIX` to the [Google Cloud Logging Service](https://cloud.google.com/logging/).
+
+This plugin provides the ability to push Log data as a batch to Google Cloud logging Service.
+
+For more info on Batch-Processor in Apache APISIX please refer:
+[Batch-Processor](../batch-processor.md)
+
+## Attributes
+
+| Name | Requirement | Default | Description |
+| ----------------------- | ------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| auth_config | Semi-optional | | one of `auth_config` or `auth_file` must be configured |
+| auth_config.private_key | Required | | the private key parameters of the Google service account |
+| auth_config.project_id | Required | | the project id parameters of the Google service account |
+| auth_config.token_uri | Optional | https://oauth2.googleapis.com/token | the token uri parameters of the Google service account |
+| auth_config.entries_uri | Optional | https://logging.googleapis.com/v2/entries:write | google logging service API |
+| auth_config.scopes | Optional | ["https://www.googleapis.com/auth/logging.read","https://www.googleapis.com/auth/logging.write","https://www.googleapis.com/auth/logging.admin","https://www.googleapis.com/auth/cloud-platform"] | the access scopes parameters of the Google service account, refer to: [OAuth 2.0 Scopes for Google APIs](https://developers.google.com/identity/protocols/oauth2/scopes#logging) |
+| auth_file | Semi-optional | | path to the google service account json file（Semi-optional, one of auth_config or auth_file must be configured） |
+| resource | Optional | {"type": "global"} | the Google monitor resource, refer to: [MonitoredResource](https://cloud.google.com/logging/docs/reference/v2/rest/v2/MonitoredResource) |
+| log_id | Optional | apisix.apache.org%2Flogs | google logging id, refer to: [LogEntry](https://cloud.google.com/logging/docs/reference/v2/rest/v2/LogEntry) |
+| max_retry_count | Optional | 0 | max number of retries before removing from the processing pipe line |
+| retry_delay | Optional | 1 | number of seconds the process execution should be delayed if the execution fails |
+| buffer_duration | Optional | 60 | max age in seconds of the oldest entry in a batch before the batch must be processed |
+| inactive_timeout | Optional | 10 | max age in seconds when the buffer will be flushed if inactive |
+| batch_max_size | Optional | 100 | max size of each batch |
+| | | | |
Review comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] shuaijinchao commented on a change in pull request #5538: feat(plugin): support google cloud logging service
Posted by GitBox <gi...@apache.org>.
shuaijinchao commented on a change in pull request #5538:
URL: https://github.com/apache/apisix/pull/5538#discussion_r754755130
##########
File path: apisix/plugins/google-cloud-logging.lua
##########
@@ -0,0 +1,326 @@
+--
+-- Licensed to the Apache Software Foundation (ASF) under one or more
+-- contributor license agreements. See the NOTICE file distributed with
+-- this work for additional information regarding copyright ownership.
+-- The ASF licenses this file to You under the Apache License, Version 2.0
+-- (the "License"); you may not use this file except in compliance with
+-- the License. You may obtain a copy of the License at
+--
+-- http://www.apache.org/licenses/LICENSE-2.0
+--
+-- Unless required by applicable law or agreed to in writing, software
+-- distributed under the License is distributed on an "AS IS" BASIS,
+-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+-- See the License for the specific language governing permissions and
+-- limitations under the License.
+--
+
+local core = require("apisix.core")
+local ngx = ngx
+local tostring = tostring
+local ipairs = ipairs
+local os_date = os.date
+local math_floor = math.floor
+local ngx_now = ngx.now
+local ngx_timer_at = ngx.timer.at
+local ngx_update_time = ngx.update_time
+local http = require("resty.http")
+local log_util = require("apisix.utils.log-util")
+local batch_processor = require("apisix.utils.batch-processor")
+local google_oauth = require("apisix.plugins.google-cloud-logging.oauth")
+
+
+local buffers = {}
+local auth_config_cache
+local stale_timer_running
+
+
+local plugin_name = "google-cloud-logging"
+local schema = {
+ type = "object",
+ properties = {
+ auth_config = {
+ type = "object",
+ properties = {
+ private_key = { type = "string" },
+ project_id = { type = "string" },
+ token_uri = {
+ type = "string",
+ default = "https://oauth2.googleapis.com/token"
+ },
+ -- https://developers.google.com/identity/protocols/oauth2/scopes#logging
+ scopes = {
+ type = "array",
+ items = {
+ description = "Google OAuth2 Authorization Scopes",
+ type = "string",
+ },
+ minItems = 1,
+ uniqueItems = true,
+ default = {
+ "https://www.googleapis.com/auth/logging.read",
+ "https://www.googleapis.com/auth/logging.write",
+ "https://www.googleapis.com/auth/logging.admin",
+ "https://www.googleapis.com/auth/cloud-platform"
+ }
+ },
+ entries_uri = {
+ type = "string",
+ default = "https://logging.googleapis.com/v2/entries:write"
+ },
+ ssl_verify = {
+ type = "boolean",
+ default = true
+ },
+ },
+ required = { "private_key", "project_id", "token_uri" }
+ },
+ auth_file = { type = "string" },
+ -- https://cloud.google.com/logging/docs/reference/v2/rest/v2/MonitoredResource
+ resource = {
+ type = "object",
+ properties = {
+ type = { type = "string" },
+ labels = { type = "object" }
+ },
+ default = {
+ type = "global"
+ },
+ required = { "type" }
+ },
+ -- https://cloud.google.com/logging/docs/reference/v2/rest/v2/LogEntry
+ log_id = {
+ type = "string",
+ default = "apisix.apache.org%2Flogs"
+ },
+ max_retry_count = {
+ type = "integer",
+ minimum = 0,
+ default = 0
+ },
+ retry_delay = {
+ type = "integer",
+ minimum = 0,
+ default = 1
+ },
+ buffer_duration = {
+ type = "integer",
+ minimum = 1,
+ default = 60
+ },
+ inactive_timeout = {
+ type = "integer",
+ minimum = 1,
+ default = 10
+ },
+ batch_max_size = {
+ type = "integer",
+ minimum = 1,
+ default = 100
+ },
+ },
+ oneOf = {
+ { required = { "auth_config" } },
+ { required = { "auth_file" } },
+ },
+}
+
+
+-- remove stale objects from the memory after timer expires
+local function remove_stale_objects(premature)
+ if premature then
+ return
+ end
+
+ for key, batch in ipairs(buffers) do
+ if #batch.entry_buffer.entries == 0 and #batch.batch_to_process == 0 then
+ core.log.warn("removing batch processor stale object, route id:", tostring(key))
+ buffers[key] = nil
+ end
+ end
+
+ stale_timer_running = false
+end
+
+
+local function send_to_google(oauth, entries)
+ local http_new = http.new()
+ local access_token = oauth:generate_access_token()
+ if not access_token then
+ return nil, "failed to get google oauth token"
+ end
+
+ local res, err = http_new:request_uri(oauth.entries_uri, {
+ ssl_verify = oauth.ssl_verify,
+ method = "POST",
+ body = core.json.encode({
+ entries = entries,
+ partialSuccess = false,
+ }),
+ headers = {
+ ["Content-Type"] = "application/json",
+ ["Authorization"] = "Bearer " .. access_token,
+ },
+ })
+
+ if err then
+ return nil, "failed to write log to google, " .. err
+ end
+
+ if res.status ~= 200 then
+ return nil, res.body
+ end
+
+ return res.body
+end
+
+
+local function get_auth_config(config)
+ if auth_config_cache then
+ return auth_config_cache
+ end
+
+ if config.auth_config then
+ auth_config_cache = config.auth_config
+ return auth_config_cache
+ end
+
+ if not config.auth_file then
+ return nil, "configuration is not defined"
+ end
+
+ local file_content, err = core.io.get_file(config.auth_file)
+ if not file_content then
+ return nil, "failed to read configuration, file: " .. config.auth_file .. " err:" .. err
+ end
+
+ local config_data
+ config_data, err = core.json.decode(file_content)
+ if not config_data then
+ return nil, "config parse failure, data: " .. file_content .. " , err: " .. err
+ end
+
+ auth_config_cache = config.auth_config
+ return auth_config_cache
+end
+
+
+local function get_logger_buffer(conf, ctx)
+ local oauth_client = google_oauth:new(auth_config_cache)
+
+ local process = function(entries)
+ return send_to_google(oauth_client, entries)
+ end
+
+ local config = {
+ name = conf.name or plugin_name,
+ retry_delay = conf.retry_delay,
+ batch_max_size = conf.batch_max_size,
+ max_retry_count = conf.max_retry_count,
+ buffer_duration = conf.buffer_duration,
+ inactive_timeout = conf.inactive_timeout,
+ route_id = ctx.var.route_id,
+ server_addr = ctx.var.server_addr,
+ }
+
+ local buffer, err = batch_processor:new(process, config)
+
+ if not buffer then
+ return nil, "error when creating the batch processor: " .. err
+ end
+
+ return buffer
+end
+
+
+local function get_utc_timestamp()
+ ngx_update_time()
+ local now = ngx_now()
+ local second = math_floor(now)
+ local millisecond = math_floor((now - second) * 1000)
+ return os_date("!%Y-%m-%dT%T.", second) .. core.string.format("%03dZ", millisecond)
+end
Review comment:
What does it mean if it is true? in addition: https://github.com/apache/apisix/blob/master/apisix/plugins/slslog/rfc5424.lua#L83 unable to format milliseconds
```lua
local dd = 1573537134.123
local df = "!%Y-%m-%dT%H:%M:%S.000Z"
print(os.date(df, dd))
--- print 2019-11-12T05:38:54.000Z
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] shuaijinchao commented on a change in pull request #5538: feat(plugin): support google cloud logging service
Posted by GitBox <gi...@apache.org>.
shuaijinchao commented on a change in pull request #5538:
URL: https://github.com/apache/apisix/pull/5538#discussion_r754077165
##########
File path: docs/en/latest/plugins/google-logging.md
##########
@@ -0,0 +1,155 @@
+---
+title: google-logging
+---
+
+<!--
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+-->
+
+## Summary
+
+- [**Name**](#name)
+- [**Attributes**](#attributes)
+- [**How To Enable**](#how-to-enable)
+- [**Test Plugin**](#test-plugin)
+- [**Disable Plugin**](#disable-plugin)
+
+## Name
+
+`google-logging` plugin is used to send the access log of `Apache APISIX` to the [Google Cloud Logging Service](https://cloud.google.com/logging/).
+
+This plugin provides the ability to push Log data as a batch to Google Cloud logging Service.
+
+For more info on Batch-Processor in Apache APISIX please refer:
+[Batch-Processor](../batch-processor.md)
+
+## Attributes
+
+| Name | Requirement | Default | Description |
+| ----------------------- | ------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| auth_config | Semi-optional | | one of `auth_config` or `auth_file` must be configured |
+| auth_config.private_key | Required | | the private key parameters of the Google service account |
+| auth_config.project_id | Required | | the project id parameters of the Google service account |
+| auth_config.token_uri | Optional | https://oauth2.googleapis.com/token | the token uri parameters of the Google service account |
+| auth_config.entries_uri | Optional | https://logging.googleapis.com/v2/entries:write | google logging service API |
+| auth_config.scopes | Optional | ["https://www.googleapis.com/auth/logging.read","https://www.googleapis.com/auth/logging.write","https://www.googleapis.com/auth/logging.admin","https://www.googleapis.com/auth/cloud-platform"] | the access scopes parameters of the Google service account, refer to: [OAuth 2.0 Scopes for Google APIs](https://developers.google.com/identity/protocols/oauth2/scopes#logging) |
+| auth_file | Semi-optional | | path to the google service account json file（Semi-optional, one of auth_config or auth_file must be configured） |
+| resource | Optional | {"type": "global"} | the Google monitor resource, refer to: [MonitoredResource](https://cloud.google.com/logging/docs/reference/v2/rest/v2/MonitoredResource) |
+| log_id | Optional | apisix.apache.org%2Flogs | google logging id, refer to: [LogEntry](https://cloud.google.com/logging/docs/reference/v2/rest/v2/LogEntry) |
+| max_retry_count | Optional | 0 | max number of retries before removing from the processing pipe line |
+| retry_delay | Optional | 1 | number of seconds the process execution should be delayed if the execution fails |
+| buffer_duration | Optional | 60 | max age in seconds of the oldest entry in a batch before the batch must be processed |
+| inactive_timeout | Optional | 10 | max age in seconds when the buffer will be flushed if inactive |
+| batch_max_size | Optional | 100 | max size of each batch |
+
+## How To Enable
+
+The following is an example on how to enable the `google-logging` for a specific route.
+
+### Full configuration
+
+```shell
+curl http://127.0.0.1:9080/apisix/admin/routes/1 -H 'X-API-KEY: edd1c9f034335f136f87ad84b625c8f1' -X PUT -d '
+{
+ "plugins": {
+ "google-logging": {
+ "auth_config":{
+ "project_id":"apisix",
+ "private_key":"-----BEGIN RSA PRIVATE KEY-----KEY-----END RSA PRIVATE KEY-----",
Review comment:
I think it is enough for the user to get the correct prompt in the document. The text of private_key has been updated from `KEY` to `your private key`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] shuaijinchao commented on a change in pull request #5538: feat(plugin): support google cloud logging service
Posted by GitBox <gi...@apache.org>.
shuaijinchao commented on a change in pull request #5538:
URL: https://github.com/apache/apisix/pull/5538#discussion_r754070036
##########
File path: docs/en/latest/plugins/google-logging.md
##########
@@ -0,0 +1,155 @@
+---
+title: google-logging
+---
+
+<!--
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+-->
+
+## Summary
+
+- [**Name**](#name)
+- [**Attributes**](#attributes)
+- [**How To Enable**](#how-to-enable)
+- [**Test Plugin**](#test-plugin)
+- [**Disable Plugin**](#disable-plugin)
+
+## Name
+
+`google-logging` plugin is used to send the access log of `Apache APISIX` to the [Google Cloud Logging Service](https://cloud.google.com/logging/).
+
+This plugin provides the ability to push Log data as a batch to Google Cloud logging Service.
Review comment:
updated
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] tokers commented on a change in pull request #5538: feat(plugin): support google cloud logging service
Posted by GitBox <gi...@apache.org>.
tokers commented on a change in pull request #5538:
URL: https://github.com/apache/apisix/pull/5538#discussion_r753885192
##########
File path: t/lib/server.lua
##########
@@ -426,6 +426,83 @@ function _M._well_known_openid_configuration()
ngx.say(openid_data)
end
+function _M.google_logging_token()
+ ngx.req.read_body()
+ local data = ngx.decode_args(ngx.req.get_body_data())
+ local jwt = require("resty.jwt")
+ local rsa_public_key = "-----BEGIN PUBLIC KEY-----\nMFwwDQYJKoZIhvcNAQEBBQADSwAwSAJBAKebDxlvQMGyEesAL1r1nIJBkSdqu3Hr\n7noq/0ukiZqVQLSJPMOv0oxQSutvvK3hoibwGakDOza+xRITB7cs2cECAwEAAQ==\n-----END PUBLIC KEY-----"
Review comment:
It's not necessary to make them public, but we can put it in the module level (a module-local variable).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] shuaijinchao commented on a change in pull request #5538: feat(plugin): support google cloud logging service
Posted by GitBox <gi...@apache.org>.
shuaijinchao commented on a change in pull request #5538:
URL: https://github.com/apache/apisix/pull/5538#discussion_r754768659
##########
File path: apisix/plugins/google-cloud-logging.lua
##########
@@ -0,0 +1,326 @@
+--
+-- Licensed to the Apache Software Foundation (ASF) under one or more
+-- contributor license agreements. See the NOTICE file distributed with
+-- this work for additional information regarding copyright ownership.
+-- The ASF licenses this file to You under the Apache License, Version 2.0
+-- (the "License"); you may not use this file except in compliance with
+-- the License. You may obtain a copy of the License at
+--
+-- http://www.apache.org/licenses/LICENSE-2.0
+--
+-- Unless required by applicable law or agreed to in writing, software
+-- distributed under the License is distributed on an "AS IS" BASIS,
+-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+-- See the License for the specific language governing permissions and
+-- limitations under the License.
+--
+
+local core = require("apisix.core")
+local ngx = ngx
+local tostring = tostring
+local ipairs = ipairs
+local os_date = os.date
+local math_floor = math.floor
+local ngx_now = ngx.now
+local ngx_timer_at = ngx.timer.at
+local ngx_update_time = ngx.update_time
+local http = require("resty.http")
+local log_util = require("apisix.utils.log-util")
+local batch_processor = require("apisix.utils.batch-processor")
+local google_oauth = require("apisix.plugins.google-cloud-logging.oauth")
+
+
+local buffers = {}
+local auth_config_cache
+local stale_timer_running
+
+
+local plugin_name = "google-cloud-logging"
+local schema = {
+ type = "object",
+ properties = {
+ auth_config = {
+ type = "object",
+ properties = {
+ private_key = { type = "string" },
+ project_id = { type = "string" },
+ token_uri = {
+ type = "string",
+ default = "https://oauth2.googleapis.com/token"
+ },
+ -- https://developers.google.com/identity/protocols/oauth2/scopes#logging
+ scopes = {
+ type = "array",
+ items = {
+ description = "Google OAuth2 Authorization Scopes",
+ type = "string",
+ },
+ minItems = 1,
+ uniqueItems = true,
+ default = {
+ "https://www.googleapis.com/auth/logging.read",
+ "https://www.googleapis.com/auth/logging.write",
+ "https://www.googleapis.com/auth/logging.admin",
+ "https://www.googleapis.com/auth/cloud-platform"
+ }
+ },
+ entries_uri = {
+ type = "string",
+ default = "https://logging.googleapis.com/v2/entries:write"
+ },
+ ssl_verify = {
+ type = "boolean",
+ default = true
+ },
+ },
+ required = { "private_key", "project_id", "token_uri" }
+ },
+ auth_file = { type = "string" },
+ -- https://cloud.google.com/logging/docs/reference/v2/rest/v2/MonitoredResource
+ resource = {
+ type = "object",
+ properties = {
+ type = { type = "string" },
+ labels = { type = "object" }
+ },
+ default = {
+ type = "global"
+ },
+ required = { "type" }
+ },
+ -- https://cloud.google.com/logging/docs/reference/v2/rest/v2/LogEntry
+ log_id = {
+ type = "string",
+ default = "apisix.apache.org%2Flogs"
+ },
+ max_retry_count = {
+ type = "integer",
+ minimum = 0,
+ default = 0
+ },
+ retry_delay = {
+ type = "integer",
+ minimum = 0,
+ default = 1
+ },
+ buffer_duration = {
+ type = "integer",
+ minimum = 1,
+ default = 60
+ },
+ inactive_timeout = {
+ type = "integer",
+ minimum = 1,
+ default = 10
+ },
+ batch_max_size = {
+ type = "integer",
+ minimum = 1,
+ default = 100
+ },
+ },
+ oneOf = {
+ { required = { "auth_config" } },
+ { required = { "auth_file" } },
+ },
+}
+
+
+-- remove stale objects from the memory after timer expires
+local function remove_stale_objects(premature)
+ if premature then
+ return
+ end
+
+ for key, batch in ipairs(buffers) do
+ if #batch.entry_buffer.entries == 0 and #batch.batch_to_process == 0 then
+ core.log.warn("removing batch processor stale object, route id:", tostring(key))
+ buffers[key] = nil
+ end
+ end
+
+ stale_timer_running = false
+end
+
+
+local function send_to_google(oauth, entries)
+ local http_new = http.new()
+ local access_token = oauth:generate_access_token()
+ if not access_token then
+ return nil, "failed to get google oauth token"
+ end
+
+ local res, err = http_new:request_uri(oauth.entries_uri, {
+ ssl_verify = oauth.ssl_verify,
+ method = "POST",
+ body = core.json.encode({
+ entries = entries,
+ partialSuccess = false,
+ }),
+ headers = {
+ ["Content-Type"] = "application/json",
+ ["Authorization"] = "Bearer " .. access_token,
+ },
+ })
+
+ if err then
+ return nil, "failed to write log to google, " .. err
+ end
+
+ if res.status ~= 200 then
+ return nil, res.body
+ end
+
+ return res.body
+end
+
+
+local function get_auth_config(config)
+ if auth_config_cache then
+ return auth_config_cache
+ end
+
+ if config.auth_config then
+ auth_config_cache = config.auth_config
+ return auth_config_cache
+ end
+
+ if not config.auth_file then
+ return nil, "configuration is not defined"
+ end
+
+ local file_content, err = core.io.get_file(config.auth_file)
+ if not file_content then
+ return nil, "failed to read configuration, file: " .. config.auth_file .. " err:" .. err
+ end
+
+ local config_data
+ config_data, err = core.json.decode(file_content)
+ if not config_data then
+ return nil, "config parse failure, data: " .. file_content .. " , err: " .. err
+ end
+
+ auth_config_cache = config.auth_config
+ return auth_config_cache
+end
+
+
+local function get_logger_buffer(conf, ctx)
+ local oauth_client = google_oauth:new(auth_config_cache)
+
+ local process = function(entries)
+ return send_to_google(oauth_client, entries)
+ end
+
+ local config = {
+ name = conf.name or plugin_name,
+ retry_delay = conf.retry_delay,
+ batch_max_size = conf.batch_max_size,
+ max_retry_count = conf.max_retry_count,
+ buffer_duration = conf.buffer_duration,
+ inactive_timeout = conf.inactive_timeout,
+ route_id = ctx.var.route_id,
+ server_addr = ctx.var.server_addr,
+ }
+
+ local buffer, err = batch_processor:new(process, config)
+
+ if not buffer then
+ return nil, "error when creating the batch processor: " .. err
+ end
+
+ return buffer
+end
+
+
+local function get_utc_timestamp()
+ ngx_update_time()
+ local now = ngx_now()
+ local second = math_floor(now)
+ local millisecond = math_floor((now - second) * 1000)
+ return os_date("!%Y-%m-%dT%T.", second) .. core.string.format("%03dZ", millisecond)
+end
Review comment:
But it’s a good idea to move to common utils
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] shuaijinchao commented on a change in pull request #5538: feat(plugin): support google cloud logging service
Posted by GitBox <gi...@apache.org>.
shuaijinchao commented on a change in pull request #5538:
URL: https://github.com/apache/apisix/pull/5538#discussion_r754770407
##########
File path: docs/en/latest/plugins/google-cloud-logging.md
##########
@@ -0,0 +1,156 @@
+---
+title: google-cloud-logging
+---
+
+<!--
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+-->
+
+## Summary
+
+- [**Name**](#name)
+- [**Attributes**](#attributes)
+- [**How To Enable**](#how-to-enable)
+- [**Test Plugin**](#test-plugin)
+- [**Disable Plugin**](#disable-plugin)
+
+## Name
+
+`google-cloud-logging` plugin is used to send the access log of `Apache APISIX` to the [Google Cloud Logging Service](https://cloud.google.com/logging/).
+
+This plugin provides the ability to push log data as a batch to Google Cloud logging Service.
+
+For more info on Batch-Processor in Apache APISIX please refer:
+[Batch-Processor](../batch-processor.md)
+
+## Attributes
+
+| Name | Requirement | Default | Description |
+| ----------------------- | ------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| auth_config | Semi-optional | | one of `auth_config` or `auth_file` must be configured |
+| auth_config.private_key | Required | | the private key parameters of the Google service account |
+| auth_config.project_id | Required | | the project id parameters of the Google service account |
+| auth_config.token_uri | Optional | https://oauth2.googleapis.com/token | the token uri parameters of the Google service account |
Review comment:
updated
##########
File path: docs/en/latest/plugins/google-cloud-logging.md
##########
@@ -0,0 +1,156 @@
+---
+title: google-cloud-logging
+---
+
+<!--
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+-->
+
+## Summary
+
+- [**Name**](#name)
+- [**Attributes**](#attributes)
+- [**How To Enable**](#how-to-enable)
+- [**Test Plugin**](#test-plugin)
+- [**Disable Plugin**](#disable-plugin)
+
+## Name
+
+`google-cloud-logging` plugin is used to send the access log of `Apache APISIX` to the [Google Cloud Logging Service](https://cloud.google.com/logging/).
+
+This plugin provides the ability to push log data as a batch to Google Cloud logging Service.
+
+For more info on Batch-Processor in Apache APISIX please refer:
+[Batch-Processor](../batch-processor.md)
+
+## Attributes
+
+| Name | Requirement | Default | Description |
+| ----------------------- | ------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| auth_config | Semi-optional | | one of `auth_config` or `auth_file` must be configured |
+| auth_config.private_key | Required | | the private key parameters of the Google service account |
Review comment:
updated
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] shuaijinchao commented on a change in pull request #5538: feat(plugin): support google cloud logging service
Posted by GitBox <gi...@apache.org>.
shuaijinchao commented on a change in pull request #5538:
URL: https://github.com/apache/apisix/pull/5538#discussion_r753890294
##########
File path: t/lib/server.lua
##########
@@ -426,6 +426,83 @@ function _M._well_known_openid_configuration()
ngx.say(openid_data)
end
+function _M.google_logging_token()
+ ngx.req.read_body()
+ local data = ngx.decode_args(ngx.req.get_body_data())
+ local jwt = require("resty.jwt")
+ local rsa_public_key = "-----BEGIN PUBLIC KEY-----\nMFwwDQYJKoZIhvcNAQEBBQADSwAwSAJBAKebDxlvQMGyEesAL1r1nIJBkSdqu3Hr\n7noq/0ukiZqVQLSJPMOv0oxQSutvvK3hoibwGakDOza+xRITB7cs2cECAwEAAQ==\n-----END PUBLIC KEY-----"
Review comment:
OK
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] juzhiyuan commented on a change in pull request #5538: feat(plugin): support google cloud logging service
Posted by GitBox <gi...@apache.org>.
juzhiyuan commented on a change in pull request #5538:
URL: https://github.com/apache/apisix/pull/5538#discussion_r753957415
##########
File path: docs/en/latest/plugins/google-logging.md
##########
@@ -0,0 +1,155 @@
+---
+title: google-logging
Review comment:
Not sure if `google-cloud-logging-service` would be better 😄
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] shuaijinchao commented on a change in pull request #5538: feat(plugin): support google cloud logging service
Posted by GitBox <gi...@apache.org>.
shuaijinchao commented on a change in pull request #5538:
URL: https://github.com/apache/apisix/pull/5538#discussion_r754069452
##########
File path: apisix/plugins/google-logging.lua
##########
@@ -0,0 +1,298 @@
+--
+-- Licensed to the Apache Software Foundation (ASF) under one or more
+-- contributor license agreements. See the NOTICE file distributed with
+-- this work for additional information regarding copyright ownership.
+-- The ASF licenses this file to You under the Apache License, Version 2.0
+-- (the "License"); you may not use this file except in compliance with
+-- the License. You may obtain a copy of the License at
+--
+-- http://www.apache.org/licenses/LICENSE-2.0
+--
+-- Unless required by applicable law or agreed to in writing, software
+-- distributed under the License is distributed on an "AS IS" BASIS,
+-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+-- See the License for the specific language governing permissions and
+-- limitations under the License.
+--
+
+local core = require("apisix.core")
+local ngx = ngx
+local tostring = tostring
+local ipairs = ipairs
+local sub_str = string.sub
+local os_date = os.date
+
+local ngx_now = ngx.now
+local ngx_timer_at = ngx.timer.at
+local ngx_update_time = ngx.update_time
+
+local http = require("resty.http")
+local log_util = require("apisix.utils.log-util")
+local batch_processor = require("apisix.utils.batch-processor")
+local google_oauth = require("apisix.plugins.google-logging.oauth")
+
+
+local buffers = {}
+local auth_config_cache
+local stale_timer_running
+
+
+local plugin_name = "google-logging"
+local schema = {
+ type = "object",
+ properties = {
+ auth_config = {
+ type = "object",
+ properties = {
+ private_key = { type = "string" },
+ project_id = { type = "string" },
+ token_uri = {
+ type = "string",
+ default = "https://oauth2.googleapis.com/token"
+ },
+ -- https://developers.google.com/identity/protocols/oauth2/scopes#logging
+ scopes = {
+ type = "array",
+ items = {
+ description = "Google OAuth2 Authorization Scopes",
+ type = "string",
+ },
+ minItems = 1,
+ uniqueItems = true,
+ default = {
+ "https://www.googleapis.com/auth/logging.read",
+ "https://www.googleapis.com/auth/logging.write",
+ "https://www.googleapis.com/auth/logging.admin",
+ "https://www.googleapis.com/auth/cloud-platform"
+ }
+ },
+ entries_uri = {
+ type = "string",
+ default = "https://logging.googleapis.com/v2/entries:write"
+ },
+ },
+ required = { "private_key", "project_id", "token_uri" }
+ },
+ auth_file = { type = "string" },
+ -- https://cloud.google.com/logging/docs/reference/v2/rest/v2/MonitoredResource
+ resource = {
+ type = "object",
+ properties = {
+ type = { type = "string" },
+ labels = { type = "object" }
+ },
+ default = {
+ type = "global"
+ },
+ required = { "type" }
+ },
+ -- https://cloud.google.com/logging/docs/reference/v2/rest/v2/LogEntry
+ log_id = { type = "string", default = "apisix.apache.org%2Flogs" },
+ max_retry_count = { type = "integer", minimum = 0, default = 0 },
+ retry_delay = { type = "integer", minimum = 0, default = 1 },
+ buffer_duration = { type = "integer", minimum = 1, default = 60 },
+ inactive_timeout = { type = "integer", minimum = 1, default = 10 },
+ batch_max_size = { type = "integer", minimum = 1, default = 100 },
+ },
+ oneOf = {
+ { required = { "auth_config" } },
+ { required = { "auth_file" } },
+ },
+}
+
+
+-- remove stale objects from the memory after timer expires
+local function remove_stale_objects(premature)
+ if premature then
+ return
+ end
+
+ for key, batch in ipairs(buffers) do
+ if #batch.entry_buffer.entries == 0 and #batch.batch_to_process == 0 then
+ core.log.warn("removing batch processor stale object, route id:", tostring(key))
+ buffers[key] = nil
+ end
+ end
+
+ stale_timer_running = false
+end
+
+
+local function send_to_google(oauth, entries)
+ local http_new = http.new()
+ local res, err = http_new:request_uri(oauth.entries_uri, {
+ ssl_verify = false,
+ method = "POST",
+ body = core.json.encode({
+ entries = entries,
+ partialSuccess = false,
+ }),
+ headers = {
+ ["Content-Type"] = "application/json",
+ ["Authorization"] = "Bearer " .. oauth:get_access_token(),
+ },
+ })
+
+ if err then
+ return nil, "failed to write log to google, ", err
+ end
+
+ if res.status ~= 200 then
+ return nil, res.body
+ end
+
+ return res.body
+end
+
+
+local function get_auth_config(config)
+ local err
+ local auth_config = {}
+ if config.auth_config then
+ auth_config = config.auth_config
+ end
+
+ if config.auth_file then
+ local file_content
+ file_content, err = core.utils.get_file(config.auth_file)
+ if file_content then
+ auth_config = core.json.decode(file_content)
+ end
+ end
+
+ if err then
+ return nil, err
+ end
+
+ return auth_config
+end
+
+
+local function get_logger_buffer(conf, ctx)
+ local oauth_client = google_oauth:new(auth_config_cache)
+
+ local process = function(entries)
+ return send_to_google(oauth_client, entries)
+ end
+
+ local config = {
+ name = conf.name or plugin_name,
+ retry_delay = conf.retry_delay,
+ batch_max_size = conf.batch_max_size,
+ max_retry_count = conf.max_retry_count,
+ buffer_duration = conf.buffer_duration,
+ inactive_timeout = conf.inactive_timeout,
+ route_id = ctx.var.route_id,
+ server_addr = ctx.var.server_addr,
+ }
+
+ local buffer, err = batch_processor:new(process, config)
+
+ if not buffer then
+ return nil, "error when creating the batch processor: " .. err
+ end
+
+ return buffer
+end
+
+
+local function get_utc_timestamp()
+ ngx_update_time()
+ local now = tostring(ngx_now())
+ local pos = core.string.rfind_char(now, ".", #now - 1)
+ local second = now
+ local millisecond = 0
+ if pos then
+ second = sub_str(now, 1, pos - 1)
+ millisecond = sub_str(now, pos + 1)
+ end
+ return os_date("!%Y-%m-%dT%T.", second) .. core.string.format("%03dZ", millisecond)
+end
+
+
+local function get_logger_entry(conf)
+ if not auth_config_cache then
+ local auth_config, err = get_auth_config(conf)
+ if err or not auth_config.project_id or not auth_config.private_key then
+ return nil, "failed to get google authentication configuration" .. err
+ end
+
+ auth_config_cache = auth_config
+ end
+
+ local entry = log_util.get_full_log(ngx, conf)
+ local google_entry = {
+ httpRequest = {
+ requestMethod = entry.request.method,
+ requestUrl = entry.request.url,
+ requestSize = entry.request.size,
+ status = entry.response.status,
+ responseSize = entry.response.size,
+ userAgent = entry.request.headers and entry.request.headers["user-agent"],
+ remoteIp = entry.client_ip,
+ serverIp = entry.upstream,
+ latency = tostring(core.string.format("%0.3f", entry.latency / 1000)) .. "s"
+ },
+ jsonPayload = {
+ route_id = entry.route_id,
+ service_id = entry.service_id,
+ },
+ labels = {
+ source = "apache-apisix-google-logging"
+ },
+ timestamp = get_utc_timestamp(),
+ resource = conf.resource,
+ insertId = entry.request.id,
Review comment:
yes, updated
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] shuaijinchao commented on a change in pull request #5538: feat(plugin): support google cloud logging service
Posted by GitBox <gi...@apache.org>.
shuaijinchao commented on a change in pull request #5538:
URL: https://github.com/apache/apisix/pull/5538#discussion_r754067678
##########
File path: apisix/plugins/google-logging.lua
##########
@@ -0,0 +1,298 @@
+--
+-- Licensed to the Apache Software Foundation (ASF) under one or more
+-- contributor license agreements. See the NOTICE file distributed with
+-- this work for additional information regarding copyright ownership.
+-- The ASF licenses this file to You under the Apache License, Version 2.0
+-- (the "License"); you may not use this file except in compliance with
+-- the License. You may obtain a copy of the License at
+--
+-- http://www.apache.org/licenses/LICENSE-2.0
+--
+-- Unless required by applicable law or agreed to in writing, software
+-- distributed under the License is distributed on an "AS IS" BASIS,
+-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+-- See the License for the specific language governing permissions and
+-- limitations under the License.
+--
+
+local core = require("apisix.core")
+local ngx = ngx
+local tostring = tostring
+local ipairs = ipairs
+local sub_str = string.sub
+local os_date = os.date
+
+local ngx_now = ngx.now
+local ngx_timer_at = ngx.timer.at
+local ngx_update_time = ngx.update_time
+
+local http = require("resty.http")
+local log_util = require("apisix.utils.log-util")
+local batch_processor = require("apisix.utils.batch-processor")
+local google_oauth = require("apisix.plugins.google-logging.oauth")
+
+
+local buffers = {}
+local auth_config_cache
+local stale_timer_running
+
+
+local plugin_name = "google-logging"
+local schema = {
+ type = "object",
+ properties = {
+ auth_config = {
+ type = "object",
+ properties = {
+ private_key = { type = "string" },
+ project_id = { type = "string" },
+ token_uri = {
+ type = "string",
+ default = "https://oauth2.googleapis.com/token"
+ },
+ -- https://developers.google.com/identity/protocols/oauth2/scopes#logging
+ scopes = {
+ type = "array",
+ items = {
+ description = "Google OAuth2 Authorization Scopes",
+ type = "string",
+ },
+ minItems = 1,
+ uniqueItems = true,
+ default = {
+ "https://www.googleapis.com/auth/logging.read",
+ "https://www.googleapis.com/auth/logging.write",
+ "https://www.googleapis.com/auth/logging.admin",
+ "https://www.googleapis.com/auth/cloud-platform"
+ }
+ },
+ entries_uri = {
+ type = "string",
+ default = "https://logging.googleapis.com/v2/entries:write"
+ },
+ },
+ required = { "private_key", "project_id", "token_uri" }
+ },
+ auth_file = { type = "string" },
+ -- https://cloud.google.com/logging/docs/reference/v2/rest/v2/MonitoredResource
+ resource = {
+ type = "object",
+ properties = {
+ type = { type = "string" },
+ labels = { type = "object" }
+ },
+ default = {
+ type = "global"
+ },
+ required = { "type" }
+ },
+ -- https://cloud.google.com/logging/docs/reference/v2/rest/v2/LogEntry
+ log_id = { type = "string", default = "apisix.apache.org%2Flogs" },
+ max_retry_count = { type = "integer", minimum = 0, default = 0 },
+ retry_delay = { type = "integer", minimum = 0, default = 1 },
+ buffer_duration = { type = "integer", minimum = 1, default = 60 },
+ inactive_timeout = { type = "integer", minimum = 1, default = 10 },
+ batch_max_size = { type = "integer", minimum = 1, default = 100 },
+ },
+ oneOf = {
+ { required = { "auth_config" } },
+ { required = { "auth_file" } },
+ },
+}
+
+
+-- remove stale objects from the memory after timer expires
+local function remove_stale_objects(premature)
+ if premature then
+ return
+ end
+
+ for key, batch in ipairs(buffers) do
+ if #batch.entry_buffer.entries == 0 and #batch.batch_to_process == 0 then
+ core.log.warn("removing batch processor stale object, route id:", tostring(key))
+ buffers[key] = nil
+ end
+ end
+
+ stale_timer_running = false
+end
+
+
+local function send_to_google(oauth, entries)
+ local http_new = http.new()
+ local res, err = http_new:request_uri(oauth.entries_uri, {
+ ssl_verify = false,
+ method = "POST",
+ body = core.json.encode({
+ entries = entries,
+ partialSuccess = false,
+ }),
+ headers = {
+ ["Content-Type"] = "application/json",
+ ["Authorization"] = "Bearer " .. oauth:get_access_token(),
+ },
+ })
+
+ if err then
+ return nil, "failed to write log to google, ", err
+ end
+
+ if res.status ~= 200 then
+ return nil, res.body
+ end
+
+ return res.body
+end
+
+
+local function get_auth_config(config)
+ local err
+ local auth_config = {}
+ if config.auth_config then
+ auth_config = config.auth_config
+ end
+
+ if config.auth_file then
+ local file_content
+ file_content, err = core.utils.get_file(config.auth_file)
+ if file_content then
+ auth_config = core.json.decode(file_content)
Review comment:
updated
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] shuaijinchao commented on a change in pull request #5538: feat(plugin): support google cloud logging service
Posted by GitBox <gi...@apache.org>.
shuaijinchao commented on a change in pull request #5538:
URL: https://github.com/apache/apisix/pull/5538#discussion_r754071275
##########
File path: t/plugin/google-logging.t
##########
@@ -0,0 +1,380 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+use t::APISIX 'no_plan';
+
+repeat_each(1);
+no_long_string();
+no_root_location();
+run_tests;
+
+
+add_block_preprocessor(sub {
+ my ($block) = @_;
+
+ if ((!defined $block->error_log) && (!defined $block->no_error_log)) {
+ $block->set_value("no_error_log", "[error]");
+ }
Review comment:
updated
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org