You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@apisix.apache.org by GitBox <gi...@apache.org> on 2022/11/25 09:53:44 UTC
[GitHub] [apisix] monkeyDluffy6017 opened a new issue, #8407: Proposal: APISIX Supports Data Encryption
monkeyDluffy6017 opened a new issue, #8407:
URL: https://github.com/apache/apisix/issues/8407
### Description
# Background
Nowadays, if a plugin needs to store sensitive information, it will design a separate set of encryption mechanism, which causes the problem of repeated development and cumbersome management. Now we consider introducing Data Encryption function, hereinafter referred to as DE, to encrypt the specified information, and any module/plugin that needs to encrypt sensitive information only needs to specify the fields to be encrypted on display, so that transparent encryption/decryption can be performed when saving/reading the information.
# Benefits
Unify the encryption mechanism of APISIX to avoid duplication of development while improving product competitiveness
# Goals
1. Turn on the DE function in the configuration file
apisix:
enable_global_data_encryption: true
2. set encrypted fields
local schema = {
type = "object",
properties = {
username = { type = "string" },
password = { type = "string", encrypted = true }, <==== Here
},
required = {"username", "password"},
}
3. Set the plugin
curl http://127.0.0.1:9180/apisix/admin/consumers -H 'X-API-KEY: edd1c9f034335f136f87ad84b625c8f1' -X PUT -d '
{
"username": "foo",
"plugins": {
"basic-auth": {
"username": "foo",
"password": "bar" <= Sensitive Information
}
}
}'
4. The plugin data read directly from ETCD is encrypted
etcdctl get /apisix/consumers/foo
{
"username":"foo",
"update_time":1668584767,
"create_time":1668584767,
"plugins":{
"basic-auth":{
"username":"foo",
"password":"da61c0083b6d19ef3db2490d0da96a71572da0fa" <= encrypted
}
}
}
5. The data surface is read normally
curl http://127.0.0.1:9180/apisix/admin/consumers -H 'X-API-KEY: edd1c9f034335f136f87ad84b625c8f1'
{
"list": [
{
"createdIndex": 8627,
"modifiedIndex": 8627,
"value": {
"username": "foo",
"update_time": 1668584767,
"create_time": 1668584767,
"plugins": {
"basic-auth": {
"username": "foo",
"password": "bar" <= Unencrypted
}
}
},
"key": "/apisix/consumers/foo"
}
],
"total": 1
}
# Detailed Design
1. Add a field in the configuration file to enable data encryption
apisix:
data_encryption: # use `encrypted = {fields}` in plugin schema to enable encryption
enable: false # If not set, the default value is `false`.
keyring:
- edd1c9f0985e76a2 # If not set, will save origin value into etcd.
- qeddd145sfvddff3 # If set this, the keyring should be an array whose elements are string, and the size is also 16, and it will encrypt fields with AES-128-CBC
# !!! So do not change it after encryption, it can't decrypt the fields have be saved if you change !!
# Only use the first key to encrypt, and decrypt in the order of the array.
2. Specify the fields to be encrypted in the schema of the plugin
local consumer_schema = {
type = "object",
title = "work with consumer object",
properties = {
username = { type = "string" },
password = { type = "string", encrypted = true }, <===== here
},
required = {"username", "password"},
}
local schema = {
}
3. If data encryption is enabled, the corresponding fields of the following plug-ins will be encrypted
jwt-auth：secret，private_key
basic-auth：password
authz-keycloak：client_secret
authz-casdoor：client_secret
openid-connect：client_secret
hmac-auth: secret_key
csrf：key
rocketmq-logger：secret_key
clickhouse-logger：password
error-log-logger：clickhouse.password
sls-logger：access_key_secret
google-cloud-logging：auth_config.private_key
elasticsearch-logger：auth.password
tencent-cloud-cls：secret_key
kafka-proxy：sasl.password
4. Encryption and Decryption
- Commonly used algorithms AES, ChaCha20, we use AES128CBC encryption algorithm, the logic is the same as ssl encryption and decryption, that is, only use the first key to encrypt, if it fails to take turns to use all keys to decrypt, consider reusing the ssl encryption and decryption function （https://github.com/apache/apisix/blob/master/apisix/ssl.lua）
- Currently, only those distributed through the apisix admin interface can be encrypted. Many users have developed their own admin and directly manipulated storage media such as etcd, so they need to implement the encryption function themselves.
- After the encryption function is enabled, the data that was not encrypted needs to be sent down again through the interface in order to be encrypted, is a refresh interface provided? Consider not providing it for now
- Combining the above, the original string is used directly after decryption failure
- Admin now only uses etcd, considering that the first version only supports etcd and only supports encryption of plugin parameters
- Encryption：
Add encryption and decryption to check_single_plugin_schema, note that there are 2 different types of schema within the plugin, namely: schema and consumer_schema
local function check_single_plugin_schema(name, plugin_conf, schema_type, skip_disabled_plugin, encrypt_or_decrypt)
...
if plugin_obj.check_schema then
...
end
if schema_type == core.schema.TYPE_CONSUMER then
ok, err = encrypt(plugin_conf, plugin_obj.consumer_schema, encrypt_or_decrypt)
else
ok, err = encrypt(plugin_conf, plugin_obj.schema, encrypt_or_decrypt)
end
return true
end
The encryption and decryption function iterates through the corresponding schema, and if it finds the encrypted = true field, it encrypts and decrypts the field in the corresponding configuration
local function encrypt(plugin_conf, schema, encrypt_or_decrypt)
if schema.type == "object" then
for name, obj in pairs(schema.properties) do
encrypt(plugin_conf[name], obj, encrypt_or_decrypt)
end
elseif schema.type = "array" then
...
end
if schema.encrypted then
aes128(plugin_conf)
end
end
Add corresponding options in check_schema and stream_check_schema, now directly called in admin, no need to modify, encrypt_or_decrypt is nil by default, i.e. encrypt
local function check_schema(plugins_conf, schema_type, skip_disabled_plugin, encrypt_or_decrypt)
for name, plugin_conf in pairs(plugins_conf) do
local ok, err = check_single_plugin_schema(name, plugin_conf,
schema_type, skip_disabled_plugin, encrypt_or_decrypt)
if not ok then
return false, err
end
end
return true
end
plugin_checker and stream_plugin_checker are used for checksumming when reading data from etcd, and are decrypted by default
function _M.plugin_checker(item, schema_type)
if item.plugins then
return check_schema(item.plugins, schema_type, true, decrypt) <=== decrypt
end
return true
end
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] tzssangglass commented on issue #8407: Proposal: APISIX Supports Global Data Encryption
Posted by GitBox <gi...@apache.org>.
tzssangglass commented on issue #8407:
URL: https://github.com/apache/apisix/issues/8407#issuecomment-1357084358
An additional question: in the admin API, do we need to return encrypted data or decrypted data? Or provide the configuration for the user to choose?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] monkeyDluffy6017 commented on issue #8407: Proposal: APISIX Supports Data Encryption
Posted by GitBox <gi...@apache.org>.
monkeyDluffy6017 commented on issue #8407:
URL: https://github.com/apache/apisix/issues/8407#issuecomment-1328064532
1. Done
2. Could we do this later?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] kwanhur commented on issue #8407: Proposal: APISIX Supports Global Data Encryption
Posted by GitBox <gi...@apache.org>.
kwanhur commented on issue #8407:
URL: https://github.com/apache/apisix/issues/8407#issuecomment-1329383739
what about the compatibility between apisix supported/unsupported DE and the data in etcd with/without DE?
| | apisix unsupported DE | apisix support DE |
| ---------------- | ---------------------- | ------------------ |
| data without DE | :thumbsup: | :question: |
| data with DE | :thumbsdown: | :thumbsup: |
we need to take more care when upgrading/downgrading apisix's version.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] spacewander closed issue #8407: Proposal: APISIX Supports Global Data Encryption
Posted by GitBox <gi...@apache.org>.
spacewander closed issue #8407: Proposal: APISIX Supports Global Data Encryption
URL: https://github.com/apache/apisix/issues/8407
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] tokers commented on issue #8407: Proposal: APISIX Supports Data Encryption
Posted by GitBox <gi...@apache.org>.
tokers commented on issue #8407:
URL: https://github.com/apache/apisix/issues/8407#issuecomment-1327259954
1. Could you reformat the proposal? It takes effort to read it.
2. Could we try to avoid adding new items to the **static** APISIX configuration file? It's not good for the management, instead of, trying to combine wit the KMS feature.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] tokers commented on issue #8407: Proposal: APISIX Supports Data Encryption
Posted by GitBox <gi...@apache.org>.
tokers commented on issue #8407:
URL: https://github.com/apache/apisix/issues/8407#issuecomment-1328220579
> Could we do this later?
Yes we can, as long as the compatibility can be sure.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] kwanhur commented on issue #8407: Proposal: APISIX Supports Global Data Encryption
Posted by GitBox <gi...@apache.org>.
kwanhur commented on issue #8407:
URL: https://github.com/apache/apisix/issues/8407#issuecomment-1329427637
similar issue #8319
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [apisix] monkeyDluffy6017 commented on issue #8407: Proposal: APISIX Supports Global Data Encryption
Posted by GitBox <gi...@apache.org>.
monkeyDluffy6017 commented on issue #8407:
URL: https://github.com/apache/apisix/issues/8407#issuecomment-1329938360
@kwanhur I have noticed that
```
Combining the above, the original string is used directly after decryption failure
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@apisix.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org