You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@camel.apache.org by ac...@apache.org on 2022/11/17 06:56:26 UTC
[camel-kamelets-examples] branch db-example-secret-refresh created (now 904d746)
This is an automated email from the ASF dual-hosted git repository.
acosentino pushed a change to branch db-example-secret-refresh
in repository https://gitbox.apache.org/repos/asf/camel-kamelets-examples.git
at 904d746 Added an example of secret refresh with a PostgreSQL database
This branch includes the following new commits:
new 904d746 Added an example of secret refresh with a PostgreSQL database
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
[camel-kamelets-examples] 01/01: Added an example of secret refresh with a PostgreSQL database
Posted by ac...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
acosentino pushed a commit to branch db-example-secret-refresh
in repository https://gitbox.apache.org/repos/asf/camel-kamelets-examples.git
commit 904d7465296a93196a71a8cdb0963f6e7a8387ef
Author: Andrea Cosentino <an...@gmail.com>
AuthorDate: Thu Nov 17 07:55:52 2022 +0100
Added an example of secret refresh with a PostgreSQL database
Signed-off-by: Andrea Cosentino <an...@gmail.com>
---
.../aws-database-admin-secrets-refresh/README.adoc | 218 +++++++++++++++++++++
.../aws-sec-integration.properties | 6 +
.../database-cred-updated.json | 4 +
.../database-cred.json | 4 +
.../populate.sql | 7 +
.../sql-query.yaml | 35 ++++
jbang/aws-database-admin-secrets-refresh/table.sql | 1 +
7 files changed, 275 insertions(+)
diff --git a/jbang/aws-database-admin-secrets-refresh/README.adoc b/jbang/aws-database-admin-secrets-refresh/README.adoc
new file mode 100644
index 0000000..f96f8e3
--- /dev/null
+++ b/jbang/aws-database-admin-secrets-refresh/README.adoc
@@ -0,0 +1,218 @@
+== AWS Secrets Manager Vault Example with Database password
+
+In this sample you'll use the AWS Secrets Manager Vault Properties Source and refresh
+
+=== Install JBang
+
+First install JBang according to https://www.jbang.dev
+
+When JBang is installed then you should be able to run from a shell:
+
+[source,sh]
+----
+$ jbang --version
+----
+
+This will output the version of JBang.
+
+To run this example you can either install Camel on JBang via:
+
+[source,sh]
+----
+$ jbang app install camel@apache/camel
+----
+
+Which allows to run CamelJBang with `camel` as shown below.
+
+=== Setup the AWS Secret Manager service
+
+Create a secret on AWS
+
+[source,sh]
+----
+aws secretsmanager create-secret --name psql --description "Create a secret" --region eu-west-1 --secret-string file://database-cred.json
+----
+
+=== Setting up the AWS credentials
+
+This example uses the ProfileCredentialsProvider from AWS SDK v2. So you'll need to have a configuration file, locally to your machine.
+
+In particular you'll need to have a file placed in `~/.aws/credentials`
+
+with a content like the following
+
+[source,sh]
+----
+[default]
+aws_access_key_id = accessKey
+aws_secret_access_key = secretKey
+----
+
+=== Setup and populate the Postgresql Database
+
+We create a PostgreSQL instance in a docker container
+
+[source,sh]
+----
+docker run -d --name psql -e POSTGRES_PASSWORD=psql123 -e PGDATA=/var/lib/postgresql/data/pgdata -v /custom/mount:/var/lib/postgresql/data postgres
+----
+
+Then we populate it
+
+[source,sh]
+----
+docker exec -i psql psql -U postgres < table.sql
+docker exec -i psql psql -U postgres < populate.sql
+----
+
+=== How to run
+
+Then you can run this example using:
+
+[source,sh]
+----
+$ camel run --properties=aws-sec-integration.properties sql-query.yaml
+----
+
+Or run it even shorter:
+
+[source,sh]
+----
+$ camel run *
+----
+
+Or run with JBang using the longer command line (without installing camel as app in JBang):
+
+[source,sh]
+----
+$ jbang camel@apache/camel run --properties=aws-sec-integration.properties sql-query.yaml
+----
+
+The application will run and consume immediately, then it will wait 120 seconds to query the database again.
+
+[source,sh]
+----
+2022-11-17 07:46:04.515 INFO 10684 --- [ main] org.apache.camel.main.MainSupport : Apache Camel (JBang) 3.19.0 is starting
+2022-11-17 07:46:04.580 INFO 10684 --- [ main] org.apache.camel.main.MainSupport : Using Java 11.0.16.1 with PID 10684. Started by oscerd in /home/oscerd/workspace/apache-camel/camel-kamelets-examples/jbang/aws-database-admin-secrets-refresh
+2022-11-17 07:46:04.592 INFO 10684 --- [ main] he.camel.cli.connector.LocalCliConnector : Camel CLI enabled (local)
+2022-11-17 07:46:05.704 INFO 10684 --- [ main] org.apache.camel.main.BaseMainSupport : Auto-configuration summary
+2022-11-17 07:46:05.704 INFO 10684 --- [ main] org.apache.camel.main.BaseMainSupport : [s-sec-integration.properties] camel.main.name=AWSExample
+2022-11-17 07:46:05.704 INFO 10684 --- [ main] org.apache.camel.main.BaseMainSupport : [s-sec-integration.properties] camel.vault.aws.defaultCredentialsProvider=true
+2022-11-17 07:46:05.704 INFO 10684 --- [ main] org.apache.camel.main.BaseMainSupport : [s-sec-integration.properties] camel.vault.aws.region=eu-west-1
+2022-11-17 07:46:05.704 INFO 10684 --- [ main] org.apache.camel.main.BaseMainSupport : [s-sec-integration.properties] camel.vault.aws.refreshEnabled=true
+2022-11-17 07:46:05.704 INFO 10684 --- [ main] org.apache.camel.main.BaseMainSupport : [s-sec-integration.properties] camel.vault.aws.refreshPeriod=1000
+2022-11-17 07:46:05.705 INFO 10684 --- [ main] org.apache.camel.main.BaseMainSupport : [s-sec-integration.properties] camel.vault.aws.secrets=psql
+2022-11-17 07:46:06.370 INFO 10684 --- [ main] pl.engine.DefaultRuntimeEndpointRegistry : Runtime endpoint registry is in extended mode gathering usage statistics of all incoming and outgoing endpoints (cache limit: 1000)
+2022-11-17 07:46:07.495 INFO 10684 --- [ main] amel.main.MainAutowiredLifecycleStrategy : Autowired property: dataSource on component: sql as exactly one instance of type: javax.sql.DataSource (org.apache.commons.dbcp2.BasicDataSource) found in the registry
+2022-11-17 07:46:07.544 INFO 10684 --- [ main] e.camel.impl.engine.AbstractCamelContext : Apache Camel 3.19.0 (AWSExample) is starting
+2022-11-17 07:46:07.681 INFO 10684 --- [ main] org.apache.camel.main.BaseMainSupport : Property-placeholders summary
+2022-11-17 07:46:07.681 INFO 10684 --- [ main] org.apache.camel.main.BaseMainSupport : [stgresql-source.kamelet.yaml] query=SELECT * FROM accounts
+2022-11-17 07:46:07.681 INFO 10684 --- [ main] org.apache.camel.main.BaseMainSupport : [stgresql-source.kamelet.yaml] dsBean=dsBean-1
+2022-11-17 07:46:07.681 INFO 10684 --- [ main] org.apache.camel.main.BaseMainSupport : [stgresql-source.kamelet.yaml] delay=120000
+2022-11-17 07:46:07.682 INFO 10684 --- [ main] org.apache.camel.main.BaseMainSupport : [stgresql-source.kamelet.yaml] password=xxxxxx
+2022-11-17 07:46:07.682 INFO 10684 --- [ main] org.apache.camel.main.BaseMainSupport : [stgresql-source.kamelet.yaml] serverName=172.17.0.2
+2022-11-17 07:46:07.682 INFO 10684 --- [ main] org.apache.camel.main.BaseMainSupport : [stgresql-source.kamelet.yaml] serverPort=5432
+2022-11-17 07:46:07.682 INFO 10684 --- [ main] org.apache.camel.main.BaseMainSupport : [stgresql-source.kamelet.yaml] databaseName=postgres
+2022-11-17 07:46:07.682 INFO 10684 --- [ main] org.apache.camel.main.BaseMainSupport : [stgresql-source.kamelet.yaml] username=xxxxxx
+2022-11-17 07:46:07.682 INFO 10684 --- [ main] org.apache.camel.main.BaseMainSupport : [log-sink.kamelet.yaml] templateId=log-sink
+2022-11-17 07:46:07.682 INFO 10684 --- [ main] org.apache.camel.main.BaseMainSupport : [log-sink.kamelet.yaml] level=INFO
+2022-11-17 07:46:07.682 INFO 10684 --- [ main] org.apache.camel.main.BaseMainSupport : [log-sink.kamelet.yaml] showHeaders=false
+2022-11-17 07:46:07.682 INFO 10684 --- [ main] org.apache.camel.main.BaseMainSupport : [log-sink.kamelet.yaml] showStreams=true
+2022-11-17 07:46:07.696 INFO 10684 --- [ main] e.camel.impl.engine.AbstractCamelContext : Routes startup (started:3)
+2022-11-17 07:46:07.696 INFO 10684 --- [ main] e.camel.impl.engine.AbstractCamelContext : Started route1 (kamelet://postgresql-source)
+2022-11-17 07:46:07.696 INFO 10684 --- [ main] e.camel.impl.engine.AbstractCamelContext : Started postgresql-source-1 (sql://SELECT%20*%20FROM%20accounts)
+2022-11-17 07:46:07.697 INFO 10684 --- [ main] e.camel.impl.engine.AbstractCamelContext : Started log-sink-2 (kamelet://source)
+2022-11-17 07:46:07.697 INFO 10684 --- [ main] e.camel.impl.engine.AbstractCamelContext : Apache Camel 3.19.0 (AWSExample) started in 1s506ms (build:115ms init:1s239ms start:152ms JVM-uptime:4s)
+2022-11-17 07:46:08.918 INFO 10684 --- [FROM%20accounts] log-sink : Exchange[ExchangePattern: InOnly, BodyType: org.apache.camel.converter.stream.InputStreamCache, Body: {"user_id":1,"username":"John","city":"New York"}]
+2022-11-17 07:46:08.921 INFO 10684 --- [FROM%20accounts] log-sink : Exchange[ExchangePattern: InOnly, BodyType: org.apache.camel.converter.stream.InputStreamCache, Body: {"user_id":2,"username":"John","city":"New York"}]
+2022-11-17 07:46:08.921 INFO 10684 --- [FROM%20accounts] log-sink : Exchange[ExchangePattern: InOnly, BodyType: org.apache.camel.converter.stream.InputStreamCache, Body: {"user_id":3,"username":"John","city":"New York"}]
+2022-11-17 07:46:08.922 INFO 10684 --- [FROM%20accounts] log-sink : Exchange[ExchangePattern: InOnly, BodyType: org.apache.camel.converter.stream.InputStreamCache, Body: {"user_id":4,"username":"John","city":"New York"}]
+2022-11-17 07:46:08.922 INFO 10684 --- [FROM%20accounts] log-sink : Exchange[ExchangePattern: InOnly, BodyType: org.apache.camel.converter.stream.InputStreamCache, Body: {"user_id":5,"username":"John","city":"New York"}]
+2022-11-17 07:46:08.923 INFO 10684 --- [FROM%20accounts] log-sink : Exchange[ExchangePattern: InOnly, BodyType: org.apache.camel.converter.stream.InputStreamCache, Body: {"user_id":6,"username":"John","city":"New York"}]
+2022-11-17 07:46:08.924 INFO 10684 --- [FROM%20accounts] log-sink : Exchange[ExchangePattern: InOnly, BodyType: org.apache.camel.converter.stream.InputStreamCache, Body: {"user_id":7,"username":"John","city":"New York"}]
+2022-11-17 07:46:08.924 INFO 10684 --- [FROM%20accounts] log-sink : Exchange[ExchangePattern: InOnly, BodyType: org.apache.camel.converter.stream.InputStreamCache, Body: {"user_id":8,"username":"John","city":"New York"}]
+2022-11-17 07:46:08.925 INFO 10684 --- [FROM%20accounts] log-sink : Exchange[ExchangePattern: InOnly, BodyType: org.apache.camel.converter.stream.InputStreamCache, Body: {"user_id":9,"username":"John","city":"New York"}]
+2022-11-17 07:46:08.925 INFO 10684 --- [FROM%20accounts] log-sink : Exchange[ExchangePattern: InOnly, BodyType: org.apache.camel.converter.stream.InputStreamCache, Body: {"user_id":10,"username":"John","city":"New York"}]
+
+----
+
+=== Modify postgres user database password and update the secret
+
+You can list the secrets in use from the AWS security vault:
+
+[source,sh]
+----
+camel get vault
+----
+
+While the integration is running you could modify the database password for postgres user
+
+[source,sh]
+----
+docker exec -it psql psql -U postgres
+\password postgres
+insert psql1234
+----
+
+and update the secret accordingly
+
+[source,sh]
+----
+aws secretsmanager put-secret-value --secret-id postgresqlsecret --secret-string file://database-cred-updated.json --region eu-west-1
+----
+
+and restart the docker container since modifying the postgres user password requires a restart
+
+[source,sh]
+----
+docker restart psql
+----
+
+Now, get back, to the running Camel application and wait for the reloading.
+
+[source,sh]
+----
+2022-11-17 07:49:24.183 INFO 10929 --- [agementLoadTask] anager.vault.CloudTrailReloadTriggerTask : Update for AWS secret: psql detected, triggering CamelContext reload
+2022-11-17 07:49:24.184 INFO 10929 --- [agementLoadTask] mel.support.DefaultContextReloadStrategy : Reloading CamelContext (AWSExample) triggered by: AWS Secrets Refresh Task
+2022-11-17 07:49:25.635 INFO 10929 --- [FROM%20accounts] log-sink : Exchange[ExchangePattern: InOnly, BodyType: org.apache.camel.converter.stream.InputStreamCache, Body: {"user_id":1,"username":"John","city":"New York"}]
+2022-11-17 07:49:25.635 INFO 10929 --- [FROM%20accounts] log-sink : Exchange[ExchangePattern: InOnly, BodyType: org.apache.camel.converter.stream.InputStreamCache, Body: {"user_id":2,"username":"John","city":"New York"}]
+2022-11-17 07:49:25.636 INFO 10929 --- [FROM%20accounts] log-sink : Exchange[ExchangePattern: InOnly, BodyType: org.apache.camel.converter.stream.InputStreamCache, Body: {"user_id":3,"username":"John","city":"New York"}]
+2022-11-17 07:49:25.636 INFO 10929 --- [FROM%20accounts] log-sink : Exchange[ExchangePattern: InOnly, BodyType: org.apache.camel.converter.stream.InputStreamCache, Body: {"user_id":4,"username":"John","city":"New York"}]
+2022-11-17 07:49:25.637 INFO 10929 --- [FROM%20accounts] log-sink : Exchange[ExchangePattern: InOnly, BodyType: org.apache.camel.converter.stream.InputStreamCache, Body: {"user_id":5,"username":"John","city":"New York"}]
+2022-11-17 07:49:25.637 INFO 10929 --- [FROM%20accounts] log-sink : Exchange[ExchangePattern: InOnly, BodyType: org.apache.camel.converter.stream.InputStreamCache, Body: {"user_id":6,"username":"John","city":"New York"}]
+2022-11-17 07:49:25.638 INFO 10929 --- [FROM%20accounts] log-sink : Exchange[ExchangePattern: InOnly, BodyType: org.apache.camel.converter.stream.InputStreamCache, Body: {"user_id":7,"username":"John","city":"New York"}]
+2022-11-17 07:49:25.638 INFO 10929 --- [FROM%20accounts] log-sink : Exchange[ExchangePattern: InOnly, BodyType: org.apache.camel.converter.stream.InputStreamCache, Body: {"user_id":8,"username":"John","city":"New York"}]
+2022-11-17 07:49:25.638 INFO 10929 --- [FROM%20accounts] log-sink : Exchange[ExchangePattern: InOnly, BodyType: org.apache.camel.converter.stream.InputStreamCache, Body: {"user_id":9,"username":"John","city":"New York"}]
+2022-11-17 07:49:25.639 INFO 10929 --- [FROM%20accounts] log-sink : Exchange[ExchangePattern: InOnly, BodyType: org.apache.camel.converter.stream.InputStreamCache, Body: {"user_id":10,"username":"John","city":"New York"}]
+----
+
+We changed the password and the Camel route was able to align itself without downtime.
+
+And the secret should also now be listed as updated form the get vault command:
+
+[source,sh]
+----
+camel get vault
+----
+
+=== Developer Web Console
+
+You can enable the developer console via `--console` flag as show:
+
+[source,sh]
+----
+$ camel run --properties=aws-sec-integration.properties sql-query.yaml --console
+----
+
+Then you can browse: http://localhost:8080/q/dev to introspect the running Camel applicaton.
+
+
+=== Help and contributions
+
+If you hit any problem using Camel or have some feedback, then please
+https://camel.apache.org/community/support/[let us know].
+
+We also love contributors, so
+https://camel.apache.org/community/contributing/[get involved] :-)
+
+The Camel riders!
diff --git a/jbang/aws-database-admin-secrets-refresh/aws-sec-integration.properties b/jbang/aws-database-admin-secrets-refresh/aws-sec-integration.properties
new file mode 100644
index 0000000..137d0c1
--- /dev/null
+++ b/jbang/aws-database-admin-secrets-refresh/aws-sec-integration.properties
@@ -0,0 +1,6 @@
+camel.vault.aws.defaultCredentialsProvider=true
+camel.vault.aws.region=eu-west-1
+camel.vault.aws.refreshEnabled=true
+camel.vault.aws.refreshPeriod=1000
+camel.vault.aws.secrets=psql
+camel.main.name = AWSExample
diff --git a/jbang/aws-database-admin-secrets-refresh/database-cred-updated.json b/jbang/aws-database-admin-secrets-refresh/database-cred-updated.json
new file mode 100644
index 0000000..d5f833f
--- /dev/null
+++ b/jbang/aws-database-admin-secrets-refresh/database-cred-updated.json
@@ -0,0 +1,4 @@
+{
+ "username":"postgres",
+ "password":"psql1234"
+}
diff --git a/jbang/aws-database-admin-secrets-refresh/database-cred.json b/jbang/aws-database-admin-secrets-refresh/database-cred.json
new file mode 100644
index 0000000..ff5d4b5
--- /dev/null
+++ b/jbang/aws-database-admin-secrets-refresh/database-cred.json
@@ -0,0 +1,4 @@
+{
+ "username":"postgres",
+ "password":"psql123"
+}
diff --git a/jbang/aws-database-admin-secrets-refresh/populate.sql b/jbang/aws-database-admin-secrets-refresh/populate.sql
new file mode 100644
index 0000000..93e7b1e
--- /dev/null
+++ b/jbang/aws-database-admin-secrets-refresh/populate.sql
@@ -0,0 +1,7 @@
+do $$
+BEGIN
+for r in 1..10 loop
+INSERT into accounts (username,city) VALUES ('John', 'New York');
+END loop;
+END;
+$$;
diff --git a/jbang/aws-database-admin-secrets-refresh/sql-query.yaml b/jbang/aws-database-admin-secrets-refresh/sql-query.yaml
new file mode 100644
index 0000000..9bbff0e
--- /dev/null
+++ b/jbang/aws-database-admin-secrets-refresh/sql-query.yaml
@@ -0,0 +1,35 @@
+## ---------------------------------------------------------------------------
+## Licensed to the Apache Software Foundation (ASF) under one or more
+## contributor license agreements. See the NOTICE file distributed with
+## this work for additional information regarding copyright ownership.
+## The ASF licenses this file to You under the Apache License, Version 2.0
+## (the "License"); you may not use this file except in compliance with
+## the License. You may obtain a copy of the License at
+##
+## http://www.apache.org/licenses/LICENSE-2.0
+##
+## Unless required by applicable law or agreed to in writing, software
+## distributed under the License is distributed on an "AS IS" BASIS,
+## WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+## See the License for the specific language governing permissions and
+## limitations under the License.
+## ---------------------------------------------------------------------------
+
+# camel-k: dependency=camel:aws-secrets-manager
+
+- route:
+ from:
+ uri: "kamelet:postgresql-source"
+ parameters:
+ serverName: "172.17.0.2"
+ username: "{{aws:psql/username}}"
+ password: "{{aws:psql/password}}"
+ query: 'SELECT * FROM accounts'
+ port: 5432
+ databaseName: postgres
+ delay: 120000
+ steps:
+ - to:
+ uri: "kamelet:log-sink"
+ parameters:
+ showStreams: true
diff --git a/jbang/aws-database-admin-secrets-refresh/table.sql b/jbang/aws-database-admin-secrets-refresh/table.sql
new file mode 100644
index 0000000..8bce6cd
--- /dev/null
+++ b/jbang/aws-database-admin-secrets-refresh/table.sql
@@ -0,0 +1 @@
+CREATE TABLE accounts ( user_id serial PRIMARY KEY, username VARCHAR ( 50 ) NOT NULL, city VARCHAR ( 50 ) NOT NULL);