You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@qpid.apache.org by "a3f (via GitHub)" <gi...@apache.org> on 2024/07/10 10:27:40 UTC
[PR] PROTON-2594: Add OpenSSL PKCS#11 PROVIDER support to enable HSM use [qpid-proton]
a3f opened a new pull request, #430:
URL: https://github.com/apache/qpid-proton/pull/430
Hardware Security Modules (HSMs) are physical computing devices that
safeguard secrets and allow performing cryptographic operations like
encryption and signing without necessarily divulging the private key
material.
PKCS#11 is a platform-independent API for cryptographic tokens like
HSMs. It defines a scheme for pkcs11: URIs that describe objects on the
token as well as an API to interact with them.
This commit adds support for transparent use of PKCS#11 URIs: Whenever a
certificate or private key path is prefixed with pkcs11: it will be
interpreted as PKCS#11 URI instead of a file path and OpenSSL will do
all necessary communication with the HSM behind the scenes.
For programs like proton that use OpenSSL, this could have been
realized in two ways:
- OpenSSL ENGINE: Introduced in OpenSSL 0.9.6 and deprecated in 3.0
- OpenSSL PROVIDER: Introduced in OpenSSL 3.0 to replace ENGINE
While both are supported in recent OpenSSL versions, it's more future
proof to use the PROVIDER API, even if this comes at the cost of having
to add an #ifdef to the code.
This has been tested on Linux/x86_64 with softhsm and Linux/arm32 with
OP-TEE, both with v0.5 of https://github.com/latchset/pkcs11-provider.
As everything is loaded dynamically, we do not link against any
PKCS#11-related shared libraries. It's expected that PKCS#11 provider
will already be loaded via OPENSSL_CONF if it's to be used.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@qpid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@qpid.apache.org
For additional commands, e-mail: dev-help@qpid.apache.org
Re: [PR] PROTON-2594: Add OpenSSL PKCS#11 PROVIDER support to enable HSM use [qpid-proton]
Posted by "cliffjansen (via GitHub)" <gi...@apache.org>.
cliffjansen commented on PR #430:
URL: https://github.com/apache/qpid-proton/pull/430#issuecomment-2475308160
This PR has some problems for which I have some proposed fixes. The main ones being compilation failures on older versions of OpenSSL and a memory leak.
If you can confirm my changes do not break anything in your environment with a real hardware security module, I will check this in and add a documentation update. My proposed changes are here:
https://github.com/cliffjansen/qpid-proton/tree/pn2594_wip
I will revert the github action back to ubuntu-latest once that advances to ubuntu-24.04 in a few weeks.
It is unfortunate the software emulation bits of PKCS11 support are sufficiently buggy/immature that you had to take extraordinary steps to build a custom package and configuration just to get the test to run. Hopefully the tests can be altered to use regular distro packages in time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@qpid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@qpid.apache.org
For additional commands, e-mail: dev-help@qpid.apache.org
Re: [PR] PROTON-2594: Add OpenSSL PKCS#11 PROVIDER support to enable HSM use [qpid-proton]
Posted by "astitcher (via GitHub)" <gi...@apache.org>.
astitcher commented on PR #430:
URL: https://github.com/apache/qpid-proton/pull/430#issuecomment-2225813526
This looks good to me - is there any way of testing this functionality in the CI job? (Rather than just testing that nothing regressed!)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@qpid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@qpid.apache.org
For additional commands, e-mail: dev-help@qpid.apache.org
Re: [PR] PROTON-2594: Add OpenSSL PKCS#11 PROVIDER support to enable HSM use [qpid-proton]
Posted by "a3f (via GitHub)" <gi...@apache.org>.
a3f commented on PR #430:
URL: https://github.com/apache/qpid-proton/pull/430#issuecomment-2479703596
Thanks a lot @astitcher & @cliffjansen!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@qpid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@qpid.apache.org
For additional commands, e-mail: dev-help@qpid.apache.org
Re: [PR] PROTON-2594: Add OpenSSL PKCS#11 PROVIDER support to enable HSM use [qpid-proton]
Posted by "astitcher (via GitHub)" <gi...@apache.org>.
astitcher commented on PR #430:
URL: https://github.com/apache/qpid-proton/pull/430#issuecomment-2445134599
Hmm, have you made sure that the CI image actually has the relevant openssl modules for the softhsm? Maybe you need to apt install something to make the test work in the CI envioronment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@qpid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@qpid.apache.org
For additional commands, e-mail: dev-help@qpid.apache.org
Re: [PR] PROTON-2594: Add OpenSSL PKCS#11 PROVIDER support to enable HSM use [qpid-proton]
Posted by "a3f (via GitHub)" <gi...@apache.org>.
a3f closed pull request #430: PROTON-2594: Add OpenSSL PKCS#11 PROVIDER support to enable HSM use
URL: https://github.com/apache/qpid-proton/pull/430
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@qpid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@qpid.apache.org
For additional commands, e-mail: dev-help@qpid.apache.org
Re: [PR] PROTON-2594: Add OpenSSL PKCS#11 PROVIDER support to enable HSM use [qpid-proton]
Posted by "astitcher (via GitHub)" <gi...@apache.org>.
astitcher commented on PR #430:
URL: https://github.com/apache/qpid-proton/pull/430#issuecomment-2457978207
I've asked @cliffjansen to take a look at this. He has more experience with the openssl code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@qpid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@qpid.apache.org
For additional commands, e-mail: dev-help@qpid.apache.org
Re: [PR] PROTON-2594: Add OpenSSL PKCS#11 PROVIDER support to enable HSM use [qpid-proton]
Posted by "a3f (via GitHub)" <gi...@apache.org>.
a3f commented on PR #430:
URL: https://github.com/apache/qpid-proton/pull/430#issuecomment-2225923971
> is there any way of testing this functionality in the CI job? (Rather than just testing that nothing regressed!)
How about we apply [this patch](https://github.com/a3f/pkcs11-provider-qpid-proton-bug-reproduction/blob/master/0001-HACK-PROTON-2594-cpp-connect_config_test-adapt-for-t.patch) and then run the test twice once with PEM files and once with PKCS#11?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@qpid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@qpid.apache.org
For additional commands, e-mail: dev-help@qpid.apache.org
Re: [PR] PROTON-2594: Add OpenSSL PKCS#11 PROVIDER support to enable HSM use [qpid-proton]
Posted by "a3f (via GitHub)" <gi...@apache.org>.
a3f commented on PR #430:
URL: https://github.com/apache/qpid-proton/pull/430#issuecomment-2446700159
> Hmm, have you made sure that the CI image actually has the relevant openssl modules for the softhsm? Maybe you need to apt install something to make the test work in the CI envioronment.
Yes, that's what the `apt install pkcs11-provider` was for. I managed to reproduce in a ubuntu-24.04 container. The pkcs11-provider shipped there was the culprit. I now build the pkcs11-provider manually and the tests pass finally: https://github.com/a3f/qpid-proton/actions/runs/11591599162.
> If at all possible I'd really like to maintain this distinction for these tests too.
I can understand that, but most users will likely not run the PKCS#11 tests anyway, because there are more prerequisites than the C++ compiler (Linux, softhsm2, pkcs11-provider, OpenSSL > 3.0).
Unfortunately, this side quest with debugging the CI has more than depleted the time I have alotted for this, so I won't be able to rewrite the test myself anytime soon.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@qpid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@qpid.apache.org
For additional commands, e-mail: dev-help@qpid.apache.org
Re: [PR] PROTON-2594: Add OpenSSL PKCS#11 PROVIDER support to enable HSM use [qpid-proton]
Posted by "cliffjansen (via GitHub)" <gi...@apache.org>.
cliffjansen commented on PR #430:
URL: https://github.com/apache/qpid-proton/pull/430#issuecomment-2479655936
Now on main. Rebased to af0124ef
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@qpid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@qpid.apache.org
For additional commands, e-mail: dev-help@qpid.apache.org
Re: [PR] PROTON-2594: Add OpenSSL PKCS#11 PROVIDER support to enable HSM use [qpid-proton]
Posted by "astitcher (via GitHub)" <gi...@apache.org>.
astitcher commented on PR #430:
URL: https://github.com/apache/qpid-proton/pull/430#issuecomment-2450758241
Thanks for the work of adding the test.
Can you squash together the last 4 commits into 1 commit leaving just 3 commits overall please. I don't think those commits standalone really - they are just part of adding the test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@qpid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@qpid.apache.org
For additional commands, e-mail: dev-help@qpid.apache.org
Re: [PR] PROTON-2594: Add OpenSSL PKCS#11 PROVIDER support to enable HSM use [qpid-proton]
Posted by "astitcher (via GitHub)" <gi...@apache.org>.
astitcher commented on code in PR #430:
URL: https://github.com/apache/qpid-proton/pull/430#discussion_r1821365884
##########
cpp/src/pkcs11_test.cpp:
##########
@@ -0,0 +1,113 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+#include "test_bits.hpp"
+
+#include "proton/connection_options.hpp"
+#include "proton/container.hpp"
+#include "proton/ssl.hpp"
+
+// The C++ API lacks a way to test for presence of extended SSL support.
+#include "proton/ssl.h"
+
+#include <cstdio>
+#include <string>
+
+#include "test_handler.hpp"
+
+#define SKIP_RETURN_CODE 127
+
+namespace {
+
+using namespace std;
+using namespace proton;
+
+// Hack to write strings with embedded '"' and newlines
+#define RAW_STRING(...) #__VA_ARGS__
Review Comment:
We currently require C++17 or above and so can now use native C++11 raw strings.
I know you just copied this from a previous c++ test so it's not a required change, more a note to myself that we can improve the test code a bit now!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@qpid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@qpid.apache.org
For additional commands, e-mail: dev-help@qpid.apache.org
Re: [PR] PROTON-2594: Add OpenSSL PKCS#11 PROVIDER support to enable HSM use [qpid-proton]
Posted by "astitcher (via GitHub)" <gi...@apache.org>.
astitcher commented on PR #430:
URL: https://github.com/apache/qpid-proton/pull/430#issuecomment-2445099977
> I added a `pkc11_test` that receives `pkcs11:` URIs for both server and client certificates and keys via environment variables. A script executed by CI adds the PEM files available in tree to a SoftHSM and exports said environment variables.
>
> The test is skipped without being marked as failure whenever the environment variables are missing as not break other users running the test suite, but lacking the prerequisites.
>
> This works fine locally, but unfortunately fails in CI and I am not sure why.
>
> @astitcher How would you go about debugging issues that only fail in CI? I tried https://github.com/nektos/act, but I don't have a docker image that looks sufficiently enough like the Github runner VM images, so it's able to execute the tests as-is.
I'm not aware of anything specific. I guess adding extra debugging output until you can figure out enough to make it fail in your local environment too. I know that's not very helpful - sorry :-(
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@qpid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@qpid.apache.org
For additional commands, e-mail: dev-help@qpid.apache.org
Re: [PR] PROTON-2594: Add OpenSSL PKCS#11 PROVIDER support to enable HSM use [qpid-proton]
Posted by "a3f (via GitHub)" <gi...@apache.org>.
a3f commented on PR #430:
URL: https://github.com/apache/qpid-proton/pull/430#issuecomment-2445066361
I added a `pkc11_test` that receives `pkcs11:` URIs for both server and client certificates and keys via environment variables. A script executed by CI adds the PEM files available in tree to a SoftHSM and exports said environment variables.
The test is skipped without being marked as failure whenever the environment variables are missing as not break other users running the test suite, but lacking the prerequisites.
This works fine locally, but unfortunately fails in CI and I am not sure why.
@astitcher How would you go about debugging issues that only fail in CI? I tried https://github.com/nektos/act, but I don't have a docker image that looks sufficiently enough like the Github runner VM images, so it's able to execute the tests as-is.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@qpid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@qpid.apache.org
For additional commands, e-mail: dev-help@qpid.apache.org
Re: [PR] PROTON-2594: Add OpenSSL PKCS#11 PROVIDER support to enable HSM use [qpid-proton]
Posted by "a3f (via GitHub)" <gi...@apache.org>.
a3f commented on PR #430:
URL: https://github.com/apache/qpid-proton/pull/430#issuecomment-2475596446
> This PR has some problems for which I have some proposed fixes. The main ones being compilation failures on older versions of OpenSSL and a memory leak.
Thanks and sorry for missing those. I now squashed your fixes directly into the commits.
> If you can confirm my changes do not break anything in your environment with a real hardware security module, I will check this in and add a documentation update. My proposed changes are here:
Your changes look fine to me. Feel free to push any commits you want to add on top of this PR.
> I will revert the github action back to ubuntu-latest once that advances to ubuntu-24.04 in a few weeks.
Sounds good.
> It is unfortunate the software emulation bits of PKCS11 support are sufficiently buggy/immature that you had to take extraordinary steps to build a custom package and configuration just to get the test to run. Hopefully the tests can be altered to use regular distro packages in time.
This is mostly because of the OpenSSL switch from ENGINE to PROVIDER. The PKCS#11 Engine is shipped universally, but is now deprecated. The provider is getting there.
Thanks again,
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@qpid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@qpid.apache.org
For additional commands, e-mail: dev-help@qpid.apache.org