imx-cst: fix remaining build failures with OpenSSL 4

Two more OpenSSL 4.0 breakages on top of the ASN.1 opacity already
handled by 0012-fix-openssl-4-asn1-opaque.patch.

The ENGINE API is gone. <openssl/engine.h> is still shipped as a
source-compatibility stub, so linking fails on ENGINE_free,
ENGINE_load_builtin_engines, ENGINE_by_id, ENGINE_init, ENGINE_finish,
ENGINE_ctrl_cmd and ENGINE_load_private_key. Define OPENSSL_ENGINE_STUBS,
which OpenSSL 4 provides for exactly this case: the declarations become
inline no-ops returning failure, so the file compiles and links unchanged.
OPENSSL_SUPPRESS_DEPRECATED is already defined there, so the stubs'
deprecation attributes do not trip -Werror. That leaves ENGINE_by_id()
returning NULL, which the existing code fed straight into ENGINE_init() -
a NULL dereference predating OpenSSL 4 - so check it and report what went
wrong, mentioning OpenSSL 4 so the failure is not mistaken for a missing
module. PKCS#11 backed signing genuinely is unavailable there, as no
provider exposes an equivalent of the pkcs11 engine's LOAD_CERT_CTRL.

X509_get_subject_name() now returns const X509_NAME *, so its result can
no longer be the destination of X509_NAME_add_entry_by_txt():

  src/tools/pki_tree/pki_helper.c:440:10: error: assigning to 'X509_NAME *' from 'const X509_NAME *' discards qualifiers

There is no mutable counterpart, so build the subject name standalone and
install it with X509_set_subject_name(); both setters copy it, so it is
freed once the issuer name has been set from it too.

AI-Generated: Uses Claude Code
Signed-off-by: Khem Raj <raj.khem@gmail.com>
This commit is contained in:
Khem Raj
2026-09-11 22:47:21 -07:00
parent 5bdb6cac38
commit 33d910cca2
3 changed files with 117 additions and 0 deletions
@@ -0,0 +1,67 @@
From: Khem Raj <raj.khem@gmail.com>
Date: Wed, 3 Sep 2026 01:30:00 +0000
Subject: Fix FTBFS with OpenSSL 4.0: ENGINE API removal
OpenSSL 4.0 removed the ENGINE API. <openssl/engine.h> is still shipped,
but only as a source-compatibility stub, so linking fails:
ld.lld: error: undefined symbol: ENGINE_free
ld.lld: error: undefined symbol: ENGINE_load_builtin_engines
ld.lld: error: undefined symbol: ENGINE_by_id
ld.lld: error: undefined symbol: ENGINE_init
ld.lld: error: undefined symbol: ENGINE_finish
ld.lld: error: undefined symbol: ENGINE_ctrl_cmd
ld.lld: error: undefined symbol: ENGINE_load_private_key
Define OPENSSL_ENGINE_STUBS, which OpenSSL 4 offers exactly for this
case: the ENGINE_* declarations become inline no-ops returning failure,
so the file compiles and links unchanged. OPENSSL_SUPPRESS_DEPRECATED is
already defined here, so the stubs' deprecation attributes do not trip
-Werror either.
That leaves ENGINE_by_id() returning NULL at runtime, which the existing
code fed straight into ENGINE_init() - a NULL dereference that predates
OpenSSL 4. Check the result and print what went wrong instead, noting the
OpenSSL 4 situation so the failure is not mistaken for a missing module.
Providers replace engines upstream, but nothing exposes an equivalent of
the pkcs11 engine's LOAD_CERT_CTRL command yet, so PKCS#11 backed signing
genuinely is unavailable with OpenSSL 4.
Upstream-Status: Pending
Signed-off-by: Khem Raj <raj.khem@gmail.com>
---
--- a/src/lib/back_end/engine.c
+++ b/src/lib/back_end/engine.c
@@ -4,6 +4,13 @@
*/
#define OPENSSL_SUPPRESS_DEPRECATED
+/*
+ * OpenSSL 4.0 removed the ENGINE API. <openssl/engine.h> still declares
+ * it for source compatibility, and defining OPENSSL_ENGINE_STUBS turns the
+ * declarations into inline no-ops so this file keeps linking. The pkcs11
+ * engine simply does not exist there, which engine_ctx_init() reports.
+ */
+#define OPENSSL_ENGINE_STUBS
#include "engine.h"
#include "err.h"
@@ -63,6 +70,16 @@
ctx->engine = ENGINE_by_id("pkcs11");
+ if (!ctx->engine) {
+ fprintf(stderr, "ERROR: cannot load the pkcs11 OpenSSL engine\n");
+#if OPENSSL_VERSION_MAJOR >= 4
+ fprintf(stderr,
+ "ERROR: OpenSSL 4.0 removed ENGINE support, so PKCS#11 "
+ "backed signing is unavailable\n");
+#endif
+ return 0;
+ }
+
#ifdef DEBUG
ENGINE_ctrl_cmd_string(ctx->engine, "VERBOSE", NULL, 0);
#endif
@@ -0,0 +1,48 @@
From: Khem Raj <raj.khem@gmail.com>
Date: Wed, 3 Sep 2026 01:30:00 +0000
Subject: Fix FTBFS with OpenSSL 4.0: const X509_get_subject_name()
OpenSSL 4.0 changed X509_get_subject_name() to return a const X509_NAME *,
so using its result as the destination of X509_NAME_add_entry_by_txt() no
longer compiles:
src/tools/pki_tree/pki_helper.c:440:10: error: assigning to 'X509_NAME *'
from 'const X509_NAME *' discards qualifiers
[-Werror,-Wincompatible-pointer-types-discards-qualifiers]
There is no mutable counterpart to the getter (unlike X509_getm_notAfter),
so build the subject name as a standalone X509_NAME and install it with
X509_set_subject_name(). Both setters copy the name, so it is freed once
the issuer name has been set from it too.
Upstream-Status: Pending
Signed-off-by: Khem Raj <raj.khem@gmail.com>
---
--- a/src/tools/pki_tree/pki_helper.c
+++ b/src/tools/pki_tree/pki_helper.c
@@ -437,11 +437,15 @@
if (X509_set_pubkey(x509, pkey) != 1)
handle_errors();
- name = X509_get_subject_name(x509);
+ name = X509_NAME_new();
+ if (!name)
+ handle_errors();
if (X509_NAME_add_entry_by_txt(name, "CN", MBSTRING_ASC,
(const unsigned char *) subj, -1, -1,
0) != 1)
handle_errors();
+ if (X509_set_subject_name(x509, name) != 1)
+ handle_errors();
if (sign_cert)
{
if (X509_set_issuer_name(x509, X509_get_subject_name(sign_cert)) != 1)
@@ -452,6 +456,7 @@
if (X509_set_issuer_name(x509, name) != 1)
handle_errors();
}
+ X509_NAME_free(name);
if (is_ca)
{
@@ -27,6 +27,8 @@ SRC_URI = "\
file://0012-fix-openssl-4-asn1-opaque.patch \
file://0013-convlb-remove-redundant-NULL-definition.patch \
file://0014-fix-pointer-sign-errors-with-clang.patch \
file://0015-fix-openssl-4-engine-api-removal.patch \
file://0016-fix-openssl-4-const-subject-name.patch \
"
SRC_URI[sha256sum] = "fd92a1a9faa10fb81bbf752c7ee1e257f17e1ec4c2964f8a47adf8a3eaa7df41"