(axis-axis2-c-core) 02/02: Add HTTP2_JSON_DEBUG.md — deployment debugging guide from live session

[ro...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

robertlazarski pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/axis-axis2-c-core.git

commit 9a9227b07c1bcbfae743c796cb46a529de0f06de
Author: Robert Lazarski <ro...@gmail.com>
AuthorDate: Wed Apr 8 04:57:04 2026 -1000

    Add HTTP2_JSON_DEBUG.md — deployment debugging guide from live session
    
    Documents 8 real problems encountered deploying Axis2/C HTTP/2 JSON services
    on Ubuntu 22.04 with packaged Apache, including:
    
    - Missing vhost config (site not enabled)
    - OCSP stapling with self-signed certs
    - Transport sender/receiver not needed in HTTP/2 mode
    - axis2_get_instance signature mismatch causing segfaults
    - json-c version incompatibility
    - Packaged vs source-built Apache path differences
    - GDB/coredumpctl backtrace procedures for systemd services
    - Post-restart health check race conditions
    
    Co-Authored-By: Claude Opus 4.6 (1M context) <no...@anthropic.com>
---
 docs/HTTP2_JSON_DEBUG.md | 346 +++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 346 insertions(+)

diff --git a/docs/HTTP2_JSON_DEBUG.md b/docs/HTTP2_JSON_DEBUG.md
new file mode 100644
index 000000000..42f7ddfb9
--- /dev/null
+++ b/docs/HTTP2_JSON_DEBUG.md
@@ -0,0 +1,346 @@
+# HTTP/2 JSON Deployment Debugging Guide
+
+Practical debugging guide for Axis2/C HTTP/2 JSON services running under
+Apache httpd on Debian/Ubuntu systems. Based on real deployment issues
+encountered on Ubuntu 22.04 with the packaged Apache (`apache2` package).
+
+## Quick Reference: Log Locations
+
+| Log | Path | What it tells you |
+|-----|------|-------------------|
+| Apache error log | `/var/log/apache2/error.log` | Module load failures, SSL errors, child crashes |
+| Apache access log | `/var/log/apache2/access.log` | Request routing (empty = requests not reaching vhost) |
+| Apache other vhosts | `/var/log/apache2/other_vhosts_access.log` | Requests hitting wrong vhost |
+| Axis2/C log | `/var/log/axis2c/axis2.log` | Service loading, transport init, worker creation |
+
+## Quick Reference: Diagnostic Commands
+
+```bash
+# Is Apache running and on which ports?
+systemctl status apache2
+ss -tlnp | grep apache2
+
+# Which vhosts are active?
+apache2ctl -S
+
+# Which modules are loaded?
+apache2ctl -M | grep -E 'ssl|http2|axis2'
+
+# Test config syntax
+apache2ctl configtest
+
+# Recent core dumps from Apache
+coredumpctl list apache2 | tail -5
+```
+
+---
+
+## Problem 1: Apache Starts but Nothing Listens / Curl Returns 000
+
+**Symptoms:**
+- `systemctl status apache2` shows active
+- `curl -k https://localhost/` returns empty or 000
+- `access.log` is empty
+
+**Diagnosis:** Check if the SSL vhost is actually loaded.
+
+```bash
+apache2ctl -S
+```
+
+If only the port 80 vhost appears and the port 443 vhost is missing:
+
+```bash
+# Is the site config file present?
+ls /etc/apache2/sites-available/axis2-services.conf
+
+# Is it enabled (symlinked)?
+ls /etc/apache2/sites-enabled/ | grep axis2
+
+# Enable it
+a2ensite axis2-services
+apache2ctl configtest && systemctl restart apache2
+```
+
+**Root cause:** The site config was never copied to `sites-available/` or
+`a2ensite` was never run. The deploy script copies from the source tree —
+if the file isn't committed to git, it won't exist on the target machine.
+
+---
+
+## Problem 2: TLS Handshake Bytes in Port 80 Log
+
+**Symptom:** `other_vhosts_access.log` shows:
+```
+penguin.alphatheory.local:80 127.0.0.1 - - [...] "\x16\x03\x01\x02" 400 499
+```
+
+**What it means:** `\x16\x03\x01` is a TLS ClientHello hitting the plain
+HTTP port 80 vhost. The client is sending HTTPS but there is no port 443
+listener, so the connection fails and may fall through to port 80.
+
+**Fix:** Ensure `mod_ssl` is loaded and the 443 vhost is active:
+```bash
+a2enmod ssl
+apache2ctl -S   # should show *:443 vhost
+```
+
+---
+
+## Problem 3: OCSP Stapling Errors with Self-Signed Certificates
+
+**Symptom in error.log:**
+```
+AH02217: ssl_stapling_init_cert: can't retrieve issuer certificate!
+AH02604: Unable to configure certificate for stapling
+```
+
+**What it means:** `SSLUseStapling on` requires a CA-signed certificate
+with an OCSP responder URL. Self-signed certificates have no issuer to
+query for stapling.
+
+**Fix:** Disable OCSP stapling in the vhost config:
+```apache
+# SSLUseStapling on
+```
+
+Re-enable when switching to a CA-signed certificate (e.g., Let's Encrypt).
+
+---
+
+## Problem 4: "Transport sender is NULL" — Worker Creation Fails
+
+**Symptom in axis2.log:**
+```
+Loading shared library libaxis2_http_sender.so Failed. No such file or directory
+Transport sender is NULL for transport http, unable to continue
+```
+
+**Symptom in error.log:**
+```
+[Axis2] Error creating mod_axis2 apache2 worker
+Child returned a Fatal error... Apache is exiting!
+```
+
+**What it means:** `axis2.xml` references `axis2_http_sender` and
+`axis2_http_receiver`, which map to shared libraries. In HTTP/2 mode these
+libraries are intentionally **not built** — Apache httpd handles all
+transport via `mod_axis2` (see `docs/HTTP2_CONDITIONAL_COMPILATION.md`).
+
+**Fix:** Comment out the `<transportSender>` and `<transportReceiver>`
+blocks in `axis2.xml`:
+```xml
+<!--transportSender name="http" class="axis2_http_sender">
+    ...
+</transportSender-->
+
+<!--transportReceiver name="http" class="axis2_http_receiver">
+    ...
+</transportReceiver-->
+```
+
+The source-controlled copy is `samples/server/axis2.xml`. After editing,
+run `make install` to propagate to `$(prefix)/share/axis2c/axis2.xml`.
+
+See also: `docs/HTTP2_AXIS2_DOT_XML.md`
+
+---
+
+## Problem 5: Child Process Segfault (SIGSEGV) Loop
+
+**Symptom in error.log:**
+```
+child pid NNNNN exit signal Segmentation fault (11), possible coredump in /etc/apache2
+```
+
+The parent process stays alive, keeps spawning children that immediately
+crash — Apache appears "running" but serves no requests.
+
+### Getting a Backtrace
+
+On systemd systems, core dumps are captured by `systemd-coredump`:
+
+```bash
+# List recent Apache crashes
+coredumpctl list apache2 | tail -10
+
+# Show crash info including stack trace
+coredumpctl info apache2 | head -80
+
+# Full GDB backtrace (if debug symbols available)
+coredumpctl debug apache2 --debugger=gdb \
+    --debugger-arguments="-batch -ex 'bt full' -ex 'quit'"
+```
+
+If `coredumpctl` is not available, enable manual core dumps:
+```bash
+# Set core dump pattern and size limit
+echo '/tmp/core.%e.%p' > /proc/sys/kernel/core_pattern
+ulimit -c unlimited
+
+# Note: systemctl-started processes inherit systemd's limits, not the shell's.
+# For systemd services, edit the unit override:
+systemctl edit apache2
+# Add:
+#   [Service]
+#   LimitCORE=infinity
+
+systemctl restart apache2
+sleep 3
+ls /tmp/core.apache2.*
+
+# Analyze with GDB
+gdb /usr/sbin/apache2 /tmp/core.apache2.* -batch -ex 'bt full' -ex 'quit'
+```
+
+### Common Segfault: axis2_get_instance Signature Mismatch
+
+**Stack trace pattern:**
+```
+#0  axis2_svc_create (libaxis2_engine.so)
+#1  axis2_<servicename>_create (lib<servicename>.so)
+#2  axutil_class_loader_create_dll (libaxutil.so)
+#3  axis2_msg_recv_load_and_init_svc_impl (libaxis2_engine.so)
+#4  axis2_apache2_worker_create (libmod_axis2.so)
+```
+
+**Root cause:** The service's `axis2_get_instance` function has the wrong
+signature. The class loader (`axutil_class_loader_create_dll`) calls it as:
+
+```c
+// Expected signature (CREATE_FUNCT from axutil_dll_desc.h):
+int axis2_get_instance(void **inst, const axutil_env_t *env);
+
+// Called as:
+create_funct(&obj, env);
+```
+
+If the service instead exports:
+```c
+// WRONG — causes segfault:
+axis2_svc_t* axis2_get_instance(const axutil_env_t *env);
+```
+
+Then `&obj` (a `void**`) is misinterpreted as `env`, and the real `env`
+goes to an unused parameter slot. Any dereference of `env` inside the
+service (e.g., `env->allocator` in `axis2_svc_create`) segfaults.
+
+**Fix:** Match the `CREATE_FUNCT` / `DELETE_FUNCT` typedefs exactly:
+```c
+AXIS2_EXTERN int AXIS2_CALL
+axis2_get_instance(void **inst, const axutil_env_t *env)
+{
+    *inst = axis2_my_service_create(env);
+    if (!(*inst)) return AXIS2_FAILURE;
+    return AXIS2_SUCCESS;
+}
+
+AXIS2_EXTERN int AXIS2_CALL
+axis2_remove_instance(void *inst, const axutil_env_t *env)
+{
+    axis2_svc_t *svc = (axis2_svc_t *)inst;
+    if (svc) axis2_svc_free(svc, env);
+    return AXIS2_SUCCESS;
+}
+```
+
+---
+
+## Problem 6: Apache Module Not Found at Install Time
+
+**Symptom during `make install`:**
+```
+Neither mod_axis2.so nor libmod_axis2.so found at /usr/local/apache2/modules/
+```
+
+**What it means:** The Makefile.am `apachemoddir` is set to a path that
+doesn't exist. Source-built Apache uses `/usr/local/apache2/modules/`;
+packaged Apache uses system paths.
+
+**Fix:** Set `apachemoddir = @prefix@/lib` in
+`src/core/transport/http/server/apache2/Makefile.am`, then `autoreconf -fiv`
+and rebuild. The `axis2-services.conf` loads the module from
+`/usr/local/axis2c/lib/libmod_axis2.so` which matches `--prefix=/usr/local/axis2c`.
+
+---
+
+## Problem 7: json-c Version Incompatibility
+
+**Symptom during `make`:**
+```
+error: 'json_tokener_error_memory' undeclared
+```
+
+**What it means:** The code uses `json_tokener_error_memory` which was
+added in json-c **0.17**, but the version guard checks for `>= 0.14`.
+Ubuntu 22.04 ships json-c 0.15 which passes the guard but lacks the enum.
+
+**Fix:** In `src/core/transport/http/util/axis2_json_reader.c`, change the
+version guard for `json_tokener_error_memory` and `json_tokener_error_size`
+from `(14 << 8)` to `(17 << 8)`:
+
+```c
+#if defined(JSON_C_VERSION_NUM) && JSON_C_VERSION_NUM >= ((0 << 16) | (17 << 8))
+    case json_tokener_error_memory:
+        return "error_memory";
+    case json_tokener_error_size:
+        return "error_size";
+#endif
+```
+
+---
+
+## Problem 8: Post-Restart Health Check Passes but Service Crashes
+
+**Symptom:** The deploy script reports "Apache is running" but verification
+curls all return 000.
+
+**What it means:** Apache's parent process starts successfully (passes
+`systemctl is-active` check), but child processes crash during worker
+initialization. With `mpm_event`, the parent keeps trying to spawn children,
+creating a crash loop visible in `error.log`.
+
+**Diagnosis:**
+```bash
+# The parent is alive but children are dying
+tail -20 /var/log/apache2/error.log   # look for "Child returned a Fatal error"
+tail -20 /var/log/axis2c/axis2.log    # look for loading errors before the crash
+```
+
+**Common causes (in order of likelihood):**
+1. Missing transport libs in `axis2.xml` (Problem 4)
+2. Service DLL signature mismatch (Problem 5)
+3. Missing service shared libraries
+4. Corrupt `axis2.xml` (syntax errors)
+
+---
+
+## Packaged Apache vs Source-Built Apache
+
+Many issues stem from path differences between these two environments:
+
+| Component | Packaged (Debian/Ubuntu) | Source-built |
+|-----------|-------------------------|--------------|
+| Binary | `/usr/sbin/apache2` | `/usr/local/apache2/bin/httpd` |
+| Modules dir | `/usr/lib/apache2/modules/` | `/usr/local/apache2/modules/` |
+| Config dir | `/etc/apache2/` | `/usr/local/apache2/conf/` |
+| Site management | `a2ensite` / `a2dissite` | Manual `Include` directives |
+| Module management | `a2enmod` / `a2dismod` | `LoadModule` in httpd.conf |
+| mod_http2 name | `mod_http2.so` (`a2enmod http2`) | `mod_h2.so` |
+| Service management | `systemctl restart apache2` | `httpd -k restart` |
+
+The `deploy-localhost.sh` script and `axis2-services.conf` are designed for
+the **packaged** Apache. The individual service build scripts
+(`build_json_service.sh`, etc.) detect source-built Apache at
+`/usr/local/apache2` and configure it automatically; otherwise they skip
+module installation and defer to `deploy-localhost.sh`.
+
+---
+
+## Related Documentation
+
+- `docs/HTTP2_AXIS2_DOT_XML.md` — Why axis2.xml doesn't need HTTP/2 transport config
+- `docs/HTTP2_CONDITIONAL_COMPILATION.md` — What's built/excluded in HTTP/2 mode
+- `docs/HTTP2_LIBRARY_LOADING.md` — How Axis2/C loads service shared libraries
+- `docs/HTTP2_APACHE2_WORKER.md` — Apache worker initialization details
+- `docs/userguide/json-httpd-h2-userguide.md` — Full deployment guide