You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@ambari.apache.org by ji...@apache.org on 2026/10/10 09:25:07 UTC
(ambari-mpacks) 23/35: AMBARI-26663: Record native configuration contracts and cluster acceptance
This is an automated email from the ASF dual-hosted git repository.
asf-gitbox-commits pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/ambari-mpacks.git
commit bd595d86a1de08519245b99f3b24dff4e1988011
Author: jialiang <ji...@gmail.com>
AuthorDate: Mon Sep 28 14:48:18 2026 +0800
AMBARI-26663: Record native configuration contracts and cluster acceptance
---
README.md | 4 ++
docs/native-configuration-acceptance.md | 123 ++++++++++++++++++++++++++++++++
docs/native-configuration.md | 100 ++++++++++++++++++++++++++
tests/test_native_metadata.py | 47 ++++++++++++
4 files changed, 274 insertions(+)
diff --git a/README.md b/README.md
index 002c12c..783ac07 100644
--- a/README.md
+++ b/README.md
@@ -18,6 +18,10 @@ limitations under the License.
# Ambari Third-Party Management Packs
+The [native configuration guide](docs/native-configuration.md) describes the
+Basic Settings, Native Settings, Configuration Files and custom-property
+editors for the eight expanded service packs.
+
This independent repository contains management definitions and runtime installers.
Core delivery is tracked in the Ambari worktree's `docs/mpack` baseline; the
[component expansion results](docs/component-expansion-results.md) record the
diff --git a/docs/native-configuration-acceptance.md b/docs/native-configuration-acceptance.md
new file mode 100644
index 0000000..541e6ca
--- /dev/null
+++ b/docs/native-configuration-acceptance.md
@@ -0,0 +1,123 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to You under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+http://www.apache.org/licenses/LICENSE-2.0
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# Native Configuration Acceptance
+
+Date: 2026-09-28. This deliverable follows the request to replace minimal
+installation-only configuration forms with practical native configuration
+management. The implementation uses the service's own files and metadata;
+the am-pro Kyuubi implementation was a reference, not a copied dependency.
+
+## Result
+
+Eight packs now provide Basic Settings and Native Settings tabs, complete-file
+editors where applicable, and custom native properties. Save/restart/application
+and restoration were exercised on the running Rocky 8 ARM64 acceptance clusters.
+The UI fixes preserve custom-property entry when all predefined fields are
+themed, honor adding_forbidden, and expose defaults for newly introduced
+configuration types on the current configuration version.
+
+| Pack | Final definition | Final service-check request / task |
+| --- | --- | --- |
+| Kyuubi | 1.0.0.11 | 274 / 556 |
+| Airflow | 1.0.0.6 | 257 / 535 |
+| Celeborn | 1.0.0.2 | 277 / 561 |
+| DolphinScheduler | 1.0.0.5 | 271 / 551 |
+| Trino | 1.0.0.4 | 259 / 537 |
+| Doris | 1.0.0.7 | 262 / 542 |
+| Elasticsearch | 1.0.0.5 | 265 / 545 |
+| MinIO | 1.0.0.5 | 268 / 548 |
+
+All eight final checks reported COMPLETED and APPLIED with a matching task ID,
+canonical execution UUID and definition snapshot:
+`3f61f0944eab2ea5c74edf0a163c3811d63e7f8a3aed37d271ba82cc8d68bd62`.
+
+Elasticsearch and MinIO run in `mpack_five_generic_20260928`; the other BIGTOP
+services run in `mpack_kyuubi`. Airflow acceptance uses
+`mpack_airflow_clean_20260926`. The older stopped Airflow fixture was not
+started for this acceptance.
+
+## Browser And Runtime Evidence
+
+The browser visited all eight service configuration pages, checked the new
+fields and complete-file editors, and verified that custom-property categories
+exist only for extensible types. Every service had a representative setting
+changed through the UI and saved as a new version. Existing values were restored
+through the same UI and applied again.
+
+| Service | Change / restored value | Observation |
+| --- | --- | --- |
+| Kyuubi | Spark shuffle partitions 3 / 2 | Real JDBC SET query; session UUID and server operation UUID; separate execution UUID echoed by SQL |
+| Trino | Added memory catalog / removed it | SQL against system.metadata.catalogs returned 1 / 0; TPCH service check remained valid |
+| Doris | Slow-query threshold 6000 / 5000 ms | ADMIN SHOW FRONTEND CONFIG returned the expected Value |
+| Elasticsearch | search.max_buckets 65000 / 65536; added / deleted node.attr.acceptance | Authenticated HTTPS node-settings API, stable node ID, attribute present / absent |
+| MinIO | API request deadline 11s / 10s | Matching systemd service PID and its actual process environment |
+| Celeborn | Worker flusher threads 3 / 2 | Typed /api/v1/conf response on the assigned worker |
+| Airflow | Log level WARNING / INFO | Installed Airflow configuration API plus real health-DAG service check |
+| DolphinScheduler | Master execution threads 101 / 100 | Rendered application.properties plus authenticated API and master/worker registration check |
+
+DolphinScheduler's thread-pool size was verified at the configuration-file
+boundary; this run did not introspect the JVM thread pool or submit a real
+DolphinScheduler workflow. Airflow's configuration readback used the installed
+configuration library; its separate service check executed the health DAG.
+
+New-type default visibility, deleted-property preservation, history behavior,
+read-only contexts and forbidden custom categories also have frontend regression
+coverage. The live fixture's initial new types were seeded through the ordinary
+configuration API with existing values preserved; no database tables were edited.
+
+An initial Kyuubi check (request 224) failed during listener startup. The final
+pack adds a bounded listener wait before the existing identity-checked SQL
+operation. Subsequent checks, including the final immediate post-start check,
+passed. Draft definitions used during acceptance remain immutable; corrected
+definitions were imported at new version numbers.
+
+## Automated Verification
+
+Commands executed successfully:
+
+- `python3 -m unittest discover -s tests`: 77 tests passed on macOS.
+- `docker exec -e TMPDIR=/var/lib ambari-mpack-five-bigtop-20260928 /usr/bin/python3.11 -m unittest discover -s /mpacks/tests`: 77 tests passed on Rocky 8. The pinned Airflow constraints fixture was copied into the container before the final run; no final skips.
+- `npm test` in ambari-web/latest: 251 files and 1,409 tests passed.
+- `npm run build`: TypeScript and Vite production build passed. Existing Sass deprecation and bundle-size warnings remain.
+- Two complete eleven-pack store builds compared byte-for-byte using `cmp`.
+- Final configuration/Theme metadata coverage validates each declared property
+ placement, dependencies, and the service-level Theme declaration.
+
+The rendering tests cover managed-key conflicts, newline/NUL rejection, Unicode
+property escaping, literal environment exports, invalid XML/JVM input, duplicate
+or traversing catalog names, catalog removal, and retry after interrupted writes.
+
+## Local Artifacts
+
+Evidence, screenshots, immutable draft archives and final archives:
+`~/.local/share/ambari-mpack-acceptance/native-config-20260928/`.
+
+Final store:
+`final-b/mpackstore.bundle.tar.gz`
+
+SHA-256:
+`8f2eb9fba390df762f97c1e4cd3b010c01404f65c8a26c4238e5a595a5b2dbfb`
+
+Key evidence files include `browser.json`, `final-checks.json`,
+`*-runtime-changed.json`, `*-runtime-restored.json` and
+`*-ui-restore.json`. Private cluster configuration evidence remains outside Git.
+
+The existing prebuilt cluster was reused. Only frontend assets were replaced;
+management definitions were updated through the mpack API. No new Ambari RPM
+was built. See [the configuration guide](native-configuration.md) for supported
+files and the retained topology/authentication limits.
+
+Code and this conclusion are committed locally. The mpack store is not pushed.
diff --git a/docs/native-configuration.md b/docs/native-configuration.md
new file mode 100644
index 0000000..c8ab872
--- /dev/null
+++ b/docs/native-configuration.md
@@ -0,0 +1,100 @@
+<!--
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to You under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+http://www.apache.org/licenses/LICENSE-2.0
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# Native Configuration Management
+
+The Kyuubi, Airflow, Celeborn, DolphinScheduler, Trino, Doris, Elasticsearch and
+MinIO packs expose native configuration through Ambari. Basic settings remain
+compatible with earlier releases. Service-owned configuration types travel
+with each Agent task and are consumed by the runtime renderer.
+
+## Using Configs
+
+- **Basic Settings** contains installation paths, topology, listener ports,
+ credentials and the original memory settings.
+- **Native Settings** contains common application properties, with descriptions
+ and numeric or boolean controls where appropriate.
+- **Configuration Files** contains full logging documents, additional JVM
+ options, Trino catalogs and the Airflow FAB configuration.
+- **Advanced** accepts custom native properties for extensible configuration
+ types. File-content types accept their single content editor. Custom
+ environment settings use the separate environment type.
+
+Save a configuration version with a note, review any advisor warnings, and
+restart the affected service. Configuration saving does not itself restart a
+service. The installed service's native settings, configuration history and
+restart indicators remain Ambari's normal interfaces.
+
+For an existing installation, update the management definition first, reopen
+Configs and save the newly exposed defaults together with the existing values.
+Do not replace existing cluster configurations wholesale with XML defaults.
+The basic settings retain their names, so existing credentials, paths, ports
+and topology do not require renaming. Existing custom properties are preserved.
+
+## Managed Files
+
+| Service | Configuration types and behavior |
+| --- | --- |
+| Kyuubi | `kyuubi-defaults` writes every supplied native property; `kyuubi-environment` writes literal shell exports into `kyuubi-env.sh`; `kyuubi-log4j2` writes the complete logging file. JVM options are editable. |
+| Trino | `trino-config`, `trino-node`, `trino-log`, `trino-jvm` and `trino-catalogs` manage the corresponding files. Each catalog is a name-to-properties entry in the JSON editor. |
+| Doris | `doris-fe` and `doris-be` overlay the pinned vendor files. Replaced keys are removed before new values are written, preserving vendor comments and version-specific defaults. |
+| Elasticsearch | `elasticsearch-settings` writes YAML settings using quoted scalars and JSON-compatible arrays; `elasticsearch-jvm` writes an options fragment; `elasticsearch-log4j2` manages the complete vendor logging document. |
+| DolphinScheduler | Separate master, worker, API and alert types write Spring `application.properties` overrides beside vendor YAML. The common type overlays `common.properties`. Each role has its own full Logback editor; the runtime and JVM types configure process memory and options. |
+| MinIO | `minio-environment` supplies native `MINIO_*` variables to the running server through a protected runtime document. Basic credentials and listener arguments remain authoritative. |
+| Airflow | `airflow-settings` uses `section.option` keys to manage `airflow.cfg`; `airflow-webserver` manages the full FAB Python configuration. |
+| Celeborn | `celeborn-defaults` writes native properties; `celeborn-environment` manages additional exports; `celeborn-log4j2` manages logging. Master and worker assignments remain in basic settings. |
+
+Common fields are a curated subset for the pinned software versions. Additional
+upstream options use the custom-property editor; the package does not enumerate
+every connector, plugin or future upstream option. Arbitrary user-supplied
+options remain subject to the application's own validation.
+
+## Consistency And Recovery
+
+Configuration types are scoped to their service; unrelated cluster credentials
+are not forwarded to runtime helpers. Properties are validated and escaped,
+including Java property Unicode escapes. Structured action results retain
+execution identity and do not echo the input configuration.
+
+Basic topology, installation paths and service identity take precedence through
+explicit conflict rejection. Heap options for Trino and Elasticsearch belong
+to Basic Settings; their JVM editors accept additional options. DolphinScheduler
+database and registry settings use the existing basic fields. Invalid JVM input,
+malformed catalog JSON, duplicate catalog names and unsafe catalog paths fail.
+
+Trino records its owned catalog names before writing files, allowing retry
+after an interrupted write. It removes catalogs deleted from the editor and
+refuses to adopt pre-existing unowned files. The TPCH catalog is retained for
+the service check.
+
+Kyuubi waits for a reachable JDBC listener before attempting the SQL service
+check. A listening socket is only a prerequisite: success still requires the
+real SQL result and matching session, operation and execution identities.
+
+## Scope
+
+These configuration improvements retain the previously accepted single-node
+or small-cluster topology. HA, new authentication modes and additional runtime
+dependencies require their corresponding lifecycle support. For example,
+changing an arbitrary property does not provision an Airflow Celery broker,
+enable Kyuubi Kerberos or distribute Elasticsearch certificates.
+
+The complete store remains a transport bundle of management definitions.
+Software artifacts continue to use their pinned upstream sources. Published
+archive bytes are immutable; configuration changes ship as new definition
+versions. No RPM rebuild is required for these definition updates.
+
+Acceptance evidence and exact final versions are recorded in
+[native configuration acceptance](native-configuration-acceptance.md).
diff --git a/tests/test_native_metadata.py b/tests/test_native_metadata.py
new file mode 100644
index 0000000..b43edd4
--- /dev/null
+++ b/tests/test_native_metadata.py
@@ -0,0 +1,47 @@
+"""
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to You under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+http://www.apache.org/licenses/LICENSE-2.0
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+"""
+
+import json
+import unittest
+from pathlib import Path
+from xml.etree import ElementTree
+
+ROOT = Path(__file__).resolve().parents[1]
+
+
+def parameters(types):
+ return {"native_configs": {"schema_version": 1, "types": types}}
+
+
+
+class NativeMetadataTests(unittest.TestCase):
+ def test_metadata_declares_every_type_and_theme_property(self):
+ for name in ("kyuubi", "trino", "doris", "elasticsearch", "minio", "airflow", "celeborn", "dolphinscheduler"):
+ base = ROOT / "mpacks" / name / "extensions" / name.upper() / "1.0/services" / name.upper()
+ types = {p.stem: ElementTree.parse(p).getroot() for p in (base / "configuration").glob("*.xml")}
+ metadata = ElementTree.parse(base / "metainfo.xml")
+ self.assertEqual(metadata.findtext("services/service/themes/theme/fileName"), "theme.json")
+ self.assertTrue(all(not list(e) for e in metadata.findall("services/service/requiredServices/service")))
+ declared = {e.text for e in ElementTree.parse(base / "metainfo.xml").findall(".//configuration-dependencies/config-type")}
+ self.assertTrue(set(types) <= declared)
+ props = {kind + "/" + p.findtext("name") for kind, root in types.items() for p in root.findall("property")}
+ theme = json.loads((base / "themes/theme.json").read_text())["configuration"]
+ placed = [p["config"] for p in theme["placement"]["configs"]]
+ self.assertEqual(set(placed), props)
+ self.assertEqual(len(placed), len(set(placed)))
+
+
+if __name__ == "__main__":
+ unittest.main()
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@ambari.apache.org
For additional commands, e-mail: commits-help@ambari.apache.org