You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@knox.apache.org by "samuhasavak5 (via GitHub)" <gi...@apache.org> on 2026/09/17 19:29:23 UTC
[PR] KNOX-3474: Add spark4historyui Knox service definition [knox]
samuhasavak5 opened a new pull request, #1413:
URL: https://github.com/apache/knox/pull/1413
KNOX-3474 - Add spark4historyui Knox service definition
Depends on #1411, #1412
## What changes were proposed in this pull request?
This PR adds upstream Knox support for the dedicated Spark 4 History Server Web UI (role: SPARK4HISTORYUI).
Problem: Knox has no service definition for Spark 4 History Server. Deployments running a separate Spark 4 History Server cannot be proxied or discovered through Knox, YARN UI links to Spark 4 history jobs do not route correctly, and the Knox Homepage cannot show a dedicated Spark 4 History tile.
Solution: Introduce spark4historyui by mirroring the spark3historyui service definition with Spark 4 naming, CM discovery, admin UI registration, homepage logo, YARN rewrite rules, and unit tests.
### Changes
| File | Change |
|------|--------|
| gateway-service-definitions/.../spark4historyui/3.0.0/service.xml | New service definition (metadata, routes) |
| gateway-service-definitions/.../spark4historyui/3.0.0/rewrite.xml | Inbound/outbound rewrite rules for REST API and UI |
| gateway-discovery-cm/.../Spark4HistoryUIServiceModelGenerator.java | CM discovery for SPARK4HISTORYUI |
| gateway-discovery-cm/.../Spark4HistoryUIServiceModelGeneratorTest.java | Unit tests for model generator |
| gateway-discovery-cm/.../ServiceModelGenerator (SPI) | Register Spark4HistoryUIServiceModelGenerator |
| ClouderaManagerServiceDiscoveryTest.java | Add testSpark4HistoryUIDiscovery and testSpark4HistoryUIDiscoverySSL |
| gateway-service-definitions/.../yarnui/2.7.0/rewrite.xml | Route ?spark4=true history links to /spark4history/ |
| gateway-service-definitions/.../yarnuiv2/3.0.0/rewrite.xml | Same YARN rewrite rule |
| gateway-admin-ui/.../new-desc-wizard.component.ts | Add SPARK4HISTORYUI to descriptor wizard |
| knox-homepage-ui/home/assets/service-logos/spark4historyui.png | Homepage logo for service role SPARK4HISTORYUI |
### Service details
| Item | Value |
|------|-------|
| Service role | SPARK4HISTORYUI |
| Service name / path | spark4history |
| CM service type | SPARK4_ON_YARN |
| CM role type | SPARK4_YARN_HISTORY_SERVER |
| Metadata type | UI |
| Homepage context | /spark4history/ |
### Design notes
* Mirrors spark3historyui with Spark 4 identifiers only
* Additive only — no changes to spark3historyui or sparkhistoryui service definitions
* YARN rewrite — spark4=true query param routes to spark4history (parallel to CDH spark3=true → spark3history pattern)
* Homepage logo — spark4historyui.png (same Spark branding as Spark 3 History)
### Compatibility
* No changes to existing sparkhistoryui or spark3historyui behavior
* New service is opt-in via topology / CM discovery configuration
* Backward compatible for clusters that do not deploy Spark 4 History Server
JIRA: https://issues.apache.org/jira/browse/KNOX-3474
## How was this patch tested?
### Build
mvn -s mvn-settings-central.xml -U -pl gateway-service-definitions,gateway-discovery-cm -am test -DskipITs
Result: BUILD SUCCESS
### Unit tests
| Test | Coverage |
|------|----------|
| Spark4HistoryUIServiceModelGeneratorTest | CM model generator metadata and URL generation |
| ClouderaManagerServiceDiscoveryTest.testSpark4HistoryUIDiscovery | HTTP discovery for SPARK4HISTORYUI |
| ClouderaManagerServiceDiscoveryTest.testSpark4HistoryUIDiscoverySSL | HTTPS discovery for SPARK4HISTORYUI |
| Full reactor (gateway-service-definitions, gateway-discovery-cm + dependencies) | 351 tests passed, 0 failures |
### Manual verification
* Knox Homepage on spark4knoxui cluster shows "Spark 4 History Server Web UI" tile with logo
* Tile click opens Spark 4 History UI through Knox gateway path /spark4history/
* YARN UI history links with spark4=true route to /spark4history/history/
## Integration Tests
No new integration test workflow added. Coverage is provided by existing unit tests in gateway-discovery-cm and gateway-service-definitions.
## UI changes
Yes — Knox Homepage shows a new Spark 4 History Server tile with logo when SPARK4HISTORYUI is configured in the topology.
## Was this patch authored or co-authored using generative AI tooling?
Yes, this was co-authored using Cursor to help port the service definition from spark3historyui, discovery generator, tests, YARN rewrite rules, and PR description.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@knox.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KNOX-3474: Add spark4historyui Knox service definition [knox]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #1413:
URL: https://github.com/apache/knox/pull/1413#issuecomment-5726405908
## Test Results
4 files 4 suites 25s ⏱️
98 tests 98 ✅ 0 💤 0 ❌
116 runs 116 ✅ 0 💤 0 ❌
Results for commit ff441687.
[test-results]:data:application/gzip;base64,H4sIAHPhrGoC/13MyQ6CMBSF4VchXbug0On6MqZj0gjUdFgR391CAKvL7z/JWZHzk03o3pFbh1Lx+YIpUWYflsqBVtclbxuIE49UtP4rT/+qpb+Ck376CTbGEI8Sy7I9YswOnI9t+T7ubg53t386zLPPFcg5QjAT3IAaRgnAAKzhanSul2CF5FpRpqhG7w+wvkYZAgEAAA==
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@knox.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KNOX-3474: Add spark4historyui Knox service definition [knox]
Posted by "samuhasavak5 (via GitHub)" <gi...@apache.org>.
samuhasavak5 commented on PR #1413:
URL: https://github.com/apache/knox/pull/1413#issuecomment-5720235448
Cluster validation: spark4knoxui — Jenkins #11582 SUCCESS; Spark 4 History Server homepage tile verified.
https://master-06.jenkins.cloudera.com/job/Cluster-Setup-CDPD/11582/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@knox.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KNOX-3474: Add spark4historyui Knox service definition [knox]
Posted by "samuhasavak5 (via GitHub)" <gi...@apache.org>.
samuhasavak5 commented on PR #1413:
URL: https://github.com/apache/knox/pull/1413#issuecomment-5728780685
> @samuhasavak5
>
> I took a quick look at this PR and have a few quick comments, could you please update this PR with the following changes?
>
> * Remove LIVY_FOR_SPARK3 and LIVY_FOR_SPARK4 modifications so this PR strictly contains SPARK4HISTORYUI.
> * Use directory name 4.0.0 instead of 3.0.0 here: gateway-service-definitions/src/main/resources/services/spark4historyui/4.0.0/
Thanks for the review
1. Rebased onto upstream/master so this PR contains only the KNOX-3474 / SPARK4HISTORYUI commit (removed stacked LIVY_FOR_SPARK3 and LIVY_FOR_SPARK4 changes; those remain in #1411 and #1412).
2. Moved spark4historyui to gateway-service-definitions/.../spark4historyui/4.0.0/ and set service version to 4.0.0.
Please take another look when you have a moment
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@knox.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KNOX-3474: Add spark4historyui Knox service definition [knox]
Posted by "samuhasavak5 (via GitHub)" <gi...@apache.org>.
samuhasavak5 commented on code in PR #1413:
URL: https://github.com/apache/knox/pull/1413#discussion_r4061097512
##########
gateway-discovery-cm/src/test/java/org/apache/knox/gateway/topology/discovery/cm/model/spark/Spark4HistoryUIServiceModelGeneratorTest.java:
##########
@@ -0,0 +1,54 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership. The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations under
+ * the License.
+ */
+package org.apache.knox.gateway.topology.discovery.cm.model.spark;
+
+import org.apache.knox.gateway.topology.discovery.cm.ServiceModelGenerator;
+import org.apache.knox.gateway.topology.discovery.cm.model.AbstractServiceModelGeneratorTest;
+import org.junit.Test;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Map;
+
+public class Spark4HistoryUIServiceModelGeneratorTest extends AbstractServiceModelGeneratorTest {
+
+ @Test
+ public void testServiceModelMetadata() {
+ final Map<String, String> serviceConfig = Collections.emptyMap();
+ final Map<String, String> roleConfig = new HashMap<>();
+ roleConfig.put(Spark4HistoryUIServiceModelGenerator.SSL_ENABLED, "false");
+ roleConfig.put(Spark4HistoryUIServiceModelGenerator.HISTORY_SERVER_PORT, "18089");
Review Comment:
Done, please check
##########
gateway-discovery-cm/src/test/java/org/apache/knox/gateway/topology/discovery/cm/model/spark/Spark4HistoryUIServiceModelGeneratorTest.java:
##########
@@ -0,0 +1,54 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership. The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations under
+ * the License.
+ */
+package org.apache.knox.gateway.topology.discovery.cm.model.spark;
+
+import org.apache.knox.gateway.topology.discovery.cm.ServiceModelGenerator;
+import org.apache.knox.gateway.topology.discovery.cm.model.AbstractServiceModelGeneratorTest;
+import org.junit.Test;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Map;
+
+public class Spark4HistoryUIServiceModelGeneratorTest extends AbstractServiceModelGeneratorTest {
+
+ @Test
+ public void testServiceModelMetadata() {
+ final Map<String, String> serviceConfig = Collections.emptyMap();
+ final Map<String, String> roleConfig = new HashMap<>();
+ roleConfig.put(Spark4HistoryUIServiceModelGenerator.SSL_ENABLED, "false");
+ roleConfig.put(Spark4HistoryUIServiceModelGenerator.HISTORY_SERVER_PORT, "18089");
+ roleConfig.put(Spark4HistoryUIServiceModelGenerator.SSL_SERVER_PORT, "18489");
Review Comment:
Done, please check
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@knox.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KNOX-3474: Add spark4historyui Knox service definition [knox]
Posted by "roczei (via GitHub)" <gi...@apache.org>.
roczei commented on code in PR #1413:
URL: https://github.com/apache/knox/pull/1413#discussion_r4060574551
##########
gateway-discovery-cm/src/test/java/org/apache/knox/gateway/topology/discovery/cm/model/spark/Spark4HistoryUIServiceModelGeneratorTest.java:
##########
@@ -0,0 +1,54 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership. The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations under
+ * the License.
+ */
+package org.apache.knox.gateway.topology.discovery.cm.model.spark;
+
+import org.apache.knox.gateway.topology.discovery.cm.ServiceModelGenerator;
+import org.apache.knox.gateway.topology.discovery.cm.model.AbstractServiceModelGeneratorTest;
+import org.junit.Test;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Map;
+
+public class Spark4HistoryUIServiceModelGeneratorTest extends AbstractServiceModelGeneratorTest {
+
+ @Test
+ public void testServiceModelMetadata() {
+ final Map<String, String> serviceConfig = Collections.emptyMap();
+ final Map<String, String> roleConfig = new HashMap<>();
+ roleConfig.put(Spark4HistoryUIServiceModelGenerator.SSL_ENABLED, "false");
+ roleConfig.put(Spark4HistoryUIServiceModelGenerator.HISTORY_SERVER_PORT, "18089");
+ roleConfig.put(Spark4HistoryUIServiceModelGenerator.SSL_SERVER_PORT, "18489");
Review Comment:
It's a similar situation for port 18489, could you change that to 18490 as well?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@knox.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KNOX-3474: Add spark4historyui Knox service definition [knox]
Posted by "roczei (via GitHub)" <gi...@apache.org>.
roczei commented on code in PR #1413:
URL: https://github.com/apache/knox/pull/1413#discussion_r4060573109
##########
gateway-discovery-cm/src/test/java/org/apache/knox/gateway/topology/discovery/cm/model/spark/Spark4HistoryUIServiceModelGeneratorTest.java:
##########
@@ -0,0 +1,54 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership. The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations under
+ * the License.
+ */
+package org.apache.knox.gateway.topology.discovery.cm.model.spark;
+
+import org.apache.knox.gateway.topology.discovery.cm.ServiceModelGenerator;
+import org.apache.knox.gateway.topology.discovery.cm.model.AbstractServiceModelGeneratorTest;
+import org.junit.Test;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.Map;
+
+public class Spark4HistoryUIServiceModelGeneratorTest extends AbstractServiceModelGeneratorTest {
+
+ @Test
+ public void testServiceModelMetadata() {
+ final Map<String, String> serviceConfig = Collections.emptyMap();
+ final Map<String, String> roleConfig = new HashMap<>();
+ roleConfig.put(Spark4HistoryUIServiceModelGenerator.SSL_ENABLED, "false");
+ roleConfig.put(Spark4HistoryUIServiceModelGenerator.HISTORY_SERVER_PORT, "18089");
Review Comment:
Port 18089 is currently being used for spark3 testing, could you please change it to 18090?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@knox.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KNOX-3474: Add spark4historyui Knox service definition [knox]
Posted by "samuhasavak5 (via GitHub)" <gi...@apache.org>.
samuhasavak5 commented on PR #1413:
URL: https://github.com/apache/knox/pull/1413#issuecomment-5726670792
@roczei, @gyogal and @ArnavBalyan — could you please review the upstream Knox PRs when you get a chance?
JIRA comments are updated with build and spark4knoxui validation details.
Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@knox.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KNOX-3474: Add spark4historyui Knox service definition [knox]
Posted by "roczei (via GitHub)" <gi...@apache.org>.
roczei commented on PR #1413:
URL: https://github.com/apache/knox/pull/1413#issuecomment-5728543633
@samuhasavak5
I took a quick look at this PR and have a few quick comments, could you please update this PR with the following changes?
- Remove LIVY_FOR_SPARK3 and LIVY_FOR_SPARK4 modifications so this PR strictly contains SPARK4HISTORYUI.
- Use directory name 4.0.0 instead of 3.0.0 here: gateway-service-definitions/src/main/resources/services/spark4historyui/4.0.0/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@knox.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KNOX-3474: Add spark4historyui Knox service definition [knox]
Posted by "smolnar82 (via GitHub)" <gi...@apache.org>.
smolnar82 merged PR #1413:
URL: https://github.com/apache/knox/pull/1413
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@knox.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org