You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@knox.apache.org by "ASF GitHub Bot (Jira)" <ji...@apache.org> on 2020/01/10 18:09:00 UTC
[jira] [Work logged] (KNOX-2160) Introduce a Hadoop XML type
descriptor format
[ https://issues.apache.org/jira/browse/KNOX-2160?focusedWorklogId=369960&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-369960 ]
ASF GitHub Bot logged work on KNOX-2160:
----------------------------------------
Author: ASF GitHub Bot
Created on: 10/Jan/20 18:08
Start Date: 10/Jan/20 18:08
Worklog Time Spent: 10m
Work Description: smolnar82 commented on pull request #236: KNOX-2160 - Introducing Hadoop XML type descriptor format
URL: https://github.com/apache/knox/pull/236
## What changes were proposed in this pull request?
Implemented a new descriptor type - using Hadoop XML configuration - as described in [KNOX-2160 ](https://issues.apache.org/jira/browse/KNOX-2160).
Two new Maven modules were created:
- `gateway-cm-integration` - Clouder Manager integration-related resources should be added here. The first step was to add the new XML descriptor processing here.
- `gateway-topology-simple` - simple topology generation related resources were moved here from `gateway-server` So that the new `gateway-cm-integration` can see them too.
## How was this patch tested?
Added and updated JUnit tests then executed them:
```
$ mvn clean -Dshellcheck=true -T1C verify -Prelease,package
...
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 20:06 min (Wall Clock)
[INFO] Finished at: 2020-01-10T19:02:35+01:00
[INFO] Final Memory: 385M/2054M
[INFO] ------------------------------------------------------------------------
```
Additionally, the following E2E testing took place locally:
1. deployed and started Knox with my changes
2. created `topology1-rpvider` and `topology2-provider` provider configurations
3. saved the following Hadoop XML configuration in Knox's descriptor directory as `testDescriptor.xml`:
```
<configuration>
<property>
<name>topology1</name>
<value>
discoveryType=ClouderaManager;
discoveryAddress=http://host:123;
discoveryUser=user;
discoveryPasswordAlias=alias;
cluster=Cluster 1;
providerConfigRef=topology1-provider;
app:knoxauth:param1.name=param1.value;
HIVE:url=http://localhost:456;
HIVE:version=1.0;
HIVE:httpclient.connectionTimeout=5m;
HIVE:httpclient.socketTimeout=100m
</value>
</property>
<property>
<name>topology2</name>
<value>
providerConfigRef=topology2-provider;
HDFS:url=http://localhost:456;
HDFS:httpclient.connectionTimeout=5m;
HDFS:httpclient.socketTimeout=100m
</value>
</property>
</configuration>
```
4. Confirmed that
- descriptor files `topology1.json` and `topology2.json` created within Knox's descriptor folder
- topology generation finished successfully (based on log messages and Admin UI)
- `topology1` and `topology2` descriptors appeared on Admin UI as well as the corresponding topologies
Relevant `gateway.log` messages:
```
2020-01-10 17:28:45,359 INFO knox.gateway (XmlDescriptorParser.java:parse(53)) - Parsing XML descriptor /Users/smolnar/test/knoxGateway/conf/descriptors/testDescriptor.xml
2020-01-10 17:28:45,361 INFO knox.gateway (XmlDescriptorParser.java:parse(59)) - Found descriptors topology1, topology2 in /Users/smolnar/test/knoxGateway/conf/descriptors/testDescriptor.xml
2020-01-10 17:28:45,367 ERROR discovery.cm (DiscoveryApiClient.java:configure(106)) - No password configured for Cloudera Manager service discovery.
2020-01-10 17:28:45,367 INFO discovery.cm (ClouderaManagerServiceDiscovery.java:discoverCluster(170)) - Performing cluster discovery for "Cluster 1"
2020-01-10 17:28:45,369 ERROR discovery.cm (ClouderaManagerServiceDiscovery.java:getClusterServices(218)) - Failed to access the service configurations for cluster (Cluster 1) discovery: com.cloudera.api.swagger.client.ApiException: java.net.UnknownHostException: host: nodename nor servname provided, or not known
2020-01-10 17:28:45,475 INFO knox.gateway (DefaultTopologyService.java:onFileChange(868)) - Generated topology topology1.xml because the associated descriptor topology1.json changed.
2020-01-10 17:28:45,478 ERROR discovery.ambari (AmbariServiceDiscovery.java:discover(237)) - No address for Ambari service discovery has been configured.
2020-01-10 17:28:45,478 ERROR discovery.ambari (AmbariServiceDiscovery.java:discover(247)) - No cluster for Ambari service discovery has been configured.
2020-01-10 17:28:45,480 ERROR topology.simple (SimpleDescriptorHandler.java:handle(107)) - Unable to complete service discovery for cluster topology2.
2020-01-10 17:28:45,586 INFO knox.gateway (DefaultTopologyService.java:onFileChange(868)) - Generated topology topology2.xml because the associated descriptor topology2.json changed.
2020-01-10 17:28:46,321 INFO knox.gateway (GatewayServer.java:handleCreateDeployment(998)) - Deploying topology topology1 to /Users/smolnar/test/knoxGateway/data/deployments/topology1.topo.16f90495e48
2020-01-10 17:28:46,322 INFO knox.gateway (GatewayServer.java:internalDeactivateTopology(917)) - Deactivating topology topology1
2020-01-10 17:28:46,419 INFO knox.gateway (DefaultGatewayServices.java:initializeContribution(164)) - Credential store found for the cluster: topology1 - no need to create one.
2020-01-10 17:28:46,707 INFO knox.gateway (GatewayServer.java:internalActivateTopology(883)) - Activating topology topology1
2020-01-10 17:28:46,707 INFO knox.gateway (GatewayServer.java:internalActivateArchive(893)) - Activating topology topology1 archive %2Fknoxauth
2020-01-10 17:28:46,762 INFO knox.gateway (GatewayServer.java:handleCreateDeployment(998)) - Deploying topology topology2 to /Users/smolnar/test/knoxGateway/data/deployments/topology2.topo.16f90495e48
2020-01-10 17:28:46,762 INFO knox.gateway (GatewayServer.java:internalDeactivateTopology(917)) - Deactivating topology topology2
2020-01-10 17:28:46,850 INFO knox.gateway (GatewayServer.java:internalActivateTopology(883)) - Activating topology topology2
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Issue Time Tracking
-------------------
Worklog Id: (was: 369960)
Time Spent: 1.5h (was: 1h 20m)
> Introduce a Hadoop XML type descriptor format
> ---------------------------------------------
>
> Key: KNOX-2160
> URL: https://issues.apache.org/jira/browse/KNOX-2160
> Project: Apache Knox
> Issue Type: New Feature
> Components: Server
> Affects Versions: 1.1.0, 1.2.0, 1.3.0
> Reporter: Sandor Molnar
> Assignee: Sandor Molnar
> Priority: Major
> Fix For: 1.4.0
>
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> To support topology management in Cloudera Manager it'd be beneficial if Knox was able to process a descriptor that CM can generate natively. As of now, CM's CSD framework is capable of producing a file of its parameters in the following [formats|https://github.com/cloudera/cm_ext/wiki/Service-Descriptor-Language-Reference#configWriterFormat]:
> * Hadoop XML
> * properties
> * gflags
> As the {{gateway-site.xml}} is a Hadoop XML it's quite obvious that the first option is the one that fits the most.
> One XML type descriptor file may contain one or more Knox descriptors using the following structure:
> * the configuration name would indicate the descriptor (topology) name
> * the configuration value would list all properties of a Knox descriptor
> ** service discovery related information (type, address, cluster, user/password alias)
> ** services
> *** name
> *** url
> *** version (optional)
> *** parameters (optional)
> ** applications (optional)
> *** name
> *** parameters (optional)
> A sample descriptor file would look like this:
> {code:xml}
> <configuration>
> <property>
> <name>topology1</name>
> <value>
> discoveryType=ClouderaManager;
> discoveryAddress=http://host:123;
> discoveryUser=user;
> discoveryPasswordAlias=alias;
> cluster=Cluster 1;
> providerConfigRef=topology1-provider;
> app:knoxauth:param1.name=param1.value;
> app:KNOX;
> HIVE:url=http://localhost:389;
> HIVE:version=1.0;
> HIVE:httpclient.connectionTimeout=5m;
> HIVE:httpclient.socketTimeout=100m
> </value>
> </property>
> <property>
> <name>topology2</name>
> <value>
> discoveryType=ClouderaManager;
> discoveryAddress=http://host:123;
> discoveryUser=user;
> discoveryPasswordAlias=alias;
> cluster=Cluster 1;
> providerConfigRef=topology2-provider;
> app:KNOX;
> HDFS.url=https://localhost:443;
> HDFS:httpclient.connectionTimeout=5m;
> HDFS:httpclient.socketTimeout=100m
> </value>
> </property>
> </configuration>
> {code}
>
> Workflow:
> # this kind of descriptor should also be placed in Knox's descriptor directory
> # once it's added or modified Knox's existing descriptor monitor should parse the XML and build one or more instance sof {{org.apache.knox.gateway.topology.simple.SimpleDescriptor}}
> # after the Java object(s) got created it (they) should be saved in the Knox descriptor directory in JSON format. As a result, the same monitor should parse the new/modified JSON descriptor(s) and re-deploys it (them) using the already existing mechanism
--
This message was sent by Atlassian Jira
(v8.3.4#803005)