You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@knox.apache.org by "ASF GitHub Bot (Jira)" <ji...@apache.org> on 2020/03/10 15:21:00 UTC
[jira] [Work logged] (KNOX-2284) CM descriptors do not get picked
up at Gateway start
[ https://issues.apache.org/jira/browse/KNOX-2284?focusedWorklogId=400764&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-400764 ]
ASF GitHub Bot logged work on KNOX-2284:
----------------------------------------
Author: ASF GitHub Bot
Created on: 10/Mar/20 15:20
Start Date: 10/Mar/20 15:20
Worklog Time Spent: 10m
Work Description: smolnar82 commented on pull request #285: KNOX-2284 - Handling CM descriptors after Knox shared-provider/descriptor/topology monitors are started
URL: https://github.com/apache/knox/pull/285
## What changes were proposed in this pull request?
Handling CM descriptors after Knox shared-provider/descriptor/topology monitors are started
## How was this patch tested?
Tested manually:
1. redeployed Knox with my changes in place; brand new installation, only the OOTB topologies existed
2. produced a CM descriptor (cm-topologies.cm) with the following content:
```
<configuration>
<property>
<name>topology1</name>
<value>
providerConfigRef=default-providers;
app:knoxauth:param1.name=param1.value;
HIVE:url=http://localhost:456;
HIVE:version=1.0;
HIVE:httpclient.connectionTimeout=5m;
HIVE:httpclient.socketTimeout=100m
</value>
</property>
<property>
<name>topology2</name>
<value>
providerConfigRef=default-providers;
HDFS:url=http://localhost:456;
HDFS:httpclient.connectionTimeout=5m;
HDFS:httpclient.socketTimeout=100m
</value>
</property>
</configuration>
```
3. started Knox and confirmed (from logs and using the Admin UI):
- `$KNOX_HOME/conf/descriptors/topology1.json` and `$KNOX_HOME/conf/descriptors/topology2.json` got persisted
- Knox picked them up and generated `$KNOX_HOME/data/deployments/topology1.topo.170c48f42c0` and `$KNOX_HOME/data/deployments/topology2.topo.170c48f42c0`
Relevant logs:
```
2020-03-10 14:07:50,348 INFO knox.gateway (DefaultTopologyService.java:init(648)) - Monitoring simple descriptors in directory: /Users/smolnar/test/knoxGateway/conf/descriptors
2020-03-10 14:07:50,577 INFO knox.gateway (DefaultTopologyService.java:init(653)) - Monitoring shared provider configurations in directory: /Users/smolnar/test/knoxGateway/conf/shared-providers
...
2020-03-10 14:08:02,899 INFO knox.gateway (GatewayServer.java:start(700)) - Monitoring topologies in directory: /Users/smolnar/test/knoxGateway/conf/topologies
2020-03-10 14:08:02,920 INFO discovery.advanced (AdvancedServiceDiscoveryConfigurationMonitor.java:setupMonitor(73)) - Monitoring /Users/smolnar/test/knoxGateway/conf/auto-discovery-advanced-configuration-* for advanced service discovery configuration changes.
2020-03-10 14:08:02,925 INFO knox.gateway (ClouderaManagerDescriptorMonitor.java:setupMonitor(66)) - Monitoring Cloudera Manager descriptors in /Users/smolnar/test/knoxGateway/conf/descriptors ...
2020-03-10 14:08:02,925 INFO knox.gateway (ClouderaManagerDescriptorParser.java:parse(79)) - Parsing Cloudera Manager descriptor /Users/smolnar/test/knoxGateway/conf/descriptors/cm-topologies.cm. Looking up all topologies...
2020-03-10 14:08:02,925 INFO knox.gateway (GatewayServer.java:startGateway(392)) - Started gateway on port 8,443.
2020-03-10 14:08:02,936 INFO knox.gateway (ClouderaManagerDescriptorParser.java:parse(84)) - Found Knox descriptors topology1, topology2 in /Users/smolnar/test/knoxGateway/conf/descriptors/cm-topologies.cm
2020-03-10 14:08:03,125 INFO knox.gateway (ClouderaManagerDescriptorMonitor.java:lambda$1(100)) - Saved Knox descriptor /Users/smolnar/test/knoxGateway/conf/descriptors/topology1.json
2020-03-10 14:08:03,125 INFO knox.gateway (ClouderaManagerDescriptorMonitor.java:lambda$1(100)) - Saved Knox descriptor /Users/smolnar/test/knoxGateway/conf/descriptors/topology2.json
2020-03-10 14:08:08,011 INFO knox.gateway (DefaultTopologyService.java:onFileChange(872)) - Generated topology topology1.xml because the associated descriptor topology1.json changed.
...
2020-03-10 14:08:14,278 INFO knox.gateway (GatewayServer.java:handleCreateDeployment(1013)) - Deploying topology topology1 to /Users/smolnar/test/knoxGateway/data/deployments/topology1.topo.170c48f42c0
2020-03-10 14:08:14,278 INFO knox.gateway (GatewayServer.java:internalDeactivateTopology(932)) - Deactivating topology topology1
2020-03-10 14:08:14,450 INFO knox.gateway (DefaultGatewayServices.java:initializeContribution(161)) - Creating credential store for the cluster: topology1
2020-03-10 14:08:14,753 INFO knox.gateway (GatewayServer.java:internalActivateTopology(898)) - Activating topology topology1
2020-03-10 14:08:14,753 INFO knox.gateway (GatewayServer.java:internalActivateArchive(908)) - Activating topology topology1 archive %2Fknoxauth
2020-03-10 14:08:14,833 INFO knox.gateway (GatewayServer.java:handleCreateDeployment(1013)) - Deploying topology topology2 to /Users/smolnar/test/knoxGateway/data/deployments/topology2.topo.170c48f42c0
2020-03-10 14:08:14,833 INFO knox.gateway (GatewayServer.java:internalDeactivateTopology(932)) - Deactivating topology topology2
2020-03-10 14:08:14,940 INFO knox.gateway (GatewayServer.java:internalActivateTopology(898)) - Activating topology topology2
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Issue Time Tracking
-------------------
Worklog Id: (was: 400764)
Remaining Estimate: 0h
Time Spent: 10m
> CM descriptors do not get picked up at Gateway start
> ----------------------------------------------------
>
> Key: KNOX-2284
> URL: https://issues.apache.org/jira/browse/KNOX-2284
> Project: Apache Knox
> Issue Type: Bug
> Components: Server
> Affects Versions: 1.4.0
> Reporter: Sandor Molnar
> Assignee: Sandor Molnar
> Priority: Major
> Fix For: 1.4.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> When there is a CM descriptor in the KNOX_DESCRIPTORS_DIR at startup time and the KNOX descriptors found in this CM descriptor did not exist before then Knox does not pick up them.
> The reason is: the shared-provider/descriptor/topology monitor starts after the CM descriptor is being parsed and the Knox descriptors are persisted.
> Recommended solution: move the CM descriptor handling after the monitors are started.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)