You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@nifi.apache.org by kevdoran <gi...@git.apache.org> on 2018/03/16 19:58:40 UTC
[GitHub] nifi-minifi pull request #118: MINIFI-444 C2 Data Model and REST API
GitHub user kevdoran opened a pull request:
https://github.com/apache/nifi-minifi/pull/118
MINIFI-444 C2 Data Model and REST API
Adds the foundation of a data model and REST API that would support
discussed features such as a MiNiFi Agent C2 Protocol and
MiNiFi Flow Designer.
Thank you for submitting a contribution to Apache NiFi - MiNiFi.
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?
- [ ] Does your PR title start with MINIFI-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.
- [ ] Has your PR been rebased against the latest commit within the target branch (typically master)?
- [ ] Is your initial contribution a single, squashed commit?
### For code changes:
- [ ] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi-minifi folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under minifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under minifi-assembly?
### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in which it is rendered?
### Note:
Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/kevdoran/nifi-minifi MINIFI-444-c2-data-model
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/nifi-minifi/pull/118.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #118
----
commit b765407b29a8dc8876cff340275f1937e28633c8
Author: Kevin Doran <kd...@...>
Date: 2018-02-27T19:09:50Z
MINIFI-444 C2 Data Model and REST API
The foundation of a data model and REST API that would support
discussed features such as a MiNiFi Agent C2 Protocol and
MiNiFi Flow Designer.
----
---
[GitHub] nifi-minifi pull request #118: MINIFI-444 C2 Data Model and REST API
Posted by phrocker <gi...@git.apache.org>.
Github user phrocker commented on a diff in the pull request:
https://github.com/apache/nifi-minifi/pull/118#discussion_r175201509
--- Diff: minifi-c2/minifi-c2-commons/src/main/java/org/apache/nifi/minifi/c2/model/AgentRepositoryStatus.java ---
@@ -0,0 +1,63 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.minifi.c2.model;
+
+import io.swagger.annotations.ApiModel;
+import io.swagger.annotations.ApiModelProperty;
+
+@ApiModel
+public class AgentRepositoryStatus {
+
+ private Long size;
--- End diff --
keep in mind that java is a 64-bit two's complement so you could get values here that exceed the max. Should either comment re this or use Long's unsigned comparator methods.
---
[GitHub] nifi-minifi pull request #118: MINIFI-444 C2 Data Model and REST API
Posted by apiri <gi...@git.apache.org>.
Github user apiri commented on a diff in the pull request:
https://github.com/apache/nifi-minifi/pull/118#discussion_r176267863
--- Diff: minifi-c2/minifi-c2-commons/src/main/java/org/apache/nifi/minifi/c2/model/FlowUri.java ---
@@ -0,0 +1,54 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.minifi.c2.model;
+
+import io.swagger.annotations.ApiModel;
+import io.swagger.annotations.ApiModelProperty;
+
+@ApiModel("Uniform Resource Identifier for NiFi Versioned Flows saved to a NiFi Registry")
+public class FlowUri {
--- End diff --
Not sure of how this is represented by Registry but is there an actual URI form?
https://registry.myorganization.org/nifi-registry/<bucket>/<flow>?
---
[GitHub] nifi-minifi pull request #118: MINIFI-444 C2 Data Model and REST API
Posted by kevdoran <gi...@git.apache.org>.
Github user kevdoran commented on a diff in the pull request:
https://github.com/apache/nifi-minifi/pull/118#discussion_r175503881
--- Diff: minifi-c2/minifi-c2-commons/src/main/java/org/apache/nifi/minifi/c2/model/AgentManifest.java ---
@@ -0,0 +1,82 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.minifi.c2.model;
+
+import io.swagger.annotations.ApiModel;
+import io.swagger.annotations.ApiModelProperty;
+import org.apache.nifi.minifi.c2.model.extension.Bundle;
+
+import java.util.List;
+
+@ApiModel
+public class AgentManifest {
+
+ private String identifier;
--- End diff --
No, this is a new identifier that will only be used by the C2 server. It assumes the agent manifests are going to be persistable and reusable. It probably would not be present when an agent manifest is being uploaded to a C2 server (e.g., in a C2 heartbeat from a MiNiFi agent or as the output of a MiNiFi agent build that some build script uploads), but when the C2 server persists it, it will serve as a unique ID that can be used to retrieve that agent manifest again.
The primary use case would be when using a Flow Designer in NiFi, the user chooses an agent class name from a list. That selection then returns one or more agent manifests that are associated with that class name. So the way I am picturing it the mapping of agent to agent manifest depends on context: from the perspecitive of an agent instance, it's going to be 1-to-1 where the manifest is always what it currently consists of. from the c2 server's perspective, it will be 1-to-many, where it is keeping track of all manifests for a given class name based on what has been uploaded to it.
These are still very new concepts, so if you have an input to help formulate them that would be welcome! This is just an initial idea and I tried to represent it the way that will support this use case. I'm sure it will change a bit before the minifi-c2-server branch could be considered "done" and merged to master.
---
[GitHub] nifi-minifi pull request #118: MINIFI-444 C2 Data Model and REST API
Posted by phrocker <gi...@git.apache.org>.
Github user phrocker commented on a diff in the pull request:
https://github.com/apache/nifi-minifi/pull/118#discussion_r175207806
--- Diff: minifi-c2/minifi-c2-web-api/src/main/java/org/apache/nifi/minifi/c2/web/api/C2ProtocolResource.java ---
@@ -0,0 +1,119 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.minifi.c2.web.api;
+
+import io.swagger.annotations.Api;
+import io.swagger.annotations.ApiOperation;
+import io.swagger.annotations.ApiParam;
+import io.swagger.annotations.ApiResponse;
+import io.swagger.annotations.ApiResponses;
+import org.apache.nifi.minifi.c2.model.C2Heartbeat;
+import org.apache.nifi.minifi.c2.model.C2HeartbeatResponse;
+import org.apache.nifi.minifi.c2.model.C2OperationAck;
+import org.springframework.stereotype.Component;
+
+import javax.ws.rs.Consumes;
+import javax.ws.rs.POST;
+import javax.ws.rs.Path;
+import javax.ws.rs.Produces;
+import javax.ws.rs.core.MediaType;
+import javax.ws.rs.core.Response;
+
+@Component
+@Path("/c2-protocol")
--- End diff --
Is there going to be a transformation layer for different protocol types? Will that imply live in the protocol implementation?
---
[GitHub] nifi-minifi pull request #118: MINIFI-444 C2 Data Model and REST API
Posted by phrocker <gi...@git.apache.org>.
Github user phrocker commented on a diff in the pull request:
https://github.com/apache/nifi-minifi/pull/118#discussion_r175207121
--- Diff: minifi-c2/minifi-c2-commons/src/main/java/org/apache/nifi/minifi/c2/model/extension/ExtensionComponent.java ---
@@ -0,0 +1,85 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.minifi.c2.model.extension;
+
+import io.swagger.annotations.ApiModel;
+import io.swagger.annotations.ApiModelProperty;
+
+import java.util.List;
+import java.util.Set;
+
+/**
+ * A component provided by an extension bundle
+ */
+@ApiModel
+public class ExtensionComponent extends DefinedType {
+
+ // TODO, does arch/binary/compiler metadata need to be added here?
+
+ private List<DefinedType> providedApiImplementations;
+
+ private String description;
+ private Set<String> tags;
+
+ private Boolean deprecated;
+ private String deprecationReason;
+
+ @ApiModelProperty("If this type represents a provider for an interface, this lists the APIs it implements")
+ public List<DefinedType> getProvidedApiImplementations() {
+ return providedApiImplementations;
+ }
+
+ public void setProvidedApiImplementations(List<DefinedType> providedApiImplementations) {
+ this.providedApiImplementations = providedApiImplementations;
--- End diff --
How is this "API" implementation used? What if api implementations change will we need to require that an agreed upon versioning is used ? Is this a public API?
---
[GitHub] nifi-minifi pull request #118: MINIFI-444 C2 Data Model and REST API
Posted by kevdoran <gi...@git.apache.org>.
Github user kevdoran commented on a diff in the pull request:
https://github.com/apache/nifi-minifi/pull/118#discussion_r175509585
--- Diff: minifi-c2/minifi-c2-commons/src/main/java/org/apache/nifi/minifi/c2/model/AgentInfo.java ---
@@ -0,0 +1,81 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.minifi.c2.model;
+
+import io.swagger.annotations.ApiModel;
+import io.swagger.annotations.ApiModelProperty;
+
+import java.util.Collection;
+
+@ApiModel
+public class AgentInfo {
+
+ private String identifier;
+ // TODO, do we also need identity. e.g., cert DN
+ private String agentClass;
--- End diff --
_[Some background to those who are new to the terminology that's starting to formulate around these features: In this context, class != OO type. Rather, it's intended as a category/purpose label for a MiNiFi agent that could be used for a variety of purposes from a C2 server: e.g., seeing summary views of a class, showing all agents of a particular class, scheduling a bulk operation (such as deploying flow update) to all agents that match a particular class.]_
In the data model, I do have an AgentClass type that could be used here, but I think that will really just be metadata that one can associate with a class in the C2 server. We'll have to enforce agent class name uniqueness, so that in contexts where the full agent class metadata is not needed, just the class name can be used.
---
[GitHub] nifi-minifi pull request #118: MINIFI-444 C2 Data Model and REST API
Posted by phrocker <gi...@git.apache.org>.
Github user phrocker commented on a diff in the pull request:
https://github.com/apache/nifi-minifi/pull/118#discussion_r175203879
--- Diff: minifi-c2/minifi-c2-commons/src/main/java/org/apache/nifi/minifi/c2/model/C2Operation.java ---
@@ -0,0 +1,99 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.minifi.c2.model;
+
+import io.swagger.annotations.ApiModel;
+import io.swagger.annotations.ApiModelProperty;
+
+import javax.validation.constraints.NotBlank;
+import java.util.Map;
+
+@ApiModel
+public class C2Operation {
+
+ private String identifier;
--- End diff --
The cpp agent has a nested operations. we will likely need to keep that.
---
[GitHub] nifi-minifi pull request #118: MINIFI-444 C2 Data Model and REST API
Posted by kevdoran <gi...@git.apache.org>.
Github user kevdoran commented on a diff in the pull request:
https://github.com/apache/nifi-minifi/pull/118#discussion_r175938847
--- Diff: minifi-c2/minifi-c2-commons/src/main/java/org/apache/nifi/minifi/c2/model/AgentRepositoryStatus.java ---
@@ -0,0 +1,63 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.minifi.c2.model;
+
+import io.swagger.annotations.ApiModel;
+import io.swagger.annotations.ApiModelProperty;
+
+@ApiModel
+public class AgentRepositoryStatus {
+
+ private Long size;
+ private Long sizeMax;
+ private Long count;
+ private Long countMax;
--- End diff --
For now I went with these field names:
private Long size;
private Long sizeMax;
private Long dataSize;
private Long dataSizeMax;
Thoughts?
---
[GitHub] nifi-minifi pull request #118: MINIFI-444 C2 Data Model and REST API
Posted by phrocker <gi...@git.apache.org>.
Github user phrocker commented on a diff in the pull request:
https://github.com/apache/nifi-minifi/pull/118#discussion_r175202203
--- Diff: minifi-c2/minifi-c2-commons/src/main/java/org/apache/nifi/minifi/c2/model/AgentManifest.java ---
@@ -0,0 +1,82 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.minifi.c2.model;
+
+import io.swagger.annotations.ApiModel;
+import io.swagger.annotations.ApiModelProperty;
+import org.apache.nifi.minifi.c2.model.extension.Bundle;
+
+import java.util.List;
+
+@ApiModel
+public class AgentManifest {
+
+ private String identifier;
+ private String agentType;
+ private String version;
+ private BuildInfo buildInfo;
+ private List<Bundle> bundles;
+
+ @ApiModelProperty(
+ value = "A unique identifier for the manifest",
+ readOnly = true)
+ public String getIdentifier() {
+ return identifier;
+ }
+
+ public void setIdentifier(String identifier) {
+ this.identifier = identifier;
+ }
+
+ @ApiModelProperty(
+ value = "The type of the agent binary, e.g., 'minifi-java' or 'minifi-cpp'",
+ notes = "Usually set when the agent is built.")
+ public String getAgentType() {
+ return agentType;
+ }
+
+ public void setAgentType(String agentType) {
+ this.agentType = agentType;
+ }
+
+ @ApiModelProperty(
+ value = "The version of the agent binary, e.g., '1.0.1'",
+ notes = "Usually set when the agent is built.")
+ public String getVersion() {
+ return version;
+ }
+
+ public void setVersion(String version) {
+ this.version = version;
+ }
+
+ @ApiModelProperty("Build summary for this agent binary")
+ public BuildInfo getBuildInfo() {
+ return buildInfo;
+ }
+
+ public void setBuildInfo(BuildInfo buildInfo) {
+ this.buildInfo = buildInfo;
+ }
+
+ @ApiModelProperty("All extension bundles included with this agent")
+ public List<Bundle> getBundles() {
+ return bundles;
--- End diff --
can we return an immutable list here?
---
[GitHub] nifi-minifi pull request #118: MINIFI-444 C2 Data Model and REST API
Posted by kevdoran <gi...@git.apache.org>.
Github user kevdoran commented on a diff in the pull request:
https://github.com/apache/nifi-minifi/pull/118#discussion_r175812152
--- Diff: minifi-c2/minifi-c2-commons/src/main/java/org/apache/nifi/minifi/c2/model/C2Heartbeat.java ---
@@ -0,0 +1,66 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.minifi.c2.model;
+
+import io.swagger.annotations.ApiModel;
+import io.swagger.annotations.ApiModelProperty;
+
+import java.util.Map;
+
+@ApiModel
+public class C2Heartbeat {
+
+ // TODO, timestamp?
+ private DeviceInfo deviceInfo;
+ private AgentInfo agentInfo;
+ private FlowInfo flowInfo;
+ private Map<String, String> metrics;
--- End diff --
Typed metrics have been incorporated as nested objects of the top-level DeviceInfo, AgentInfo, and FlowInfo objects.
The thinking of the additional `Map<String, String> metrics` fields was that it might be useful for agents to include some custom measurements in a heartbeat that are not "core" to the MiNiFi agent, but are relevant to the particular deployment use case or scenario. For example:
"metrics": {
"com.example.device.battery.health": "0.8",
"com.example.device.battery.cycle-count": "334",
"com.example.device.battery.charge": "0.2"
}
I was thinking that if we built such a framework, all metrics would be produced as string values (to make it easy to recieve adhoc metrics without typing metadata), so the it would be the responsibility of the consumer, presumably customize, to cast the metrics to the desired type. The idea was that the C2 server would just be an input point for these adhoc metrics that would then get forwarded to some other time-series metrics store such as Ambari Metrics System, Graphite, Datadog, etc. Those storage systems usually provide their own type conversion capabilities, so I was thinking it could keep the requirements for of defining these customized metrics lightweight and pass the interpretation logic downstream.
I don't think there is a need/use for this field yet, so I'll remove this for this PR. At a minimum, it requires work on the agent side to enable a way to add custom metrics to a heartbeat, so we don't need to officially put it into the C2 protocol until we have a need and end-to-end design to support it. There also might be a different way to solve this problem (I could think of ways to do it without C2 just using site-to-site, or maybe C2 heartbeats is the right approach, but accurate typing is needed) We can discuss it as a feature proposal and re-introduce it if people have a need for a way to add custom metrics to be collected by the MiNiFi agent to be reported in the heartbeat to the C2 Server.
---
[GitHub] nifi-minifi pull request #118: MINIFI-444 C2 Data Model and REST API
Posted by kevdoran <gi...@git.apache.org>.
Github user kevdoran commented on a diff in the pull request:
https://github.com/apache/nifi-minifi/pull/118#discussion_r176278082
--- Diff: minifi-c2/minifi-c2-commons/src/main/java/org/apache/nifi/minifi/c2/model/extension/ExtensionComponent.java ---
@@ -0,0 +1,85 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.minifi.c2.model.extension;
+
+import io.swagger.annotations.ApiModel;
+import io.swagger.annotations.ApiModelProperty;
+
+import java.util.List;
+import java.util.Set;
+
+/**
+ * A component provided by an extension bundle
+ */
+@ApiModel
+public class ExtensionComponent extends DefinedType {
+
+ // TODO, does arch/binary/compiler metadata need to be added here?
--- End diff --
> ... would it make this an implementation of N types of components based on flags/architectures or a singular object which is encapsulated elsewhere, with the necessary information...
Great question, and one I am still grappling with myself. I'd like to abstract these details away from the user as well and make it as easy as possible for both extension authors and extension users. I think I would lean towards something along the lines that an extension should define an interface (eg, the metadata in this model) that can be implemented on multiple platforms. In otherwords, if I choose to support the JVM, x86, and ARM, I could publish that as a single extension (MyCompany, MyExtension, v1.0) and there is some metadata that lets the MiNiFi/NiFi ecosystem know about the platform-specific binaries. I think this is heading down the path of a full-fledged Extension Registry though, which might be biting of a bit to much for this initial version of the C2 effort (we aren't even attempting to host binaries or facilitate centralized distribution and dynamic installation). That said, If anyone can think of a minimal version of that concept that gets us on the right path,
I'll be happy to include it!
---
[GitHub] nifi-minifi pull request #118: MINIFI-444 C2 Data Model and REST API
Posted by kevdoran <gi...@git.apache.org>.
Github user kevdoran commented on a diff in the pull request:
https://github.com/apache/nifi-minifi/pull/118#discussion_r175843712
--- Diff: minifi-c2/minifi-c2-commons/src/main/java/org/apache/nifi/minifi/c2/model/extension/DefinedType.java ---
@@ -0,0 +1,69 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.minifi.c2.model.extension;
+
+import io.swagger.annotations.ApiModel;
+import io.swagger.annotations.ApiModelProperty;
+
+/**
+ * A reference to a defined type identified by bundle and fully qualified class type identifiers
+ */
+@ApiModel
+public class DefinedType {
+
+ private String group;
+ private String artifact;
+ private String version;
+ private String type;
--- End diff --
I think in practice it will just be a class that is deployable, i.e., via a flow config.yaml. So basically a processor, controller service, or reporting task.
But it could be more than that. For example, in NiFi, some of the core framework interface implementations are actually built as NARs and deployed by including them in the /lib dir. So givent that the model in the `org.apache.nifi.minifi.c2.model.extension` package is designed to be a prototype for a model that could eventually live in an Extension Registry, these could be used in the NiFi case as well. But for our purposes, just the types that would be used as Flow Components
---
[GitHub] nifi-minifi pull request #118: MINIFI-444 C2 Data Model and REST API
Posted by phrocker <gi...@git.apache.org>.
Github user phrocker commented on a diff in the pull request:
https://github.com/apache/nifi-minifi/pull/118#discussion_r175206733
--- Diff: minifi-c2/minifi-c2-commons/src/main/java/org/apache/nifi/minifi/c2/model/extension/ExtensionComponent.java ---
@@ -0,0 +1,85 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.minifi.c2.model.extension;
+
+import io.swagger.annotations.ApiModel;
+import io.swagger.annotations.ApiModelProperty;
+
+import java.util.List;
+import java.util.Set;
+
+/**
+ * A component provided by an extension bundle
+ */
+@ApiModel
+public class ExtensionComponent extends DefinedType {
+
+ // TODO, does arch/binary/compiler metadata need to be added here?
--- End diff --
I'm afraid if we don't include this we will likely not have deployable C++ agents across platforms.
---
[GitHub] nifi-minifi pull request #118: MINIFI-444 C2 Data Model and REST API
Posted by phrocker <gi...@git.apache.org>.
Github user phrocker commented on a diff in the pull request:
https://github.com/apache/nifi-minifi/pull/118#discussion_r175205789
--- Diff: minifi-c2/minifi-c2-commons/src/main/java/org/apache/nifi/minifi/c2/model/extension/Bundle.java ---
@@ -0,0 +1,106 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.minifi.c2.model.extension;
+
+import io.swagger.annotations.ApiModel;
+import io.swagger.annotations.ApiModelProperty;
+
+import java.util.Objects;
+
+@ApiModel
+public class Bundle {
+
+ public static final String DEFAULT_GROUP = "default";
+ public static final String DEFAULT_ARTIFACT = "unknown";
+ public static final String DEFAULT_VERSION = "unversioned";
+
+ private String group;
+ private String artifact;
+ private String version;
+ // TODO md5sum?
+ private BundleManifest bundleManifest;
+
+ public Bundle() {}
+
+ public Bundle(String group, String artifact, String version) {
+ this.group = group;
+ this.artifact = artifact;
+ this.version = version;
+ }
+
+ public static Bundle defaultBundle() {
+ return new Bundle(DEFAULT_GROUP, DEFAULT_ARTIFACT, DEFAULT_VERSION);
+ }
+
+ @ApiModelProperty(
+ value = "The group id of the bundle",
+ notes = "A globally unique group namespace, e.g., org.apache.nifi",
+ required = true)
+ public String getGroup() {
+ return group;
+ }
+
+ public void setGroup(String group) {
+ this.group = group;
+ }
+
+ @ApiModelProperty(
+ value = "The artifact id of the bundle",
+ notes = "Unique within the group",
+ required = true)
+ public String getArtifact() {
+ return artifact;
+ }
+
+ public void setArtifact(String artifact) {
+ this.artifact = artifact;
+ }
+
+ @ApiModelProperty("The version of the bundle artifact")
+ public String getVersion() {
+ return version;
+ }
+
+ public void setVersion(String version) {
+ this.version = version;
+ }
+
+ @ApiModelProperty(value = "The full specification of the bundle contents",
+ notes = "This is optional, as the group, artifact, and version are " +
+ "also enough to reference a bundle in the case the bundle " +
+ "specification has been published to a registry.")
+ public BundleManifest getBundleManifest() {
+ return bundleManifest;
+ }
+
+ public void setBundleManifest(BundleManifest bundleManifest) {
+ this.bundleManifest = bundleManifest;
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (this == o) return true;
--- End diff --
Equals builder can make this a little cleaner. I see that commons-lang is in the root pom.
---
[GitHub] nifi-minifi pull request #118: MINIFI-444 C2 Data Model and REST API
Posted by kevdoran <gi...@git.apache.org>.
Github user kevdoran commented on a diff in the pull request:
https://github.com/apache/nifi-minifi/pull/118#discussion_r175509670
--- Diff: minifi-c2/minifi-c2-commons/src/main/java/org/apache/nifi/minifi/c2/model/AgentManifest.java ---
@@ -0,0 +1,82 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.minifi.c2.model;
+
+import io.swagger.annotations.ApiModel;
+import io.swagger.annotations.ApiModelProperty;
+import org.apache.nifi.minifi.c2.model.extension.Bundle;
+
+import java.util.List;
+
+@ApiModel
+public class AgentManifest {
+
+ private String identifier;
+ private String agentType;
+ private String version;
+ private BuildInfo buildInfo;
+ private List<Bundle> bundles;
+
+ @ApiModelProperty(
+ value = "A unique identifier for the manifest",
+ readOnly = true)
+ public String getIdentifier() {
+ return identifier;
+ }
+
+ public void setIdentifier(String identifier) {
+ this.identifier = identifier;
+ }
+
+ @ApiModelProperty(
+ value = "The type of the agent binary, e.g., 'minifi-java' or 'minifi-cpp'",
+ notes = "Usually set when the agent is built.")
+ public String getAgentType() {
+ return agentType;
+ }
+
+ public void setAgentType(String agentType) {
+ this.agentType = agentType;
+ }
+
+ @ApiModelProperty(
+ value = "The version of the agent binary, e.g., '1.0.1'",
+ notes = "Usually set when the agent is built.")
+ public String getVersion() {
+ return version;
+ }
+
+ public void setVersion(String version) {
+ this.version = version;
+ }
+
+ @ApiModelProperty("Build summary for this agent binary")
+ public BuildInfo getBuildInfo() {
+ return buildInfo;
+ }
+
+ public void setBuildInfo(BuildInfo buildInfo) {
+ this.buildInfo = buildInfo;
+ }
+
+ @ApiModelProperty("All extension bundles included with this agent")
+ public List<Bundle> getBundles() {
+ return bundles;
--- End diff --
Yes, will do.
---
[GitHub] nifi-minifi pull request #118: MINIFI-444 C2 Data Model and REST API
Posted by kevdoran <gi...@git.apache.org>.
Github user kevdoran commented on a diff in the pull request:
https://github.com/apache/nifi-minifi/pull/118#discussion_r175506629
--- Diff: minifi-c2/minifi-c2-commons/src/main/java/org/apache/nifi/minifi/c2/model/AgentInfo.java ---
@@ -0,0 +1,81 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.minifi.c2.model;
+
+import io.swagger.annotations.ApiModel;
+import io.swagger.annotations.ApiModelProperty;
+
+import java.util.Collection;
+
+@ApiModel
+public class AgentInfo {
+
+ private String identifier;
+ // TODO, do we also need identity. e.g., cert DN
+ private String agentClass;
+ private Collection<String> tags;
--- End diff --
I was thinking it would be nice for an operator to be able to tag agents with arbitrary labels that they can then use as filters in future queries. I'm not sure those tags would be just on the C2 server in the agents it is tracking, or if they could also be set on agents when they are provisioned and deployed, similar to the agent class labels.
Thanks for calling this out. Now that I think about it, I'd like to drop this field for now so that we can properly discuss (on the mailing list, JIRA, etc) if this is valuable and if so how to implement it.
---
[GitHub] nifi-minifi pull request #118: MINIFI-444 C2 Data Model and REST API
Posted by phrocker <gi...@git.apache.org>.
Github user phrocker commented on a diff in the pull request:
https://github.com/apache/nifi-minifi/pull/118#discussion_r175206927
--- Diff: minifi-c2/minifi-c2-commons/src/main/java/org/apache/nifi/minifi/c2/model/extension/DefinedType.java ---
@@ -0,0 +1,69 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.minifi.c2.model.extension;
+
+import io.swagger.annotations.ApiModel;
+import io.swagger.annotations.ApiModelProperty;
+
+/**
+ * A reference to a defined type identified by bundle and fully qualified class type identifiers
+ */
+@ApiModel
+public class DefinedType {
+
+ private String group;
+ private String artifact;
+ private String version;
+ private String type;
--- End diff --
A defined type is effectively a version for a class that's deployable or any class in the software stack?
---
[GitHub] nifi-minifi pull request #118: MINIFI-444 C2 Data Model and REST API
Posted by kevdoran <gi...@git.apache.org>.
Github user kevdoran commented on a diff in the pull request:
https://github.com/apache/nifi-minifi/pull/118#discussion_r176280748
--- Diff: minifi-c2/minifi-c2-commons/src/main/java/org/apache/nifi/minifi/c2/model/FlowStatus.java ---
@@ -0,0 +1,51 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.minifi.c2.model;
+
+import io.swagger.annotations.ApiModel;
+import io.swagger.annotations.ApiModelProperty;
+
+import java.util.Map;
+
+/**
+ * Status of the aspects of the flow that are controllable by the C2 server, ie:
+ * - Processors that can be started/stopped and their current state
+ * - Queues that can be cleared and their current state
+ */
+@ApiModel
+public class FlowStatus {
--- End diff --
OK, I got it. The current structure assumes to much and is limiting. Will take another look at that, thanks.
Yes, I don't think these DTOs need to worry about concurrent access. Official model and state and would be managed at a lower level in the C2 server (persisted or cached and accessed through thread safe interfaces). These objects represent a snapshot copy for I/O. It should become clearer in my next PR that adds some of the service and persistence layer foundations.
---
[GitHub] nifi-minifi pull request #118: MINIFI-444 C2 Data Model and REST API
Posted by phrocker <gi...@git.apache.org>.
Github user phrocker commented on a diff in the pull request:
https://github.com/apache/nifi-minifi/pull/118#discussion_r175202823
--- Diff: minifi-c2/minifi-c2-commons/src/main/java/org/apache/nifi/minifi/c2/model/BuildInfo.java ---
@@ -0,0 +1,77 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.minifi.c2.model;
+
+import io.swagger.annotations.ApiModel;
+import io.swagger.annotations.ApiModelProperty;
+
+@ApiModel
+public class BuildInfo {
--- End diff --
Is this an agent level class? The definition below makes me wonder if we're modeling extensions differently than agent information? That model doesn't seem to fit what we can/will need to actually do.
---
[GitHub] nifi-minifi pull request #118: MINIFI-444 C2 Data Model and REST API
Posted by kevdoran <gi...@git.apache.org>.
Github user kevdoran commented on a diff in the pull request:
https://github.com/apache/nifi-minifi/pull/118#discussion_r175587202
--- Diff: minifi-c2/minifi-c2-commons/src/main/java/org/apache/nifi/minifi/c2/model/extension/ExtensionComponent.java ---
@@ -0,0 +1,85 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.minifi.c2.model.extension;
+
+import io.swagger.annotations.ApiModel;
+import io.swagger.annotations.ApiModelProperty;
+
+import java.util.List;
+import java.util.Set;
+
+/**
+ * A component provided by an extension bundle
+ */
+@ApiModel
+public class ExtensionComponent extends DefinedType {
+
+ // TODO, does arch/binary/compiler metadata need to be added here?
+
+ private List<DefinedType> providedApiImplementations;
+
+ private String description;
+ private Set<String> tags;
+
+ private Boolean deprecated;
+ private String deprecationReason;
+
+ @ApiModelProperty("If this type represents a provider for an interface, this lists the APIs it implements")
+ public List<DefinedType> getProvidedApiImplementations() {
+ return providedApiImplementations;
+ }
+
+ public void setProvidedApiImplementations(List<DefinedType> providedApiImplementations) {
+ this.providedApiImplementations = providedApiImplementations;
--- End diff --
The intention of this field is to allow a component to declare that it provides an implementation of an interface defined by another DefinedType.
The goal (the details of which can change) is to allow bundles to declare interfaces/headers as DefinedTypes that other components can provide implementations for.
This is used in the NiFi UI, so will also likely be useful in the [Flow Designer](https://lists.apache.org/thread.html/b720ff7b6e4e88fb23163b279f2d71eb125df18d1be8a7fca1c9b28d@%3Cdev.nifi.apache.org%3E). In NiFi, it is used so that property descriptors can mark a property as requiring an API implementation selection as its value, so that those options can be presented in a drop down selection. That is one thought of how this will be used in this context.
Another possible use case of this is defining an interface in version 1.0 of and extension bundle and letting implementations of that interface explicitly declare support for in future versions of the bundle. For example, you could have:
MyBundle v1.0.0
apis: [org::apache:nifi:minifi::DatabaseService]
controllerServices: org::apache:nifi:minifi::DatabaseServiceImpl
provides: [MyBundle/v1.0.0/org::apache:nifi:minifi::DatabaseService]
MyBundle v1.0.1
controllerServices: org::apache:nifi:minifi::DatabaseServiceImpl
provides: [MyBundle/v1.0.0/org::apache:nifi:minifi::DatabaseService]
MyBundle v1.1.0
apis: [org::apache:nifi:minifi::DatabaseService]
controllerServices: org::apache:nifi:minifi::DatabaseServiceImpl
provides: [MyBundle/v1.0.0/org::apache:nifi:minifi::DatabaseService,
MyBundle/v1.1.0/org::apache:nifi:minifi::DatabaseService]
MyBundle v2.0.0
apis: [org::apache:nifi:minifi::DatabaseService]
controllerServices: org::apache:nifi:minifi::DatabaseServiceImpl
provides: [MyBundle/v2.0.0/org::apache:nifi:minifi::DatabaseService]
In this example, an API is defined in a 1.0, improved in a patch release, evolved in a minor release, and then ultimately rethought with breaking changes in a 2.0. The implementation of the ControllerService API, which happens to be in the same bundle in this example, can declare compatibility with current or previous versions.
---
[GitHub] nifi-minifi pull request #118: MINIFI-444 C2 Data Model and REST API
Posted by phrocker <gi...@git.apache.org>.
Github user phrocker commented on a diff in the pull request:
https://github.com/apache/nifi-minifi/pull/118#discussion_r175204543
--- Diff: minifi-c2/minifi-c2-commons/src/main/java/org/apache/nifi/minifi/c2/model/FlowQueueStatus.java ---
@@ -0,0 +1,63 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.minifi.c2.model;
+
+import io.swagger.annotations.ApiModel;
+import io.swagger.annotations.ApiModelProperty;
+
+@ApiModel
+public class FlowQueueStatus {
+
+ private Long size;
--- End diff --
same here regarding sizes. I think the queues are less likely to be an issue but if they are set to zero then they will be system long max.
---
[GitHub] nifi-minifi pull request #118: MINIFI-444 C2 Data Model and REST API
Posted by kevdoran <gi...@git.apache.org>.
Github user kevdoran commented on a diff in the pull request:
https://github.com/apache/nifi-minifi/pull/118#discussion_r175586309
--- Diff: minifi-c2/minifi-c2-commons/src/main/java/org/apache/nifi/minifi/c2/model/extension/PropertyDescriptor.java ---
@@ -0,0 +1,123 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.minifi.c2.model.extension;
+
+import io.swagger.annotations.ApiModel;
+import io.swagger.annotations.ApiModelProperty;
+
+import java.util.List;
+
+@ApiModel
+public class PropertyDescriptor {
+
+ private String name;
+ private String displayName;
+ private String description;
+ private List<PropertyAllowableValue> allowableValues;
+ private String defaultValue;
+ private Boolean required;
+ private Boolean sensitive;
+ private Boolean supportsEl;
+ private DefinedType identifiesControllerServiceApi;
--- End diff --
No, but I can see how the field name is misleading. The intention of this field is to indicate if the property value identifies a Controller Service API, and if so, which DefinedType that API is. This is used in NiFi to present a drop down menu of any ExtensionComponents that implement that API.
See the comment about which is related to this field.
If this API definition+implementation metadata complexity is not needed in this initial version of this workflow, or if it is too difficult to provide, we could leave it out and reintroduce it in a future version of C2 server.
---
[GitHub] nifi-minifi pull request #118: MINIFI-444 C2 Data Model and REST API
Posted by phrocker <gi...@git.apache.org>.
Github user phrocker commented on a diff in the pull request:
https://github.com/apache/nifi-minifi/pull/118#discussion_r176270398
--- Diff: minifi-c2/minifi-c2-commons/src/main/java/org/apache/nifi/minifi/c2/model/C2Operation.java ---
@@ -0,0 +1,99 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.minifi.c2.model;
+
+import io.swagger.annotations.ApiModel;
+import io.swagger.annotations.ApiModelProperty;
+
+import javax.validation.constraints.NotBlank;
+import java.util.Map;
+
+@ApiModel
+public class C2Operation {
+
+ private String identifier;
--- End diff --
Think of it like this
OpA {
OpB {
OpC
}
OpD{}}
}
OpC would not execute if B fails
---
[GitHub] nifi-minifi pull request #118: MINIFI-444 C2 Data Model and REST API
Posted by kevdoran <gi...@git.apache.org>.
Github user kevdoran closed the pull request at:
https://github.com/apache/nifi-minifi/pull/118
---
[GitHub] nifi-minifi pull request #118: MINIFI-444 C2 Data Model and REST API
Posted by phrocker <gi...@git.apache.org>.
Github user phrocker commented on a diff in the pull request:
https://github.com/apache/nifi-minifi/pull/118#discussion_r175200116
--- Diff: minifi-c2/minifi-c2-commons/src/main/java/org/apache/nifi/minifi/c2/model/AgentInfo.java ---
@@ -0,0 +1,81 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.minifi.c2.model;
+
+import io.swagger.annotations.ApiModel;
+import io.swagger.annotations.ApiModelProperty;
+
+import java.util.Collection;
+
+@ApiModel
+public class AgentInfo {
+
+ private String identifier;
+ // TODO, do we also need identity. e.g., cert DN
+ private String agentClass;
+ private Collection<String> tags;
--- End diff --
What are tags?
---
[GitHub] nifi-minifi pull request #118: MINIFI-444 C2 Data Model and REST API
Posted by phrocker <gi...@git.apache.org>.
Github user phrocker commented on a diff in the pull request:
https://github.com/apache/nifi-minifi/pull/118#discussion_r175200297
--- Diff: minifi-c2/minifi-c2-commons/src/main/java/org/apache/nifi/minifi/c2/model/AgentManifest.java ---
@@ -0,0 +1,82 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.minifi.c2.model;
+
+import io.swagger.annotations.ApiModel;
+import io.swagger.annotations.ApiModelProperty;
+import org.apache.nifi.minifi.c2.model.extension.Bundle;
+
+import java.util.List;
+
+@ApiModel
+public class AgentManifest {
+
+ private String identifier;
--- End diff --
is this referential to the identifiers above?
---
[GitHub] nifi-minifi pull request #118: MINIFI-444 C2 Data Model and REST API
Posted by phrocker <gi...@git.apache.org>.
Github user phrocker commented on a diff in the pull request:
https://github.com/apache/nifi-minifi/pull/118#discussion_r175208131
--- Diff: minifi-c2/minifi-c2-commons/src/main/java/org/apache/nifi/minifi/c2/model/C2OperationAck.java ---
@@ -0,0 +1,35 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.minifi.c2.model;
+
+import io.swagger.annotations.ApiModel;
+import io.swagger.annotations.ApiModelProperty;
+
+@ApiModel
+public class C2OperationAck {
+
+ private String operationId;
+ // TODO, result, outcome, or status of the operation? eg SUCCESS/FAIL?
--- End diff --
yeah agreed. probably need something to provide more information.
---
[GitHub] nifi-minifi pull request #118: MINIFI-444 C2 Data Model and REST API
Posted by phrocker <gi...@git.apache.org>.
Github user phrocker commented on a diff in the pull request:
https://github.com/apache/nifi-minifi/pull/118#discussion_r176273652
--- Diff: minifi-c2/minifi-c2-commons/src/main/java/org/apache/nifi/minifi/c2/model/extension/ExtensionComponent.java ---
@@ -0,0 +1,85 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.minifi.c2.model.extension;
+
+import io.swagger.annotations.ApiModel;
+import io.swagger.annotations.ApiModelProperty;
+
+import java.util.List;
+import java.util.Set;
+
+/**
+ * A component provided by an extension bundle
+ */
+@ApiModel
+public class ExtensionComponent extends DefinedType {
+
+ // TODO, does arch/binary/compiler metadata need to be added here?
--- End diff --
There may be additional toolchain formation we may need to carry over. I'm not 100% certain at this time if we would need all of that. It comes down to making a decision based on the compile time flags to know if an extension or implementation of an extension is translatable to an agent's architecture. I'm a little fuzzy on how this model views ExtensionComponent as it relates to an Extension of architectures types. In that case would it make this an implementation of N types of components based on flags/architectures or a singular object which is encapsulated elsewhere, with the necessary information...
---
[GitHub] nifi-minifi pull request #118: MINIFI-444 C2 Data Model and REST API
Posted by kevdoran <gi...@git.apache.org>.
Github user kevdoran commented on a diff in the pull request:
https://github.com/apache/nifi-minifi/pull/118#discussion_r175587026
--- Diff: minifi-c2/minifi-c2-commons/src/main/java/org/apache/nifi/minifi/c2/model/extension/DefinedType.java ---
@@ -0,0 +1,69 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.minifi.c2.model.extension;
+
+import io.swagger.annotations.ApiModel;
+import io.swagger.annotations.ApiModelProperty;
+
+/**
+ * A reference to a defined type identified by bundle and fully qualified class type identifiers
+ */
+@ApiModel
+public class DefinedType {
+
+ private String group;
+ private String artifact;
+ private String version;
+ private String type;
+
+ @ApiModelProperty("The group name of the bundle that provides the referenced type.")
+ public String getGroup() {
+ return group;
+ }
+
+ public void setGroup(String group) {
+ this.group = group;
+ }
+
+ @ApiModelProperty("The artifact name of the bundle that provides the referenced type.")
+ public String getArtifact() {
+ return artifact;
+ }
+
+ public void setArtifact(String artifact) {
+ this.artifact = artifact;
+ }
+
+ @ApiModelProperty("The version of the bundle that provides the referenced type.")
+ public String getVersion() {
+ return version;
+ }
+
+ public void setVersion(String version) {
+ this.version = version;
+ }
+
+ @ApiModelProperty(
+ value = "The fully-qualified class type",
+ required = true,
+ notes = "For example, 'org.apache.nifi.GetFile' or 'org::apache:nifi::minifi::GetFile'")
--- End diff --
See above, it's really only intended to declare that a bundle contains an API definition, not the API definition itself. Presumable the interface type would reference a fully qualified class interface defined in a .h or .java file.
---
[GitHub] nifi-minifi pull request #118: MINIFI-444 C2 Data Model and REST API
Posted by kevdoran <gi...@git.apache.org>.
Github user kevdoran commented on a diff in the pull request:
https://github.com/apache/nifi-minifi/pull/118#discussion_r175587496
--- Diff: minifi-c2/minifi-c2-commons/src/main/java/org/apache/nifi/minifi/c2/model/BuildInfo.java ---
@@ -0,0 +1,77 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.minifi.c2.model;
+
+import io.swagger.annotations.ApiModel;
+import io.swagger.annotations.ApiModelProperty;
+
+@ApiModel
+public class BuildInfo {
--- End diff --
Yeah, good point. We probably need it at the extensioncomponent level as well. See my comment below.
---
[GitHub] nifi-minifi pull request #118: MINIFI-444 C2 Data Model and REST API
Posted by kevdoran <gi...@git.apache.org>.
Github user kevdoran commented on a diff in the pull request:
https://github.com/apache/nifi-minifi/pull/118#discussion_r176278810
--- Diff: minifi-c2/minifi-c2-commons/src/main/java/org/apache/nifi/minifi/c2/model/C2Operation.java ---
@@ -0,0 +1,99 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.minifi.c2.model;
+
+import io.swagger.annotations.ApiModel;
+import io.swagger.annotations.ApiModelProperty;
+
+import javax.validation.constraints.NotBlank;
+import java.util.Map;
+
+@ApiModel
+public class C2Operation {
+
+ private String identifier;
--- End diff --
OK got it, thanks. It seems that a protocol distinction here is that operations in the top level requestedOperations list are always executed, but nested operations only execute if their parent succeeds?
---
[GitHub] nifi-minifi pull request #118: MINIFI-444 C2 Data Model and REST API
Posted by kevdoran <gi...@git.apache.org>.
Github user kevdoran commented on a diff in the pull request:
https://github.com/apache/nifi-minifi/pull/118#discussion_r175588599
--- Diff: minifi-c2/minifi-c2-commons/src/main/java/org/apache/nifi/minifi/c2/model/C2Operation.java ---
@@ -0,0 +1,99 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.minifi.c2.model;
+
+import io.swagger.annotations.ApiModel;
+import io.swagger.annotations.ApiModelProperty;
+
+import javax.validation.constraints.NotBlank;
+import java.util.Map;
+
+@ApiModel
+public class C2Operation {
+
+ private String identifier;
--- End diff --
Interesting. I guess I missed that. Can you provide an example?
---
[GitHub] nifi-minifi pull request #118: MINIFI-444 C2 Data Model and REST API
Posted by kevdoran <gi...@git.apache.org>.
Github user kevdoran commented on a diff in the pull request:
https://github.com/apache/nifi-minifi/pull/118#discussion_r176279968
--- Diff: minifi-c2/minifi-c2-commons/src/main/java/org/apache/nifi/minifi/c2/model/extension/BundleManifest.java ---
@@ -0,0 +1,65 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.minifi.c2.model.extension;
+
+import io.swagger.annotations.ApiModel;
+import io.swagger.annotations.ApiModelProperty;
+
+import java.util.List;
+
+@ApiModel
+public class BundleManifest {
+
--- End diff --
OK, I like that too. I'll add top level components to the AgentManifest, and devs can choose which makes more sense for them. Flow Designer can adapt to both.
---
[GitHub] nifi-minifi pull request #118: MINIFI-444 C2 Data Model and REST API
Posted by phrocker <gi...@git.apache.org>.
Github user phrocker commented on a diff in the pull request:
https://github.com/apache/nifi-minifi/pull/118#discussion_r175201914
--- Diff: minifi-c2/minifi-c2-commons/src/main/java/org/apache/nifi/minifi/c2/model/AgentRepositoryStatus.java ---
@@ -0,0 +1,63 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.minifi.c2.model;
+
+import io.swagger.annotations.ApiModel;
+import io.swagger.annotations.ApiModelProperty;
+
+@ApiModel
+public class AgentRepositoryStatus {
+
+ private Long size;
+ private Long sizeMax;
+ private Long count;
+ private Long countMax;
--- End diff --
Count max seems confusing. I imagine it's not the max count seen. Realize you set a description in the model prop annotation, but you probably should tend toward a different name. Similarly sizemax and countmax seem to have overlap. Would storageMax and size max better reflect their intent?
---
[GitHub] nifi-minifi pull request #118: MINIFI-444 C2 Data Model and REST API
Posted by phrocker <gi...@git.apache.org>.
Github user phrocker commented on a diff in the pull request:
https://github.com/apache/nifi-minifi/pull/118#discussion_r175200073
--- Diff: minifi-c2/minifi-c2-commons/src/main/java/org/apache/nifi/minifi/c2/model/AgentInfo.java ---
@@ -0,0 +1,81 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.minifi.c2.model;
+
+import io.swagger.annotations.ApiModel;
+import io.swagger.annotations.ApiModelProperty;
+
+import java.util.Collection;
+
+@ApiModel
+public class AgentInfo {
+
+ private String identifier;
+ // TODO, do we also need identity. e.g., cert DN
+ private String agentClass;
--- End diff --
Any reason why this is not more strongly typed?
---
[GitHub] nifi-minifi pull request #118: MINIFI-444 C2 Data Model and REST API
Posted by phrocker <gi...@git.apache.org>.
Github user phrocker commented on a diff in the pull request:
https://github.com/apache/nifi-minifi/pull/118#discussion_r176272024
--- Diff: minifi-c2/minifi-c2-commons/src/main/java/org/apache/nifi/minifi/c2/model/extension/ExtensionComponent.java ---
@@ -0,0 +1,85 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.minifi.c2.model.extension;
+
+import io.swagger.annotations.ApiModel;
+import io.swagger.annotations.ApiModelProperty;
+
+import java.util.List;
+import java.util.Set;
+
+/**
+ * A component provided by an extension bundle
+ */
+@ApiModel
+public class ExtensionComponent extends DefinedType {
+
+ // TODO, does arch/binary/compiler metadata need to be added here?
--- End diff --
For extensions we probably need to carry the type of built binary, to include arch built, compiler flags that could limit cross platform compatibility, and anything else that could possibly limit movement of that object. Ideally this is all abstracted from the user, so whether that lives here or elsewhere is unclear; however we would need to be certain a user isn't getting x86 shared object when they're running on a pi.
---
[GitHub] nifi-minifi pull request #118: MINIFI-444 C2 Data Model and REST API
Posted by kevdoran <gi...@git.apache.org>.
Github user kevdoran commented on a diff in the pull request:
https://github.com/apache/nifi-minifi/pull/118#discussion_r175573944
--- Diff: minifi-c2/minifi-c2-web-api/src/main/java/org/apache/nifi/minifi/c2/web/api/AgentClassResource.java ---
@@ -0,0 +1,132 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.minifi.c2.web.api;
+
+import io.swagger.annotations.Api;
+import io.swagger.annotations.ApiOperation;
+import io.swagger.annotations.ApiParam;
+import org.apache.nifi.minifi.c2.model.AgentClass;
+import org.springframework.stereotype.Component;
+
+import javax.ws.rs.Consumes;
+import javax.ws.rs.DELETE;
+import javax.ws.rs.GET;
+import javax.ws.rs.NotSupportedException;
+import javax.ws.rs.PATCH;
+import javax.ws.rs.POST;
+import javax.ws.rs.PUT;
+import javax.ws.rs.Path;
+import javax.ws.rs.PathParam;
+import javax.ws.rs.Produces;
+import javax.ws.rs.core.MediaType;
+import javax.ws.rs.core.Response;
+
+@Component
+@Path("/agent-classes")
+@Api(value = "Agent Classes", description = "Register and manage agent class definitions")
+public class AgentClassResource {
+
+ @POST
+ @Consumes(MediaType.APPLICATION_JSON)
+ @Produces(MediaType.APPLICATION_JSON)
+ @ApiOperation(
+ value = "Register a MiNiFi agent class with this C2 server",
+ notes = "This can also be done with a heartbeat, which will register a MiNiFi agent class the first time it is seen in a heartbeat.",
+ response = AgentClass.class
+ )
+ public Response createAgentClass(
+ @ApiParam(value = "The class to create", required = true)
+ AgentClass agentClass) {
+ throw new NotSupportedException("This method is not yet implemented for this resource.");
+
+ }
+
+ @GET
+ @Produces(MediaType.APPLICATION_JSON)
+ @ApiOperation(
+ value = "Get all MiNiFi agent classes that are registered with this C2 server",
+ response = AgentClass.class,
+ responseContainer = "List"
+ )
+ public Response getAgentClasses() {
--- End diff --
- [ ] Add ReadOnly NiFi Registry URL, Bucket, and Flow ID metadata when retrieving Agent classes (so that the flow designer knows where to read/save a flow)
---
[GitHub] nifi-minifi pull request #118: MINIFI-444 C2 Data Model and REST API
Posted by kevdoran <gi...@git.apache.org>.
Github user kevdoran commented on a diff in the pull request:
https://github.com/apache/nifi-minifi/pull/118#discussion_r175517256
--- Diff: minifi-c2/minifi-c2-commons/src/main/java/org/apache/nifi/minifi/c2/model/AgentRepositoryStatus.java ---
@@ -0,0 +1,63 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.minifi.c2.model;
+
+import io.swagger.annotations.ApiModel;
+import io.swagger.annotations.ApiModelProperty;
+
+@ApiModel
+public class AgentRepositoryStatus {
+
+ private Long size;
+ private Long sizeMax;
+ private Long count;
+ private Long countMax;
--- End diff --
I'm not tied to the names of these fields. I agree that storageMax is better than sizeMax and that 'size' in general is ambiguous without reading the description. Better to avoid confusion.
I want to make this change so that the names stay consistent with the FlowQueueStatus fields if possible. My understanding (correct me if this is wrong) is that both queues and repositories have two important metrics:
1. number of items they currently contain
2. the data/storage footprint of those items
And that both of these metrics can number configurable caps (i.e., the "max value") for both queues and repositories.
If that understanding is not correct, let me know as I'll need to correct these model classes. If it is correct, what would be the best names to use that would be consistent with how these values are labeled in MiNiFi's configuration?
---
[GitHub] nifi-minifi pull request #118: MINIFI-444 C2 Data Model and REST API
Posted by apiri <gi...@git.apache.org>.
Github user apiri commented on a diff in the pull request:
https://github.com/apache/nifi-minifi/pull/118#discussion_r176286241
--- Diff: minifi-c2/minifi-c2-commons/src/main/java/org/apache/nifi/minifi/c2/model/FlowUri.java ---
@@ -0,0 +1,54 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.minifi.c2.model;
+
+import io.swagger.annotations.ApiModel;
+import io.swagger.annotations.ApiModelProperty;
+
+@ApiModel("Uniform Resource Identifier for NiFi Versioned Flows saved to a NiFi Registry")
+public class FlowUri {
--- End diff --
I dig the constructor/toString idea. Gives us more flexibility and can tack things on as needed for whatever reason.
---
[GitHub] nifi-minifi pull request #118: MINIFI-444 C2 Data Model and REST API
Posted by kevdoran <gi...@git.apache.org>.
Github user kevdoran commented on a diff in the pull request:
https://github.com/apache/nifi-minifi/pull/118#discussion_r176286360
--- Diff: minifi-c2/minifi-c2-commons/src/main/java/org/apache/nifi/minifi/c2/model/FlowUri.java ---
@@ -0,0 +1,54 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.minifi.c2.model;
+
+import io.swagger.annotations.ApiModel;
+import io.swagger.annotations.ApiModelProperty;
+
+@ApiModel("Uniform Resource Identifier for NiFi Versioned Flows saved to a NiFi Registry")
+public class FlowUri {
--- End diff --
Will do!
---
[GitHub] nifi-minifi pull request #118: MINIFI-444 C2 Data Model and REST API
Posted by phrocker <gi...@git.apache.org>.
Github user phrocker commented on a diff in the pull request:
https://github.com/apache/nifi-minifi/pull/118#discussion_r175202384
--- Diff: minifi-c2/minifi-c2-commons/src/main/java/org/apache/nifi/minifi/c2/model/AgentInfo.java ---
@@ -0,0 +1,81 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.minifi.c2.model;
+
+import io.swagger.annotations.ApiModel;
+import io.swagger.annotations.ApiModelProperty;
+
+import java.util.Collection;
+
+@ApiModel
+public class AgentInfo {
+
+ private String identifier;
+ // TODO, do we also need identity. e.g., cert DN
+ private String agentClass;
+ private Collection<String> tags;
+ private AgentManifest agentManifest;
+ private AgentStatus status;
+ // private Map<String, String> configProperties; // TODO we should add this information eventually, but we need to handle the best way to handle sharing sensitive properties.
+
+ @ApiModelProperty(
+ value = "A unique identifier for the Agent",
+ notes = "Usually set when the agent is provisioned and deployed")
+ public String getIdentifier() {
+ return identifier;
+ }
+
+ public void setIdentifier(String identifier) {
+ this.identifier = identifier;
+ }
+
+ @ApiModelProperty(
+ value = "The class or category label of the agent, e.g., 'sensor-collector'",
+ notes = "Usually set when the agent is provisioned and deployed")
+ public String getAgentClass() {
+ return agentClass;
+ }
+
+ public void setAgentClass(String agentClass) {
+ this.agentClass = agentClass;
+ }
+
+ @ApiModelProperty("Arbitrary tags that describe the agent in addition to the class")
+ public Collection<String> getTags() {
+ return tags;
--- End diff --
can this be an immutable collection ? Preferably a set since I imagine similar tags have no real meaning?
---
[GitHub] nifi-minifi pull request #118: MINIFI-444 C2 Data Model and REST API
Posted by phrocker <gi...@git.apache.org>.
Github user phrocker commented on a diff in the pull request:
https://github.com/apache/nifi-minifi/pull/118#discussion_r175203025
--- Diff: minifi-c2/minifi-c2-commons/src/main/java/org/apache/nifi/minifi/c2/model/C2Heartbeat.java ---
@@ -0,0 +1,66 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.minifi.c2.model;
+
+import io.swagger.annotations.ApiModel;
+import io.swagger.annotations.ApiModelProperty;
+
+import java.util.Map;
+
+@ApiModel
+public class C2Heartbeat {
+
+ // TODO, timestamp?
+ private DeviceInfo deviceInfo;
+ private AgentInfo agentInfo;
+ private FlowInfo flowInfo;
+ private Map<String, String> metrics;
--- End diff --
Metrics are currently typed, is there a need for bag of words? can we have that be metadata, imo that has a better meaning long term.
---
[GitHub] nifi-minifi pull request #118: MINIFI-444 C2 Data Model and REST API
Posted by kevdoran <gi...@git.apache.org>.
Github user kevdoran commented on a diff in the pull request:
https://github.com/apache/nifi-minifi/pull/118#discussion_r176281479
--- Diff: minifi-c2/minifi-c2-commons/src/main/java/org/apache/nifi/minifi/c2/model/FlowUri.java ---
@@ -0,0 +1,54 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.minifi.c2.model;
+
+import io.swagger.annotations.ApiModel;
+import io.swagger.annotations.ApiModelProperty;
+
+@ApiModel("Uniform Resource Identifier for NiFi Versioned Flows saved to a NiFi Registry")
+public class FlowUri {
--- End diff --
Yes it is {base_url}/buckets/{bucketId}/flows/{flowId}. We could use that in place of this object if preferred. or we could use this object but have a constructor / toString that is capable of going from/to that form.
---
[GitHub] nifi-minifi pull request #118: MINIFI-444 C2 Data Model and REST API
Posted by phrocker <gi...@git.apache.org>.
Github user phrocker commented on a diff in the pull request:
https://github.com/apache/nifi-minifi/pull/118#discussion_r175207453
--- Diff: minifi-c2/minifi-c2-commons/src/main/java/org/apache/nifi/minifi/c2/model/extension/PropertyDescriptor.java ---
@@ -0,0 +1,123 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.minifi.c2.model.extension;
+
+import io.swagger.annotations.ApiModel;
+import io.swagger.annotations.ApiModelProperty;
+
+import java.util.List;
+
+@ApiModel
+public class PropertyDescriptor {
+
+ private String name;
+ private String displayName;
+ private String description;
+ private List<PropertyAllowableValue> allowableValues;
+ private String defaultValue;
+ private Boolean required;
+ private Boolean sensitive;
+ private Boolean supportsEl;
+ private DefinedType identifiesControllerServiceApi;
--- End diff --
Should this be a boolean?
---
[GitHub] nifi-minifi pull request #118: MINIFI-444 C2 Data Model and REST API
Posted by kevdoran <gi...@git.apache.org>.
Github user kevdoran commented on a diff in the pull request:
https://github.com/apache/nifi-minifi/pull/118#discussion_r175519408
--- Diff: minifi-c2/minifi-c2-commons/src/main/java/org/apache/nifi/minifi/c2/model/AgentStatus.java ---
@@ -0,0 +1,50 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.minifi.c2.model;
+
+import io.swagger.annotations.ApiModel;
+import io.swagger.annotations.ApiModelProperty;
+
+import java.util.Map;
+
+/**
+ * Status of the aspects of the agent, including any agent components that are controllable by the C2 server, ie:
+ * - Repositories that can be cleared and their current state
+ */
+@ApiModel
+public class AgentStatus {
+
+ private long uptime;
+ private Map<String, AgentRepositoryStatus> repositories;
--- End diff --
No good reason. You're right, the repos are fixed. Good catch!
- [ ] Replace map for AgentRepositoryStatus with fields for each specific repo.
---
[GitHub] nifi-minifi pull request #118: MINIFI-444 C2 Data Model and REST API
Posted by phrocker <gi...@git.apache.org>.
Github user phrocker commented on a diff in the pull request:
https://github.com/apache/nifi-minifi/pull/118#discussion_r175206480
--- Diff: minifi-c2/minifi-c2-commons/src/main/java/org/apache/nifi/minifi/c2/model/extension/BundleManifest.java ---
@@ -0,0 +1,65 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.minifi.c2.model.extension;
+
+import io.swagger.annotations.ApiModel;
+import io.swagger.annotations.ApiModelProperty;
+
+import java.util.List;
+
+@ApiModel
+public class BundleManifest {
+
--- End diff --
How would one define an extension outside of the confines of a manifest?
---
[GitHub] nifi-minifi pull request #118: MINIFI-444 C2 Data Model and REST API
Posted by apiri <gi...@git.apache.org>.
Github user apiri commented on a diff in the pull request:
https://github.com/apache/nifi-minifi/pull/118#discussion_r176268253
--- Diff: minifi-c2/minifi-c2-commons/src/main/java/org/apache/nifi/minifi/c2/model/Device.java ---
@@ -0,0 +1,55 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.minifi.c2.model;
+
+import io.swagger.annotations.ApiModel;
+import io.swagger.annotations.ApiModelProperty;
+
+@ApiModel
+public class Device extends DeviceInfo {
+
+ private String name;
+ private String firstSeen; // TODO, decide on standard JSON format for timestamps in the C2 REST API
--- End diff --
My inclination would be to just do this in epoch millis UTC and munge it however needed.
---
[GitHub] nifi-minifi pull request #118: MINIFI-444 C2 Data Model and REST API
Posted by phrocker <gi...@git.apache.org>.
Github user phrocker commented on a diff in the pull request:
https://github.com/apache/nifi-minifi/pull/118#discussion_r175204853
--- Diff: minifi-c2/minifi-c2-commons/src/main/java/org/apache/nifi/minifi/c2/model/FlowStatus.java ---
@@ -0,0 +1,51 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.minifi.c2.model;
+
+import io.swagger.annotations.ApiModel;
+import io.swagger.annotations.ApiModelProperty;
+
+import java.util.Map;
+
+/**
+ * Status of the aspects of the flow that are controllable by the C2 server, ie:
+ * - Processors that can be started/stopped and their current state
+ * - Queues that can be cleared and their current state
+ */
+@ApiModel
+public class FlowStatus {
--- End diff --
Do we intent to have an analog for non flow component statuses?
---
[GitHub] nifi-minifi pull request #118: MINIFI-444 C2 Data Model and REST API
Posted by kevdoran <gi...@git.apache.org>.
Github user kevdoran commented on a diff in the pull request:
https://github.com/apache/nifi-minifi/pull/118#discussion_r175813531
--- Diff: minifi-c2/minifi-c2-commons/src/main/java/org/apache/nifi/minifi/c2/model/FlowStatus.java ---
@@ -0,0 +1,51 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.minifi.c2.model;
+
+import io.swagger.annotations.ApiModel;
+import io.swagger.annotations.ApiModelProperty;
+
+import java.util.Map;
+
+/**
+ * Status of the aspects of the flow that are controllable by the C2 server, ie:
+ * - Processors that can be started/stopped and their current state
+ * - Queues that can be cleared and their current state
+ */
+@ApiModel
+public class FlowStatus {
--- End diff --
Not sure I follow. Do you have an example of something that might not into the proposed model?
---
[GitHub] nifi-minifi pull request #118: MINIFI-444 C2 Data Model and REST API
Posted by kevdoran <gi...@git.apache.org>.
Github user kevdoran commented on a diff in the pull request:
https://github.com/apache/nifi-minifi/pull/118#discussion_r176297389
--- Diff: minifi-c2/minifi-c2-commons/src/main/java/org/apache/nifi/minifi/c2/model/extension/ExtensionComponent.java ---
@@ -0,0 +1,85 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.minifi.c2.model.extension;
+
+import io.swagger.annotations.ApiModel;
+import io.swagger.annotations.ApiModelProperty;
+
+import java.util.List;
+import java.util.Set;
+
+/**
+ * A component provided by an extension bundle
+ */
+@ApiModel
+public class ExtensionComponent extends DefinedType {
+
+ // TODO, does arch/binary/compiler metadata need to be added here?
--- End diff --
Ok, so here is what I came up with for now:
An extension component is uniquely identified by (bundle, type), where bundle = (group, artifact, version) and type is a fully qualified class name. Bundle is optional in the case that you have an extension component outside of a bundle (common in the C++ case for now), in which case the fully qualified type must be globally unique.
An extension component implementation is uniquely identified (bundle, type, build) where build = (version, revision, timestamp, target architecture, compiler, compiler flags)
In general, a flow designer should only have to target an extension component in order to author a flow. Whether or not a flow can run on a given agent depends on if the extension component has an available implementation on that agent.
It's still a bit hand-wavy (at least for now), but I think this distinction should work once we have more dynamic authoring and deployment logic that leverages a fully featured device registry and extension registry. For now, we will focus on agent class name compatibility, so we will have to assume that manifests for classes don't change (or at least, don't remove capabilities) and use the intersection of all agent manifest capabilities when designing against a an agent class label.
I need to flesh it out a bit more, but I think it should work. To get a sense of what some of this looks like in code, look at the equals/hashcode methods for DefinedType and ExtensionComponent after I push my next commit.
---
[GitHub] nifi-minifi pull request #118: MINIFI-444 C2 Data Model and REST API
Posted by kevdoran <gi...@git.apache.org>.
Github user kevdoran commented on a diff in the pull request:
https://github.com/apache/nifi-minifi/pull/118#discussion_r175534416
--- Diff: minifi-c2/minifi-c2-commons/src/main/java/org/apache/nifi/minifi/c2/model/extension/Bundle.java ---
@@ -0,0 +1,106 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.minifi.c2.model.extension;
+
+import io.swagger.annotations.ApiModel;
+import io.swagger.annotations.ApiModelProperty;
+
+import java.util.Objects;
+
+@ApiModel
+public class Bundle {
+
+ public static final String DEFAULT_GROUP = "default";
+ public static final String DEFAULT_ARTIFACT = "unknown";
+ public static final String DEFAULT_VERSION = "unversioned";
+
+ private String group;
+ private String artifact;
+ private String version;
+ // TODO md5sum?
+ private BundleManifest bundleManifest;
+
+ public Bundle() {}
+
+ public Bundle(String group, String artifact, String version) {
+ this.group = group;
+ this.artifact = artifact;
+ this.version = version;
+ }
+
+ public static Bundle defaultBundle() {
+ return new Bundle(DEFAULT_GROUP, DEFAULT_ARTIFACT, DEFAULT_VERSION);
+ }
+
+ @ApiModelProperty(
+ value = "The group id of the bundle",
+ notes = "A globally unique group namespace, e.g., org.apache.nifi",
+ required = true)
+ public String getGroup() {
+ return group;
+ }
+
+ public void setGroup(String group) {
+ this.group = group;
+ }
+
+ @ApiModelProperty(
+ value = "The artifact id of the bundle",
+ notes = "Unique within the group",
+ required = true)
+ public String getArtifact() {
+ return artifact;
+ }
+
+ public void setArtifact(String artifact) {
+ this.artifact = artifact;
+ }
+
+ @ApiModelProperty("The version of the bundle artifact")
+ public String getVersion() {
+ return version;
+ }
+
+ public void setVersion(String version) {
+ this.version = version;
+ }
+
+ @ApiModelProperty(value = "The full specification of the bundle contents",
+ notes = "This is optional, as the group, artifact, and version are " +
+ "also enough to reference a bundle in the case the bundle " +
+ "specification has been published to a registry.")
+ public BundleManifest getBundleManifest() {
+ return bundleManifest;
+ }
+
+ public void setBundleManifest(BundleManifest bundleManifest) {
+ this.bundleManifest = bundleManifest;
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (this == o) return true;
--- End diff --
- [ ] Change equals method implementations to utilize commons-lang equals builder.
---
[GitHub] nifi-minifi pull request #118: MINIFI-444 C2 Data Model and REST API
Posted by phrocker <gi...@git.apache.org>.
Github user phrocker commented on a diff in the pull request:
https://github.com/apache/nifi-minifi/pull/118#discussion_r176271235
--- Diff: minifi-c2/minifi-c2-commons/src/main/java/org/apache/nifi/minifi/c2/model/FlowStatus.java ---
@@ -0,0 +1,51 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.minifi.c2.model;
+
+import io.swagger.annotations.ApiModel;
+import io.swagger.annotations.ApiModelProperty;
+
+import java.util.Map;
+
+/**
+ * Status of the aspects of the flow that are controllable by the C2 server, ie:
+ * - Processors that can be started/stopped and their current state
+ * - Queues that can be cleared and their current state
+ */
+@ApiModel
+public class FlowStatus {
--- End diff --
Perhaps it is the naming, but the name FlowComponentStatus implies that it is part of the flow, but there may be components that run that are flow independent. For example think of things that may run a procedure. That component could be tracked in the FlowComponentStatus map components, but perhaps not.
Additionally, not sure I mentioned this already but I see that tracking maps are not concurrent map impls. Should I assume that state is monitored/controlled elsewhere?
---
[GitHub] nifi-minifi pull request #118: MINIFI-444 C2 Data Model and REST API
Posted by apiri <gi...@git.apache.org>.
Github user apiri commented on a diff in the pull request:
https://github.com/apache/nifi-minifi/pull/118#discussion_r176267101
--- Diff: minifi-c2/minifi-c2-commons/src/main/java/org/apache/nifi/minifi/c2/model/extension/BundleManifest.java ---
@@ -0,0 +1,65 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.minifi.c2.model.extension;
+
+import io.swagger.annotations.ApiModel;
+import io.swagger.annotations.ApiModelProperty;
+
+import java.util.List;
+
+@ApiModel
+public class BundleManifest {
+
--- End diff --
I think I like the idea of the direct access to add components. Bundles are nice for the Java side of the house but don't necessarily map well to C++. From the standpoint of the flow designer, the notions of bundles should be minimal but some defined tuple (class, version, etc) would be all that matters as a keyed extension (which has a fuller model as outlined with the other work that has gone on in this PR).
---
[GitHub] nifi-minifi pull request #118: MINIFI-444 C2 Data Model and REST API
Posted by phrocker <gi...@git.apache.org>.
Github user phrocker commented on a diff in the pull request:
https://github.com/apache/nifi-minifi/pull/118#discussion_r175205532
--- Diff: minifi-c2/minifi-c2-commons/src/main/java/org/apache/nifi/minifi/c2/model/SystemInfo.java ---
@@ -0,0 +1,55 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.minifi.c2.model;
+
+import io.swagger.annotations.ApiModel;
+import io.swagger.annotations.ApiModelProperty;
+
+@ApiModel
+public class SystemInfo {
+
+ private String machineArch;
+ private long physicalMem;
+ private int vCores;
+ // TODO timezone / UTC offset?
--- End diff --
configured timezone isn't a bad idea but this can change.
---
[GitHub] nifi-minifi pull request #118: MINIFI-444 C2 Data Model and REST API
Posted by kevdoran <gi...@git.apache.org>.
Github user kevdoran commented on a diff in the pull request:
https://github.com/apache/nifi-minifi/pull/118#discussion_r175529769
--- Diff: minifi-c2/minifi-c2-commons/src/main/java/org/apache/nifi/minifi/c2/model/Device.java ---
@@ -0,0 +1,48 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.minifi.c2.model;
+
+public class Device extends DeviceInfo {
+
+ private String name;
+ private String firstSeen;
--- End diff --
Yes, they should be timestamps. I meant to ask this question, so I'll just ask this here: what should the preferred JSON format of timestamps be for the C2 server REST API? We can go with Epoch time or another standard representation such as [RFC 3339](https://tools.ietf.org/html/rfc3339), which is a stricter version of ISO 8601, e.g., 2018-03-19T17:51:00+00:00. If Epoch time, this field would change to a long of course, but should the units be seconds or milliseconds since epoch?
---
[GitHub] nifi-minifi pull request #118: MINIFI-444 C2 Data Model and REST API
Posted by kevdoran <gi...@git.apache.org>.
Github user kevdoran commented on a diff in the pull request:
https://github.com/apache/nifi-minifi/pull/118#discussion_r175531893
--- Diff: minifi-c2/minifi-c2-commons/src/main/java/org/apache/nifi/minifi/c2/model/OperationRequest.java ---
@@ -0,0 +1,72 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.minifi.c2.model;
+
+import io.swagger.annotations.ApiModel;
+import io.swagger.annotations.ApiModelProperty;
+
+@ApiModel
+public class OperationRequest {
--- End diff --
This is C2 server specific for now. I wanted to try to formulate a basic mechanism that someone could manually submit an operation request to the C2 server using it's rest api, which would then be queued in the C2 Server and eventually sent to the targeted MiNiFi agent over the C2 protocol (e.g., in a heartbeat response). I don't think there is a need to push this into the C2 protocol yet, but I could see how that would be useful eventually, so if you have an idea for how to do that I'd be happy to implement support in the C2 server.
---
[GitHub] nifi-minifi pull request #118: MINIFI-444 C2 Data Model and REST API
Posted by phrocker <gi...@git.apache.org>.
Github user phrocker commented on a diff in the pull request:
https://github.com/apache/nifi-minifi/pull/118#discussion_r175205415
--- Diff: minifi-c2/minifi-c2-commons/src/main/java/org/apache/nifi/minifi/c2/model/OperationRequest.java ---
@@ -0,0 +1,72 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.minifi.c2.model;
+
+import io.swagger.annotations.ApiModel;
+import io.swagger.annotations.ApiModelProperty;
+
+@ApiModel
+public class OperationRequest {
--- End diff --
This looks to be C2 server specific, is that true? If there is the need to push this further to a quorum of agents we will need a source identifier and then what's referred to as a request path.
---
[GitHub] nifi-minifi pull request #118: MINIFI-444 C2 Data Model and REST API
Posted by kevdoran <gi...@git.apache.org>.
Github user kevdoran commented on a diff in the pull request:
https://github.com/apache/nifi-minifi/pull/118#discussion_r175555010
--- Diff: minifi-c2/minifi-c2-web-api/src/main/java/org/apache/nifi/minifi/c2/web/api/C2ProtocolResource.java ---
@@ -0,0 +1,119 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.minifi.c2.web.api;
+
+import io.swagger.annotations.Api;
+import io.swagger.annotations.ApiOperation;
+import io.swagger.annotations.ApiParam;
+import io.swagger.annotations.ApiResponse;
+import io.swagger.annotations.ApiResponses;
+import org.apache.nifi.minifi.c2.model.C2Heartbeat;
+import org.apache.nifi.minifi.c2.model.C2HeartbeatResponse;
+import org.apache.nifi.minifi.c2.model.C2OperationAck;
+import org.springframework.stereotype.Component;
+
+import javax.ws.rs.Consumes;
+import javax.ws.rs.POST;
+import javax.ws.rs.Path;
+import javax.ws.rs.Produces;
+import javax.ws.rs.core.MediaType;
+import javax.ws.rs.core.Response;
+
+@Component
+@Path("/c2-protocol")
--- End diff --
The internal architecture of the C2 Server will be based on these layered:
Web/Network Protocols (REST, MQTT, etc)
Services
Providers (e.g., database or persistence providers)
The bulk of the C2 Protocol implementation logic will live in the service layer, which will expose Java APIs for the Web protocol layers to interact will. The Web protocol implementations will provide very minimal/lightweight adapters, such as deserializing whatever wire-format they use into the in-memory objects that are required for calling the service layer API. External clients, such as MiNiFi agents, will only interact with the web protocols we choose to define, expose, and support. Everything else at the service level and below is internal to the C2 server implementation.
---
[GitHub] nifi-minifi pull request #118: MINIFI-444 C2 Data Model and REST API
Posted by phrocker <gi...@git.apache.org>.
Github user phrocker commented on a diff in the pull request:
https://github.com/apache/nifi-minifi/pull/118#discussion_r175202507
--- Diff: minifi-c2/minifi-c2-commons/src/main/java/org/apache/nifi/minifi/c2/model/AgentStatus.java ---
@@ -0,0 +1,50 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.minifi.c2.model;
+
+import io.swagger.annotations.ApiModel;
+import io.swagger.annotations.ApiModelProperty;
+
+import java.util.Map;
+
+/**
+ * Status of the aspects of the agent, including any agent components that are controllable by the C2 server, ie:
+ * - Repositories that can be cleared and their current state
+ */
+@ApiModel
+public class AgentStatus {
+
+ private long uptime;
+ private Map<String, AgentRepositoryStatus> repositories;
+
+ @ApiModelProperty("The number of milliseconds since the agent started.")
+ public long getUptime() {
+ return uptime;
+ }
+
+ public void setUptime(long uptime) {
+ this.uptime = uptime;
+ }
+
+ @ApiModelProperty("Status and metrics for each repository")
+ public Map<String, AgentRepositoryStatus> getRepositories() {
+ return repositories;
--- End diff --
immutable map?
---
[GitHub] nifi-minifi pull request #118: MINIFI-444 C2 Data Model and REST API
Posted by kevdoran <gi...@git.apache.org>.
Github user kevdoran commented on a diff in the pull request:
https://github.com/apache/nifi-minifi/pull/118#discussion_r175459177
--- Diff: minifi-c2/minifi-c2-commons/src/main/java/org/apache/nifi/minifi/c2/model/AgentInfo.java ---
@@ -0,0 +1,81 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.minifi.c2.model;
+
+import io.swagger.annotations.ApiModel;
+import io.swagger.annotations.ApiModelProperty;
+
+import java.util.Collection;
+
+@ApiModel
+public class AgentInfo {
+
+ private String identifier;
+ // TODO, do we also need identity. e.g., cert DN
+ private String agentClass;
+ private Collection<String> tags;
+ private AgentManifest agentManifest;
+ private AgentStatus status;
+ // private Map<String, String> configProperties; // TODO we should add this information eventually, but we need to handle the best way to handle sharing sensitive properties.
+
+ @ApiModelProperty(
+ value = "A unique identifier for the Agent",
+ notes = "Usually set when the agent is provisioned and deployed")
+ public String getIdentifier() {
+ return identifier;
+ }
+
+ public void setIdentifier(String identifier) {
+ this.identifier = identifier;
+ }
+
+ @ApiModelProperty(
+ value = "The class or category label of the agent, e.g., 'sensor-collector'",
+ notes = "Usually set when the agent is provisioned and deployed")
+ public String getAgentClass() {
+ return agentClass;
+ }
+
+ public void setAgentClass(String agentClass) {
+ this.agentClass = agentClass;
+ }
+
+ @ApiModelProperty("Arbitrary tags that describe the agent in addition to the class")
+ public Collection<String> getTags() {
+ return tags;
--- End diff --
Good idea, will do. I'll review all getters to see which ones should return immutable collections.
---
[GitHub] nifi-minifi pull request #118: MINIFI-444 C2 Data Model and REST API
Posted by kevdoran <gi...@git.apache.org>.
Github user kevdoran commented on a diff in the pull request:
https://github.com/apache/nifi-minifi/pull/118#discussion_r175548117
--- Diff: minifi-c2/minifi-c2-commons/src/main/java/org/apache/nifi/minifi/c2/model/extension/ExtensionComponent.java ---
@@ -0,0 +1,85 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.minifi.c2.model.extension;
+
+import io.swagger.annotations.ApiModel;
+import io.swagger.annotations.ApiModelProperty;
+
+import java.util.List;
+import java.util.Set;
+
+/**
+ * A component provided by an extension bundle
+ */
+@ApiModel
+public class ExtensionComponent extends DefinedType {
+
+ // TODO, does arch/binary/compiler metadata need to be added here?
--- End diff --
Noted. I'll add it. What do you think are the required/optional fields?
---
[GitHub] nifi-minifi pull request #118: MINIFI-444 C2 Data Model and REST API
Posted by kevdoran <gi...@git.apache.org>.
Github user kevdoran commented on a diff in the pull request:
https://github.com/apache/nifi-minifi/pull/118#discussion_r175511730
--- Diff: minifi-c2/minifi-c2-commons/src/main/java/org/apache/nifi/minifi/c2/model/AgentRepositoryStatus.java ---
@@ -0,0 +1,63 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.minifi.c2.model;
+
+import io.swagger.annotations.ApiModel;
+import io.swagger.annotations.ApiModelProperty;
+
+@ApiModel
+public class AgentRepositoryStatus {
+
+ private Long size;
--- End diff --
Excellent point. Will review this and update
- [ ] For numeric primitive types, should document how it will be interpreted so that serialized values are valid and interpreted correctly when deserialized by the Java C2 (e.g., signed vs unsigned)
---
[GitHub] nifi-minifi pull request #118: MINIFI-444 C2 Data Model and REST API
Posted by kevdoran <gi...@git.apache.org>.
Github user kevdoran commented on a diff in the pull request:
https://github.com/apache/nifi-minifi/pull/118#discussion_r175459214
--- Diff: minifi-c2/minifi-c2-commons/src/main/java/org/apache/nifi/minifi/c2/model/AgentStatus.java ---
@@ -0,0 +1,50 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.minifi.c2.model;
+
+import io.swagger.annotations.ApiModel;
+import io.swagger.annotations.ApiModelProperty;
+
+import java.util.Map;
+
+/**
+ * Status of the aspects of the agent, including any agent components that are controllable by the C2 server, ie:
+ * - Repositories that can be cleared and their current state
+ */
+@ApiModel
+public class AgentStatus {
+
+ private long uptime;
+ private Map<String, AgentRepositoryStatus> repositories;
+
+ @ApiModelProperty("The number of milliseconds since the agent started.")
+ public long getUptime() {
+ return uptime;
+ }
+
+ public void setUptime(long uptime) {
+ this.uptime = uptime;
+ }
+
+ @ApiModelProperty("Status and metrics for each repository")
+ public Map<String, AgentRepositoryStatus> getRepositories() {
+ return repositories;
--- End diff --
Yep will do.
---
[GitHub] nifi-minifi pull request #118: MINIFI-444 C2 Data Model and REST API
Posted by phrocker <gi...@git.apache.org>.
Github user phrocker commented on a diff in the pull request:
https://github.com/apache/nifi-minifi/pull/118#discussion_r175208409
--- Diff: minifi-c2/minifi-c2-commons/src/main/java/org/apache/nifi/minifi/c2/model/extension/DefinedType.java ---
@@ -0,0 +1,69 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.minifi.c2.model.extension;
+
+import io.swagger.annotations.ApiModel;
+import io.swagger.annotations.ApiModelProperty;
+
+/**
+ * A reference to a defined type identified by bundle and fully qualified class type identifiers
+ */
+@ApiModel
+public class DefinedType {
+
+ private String group;
+ private String artifact;
+ private String version;
+ private String type;
+
+ @ApiModelProperty("The group name of the bundle that provides the referenced type.")
+ public String getGroup() {
+ return group;
+ }
+
+ public void setGroup(String group) {
+ this.group = group;
+ }
+
+ @ApiModelProperty("The artifact name of the bundle that provides the referenced type.")
+ public String getArtifact() {
+ return artifact;
+ }
+
+ public void setArtifact(String artifact) {
+ this.artifact = artifact;
+ }
+
+ @ApiModelProperty("The version of the bundle that provides the referenced type.")
+ public String getVersion() {
+ return version;
+ }
+
+ public void setVersion(String version) {
+ this.version = version;
+ }
+
+ @ApiModelProperty(
+ value = "The fully-qualified class type",
+ required = true,
+ notes = "For example, 'org.apache.nifi.GetFile' or 'org::apache:nifi::minifi::GetFile'")
--- End diff --
How will this be used to define an API?
---
[GitHub] nifi-minifi pull request #118: MINIFI-444 C2 Data Model and REST API
Posted by phrocker <gi...@git.apache.org>.
Github user phrocker commented on a diff in the pull request:
https://github.com/apache/nifi-minifi/pull/118#discussion_r175202125
--- Diff: minifi-c2/minifi-c2-commons/src/main/java/org/apache/nifi/minifi/c2/model/AgentStatus.java ---
@@ -0,0 +1,50 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.minifi.c2.model;
+
+import io.swagger.annotations.ApiModel;
+import io.swagger.annotations.ApiModelProperty;
+
+import java.util.Map;
+
+/**
+ * Status of the aspects of the agent, including any agent components that are controllable by the C2 server, ie:
+ * - Repositories that can be cleared and their current state
+ */
+@ApiModel
+public class AgentStatus {
+
+ private long uptime;
+ private Map<String, AgentRepositoryStatus> repositories;
--- End diff --
why a map versus calling out the specific repos?
---
[GitHub] nifi-minifi pull request #118: MINIFI-444 C2 Data Model and REST API
Posted by phrocker <gi...@git.apache.org>.
Github user phrocker commented on a diff in the pull request:
https://github.com/apache/nifi-minifi/pull/118#discussion_r175204137
--- Diff: minifi-c2/minifi-c2-commons/src/main/java/org/apache/nifi/minifi/c2/model/Device.java ---
@@ -0,0 +1,48 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.minifi.c2.model;
+
+public class Device extends DeviceInfo {
+
+ private String name;
+ private String firstSeen;
--- End diff --
what do first seen and last seen mean in this context? I assume they are timestamps?
---
[GitHub] nifi-minifi pull request #118: MINIFI-444 C2 Data Model and REST API
Posted by kevdoran <gi...@git.apache.org>.
Github user kevdoran commented on a diff in the pull request:
https://github.com/apache/nifi-minifi/pull/118#discussion_r175844957
--- Diff: minifi-c2/minifi-c2-commons/src/main/java/org/apache/nifi/minifi/c2/model/extension/BundleManifest.java ---
@@ -0,0 +1,65 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.minifi.c2.model.extension;
+
+import io.swagger.annotations.ApiModel;
+import io.swagger.annotations.ApiModelProperty;
+
+import java.util.List;
+
+@ApiModel
+public class BundleManifest {
+
--- End diff --
::sigh:: I typed a long response that I think got lost before I hit the button.
The gist of it is -> I need to think about this. Currently the data model requires defining an extension in the context of a bundle. If there is no actual bundle, you could use a default bundle (I even included a constructor `defaultBundle()` to create one) that indicates there is no actual bundle used.
Alternatively, we could add ExtensionComponents directly to AgentManifest so you could include them without creating fake bundles to act as containers. Thoughts?
---