You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@nifi.apache.org by GitBox <gi...@apache.org> on 2022/11/04 22:44:27 UTC
[GitHub] [nifi] MikeThomsen commented on a diff in pull request #6544: NIFI-9398 add verification to ElasticSearchClientService (with integration tests) and Elasticsearch REST API processors
MikeThomsen commented on code in PR #6544:
URL: https://github.com/apache/nifi/pull/6544#discussion_r1012410138
##########
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-client-service/src/main/java/org/apache/nifi/elasticsearch/ElasticSearchClientServiceImpl.java:
##########
@@ -416,6 +495,26 @@ public void refresh(final String index, final Map<String, String> requestParamet
}
}
+ @Override
+ public boolean exists(final String index, final Map<String, String> requestParameters) {
+ try {
+ final Response response = performRequest("HEAD", "/" + index, requestParameters, null);
Review Comment:
Should probably check to make sure that `index` has no leading or trailing `/`
##########
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-test-utils/pom.xml:
##########
@@ -0,0 +1,70 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!-- Licensed to the Apache Software Foundation (ASF) under one or more contributor
+license agreements. See the NOTICE file distributed with this work for additional
+information regarding copyright ownership. The ASF licenses this file to
+You under the Apache License, Version 2.0 (the "License"); you may not use
+this file except in compliance with the License. You may obtain a copy of
+the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required
+by applicable law or agreed to in writing, software distributed under the
+License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS
+OF ANY KIND, either express or implied. See the License for the specific
+language governing permissions and limitations under the License. -->
+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <modelVersion>4.0.0</modelVersion>
+ <parent>
+ <artifactId>nifi-elasticsearch-bundle</artifactId>
+ <groupId>org.apache.nifi</groupId>
+ <version>1.19.0-SNAPSHOT</version>
+ </parent>
+
+ <artifactId>nifi-elasticsearch-test-utils</artifactId>
+ <packaging>jar</packaging>
+
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-mock</artifactId>
+ <version>1.19.0-SNAPSHOT</version>
+ <scope>compile</scope>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.httpcomponents</groupId>
+ <artifactId>httpclient</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>com.fasterxml.jackson.core</groupId>
+ <artifactId>jackson-databind</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.elasticsearch.client</groupId>
+ <artifactId>elasticsearch-rest-client</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.junit.jupiter</groupId>
+ <artifactId>junit-jupiter-api</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>org.testcontainers</groupId>
+ <artifactId>testcontainers</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>org.testcontainers</groupId>
+ <artifactId>elasticsearch</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>org.testcontainers</groupId>
Review Comment:
Should be made into a compile time dependency w/ a version number.
##########
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-restapi-processors/src/main/java/org/apache/nifi/processors/elasticsearch/ElasticsearchRestProcessor.java:
##########
@@ -117,13 +123,66 @@ default String getQuery(final FlowFile input, final ProcessContext context, fina
}
default Map<String, String> getUrlQueryParameters(final ProcessContext context, final FlowFile flowFile) {
+ return getUrlQueryParameters(context, flowFile != null ? flowFile.getAttributes() : null);
+ }
+
+ default Map<String, String> getUrlQueryParameters(final ProcessContext context, final Map<String, String> attributes) {
return context.getProperties().entrySet().stream()
// filter non-null dynamic properties
.filter(e -> e.getKey().isDynamic() && e.getValue() != null)
// convert to Map of URL parameter keys and values
.collect(Collectors.toMap(
- e -> e.getKey().getName(),
- e -> context.getProperty(e.getKey()).evaluateAttributeExpressions(flowFile).getValue()
+ e -> e.getKey().getName(),
+ e -> context.getProperty(e.getKey()).evaluateAttributeExpressions(attributes).getValue()
));
}
+
+ String VERIFICATION_STEP_INDEX_EXISTS = "Elasticsearch Index Exists";
+
+ @Override
+ default List<ConfigVerificationResult> verify(final ProcessContext context, final ComponentLog verificationLogger, final Map<String, String> attributes) {
Review Comment:
I have reservations about this approach for verification because it's very possible users could end up using EL to specify a dynamic index name based on an Elastic template. How do you see that scenario playing out with verification since the index may not exist yet for a good reason? Also, this could potentially block users who are iterating quickly and relying on Elastic to autogenerate the schema during their initial setup before formalizing one.
##########
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-restapi-processors/src/main/java/org/apache/nifi/processors/elasticsearch/GetElasticsearch.java:
##########
@@ -213,7 +273,7 @@ private void handleElasticsearchException(final ElasticsearchException ese, Flow
getLogger().error(msg, ese);
if (input != null) {
session.penalize(input);
- input = session.putAttribute(input, "elasticsearch.get.error", ese.getMessage());
+ session.putAttribute(input, "elasticsearch.get.error", ese.getMessage());
Review Comment:
Why did you drop the assignment?
##########
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-restapi-processors/src/main/java/org/apache/nifi/processors/elasticsearch/ElasticsearchRestProcessor.java:
##########
@@ -117,13 +123,66 @@ default String getQuery(final FlowFile input, final ProcessContext context, fina
}
default Map<String, String> getUrlQueryParameters(final ProcessContext context, final FlowFile flowFile) {
+ return getUrlQueryParameters(context, flowFile != null ? flowFile.getAttributes() : null);
+ }
+
+ default Map<String, String> getUrlQueryParameters(final ProcessContext context, final Map<String, String> attributes) {
return context.getProperties().entrySet().stream()
// filter non-null dynamic properties
.filter(e -> e.getKey().isDynamic() && e.getValue() != null)
// convert to Map of URL parameter keys and values
.collect(Collectors.toMap(
- e -> e.getKey().getName(),
- e -> context.getProperty(e.getKey()).evaluateAttributeExpressions(flowFile).getValue()
+ e -> e.getKey().getName(),
+ e -> context.getProperty(e.getKey()).evaluateAttributeExpressions(attributes).getValue()
));
}
+
+ String VERIFICATION_STEP_INDEX_EXISTS = "Elasticsearch Index Exists";
+
+ @Override
+ default List<ConfigVerificationResult> verify(final ProcessContext context, final ComponentLog verificationLogger, final Map<String, String> attributes) {
+ final List<ConfigVerificationResult> results = new ArrayList<>();
+ final ConfigVerificationResult.Builder indexExistsResult = new ConfigVerificationResult.Builder()
+ .verificationStepName(VERIFICATION_STEP_INDEX_EXISTS);
+
+ ElasticSearchClientService verifyClientService = null;
+ String index = null;
+ boolean indexExists = false;
+ if (context.getProperty(CLIENT_SERVICE).isSet()) {
+ verifyClientService = context.getProperty(CLIENT_SERVICE).asControllerService(ElasticSearchClientService.class);
+ if (context.getProperty(INDEX).isSet()) {
+ index = context.getProperty(INDEX).evaluateAttributeExpressions(attributes).getValue();
+ try {
+ if (verifyClientService.exists(index, getUrlQueryParameters(context, attributes))) {
+ indexExistsResult.outcome(ConfigVerificationResult.Outcome.SUCCESSFUL)
+ .explanation(String.format("Index [%s] exists", index));
+ indexExists = true;
+ } else {
+ indexExistsResult.outcome(ConfigVerificationResult.Outcome.SUCCESSFUL)
Review Comment:
Related to the point above, I guess the goal here is to just flex the REST api and make sure you can get an expected response (even if it's an index doesn't exist exception)?
##########
nifi-nar-bundles/nifi-elasticsearch-bundle/nifi-elasticsearch-test-utils/pom.xml:
##########
@@ -0,0 +1,70 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!-- Licensed to the Apache Software Foundation (ASF) under one or more contributor
+license agreements. See the NOTICE file distributed with this work for additional
+information regarding copyright ownership. The ASF licenses this file to
+You under the Apache License, Version 2.0 (the "License"); you may not use
+this file except in compliance with the License. You may obtain a copy of
+the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required
+by applicable law or agreed to in writing, software distributed under the
+License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS
+OF ANY KIND, either express or implied. See the License for the specific
+language governing permissions and limitations under the License. -->
+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
+ <modelVersion>4.0.0</modelVersion>
+ <parent>
+ <artifactId>nifi-elasticsearch-bundle</artifactId>
+ <groupId>org.apache.nifi</groupId>
+ <version>1.19.0-SNAPSHOT</version>
+ </parent>
+
+ <artifactId>nifi-elasticsearch-test-utils</artifactId>
+ <packaging>jar</packaging>
+
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.nifi</groupId>
+ <artifactId>nifi-mock</artifactId>
+ <version>1.19.0-SNAPSHOT</version>
+ <scope>compile</scope>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.httpcomponents</groupId>
+ <artifactId>httpclient</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>com.fasterxml.jackson.core</groupId>
+ <artifactId>jackson-databind</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.elasticsearch.client</groupId>
+ <artifactId>elasticsearch-rest-client</artifactId>
+ </dependency>
+
+ <dependency>
+ <groupId>org.junit.jupiter</groupId>
+ <artifactId>junit-jupiter-api</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>org.testcontainers</groupId>
+ <artifactId>testcontainers</artifactId>
+ </dependency>
+ <dependency>
+ <groupId>org.testcontainers</groupId>
+ <artifactId>elasticsearch</artifactId>
Review Comment:
Should be made into a compile time dependency w/ a version number.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org