[GitHub] [nifi-minifi-cpp] fgerlits commented on a diff in pull request #1349: MINIFICPP-1843 Implement PostElasticsearch

[GitBox <gi...@apache.org>]

fgerlits commented on code in PR #1349:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1349#discussion_r907494696


##########
bstrp_functions.sh:
##########
@@ -386,6 +386,7 @@ show_supported_features() {
   echo "AB. Kubernetes Support .........$(print_feature_status KUBERNETES_ENABLED)"
   echo "AC. Google Cloud Support .......$(print_feature_status GCP_ENABLED)"
   echo "AD. ProcFs Support .............$(print_feature_status PROCFS_ENABLED)"
+  echo "AE. Elasticsearch Support ......$(print_feature_status ELASTIC_ENABLED)"

Review Comment:
   lines 412 and 467 need to be updated



##########
docker/test/integration/features/elasticsearch.feature:
##########
@@ -0,0 +1,93 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+@no-ci  # Elasticsearch container requires more RAM than what the CI environment has
+Feature: Managing documents on Elasticsearch with PostElasticsearch
+
+  Background:
+    Given the content of "/tmp/output" is monitored
+
+  Scenario: MiNiFi instance indexes a document on Elasticsearch using Basic Authentication
+    Given an Elasticsearch server is set up and running
+    And a GetFile processor with the "Input Directory" property set to "/tmp/input"
+    And a file with the content "{ "field1" : "value1" }" is present in "/tmp/input"
+    And a PostElasticsearch processor
+    And the "Index" property of the PostElasticsearch processor is set to "my_index"
+    And the "Identifier" property of the PostElasticsearch processor is set to "my_id"
+    And the "Action" property of the PostElasticsearch processor is set to "index"
+    And a SSL context service is set up for PostElasticsearch and Elasticsearch
+    And an ElasticsearchCredentialsService is set up for PostElasticsearch with Basic Authentication
+    And a PutFile processor with the "Directory" property set to "/tmp/output"
+    And the "success" relationship of the GetFile processor is connected to the PostElasticsearch
+    And the "success" relationship of the PostElasticsearch processor is connected to the PutFile
+
+    When both instances start up
+    Then a flowfile with the content "{ "field1" : "value1" }" is placed in the monitored directory in less than 20 seconds
+    And Elasticsearch has a document with "my_id" in "my_index" that has "value1" set in "field1"
+
+  Scenario: MiNiFi instance creates a document on Elasticsearch using API Key authentication

Review Comment:
   Are these two scenarios identical except for `Basic Authentication` / `ApiKey`?  If so, then they could be merged into a single Scenario Outline with two Examples.



##########
docker/DockerVerify.sh:
##########
@@ -67,7 +67,7 @@ TEST_DIRECTORY="${docker_dir}/test/integration"
 export TEST_DIRECTORY
 
 # Add --no-logcapture to see logs interleaved with the test output
-BEHAVE_OPTS=(-f pretty --logging-level INFO --logging-clear-handlers)
+BEHAVE_OPTS=(-f pretty --logging-level INFO --logging-clear-handlers --tags ~@no-ci)

Review Comment:
   Can we have a command-line argument or environment variable to control this?  I would like `make docker-verify` to include these tests on my computer, but skip them in CI jobs.



##########
docker/test/integration/steps/steps.py:
##########
@@ -410,6 +411,62 @@ def step_impl(context):
     context.test.acquire_container("fake-gcs-server", "fake-gcs-server")
 
 
+# elasticsearch
+@given('an Elasticsearch server is set up and running')
+@given('an Elasticsearch server is set up and a single document is present with "preloaded_id" in "my_index"')
+@given('an Elasticsearch server is set up and a single document is present with "preloaded_id" in "my_index" with "value1" in "field1"')
+def step_impl(context):
+    context.test.start_elasticsearch()
+    context.test.create_doc_elasticsearch("elasticsearch", "my_index", "preloaded_id")
+
+
+# opensearch
+@given('an Opensearch server is set up and running')
+@given('an Opensearch server is set up and a single document is present with "preloaded_id" in "my_index"')
+@given('an Opensearch server is set up and a single document is present with "preloaded_id" in "my_index" with "value1" in "field1"')
+def step_impl(context):
+    context.test.start_opensearch()
+    context.test.add_elastic_user_to_opensearch("opensearch")
+    context.test.create_doc_elasticsearch("opensearch", "my_index", "preloaded_id")
+
+
+@given(u'a SSL context service is set up for PostElasticsearch and Elasticsearch')
+def step_impl(context):
+    minifi_crt_file = '/tmp/resources/elasticsearch/minifi_client.crt'
+    minifi_key_file = '/tmp/resources/elasticsearch/minifi_client.key'
+    root_ca_crt_file = '/tmp/resources/elasticsearch/root_ca.crt'
+    ssl_context_service = SSLContextService(cert=minifi_crt_file, ca_cert=root_ca_crt_file, key=minifi_key_file)
+    put_elasticsearch_json = context.test.get_node_by_name("PostElasticsearch")
+    put_elasticsearch_json.controller_services.append(ssl_context_service)
+    put_elasticsearch_json.set_property("SSL Context Service", ssl_context_service.name)

Review Comment:
   very minor, but why are the variables containing the `PostElasticsearch` processor called `put_elastic...` instead of `post_elastic...` everywhere in this file?



##########
docker/test/integration/features/opensearch.feature:
##########
@@ -0,0 +1,97 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+@no-ci  # Opensearch container requires more RAM than what the CI environment has
+Feature: PostElasticsearch works on Opensearch (Opensearch doesnt support API Keys)
+
+  Background:
+    Given the content of "/tmp/output" is monitored
+
+  Scenario: MiNiFi instance indexes a document on Opensearch using Basic Authentication
+    Given an Opensearch server is set up and running
+    And a GetFile processor with the "Input Directory" property set to "/tmp/input"
+    And a file with the content "{ "field1" : "value1" }" is present in "/tmp/input"
+    And a PostElasticsearch processor
+    And the "Hosts" property of the PostElasticsearch processor is set to "https://opensearch:9200"
+    And the "Index" property of the PostElasticsearch processor is set to "my_index"
+    And the "Identifier" property of the PostElasticsearch processor is set to "my_id"
+    And the "Action" property of the PostElasticsearch processor is set to "index"
+    And a SSL context service is set up for PostElasticsearch and Opensearch
+    And an ElasticsearchCredentialsService is set up for PostElasticsearch with Basic Authentication
+    And a PutFile processor with the "Directory" property set to "/tmp/output"
+    And the "success" relationship of the GetFile processor is connected to the PostElasticsearch
+    And the "success" relationship of the PostElasticsearch processor is connected to the PutFile
+
+    When both instances start up
+    Then a flowfile with the content "{ "field1" : "value1" }" is placed in the monitored directory in less than 20 seconds
+    And Opensearch has a document with "my_id" in "my_index" that has "value1" set in "field1"
+
+  Scenario: MiNiFi instance creates a document on Opensearch using Basic Authentication

Review Comment:
   Here, too, it looks like the only difference between the first two Scenarios is the value of the Action property, so they could be merged into a single Scenario Outline.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org